SSL errors when using Nessus and OpenSSL 0.9.8

[Javier Fernandez-Sanguino]

It's been talked about in the list in the past, but I've been hitten by 
this bug when transitioning (in Debian) from 0.9.7 to 0.9.8 and wanted 
to provide a summary here.

A user first reported an issue with the Debian Nessus 2.2.5 packages 
here http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=343487

Even though there are some issues with the Debian packages (the binary 
gets linked to both OpenSSL versions due to the transition) the end 
issue is the same that has been reported here with Mac OS X and other 
platforms. When the client tries to connect to the server it aborts and 
prints these errors:

SSL_connect: error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert 
bad record mac
   nessus : SSL error

[ BTW, the same error shows up if the client cannot connect to the 
server due to the tcp-wrappers configuration, but then you will see this 
in the server logs:
  Connection from 127.0.0.1 rejected by libwrap
That is easily fixed chaning your tcpwrapper's hosts.allow config ]

When you see the SSL error above and *don't* see any error in the server 
logs then you've been bitten by this bug:
"libssl0.9.8: bad record mac because of wrong 
SSL_OP_TLS_BLOCK_PADDING_BUG handling"
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=338006
Which is also open in the OpenSSL tracking system:
http://www.aet.tu-cottbus.de/rt2/Ticket/Display.html?id=1204

There seems to be no workaround other than recompiling against 0.9.7. If 
you are using Debian, and since unstable/testing in in transition to 
OpenSSL 0.9.8, you need to use the libssl-dev packages from *stable* [1] 
in order to get it compiled against 0.9.7 and work.

Another option (for both Debian and other OSes) is to take the 0.9.7 
sources, build them, and then build the Nessus sources. Make sure that 
you removed 0.9.8 completely from your system (review shared libraries 
directories just in case). If you don't want to mess up your system, do 
this in a chroot environment (in Debian it's easy to make a development 
chroot environment with debootrstrap, don't know about others).

Summarising: if you try to use Nessus (either compiled from sources or 
binary packages) with OpenSSL 0.9.8 you will get bitten by this bug, you 
will need to recompile using the 0.9.7 OpenSSL version to get it working 
until this bug is fixed.

It seems many daemon servers using client-side certificates are being 
hit by this bug (there are reports associated with Apache, with 
Cyrus-Imapd, and others) so, hopefully, the bug will be solved promptly.

Hope this helps

Javier

[1] http://packages.debian.org/stable/libdevel/libssl-dev