Forum OpenACS Q&A: https is down
Here's the error I get while trying to connect:
[17/Sep/2003:09:30:42][17509.5126][-conn2-] Error: nsopenssl: error -1/1 during SSL handshake
I'm on Debian stable:
# apt-cache showpkg openssl
Any suggestions? I hope this isn't related to the recent OpenSSH vulnerability...
I hope to get this up soon... my company is understandibly vexed. :(
I did not upgrade my openssl libraries, and I guess we can't do so until a new version of nsopenssl comes out. Scott G has a workaround for the new changes, according to what I heard on IRC this morning.
According to scottg, the new openssl libs have changed the way they report if they are compiled with threads or not. Removing the ifdefs that check for threads is a temporary workaround. (Thanks to CR Oldham for this info.)
I'm trying to recompile nsopenssl now and hope that fixes the problem.
We started to see the same problems recently on our servers (RedHat) ...
Indeed just restarting nsd seems to make the issue go away but we haven't found yet the reason. When these errors occur, it also matches with some strange logs in the access_log file.
Using nsopenssl-2.1a here ... and:
Dang. Recompiled nsopenssl, and it was stable for about 6 hours before crashing again. I guess I could always revert to older openssl libs, but am certainly reluctant to do so. 😟 If anyone has any ideas, please let me know. My configuration is very similar to Jade's.
For the record, I've included the message I sent to the AOLserver list below.
I'm having trouble resolving an openssl issue. It seems to be related to a recent system update of openssl. At first I thought recompiling nsopenssl against the new openssl libraries would fix the issue, however I am still seeing the problem. The system is: Debian 3.0 OpenSSL 0.9.6c-2 nsopenssl 2.1 (2.1a requires OpenSSL 0.9.7x and higher) AOLserver 3.4.2-1 Once AOLserver starts, nsopenssl will usually remain stable for a couple of hours. It inevitably crashes however, giving the client this error: "ssldomain.com received a message with incorrect Message Authentication Code. If the error occurs frequently, contact the website administrator." The server log reports: "Error: nsopenssl: error -1/1 during SSL handshake" Previous to the update to openssl, nsopenssl had been rock solid. Any suggestions? I hope I'm just stupid and am missing something obvious. Thanks in advance, and thanks already to Scott Goodwin for his previous help.
That's a good question Andrew P. The previous version (in woody) is also 0.9.6c-2, but does not have the patch for RSA Blinding. And checking that, I found this tidbit:
Unfortunately, RSA blinding is not thread-safe and will cause failures for programs that use threads and OpenSSL such as stunnel. However, since the proposed fix would change the binary interface (ABI), programs that are dynamically linked against OpenSSL won't run anymore. This is a dilemma we can't solve.
You will have to decide whether you want the security update which is not thread-safe and recompile all applications that apparently fail after the upgrade, or fetch the additional source packages at the end of this advisory, recompile it and use a thread-safe OpenSSL library again, but also recompile all applications that make use of it (such as apache-ssl, mod_ssl, ssh etc.).
However, since only very few packages use threads and link against the OpenSSL library most users will be able to use packages from this update without any problems.
So, there seem to be a couple of options that will (hopefully) solve the problem. Thanks for setting me on the right path, Andrew P.
I need to double check that again but they are telling me they haven't done it and at least for sure not within a few month of the time when we started to get these errors.
So perhaps it's that some folks are upgrading OpenSSL at the same time they upgraded OpenSSH, and then breaking OpenSSL thread-safety due to silly header file and build changes in OpenSSL?
[grumble grumble] My fairly un-informed take on this is that some of these rushed in security patches are not all that well thought out. Breaking thread-safety by default in a security patch to a formerly thread-safe library strikes me as really obnoxious.
So I'm damned if I do and damned if I don't?
I have to compile it from source. That's easy, but it sucks because I lose the easy Debian-style upgrades..
Strange ... the OpenSSL version that we are running was released: https://rhn.redhat.com/errata/RHSA-2003-101.html so April or something and we started to see problems only 4 weeks ago ... but that is strangely familiar to the time when OpenSSH was upgraded ... I will investigate further ... Have to run now, so more tomorrow (hopefully).
I'm running RH8. When I tried to upgrade openssh, it required a newer version of openssl. So if RPM's are used in the upgrade, openssl will be upgraded to get the new openssh. Otherwise, you can get the patch for openssh and recompile that. Here is the openssh patch.
Considering how many packages depend on openssl, this is probably the easiest way to upgrade.
#define RSA_FLAG_THREAD_SAFE 0x10" only ever appears in exactly one place:
but that's the only occurrence of the string "RSA_FLAG_THREAD_SAFE" anywhere in the sources. I thought perhaps the build process might do something funky to use that, so I did:$ find . -type f -print | xargs grep RSA_FLAG_THREAD_SAFE ./crypto/rsa/rsa.h:#define RSA_FLAG_THREAD_SAFE 0x10
and checked again. Nope, no difference. Whatever that RSA_FLAG_THREAD_SAFE is supposed to be for, it appears to never be used. (On this Red Hat 7.3 system anyway.)$ ./config threads shared $ make
The openssl-0.9.7b CHANGES file has this to say:
Changes between 0.9.7a and 0.9.7b [10 Apr 2003] *) Turn on RSA blinding by default in the default implementation to avoid a timing attack. Applications that don't want it can call RSA_blinding_off() or use the new flag RSA_FLAG_NO_BLINDING. They would be ill-advised to do so in most cases. [Ben Laurie, Steve Henson, Geoff Thorpe, Bodo Moeller] *) Change RSA blinding code so that it works when the PRNG is not seeded (in this case, the secret RSA exponent is abused as an unpredictable seed -- if it is not unpredictable, there is no point in blinding anyway). Make RSA blinding thread-safe by remembering the creator's thread ID in rsa->blinding and having all other threads use local one-time blinding factors (this requires more computation than sharing rsa->blinding, but avoids excessive locking; and if an RSA object is not shared between threads, blinding will still be very fast). [Bodo Moeller]
If that's true, and it works, then no one should be seeing any thread safety problems. Thoughts?
I've had a nsopenssl 2.1a dynamically linked against openssl-0.9.7b and running on a Dev server with no problems for a week or so now, but it hardly gets any load at all so that doesn't prove much.
Bruno didn't say exactly what version he was using, but the Red Hat link he gave seems to show that the latest updated packages for all consumer Red Hat distributions prior to Red Hat 9.0 are using an OpenSSL older than 0.9.7x. Which is suspicious, as it's possible that RH backported the RSA blinding security patch but not the thread-safety fix.
So the fix should be simple, just upgrade to 0.9.7b. Note that Scott Goodwin says that nsopenssl 2.1a requires OpenSSL 0.9.7x or later anyway.
You can't do it using apt-get, because 0.9.7 is on unstable and testing, and is compiled against gcc 3.x, etc..
So Debian users either have to compile it themselves, and lose the benefits of Debian package management, or continually restart their servers.
But in this case, the Right Way would be to build your own OpenSSL 0.9.7b binary package for Debian Stable. Probably using the same source package that's in testing now, you just need to rebuild with the older Stable tool chain.
Personally, I haven't gotten around to actually doing that sort of thing yet, but my understanding is it's not bad. At least, several people here do that sort of thing, and I've read (on the Beowulf mailing list, other places) that it's not unusual for admins of large clusters to choose to rebuild most of their binary packages from source packages as a matter of course.
For Debian though, I bet someone has already done it, and if you like can just add a line to your /etc/apt/sources.list and download their binary package.