Forum OpenACS Q&A: https is down

Collapse
1: https is down
Posted by Jade Rubick on 09/17/03 06:29 PM
For some reason, https gave out on Monday evening.

Here's the error I get while trying to connect:

[17/Sep/2003:09:30:42][17509.5126][-conn2-] Error: nsopenssl: error -1/1 during SSL handshake

I'm on Debian stable:

# apt-cache showpkg openssl
0.9.6c-2.woody.3(/var/lib/apt/lists/security.debian.org_dists_stable_updates_main_binary-i386_Packages)(/var/lib/dpkg/status)
0.9.6c-2(/var/lib/apt/lists/debian.orst.edu_debian_dists_stable_main_binary-i386_Packages)(/var/lib/apt/lists/linux.csua.berkeley.edu_debian_dists_stable_main_binary-i386_Packages)

nsopenssl 2.1

Any suggestions? I hope this isn't related to the recent OpenSSH vulnerability...

I hope to get this up soon... my company is understandibly vexed. :(

Collapse
2: Re: https is down (response to 1)
Posted by Jade Rubick on 09/17/03 06:51 PM
Restarting solves the problem, but I'm concerned that it happened in the first place.

I did not upgrade my openssl libraries, and I guess we can't do so until a new version of nsopenssl comes out. Scott G has a workaround for the new changes, according to what I heard on IRC this morning.

Collapse
3: Re: https is down (response to 1)
Posted by Andrew Spencer on 09/17/03 06:56 PM
Jade, unfortunately restarting does not solve the problem. nsopenssl will eventually die again (from my experience).

According to scottg, the new openssl libs have changed the way they report if they are compiled with threads or not. Removing the ifdefs that check for threads is a temporary workaround. (Thanks to CR Oldham for this info.)

I'm trying to recompile nsopenssl now and hope that fixes the problem.

Collapse
4: Re: https is down (response to 1)
Posted by Bruno Mattarollo on 09/17/03 06:56 PM

We started to see the same problems recently on our servers (RedHat) ...

Indeed just restarting nsd seems to make the issue go away but we haven't found yet the reason. When these errors occur, it also matches with some strange logs in the access_log file.

Using nsopenssl-2.1a here ... and:

openssl-0.9.6b-32.7
RedHat 7.3

Collapse
5: Re: https is down (response to 4)
Posted by Bruno Mattarollo on 09/17/03 06:58 PM
Sorry, just saw the post made while I was typing ... yes, indeed, restarting solves the issue temporarily ... not definitively!
Collapse
6: Re: https is down (response to 1)
Posted by Andrew Spencer on 09/18/03 07:26 AM

Dang. Recompiled nsopenssl, and it was stable for about 6 hours before crashing again. I guess I could always revert to older openssl libs, but am certainly reluctant to do so. 😟 If anyone has any ideas, please let me know. My configuration is very similar to Jade's.

For the record, I've included the message I sent to the AOLserver list below.

I'm having trouble resolving an openssl issue. It seems to be
related to a recent system update of openssl. At first I thought
recompiling nsopenssl against the new openssl libraries would
fix the issue, however I am still seeing the problem.

The system is:
Debian 3.0
OpenSSL 0.9.6c-2
nsopenssl 2.1 (2.1a requires OpenSSL 0.9.7x and higher)
AOLserver 3.4.2-1

Once AOLserver starts, nsopenssl will usually remain stable for a
couple of hours. It inevitably crashes however, giving the client
this error:

"ssldomain.com received a message with incorrect Message
Authentication Code. If the error occurs frequently, contact the
website administrator."

The server log reports:

"Error: nsopenssl: error -1/1 during SSL handshake"

Previous to the update to openssl, nsopenssl had been rock solid.
Any suggestions? I hope I'm just stupid and am missing something
obvious. Thanks in advance, and thanks already to Scott Goodwin
for his previous help.
Collapse
7: Re: https is down (response to 1)
Posted by Andrew Piskorski on 09/18/03 09:12 AM
What a minute here, what was the change that instigated all these nsopenssl problems? You all upgrade your OpenSSL libraries, and then started seeing problems with nsopenssl? But from which version of OpenSSL to which version?
Collapse
8: Re: https is down (response to 1)
Posted by Andrew Spencer on 09/18/03 02:47 PM

That's a good question Andrew P. The previous version (in woody) is also 0.9.6c-2, but does not have the patch for RSA Blinding. And checking that, I found this tidbit:

Unfortunately, RSA blinding is not thread-safe and will cause failures for programs that use threads and OpenSSL such as stunnel. However, since the proposed fix would change the binary interface (ABI), programs that are dynamically linked against OpenSSL won't run anymore. This is a dilemma we can't solve.

You will have to decide whether you want the security update which is not thread-safe and recompile all applications that apparently fail after the upgrade, or fetch the additional source packages at the end of this advisory, recompile it and use a thread-safe OpenSSL library again, but also recompile all applications that make use of it (such as apache-ssl, mod_ssl, ssh etc.).

However, since only very few packages use threads and link against the OpenSSL library most users will be able to use packages from this update without any problems.

So, there seem to be a couple of options that will (hopefully) solve the problem. Thanks for setting me on the right path, Andrew P.

Collapse
9: Re: https is down (response to 7)
Posted by Bruno Mattarollo on 09/18/03 06:15 PM
The strange thing in our case is that our provider, that does the maintenance of the servers, hasn't upgraded OpenSSL ... :-?

I need to double check that again but they are telling me they haven't done it and at least for sure not within a few month of the time when we started to get these errors.

Collapse
10: Re: https is down (response to 1)
Posted by Janine Ohmer on 09/18/03 06:21 PM
Bruno, I think it is OpenSSH you need to ask them about.  There was a root exploit in it earlier in the week, so if they *haven't* updated that then you have some other questions to ask.  I have gathered from reading the threads on this that it is upgrading OpenSSH that is breaking OpenSSL, or at least that's how it sounds to me.
Collapse
11: Re: https is down (response to 10)
Posted by Andrew Piskorski on 09/18/03 07:20 PM
Janine, I believe that OpenSSH uses the OpenSSL libraries, but not vice-versa. I am not at all certain, but the Debian package dependencies in the links above appear to show this.

So perhaps it's that some folks are upgrading OpenSSL at the same time they upgraded OpenSSH, and then breaking OpenSSL thread-safety due to silly header file and build changes in OpenSSL?

Collapse
12: Re: https is down (response to 8)
Posted by Andrew Piskorski on 09/18/03 07:24 PM
Andrew S., where did you find that info about "RSA blinding"? Oh, it was here, from April, and the original 17 March advisory. It would be nice to know how serious a security vulnerabilty this really is. From the discussion in Vulnerability Note VU#997481, I suspect it is pretty low risk for most web servers on the Internet. Various googles show a lot of info from back in March.

[grumble grumble] My fairly un-informed take on this is that some of these rushed in security patches are not all that well thought out. Breaking thread-safety by default in a security patch to a formerly thread-safe library strikes me as really obnoxious.

Collapse
13: Re: https is down (response to 1)
Posted by Jade Rubick on 09/18/03 07:27 PM
What's really weird is that we didn't update anything before the server went down. In fact, we haven't updated anything in a little while (longer than I would like anyway).

So I'm damned if I do and damned if I don't?

I have to compile it from source. That's easy, but it sucks because I lose the easy Debian-style upgrades..

Collapse
14: Re: https is down (response to 11)
Posted by Bruno Mattarollo on 09/18/03 07:49 PM

Strange ... the OpenSSL version that we are running was released: https://rhn.redhat.com/errata/RHSA-2003-101.html so April or something and we started to see problems only 4 weeks ago ... but that is strangely familiar to the time when OpenSSH was upgraded ... I will investigate further ... Have to run now, so more tomorrow (hopefully).

Collapse
15: Re: https is down (response to 1)
Posted by Tom Jackson on 09/18/03 08:03 PM

I'm running RH8. When I tried to upgrade openssh, it required a newer version of openssl. So if RPM's are used in the upgrade, openssl will be upgraded to get the new openssh. Otherwise, you can get the patch for openssh and recompile that. Here is the openssh patch.

Considering how many packages depend on openssl, this is probably the easiest way to upgrade.

Collapse
16: RSA_FLAG_THREAD_SAFE is not used (response to 1)
Posted by Andrew Piskorski on 09/22/03 11:24 PM
Curiously, in the openssl-0.9.7b sources, the line "#define RSA_FLAG_THREAD_SAFE 0x10" only ever appears in exactly one place: 
$ find . -type f -print | xargs grep RSA_FLAG_THREAD_SAFE
./crypto/rsa/rsa.h:#define RSA_FLAG_THREAD_SAFE         0x10
but that's the only occurrence of the string "RSA_FLAG_THREAD_SAFE" anywhere in the sources. I thought perhaps the build process might do something funky to use that, so I did: 
$ ./config threads shared
$ make
and checked again. Nope, no difference. Whatever that RSA_FLAG_THREAD_SAFE is supposed to be for, it appears to never be used. (On this Red Hat 7.3 system anyway.)
Collapse
17: RSA blinding in openssl-0.9.7b is supposedly thread-safe (response to 1)
Posted by Andrew Piskorski on 09/22/03 11:39 PM
The RSA_FLAG_NO_BLINDING and RSA_FLAG_BLINDING defines, on the other hand, are indeed used in the code.

The openssl-0.9.7b CHANGES file has this to say: 

Changes between 0.9.7a and 0.9.7b  [10 Apr 2003]   

 *) Turn on RSA blinding by default in the default implementation 
    to avoid a timing attack. Applications that don't want it can call 
    RSA_blinding_off() or use the new flag RSA_FLAG_NO_BLINDING. 
    They would be ill-advised to do so in most cases. 
    [Ben Laurie, Steve Henson, Geoff Thorpe, Bodo Moeller] 

 *) Change RSA blinding code so that it works when the PRNG is not 
    seeded (in this case, the secret RSA exponent is abused as 
    an unpredictable seed -- if it is not unpredictable, there 
    is no point in blinding anyway).  Make RSA blinding thread-safe 
    by remembering the creator's thread ID in rsa->blinding and 
    having all other threads use local one-time blinding factors 
    (this requires more computation than sharing rsa->blinding, but 
    avoids excessive locking; and if an RSA object is not shared 
    between threads, blinding will still be very fast). 
    [Bodo Moeller]

If that's true, and it works, then no one should be seeing any thread safety problems. Thoughts?

I've had a nsopenssl 2.1a dynamically linked against openssl-0.9.7b and running on a Dev server with no problems for a week or so now, but it hardly gets any load at all so that doesn't prove much.

Collapse
18: Re: RSA blinding in openssl-0.9.7b is supposedly thread-safe (response to 17)
Posted by Andrew Piskorski on 09/22/03 11:59 PM
A Google groups search seems to show that the OpenSSL folks do indeed believe that the RSA blinding thread-safety issues were fixed back in April.
Collapse
19: Re: https is down (response to 1)
Posted by Andrew Piskorski on 09/23/03 12:14 AM
Ah, silly me! My eye missed it every other time I read this thread - seems that just about everyone above who was reporting crashing AOLservers also said they were using OpenSSL 0.9.6x.

Bruno didn't say exactly what version he was using, but the Red Hat link he gave seems to show that the latest updated packages for all consumer Red Hat distributions prior to Red Hat 9.0 are using an OpenSSL older than 0.9.7x. Which is suspicious, as it's possible that RH backported the RSA blinding security patch but not the thread-safety fix.

So the fix should be simple, just upgrade to 0.9.7b. Note that Scott Goodwin says that nsopenssl 2.1a requires OpenSSL 0.9.7x or later anyway.

Collapse
20: Re: https is down (response to 1)
Posted by Jade Rubick on 09/23/03 02:28 AM
On Debian, apparently it's not so easy to just upgrade to 0.9.7

You can't do it using apt-get, because 0.9.7 is on unstable and testing, and is compiled against gcc 3.x, etc..

So Debian users either have to compile it themselves, and lose the benefits of Debian package management, or continually restart their servers.

Ug.

Collapse
21: Re: https is down (response to 20)
Posted by Andrew Piskorski on 09/23/03 03:40 AM
Actually, on Debian, it's not any harder than on, say, Solaris. :) Yes, so far I've always fallen back to installing from source, mostly because I already knew how to do it, and at least that way I could do it exactly (well, almost...) the same on Solaris, Debian, and Red Hat.

But in this case, the Right Way would be to build your own OpenSSL 0.9.7b binary package for Debian Stable. Probably using the same source package that's in testing now, you just need to rebuild with the older Stable tool chain.

Personally, I haven't gotten around to actually doing that sort of thing yet, but my understanding is it's not bad. At least, several people here do that sort of thing, and I've read (on the Beowulf mailing list, other places) that it's not unusual for admins of large clusters to choose to rebuild most of their binary packages from source packages as a matter of course.

For Debian though, I bet someone has already done it, and if you like can just add a line to your /etc/apt/sources.list and download their binary package.