Forum OpenACS Q&A: How to test Jerry's nsvhr/nsunix virtual hosting on development box?

After working through Jerry's great How-to (http://www.theashergroup.com/tag/articles/nsvhr/virtual-hosting-howto.adp) and reading through the archives, I'm still having difficulty where others seem to have none. ("Dammit Jim, I'm an applications developer not a sysadmin." Or else I'm just a couple bottles short of a six-pack.)

I'm trying to test a server setup of two sites (firstserver.org and secondserver.org) on a development box in an internal network with unrouteable ip addresses. Neither of these domain names is real, but I resolve them with appropriate entries in /etc/hosts with /etc/host.conf and /etc/nsswitch.conf set to resolve locally with files before dns. (I'm using DNS from the ISP, not running it locally--in case that matters.)

I've installed and patched AOLServer 3.3ad13 and I've amended the stock nsd.tcl file for each server only by editing the various items pointed out in Jerry's document. AOLServer serves pages from either site when I start either one alone via nssock. I thus know that AOLServer works and the name resolution works. When I set up the third master server (running nsvhr) and start the two "slave" processes (using the nsunix module instead of the nssock module), it *appears* that the three processes seem to work (though they don't):

  • I see the appropriate firstserver.nsunix and secondserver.nsunix socket files in /usr/local/aolserver/modules/nsunix.
  • In standard out during the master server startup, I see notices that nsvhr is redirecting each of the mapped domains to the appropriate nsunix socket file.
  • However, earlier in the startup process for each slave server, I see in standard out: "Warning: nsunix: missing Location and Port parameters". (However, I do have the default port 8000 set in each case.)
  • Nevertheless, later on in standard out during each slave server startup, I can see "Notice: nsunix: DrvStart starting: listenSocket = 12, modules/nsunix/firstserver.nsunix" for one and "Notice: nsunix: DrvStart starting: listenSocket = 12, modules/nsunix/firstserver.nsunix" for the other (suggesting that some default has kicked in to provide the missing Location and Port parameters??) However, both slave servers appear to have been given somehow the same listenSocket (=12) (Is this correct; seems unlikely.)
  • One other confusing clue is that I don't see separate server.log files in /usr/local/aolserver/log/ even if I include a path in the ns_section "ns/parameters" section. In fact, no logs are getting written by any of the servers. What can this mean?

Anyway, when I then try to hit either slave server with a browser on the same server box, I get the error "Netscape's network connection was refused by the server www.firstserver.org:8000. The server may not be accepting connections or may be busy."

If this is a simple config problem, I can't spot it after staring at things for a long time. So here are my questions:

  • Are there additional parameters I'm supposed to give the nsunix module in each of the slave server's startup .tcl files (I'm using separate "nsd.tcl" files for each of the three servers.)? I'm just using the ones Jerry indicates:
       ns_section "ns/server/${servername}/module/nsunix"
       ns_param hostname www.firstserver.org
       ns_param socketfile firstserver.nsunix
    
  • Is this some problem with how I'm trying to dodge DNS by using /etc/hosts in this manner? How do the rest of you handle this when working on sites on development boxes that you then mirror up to your real servers? To date I've done all my development on the live site, but this obviously isn't a good long-term approach.
  • What else am I'm missing?
  • Would someone be willing to post their "nsd.tcl" files if they've set them up separately for the master server and slave servers?
Many thanks for any pointers!

Stan

Looks like nssock of the master server is not correctly configured. Does it say nssock: listening on port xxx as desired after startup?

Do you want the master server to listen on port 80 or on port 8000? If it listens on 80 then you should be able to request www.firstserver.com and www.secondserver.com, if it listens on 8000 then it should answer on www.firstserver.com:8000 like your wrote above, but I wonder what the point of this would be, since the only benefit of virtual hosting is that you can leave out the port number. Otherwise you could just setup localhost:8001 for your first test server and localhost:8002 for the second, without the need for a master server and nsvhr.

To be sure I'd also test this with telnet firstname.com 8000 and also with telnet localhost 8000 to make sure it's not a problem with name resolution.

When the master is correctly accepting connections then it should at least answer your request, even if it cannot connect to the slave server for whatever reason. You would get a 404 Not Found or custom error page if configured, but never a connection refused.

However, earlier in the startup process for each slave server, I see in standard out: "Warning: nsunix: missing Location and Port parameters". (However, I do have the default port 8000 set in each case.)

There is an additional Port parameter in my setup:

ns_section "ns/server/${servername}/module/nsunix"
        ns_param Port 80
        ns_param hostname "bla" 
        ns_param socketfile "bla.nsunix"
I have no idea where it comes from nor what it does, besides generating the above warning if you leave it out (i just tried that).

About the log files not being written: maybe you are running with the -f option? The server log seems not to be written in foreground mode.

Looks like nssock of the master server is not correctly configured. Does it say nssock: listening on port xxx as desired after startup?

Doh. No it doesn't, and the reason (I'm completely embarrassed to admit) is that I'd somehow commented out nssock in the ns/server/${servername}/modules section. Here I was looking at the slave config files and the problem was in the master file. Isn't that always the way it is?

Do you want the master server to listen on port 80 or on port 8000? If it listens on 80 then you should be able to request www.firstserver.com and www.secondserver.com, if it listens on 8000 then it should answer on www.firstserver.com:8000 like your wrote above, but I wonder what the point of this would be, since the only benefit of virtual hosting is that you can leave out the port number. Otherwise you could just setup localhost:8001 for your first test server and localhost:8002 for the second, without the need for a master server and nsvhr.

You are of course correct, and it was my intent to run everything on standard port 80. I just left things at 8000 "to keep it simple". Yeah, right. Too simple by half!

When the master is correctly accepting connections then it should at least answer your request, even if it cannot connect to the slave server for whatever reason. You would get a 404 Not Found or custom error page if configured, but never a connection refused.
And indeed, now things work beautifully!
There is an additional Port parameter in my setup:
     ns_section "ns/server/${servername}/module/nsunix"
             ns_param Port 80
             ns_param hostname "bla" 
             ns_param socketfile "bla.nsunix"
I have no idea where it comes from nor what it does, besides generating the above warning if you leave it out (i just tried that).

Excellent! This addition stops the warning just as you say!

About the log files not being written: maybe you are running with the -f option? The server log seems not to be written in foreground mode.

Once again you are correct.

Tilmann, you are a gentleman and a scholar! Many thanks!!

I haven't gotten quite this far, but my plan was to put the first virtual server on port 8000, and the next on 8001, etc.. I thought I understood that each of the virtuals would be listening on its own port, in fact had to be listening each on a different port, and that the master would listen on 80 and route to the correct port based on incoming name.

Starting tonight I'm going to launch into the OACS 4.5 install, shooting for a working system on 8000 - I'll worry about getting the master working per Jerry's directions after I have some experience behind me.

As mention in my howto, I recommend that you start first with an AOL approved nsvhr/nssock setup before attempting the Salchow.  That's what Brian Boitano would do.

Nsvhr/nssock will take you 95% of the way there and you won't encounter any of the weirdities involving unix domain sockets and where the unix domain socket is placed in the directory, or any of that other crap required to configure the damn thing.  It's probably fast enough for your needs for quite a while, and it gives you a fallback position to take anytime you think my dubious patches are more dubious than usual.

That fallback position is a valuable thing to have, or rather it's a valuable thing for me if you have it.  It lets you differentiate bugs I've introduced into your system from bugs you introduced into your system and thus lets me get on with work of creating more GPLed bugs.

Tonight on Chin Wag, Theater Daniel Pinkwater said that it is just a myth that a polar bear's white fur acts like fiber optics in directing/trapping/channeling solar energy.  It's now been measured and it isn't true.  And underneath that white fur is black skin.  Go figure.

I just setup a aolserver3.3ad13 to work with Jerry's nsvhr/nsunix.

It works all right, with a master and a slave (and soon 2 slaves).

My question (mainly to Jerry): is it possible to do https virtual hosting, ie: one slave with http, and one slave with https ?

If yes, any particular thing I should know ?

thanx

Nsvhr/nssock will take you 95% of the way there and you won't encounter any of the weirdities involving unix domain sockets and where the unix domain socket is placed in the directory, or any of that other crap required to configure the damn thing. It's probably fast enough for your needs for quite a while, and it gives you a fallback position to take anytime you think my dubious patches are more dubious than usual.

Jerry, is this just your usual endearing modest understatement of your formidable technical achievements, or are there limitations that one runs into with the nsvhr/nsunix solution?

  • Does it choke at high levels of throughput?
  • Does it not work with SSL?
  • Is the IE 5 bug a commonly-encountered problem?
If it works, it works, isn't that right?

Anyway, my hat's off to you for this major contribution you've made to the community! I particularly like the nsvhr/nsunix approach because

  • It's wonderfully modular; to add a domain/site, you add a new config file in /usr/local/aolserver, a new pageroot for the OpenACS code, and a new PG DB, and you're golden.
  • You can start and stop each site separately.
  • You can use the same procedures to migrate new sites to a new box if things get busy as you do to add the second site to an idle box (and the fewer ways I have to learn to do something, the better).
What's not to love about that?
<em>My question (mainly to Jerry): is it possible to do https virtual hosting, ie: one slave with http, and one slave with https ?
</em>

<p>No, you can't do this, because the SSL handshake takes place before the Host: header is transferred, so the virtual hosting modules can't know which server to hand the request to.
</p>

I've posted config files for the These seem to work for me. They're the stock nsd.tcl file that comes with AOLServer without the OpenACS mods but with Jerry's mods. I presume that once I add all the other OpenACS stuff, things will still work.

Hope this saves someone some time. (Like me next time when I need to do this again and can't remember how I did it. 😉

Regarding the IE5 bug, that's the one tremendous hurt associated with the current nsvhr that I know of, however, it's also a hurt associated with Apache 2.0, and one that at least one Apache developer is trying to eliminate.  Fun, is that there was a fix for it in Apache 1.xx, but no one seems to know what the fix was or the details why.  (That's that IE 5 and IE 6 (I believe) will occasionally have problems POSTing through nsvhr if the POST is large.  Unfortunately, it's not clear what large is, it appears to occur somewhere over 1K.)  Does it occur commonly?  I believe it depends on your site and what you ask the user's to POST.  I encounter it with OpenACS  3.2.5 using the prototyper in which POSTS commonly are about 8K long consisting of a bunch of autogenned Tcl.

No, you can't use it with SSL.

I don't know of any other problems with nsvhr, but the cool part of my development with nsvhr (and almost any other piece of software) has always been that just as I think I've seen no bugs for many many months, along comes some new (to me) behavior that is just completely bonkers.

There's a small sweet spot for nsvhr/nsunix.  If your sites get so popular you want to put them on their own machine, than you have to use a tcp socket based solution: nsvhr/nssock, Apache, or SQUID.  Also, why it is very nice that nsvhr makes adding a new proxy easy to do and doesn't require taking down ANY aolservers, it's also a bit of a pain, that each AOLserver becomes rather piggy.  Most likely you can only have somewhere between 4 and 16, or 4 and 32 AOLserver sites hosted on one machine.  Contrast that with an inprocess virtual hosting technology....

To sum it up, if I had to do it again, I'd have used an Apache or SQUID reverse proxy myself.  I only got into it because AOL released it in their first 3.0 release and claimed it worked, and then it didn't, but someone came up with one patch that was supposed to have fixed it, and then the gods cursed me and I broke out gdb one day trying to understand why I was the one that couldn't get this released package to work.

Sounds like for high-volume sites that need SSL, the thing to do is buy some more hardware, get some more ip addresses and spread the sites around. In the near term, it's the IE5 POST limitation that worries me. I don't want to have to make major hosting infrastructure decisions based on flawed M$ code.
Yes, that's my evaluation too.  I believe you can eliminate the POST problem if you use Apache 1.xx as your reverse proxy.  I don't know if the problem is in SQUID or not.  I would also think that if your site demands an external load balancer, you will probably not see the problem.  I suspect it's a function of what the solution developer can afford in terms of TCP expertise and various TCP sniffers and analyzers.  Of course, I would be completely happy to be shown that it was something in my patches or in AOLserver that could be controlled or fixed....
Oh, to answer one more question asked, nsvhr/nsunix certainly doesn't choke at high throughputs.  I did some benchmarking of it against Apache and at high hit rates, it just stomps the competition by at least a factor of two to three in speed.  Under appropriate conditions nsvhr/nsunix can realistically save you the money you might need for either additional IP addresses, additional systems/processors, or both.  nsunix works it's magic by eliminating at least one copy of the content through the proxy, so efficiency goes up the more your content tends to be larger rather than smaller amounts.

That's why the fact I can't figure how to solve the IE POST bug is so annoying.

Jerry, the redirect problem goes back to being the USUAL redirect problem (i.e. need to check for MSIE and do a meta refresh instead) as long as you ns_returnredirect with the full url not just a relative url, right?
I am not sure about that.  What I see is a problem POSTing data, and right now, I cannot say that it only occurs if something responds to that POST with a redirect, but I do not believe that to be the case.
Collapse
Posted by Stan Kaufman on
For multiple ACS installations, Jerry's How-to suggests creating separate users to go with the separate tablespaces (aka DBs in PG).

Should this separation extend to the users which own the nsd processes (and not just the users for logging into the DBs)? Or should the main AOLServer user (traditionally nsadmin) own all the nsd processes? If you put all your master/slave configs in a single nsd.tcl file, obviously you will start them all with a single user. But if you separate the files, you can have a different user per server instance.

So, which should it be:

./bin/nsd -kt nsd-master.tcl -u nsadmin -g web
./bin/nsd -kt nsd-slave1.tcl -u nsadmin -g web
./bin/nsd -kt nsd-slave2.tcl -u nsadmin -g web

or

./bin/nsd -kt nsd-master.tcl -u nsadmin -g web
./bin/nsd -kt nsd-slave1.tcl -u slave1 -g web
./bin/nsd -kt nsd-slave2.tcl -u slave2 -g web

(presuming you have users slave1 and slave2)

And what factors are important here: maintainability, clarity, efficiency, style, other? Does this really matter or not?