Forum OpenACS Q&A: Gush: vservers as development/production environment

I wanted to point out a couple of good uses for vservers other than for hosting.  (Obviously, I think they rock for hosting, but that's a different story.)

First, a little background.  The Linux vserver project (http://www.linux-vserver.org) involves a patched kernel and some userspace tools for managing vservers.  The result is servers that essentially behave like native linux (in just about any flavor) except that they can't interact with the kernel or reconfigure the hardware.  There's essentially no performance hit, unlike vmware, since you just run one kernel.

Pros: Each vserver behaves for most purposes like an isolated server.  Each developer on a shared server can have be root in his/her own vserver.  Different versions of just about everything?  No problem.  Want to dist-upgrade debian today?  Sure, why not?
Most applications work out-of-the-box.  Vservers look and feel exactly like a 'real' server, until you really kick the tires.

Cons:
-Extra disk space.  There are some file unification options, but you *will* need more disk.
-You do have to build a kernel, and if you want a vserver flavor that you can't find a build script for, it can be finicky to set the first one up.  If you haven't built a kernel before, don't try it on a remote server!  (What do you mean I forgot to compile in the network drivers? ARGH.)
-Some things (like IPTables) only work in the parent server.
-Each vserver needs an IP address.

Things I like vservers for:

Development-ish stuff:
- isolated development environments.  You can spawn an exact duplicate of your production server and upgrade/change/etc without risking the production server.  (You can change glibc for kicks, and if you don't like the results, just delete the vserver.  All the "I wonder what would happen if..." questions are suddenly much safer to answer.)
- since vservers can be exact clones, no issues with discrepancies in libraries, permissions, etc.
- simulate distributed systems.

Production-ish stuff:
- Small now, big later:  Put AOLserver in one vserver, postgreSQL in another.  When you're ready to split into two physical servers, just rsync one vserver to the new machine.  Vservers are very easy to relocate.  You don't even have to pg_dump.  Also useful for upgrading to a beefier machine.  Build a kernel on the new machine, rsync the vserver, and start it up.  (Ok, test, test, test first.)
- Keep a backup of your site ready to go, just in case.  If your production site lives in a vserver, having a backup of the whole vserver allows you to put the backup online as quickly as you can get AOLserver to start, if something happens to mess up your install.
- run services in separate vservers for security.  An attacker who gets into one vserver doesn't automatically get the whole machine.

Aside: there's nothing more annoying than knowing you checked in those bug fixes on _some_ development vserver, that you might or might not have deleted already.  A centralized CVS repository (running in a vserver..?) is a good idea.

Anyway, I think they're really useful.  More information at http://www.linux-vserver.org if anyone wants it.

Collapse
Posted by Ola Hansson on
Cathy,

I've always thought it's a mighty interesting technology you're using for your hosting and I've been meaning to try it out for a long time, thanks for reminding me to give it a shot  .

Is there any sign on the horizon that the security context patch (vserver) will become a standard part of the linux kernel?

Also, does unification (sharing of files between vservers) work well on Debian yet? (I read that it had some shortcomings about a year ago.)

I think I'm going to try it out on a RAID 1 device running the XFS file system and LVM on 2.4.26 kernel. Do you see any problems with that configuration?

How would you suggest that one partition the device? One partition per vserver, or..?

Thanks.

Collapse
Posted by James O'Kane on
Do you know anyone who has compared vservers to User-mode-linux? One of your customers pointed me to this article, so I'll probably bug him for more details of the vserver from a guest vserver point of view.
googling for 'vserver UML' minus quotes gives a bunch of pertinent results.
Collapse
Posted by Chris Davies on
Vserver does do quite nicely -- We run 5 different vserver OSes on one machine including Debian, Mandrake, Redhat, Fedora and Gentoo (for a development project).  Security-wise, it is fairly decent and the servers are fairly isolated.

UML in production use requires a lot more hardware to perform the same job.  You have more control over Security and Resources.  With SKAS mode, you can eek out a bit more performance than plain UML.

Then there's VMWare, which we have found to be great for development using multiple OSs.  I wouldn't attempt to run it for any production mode servers.

We run 2.4.27, tux and openwall using XFS and a few kernel tweaks.  I think your config will work just fine for Vserver.  I've had some strange issues with Hardware and Software Raid1 & SATA with 2.4.26.  I've not tested that particular combination with 2.4.27 and won't be doing SATA Raid again for quite some time.

James, read up a bit on both Vservers and User Mode Linux, the differences should quickly become clear. Usually, it should pretty much boil down to:

If each user needs his own different Linux kernel (e.g., for kernel testing), you need UML. If not, use Vservers instead. UML will incur a much larger performance hit.

As Xen demonstrates, it's entirely possibly to run a fully virtualized Linux (or Windows for that matter), kernel and all, with very little performance loss vs. running on the bare metal. But that's really not what UML is intended for.

If you're interested in this sort of stuff, check out the many links on John Sequeira's Virtualization Wiki.

Collapse
Posted by xx xx on
Will BSD Jails do what Cathy does (on FreeBSD)? Does anybody have experience with it?

In any case Building Systems to be Shared Securely is an interesting article to read.

Collapse
Posted by Chris Davies on
BSD Jail is a little more sophisticated than vserver, but essentially does the same thing.
Solaris 10 will have "zones" which are similar to jails, plus will have a "container" setup where you have a zone located inside a container, and can configure RAM usage, CPU, number of simultaneous processes that can be run, etc. for each zone.    If a zone takes up more than the configured amount of RAM, it will have its least active pages swapped out.

Rebooting a zone will take a few seconds.  You will be able to have up to 8191 zones per machine.

Collapse
Posted by James O'Kane on
I've done some experimentation with UML already. It seemed a bit too fragile to work with. I've also used VMWare for years for various other projects. vserver is something new to me.

I'll do the google thing shortly. I was just asking around to see if anyone had any personal experience or pointers. From what I've read and heard so far, vserver seems better suited for my purposes.

Collapse
Posted by Chris Davies on
Another possibility that we are reviewing is:

http://www.freevps.com/

From the writeup, it appears to be promising.

If you can afford it, Vmware ESX is _excellent_. The remote console administration tools kick butt, and overall it's just a really stellar product.

We put essentially an entire rack on one Dell 6650 server, and have been quite happy with the results. One easy thing to do is clone existing servers; our Aolserver installations are a clone of a master redhat image, which then NFS mount the proper aolserver software. This means it takes us about 15 minutes total to set up a new virtual box with openACS running, and we get CPU, memory and disk constraints for each virtual machine from a handy web-admin page.

I'm not sure I can properly communicate how well it works in this post, but we would never go back.

What a coincidence ?

A friend of mine is subscibed to VPS hosting at Westhost.com.
He wants to convert his Apache/PHP site to AOLserver/OpenACS after seeing all the neat stuff he can do with it. :)

The problem is that after setting up everything, AOLServer and Postgresql, he wasn't able to install OpenACS because db_source_sql_file won't run.

I helped him debug and I found that an error "child killed : segmentation violation" occurs when the install gets to the point of installing the acs kernel sql file.

We tried changing AOLServer from 4.0.7 to ad3.3+13 and tcl from 8.4.5 to 8.4.7 but it's always the same problem.

He would've gladly subscribed to the openacs hosting companies I recommended to him but he's stuck with his hosting firm for about another year. Bummer.

sounds like they are running either openwall or grsecurity patches and they have disabled trampolines or some other tricky optimization.

Might have him try to recompile with more sedate options, or, have the admins chpax it to allow it to bypass the particular check that is causing the segfault.

Peter,

Can you give us some more information on the 6650?  RAM, CPU, disk configuration, etc?  And how many virtual machines does it host?

I am finally taking the plunge and have a couple of initial questions about a Vserver setup. I have a ProLiant DL360 (as lound as a vacuum cleaner) on my dinning room table that I want to move to the server room asap (before my neighbors start to complain). It has U320 SCSI drives - RAID 1 (146.8GB mirrored) and 8GB RAM.

1. How much space do you typically give the vserver host OS?

2. Have any file system recommendations (e.g. ext3 vs. ReiserFS vs. XFS)? I am tending towards XFS (after doing some reading).

3. Any tips on partitioning? Documentation on the web tends to recommend a partition per vserver. Possibly relevant: I plan to separate the DB Server from the web server(s) and plan to keep CR files on the FS.

I am going to have a real system admin helping me set this up, but any preliminary vserver related input would be great (this is stuff I have to do before he can log on).

Thanks,
Carl

P.S. How easy is it to change the size of a Vserver guest OS?

Some feedback I got per email:

1. How much space do you typically give the vserver host OS?

--It's going to depend on what the host OS is running. Allow ample room for log files, but unless you plan to do anything substantive in the host, it doesn't need all that much.

2. Have any file system recommendations (e.g. ext3 vs. ReiserFS vs. XFS)? I am tending towards XFS (after doing some reading).

--There were problems at one point with vservers and JFS. Beyond that, no specific advice. I use ext3 since that's what is usually installed on the box when the datacenter turns it over.

3. Any tips on partitioning? Documentation on the web tends to recommend a partition per vserver. Possibly relevant: I plan to separate the DB Server from the web server(s) and plan to keep CR files on the FS.

--If you trust your users (or yourself, if these are all yours), don't. You get a lot more flexibility if you don't chop your space into little bits. If you feel partitioning is essential, you're going to have to work out what your space needs are going to be.