Forum OpenACS Q&A: Help! no SSH Telnet Access

Collapse
Posted by MaineBob OConnor on

Big Oh Oh....

This morning, I can NOT get ssh1 telnet access to my remote server. (Port 22 - ssh1 3DES) I ran a scan with ws_ping propack and I get access via these methods:

FTP POP3 SMTP HTTP

And yes, I can FTP to the server, it's still serving web pages, and I can get ftp ROOT access

Yes, FTP root shouldn't be allowed but maybe this will save me! ???

Non-Secure telnet is disabled.

This is a RedHat 7.0 system with Openacs 3.2.4 /Pg 7.0 / AOLserver with ad13 with Jerry's virtual mods...

Also port 10000 webmin seems to be working but it won't acccept passwords to root or to two other user accounts.

I'm thinking cracker but maybe it's just some minor corruption or possibly an over full partition?

Suggestions welcome and needed....

TIA!
-Bob

Collapse
Posted by David Walker on
If you have access to the entire file system put a script in /etc/cron.hourly
that launches ssh.

In my case I had to hack around until I managed to open a shell listening on
some port. (I wrote a tcl script that used the exec command.  I can't remember
exactly how I did it now.)

My solution is to have 2 ssh listeners on different ports.  The 2nd sshd is
there in case something happens to the first one or I have to restart it.

Collapse
Posted by MaineBob OConnor on

Thanks David,

I'm now trying this script to restart ssh:

#
# Restart ssh
# 
/usr/local/sbin/sshd

Only time will tell if it works...

Is there anything else to add to this simple script?

-Bob

Collapse
Posted by MaineBob OConnor on

Oops... I see that FTP has put it in /etc/cron.hourly with these permissions: -rw-r--r--

How do I set the executable flag or is it needed for files in this directory?

-Bob

Collapse
Posted by MaineBob OConnor on

Duh Bob,

I found the solution, FTP allows the use of chmod!

Now, to see if it works! waiting....

-Bob

Collapse
Posted by Jon Griffin on
And yes, I can FTP to the server, it's still serving
web pages, and I can get ftp ROOT access 
 Yes, FTP root shouldn't be allowed but maybe this will save
me! ???
 Non-Secure telnet is disabled.   
-----
There is no such thing as secure ftp or telnet.

I suspect foul play and webadmin is not a thing I would want to run on the open net. It has had its problems with security and ...

If I were you I would look for trojans, check your /etc/passwd,groups etc to see who might have gotten in. Also, check your logs, although they are likely to be trojaned as well.

Be prepared to do a fresh install and if you do, make sure you run tripwire before you connect to the net.

I don't think it is a corruption as I have run out of disk multiple times and can still ssh in as root (this is why root has its own directory).

If you installed the ssh RPM, it should have set up a /etc/init.d/sshd file and that should have started. If it is dying, (I have never seen this daemon die, but anything is possible) then I would find the reason for that. cron is a band-aid, as sshd is very stable.

Collapse
Posted by David Walker on
I have seen sshd die.  I have a firewall machine with only 24mb of ram and
an out of memory situation can result in either being unable to log in with
sshd or sshd being killed entirely.

(Tip for any programmers or script writers out there.  Infinitely spawning
processes may cause problems and use up all available memory.)

Collapse
Posted by MaineBob OConnor on

Ok, I have basic non SSH Telnet running...yes!
When I do:

# /etc/rc.d/init.d/sshd status
sshd is stopped
# /etc/rc.d/init.d/sshd start 
Starting sshd:execvp: No such file or directory
[FAILED]

Any hints about execvp?

There also is a "condrestart" option that appears to do nothing.

So far, I've found no evidence of a cracker, but I'll be checking further...

-Bob

Collapse
Posted by Jon Griffin on
I think it is trying to start a virtual process. Can you run top and see what it says?
Collapse
Posted by MaineBob OConnor on

Hi Joh,
Here is a condensed version of top without the redundancies:

USER    SHARE   TIME COMMAND
nsatgn   1064   0:00 top
root      300   7:51 init
root        0   1:41 kflushd
root        0   7:05 kupdate
root        0   0:00 kpiod
root        0   3:53 kswapd
root        0   0:00 mdrecoveryd
rpc       248   0:00 portmap
nobody      0   0:03 identd
nobody      0   0:06 identd <defunct>
daemon     56   0:00 atd
root      120   0:02 crond
root        0   0:00 mingetty
postgres  348   0:51 postmaster
nsamain  6880   0:00 nsamain
nsaerc    27M   0:01 nsaerc
root      420   9:58 master
postfix  1328   2:25 qmgr
nsatgn    48M   0:02 nsatgn
root     1028   0:00 xinetd
root      668   0:00 in.telnetd
root     1392   0:00 bash
root      616   0:14 syslogd
root      820   0:00 klogd
postfix   812   0:00 pickup
postgres 5208   1:02 postmaster

I killed the identd but it won't die!
kill -9 349
the prompt returns and running top again and it's still there.

Now a hint that a cracker may be at work.. 😟 The RH7.0 server has been up for 208 days, yet the files in /etc/rc3.d are ALL timestamped earlier today except for ...postgres and s99local. The files in other directories rc1..2..4..5..6 are all dated with the server birthdate. I don't see anything else out of the ordinary but I may not be looking in the right places...

So now onto doing an extensive backup...... for the worst case...
Any suggestions for making a backup, that will be easy to restore?

I've backed up the pg data but what about a complicated virtual system (Jerry's)... I'm tarring the whole /web tree that contains the multiple systems and.... oops tar just crashed... on /web due to:
tar: Error exit delayed from previous errors
The tar file got to 139 Megabytes before failing... I guess I'll have to do it in pieces....

-Bob

Collapse
Posted by Jon Griffin on
What was the server load and etc (i.e. top lines) from top. I need to know how much swap space and memory etc.

As for backup you probably ran out of disk space, try to tar from a larger partition if you have any. tar should work, size isn't a problem as I have tarred 1.5 gig. Of course 2 gig is the normal ext2 size limit.

Collapse
Posted by David Walker on
upload clean copies of netstat and ps so you can see if there are any
processes running that are hidden by rootkit versions of those programs.
Collapse
Posted by MaineBob OConnor on

  8:22pm  up 208 days, 20:45,  4 users,  load average: 0.14, 0.19, 0.20
87 processes: 85 sleeping, 1 running, 1 zombie, 0 stopped
CPU states:  0.9% user,  6.0% system,  2.1% nice, 90.7% idle
Mem:   516140K av,  467080K used,   49060K free,   47272K shrd,  114244K buff
Swap:  265032K av,    1504K used,  263528K free                  227952K cached

-Bob

Collapse
Posted by Jon Griffin on
Everything looks ok on top. I would suspect something else.

It is a shame to kill your uptime, but I think it is time to update some stuff.

Is this box where you can get at it physically?

Which kernel are you running

Collapse
Posted by MaineBob OConnor on

Cracker found!:
I was searching through the /var/log/message file and found these entry from yesterday and today:

sshd[17126]: Disconnecting: crc32 compensation attack: network attack detected
sshd[17139]: Disconnecting: Corrupted check bytes on input.
adduser[17163]: new group: name=liq, gid=521 
adduser[17163]: new user: name=liq, uid=521, gid=521, home=/home/liq, shell=/bin/bash 
new group: name=liq1, gid=522 
new user: name=liq1, uid=0, gid=522, home=/home/liq1, shell=/bin/bash 
Accepted password for liq from 212.199.171.187 port 1214
sshd[17186]: Disconnecting: Corrupted check bytes on input.
Could not reverse map address 212.199.171.187.
PAM_unix[17297]: (system-auth) session opened for user liq by (uid=0)
PAM_unix[17322]: (system-auth) session opened for user liq1 by liq(uid=521)
adduser[17439]: new group: name=satan, gid=523 
adduser[17439]: new user: name=satan, uid=522, gid=523, home=/home/satan, shell=/bin/bash 
userdel[17467]: delete usersatan' 
userdel[17467]: remove groupsatan' 
adduser[17470]: new group: name=satan, gid=523 
adduser[17470]: new user: name=satan, uid=522, gid=523, home=/home/satan, shell=/bin/bash 

A bigger excerpt is here:

www.rocnet.com/hack/crack1.html

So NOW what?
* These entries for this user are gone from group and passwd
* DNS reverse lookup turns up NOTHING for 212.199.171.187
AND

Do I really need to start over or is it possible to clean up this mess?

Expletives [*****************] here!

-Bob

Collapse
Posted by MaineBob OConnor on

I have no physical access to the box. My friend Bill would need to drive about 2 hours to get to the server. The server is the "white sheep", the only Linux box in a farm with "black sheep" MS servers. He was the one who did the install early this year. I don't know which kernel but it was whatever came with RH7.0. As I understand it, he would need to pull the box from the Rack and put it on the bench with keyboard/monitor to do an upgrade... 'cause that is the way he did it ....

Now, I'm thinking maybe I need to get a NEW box quick and move the good stuff while I can....

Hey.... this appears to be an attack via SSH? I thought that SSH was secure?

-Bob

Collapse
Posted by David Walker on
A search on http://www.arin.net for 212.199.171.187 points to www.ripe.net
which indicates that that IP address belongs to what I believe is an ISP called
Golden Lines in Israel.

I would not trust a cracker to come in, look, and leave without installing a
rootkit and giving himself a back door to return to your system.

psyBNC (http://www.netknowledgebase.com/tutorials/psybnc.html ), the
program that he uploaded, allows him to pretend to be coming from your
network while irc chatting.  This could be for entertainment purposes or to
protect himself if he is using an irc server to control zombie computers.
(Assuming the filename of the file he uploaded matches the contents.)

Hard to say what term.c was.  You'll need to do a new installation on this
box.

No program is perfect but ssh does encrypt your information so that it cannot
be sniffed on the network.  Telnet, ftp, and pop email all expose your
password in plain text.

Looks like this advisory from bugtraq might cover your problem.
http://cert.uni-stuttgart.de/archive/bugtraq/2001/02/msg00179.html

Collapse
Posted by David Walker on
Another recommendation. Move ssh to a different port. Port 22 is too obvious. It's the first place they'll look. Pick a five digit number (but less than 65536). Make them work for it.

If you have a couple of locations with static IP addresses then put in ipchains rules to restrict access to the port you selected with ssh to only those IPs.

Remove ftp from your servers. use scp for file transfer or look into the new gftp that supports sftp. http://gftp.seul.org
Collapse
Posted by Jon Griffin on
You have to start over.

The kernel that came with 7.0 had root exploits, I am sure that some of the stuff installed is also insecure. openSSH itself had an exploit at around 2.3 or 4, and if you have it set to use ssh1 protocol as the default, there are many exploits.

I am not sure though that it is an ssh exploit per se. I would get a new box and secure it before you start. I have (old as it is) some info on hardening an RH box on jongriffin.com and some newer links on dev.jongriffin.com.

Mainly, deinstall all the crap that probably got installed. Run tripwire or the equivalant before you hook it up to the net.

If you are using RH 7.2, upgrade to a non-modular kernel 2.4.13 is good and install the grsecurity patch, you can get the link from my site.

If you have any questions, please contact me either at my email or on the list.

Whatever you do, get rid of that system somehow. Your PG stuff and acs are most likely fine as this appears to be a script kiddie or your log file would have been erased.

Collapse
Posted by Jon Griffin on
Bob,
I wanted to get that last bit out before I did some investigating of my own.

I really don't think that ssh was the problem as I could only find theoretical attacks, but nothing in the wild. More likely they got in another way.

  • pam
  • xinetd
  • ftp - this seems like the most likely
It is an interesting puzzle that I am going to try to figure out. I really think it was your ftp that allowed them in and then a bad pam that let them get root.

FTP is evil and should be taken out of every distribution ever created and if the MS users don't like it they can get their own servers!

Anyway, it shows that even in the linux world you need to keep up with patches. I am also a little concerned at the CWD of pgsql and the session close for nsadmin, unless you did that why would a hacker give a crap about aolserver unless some of your pages were defaced.

Collapse
Posted by Patrick Giagnocavo on
Just remember that if the people that are at the colo place where you have the box have minimal PC experience, you could just build a new install on an IDE drive and ship it to them to replace the current drive with the new one (you already know the IP address, what ethernet card driver you need, etc).  Then have them ship the old drive back to you so you can examine it's contents at leisure.  Of course, you can use ssh and scp to copy the old data to the new drive, if you haven't just pulled the old data off already as a precaution.
Collapse
Posted by Marc Spitzer on
I have to agree with everyone else, you are screwed reinstall the box from scratch.  Pull your data off and blow the partitions.  And examin any data files carfully, especialy if they have an exicute bit set.  Set up sftp instead of ftp on this new box.  Run tripwire and store the database some where else, burn a CD.  Then if you ever wonder you have a base line to check against, this is in addition to keeping a local copy and doing a nightly check.  As far as I know freebsd has some neat features security wise that linux could use.  2 that come to mind are security levels and the immutable flag for files.  If you get rooted you can prevent yourself from geting root kited, the cracker cannot delete or overwrite files that have the immutable bit set with a high enough security level, even as root.
Collapse
Posted by S. Y. on

An installation from scratch is necessary; the damage is done. There are a bunch of root exploits, some to do with certain kernel versions, others to do with certain versions of OpenSSH (although, as Jon claimed, they tend to be theoretical rather than in-the-wild).

I'd run Tripwire and then run a vulnerability assessment scan with Nessus before putting the machine live on the 'net. I don't consider Webmin to be a security-conscious, gotta-have-it service on a production web server.

Also, considering that this appears to be your second break-in, I suggest that you find someone else to lock down your box. But that's just me.

Collapse
24: Bad Pam, a very bad pam (response to 1)
Posted by Jerry Asher on
Jon,

Assuming you're not talking about the Pam we all have tapes of (sorry Sean, I'll get those back to you soon), can you be more specific about what you mean by a bad pam?  I know PAM as the authentication module, just how can that turn bad?

Thanks

Collapse
Posted by S. Y. on
Not sure if Jon has a specific exploit in mind, but here's a quick list:

http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=pam

I don't think that it was really an SSH-related exploit that happened here, but I note that the use of the SSH1 protocol is not pretty much "officially" frowned upon.

P.S. hehe, don't rush on those tapes, Jerry.

Collapse
Posted by Jerry Asher on
Can anyone recommend appropriate firewalls for "typical" ACS installations, that is, an ACS website on one "standard" Linux box serving up a million or so hits per day (plus or minus half a million?)

What are the pros and cons of installing a firewall in front of your server (in addition to say, using tripwire, and snort on your server)?  What are the expected costs of a firewall?

Collapse
Posted by Jon Griffin on
I don't have any specific pam exploits because as Sean pointed out there are a few.

Rule one in security, don't trust anyone,anything and certainly not any program.

This brings me to firewall, IPchains/tables is a good start. I don't think that firewalls are really necessary for a single web server. Shut everything off you don't need and then grant access as needed.

Set up an alternate userid 0 and always use that as your login instead of root. That way if you get a log message saying someone logged in as root....

Security is a topic much to broad to discuss in a meaningful way here, but feel free to post questions.

BTW , Linux also has access levels and immutable bits, and lots of other stuff such as random PIDS, random icmp sequences and etc. You just have to know where to get the patches.

BSD's are fine, but so is Linux.

Collapse
28: Professional firewalls (response to 1)
Posted by Stephen van Egmond on
Professional firewalls can mean very different things. If you believe the marketing, the checkpoint firewall is "very professional".  I know it impresses banks, but it is also very easy to misconfigure.

IMHO the best firewall is one that combines load balancing and failover with firewalling.  And to me that means Big/IP.  You'll need two.  See f5.com.  It will cost you $10K or so to deploy but if you really are getting a million hits per day, that should be  chump change to you.

If you don't see the point of load balancing or failover, then you'll probably be fine with a cheap Linux box (run Debian, please) doing firewalling.

For my tastes, cheap linux boxes are too cheap and have too many moving parts and require too much maintenance and are too tempting to install software onto.  I use a Linksys router (they sell them as cable-modem routers but that's not all they're good for).  They do packet forwarding from arbitrary ports to arbitrary systems behind the firewall, so if you feel the need to separate web, mail, dns, and login, you can.  The little devices can be configured with lynx, or remotely at slightly increased security risk.  They have no fans and no hard drives to break down.  They're fantastic, I love them.

Collapse
Posted by Marc Spitzer on
BTW , Linux also has access levels and immutable bits, and lots of other stuff such as random PIDS, random icmp sequences and etc. You just have to know where to get the patches.

Not to be a shmuck, but if I have to patch the kernel then it is not in the offical tested distribution of the kernel and I introduce more risk that something else will break.

If you want to build a stateful firewall my personal favorate is freebsd and ipfilter. OpenBSD probably has a slightly more hardened default install. But I do not trust ther ipfilter replacement(packet filter) yet, its too new for me to use in production.

Collapse
Posted by Patrick Giagnocavo on
I hate sounding like an advocate, but OpenBSD is secure from remote exploits out of the box.  And a default install includes a recent version of OpenSSH, which is immune to the possible reason that this box was cracked (though as someone else mentioned it might have been PAM).

While adding OpenACS and PG might reduce that level of security, you can use packet filtering (also built in) to block any attempts to connect on ports that should not be connected to.

OpenBSD follows a six-month schedule, with major releases every six months.

One other point in favor of OpenBSD is that documentation is excellent.  The man page for the software raid driver, for instance, is complete enough that you can follow it to end up with a perfectly configured RAID array by the time you reach the end of the man page.

I am not a "professional" security guru, but am someone who has to admin many machines for different OpenACS customers - I chose what worked, for me.

Locking down services I don't need, packet filtering, and working from a relatively secure base OS is something I feel comfortable recommending.

Collapse
Posted by S. Y. on

Just a few comments on topics that others have touched here.

  1. A firewall for a standalone web server might not be a great benefit. As Jon mentioned, it's probably just as well to lock down the box and not run anything you need; the basic premise of a firewall is to provide a protective barrier to other systems.
  2. The redundancies in top are not insignificant. A cracker could easily upload a binary and rename it whatever (init, identd, etc.) and without doing careful analysis, makes a top list with redundancies removed useless.
  3. Concerning 2.4 Linux kernel releases and the notion of "official tested distribution", I don't think there's such a thing. Linux kernels do not submit to any QA process. Two of the seventeen 2.4 releases have been flagged as "don't use" -- i.e., showstoppers (the last being 2.4.15 which had a filesystem corruption bug). This might be an argument to switch to *BSD, but I'm not sure about that.
  4. In Bob's case, it appears that this is a production box. Telling him to switch to *BSD isn't really an immediate solution. It might be a longer term alternative for him to consider. (Refer to point 6, below though.)
  5. Switching to *BSD might not be all that helpful. Bob has publicly disclosed two incidences of his boxes getting cracked. There is evidence of what I consider to be shortcomings -- use of ftpd, old kernels, SSH1 protocol, use of Webmin -- some of which are egregious (particularly ftpd and old kernels).
  6. However, there is clear indication here that whoever is responsible for server security simply isn't good enough to do the job, whether on OpenBSD, Linux, Solaris, you name it. I mean, there's very elementary stuff not being done here (e.g., banishing ftpd).

I think that Linux can be a relatively secure OS, but it certainly is not from a shrink-wrapped distribution (especially an older one like RH 7.0 that hasn't been actively upgraded and fed the latest kernels). Nowadays, Red Hat is marginally more aware of security issues for its basic software installation, but everyone who installs it must realize that the software installer is designed for maximum hardware compatibility, and not optimized for performance nor security.

Jerry, I thought you were a big fan of SonicWall's stuff. Whence your question? Do you have reservations about whether their solutions are "professional"?
Collapse
Posted by Marc Spitzer on
Concerning 2.4 Linux kernel releases and the notion of "official tested distribution", I don't think there's such a thing. Linux kernels do not submit to any QA process. Two of the seventeen 2.4 releases have been flagged as "don't use" -- i.e., showstoppers (the last being 2.4.15 which had a filesystem corruption bug). This might be an argument to switch to *BSD, but I'm not sure about that.

I am not saying that *bsd does not have its problems, but that is a very scary attitude to have about QA for mission critical parts of any OS.

Now I was not saying that Bob should change to *bsd, he needs to get back up asap. Debian might be worth a look though.

Now iff I take snort, swatch, ipfilter and a small amount of glue code I could, after I have turned everything unnessary off, add dynamic firewall rules to the mix. So if I see a web based attack I can turn off that ip at the firwall and protect what is behind it, even if it has been compromised. Also with statefull firewalls, that are configured correctly, if a trojan/worm gets intalled you still cannot get to it, in or out, because the firewall will not allow the connection.

Collapse
Posted by Jon Griffin on
I don't really care about os wars but I must make some comments.

BSD has its problems, compatibility and mind share being the major ones. I will grant you that OpenBSD is reasonably secure out of the box but they only audit kernel code and are just as vulnerable as any other os for their non core code. Some programs are just not easy to use under BSD due to differences in libs

Debian isn't a panacea for anything. If you run anyones stock distribution, you shouldn't call yourself a sysadmin and better hire someone who can compile a new one with whatever patches you need.

Any distribution that uses loadable modules must be fixed to run without the ability to insert code into the kernel (this includes Debian).

Debian is just as vulnerable as any other distribution to any Linux kernel problems. They use a different packaging system and in many cases older versions of software that are "stable". I think you can find many exploits for Debian as well as Mandrake, Suse and all the other distributions.

The bottom line is: Either learn security or hire someone who knows it no matter what OS you use.

Collapse
Posted by S. Y. on
The intent of my second post was not to start an OS jihad.

About Linux kernels not being QA-ed revolves around the stuff at www.kernel.org (and mirrors). Certainly the major distributions put in a certain amount of effort in QA-ing their releases, but there are only varying degrees of out-of-the-box security in Linux distros and nothing that anyone should ever expect to be sufficient for a production server live on the 'net. This is meant to scare anyone away from Linux, but it's simply a wakeup call/caveat emptor.

I wouldn't be surprised if OpenBSD is more secure than a fresh install of whatever Linux distribution, but there's a certain amount of security aptitude necessary to bring both boxes up to production level security in both cases. OpenBSD gives the sysadmin a head-start, but a competent UNIX sysadmin should/must be able to get a Linux/Solaris/IRIX/whatever box up to the same level; you just start at different places on that path depending upon your choice of OS poison.

Collapse
Posted by Jerry Asher on
My personal experience sysadminning firewalls has mostly been as end user protecting various low end systems connected to the net with DSL or frame relay.

To that end about three years ago I picked up the first Sonicwall and I have been very pleased with it.  It was pricey then, but well worth every penny in being simple to configure (they would claim out of the box and configured in 15 minutes), while offering some sophisticated abilities: stateful inspection, vpn, dhcp, ppoe support, pinholing, and protection against DOS, spoofed ip addresses, and all sorts of other stuff.  The sonicwall appears to offer much better protection and its been easier to administer than all those firewalls at the various companies I've worked for, where all the "professional IT sysadmins" can say is, "Can't do it, because of the firewall."

Hey, this firewall got me onto the CBS Evening news with Dan Rather!  About two years ago, my sonicwall started telling me about more and more frequent attacks.  As in lots of attacks, many times per day.  I traced the attacks, and if the addresses weren't spoofed, they were coming from Korea, Iraq, Serbia, real cool right?  I wrote off to some journalists and I said it was an example of how PacBell (and other ISPs most likely) wasn't educating their consumers (esp. their DSL consumers) about the dangers of putting a computer on the net.  One or two small articles were written about the attacks I experienced and that was that.

Three weeks later, Yahoo, EBay, and CNN dropped off the net.  So I got a six am phone call from some producer in New York wanting to interview me because according to LexisNexis, I was about the only person in the US who ever "firewall+internet+attack+DOS" or something like that.  But I pretty much got the whole story wrong, because no one knew about the DSL zombies on the first or second days, so I said that my attacks were probably benign script kiddies.  D'oh!

Hey it was cool, some independents drove out to my place, were here for about three hours, interviewed me for about thirty minutes, and then they showed about fifteen seconds of the interview, but they did include screen shots of my logs and traces showing attackers from Serbia, Iraq, and Korea trying to take me down!  I was so dumb, I bet they would have shown more if I hadn't said that what I experienced was probably just kids.  Dumb, dumb, dumb!  I should have said, "oooh, scary hacker spies from communist satellites are out to get us all!"  What was really neat was how my picture appeared in four or five completely different segments on CBS and local news for the next month.  "Hey Carol, we need some canned video of geek in suit, in five minutes"  "Okay, I got that right here"  And for having shilled Sonicwall on the air, they sent me two XL sweatshirts!

That said, I don't know how their newer high end products stack up in ability or price to the competition.  I would expect that Sonicwall has some excellent products, and I appreciate that three years after purchasing my first, they still support it with new firmware releases with bugs fixed, and new features added, but I just don't know what the competition is in the market of higher end web application level firewalls.

I look at that purchase of a $400 residential firewall as my hiring of all of sonicwall's network and security engineers.  While I could have secured my system on my own, there is no way I could have done it as well, and protected the system from the various DOS attacks that the sonicwall supports for $400 of my own time, plus give me access to a technical support team that at the time could answer my questions or educate me as to other network issues.

There are a few reasons I still favor a separate firewall in front of a webserver.  One, techsupport if they are any good can be very useful.  Two, they are watching and constantly fixing the bugs in their firewall and enable me to let up on my scanning the various buglists looking for problems with say ipchains.  Three is nice sweatshirts.  Four, when it comes time to troubleshooting your system after the damn thing has gone live, the firewall is a completely independent node.  I can test my system with the firewall in place, and I can tweak the rest of my system without having to worry that my firewall has been compromised or disabled by my own actions.  That means that I can temporarily at least, install or configure vulnerable software and not have to worry about those vulnerabilities leading to cracks.

And should I suspect a firewall problem, it's very easy to test it in place, or swap it out for another black box when I need to.  (I actually found that old AOLserver beta 3 servers could crash the Sonicwall as a very old version of ns_httpget sent out bad headers with LFs but not CRs and the Sonicwall wasn't tolerant of that.)

It's been a great sleep aid.

Collapse
Posted by Jerry Asher on
By the way, when I wrote that my $400 was like hiring all of their network engineers, I was not saying that that eliminated my responsibility to secure my own system regardless of what the firewall provides. I removed ftp, sendmail, and eliminated as much as I knew.

I completely agree with Jon when he writes,

If you run anyones stock distribution, you shouldn't call yourself a sysadmin and better hire someone who can compile a new one with whatever patches you need.

The bottom line is: Either learn security or hire someone who knows it no matter what OS you use.

So I don't call myself a sysadmin, and I hate wearing pagers and cellphones too.
Collapse
38: Another SonicWall plug (response to 1)
Posted by C. R. Oldham on
Let me also put in a plug for the SonicWalls.  We just got one and I've been extraordinarily happy with it.  And I sleep better at night, too. 😊

One disadvantage, however, is that they license the Windows VPN client on a per-user basis, and it does not currently work under Windows XP.  This is a pain, and the client is expensive.  I did read on the FreeS/WAN list, though, that there has been some success connecting Linux with FreeS/WAN to a SonicWall via IPSEC, so you may not need the Win32 client.

Collapse
Posted by S. Y. on
In light of Jerry's lucid explanation, I must retract my earlier opinion that firewalls in front of standalone web servers are unnecessary. That extra cushion of security is worth the $400. I'm not running a server on the 'net anymore, but if I did again for some reason, I would seriously consider it. I don't really call myself a sysadmin anymore, and certainly never said I was a security guru. However, between something like a SonicWall and my own measly skills, I don't think I'd lose any sleep about a personal hobby-type web server. When I was responsible for a corporate web server, I certainly had other people set up security.
Collapse
Posted by Jamie Ross on
I would like to add another option.  Smoothwall is a hardened Linux based firewall and includes IPSec, Snort , IPchains , forwarding etc AND it runs on older machines. The system is designed to be the sole resident and has a nice web interface.  I have it running on a P100 with 32 Mbytes of RAM with no problems and I believe it is very secure. It is also very easy to set up and supports DMZ zone.  You can download it from http://www.smoothwall.org and I highly recommend a donation if you find it useful to keep the project going.

I run a couple of different servers (AOLserver,Postgresql etc) on different machines and seems to work fine.  We also replaced Gauntlet at work with Smoothwall and good luck there.

cheers
Jamie

Collapse
Posted by C. R. Oldham on
Thanks, Jamie.  Any idea if the IPSEC implementation in Smoothwall will interoperate with the SonicWall VPN Client?
Collapse
Posted by Louis Gabriel on
Sean, (and anyone who likes the BSDs and/or secrity focused Linux distros)

Not to start an OS jihad, but I am interested in your opinion.  If you were to pick a flavor of BSD for an OpenACS/PG server with security uppermost in mind, which one would you pick?  Same question for the Linux distro; which one would you pick and why?  Thank you for sharing.

I would appreciate the opinions of others here also.  Assume a "blank slate" for your choice of OS(S); Heck Trusted Solaris or any other *nix is fair game, too.

Note to all:  While some may object to this question due to some going off an an OS Jihad, I do think it's both central and crucial to enabling more use of the internet by various organizations.  Security concerns are a big worry to lots of folks who would otherwise use the net more for business/organizational use.

Moreover, if useage of the net evolves into lots of interconnected non-pc devices all communicating with each other and conducting important and/or finacial transactions -- like your fridge ordering what you're out of or your pacemaker being "fine tuned" remotely for health's sake -- security has got to be "whipped" as an issue or progress will stall big time.

Again, thanks for sharing.

BTW,  hope all is going great for you,

Louis

Collapse
Posted by Stephen van Egmond on
I favour Debian for nearly all my work. I've written an article, one of the sections of which details the Zen of Debian.

I've personally found that the distribution is the easiest to maintain. I agree with other posters that security is difficult and requires attention. Debian allows fast, easy upgrades through its stellar package management system.

In my day job and at home, I use it for production systems and it has never, ever disappointed me.

Collapse
Posted by Talli Somekh on
If I may make a request as the community's (sometime) self-apointed PC lefty-liberal, let's try not to use the term "Jihad" for anything other than holy wars waged by religious psycopaths.

Those of us living in Afghanistan and New York City live a little too close to the real deal to feel that a Slashdot-esque discussion of OS utility is anything more than a some (bearded) sysadmin's flaming opinion.

thanks.

talli

Collapse
Posted by Talli Somekh on
BTW, Stephen, dope article! Debian does rock!

(of course, if someone could tell me how to get XFree86 4.1 to run on my Dell C600, Debian and that person would rock even harder.)

talli

Collapse
Posted by Patrick Giagnocavo on
Louis asks "which BSD"?  In my opinion, if you have x86-based servers, go with OpenBSD 3.0 for its emphasis on security; although between NetBSD, FreeBSD and OpenBSD I would say that there are no bad choices. The recent changes to their filesystem code greatly increase performance while maintaining filesystem integrity.  Note that you need to apply the one-line fix that Connie Hentosh posted on this bboard to have the best reliability, and should probably add the tcl exec fix if you are using exec a lot, or running OpenACS4.

For Linux, Debian is your best bet. Their package management is just the way to go, period.

Collapse
Posted by Marc Spitzer on
Well I would go with FreeBSD and IPF, PF in openbsd is just too new for me to trust it yet.  And IPF also is ported to a bunch of other unix's.  For a pc server it is hard to beat FBSD, when done correctly it is just so quite.  And Freebsd has smp support, openbsd may have it in 3.0
Collapse
Posted by Louis Gabriel on
Talli,

My sincerest apologies at offending/irritating you by the use of the word Jihad to refer to OS wars.  I was simply repeating the common venacular.  However, "OS wars" suffices just as well without offending anyones sensibilities.

Thanks for bringing it to my attention, as the last thing I want to do is offend anyone's religious or cultural sensibilities or beliefs here -- or anywhere else for that matter.  I'll make it a point to use -- in retrospect -- the more appropriate "OS wars".

Take care and best wishes,

Louis

Collapse
Posted by Talli Somekh on
Louis, I didn't mean to single you out or make anyone feel bad about their word choice. Before 9/11, I used that word plenty, too, for software and other stuff. But I guess enough things have changed...

I really appreciate your response. Thanks.

talli

Collapse
Posted by S. Y. on
I apologize for my use of the "j" word; I've been using it for years and I errantly assumed that people would read it today the same way they did in August. My mistake. Okay, moving on...

Holy crow! 42 responses?!?

Here goes... When I ordered my last set of Red Hat 7.2 CDs from www.cheapbytes.com, I tossed in a set of FreeBSD 4.4(?) CDs, so I guess you have my answer. FreeBSD appears to be the most popular "distro" of *BSD, so I thought I'd give it a shot. I've played with the installer once or twice, and have gotten it to boot with X and KDE, but that's about as far as I went.

However, my motives for trying out FreeBSD aren't really security-related. Someday I might switch to Mac OS X/Darwin if I can be reasonably assured that I can run qmail+nmh+exmh. The security stuff is really a hobby for me and the more I can learn about it, the better.

Note that I've stuck with RH Linux for over three years for one reason. Because Oracle qual-ed their RDBMS on it (three years ago, in the dark ages of Oracle 8.0.5). I've gotten rather used to the convenience of RPMs (even if I think rpm is a relatively stupid installer compared to SGI's inst); however, I pretty much always compile my security software from scratch.

I enjoy the convenience and widespread acceptance of RH Linux, but I don't have a good time locking stuff down every time I do a fresh install. There's always a cracker out there whose smarter and craftier than you, but I'm willing to live with those risks right now.

If I had to give Linux distros a second go around, I'd consider Gentoo Linux (www.gentoo.org), although those guys have been in pre-release state for years now and their installation documentation sucked large rocks (well, at least until recently). I actually haven't evaluated how secure the distro is, but even a year ago (when I last tried it) they seem to have some ideas I liked: BSD-style ports system, qmail, root partition on a ReiserFS filesystem, etc.

SGI IRIX wasn't considered a particularly "secure" UNIX when I was using it. The SGI boxes were sure fun to use though. Maybe since those p0Rn sites (www.danni.com) have been using it for a while, IRIX has gotten better. If I had to run a production-grade web site, I'm pretty sure I'd pick Solaris again. I don't know much about Solaris, but there are plenty of good Solaris sysadmins out there.

But this is pretty much moot nowadays, since I've resigned to live on a cheapass Windows-only dial-up modem connection. Hope that answers the question.

Collapse
Posted by Louis Gabriel on
Thanks, Sean.  I appreciate you're sharing your insights.

Best Wishes and take care!

Louis

Collapse
Posted by Louis Gabriel on
Oops.  I do know the difference between your and you're.

Sometimes it's just not wise to post before moring caffeine!

Take care,

Louis

Collapse
Posted by Roberto Mello on
Bob, I don't know if this was mentioned in this thread or not, but you really shouldn't be allowing ssh v1 connections.
Collapse
Posted by MaineBob OConnor on

Wow! This thread became long is a great discussion! Thanks....

For followup to my starter post, Jon helped me harden my current server and stop many unnecessary processes including FTP and hopefully stop the crackers.... last night, a couple more from Canada gained access via FTP...

As a replacement for my windows FTP program I'm now using WinSCP and like it *better* than my old FTP client.

    http://winscp.vse.cz/eng/
    WinSCP is freeware SCP (Secure CoPy) client for Windows 95/98/2000/NT using SSH (Secure SHell). Its main function is safe copying files between local and remote computer....

I'm also, on a not so urgent schedule, planning to move this production site to a new *hardened* server right from the start!

Thank you!
-Bob

Collapse
Posted by David Walker on
I've been looking for that program.  Thanks Bob!
Collapse
Posted by Tilmann Singer on
Roberto, could you give me a clue why ssh v1 is a bad thing?

I am running debian potato on my production server, and the only ssh version available there is OpenSSH-1.2.3 which does not support protocol version 2. I wonder if I should upgrade this with a package from a newer debian distribution or install from source (reluctantly, because then I would have to check for security updates myself instead of being able to rely on security.debian.org).

Collapse
Posted by Jon Griffin on
ssh1 has a variety of exploits. You are safer than telnet, but how safe do you want to be?
Collapse
Posted by S. Y. on
http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ssh

I wouldn't rely on your distribution's provider to keep you up to date on security patches, etc. Also, it's very easy to subscribe to the OpenSSH announce e-mail list and you'll only get mail when there's a new release. I'm on the Nessus, OpenSSL, OpenSSH, and qmail announce lists and I probably don't get more than ten e-mails per year.

I visit www.osdn.com every day so I see most of the major security bulletins. www.securityfocus.com isn't a bad site either. If you want security, don't expect to be spoon-fed by someone else. The burden is on *you* to keep up to date.

Collapse
Posted by Roberto Mello on
Tillman, http://www.debian.org/security/2001/dsa-086 says that a patched ssh-nonfree package in potato has been made available.

I wouldn't run potato in production just because it takes too long for packages to make it there, whereas woody and sid don't have this problem.

I run several production servers with Debian sid (unstable) with no problems whatsoever. In Debian's context "unstable" means "changes a lot" and not "broken". If you are careful when apt-get-upgrading then you are fine.

I found that Debian unstable is more stable than most other distribution's stable releases.

Collapse
60: Smoothwall (response to 1)
Posted by Jamie Ross on
Smoothwall (http://www.smoothwall.org) uses freeswan (http://www.freeswan.org/) as its VPN component so any VPN client with works with freeswan should work.

One nice feature is the painfree install (it even partitions the drive automatically) and provides web based updates which means keeping up with patches is very easy