Skip to content
December 21, 2010 / gus3

Linux Out, FreeBSD In

Don’t let the title scare you. This is only on my main file server, which holds both shared material (source tarballs and update packages) and 116G of movie files. It has recently begun to mis-behave under Linux. The specific symptom is a network freeze under heavy I/O load. This is not unknown with the Realtek 8139 Ethernet chipset, but it’s a big hassle when a movie player stops hard because NFS has suddenly disappeared.

Copying files off for a system wipe took no less than 3-1/2 days. The happy medium I had to live with was rebooting the file server every 20 minutes, and limiting rsync to 1.5MB/s. Most of the time, NFS (a stateless protocol) didn’t have a problem with resuming transmission after a reboot. However, after about 8 hours, the ext3 filesystem was expecting to be checked, which bogged down the process even worse. Having all the files copied was a giant relief.

That being accomplished, I pulled the server system off the network and installed FreeBSD 8.1-RELEASE on it, hoping it had OpenSSH and NFS servers in the base install. Sure enough, it does. The install process was fairly easy, but I did face two hiccups along the way:

  1. Using the same hostname and IP address meant I had to delete the old SSH key from ~/.ssh/known_keys.
  2. By default, only non-superusers are allowed to log in directly via SSH. However, only users that are members of the “wheel” group are allowed to use “su”. I forgot that second part, so I had no way to gain superuser privileges through the network. On a headless system, that’s a no-go.

Once I had it set up for headless access, getting NFS configured was trivial for my purposes.

I started copying files back around 12:30pm. As I type this, it’s a little after 7:30pm, and I have yet to reboot the system due to a network lock-up. I expect all the files to be copied back before I go to bed tonight.

The score on my home network is now Linux 2, FreeBSD 1.

Update 12/23: It turns out Linux programmers aren’t the only ones who find the RealTek 8139 chips frustrating. While perusing the FreeBSD code for the 8139 driver, I found a comment at the top of a source file to the effect that the 8139 “brings new meaning to the term ‘low-end.'” Ouch.

Update 12/26: I’m done dealing with comments that say, “Gee, I didn’t have a problem with the 8139.” I don’t care how many problems you didn’t have with the 8139. I had persistent Ethernet lockup problems under a saturated network load, and it was trivial to induce a lockup. The same goes for comments saying I should have just put in another network card that I don’t have and can’t buy at this time. Those comments are finished. Any comments saying either of these things, will be deleted, no matter how much other good stuff might be in them.

About these ads

22 Comments

Leave a Comment
  1. AlphaX / Dec 22 2010 12:40 am

    I 100% agree with the realtek network cards being poorly supported in Linux… well when I mean poorly supported I really mean that the cards crash regularly. I have a Windows 2008 R2 server and a Linux server both running on the same hardware (Intel D510 based low power servers). Both servers have identical hardware firmware etc.. both suffer from network related ‘pauses’ or lock ups. I however appear to have reduced the problem to being next to non-existent on Linux by simply putting in an intel card.. ha ha..

  2. David Dreggors / Dec 22 2010 1:05 am

    1. Using the same hostname and IP address meant I had to delete the old SSH key from ~/.ssh/known_keys.

    Yeah that is less “hiccup” and more “should be expected” considering a new ssh key will be generated.

    2. By default, only non-superusers are allowed to log in directly via SSH. However, only users that are members of the “wheel” group are allowed to use “su”. I forgot that second part, so I had no way to gain superuser privileges through the network. On a headless system, that’s a no-go.

    This is very good practice. You should ssh in as a non-root user, then you can “su – root” (note not using sudo yet). Once you supply root password you simple “moduser -G wheel myuser” and now you have sudo access! Again, this is not a “hiccup”, it is as expected for security concerns!

    I wish Linux did this by default, it would save me the time of editing sshd_config.

    Hiccup is usually used to describe a problem. I see no problem here, all was acting as expected.

    Nice job on the move to BSD though, you should report how all goes after a while.
    Please report things like:

    1. Network speed differences (if any)
    2. How the OS handles constant reads/writes vs Linux
    3. How NFS in particular handles on BSD (again vs Linux)

    I would be interested to hear your take on these areas once you have used it for a while.

  3. its / Dec 22 2010 1:20 am

    Nice to see someone publicly appreciate the best server OS on the planet. I highly recommend ZFS for your file shares and VirtualBox (in the ports collection,) for virtualization. MediaTomb (also in the ports,) hosts up a sweet upnp server with transcoding for your living room.

  4. KLaus / Dec 22 2010 9:09 am

    Under set up when you are asked to add user accounts you are also asked if the user should be inviteted into other groups, like wheel, and then you have no problem. Next you should look at is sudo, then you will rarely need to use the root account at all.

  5. gus3 / Dec 22 2010 9:37 am

    Wow, thanks for the quick comments!

    I already knew (but had temporarily forgotten) about “su” being denied for all but the “wheel” group. The last time I used FreeBSD regularly was nine years ago (version 5?). So in this case, the “hiccup” was with my memory.

    I did notice one odd thing about NFS that I didn’t see with a Linux server. Using “rsync” to an NFS mount filled local cache quickly, then stalled while it flushed the entire cache through the network. The first symptom was that 35 MB/s on the “rsync” seems a bit fast for a 100baseT Ethernet, especially when that Ethernet shows as idle. When I remounted the NFS shares with “sync”, everything started behaving much more smoothly, between 11 and 12 MB/s of file data on a consistently saturated Ethernet.

    As I predicted, the entire transfer was finished before 8:30pm, so it took a little less than 8 hours of actual copying/rsync-ing to restore the server contents.

  6. wumux / Dec 22 2010 9:41 am

    I have used dozens of Realtek NICs thru the years, including 8138 w/o any problems. You could try ext4, xfs, reiserfs, and jfs if you had problems with ext3. And I hope a firewall was configured properly on you NFS server if you had any. In other words, I expect you were too quick to quit Linux.

  7. gus3 / Dec 22 2010 11:36 am

    I’ve been using Linux full-time at home for twelve years now. It wasn’t a conflict between the network card and a filesystem. The system has had over 90% uptime over the past three years, and served as my desktop system when I first built it. It has lots of wear on it, so it’s more susceptible to marginal behaviors now than when it was new.

    As for the firewall, that had little to nothing to do with anything. The network is totally trusted, since I don’t have any outside access right now.

    The RTL8139 driver in Linux is known to have high-load issues. The driver’s configuration options in the kernel configurator shows that the chip itself has been revised more than a few times. The driver itself is likely to cause the card to drop an Ethernet frame, resulting in an invalid state, rendering the card functionally off-line. I already supplied the link showing Ubuntu’s bug list for the 8139, but a Google search also turns up this Red Hat problem.

    The only logical conclusion is that Linux’s 8139too driver is defective.

    • Bob Robertson / Dec 26 2010 8:24 am

      “The only logical conclusion is that Linux’s 8139too driver is defective.”

      Not the only logical conclusion.

      Some other logical conclusions are that the 8139 is defective;
      and/or Realtek is defective for writing only proprietary drivers;
      and/or Realtek is defective for not releasing thorough specifications for the F/OSS programmers to write drivers.

      It really all depends upon where you wish to assume blame lies.

      Considering just how well F/OSS authors of all open OSs have done with squirrelly hardware over the decades with little or no help, and often what seems deliberate obfuscation, by manufacturers, I believe it to be long past time to put the blame squarely where it needs to be, loudly and often:

      Uncooperative hardware manufacturers.

  8. PB / Dec 26 2010 6:59 am

    Might have been easier to try a different nic. :-)

  9. qwerty / Dec 26 2010 8:39 am

    # sysctl -a | grep read_max
    vfs.read_max: 8

    # sysctl vfs.read_max=32

    # vi /etc/sysctl…

    Good luck, and remember, this isn’t Linux; you’re expect to do a little homework.

  10. Kurt / Dec 26 2010 10:47 am

    Curious as to why you didn’t just drop in a different, better supported network card? They are cheap, and it would have been so much easier and faster. ……

    Kurt

    • gus3 / Dec 26 2010 2:25 pm

      Because I don’t have one lying around, and I can’t afford to go buy one, not even a cheap one. If I’m going to use this system as a file server, I need to make it work with the hardware it has.

  11. Howard / Dec 26 2010 11:32 am

    I’m not sure what the deal is with the author and RealTek drivers. My file server (NFS and Samba), a 2007-vintage Dell Inspiron 530, came equipped with a built-in RealTek RTL8139 chipset. I have run nothing but Linux on this host (Ubuntu server edition first, now Slackware 13.1 x86-64), and I have had absolutely zero issues with the RealTek driver.

    8139too Fast Ethernet driver 0.9.28
    eth0: RealTek RTL8139 at 0xce00, 00:40:05:81:6c:59, IRQ 21
    eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
    eth0: no IPv6 routers present

    Btw, it is considered a security no-no to allow super users to login directly with ssh. Strangely, Slackware is setup by default to allow root to login via ssh, and you have to edit the /etc/ssh/sshd_config file to disable this ability.

    • gus3 / Dec 26 2010 2:42 pm

      The source code shipped for Slackware’s SSH package has a configuration that allows remote root login, shown as the default on line 39 of the configuration file sshd_config. This is true also for v5.6p1 from the OpenSSH source tarball. Also, the man page indicates that PermitRootLogin defaults to “yes”, even for the OpenBSD (non-portable) source.

  12. Stanislaus Brown / Dec 26 2010 5:32 pm

    So, the time taken to do all this, cost you less then a new NIC card? You really value your time cheaply. Before you get upset, a word of advice. You can buy network cards, you cannot buy a single second more of time.

    Next time, apply Occam’s Razor = the simplest solution is the best. At the very least, you could have borrowed a NIC from a friend and tested if the problem followed the NIC or the OS.

    I have nothing against FreeBSD. But don’t blame the NIC or the OS when you’re not applying the scientific method properly. You solved the wrong problem.

    Or at least, be a man and say you were bored and just wanted a change.

    I’ve never had a problem with RealTek NICs either.

    • gus3 / Dec 26 2010 5:50 pm

      How many ways do I have to put this? I HAVE NO MONEY TO SPARE. Zip, zero. Every penny I can put into my bank account is spoken for, for the next three months at least. I even moved this weblog to free hosting, because I knew this was coming.

      So Occam’s Razor suggests I find something else to spend in search of a solution; I spent the time I have, rather than money I don’t. If you’re so worried about whether or not I’m wasting my time (by your estimation), does it occur to you that you’re wasting your own time worrying about how I spend mine?

      But you’re right about one thing: I did want a change. I wanted a file server that wouldn’t hang at the least provocation. I now have that.

      I don’t care that you never had a problem with the 8139 under Linux. I did. It was trivially reproducible on my server, running 2.6.36 and 2.6.37-rcX kernels built from un-patched sources, as well as the Slackware 2.6.33.4 stock kernel. Looking through various bug-reporting sites, it’s obvious I’m not the only one that consistently had problems with the 8139 under Linux.

      That’s good enough for me. If not for you, too bad.

      • TucsonMatt / Dec 27 2010 2:16 am

        I agree with you, gus3. It always bugs me when people assume that because THEY have enough extra money to buy another NIC that you do to. Or, since THEY would rather spend money than time, you should feel the same way. And, like you, the thing that REALLY bugs me is when people assume since they’ve never had a problem with something that you having a problem means you screwed up somewhere.

        I’ve had many times when one of my clients experiences a problem with something that I’ve never had even though they’re using the same thing I do. Rather than tell them they must be nuts because I’ve never had the problem, I figure out whey they’re having the problem and fix it.

        The bottom line is, you had a problem and you fixed it in a way that works for you with the limitations and resources you had. Therefore, it was the best solution for you and the fact that someone else has never had the problem with the same hardware, or they have enough disposable income to do it a different way is immaterial.

        Thanks for the write-up and a reminder that while there is more than one way to skin a cat, in the end, the cat ends up skinned! :)

    • Micheas / Dec 27 2010 7:15 pm

      Although if you read the long screed about realtek ethernet cards in the source code for the freebsd driver, it is easy to see how other operating systems might not deal with the limitations of a card that “redefines lowend” ethernet cards.

      The author of the FreeBSD driver estimates that the 100baseT card uses about 300mhz of cpu when in operation.

      I can easily see a driver that does not have the limitations documented getting refactored so that it does not work reliably.

Trackbacks

  1. Tweets that mention Linux Out, FreeBSD In « I Am, Therefore I Think -- Topsy.com
  2. Links 23/12/2010: Linux/Android Gains in ARM, Nautilus 3.0 Mockups Debated | Techrights
  3. Linux News » Linux Out, FreeBSD In
  4. Links 30/12/2010: PlayStation 3 Cracked for Linux, 2011 Looks Great for Android | Techrights

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: