NFS send error #55

[Nicholas Merrill <lists_(_at_)_mojo_(_dot_)_calyx_(_dot_)_net>]

Hi All

I've been struggling with this NFS problem for a while, and I've attempted
to deal with it in several ways but to no avail so far.

My setup is this-  I've got an NFS server running FreeBSD 3.0-RELEASE
(only because I needed an OS with drivers for a DPT Raid PCI card ;)  )
and it is serving up dedicated share partitions to 4 multi-purpose clients
(shell, mail, web hosting) which are all OpenBSD 2.5

On one of the OpenBSD NFS clients, which happens to be the most active
one, I get the following errors constantly:

Jun 17 15:29:47 host /bsd: nfs send error 55 for server
192.x.x.x:/share/host
Jun 17 15:29:47 host last message repeated 3 times
Jun 17 15:47:55 host last message repeated 185 times

I've tried tweaking some sysctl values in an attempt to fix the problem,
using the commands:

/usr/sbin/sysctl -w net.inet.udp.sendspace=65535
/usr/sbin/sysctl -w net.inet.udp.recvspace=65535

on the clients, and

/sbin/sysctl -w net.inet.udp.recvspace=65535

on the server (FreeBSD only has recvspace and not sendspace for whatever
reason)

but that has not helped.  I also tried using TCP instead of UDP mounts
but that was actually worse.

The NFS service is using it's own dedicated switch, and each machine
has it's own 10Mbps switch port.  I am running MRTG against that switch,
and it tells me that the NFS server averages 180.1Kb/s incoming and 99.7Kb
outgoing over the course of the month.  I was considering moving the NFS
server to the 100Mbps port, but at these speeds it seems pointless.

All the machines are using Kingston PCI NIC's based on Dec 21140 chips.

The NFS server always has a very low load, like 0.00.  It's a 300Mhz PII
with 128MB of RAM, and does nothing but run nfsd, syslogd, rpc.lockd,
sshd, mountd, portmap and crond.  The shared storage is RAID level 5
striped across 9 or so 4GB disks.

Right at this moment there are 20 nfsd processes on the server, and
10 nfsiod processes on the problem server.  Should I add more nfsiod's
or something?

Thanks for any advice that anybody can offer.

----------------------------------------------------------------------------
Nicholas Merrill       http://www.calyx.net            Voice: 212-966-1900
President / CEO        http://www.calyx.nl             Fax  : 212-966-3965
Calyx Internet Access  13 Laight St. NYC, NY 10013     Email: nick_(_at_)_calyx_(_dot_)_net
----------------------------------------------------------------------------