request for help tracking down elusive problem

[Simon Lok <simonlok@mac.com>]

I have numerous OpenBSD machines setup as routers.  Everything
works fine most of the time.  Once in a while, a machine "stops
responding."  Pings to all interfaces return.  No high level services
respond (not even SSH).  If you walk up to the console and hit enter,
the login prompt scrolls up, maybe once or twice, and then that's
it, it's over, you hit enter and nothing happens.  Meanwhile, ping
still responds and no SSH, nor web, not anything else other
than ping.

Once or twice I had a person report and interesting clue, he said
that the machine was routing traffic fine, then over the period of
about 30 minutes it start to slow down and slow down and then
finally his transfer stalled.  This could just be his download
speedometer in his browser or whatever not registering a stall,
or maybe not.

My first thought is that maybe I'm somehow forkbombing the
routers.  Is there a setting that might prevent this?  I recall
from my Solaris days there was something like this but I
have no idea where to even begin to look for something like
this on OpenBSD.

My second though was it might be this NMBCLUSTERS
business.  Would a lack of NMBCLUSTERS result in
behavior where ping would respond but nothing else?  I
am not getting any messages to the console or any logs
that give me any details.

The odd thing is that I think this has happened on different
architectures even.  So it's not like it only occurs on one of
our batches of machines.  Attached is a dmesg from one of our
machines.  Any help would be very much appreciated.

--SL

OpenBSD 3.4-stable (GENERIC) #8: Tue Jan 20 08:42:35 EST 2004
     root@localhost.localdomain:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III (Tualatin) ("GenuineIntel" 686-class) 1.41 GHz
cpu0:  
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36, 
MMX,FXS
real mem  = 535277568 (522732K)
avail mem = 490446848 (478952K)
using 4278 buffers containing 26865664 bytes (26236K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 10/09/02, BIOS32 rev. 0 @  
0xfdb80
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev. 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev. 1.0 @ 0xf3790/224 (12 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801AA LPC" rev  
0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x6000!
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82815 Hub" rev 0x04: rng active,  
7Kb/sec
vga1 at pci0 dev 2 function 0 "Intel 82815 Graphics" rev 0x04: aperture  
at 0xf8000000
wsdisplay0 at vga1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x05
pci1 at ppb0 bus 1
iop0 at pci1 dev 0 function 0 "DPT SmartRAID (I2O)" rev 0x02: I2O  
adapter <ADAPTEC 24
iop0: interrupting at irq 11
ppb1 at pci1 dev 0 function 1 "DPT PCI-PCI bridge" rev 0x02
pci2 at ppb1 bus 2
fxp0 at pci1 dev 3 function 0 "Intel 82557" rev 0x08: irq 3, address  
00:30:48:41:e8:0
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
fxp1 at pci1 dev 4 function 0 "Intel 82557" rev 0x08: irq 11, address  
00:30:48:41:e8:
inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 4
pcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x05
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x05: DMA,  
channel 0 wired
ired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x05: irq 3
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
"Intel 82801BA SMBus" rev 0x05 at pci0 dev 31 function 3 not configured
uhci1 at pci0 dev 31 function 4 "Intel 82801BA USB2" rev 0x05: irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lm0 at isa0 port 0x290/8: W83627HF
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
px0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask c48 netmask c48 ttymask c4a
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
iop0: configuring...
ioprbs0 at iop0 tid 521: <ADAPTEC, RAID-5, 3A0L> direct access, fixed
scsibus0 at ioprbs0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <I2O, Container #00, > SCSI2 0/direct  
fixed
sd0: 572346MB, 72963 cyl, 255 head, 63 sec, 512 bytes/sec, 1172164608  
sec total
device (class 0x80) at iop0 tid 8 not configured
device (class 0x80) at iop0 tid 9 not configured
device (class 0x80) at iop0 tid 10 not configured
device (class 0x80) at iop0 tid 11 not configured
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02