debugging network problems on firewall

[Cheryl Trooskin <sev@byz.org>]

I've been trying to debug a difficult-to-reproduce problem on my 
OpenBSD 2.7 firewall and I'm stumped as to where to look next.

The basic problem:  Occasionally, but not all the time, when I've got
fairly heavy network traffic going from outside to inside (for instance,
when there's ftp or rsync of large files or lots of files), the external 
interface just...stops moving packets.  There's nothing but the same 
overflow message I always get in the logfiles (below), ifconfig on 
the interface shows nothing unusual...but no traffic moves.  I get this
once every month or two, though we move data into the network at that
rate at least several times a week at the moment.

Whenever there's a lot of incoming traffic, the system message
buffer fills up with "le0: overflow":
Jan  6 13:57:07 firewall /bsd: le0: overflow
Jan  6 13:57:38 firewall last message repeated 70 times
Jan  6 13:59:22 firewall last message repeated 352 times
Jan  6 13:59:25 firewall last message repeated 277 times
Jan  6 13:59:25 firewall last message repeated 14 times

I see this far more often than I actually see the interface stop
working.  I've been working on the assumption that these messages and 
the interface problems are related.

I'm running a 2.7 kernel with only the earliest of the general patches
installed (obviously this is something I should remedy :), and ipf v.
3.3.14.

I'm really just not sure where to look next when I can't reproduce the
problem.  Actually, I'm not even sure what to look for next time I *do*
reproduce the problem. :)  Suggestions on how to reproduce it, or what to
look at/for next time it happens, would be greatly appreciated.  Or
even suggestions on how, other than rebooting, I might be able to
kick the interface into behaving again.

the kernel output:

penBSD 2.7 (conf) #7: Sun Jun 25 17:49:28 PDT 2000
    root@firewall:/usr/src/sys/arch/i386/conf
cpu0: F00F bug workaround installed
cpu0: Intel Pentium (P54C) ("GenuineIntel" 586-class) 150 MHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,MCE,CX8
real mem  = 16363520 (15980K)
avail mem = 12062720 (11780K)
using 225 buffers containing 921600 bytes (900K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 04/02/98
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Compaq Triflex Host-PCI" rev 0x01
pchb1 at pci0 dev 0 function 1 "Compaq Triflex Host-PCI" rev 0x01
pchb2 at pci0 dev 0 function 2 "Compaq Triflex Host-PCI" rev 0x01
pchb3 at pci0 dev 0 function 3 "Compaq Triflex Host-PCI" rev 0x01
pchb4 at pci0 dev 0 function 4 "Compaq Triflex Host-PCI" rev 0x01
pchb5 at pci0 dev 0 function 5 "Compaq Triflex Host-PCI" rev 0x01
pchb6 at pci0 dev 0 function 6 "Compaq Triflex Host-PCI" rev 0x01
pchb7 at pci0 dev 0 function 7 "Compaq Triflex Host-PCI" rev 0x01
le0 at pci0 dev 11 function 0 "AMD 79c970 PCnet-PCI LANCE" rev 0x02
le0: address 00:80:5f:66:28:e8
le0: 8 receive buffers, 2 transmit buffers
le0: interrupting at irq 5
ncr0 at pci0 dev 12 function 0 "Symbios Logic 53c810" rev 0x02: fast scsi, irq 1
0
scsibus0 at ncr0: 8 targets
probe(ncr0:0:0): 5.0 MB/s (200 ns, offset 8)
sd0 at scsibus0 targ 0 lun 0: <COMPAQ, ST15150N, 5216> SCSI2 0/direct fixed
sd0(ncr0:0:0): 10.0 MB/s (100 ns, offset 8)
sd0: 4094MB, 3712 cyl, 21 head, 107 sec, 512 bytes/sec, 8386000 sec total
cd0 at scsibus0 targ 5 lun 0: <COMPAQ, CRD-254V, 1.06> SCSI2 5/cdrom removable
probe(ncr0:5:1): asynchronous.
rl0 at pci0 dev 14 function 0 "Accton Technology MPX 5030/5038" rev 0x10: irq 3
address 00:e0:29:1c:b8:84
rlphy0 at rl0 phy 0: RTL internal phy
"Compaq PCI-EISA" rev 0x07 at pci0 dev 15 function 0 not configured
eisa0 at mainbus0
eisa0: can't map I/O space for slot 5
eisa0: can't map I/O space for slot 7
isa0 at mainbus0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
vt0 at isa0 port 0x60/16 irq 1: vga 80/132 col, mono, 8 scr, mf2-kbd
pms0 at vt0 irq 12
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 440 netmask 468 ttymask 14ea
pctr: 586-class performance counters and user-level cycle counter enabled
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
IP Filter: already initialized