[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
kernel/3696: Kernel panic while syncing disks
- To: gnats@openbsd.org
- Subject: kernel/3696: Kernel panic while syncing disks
- From: Søren Thing Andersen <soeren@thing.dk>
- Date: Mon, 1 Mar 2004 22:59:41 +0100 (CET)
- Resent-Date: Mon, 1 Mar 2004 15:10:04 -0700 (MST)
- Resent-From: gnats@cvs.openbsd.org (GNATS Filer)
- Resent-Message-Id: <200403012210.i21MA4tZ005929@cvs.openbsd.org>
- Resent-Reply-To: gnats@cvs.openbsd.org, soeren@thing.dk
- Resent-To: bugs@cvs.openbsd.org
>Number: 3696
>Category: kernel
>Synopsis: Kernel panic while syncing disks: fatal page fault in supervisor mode
>Confidential: yes
>Severity: non-critical
>Priority: low
>Responsible: bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Mon Mar 01 22:10:02 GMT 2004
>Closed-Date:
>Last-Modified:
>Originator: Søren Thing Andersen
>Release: OpenBSD 3.4 GENERIC#1 i386
>Organization:
net
>Environment:
System : OpenBSD 3.4
Architecture: OpenBSD.i386
Machine : i386
OPENBSD_3_4 Stable from February 6th 2004 +current (20031202) sys/dev/pci/{if_bge,pcidevs}* sys/dev/mii/mii*
>Description:
httpd child processes could not be killed, presumably because
my NFS-server had a problem with its network card.
I decided to reboot the webserver to get it running again.
While rebooting it panic'ed. Output from serial console, gdb where,
vmstat and ps:
Mar 1 15:37:32 moses reboot: rebooted by thing
/etc/rc.shutdown in progress...
/etc/rc.shutdown complete.
Mar 1 15:37:33 moses syslogd: exiting on signal 15
syncing disks... uvm_fault(0xe41a9eec, 0x0, 0, 1) -> e
fatal page fault in supervisor mode
trap type 6 code 0 eip d02673c0 cs 8 eflags 10246 cr2 8 cpl 0
panic: trap type 6, code=0, pc=d02673c0
Starting stack trace...
panic(0,e41d0b4c,6,d02673c0,8,10246,e4296ad0,d02df743,d04804fb,6,0,d02673c0,
0,d14eb400,d0b7bf00,d14eb400,d0b7bf00,d01d7d15,0,0,0,e41a9eec,e41a9eec,8,0,1,
1,e4295000,0,0,e4296b48,d0100ced,d01d0010,10,d0b7bf00,d14eb400,e4296b48,0,0,
e45c76e0,0,6,0,d02673c0,8,10246,e41d0b4c,d14eb400,d0b7bf00,0,0,0,0,0,d0bbda00,
0,4,1,0,0,e4296bf8) at panic+0x8e
panic(d04804fb,6,0,d02673c0,0) at panic+0x8e
trap() at trap+0x243
--- trap (number 6) ---
nfs_reply(d14eb400,d0b71200,d1073e00,d14eb400,e4296c58) at nfs_reply+0xcc
nfs_request(e4155308,d0b71c00,4,e41d0b4c,e450ea58) at nfs_request+0x400
nfs_access(e4296c94,e41d0b4c,e4296ca8,d01e8a3c,0) at nfs_access+0x426
VOP_ACCESS(e4155308,40,e450ea58,e41d0b4c,e4155308) at VOP_ACCESS+0x3b
nfs_lookup(e4296de8,e41d0b4c,e4296df8,d01e1f14,e4296ee0) at nfs_lookup+0x171
VOP_LOOKUP(e4155308,e4296ef0,e4296f04,e41d0b4c,e3c68180) at VOP_LOOKUP+0x35
lookup(e4296ee0,d0cc0c2e,1a,e4296ef8,e4296f88) at lookup+0x275
namei(e4296ee0,ffffffff,ffffffff,ffffffff,e41d0b4c) at namei+0x333
sys_access(e41d0b4c,e4296f88,e4296f7c,1368,2d5dc5c8) at sys_access+0x6f
syscall() at syscall+0x21d
--- syscall (number 33) ---
0xf97088d:
End of stack trace.
GNU gdb 4.16.1
Copyright 1996 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd3.4".
(gdb) file /var/crash/bsd.0
Reading symbols from /var/crash/bsd.0...(no debugging symbols found)...done.
(gdb) target kcore /var/crash/bsd.0.core
panic: trap type 6, code=0, pc=d02673c0
#0 0x1000 in ?? ()
(gdb) where
#0 0x1000 in ?? ()
#1 0xd02d8032 in boot ()
#2 0xd01c5bf3 in panic ()
#3 0xd02df743 in trap ()
#4 0xd0100ced in alltraps ()
#5 0xd0267a88 in nfs_request ()
#6 0xd0270f76 in nfs_access ()
#7 0xd01e8c63 in VOP_ACCESS ()
#8 0xd0273359 in nfs_lookup ()
#9 0xd01e8b21 in VOP_LOOKUP ()
#10 0xd01e0d5d in lookup ()
#11 0xd01e0a33 in namei ()
#12 0xd01e6737 in sys_access ()
#13 0xd02dfe49 in syscall ()
#14 0xd0100d93 in Xsyscall ()
can not access 0xcfbed6ec, invalid address (cfbed6ec)
can not access 0xcfbed6ec, invalid address (cfbed6ec)
Cannot access memory at address 0xcfbed6ec.
(gdb)
sudo vmstat -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 8424 4120 23148052 1280 201
32 6964 332 7187062 640 422
64 2711 809 12712486 320 1097
128 583 409 6705415 160 7815
256 5966 338 3255341 80 45741
512 712 632 1316821 40 377508
1024 306 202 86276460 20 53639945
2048 10 80 368487 10 178381
4096 27 296 144553 5 144414
8192 68 212 297264 5 295574
16384 363 0 141583 5 0
32768 1 0 1063 5 0
65536 0 0 95 5 0
131072 0 0 3 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, ifaddr, sysctl, namecache, UFS mount, sem,
proc, in_multi, exec, xform_data, VM swap, UVM amap, UVM aobj, USB,
temp
32 devbuf, pcb, routetbl, fragtbl, ifaddr, vnodes, UFS mount, sem,
VFS cluster, exec, xform_data, VM swap, UVM amap, USB, temp
64 devbuf, pcb, routetbl, ifaddr, NFS req, shm, sem, lockf, in_multi,
pfkey data, UVM amap, USB, NDP, temp
128 devbuf, routetbl, ifaddr, vnodes, shm, proc, ttys, exec, inodedep,
UVM amap, USB, USB device, NDP, temp
256 devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, VM map, file,
file desc, NFS srvsock, NFS daemon, ttys, newblk, UVM amap, temp
512 devbuf, pcb, ifaddr, ioctlops, mount, NFS mount, UFS mount, file desc,
proc, exec, UVM amap, USB device, temp
1024 devbuf, namei, ioctlops, file, file desc, ttys, exec, pagedep,
UVM amap, UVM aobj, crypto data, temp
2048 devbuf, UFS mount, file, file desc, MSDOSFS mount, VM swap, UVM amap
4096 devbuf, NFS node, namecache, UFS quota, UFS mount, file, ISOFS mount,
inodedep, UVM amap, temp
8192 namecache, file, UVM amap
16384 devbuf, indirdep, UVM amap
32768 devbuf, UVM amap
65536 UVM amap
131072 UVM amap
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 2083 475K 475K 39243K 2134 0 0 16,32,64,128,256,512,1024,2048,4096,16384,32768
pcb 189 24K 267K 39243K 1641034 0 0 16,32,64,256,512
routetbl 10786 1516K 1516K 39243K 27429 0 0 16,32,64,128,256
fragtbl 0 0K 1K 39243K 43775 0 0 32
ifaddr 109 22K 22K 39243K 113 0 0 16,32,64,128,256,512
sysctl 2 1K 1K 39243K 2 0 0 16,256
namei 128 128K 130K 39243K 85412215 0 0 1024
ioctlops 0 0K 1K 39243K 150 0 0 512,1024
mount 5 3K 4K 39243K 10 0 0 512
NFS req 128 8K 8K 39243K 4081755 0 0 64
NFS mount 0 0K 2K 39243K 3 0 0 512
NFS node 1 4K 4K 39243K 1 0 0 4096
vnodes 56 8K 46K 39243K 37161 0 0 32,128,256
namecache 3 13K 13K 39243K 3 0 0 16,4096,8192
UFS quota 1 4K 4K 39243K 1 0 0 4096
UFS mount 21 37K 37K 39243K 21 0 0 16,32,512,2048,4096
shm 2 1K 1K 39243K 2 0 0 64,128
VM map 4 1K 1K 39243K 4 0 0 256
sem 2 1K 2K 39243K 857284 0 0 16,32,64
file 0 0K 8K 39243K 15578 0 0 256,1024,2048,4096,8192
file desc 128 64K 77K 39243K 284525 0 0 256,512,1024,2048
lockf 83 6K 12K 39243K 1199940 0 0 64
proc 6 2K 2K 39243K 1688 0 0 16,128,512
VFS cluster 0 0K 5K 39243K 2382339 0 0 32
NFS srvsock 2 1K 1K 39243K 2 0 0 256
NFS daemon 1 1K 1K 39243K 1 0 0 256
in_multi 22 1K 1K 39243K 22 0 0 16,64
ISOFS mount 1 4K 4K 39243K 1 0 0 4096
MSDOSFS mount 1 2K 2K 39243K 1 0 0 2048
ttys 330 193K 193K 39243K 330 0 0 128,256,1024
exec 0 0K 3K 39243K 635673 0 0 16,32,128,512,1024
pfkey data 1 1K 1K 39243K 2 0 0 64
xform_data 0 0K 1K 39243K 1764 0 0 16,32
pagedep 1 1K 1K 39243K 1 0 0 1024
inodedep 1 4K 5K 39243K 298 0 0 128,4096
newblk 1 1K 1K 39243K 1 0 0 256
indirdep 0 0K 240K 39243K 4624 0 0 16384
VM swap 5 3K 3K 39243K 5 0 0 16,32,2048
UVM amap 11966 5278K 6690K 39243K 35815129 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072
UVM aobj 2 2K 2K 39243K 2 0 0 16,1024
USB 12 1K 1K 39243K 12 0 0 16,32,64,128
USB device 4 2K 2K 39243K 4 0 0 128,512
crypto data 1 1K 1K 39243K 1 0 0 1024
NDP 24 2K 2K 39243K 25 0 0 64,128
temp 23 5K 10K 39243K 9109620 0 0 16,32,64,128,256,512,1024,4096
Memory Totals: In Use Free Requests
7806K 3819K 141554685
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 40 1298 0 838 8 1 7 7 0 inf 0
extentpl 20 52616 0 52284 6 0 6 6 0 inf 0
pmappl 72 380256 0 380126 9 5 4 4 0 inf 0
vmsppl 212 380256 0 380126 31 21 10 12 0 inf 1
vmmpepl 88 45524335 0 45502510 1496 900 596 596 0 inf 69
vmmpekpl 88 992817 0 992181 17 1 16 16 0 inf 0
uaoeltpl 84 8 0 8 1 0 1 1 0 inf 1
aobjpl 52 1 0 0 1 0 1 1 0 inf 0
amappl 40 10405840 0 10401869 83 26 57 57 0 inf 2
bufpl 116 20469 0 20469 11 7 4 4 0 inf 4
mbpl 256 116553820 0 116553163 223 169 54 89 1 inf 2
mclpl 2048 51338937 1 51338645 478 313 165 260 4 1024 19
sockpl 200 1641029 0 1640727 94 67 27 56 0 inf 2
procpl 320 380268 0 380126 42 26 16 20 0 inf 0
zombiepl 72 380126 0 380126 3 2 1 2 0 inf 1
ucredpl 80 79020 0 78890 10 6 4 5 0 inf 1
pgrppl 24 19043 0 19039 1 0 1 1 0 inf 0
sessionpl 48 18090 0 18086 2 1 1 2 0 inf 0
pcredpl 24 380268 0 380126 4 2 2 2 0 inf 0
filepl 48 44556466 0 44555981 26 14 12 22 0 inf 0
fdescpl 320 380269 0 380126 44 27 17 20 0 inf 1
pipepl 76 882858 0 882855 7 6 1 3 0 inf 0
sigapl 308 380256 0 380126 37 23 14 17 0 inf 0
wdcspl 96 11446230 0 11446229 1 0 1 1 0 inf 0
scxspl 128 5 0 5 1 1 0 1 0 inf 0
vnodes 192 2556 0 0 122 0 122 122 0 inf 0
nchpl 64 1310 0 0 21 0 21 21 0 inf 0
ffsino 280 1939644 0 1937104 186 4 182 182 0 inf 0
pagedeppl 64 130629 0 130629 3 2 1 2 0 inf 1
inodedeppl 84 235876 0 235876 41 36 5 15 0 inf 5
newblkpl 32 1325512 0 1325512 2 1 1 1 0 inf 1
bmsafemappl 32 107473 0 107473 2 1 1 1 0 inf 1
allocdirectpl 60 343845 0 343845 22 20 2 13 0 inf 2
indirdeppl 28 8304 0 8304 3 2 1 1 0 inf 1
allocindirpl 52 981667 0 981667 493 463 30 169 0 inf 30
freefragpl 32 63167 0 63167 1 0 1 1 0 inf 1
freeblkspl 108 193491 0 193491 58 56 2 19 0 inf 2
freefilepl 28 154163 0 154163 10 9 1 5 0 inf 1
diraddpl 32 191605 0 191605 7 5 2 4 0 inf 2
mkdirpl 28 58986 0 58986 2 1 1 1 0 inf 1
dirrempl 32 192351 0 192351 12 11 1 6 0 inf 1
nfsnodepl 264 26668 0 26668 122 121 1 47 0 inf 1
semapl 64 110 0 110 1 0 1 1 0 inf 1
semupl 100 409627 0 409627 1 0 1 1 0 inf 1
ipqepl 20 501922 0 501922 2 1 1 1 0 inf 1
tcpcbpl 356 763468 0 763411 166 155 11 95 0 inf 2
sackhlpl 20 43199 0 43199 11 10 1 1 0 inf 1
plimitpl 152 7047 0 7044 1 0 1 1 0 inf 0
kqeuepl 192 2 0 2 2 2 0 1 0 inf 0
knotepl 64 4 0 4 2 2 0 1 0 inf 0
In use 4394K, total allocated 5636K; utilization 78.0%
sudo ps -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -axww -O paddr
PID PADDR TT STAT TIME COMMAND
23449 2449a6b8 ?? R> 0:00.00 (httpd)
13990 241d0dcc ?? R> 0:00.00 (httpd)
4559 24213010 ?? R> 0:00.00 (httpd)
12967 241d050c ?? R> 0:00.00 (httpd)
20723 24213510 ?? R> 0:00.00 (httpd)
31035 24184288 ?? R> 0:00.00 (httpd)
22346 24184788 ?? R> 0:00.00 (httpd)
21529 241d0b4c ?? R> 0:00.00 (httpd)
20603 241d014c ?? R> 0:00.00 (httpd)
12069 241d0c8c ?? R> 0:00.00 (httpd)
28386 241d064c ?? R> 0:00.00 (httpd)
14041 241d0a0c ?? R> 0:00.00 (httpd)
28588 241d08cc ?? R> 0:00.00 (httpd)
14211 241d078c ?? R> 0:00.00 (httpd)
16018 241848c8 ?? R> 0:00.00 (httpd)
26932 240bec84 ?? R> 0:00.00 (httpd)
3268 24213a10 ?? R> 0:00.00 (httpd)
5910 242683d4 ?? R> 0:00.00 (httpd)
7028 240be8c4 ?? R> 0:00.00 (httpd)
31908 24268b54 ?? R> 0:00.00 (httpd)
27178 242688d4 ?? R> 0:00.00 (httpd)
29448 24268a14 ?? R> 0:00.00 (httpd)
32485 24268dd4 ?? R> 0:00.00 (httpd)
19777 24268794 ?? R> 0:00.00 (httpd)
29067 2455a55c ?? R> 0:00.00 (httpd)
19477 2455a19c ?? R> 0:00.00 (httpd)
22166 240bea04 ?? R> 0:00.00 (httpd)
29150 24184648 ?? R> 0:00.00 (httpd)
29608 24184c88 ?? R> 0:00.00 (httpd)
9048 2455a41c ?? R> 0:00.00 (httpd)
12798 2455acdc ?? R> 0:00.00 (httpd)
18709 2455a69c ?? R> 0:00.00 (httpd)
6527 24213150 ?? R> 0:00.00 (httpd)
5667 2455a91c ?? R> 0:00.00 (httpd)
28288 24184508 ?? R> 0:00.00 (httpd)
28459 24184a08 ?? R> 0:00.00 (httpd)
28741 2455a7dc ?? R> 0:00.00 (httpd)
13596 2455ae1c ?? R> 0:00.00 (httpd)
13804 241bab94 ?? R> 0:00.00 (httpd)
27824 241ba914 ?? R> 0:00.00 (httpd)
6051 2449a2f8 ?? R> 0:00.00 (httpd)
23321 241ba054 ?? R> 0:00.00 (httpd)
19873 241ba414 ?? R> 0:00.00 (httpd)
6718 2455ab9c ?? R> 0:00.00 (httpd)
15481 242138d0 ?? R> 0:00.00 (httpd)
28340 2455aa5c ?? R> 0:00.00 (httpd)
16465 2449a578 ?? R> 0:00.00 (httpd)
13885 2449ae38 ?? R> 0:00.00 (httpd)
31427 2449a078 ?? R> 0:00.00 (httpd)
13133 241ba7d4 ?? R> 0:00.00 (httpd)
13395 241ba2d4 ?? R> 0:00.00 (httpd)
31463 24529cd8 ?? R> 0:00.00 (httpd)
29947 245292d8 ?? R> 0:00.00 (httpd)
7924 24585060 ?? R> 0:00.00 (httpd)
16316 24585ce0 ?? R> 0:00.00 (httpd)
7080 24506910 ?? R> 0:00.00 (httpd)
24642 24506550 ?? R> 0:00.00 (httpd)
12245 24506cd0 ?? R> 0:00.00 (httpd)
16138 24506050 ?? R> 0:00.00 (httpd)
10611 245067d0 ?? R> 0:00.00 (httpd)
10355 24506190 ?? R> 0:00.00 (httpd)
2859 24506b90 ?? R> 0:00.00 (httpd)
1728 245062d0 ?? R> 0:00.00 (httpd)
27080 2455a05c ?? R> 0:00.00 (httpd)
21652 24506a50 ?? R> 0:00.00 (httpd)
22639 24585e20 ?? R> 0:00.00 (httpd)
15947 24506e10 ?? R> 0:00.00 (httpd)
31491 245852e0 ?? R> 0:00.00 (httpd)
9596 245856a0 ?? R> 0:00.00 (httpd)
2769 24585a60 ?? R> 0:00.00 (httpd)
22682 24585420 ?? R> 0:00.00 (httpd)
25356 24585ba0 ?? R> 0:00.00 (httpd)
31505 241ba554 ?? R> 0:00.00 (httpd)
15987 241bacd4 ?? R> 0:00.00 (httpd)
25577 241bae14 ?? R> 0:00.00 (httpd)
9735 245851a0 ?? R> 0:00.00 (httpd)
10116 24506410 ?? R> 0:00.00 (httpd)
16357 240beb44 ?? R> 0:00.00 (httpd)
24012 245857e0 ?? R> 0:00.00 (httpd)
8386 24585920 ?? R> 0:00.00 (httpd)
26810 24529918 ?? R> 0:00.00 (httpd)
2421 24529b98 ?? R> 0:00.00 (httpd)
4012 24529e18 ?? R> 0:00.00 (httpd)
10468 24529198 ?? R> 0:00.00 (httpd)
15980 24529058 ?? R>s 0:00.00 (sftp-server)
12293 2455a2dc ?? R> 0:00.00 (httpd)
20622 241ba694 ?? R> 0:00.00 (httpd)
18732 24268514 ?? R> 0:00.00 (httpd)
2468 24529a58 ?? R> 0:00.00 (httpd)
9199 2449aa78 ?? R> 0:00.00 (httpd)
13005 24268654 ?? R> 0:00.00 (httpd)
19343 24585560 ?? R> 0:00.00 (httpd)
32586 24538a94 ?? R> 0:00.00 (httpd)
13007 2449abb8 ?? R> 0:00.00 (httpd)
20433 2449acf8 ?? R> 0:00.00 (httpd)
3046 2449a7f8 ?? R> 0:00.00 (httpd)
5816 2449a1b8 ?? R> 0:00.00 (httpd)
1 2406d000 ?? I>s 0:00.00 (init)
15290 245297d8 ?? R> 0:00.00 (httpd)
6358 24538314 ?? R> 0:00.00 (httpd)
9107 2456a6dc ?? R> 0:00.00 (httpd)
30250 245c71e0 ?? R> 0:00.00 (httpd)
21306 245c7460 ?? R> 0:00.00 (httpd)
21565 241baa54 ?? R> 0:00.00 (httpd)
19259 245c70a0 ?? R> 0:00.00 (httpd)
18225 2449a438 ?? R> 0:00.00 (httpd)
25007 245c7be0 ?? R> 0:00.00 (httpd)
10584 245c75a0 ?? R> 0:00.00 (httpd)
25801 245c7320 ?? R> 0:00.00 (httpd)
16603 24538094 ?? R> 0:00.00 (httpd)
18147 2456a09c ?? R> 0:00.00 (httpd)
4906 245c7d20 ?? R> 0:00.00 (httpd)
25511 245c76e0 ?? R> 0:00.00 (httpd)
4841 245c7aa0 ?? R> 0:00.00 (httpd)
18336 24538e54 ?? R> 0:00.00 (httpd)
22884 24685824 ?? R> 0:00.00 (httpd)
21242 2449a938 ?? R> 0:00.00 (httpd)
3347 24268c94 ?? R> 0:00.00 (httpd)
17466 245c7820 ?? R> 0:00.00 (httpd)
11659 245c7e60 ?? R> 0:00.00 (httpd)
27405 24529698 ?? R> 0:00.00 (httpd)
7627 24529558 ?? R> 0:00.00 (httpd)
27597 24529418 ?? R> 0:00.00 (httpd)
18201 24538594 ?? R> 0:00.00 (httpd)
10686 24538814 ?? R> 0:00.00 (httpd)
8573 245381d4 ?? R> 0:00.00 (httpd)
5026 2456a81c ?? R> 0:00.00 (httpd)
677 2456a95c ?? R> 0:00.00 (httpd)
15969 246855a4 ?? R> 0:00.00 (httpd)
1330 240be284 p1- R> 0:00.00 (reboot)
If I missed anything please ask for more details.
>How-To-Repeat:
Unknown
>Fix:
Unknown
>Release-Note:
>Audit-Trail:
>Unformatted: