[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Network performance openbsd 3.6



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there,

I`ve been using openbsd for quite some time now.
I used to tweak my machines alot to get as much out of it as i could.
But since the 3.6 release some options has been removed from the
kernel and been replaced by sysctl settings.
There for i was trying to figure out what option would actually raise
the nmbclusters.
Unfortunally i wasn`t able to find any documentation on this and from
reading the mailing list alot of people say you shouldn`t tweak
anything anymore and just be happy with the defaults.
After talking to some people they pointed out to tune kern.maxclusters
what would be in effect nmbclusters.

My main concern at the moment is this weird report netstat gives me.

- - ( Test machine ) Bridged setup between em interfaces and 1 fxp
management interface

With a newly build kernel without unnessecary hardware as well as no
options tweaked.

OpenBSD z1c-a20.fw 3.6 Z1CA20.MP#4 i386
real mem  = 1072717824 (1047576K)
avail mem = 975048704 (952196K)
mainbus0: Intel MP Specification (Version 1.4) (INTEL    Lindenhurst )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200 MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,PNI,MWAIT,CNXT-ID

reports:
=====

netstat -m :

602 mbufs in use:
~        576 mbufs allocated to data
~        1 mbuf allocated to packet headers
~        25 mbufs allocated to socket names and addresses
576/592/6144 mbuf clusters in use (current/peak/max)
1344 Kbytes allocated to network (96% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

vmstat -m:

Memory statistics by bucket size
~    Size   In Use   Free           Requests  HighWater  Couldfree
~      16     2096    720              65135    1280          0
~      32      392   1272              18746     640        843
~      64     1174    106              18718     320          0
~     128     1335     73              13642     160          0
~     256      185     23               4525      80          0
~     512     1255     97              10125      40       3693
~    1024      246     34               2076      20        269
~    2048       27      1                232      10          0
~    4096       24      1                375       5          0
~    8192        3      0                  3       5          0
~   16384        7      0                  7       5          0
~   32768        3      0                  3       5          0

Memory usage type by bucket size
~    Size  Type(s)
~      16  devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, namecache,
UFS mount,
~          dirhash, proc, in_multi, exec, xform_data, VM swap, UVM amap,
~          UVM aobj, USB, packet tags, temp
~      32  devbuf, pcb, routetbl, ifaddr, vnodes, UFS mount, sem, dirhash,
~          VFS cluster, ether_multi, xform_data, VM swap, UVM amap,
USB, temp
~      64  devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash,
in_multi,
~          exec, pfkey data, UVM amap, USB, NDP, temp
~     128  devbuf, routetbl, ifaddr, vnodes, UFS mount, dirhash, ttys,
UVM amap,
~          USB, USB device, NDP
~     256  devbuf, routetbl, ifaddr, sysctl, ioctlops, vnodes, shm, VM
map,
~          dirhash, file desc, NFS srvsock, NFS daemon, ttys, newblk,
UVM amap,
~          USB, USB device, temp
~     512  devbuf, pcb, ifaddr, ioctlops, mount, UFS mount, shm, dirhash,
~          file desc, proc, exec, UVM amap, USB device, temp
~    1024  devbuf, sysctl, ioctlops, file, ttys, exec, UVM amap, UVM aobj,
~          crypto data, temp
~    2048  devbuf, ifaddr, ioctlops, UFS mount, proc, pagedep, VM swap,
UVM amap,
~          temp
~    4096  devbuf, ioctlops, UFS mount, temp
~    8192  devbuf, MSDOSFS mount
~   16384  devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount,
~          inodedep
~   32768  devbuf, namecache

Memory statistics by type                           Type  Kern
~          Type InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
~        devbuf  4528  1016K   1017K 39322K    10228    0     0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
~           pcb    99     8K     10K 39322K     1195    0     0
16,32,64,512
~      routetbl    83     7K      8K 39322K     1813    0     0
16,32,64,128,256
~        ifaddr   115    19K     19K 39322K      117    0     0
16,32,64,128,256,512,2048
~        sysctl     3     2K      2K 39322K        3    0     0
16,256,1024
~      ioctlops     0     0K      4K 39322K     2111    0     0
256,512,1024,2048,4096
~         mount     5     3K      3K 39322K        7    0     0  512
~      NFS node     1    16K     16K 39322K        1    0     0  16384
~        vnodes    53     8K     46K 39322K     5064    0     0
16,32,128,256
~     namecache     3    49K     49K 39322K        3    0     0
16,16384,32768
~     UFS quota     1    16K     16K 39322K        1    0     0  16384
~     UFS mount    21    49K     49K 39322K       21    0     0
16,32,64,128,512,2048,4096,16384
~           shm     2     1K      1K 39322K        2    0     0  256,512
~        VM map     3     1K      1K 39322K        3    0     0  256
~           sem     2     1K      1K 39322K        2    0     0  32,64
~       dirhash    57    11K     51K 39322K      642    0     0
16,32,64,128,256,512
~          file     0     0K      1K 39322K        2    0     0  1024
~     file desc     4     2K      8K 39322K      199    0     0  256,512
~          proc     8     5K      5K 39322K       14    0     0
16,512,2048
~   VFS cluster     0     0K      1K 39322K      108    0     0  32
~   NFS srvsock     2     1K      1K 39322K        2    0     0  256
~    NFS daemon     1     1K      1K 39322K        1    0     0  256
~      in_multi    36     2K      2K 39322K       36    0     0  16,64
~   ether_multi    10     1K      1K 39322K       10    0     0  32
~   ISOFS mount     1    16K     16K 39322K        1    0     0  16384
~ MSDOSFS mount     1     8K      8K 39322K        1    0     0  8192
~          ttys   420   245K    245K 39322K      420    0     0
128,256,1024
~          exec     0     0K      2K 39322K     1864    0     0
16,64,512,1024
~    pfkey data     1     1K      1K 39322K        2    0     0  64
~    xform_data     0     0K      1K 39322K       36    0     0  16,32
~       pagedep     1     2K      2K 39322K        1    0     0  2048
~      inodedep     1    16K     16K 39322K        1    0     0  16384
~        newblk     1     1K      1K 39322K        1    0     0  256
~       VM swap     5     3K      3K 39322K        5    0     0
16,32,2048
~      UVM amap  1157    66K    122K 39322K   100572    0     0
16,32,64,128,256,512,1024,2048
~      UVM aobj     2     2K      2K 39322K        2    0     0  16,1024
~           USB    68     7K      7K 39322K       68    0     0
16,32,64,128,256
~    USB device    20     9K      9K 39322K       20    0     0
128,256,512
~   crypto data     1     1K      1K 39322K        1    0     0  1024
~   packet tags     0     0K      1K 39322K        4    0     0  16
~           NDP    13     2K      2K 39322K       16    0     0  64,128
~          temp    21     7K     11K 39322K     8993    0     0
16,32,64,256,512,1024,2048,4096

Memory Totals:  In Use    Free    Requests
~                 1588K    161K      133593
Memory resource pool statistics
Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
extentpl      20      187    0      150     1     0     1     1    
0     8    0
phpool        40      772    0       89     8     0     8     8    
0     8    0
pmappl        76     1730    0     1711     1     0     1     1    
0     8    0
vmsppl       216     1730    0     1711     2     0     2     2    
0     8    0
vmmpepl       88    72710    0    72020    29     6    23    25    
0     8    8
vmmpekpl      88     5161    0     5146     1     0     1     1    
0     8    0
aobjpl        52        1    0        0     1     0     1     1    
0     8    0
amappl        40    29958    0    29583     6     0     6     6    
0     8    0
bufpl        116       11    0       11     1     0     1     1    
0     8    1
mbpl         256    91316    0    90711    40     0    40    40    
1     8    2
mclpl       2048    56089    0    55512   296     0   296   296     4
3072    7
sockpl       200     2440    0     2344     7     0     7     7    
0     8    2
procpl       328     1743    0     1711     4     0     4     4    
0     8    1
zombiepl      72     1711    0     1711     1     0     1     1    
0     8    1
ucredpl       80      464    0      404     2     0     2     2    
0     8    0
pgrppl        24      114    0      100     1     0     1     1    
0     8    0
sessionpl     48       59    0       46     1     0     1     1    
0     8    0
pcredpl       24     1743    0     1711     1     0     1     1    
0     8    0
lockfpl       48     2112    0     2110     1     0     1     1    
0     8    0
filepl        48    66512    0    66374     3     0     3     3    
0     8    1
fdescpl      292     1744    0     1711     4     0     4     4    
0     8    1
pipepl        72     1706    0     1700     1     0     1     1    
0     8    0
sigapl       316     1730    0     1711     3     0     3     3    
0     8    1
scxspl       128    33442    0    33442     1     0     1     1    
0     8    1
wdcspl        96        8    0        8     1     0     1     1    
0     8    1
namei       1024   268025    0   268025     2     0     2     2    
0     8    2
vnodes       196     4191    0        0   210     0   210   210    
0     8    0
nchpl         64     2398    0        0    39     0    39    39    
0     8    0
ffsino       284   192215    0   188031   299     0   299   299    
0     8    0
dirhash     1024      744    0      685   104    89    15    93    
0     8    0
pfrulepl     556      154    0      118    11     0    11    11    
0     8    3
pfstatepl    256     1784    0     1783     5     0     5     5     0
1250    4
pfrktable   1240       21    0       16     3     0     3     3    
0     8    0
pfrkentry    152       28    0       21     1     0     1     1    
0     8    0
pfosfpen     108     1725    0     1380    18     8    10    10    
0     8    0
pfosfp        28      940    0      752     2     0     2     2    
0     8    0
rtentpl      108       36    0        2     1     0     1     1    
0     8    0
tcpcbpl      360      351    0      346     1     0     1     1    
0     8    0
tcpqepl       20        2    0        2     1     0     1     1    
0    16    1
synpl        180        3    0        3     1     0     1     1    
0     8    1
plimitpl     152      247    0      237     1     0     1     1    
0     8    0
inpcbpl      216     1254    0     1248     1     0     1     1    
0     8    0

In use 3718K, total allocated 4056K; utilization 91.7%

I am totaly confused about this utilization, because how can it be 91%
in use without anything running.
This even when pf is disabled and the only connection to the machine
is my ssh connection.

devices:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
~        inet 127.0.0.1 netmask 0xff000000
~        inet6 ::1 prefixlen 128
~        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
~        address: 00:30:48:2c:53:00
~        media: Ethernet autoselect (100baseTX full-duplex)
~        status: active
~        inet6 fe80::230:48ff:fe2c:5300%em0 prefixlen 64 scopeid 0x1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
~        address: 00:30:48:2c:53:01
~        media: Ethernet autoselect (1000baseT full-duplex)
~        status: active
~        inet6 fe80::230:48ff:fe2c:5301%em1 prefixlen 64 scopeid 0x2
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
~        address: 00:0e:0c:73:f3:a2
~        media: Ethernet autoselect (100baseTX full-duplex)
~        status: active
~        inet 192.168.255.131 netmask 0xffffff00 broadcast 192.168.255.255
~        inet6 fe80::20e:cff:fe73:f3a2%fxp0 prefixlen 64 scopeid 0x3
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=0<> mtu 2020
enc0: flags=0<> mtu 1536
bridge0: flags=41<UP,RUNNING> mtu 1500
bridge1: flags=0<> mtu 1500

- --
When i try to tweak some options:

kern.maxclusters=16384
kern.somaxconn=512
kern.maxfiles=16384


The usage reports:
=============

netstat -m:

602 mbufs in use:
~        576 mbufs allocated to data
~        1 mbuf allocated to packet headers
~        25 mbufs allocated to socket names and addresses
576/588/16384 mbuf clusters in use (current/peak/max)
1336 Kbytes allocated to network (97% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

vmstat -m:
Memory statistics by bucket size
~    Size   In Use   Free           Requests  HighWater  Couldfree
~      16     2056    248               8306    1280          0
~      32     1602     62               2812     640          0
~      64     1174     42               3170     320          0
~     128     1335      9               2081     160          0
~     256      185     23               1509      80          0
~     512     1244     12               1641      40          0
~    1024      245      3                386      20          0
~    2048       27      1                 82      10          0
~    4096       24      1                264       5          0
~    8192        3      0                  3       5          0
~   16384        7      0                  7       5          0
~   32768        3      0                  3       5          0

Memory usage type by bucket size
~    Size  Type(s)
~      16  devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, namecache,
UFS mount,
~          dirhash, proc, in_multi, exec, xform_data, VM swap, UVM amap,
~          UVM aobj, USB, packet tags, temp
~      32  devbuf, pcb, routetbl, ifaddr, vnodes, UFS mount, sem, dirhash,
~          VFS cluster, ether_multi, xform_data, VM swap, UVM amap,
USB, temp
~      64  devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash,
in_multi,
~          exec, pfkey data, UVM amap, USB, NDP, temp
~     128  devbuf, routetbl, ifaddr, vnodes, UFS mount, ttys, UVM amap,
USB,
~          USB device, NDP
~     256  devbuf, routetbl, ifaddr, sysctl, ioctlops, vnodes, shm, VM
map,
~          file desc, NFS srvsock, NFS daemon, ttys, newblk, UVM amap,
USB,
~          USB device, temp
~     512  devbuf, pcb, ifaddr, ioctlops, mount, UFS mount, shm, dirhash,
~          file desc, proc, exec, UVM amap, USB device, temp
~    1024  devbuf, ioctlops, file, ttys, exec, UVM amap, UVM aobj,
crypto data,
~          temp
~    2048  devbuf, ifaddr, ioctlops, UFS mount, proc, pagedep, VM swap,
UVM amap,
~          temp
~    4096  devbuf, ioctlops, UFS mount, temp
~    8192  devbuf, MSDOSFS mount
~   16384  devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount,
~          inodedep
~   32768  devbuf, namecache

Memory statistics by type                           Type  Kern
~          Type InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
~        devbuf  4530  1016K   1017K 39322K     4631    0     0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
~           pcb    99     8K      8K 39322K      135    0     0
16,32,64,512
~      routetbl    83     7K      8K 39322K      187    0     0
16,32,64,128,256
~        ifaddr   115    19K     19K 39322K      117    0     0
16,32,64,128,256,512,2048
~        sysctl     2     1K      1K 39322K        2    0     0  16,256
~      ioctlops     0     0K      4K 39322K      817    0     0
256,512,1024,2048,4096
~         mount     5     3K      3K 39322K        7    0     0  512
~      NFS node     1    16K     16K 39322K        1    0     0  16384
~        vnodes  1264    46K     46K 39322K     1266    0     0
16,32,128,256
~     namecache     3    49K     49K 39322K        3    0     0
16,16384,32768
~     UFS quota     1    16K     16K 39322K        1    0     0  16384
~     UFS mount    21    49K     49K 39322K       21    0     0
16,32,64,128,512,2048,4096,16384
~           shm     2     1K      1K 39322K        2    0     0  256,512
~        VM map     3     1K      1K 39322K        3    0     0  256
~           sem     2     1K      1K 39322K        2    0     0  32,64
~       dirhash    27     5K      5K 39322K       27    0     0
16,32,64,512
~          file     0     0K      1K 39322K        2    0     0  1024
~     file desc     4     2K      2K 39322K        5    0     0  256,512
~          proc     8     5K      5K 39322K       10    0     0
16,512,2048
~   VFS cluster     0     0K      1K 39322K       60    0     0  32
~   NFS srvsock     2     1K      1K 39322K        2    0     0  256
~    NFS daemon     1     1K      1K 39322K        1    0     0  256
~      in_multi    36     2K      2K 39322K       36    0     0  16,64
~   ether_multi    10     1K      1K 39322K       10    0     0  32
~   ISOFS mount     1    16K     16K 39322K        1    0     0  16384
~ MSDOSFS mount     1     8K      8K 39322K        1    0     0  8192
~          ttys   420   245K    245K 39322K      420    0     0
128,256,1024
~          exec     0     0K      2K 39322K      348    0     0
16,64,512,1024
~    pfkey data     1     1K      1K 39322K        2    0     0  64
~    xform_data     0     0K      1K 39322K       18    0     0  16,32
~       pagedep     1     2K      2K 39322K        1    0     0  2048
~      inodedep     1    16K     16K 39322K        1    0     0  16384
~        newblk     1     1K      1K 39322K        1    0     0  256
~       VM swap     5     3K      3K 39322K        5    0     0
16,32,2048
~      UVM amap  1130    65K     67K 39322K    10731    0     0
16,32,64,128,256,512,1024,2048
~      UVM aobj     2     2K      2K 39322K        2    0     0  16,1024
~           USB    68     7K      7K 39322K       68    0     0
16,32,64,128,256
~    USB device    20     9K      9K 39322K       20    0     0
128,256,512
~   crypto data     1     1K      1K 39322K        1    0     0  1024
~   packet tags     0     0K      1K 39322K        4    0     0  16
~           NDP    13     2K      2K 39322K       16    0     0  64,128
~          temp    21     7K     11K 39322K     1279    0     0
16,32,64,256,512,1024,2048,4096

Memory Totals:  In Use    Free    Requests
~                 1619K     31K       20267
Memory resource pool statistics
Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
extentpl      20      187    0      150     1     0     1     1    
0     8    0
phpool        40      511    0        0     6     0     6     6    
0     8    0
pmappl        76      207    0      188     1     0     1     1    
0     8    0
vmsppl       216      207    0      188     2     0     2     2    
0     8    0
vmmpepl       88     9511    0     8833    17     0    17    17    
0     8    2
vmmpekpl      88      513    0      502     1     0     1     1    
0     8    0
aobjpl        52        1    0        0     1     0     1     1    
0     8    0
amappl        40     3140    0     2774     4     0     4     4    
0     8    0
bufpl        116       10    0       10     1     0     1     1    
0     8    1
mbpl         256     2538    0     1934    40     0    40    40    
1     8    2
mclpl       2048     1035    0      458   294     0   294   294     4
8192    4
sockpl       200      164    0       68     5     0     5     5    
0     8    0
procpl       328      220    0      188     3     0     3     3    
0     8    0
zombiepl      72      188    0      188     1     0     1     1    
0     8    1
ucredpl       80       67    0        9     2     0     2     2    
0     8    0
pgrppl        24       22    0        8     1     0     1     1    
0     8    0
sessionpl     48       15    0        2     1     0     1     1    
0     8    0
pcredpl       24      220    0      188     1     0     1     1    
0     8    0
lockfpl       48        6    0        4     1     0     1     1    
0     8    0
filepl        48     2611    0     2473     2     0     2     2    
0     8    0
fdescpl      292      221    0      188     3     0     3     3    
0     8    0
pipepl        72      126    0      120     1     0     1     1    
0     8    0
sigapl       316      207    0      188     2     0     2     2    
0     8    0
scxspl       128     1597    0     1597     1     0     1     1    
0     8    1
wdcspl        96        8    0        8     1     0     1     1    
0     8    1
namei       1024     6404    0     6404     2     0     2     2    
0     8    2
vnodes       196     2102    0        0   106     0   106   106    
0     8    0
nchpl         64     2398    0        0    39     0    39    39    
0     8    0
ffsino       284     2093    0       36   147     0   147   147    
0     8    0
dirhash     1024       32    0        0     8     0     8     8    
0     8    0
pfrulepl     556       46    0       10     7     0     7     7    
0     8    1
pfstatepl    256       11    0        7     1     0     1     1     0
1250    0
pfrktable   1240        9    0        4     3     0     3     3    
0     8    0
pfrkentry    152        7    0        0     1     0     1     1    
0     8    0
pfosfpen     108      690    0      345    12     2    10    10    
0     8    0
pfosfp        28      376    0      188     2     0     2     2    
0     8    0
rtentpl      108       35    0        1     1     0     1     1    
0     8    0
tcpcbpl      360        5    0        0     1     0     1     1    
0     8    0
synpl        180        1    0        1     1     0     1     1    
0     8    1
plimitpl     152       13    0        3     1     0     1     1    
0     8    0
inpcbpl      216       38    0       32     1     0     1     1    
0     8    0

In use 2694K, total allocated 2896K; utilization 93.0%

What i don`t seem to understand is how openbsd can use so much memory
with default settings.
And when tweaking it somewhat it will still report > 90 % usage.
It might be i am interperting this wrong but from what i know is when
it reaches 100 % usage it will deny connections that are being made.
The weird thing is all my freebsd systems only report 3% usage and a
different openbsd box on a via C3 proccessor only reports:

cpu0: VIA Samuel 2 ("CentaurHauls" 686-class) 533 MHz
cpu0: FPU,DE,TSC,MSR,MTRR,PGE,MMX
real mem  = 125349888 (122412K)
avail mem = 107626496 (105104K)

netstat -m:

27 mbufs in use:
~        1 mbuf allocated to data
~        1 mbuf allocated to packet headers
~        25 mbufs allocated to socket names and addresses
0/82/6144 mbuf clusters in use (current/peak/max)
208 Kbytes allocated to network (3% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines


On my freebsd systems it only reports 3%.
So i am a bit dazed on how openbsd deals with this.

There for i am wondering if any of you fokes can give me any advice on
where to look at next on how to get this usage down some what so i can
change the machine to production.

Could it perhaps have to do with SMP being enabled?
I am having my second thoughts about this em driver aswell, because
since i`ve been using em cards under openbsd it seems to take alot
more memory then the normal fxp card do.

Well let me know

In hope to figure this thing out...

With kind regards

Matthew Whittaker-Williams

- --
Unix & Network Engineer

Internet Unie
Postbus 23667
1100 ED AMSTERDAM
Tel: 020 463 0506
Fax: 020 463 2146
Email: m_(_dot_)_whittaker-williams_(_at_)_iu_(_dot_)_nl
URL: http://iu.nl
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCW7oinBpoAAipCdARAsGRAJ9U1RE0Mej2ltVXijJFDuN/pyORtgCfZXgV
R2ZmIRGYPbce30OUDoWJp6Q=
=lGc0
-----END PGP SIGNATURE-----

[demime 1.01d removed an attachment of type APPLICATION/DEFANGED which had a name of m.whittaker-williams.18253DEFANGED-vcf]



Visit your host, monkey.org