Complex setup with OpenBSD in bridge mode

[Christo Butcher <christo@fox-it.com>]

Complex setup with OpenBSD in bridge mode


We have a leased line coming in over a little Cisco router, and a /26
IP range with 62 available IP addresses: x.y.z.1 through x.y.z.62
We decided devide our machines up into two divisions:
	* inernal ones, with 10.0.0.0/8 addresses
	* DMZ machines, in the /26 range

Right behind the Cisco router we're running an OpenBSD 2.9 box that
does the routing between internal, DMZ and internet hosts. So the
following setup is created:

	[Internet]
	    |
	    |
	 [Cisco]
	    |
	    |
	[OpenBSD]---[Internal]
	    |
	    |
	  [DMZ]

For management we have given the Cisco IP address x.y.z.1
Two solutions have been tested to realise this setup:

1) Split the /26 range up into two separate IP ranges, and give one
half to the [Cisco]---[OpenBSD] cross-cable, and the other to the DMZ.
Ouch! But it does make the routing easy.  :)
This solution basicly gets what we need, but at an unreasonable cost of
half our usable IP addresses.

2) Get the OpenBSD box to function in bridged mode, so that the Cisco
is (virtually) on the same LAN with all 62 IPs as the DMZ machines.
This sounded like a great way to do it, but some strange problems have
appeared.

For the internal network the OpenBSD box is the gateway to the internet
and the DMZ. It also performs some NAT functions when the internal hosts
talk to the internet. This means that the OpenBSD machine must have an
IP address, so it's not a transparent firewall. (A big benefit is that
it can then be administered remotely)

The network setup of the OpenBSD box is as follows:

	Internet
	   |
	   |
	 (de2)
	   |
	[OpenBSD]--(de0)---> Internal
	   |
	 (de1)
	   |
	   |
	  DMZ

Where the interfaces have the following /etc/hostname.de[012]:
/etc/hostname.de0: inet 10.0.0.1 0xff000000 NONE
/etc/hostname.de1: up
/etc/hostname.de2: inet x.y.z.2 0xffffffc0 NONE

Together with the following bridge config:
/etc/bridgename.bridge0: add de2
/etc/bridgename.bridge0: add de1
/etc/bridgename.bridge0: up
(The file is three lines long)

Interface de1 wasn't given an IP address, because it was expected to be
on the same (bridged) LAN as de2. So a machine on the DMZ could just
reach the OpenBSD box by talking to x.y.z.2 (IP of de2).

The bridging function worked perfectly, a DMZ machine could
access the internet by using the Cisco as default gateway.

But for some reason the DMZ machines couldn't talk to the
OpenBSD box: a packet sent to x.y.z.2 (de2) was visible by tcpdump on
de1. It was destined for the MAC address of de2. But the OpenBSD
machine didn't accept it. This kind of traffic occured:

$ ping x.y.z.10     # A host on the DMZ
MAC(de2) -> MAC(FFF): arp who-has x.y.z.10 tell x.y.z.2
MAC(dmz) -> MAC(de2): arp reply x.y.z.10 is-at aa:bb:cc:dd:ee:ff
MAC(de2) -> MAC(FFF): arp who-has x.y.z.10 tell x.y.z.2
MAC(dmz) -> MAC(de2): arp reply x.y.z.10 is-at aa:bb:cc:dd:ee:ff
MAC(de2) -> MAC(FFF): arp who-has x.y.z.10 tell x.y.z.2
MAC(dmz) -> MAC(de2): arp reply x.y.z.10 is-at aa:bb:cc:dd:ee:ff
[etc...]

Where:
MAC(de2) is the MAC address of the de2 interface card
MAC(dmz) is the MAC address of the host on the DMZ being pinged
MAC(FFF) is the broadcast MAC address: ff:ff:ff:ff:ff:ff

These arp replies were obviously not being added to the arp cache. Even
if this was done by hand ($ arp -s x.y.z.10 aa:bb:cc:dd:ee:ff), the
replies to the pings were visible in the tcpdump, but not to the ping
process.

So packets headed for MAC(de2) came in on de1. Well, this did work in a
test setup with a bridging OpenBSD 2.8 box.


A remedy to this was giving interface de1 an IP address:
/etc/hostname.de1: inet x.y.z.3 0xffffffc0 NONE
(ifconfig would complain, because de1 and de2 are in the same IP range,
but add the IP anyway)

Packets sent onto the DMZ now came from MAC(de1), and to the Cisco
(same as before) from MAC(de2). Now everybody can talk to everybody
else. Problem solved! But the question remains: why does this OpenBSD
2.9 box have to have two separate IP addresses, whereas a test setup
with OpenBSD 2.8 worked fine with just 1 IP address?

In the test setup the network setup was exactly the same as here, except
that there was no third [Internal] network segment. Here all packets to
and from the OpenBSD 2.8 bridging box used the MAC address of the
interface that did have an IP address. (The MAC of the IP-less
interface was never seen in tcpdumps)


But now, using this OpenBSD 2.9 box with three IP addresses (de0 for
internal use, de1 on the DMZ and de2 attached to the Cisco), a working
setup has been built. Next step: ipf and ipnat. Result: lots of
confusion.  :(

>From the manpage of bridge(4):
   Before forwarding a frame, the bridge will check to see if the packet
   contains an ip(4) datagram; if so, the datagram is run through the
   ipf(4) interface so that it can be filtered. Only the ipf(4) input
   rules for the source interface are checked with the datagram; output
   rules have no effect.

Okay, well, I guess we'll have to live with that. It makes stateful
rules kinda difficult though. But we don't seem to have any choice
here. A strange thing was spotted during some ipf test with the
following stateful rule (and others, but this will be used as an
example):
/etc/ipf.rules: pass out quick proto udp all keep state

A packet coming in on de0 from [Internal] and destined for [Internet]
(that was passed by the incoming rules) would keep state on interface
de1. I thought that [Internet] would be reached over de2, but the state
was kept for de1. This meant that the reply, coming in on de2, would
NOT be let back through.

Now what about NAT? We want to map internal hosts to the single IP
address of de2: x.y.z.2, we used to use this ruleset:
/etc/ipnat.rules: map de2 de0/8 -> de2/32 portmap tcp/udp 40000:50000
/etc/ipnat.rules: map de2 de0/8 -> de2/32

But with the bridge active, these rules stopped working, and we had to
add the following:
/etc/ipnat.rules: map de1 de0/8 -> de2/32 portmap tcp/udp 50000:60000
/etc/ipnat.rules: map de1 de0/8 -> de2/32

And these got the NAT working! Can someone tell me what a packet coming
in over de0 and going to the internet (over de2, I thought) has to do
with a NAT rule for outgoing traffic on de1?? But for some reason it
did get the NAT rules working.

The problems with ipf and ipnat are probably caused by this strange
bridged setup. The IP packets are treated quite differently from
normal. Perhaps the interface de1 is choses for keep-state and NAT
mapping because it is the `first' interface of the bridge? (Lowest
number, I mean)

Can anyone enlighten us to the cause (and even better: a solution!) to
these problems? Are any due to changes between OpenBSD 2.8 and 2.9?
(Both installed from official OpenBSD CDs) Is there a different solution
to this [Internet]/[DMZ]/[Internal] network topological situation?

We would be grateful for any information or help that could be offered.

Christo Butcher
Fox-IT Forensic IT Experts B.V.