projects | github | twitter | rss | contact

Using an OpenBSD Router with AT&T U-Verse

posted to writings on mar 21st, 2019 with tags networking and openbsd and commented on 11 times

I upgraded to AT&T's U-verse Gigabit internet service in 2017 and it came with an Arris BGW-210 as the WiFi AP and router. The BGW-210 is not a terrible device, but I already had my own Airport Extreme APs wired throughout my house and an OpenBSD router configured with various things, so I had no use for this device. It's also a potentially-insecure device that I can't upgrade or fully disable remote control over.

Fully removing the BGW-210 is not possible as we'll see later, but it is possible to remove it from the routing path. This is how I did it with OpenBSD.

802.1X

When the service was originally installed by AT&T, a fiber-optic cable terminated at an ONT in my basement, which then connected to the BGW-210 via ethernet.

I tried swapping my OpenBSD router in place of the BGW-210 but it could not get a DHCP lease or receive any traffic from the ONT, despite a properly negotiated ethernet connection.

After capturing some network traffic between the ONT and the BGW-210, I learned that the BGW-210 does 802.1X EAP authentication with a certificate installed on the device and all traffic is encapsulated in an 802.1Q VLAN (with a VLAN id of zero). Until this negotiation is performed, the upstream device will not respond.

The EAP certificate on the BGW-210 cannot easily be exported, making it difficult to easily use a third-party router with AT&T's service (Hush-A-Phone, anyone?). However, since the EAP authentication is only done at the beginning of the connection, it is possible to just proxy those authentication packets between the BGW-210 and AT&T to authenticate the connection and then use a 3rd party router for all DHCP and routing afterwards.

Proxying EAP

I'm not the first person to want to use their own router on AT&T's U-verse service, so there are already some software solutions out there for proxying EAP. I went with eap_parrot since it was small and written in Go, and only needed minor changes to support OpenBSD which the author has since merged.

eap_parrot can be installed with go get (pkg_add go):

jcs@pf:~> mkdir go; cd go
jcs@pf:~/go> go get github.com/mjonuschat/eap_parrot    
jcs@pf:~/go> sudo install -m 755 -o root -g wheel bin/eap_parrot /usr/local/bin/eap_parrot

Then it just needs a small configuration file:

jcs@pf:~> cat /etc/eap_parrot.toml 
[network]
wan_interface="em0"
router_interface="em2"
vlan_id=-1
promiscuous_mode=false

[logging]
syslog=true
debug=true
debug_packets=false

[ignore]
start=false
logoff=false

An /etc/rc.d/eap_parrot script can be created and then enabled to run on boot (rcctl enable eap_parrot):

jcs@pf:~> cat /etc/rc.d/eap_parrot 
#!/bin/ksh

daemon="/usr/local/bin/eap_parrot"
daemon_flags="-config /etc/eap_parrot.toml"
rc_bg=YES

. /etc/rc.d/rc.subr

rc_reload=NO
rc_cmd $1

After a test run with eap_parrot, I was able to get an IP from OpenBSD via dhclient.

My Network

I recently upgraded and moved all of my networking and server equipment into a half-rack:

Here's an ASCII diagram of my AT&T U-verse routing setup now:

        .-[ 192.168.1.1/24 - gigabit switch for servers, wireless APs ]
        +-[vlan1]-[ 192.168.2.1/24 - guest WiFi ]
        |
        |           .-[ 192.168.4.1/24 - BGW-210 LAN port ]---------------.
        |           |                                                     |
        | .---------|-----------------------------------.   .-------------|--.
        `-|-[em1] [em3] [em5] [ix1] Supermicro E-300-8D |   |     BGW-210 |  |
.-[vlan0]-|-[em0] [em2] [em4] [ix0]  OpenBSD 6.4 - pf   |   | [uplink] [lan] |
|         `---------|-----------------------------------'   `----|-----------'
|                   |                                            |
|                   `-[ no IP - BGW-210 uplink port ]------------'
|
|                               .----------.
`-[ public IP (DHCP+DHCPv6) ]---| AT&T ONT |---[ fiber ]---[ AT&T / internet ]
                                `----------'

The uplink port of my OpenBSD router is now vlan0 which has a parent interface of em0, and em0 is configured to clone the MAC address of the BGW-210's uplink port. em0 is wired directly to the AT&T ONT, which connects out to the internet.

jcs@pf:~> cat /etc/hostname.em0
up lladdr b0:93:5b:XX:XX:XX descr "ONT"
-inet6

jcs@pf:~> cat /etc/hostname.vlan0 
parent em0 descr "vlan through ONT"
inet6 autoconf

The internal LAN is connected via em1 and wired into a gigabit switch providing connectivity to my servers and Airport WiFi APs.

jcs@pf:~> cat /etc/hostname.em1
inet 192.168.1.1 255.255.255.0 192.168.1.255 descr "switch"

vlan1 also rides over em1 and is used by the Airport APs for its guest WiFi network (with VLAN id 1003).

jcs@pf:~> cat /etc/hostname.vlan1 
inet 192.168.2.1 255.255.255.0 192.168.2.255 parent em1 vnetid 1003 descr "guest wifi"

This enables traffic to be routed to the internet from the guest WiFi network without it being able to reach my normal network.

To quarantine the BGW-210, it is wired directly to a spare port (em2) on my OpenBSD router with no IP address. EAP traffic received from em2 is proxied through to em0 via eap_parrot but no other traffic can pass in or out.

jcs@pf:~> cat /etc/hostname.em2 
up descr "at&t router (ONT interface)" -inet6

An ethernet cable is connected from one of the BGW-210's LAN ports to em3 on my OpenBSD router, but the interface is left unconfigured. In the rare case that I need to access the BGW-210's web interface, I can run dhclient em3 and get an IP in the 192.168.4.0/24 network from the BGW-210 and route to it.

IPv6

AT&T uses DHCPv6 for IPv6 connectivity, so I am using the dhcpcd package (pkg_add dhcpcd) for both IPv4 and IPv6 DHCP.

jcs@pf:~> cat /etc/dhcpcd.conf 
allowinterfaces vlan0
duid
slaac hwaddr
persistent
nooption domain_name_servers, domain_name, domain_search, host_name
option interface_mtu

# override sending OS version and other crap
hostname -
vendclass 40712 .

# don't run anything at each stage
script /usr/bin/true

interface vlan0
        ia_na 0
        ia_pd 1 vlan0/1 em1/2

The hostname and vendclass 40712 lines were added after doing a tcpdump to troubleshoot IPv6 and seeing the following in dhcpcd's DHCPv6 request:

[...]
0040: b093 5bdc b811 0008 0002 006d 0010 004a  ..[........m...J
0050: 0000 9f08 0044 6468 6370 6364 2d37 2e30  .....Ddhcpcd-7.0
0060: 2e31 3a4f 7065 6e42 5344 2d36 2e33 3a61  .1:OpenBSD-6.3:a
0070: 6d64 3634 3a49 6e74 656c 2852 2920 5865  md64:Intel(R) Xe
0080: 6f6e 2852 2920 4350 5520 442d 3135 3138  on(R) CPU D-1518
0090: 2040 2032 2e32 3047 487a 000e 0000 0003   @ 2.20GHz......
[...]

I don't know why dhcpcd sends this information by default, but these values can be overridden by setting hostname - and vendclass 40712 ..

dhcpcd must also be configured to send the same DUID that the BGW-210 does, which can be found by inspecting the router solicitation requests from the BGW-210 (shown here from Wireshark):

Frame 46: 167 bytes on wire (1336 bits), 167 bytes captured (1336 bits)
Ethernet II, Src: ArrisGro_xx:xx:xx (xx:xx:xx:xx:xx:xx), Dst: IPv6mcast_01:00:02 (33:33:00:01:00:02)
802.1Q Virtual LAN, PRI: 3, DEI: 0, ID: 0
Internet Protocol Version 6, Src: fe80::xxxx:xxxx:xxxx:xxxx, Dst: ff02::1:2
User Datagram Protocol, Src Port: 546, Dst Port: 547
DHCPv6
    Message type: Solicit (1)
    Transaction ID: 0x574b20
    Client Identifier
        Option: Client Identifier (1)
        Length: 28
        Value: 000200000de9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
        DUID: 000200000de9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
        DUID Type: assigned by vendor based on Enterprise number (2)
        Enterprise ID: The Broadband Forum (3561)
        Identifier: 30303xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Reconfigure Accept
    Elapsed time
    Option Request
    Identity Association for Prefix Delegation

The long string sent as the DUID must be configured in dhcpcd by writing it to the /var/db/dhcpcd/duid file as 28 hexadecimal digits separated by colons:

jcs@pf:~> cat /var/db/dhcpcd/duid 
00:02:00:00:0d:e9:[...]

Once dhcpcd starts up, it will fetch an IPv4 lease through DHCP, then fetch a /60 IPv6 subnet via DHCPv6 and apply the ::1 IP of a /64 to em1 for routing to the LAN. With rtadvd enabled (rcctl enable rtadvd; rcctl set rtadvd flags em1), it will send out router advertisements to the LAN for clients to autoconfigure themselves on this /64.

Comments? Contact me via Twitter or e-mail.

11 Comments

Josh Klein (authentic, via ) on march 22nd, 2019 at 18:36:54:

curious if you're doing anything to monitor the ups, e.g. with sysutils/nut

joshua stein (authentic, via ) on march 22nd, 2019 at 19:58:04:

I am not, but I should probably set that up in case of a long outage needing shutdown(s).

Josh Klein (authentic, via ) on march 22nd, 2019 at 21:19:36:

It seemed like a lot for my simple setup - so I got distracted by other projects - but I'd be interested to hear if you found it straightforward if/when you do

Adam Simpson (authentic, via ) on april 29th, 2019 at 12:44:24:

I just got ATT fiber myself. Is the vlan0 necessary for the eap proxy? Also does the interface (em0 in your post) wired to the ONT have to clone the MAC of the BGW-210? A stripped down version is to have the BGW-210 hang off a router nic and tell eap_parrot the two interfaces no?

joshua stein (authentic, via ) on april 29th, 2019 at 12:53:15:

Yes, both are necessary to get a DHCP lease

Adam Simpson (authentic, via ) on april 29th, 2019 at 13:26:22:

Perfect, thanks for the response! 🙏

Nate H (authentic, via ) on june 1st, 2019 at 13:46:30:

I was able get this working on my setup, with one exception. If I request a normal address (ia_na 1), the vlan0 interface gets assigned an address that does not seem to be routed properly by ATT. A few other online posts indicating the same. Did anyone here else observe this?

Nate H (authentic, via ) on june 1st, 2019 at 17:35:40:

In my previous tweet I meant to say "ia_na 0". Removing the normal address request was good enough to get things working. I also kick slaacd (slaacctl send solicitation vlan0) on dhcpcd REBIND6 event since w/out ia_na,it seems to take slaacd extra time to setup the default route.

fülöpke (authentic, via ) on august 16th, 2019 at 02:52:31:

I recently had a firewall appliance failure at home, but I do have a E300-8D that is currently unused and I am thinking of deploying it in a similar way to you with OpenBSD and PF.
It is a fairly meaty box, do you run other things on it too or is it solely on routing/fw duties?

joshua stein (authentic, via ) on august 16th, 2019 at 07:04:34:

Just pf, I am kind of wasting it but I don't have any other use for it

fülöpke (authentic, via ) on august 16th, 2019 at 08:52:35:

Yes, they are nice boxes, I was running ESXi on one while back which had a few OpenBSD hosts on it. A common configuration for that is a fast USB 3 thumb drive to boot and NVMe SSD for storage. Are you using NVMe? Any gotchas that you can remember?