(Of course, the Wireguard traffic still goes through Hinet. I have no idea how to represent that in the graph though.)
(The hashtags are the names used in firewall aliases, and yes, I am a fan of Azur Lane.)
Essentially, the devices are connected to the new AP, which is then connected to the server. The server then sends the traffic through a pfSense VM and out to the I-040GW operating in bridge mode. In addition, the WR841HP extends the openwireless.org guest network, and the MOD (Hinet IPTV device) is connected directly to the I-040GW in bridge mode since it only requires access to VLAN 4081 on the WAN side.
Note that only the two ports connected to the server and the MOD operate in bridge mode. “Normal” routing with a DHCP server, PPPoE, NAT, …etc, are still enabled on the other ports so that, even if something goes haywire with my server when I am out, my family can still use the Internet by plugging into the I-040GW.
However, I am currently experimenting with disabling bridge mode on the server port and using PPPoE passthrough (, which, conveniently, is enabled on the I-040GW by default). This has the benefit of being able to manage the ONT directly from the LAN.
The pfSense VM, like the other VMs on the server, is managed by libvirt/virt-manager. The installation process itself was fairly straightforward. (Not so much for setting it all up though. Some hurdles are mentioned in the next section.) The major things I had set up are listed below. (In addition, some IPv6-related settings will be mentioned in the Troubleshooting section.)
I tried passing both the Intel NIC and the Atheros Killer E2400 on my motherboard to the virtual machine. However, the latter did not seem to be receiving any packets. After quite some debugging and searching, it turned out that FreeBSD support for the E2400 was buggy, and I had to resort to MacVTap.
A side note,
virt-host-validate helped me figure out that I did not pass
intel_iommu=1 to the kernel initially.
One needs to disable hardware checksum offloading when using VirtIO NICs, which I use to connect the host with the pfSense guest. The symptoms, if it is not disabled, are pretty weird. Added to the difficulty of debugging was that my host could somehow get IPv6 addresses via SLAAC from the Killer E2400 without going through PPPoE, causing me to think that IPv6 works.
If one creates a bridge interface from
/etc/network/interface (on Debian), link a VM to it, then issue
systemctl restart networking, the VM interface (
vtnet*) can be removed from the bridge. Pretty dumb of me to not find out earlier, but this still caused quite some head-scratching. (Why does DHCP work on
vtnet but not on the bridge interface?)
Before realizing this, I also tried creating an isolated network in libvirt instead. However, IPv6 for isolated networks was disabled by default. One could enable it by manually setting an IPv6 address, but dnsmasq would take up a ton of CPU, possibly caused by this bug. I then switched to using
<network ipv6='yes'> (docs) and adding the following to
iface virbr4 inet6 auto pre-up /sbin/sysctl -w net.ipv6.conf.virbr4.disable_ipv6=0
Which worked, but felt a bit hackish compared to a manual bridge.
As the diagram shows, I wanted to have two outgoing PPPoE connections with three LAN interfaces (a VirtIO bridge and two VLANs on the Intel NIC). Aside from the prefix-size issues mentioned below, the LANs flat-out could not receive IPv6 prefixes (best case scenario, only one interface could get a prefix). After some trial and error, I realized that disabling one PPPoE connection made the LAN associated with the other PPPoE connection work reliably. This led me to yet another bug report. To work around this, I wrote my own dhcp6c configuration file and used the
shellcmd package to run the following command on boot:
pkill -f '[d]hcp6c'; ln -sf /var/run/dhcp6c.pid /var/run/dhcp6c_pppoe0.pid; ln -sf /var/run/dhcp6c.pid /var/run/dhcp6c_pppoe1.pid; /usr/local/sbin/dhcp6c -dDn -c /conf/dhcp6c_merged.conf -p /var/run/dhcp6c.pid pppoe0 pppoe1
Having more LAN interfaces than PPPoE connections, I would need to obtain prefixes shorter than
/64 for prefix delegation to work. Unfortunately, Hinet only gave out
/64 prefixes no matter what I request in the settings, so I had to either use another PPPoE connection or use DHCPv6 to hand out ULAs and apply NPt on the router.
I opted for the latter due to being able to share the same prefix between the private and guest LANs, though the extra effort may not be worth it in retrospect.6
We envision a world where, in any urban environment:
The false notion that an IP address could be used as a sole identifier is finally a thing of the past, creating a privacy-enhancing norm of shared networks.
Unfortunately (again), Hinet does not honor DUID7, so the prefix I get changes every time, and pfSense does not support dynamic NPt at the time of writing. I worked around this by writing a PHP script that updates the NPt entry and is invoked whenever the DHCPv6 client gets a new reply.
Unfortunately (yet again), Android does not support DHCPv68. Although disabling IPv6 on Android devices is fine with me, devices will still get IPv6 DNS servers via RA (router announcement) and show to the user “connected, no Internet” until it gives up and reverts to IPv4. Simply setting pfSense to announce
:: as the DNS server for RA resolved this. (Clients that do support DHCPv6 will get their DNS information from that instead.)
The workarounds used to mitigate the two issues above (dual DHCPv6 clients & dynamic NPt) will be covered in depth in a future post.
Setting up the AP itself was considerably simpler. I flashed MerlinWRT as the firmware, and for setting up VLANs, https://www.snbforums.com/threads/ssid-to-vlan.24791/ and https://gist.github.com/Jimmy-Z/6120988090b9696c420385e7e42c64c4 are good resources. (Note that the latter is for the AC86U, and has to be modified a bit for the AC66U.) However, I did not realize that I needed to disable NAT acceleration (and that the setting only seemed to kick in after a
nvram commit and a reboot), causing some headaches.
In addition, client isolation on the guest networks is enabled.
Again, there will be a future article on setting up VLANs on this AP.
Even though my current setup can probably be achieved by a custom firmware on the wireless router, the current pfSense configuration has several advantages such as the flexibility and the performance of running on a more beefy machine.
That being said, did the Wi-Fi performance improve? Well, connecting directly to the RT-AC66U, the bandwidth was greatly improved due to the switch from 802.11n to 802.11ac. As for connecting through the WR841HP, while the stability did get better, the speed, similar to before, still lingered at a mere 20Mbps compared to the 100/40Mbps plan we had. Connecting to the RT-AC66U with other devices from where the WR841HP was placed resulted in similar numbers. Therefore, it remains to be investigated whether i) moving the WR841HP, ii) adding more WDS devices, or iii) experiment with power-line Ethernet, helps.
Wiring Ethernet between floors is difficult in our case. On the other hand, it is unclear whether power-line Ethernet works reliably across breakers in our house. ↩
Hardware-wise, an i7-6700K tower. ↩
Other models on my shortlist includes the classic TP-Link Archer C7 (~NT$1900) and the Asus RT-AC1300UHP (~NT$2200, essentially a higher-power, 256-MB-RAM version of the RT-AC58U), both supporting OpenWRT instead of MerlinWRT/DD-WRT/Tomato on the RT-AC66U. The RT-AC66U has better hardware and wireless capabilities (e.g., AC1900, beamforming) though. ↩
The ISP provides us with one fixed IPv4 address and up to seven floating IPv4 addresses, all via PPPoE. ↩
That being said, static ULAs makes setting up firewall rules a bit easier. ↩
If Hinet were to honor DUID, what prefixes would I get from multiple PPPoE connections? ↩
The rationale for them not supporting DHCPv6 seems to be to discourage this kind of usage. Oh well ;) ↩
Send an email to firstname.lastname@example.org.