PFsense -> OPNsense migration
For the longest time I had some personal preference issues with Pfsense. So much so that I had a hard time recommending it to fellow homelabbers. I also enjoy the constant updates and features that Opnsense gets as well as it being open sourced I can add features that I desire (More on this later).
Another large selling point of Opnsense are their nice UI, and their super simple and easy Wireguard configurations. Much of their UI is more user friendly than the Pfsense counterpart which has largely not changed for years.
Hardware
One of the other major reasons for the change is the introduction to new hardware. In all IT fashion a company near me was upgrading their equipment and were throwing out their old Unifi UAS-XG.
The UAS-XG was Unifi's way of breaking into the server space. This was an application server for IT teams to quickly deploy and manage their application. It comes with some really cool features.
- 2 Drive trays in the front
- System LED on the front of the device
- 16GB DDR4 RAM
- 120GB NVME SSD
- 2 10Base-T ports
All of that comes in a small 1U formfactor. While being fairly efficient at an average 60W. This is a large upgrade to my Dell OptiPlex I have been running for years. So it was a clear decision to start planning my upgrade path, and switch to Opnsense.
The Problems
To start out the list of problems there was no easy migration tool from my Pfsense to Opnsense so I had a week of pain-staking manual migration. This process was fairly simple mainly had to learn how to configure stuff within Opnsense, though a lot of the configuration was pretty similar, aside from my IPsec configurations.
IPsec Changes
While I was running Pfsense I had pfblocker running for my network wide ad blocker, its was truly one of the few features I missed of pfsense. This made me re-think my DNS infrastructure. I decided to no longer run DNS blocking on my router and instead separate that out to redundant DNS servers with the help of Technitium. I wont go into too much detail but I am running them as secondary servers where my AD servers are the Primary domain servers and send TXFR records to my secondary Technitium DNS servers which see all client traffic.
Everything else went fairly smoothly migrating from Opnsense to Pfsense since I had a semi-complex config though nothing was verry specific to Pfsense. It was a lot of work re-configuring my firewall rules since my pfsense had about 10-20 rules per VLAN. I since made the configuration easier with the use of aliases so I was able to group common ports together.
Routing
In my network I am running a semi-complex routing solution. I have a Kubernetes cluster which advertises load balancer IPs via Cilium BGP operators. This allows me to have an extremely highly available load balancer since it works at the network level. This reduces the need for a external load balancer which is a single point of failure.
On my old Pfsense I had configured FRR, or Free Range Routing. I can do the same on Opnsense though there was a slight difference. Pfsense when you make a change it will update the frr.conf. If the Pfsense UI did not have a feature I needed to use I could just add in my FRR config in the frr.conf.
Opnsense handled this differently they saved the config stored in the UI, and every time you booted the router or changed the config it would load the changes from the UI. This turned out to be a severe problem. The Opnsense UI was lacking in a lot of BGP features I needed. The two main features were the use of internal or external for peer group ASN. So I could not use:
neighbor <peer-group> remote-as external
This is needed for my IPsec configurations. Me and my friends have a private WAN where I need to distribute certain subnets, and I did not want to configure this per-asn. Another problem that I ran into was the lack of BGP Listen Range. In my Kubernetes configuration I had the ideology to have my IPs to be ephemeral, so this would mean no static IPs to form a neighborship. BGP solved this problem with listen ranges, if a peer is trying to form a neighborship within a certain subnet it will auto form the neighborship and add it to a peer group.
OPNsense Needed Some Changes
With my FRR configuration the only way to get it to work how I wanted I would SSH into the machine and go into the FRR process and manually make my changes. Though this did not persist like I mentioned above so every time the router would reboot for updates my config would be reverted and my Kubernetes cluster would go offline.
Fortunately OPNsense is open source. I made 2 GitHub issues on the opnsense/plugins repository, 4713 and 4609. They got fixed really quickly, and got put in the update channel within the week. With this issue solved I have added in the last change I needed to complete my migration.
Key Takeaways
This was a really fun project that I took on when I was home for spring break. Reflecting on the project it made me re-think how I configured my firewall with heavy use of aliases. With the ease of configuration I also heavily narrowed down my firewall rules only allowing select outbound ports as well as inbound restrictions.
I have slowly tested and tried out some OPNsense features, such as Suricata. I always hated the Pfsense IDS and always wanted to test out Suricata. With this migration I added in Suricata for high security VLANs and WAN interface, and tuned it to my liking. I now have more security than just basic port blocking with an IDS/IPS.
This project also taught me a lot about the world of opensource and how easy it is to get a change into an upcoming release rather than praying to the software gods to update their project.
I am really happy with how this project turned out and this is one of the few projects that I don't have anything I miss or that I wish I had done differently. It has been running solid in my home for couple months now and I have no issues, there was one slight issue where it crashed out of no where and dropped packets but I have not had that since.