DS Log

WireGuard inside WireGuard: Accessing my homelab through my VPN provider

published

I’m going to eventually create a post where I talk about my homelab and my network setup more in depth, but I decided to write this quick post to document how I was able to connect to my network through my VPN provider.

Some context: I have a router running OPNsense that connects through WireGuard to my VPN provider, and then forces essentially all traffic destined outside my network through this WireGuard tunnel. My VPN provider lets me do port forwarding, which means I can connect to the VPN’s exit server through the port they give me, and that traffic will be sent back to my router through the WireGuard tunnel.

This port forwaring thing is pretty neat, but I wanted to use it to create a tunnel to my network. This would let me use my laptop from anywhere in the world, but still be able to connect to my homelab network. Also, it’s kinda obvious but I wanted to use WireGuard for this tunnel, which would guarantee the traffic from my laptop to my router would be secure (at least as secure as WireGuard can make it be).

To summarize, I needed to setup a WireGuard tunnel inside a WireGuard tunnel on OPNsense. It was quite tricky to achieve this, and I had to piece a bunch of content online to do it, so this post will serve as a documentation of how I did things in case I ever need to redo this. Hopefully it will also help someone else out there.

The initial OPNsense -> VPN provider tunnel

This one’s quite straightforward, and I pretty much followed the instructions on OPNsense’s documentation. If you haven’t done this already, I suggest you do it.

To deal with DNS leaks, I couldn’t figure out how to force all DNS traffic to go through the tunnel itself, so I did the second-best thing which was to set my VPN provider’s DNS server as the system nameserver (OPNsense > System > Settings > General), and then force Unbound DNS to forward queries to the system nameserver (OPNsense > Services > Unbound DNS > Query Forwarding > “Use System Nameservers”).

The system nameserver thing has a gateway setting, but even if I set it to the WireGuard tunnel gateway, apparently OPNsense will still forward the queries through the WAN interface. This is something I still need to fix, but at least the current solution is good enough for now.

To allow anyone to follow this post, I used the following names for the things created through those instructions:

WireGuard interface: WAN_WG
WireGuard gateway: WAN_WG_GATEWAY
Alias for the hosts in my homelab (they all use the tunnel): WG_VPN_HOSTS

I’ll also use the symbol <VPN_PEER_IP> to mean the WireGuard IP address that was assigned by the VPN provider to my router. This is the IP address that I had to set in WireGuard when creating the tunnel to the VPN provider.

Setting up port forwarding on the VPN provider

This is specific for each VPN provider, so just follow their instructions. I’ll use the symbol <VPN_FWD_PORT> to represent the port that I was given during the port forward process, and the symbol <VPN_EXIT_IP> to represent the IP from my VPN provider that I should use to connect to my homelab network.

Setup needed before proceeding

When setting things up for the first time, I hit quite a few OPNsense + WireGuard quirks. The following resources helped me figure out some workarounds. I’m adding them here in case future me (or you) find any instruction in this post confusing.

Before adding any configurations for the new WireGuard tunnel, make OPNsense use the wireguard-kmod package in OPNsense. This package apparently installs WireGuard as a kernel module, instead of relying on the Go implementation that OPNsense currently uses. For some reason, the only way that I got things to work was to use this package (read the links above and see for yourself the mess).

As of this post’s publish date, this package is experimental, so beware of using it for anything critical. It seems to be working pretty great already and is very likely going to be the default package used by OPNsense in the future.s

To check which WireGuard packages are currently installed, go to OPNsense > System > Firmware > Packages and filter for “wireguard”. You should see (as of this post’s publish date) os-wireguard, wireguard-go, and wireguard-tools.

I really only needed to install the wireguard-kmod package and not mess with anything else. To install it, SSH into the router (you may have to enable SSH, but that’s outside the scope of this post) and run (make sure you’re root):

pkg install wireguard-kmod

After installing the package, reboot the router and it should automatically use the kernel module. The router itself will report the wireguard-go service as not running (if you have a “Services” widget on the dashboard, for example), but that’s expected - it’s using the kernel module now.

WireGuard setup

This section sets up a new WireGuard tunnel to be used only by whatever devices from the outside that I want to connect to my homelab. This new tunnel will act as if it were just another regular WireGuard tunnel, but afterwards I’ll create some rules that force all of its traffic to be routed through the VPN WireGuard tunnel.

These steps roughly follow this guide on the OPNsense docs, so if anything becomes confusing, try checking in there too.

  1. Configure a local peer on WireGuard (this will be the WireGuard server): OPNsense > VPN > WireGuard > Local > click the + to add a peer.

    Settings to use:

    • Enabled: checked.
    • Name: I used RoamingServer, can be whatever.
    • Public Key: either leave this blank and let OPNsense generate one, or use a key that’s already generated.
    • Private Key: either leave this blank and let OPNsense generate one, or use a key that’s already generated.
    • Listen Port: pick any port that won’t conflict with anything, I chose one in the high ranges (55000+). I’ll call this the <WG_LISTEN_PORT>.
    • Tunnel Address: choose a CIDR address block that’s not used anywhere else. I used a /24 block like 10.10.10.1/24.
    • Peers: leave blank for now.
    • Disable Routes: unchecked.

  2. Now head over to OPNsense > VPN > WireGuard > Endpoints, and for each device you want to connect, add an entry by clicking the +:

    Settings to use:

    • Enabled: checked.
    • Name: give a name for each device, I followed the pattern <DeviceName>Roaming.
    • Public Key: this needs to be generated already. This is the public key used by the device.
    • Allowed IPs: use a unique IP per device. Make sure this IP is within the CIDR block of the “Tunnel Address” setting above. For example: 10.10.10.2/32.

  3. Go back to the WireGuard server entry (OPNsense > VPN > WireGuard > Local > click on the edit button for RoamingServer). For each device that was added in “Endpoints”, select it in the Peers attribute.

Make sure to restart WireGuard by disabling and then reenabling it. OPNsense should now show an extra wg1 interface (or whatever next interface number, I’ll keep calling it wg1 in the rest of the post) in the “List Configuration” tab.

At this point, it’s probably a good idea to go to every device and create their initial WireGuard config already. To do that you’ll need to know the WireGuard server public key. If you already generated the keys you’re all set, but if you let OPNsense generate the keys, go to OPNsense > VPN > WireGuard > Local and click on the edit button for RoamingServer, and copy the value in Public key.

The following is a good starting point for a client config. I’ll definitely make some extra changes by the end of this post, because this config is incomplete for now:

[Interface]
PrivateKey = <device private key, this should already be generated>
Address = 10.10.10.2/32 # Or whatever address was used in the "Endpoints" configuration in OPNsense. They have to match.

[Peer]
PublicKey = <WireGuard server public key>
Endpoint = <VPN_EXIT_IP>:<VPN_FWD_PORT>

Allowing connections from the forwarded port to reach the WireGuard server

Right now it’s likely that any new traffic coming from the VPN WireGuard tunnel is getting dropped. I’ll allow traffic coming in the forwarded port to pass through and make its way to the WireGuard server.

To make that happen, first I had to assign an interface to this new WireGuard tunnel.

  1. Go to OPNsense > Interfaces > Assignments and select the wg1 interface, give it a name, add it to the list and click the “Save” button. I named my interface WG_ROAMING, so I’ll keep using this name.
  2. There should be an entry [WG_ROAMING] under OPNsense > Interfaces. Click it and configure as follows:
    • Enable: checked.
    • IPv4 Configuration Type: None.
    • IPv6 Configuration Type: None.
    • Dynamic gateway policy: checked.
  3. Remember to click “Save”.
  4. Restart Unbound DNS to make sure it will also monitor the new WG_ROAMING interface to let devices from the outside world to resolve internal hostnames too (make sure Unbound DNS is also set to monitor all network interfaces, or make it monitor the new interface too).

With that done, create the first rule to allow traffic coming for the VPN port to be redirected to the port the WireGuard server is listening on.

  1. Go to OPNsense > Firewall > NAT > Port Forward and create a new rule.

    Settings to use:

    • Interface: WAN_WG.
    • Protocol: UDP.
    • Destination: WAN_WG address (this means the address of the router on the WAN_WG interface).
    • Destination port range: (other) <VPN_FWD_PORT> to <VPN_FWD_PORT>.
    • Redirect target IP: Single host or network: <VPN_PEER_IP>.
    • Redirect target port: (other) <WG_LISTEN_PORT>.
    • Filter rule association: None.

  2. It’s very important to ensure that “Filter rule association” is “None” in this port forward rule. I had to create the associated rule manually because the rule that OPNsense creates will work on the way in (the WireGuard server will see a device attempting to connect to it), but it won’t work on the way out (it will try to route the response through the WAN interface, and not through the VPN tunnel). Go read the links I provided earlier for more information.

  3. To create the associated rule, go to OPNsense > Firewall > Rules > WAN_WG and create a new rule.

    Settings to use:

    • Action: Pass.
    • Quick: checked.
    • Interface: WAN_WG.
    • Direction: in.
    • Protocol: UDP.
    • Source: any.
    • Destination: WAN_WG address.
    • Destination port range: (other) <WG_LISTEN_PORT> to <WG_LISTEN_PORT>.
    • Click on “Show/Hide” button for “Advanced Options”.
    • reply-to: WAN_WG_GATEWAY.

  4. Make sure the “reply-to” setting points to the VPN tunnel gateway. This will force the response to go through the VPN tunnel and will allow a device from the outside world to establish a WireGuard connection through the VPN tunnel.

Allowing outside devices to talk to devices on the homelab network

This can be accomplished with firewall rules on the new WG_ROAMING interface.

  1. Go to OPNsense > Firewall > Rules > WG_ROAMING and add a new rule.

    Settings to use:

    • Action: Pass.
    • Quick: checked.
    • Interface: WG_ROAMING.
    • Direction: in.
    • Protocol: any.
    • Source: WG_ROAMING net.
    • Destination: WG_VPN_HOSTS.

  2. This new rule lets a device from outside actually access the hosts in the homelab.

  3. I’m getting a bit ahead of myself here, but since I’m here already, I’ll also create a rule that lets devices from outside use this router as a DNS server for internal hostnames. In theory this could also be accomplished with the rule above, but I decided to use the tunnel-specific IP address for the DNS server address (I don’t really know why, it just seemed better to do it this way).

    1. Still inside OPNsense > Firewall > Rules > WG_ROAMING create a new rule with the following settings:
      • Action: Pass.
      • Quick: checked.
      • Interface: WG_ROAMING.
      • Direction: in.
      • Protocol: any.
      • Source: WG_ROAMING net.
      • Destination: WG_ROAMING address.

That should be everything required on the OPNsense side. I had to fiddle a lot with everything until I arrived at this, so hopefully this will also help someone out there. If things still don’t work, don’t forget that rebooting the router after all these settings are applied is an option. I actually did this after a moment of frustration, and it was only after rebooting that everything started working. ¯\_(ツ)_/¯

Finishing the configuration on the device

I still need to complete the configuration of the WireGuard interface on every device. I’m assuming that early config is already there, so I’ll list only extra stuff that needs to be added in each section.

The config needs to define which IPs should be routed through the WireGuard tunnel. Since in my case I’m creating this tunnel only to talk to devices in my homelab network, I only need to expand the value of WG_VPN_HOSTS. For this example, I’ll assume that it expands to 192.168.1.1/24, 192.168.2.1/24 (i.e. there are two subnets on my homelab network).

If you’re reading this and want to route EVERY IP address through the WireGuard tunnel, you’ll need to do more work than this, because there is no firewall rule that properly routes traffic from the devices that was supposed to be sent externally.

I also mentioned letting the device resolve internal hostnames. My Unbound DNS settings make it register DHCP leases and static routes, allowing essentially any internal hostname to be resolved at the DNS level.

Usually, setting a DNS server in WireGuard is done with a “DNS” setting in the “Interface” section of the config. However, in my case my device is running Ubuntu 20.04, which means it’s using systemd-resolved for DNS, but WireGuard will try to add a DNS server using resolvconf, which completely bypasses the systemd-resolved stuff.

(I said Ubuntu 20.04 specifically here, but I’m almost sure this is also the case for a bunch of other distros)

I found a workaround through this blog post: WireGuard DNS Configuration for Systemd.

For the workaround, I’m going to set the DNS server address to be the IP address of the router inside the WireGuard tunnel. This IP address is the same that I set in the “Tunnel address” part of the WireGuard server (in my case it’s 10.10.10.1).

I’m also assuming that the domain of the network is called localdomain (this can be found in OPNsense > System > Settings > General > “Domain”). And finally, I’m also assuming that there are things inside the homelab network that can be resolved through the “.homelab” TLD (it’s not a real TLD).

Just to clarify on the “.homelab” thing: I have certain overrides in Unbound DNS to resolve things such as “container1.homelab” to an IP address, so I also want to be able to resolve these things in my devices outside the homelab.

With all of this said, these are the last things that need to be added to the device WireGuard config:

[Interface]
PostUp = resolvectl dns %i 10.10.10.1; resolvectl domain %i ~homelab localdomain

[Peer]
AllowedIPs = 192.168.1.1/24, 192.168.2.1/24

Bringing up the WireGuard interface in the device should make everything work now.