BenV's notes

New Machine, New problems

One of my VMs has been running on ancient deprecated hardware for $forever now (without problems I might add), but after getting notice that it will be shutdown in 2 months together with an upgrade path that only costs me time and will result in double the specs, I decided to start the upgrade.

In order to do things properly, I started cobbling together some ansible roles for things I don’t want to repeat. One of these roles you can guess based on the title, wireguard. Needless to say things never work as you think they will, this is one of those stories.

Wireguard as point of entry

Given the SSH shenanigans ( CVE-2024-6387 ) that keep popping up (CVE-2024-6409) lately (CVE-2024-3094 as well this year), combined with the ease of exploitation, and constant port scans that have become snafu (even with fail2ban blocking countless IPs), I’ve finally decided to get rid of public SSH. If we need to run VPNs anyway, we might as well make that the only publicly exposed attack vector where possible and do the rest through internal networking. This will give the bonus that port 22 can turn into a honeypot, banning everything that tries to connect there.

Wireguard setup

My wireguard servers run through docker, specifically this one (with some local customizations to add some tooling). This allows me to have a .env file in my wireguard docker directory that has a PEERS=Peer1,Peer2,Jemoeder,Peer4 etc, and when restarting the docker it will create the peer configs for me.

The docker-compose.yaml looks a bit like this:

services:
  wireguard:
    image: linuxserver/wireguard
    container_name: WireGuard-Server
    cap_add:
      - NET_ADMIN
    environment:
      - PUID=420
      - PGID=420
      - TZ=Europe/Amsterdam
      - SERVERURL=your.mother.com
      - SERVERPORT=51820
      - PEERS=${PEERS}
      - INTERNAL_SUBNET=${INTERNAL_SUBNET}
      - ALLOWEDIPS=0.0.0.0/0
      - LOG_CONFS=true
    volumes:
      - /docker/wireguard-server/mounts/config:/config
    network_mode: host
    restart: unless-stopped

When peers are added, you’ll find yourself a directory like this:

$ ls -la /docker/wireguard-server/mounts/config
drwxr-xr-x 10  420  420 4096 Jul 14 17:46 ./
drwxr-xr-x  3 root root 4096 May 10  2023 ../
-rw-------  1  420  420  211 Jul 14 17:46 .donoteditthisfile
drwxr-xr-x  2  420  420 4096 May 10  2023 coredns/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer1/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer2/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer3/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Jemoeder/
drwxr-xr-x  2  420  420 4096 Jul 14 17:46 peer_Peer4/
drwxr-xr-x  2  420  420 4096 Mar 18 11:26 server/
drwxr-xr-x  2  420  420 4096 May 10  2023 templates/
-rw-------  1  420  420 1291 Jul 14 17:46 wg0.conf

With each peer having a set of files that are easy to use:

$ ls -la /docker/wireguard-server/mounts/config/peer_Jemoeder
drwx------  2 420 420 4096 Mar 18 11:26 ./
drwxr-xr-x 10 420 420 4096 Jul 14 17:46 ../
-rw-------  1 420 420  345 Jul 14 17:46 peer_Jemoeder.conf
-rw-------  1 420 420 1257 Jul 14 17:46 peer_Jemoeder.png
-rw-------  1 420 420   45 Mar 18 11:26 presharedkey-peer_Jemoeder
-rw-------  1 420 420   45 Mar 18 11:26 privatekey-peer_Jemoeder
-rw-------  1 420 420   45 Mar 18 11:26 publickey-peer_Jemoeder

Basically you can copy over that peer_Jemoeder.conf to the other host’s /etc/wireguard/wg0.conf, run wg-quick up wg0 and it should work.

Ansible role for wireguard

Of course this means setting up wireguard as client on all my machines that didn’t have it yet, so I need an ansible role to easily add these on both my local server and all the machines I would normally ssh to. As one does these days you let the boilerplate be coughed up by an LLM, hit it a few times with a stick to be more concise, jailbreak it to have it stop arguing about all the humans that will be killed as a result of this conversation, and at some point your junior intern might have generated something that you probably could’ve done yourself in the same time. However, rubber ducking does have its merits and I find it enjoyable at times 🙂

Long story short, I now have a role like this:

├── Makefile
├── README.md
├── group_vars
│   └── all
├── inventory
│   └── testhost
├── roles
│   └── wireguard
│       ├── defaults
│       │   └── main.yml
│       ├── tasks
│       │   ├── check_add_peer.yml
│       │   ├── configure_peer.yml
│       │   ├── main.yml
│       ├── templates
│       │   └── peer_config.conf.j2
│       └── vars
│           └── main.yml
└── site.yml

The concept being that I want to be able to run this for a random host and have it add a local entry in the peers, and generate the config file on the other end. It looked like this:

# roles/wireguard/tasks/main.yml
---
- include_tasks: check_add_peer.yml
- include_tasks: configure_peer.yml
  when: peer_added | default(false)

# roles/wireguard/tasks/check_add_peer.yml
---
- name: Read WireGuard server .env file
  ansible.builtin.slurp:
    src: "{{ wireguard_server_env_file }}"
  register: env_file_content
  delegate_to: localhost

- name: Parse PEERS from .env file
  ansible.builtin.set_fact:
    current_peers: "{{ (env_file_content['content'] | b64decode | regex_search('PEERS=([^\n]+)', '\\1')) | first | split(',') }}"

- name: Check if the new peer exists
  ansible.builtin.set_fact:
    peer_exists: "{{ wireguard_peer_name in current_peers }}"

- name: Add new peer if not present
  ansible.builtin.lineinfile:
    path: "{{ wireguard_server_env_file }}"
    regexp: '^PEERS='
    line: "PEERS={{ (current_peers + [wireguard_peer_name]) | join(',') }}"
  when: not peer_exists
  delegate_to: localhost
  register: peer_added

- name: Restart WireGuard server container
  ansible.builtin.command:
    cmd: docker compose up -d
    chdir: /docker/wireguard-server
  when: peer_added.changed
  delegate_to: localhost

# roles/wireguard/tasks/configure_peer.yml
---
- name: Wait for peer configuration files to be created
  ansible.builtin.wait_for:
    path: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/peer_{{ wireguard_peer_name }}.conf"
    state: present
    timeout: 300
  delegate_to: localhost

- name: Read WireGuard server configuration
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_path }}"
  register: wg_server_config
  delegate_to: localhost

- name: Extract peer IP address
  ansible.builtin.set_fact:
    peer_ip: >-
      {{ (wg_server_config['content'] | b64decode | regex_findall('(?m)^# friendly_name=peer_' + wireguard_peer_name + '\n^PublicKey = .*\n^PresharedKey = .*\n^AllowedIPs = ([^/\n]+)') | first) }}

- name: Read WireGuard private key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/privatekey-peer_{{ wireguard_peer_name }}"
  register: private_key_content
  delegate_to: localhost

- name: Read WireGuard SERVER's public key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/server/publickey-server"
  register: public_key_content
  delegate_to: localhost

- name: Read WireGuard preshared key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/presharedkey-peer_{{ wireguard_peer_name }}"
  register: preshared_key_content
  delegate_to: localhost

# Figure out the name of this server
- name: Get Ansible control node hostname
  ansible.builtin.command: hostname -s
  register: ansible_control_hostname
  delegate_to: localhost
  run_once: true
  changed_when: false

- name: Set fact for Ansible control node hostname
  ansible.builtin.set_fact:
    ansible_control_short_hostname: "{{ ansible_control_hostname.stdout | lower }}"

- name: Generate WireGuard peer configuration
  ansible.builtin.template:
    src: peer_config.conf.j2
    dest: "/etc/wireguard/wg-{{ ansible_control_short_hostname }}.conf"
    owner: root
    group: root
    mode: '0600'
  vars:
    wireguard_private_key: "{{ private_key_content['content'] | b64decode | trim }}"
    wireguard_public_key: "{{ public_key_content['content'] | b64decode | trim }}"
    wireguard_preshared_key: "{{ preshared_key_content['content'] | b64decode | trim }}"
    wireguard_peer_ip: "{{ peer_ip }}"

# roles/wireguard/templates/peer_config.conf.j2
#############################
### {{ ansible_managed }} ###
#############################

[Interface]
# Name = {{ wireguard_peer_name  }}
Address = {{ wireguard_peer_ip }}
PrivateKey = {{ wireguard_private_key }}

[Peer]
# friendly_name=peer_{{ wireguard_peer_name }}
PublicKey = {{ wireguard_public_key }}
PresharedKey = {{ wireguard_preshared_key }}
Endpoint = {{ wireguard_peer_endpoint }}
AllowedIPs = {{ wireguard_peer_allowed_ips }}
PersistentKeepalive = 25

The vars files are boring enough:

# roles/wireguard/defaults/main.yml
---
# Defaults for wireguard
wireguard_peer_dns: 0
wireguard_peer_endpoint: "your.mother.com:51820"
wireguard_peer_allowed_ips: "192.168.123.0/24"

# roles/wireguard/vars/main.yml
wireguard_server_env_file: "/docker/wireguard-server/.env"
wireguard_server_config_dir: "/docker/wireguard-server/mounts/config"
wireguard_server_config_path: "{{ wireguard_server_config_dir }}/wg0.conf"

With a test inventory file we can now go ahead and see if it works for our new host.

# inventory/testhost
ungrouped:
  hosts:
    new.testhost.com:
      # Temp IP override while provisioning new host
      ansible_host: 123.123.123.123
      ansible_ssh_private_key_file: /home/ansible/.ssh/id_ecdsa
      ansible_user: ansible
      ansible_become: true
      wireguard_peer_name: TestHost

Result? It works! Or does it….

Ansible results

Of course this went back and forth with the LLM a few times, but it did well. The new peer was generated by the docker, the config was parsed and the template spit out to the new test host. This test host was running Centos 9 Stream, (don’t ask – Slackware and Arch weren’t options), but wireguard-tools were installed, the kernel module loaded, and we now had a /etc/wireguard/wg-jemoeder.conf (since my server is called jemoeder obviously). Nice. And it looked good too:

# /etc/wireguard/wg-jemoeder.conf
#############################
### Ansible managed: peer_config.conf.j2 modified on 2024-07-14 18:02:45 by root on jemoeder.example.com ###
#############################

[Interface]
# Name = TestHost
Address = 10.20.50.6
PrivateKey = APcRD9qFTJzM5pNNd4s4yVmeLO8er5R61oLb1DNmT0k=

[Peer]
# friendly_name=peer_TestHost
PublicKey = 0eGCaYbRJMxDBPlUVKdEw53ucmapD3rQ3udh9cg/oEo=
PresharedKey = gg+1LT2erQng12eELThRfuP0yKt1niAStl2eCWQjQ34=
Endpoint = your.mother.com:51820
AllowedIPs = 192.168.123.0/24
PersistentKeepalive = 25

Great! Time to start it up:

$ wg-quick up wg-jemoeder
[#] ip link add wg-jemoeder type wireguard
[#] wg setconf wg-jemoeder /dev/fd/63
[#] ip -4 address add 10.20.50.6 dev wg-jemoeder
[#] ip link set mtu 1420 up dev wg-jemoeder

$ wg show
interface: wg-jemoeder
  public key: 0eGCaYbRJMxDBPlUVKdEw53ucmapD3rQ3udh9cg/oEo=
  private key: (hidden)
  listening port: 44516

$

Uhhhh….. where is my peer?

Wireguard bug

So what do we see?

• Wireguard came up
• No errors returned
• No errors or warnings in dmesg
• wg-jemoeder interface is there with the correct IP
• No new routes
• No peers, not even with wg show dump or other commands

After jumping high and low, manually running wg set commands and variants, tcpdumping, turning on kernel module debugging and going absolutely crazy for a long time, troubleshooting with LLMs which provide the usual “have you tried turning it off and on again” and “maybe you’re special, try starting from scratch” and “have you checked your wg0 config file for syntax error”, running wg through strace and seeing no errors, and scouring the internet for similar problems, there was no solution in sight.

“Well, dear BenV, what was the outcome of the battle then, certain defeat?!”

Of course not. After raging for a while and tinkering with various bobs of the config, it finally struck me. Turns out the `PublicKey` that our ansible role picked up was indeed a public key…. just the wrong one – its own instead of the server’s key.

UGH.

# in ansible roles/wireguard/tasks/configure_peer.yml

- name: Read WireGuard SERVER's public key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/server/publickey-server"
  register: public_key_content
  delegate_to: localhost

This makes ansible read the correct public key (that the server uses) as opposed to the client’s own key, and after re-running the playbook it works like a charm.

Conclusion

Is this a bug? In my opinion it is, although I can see the confusion on the wireguard side of things where it matches it own keys and somehow deals with it, but as a user this is unacceptable behavior.

I’m defining a [Peer] block, not my own interface, so it should treat it as a foreign entity. If the key matches its own public key it should complain. Is it user error? Of course, but that doesn’t mean it shouldn’t help the user out.

Would this have happened without the use of LLMs as a junior? Probably not, but then again, maybe it would (copy paste has the same effects, the 3 keys would have been copy/paste snippets even when manually writing). That said, this is still on you, Claudippityard….. :p