BenV's notes

New Machine, New problems

One of my VMs has been running on ancient deprecated hardware for $forever now (without problems I might add), but after getting notice that it will be shutdown in 2 months together with an upgrade path that only costs me time and will result in double the specs, I decided to start the upgrade.

In order to do things properly, I started cobbling together some ansible roles for things I don’t want to repeat. One of these roles you can guess based on the title, wireguard. Needless to say things never work as you think they will, this is one of those stories.

Wireguard as point of entry

Given the SSH shenanigans ( CVE-2024-6387 ) that keep popping up (CVE-2024-6409) lately (CVE-2024-3094 as well this year), combined with the ease of exploitation, and constant port scans that have become snafu (even with fail2ban blocking countless IPs), I’ve finally decided to get rid of public SSH. If we need to run VPNs anyway, we might as well make that the only publicly exposed attack vector where possible and do the rest through internal networking. This will give the bonus that port 22 can turn into a honeypot, banning everything that tries to connect there.

Wireguard setup

My wireguard servers run through docker, specifically this one (with some local customizations to add some tooling). This allows me to have a .env file in my wireguard docker directory that has a PEERS=Peer1,Peer2,Jemoeder,Peer4 etc, and when restarting the docker it will create the peer configs for me.

The docker-compose.yaml looks a bit like this:

services:
  wireguard:
    image: linuxserver/wireguard
    container_name: WireGuard-Server
    cap_add:
      - NET_ADMIN
    environment:
      - PUID=420
      - PGID=420
      - TZ=Europe/Amsterdam
      - SERVERURL=your.mother.com
      - SERVERPORT=51820
      - PEERS=${PEERS}
      - INTERNAL_SUBNET=${INTERNAL_SUBNET}
      - ALLOWEDIPS=0.0.0.0/0
      - LOG_CONFS=true
    volumes:
      - /docker/wireguard-server/mounts/config:/config
    network_mode: host
    restart: unless-stopped

When peers are added, you’ll find yourself a directory like this:

$ ls -la /docker/wireguard-server/mounts/config
drwxr-xr-x 10  420  420 4096 Jul 14 17:46 ./
drwxr-xr-x  3 root root 4096 May 10  2023 ../
-rw-------  1  420  420  211 Jul 14 17:46 .donoteditthisfile
drwxr-xr-x  2  420  420 4096 May 10  2023 coredns/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer1/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer2/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Peer3/
drwx------  2  420  420 4096 Mar 18 11:26 peer_Jemoeder/
drwxr-xr-x  2  420  420 4096 Jul 14 17:46 peer_Peer4/
drwxr-xr-x  2  420  420 4096 Mar 18 11:26 server/
drwxr-xr-x  2  420  420 4096 May 10  2023 templates/
-rw-------  1  420  420 1291 Jul 14 17:46 wg0.conf

With each peer having a set of files that are easy to use:

$ ls -la /docker/wireguard-server/mounts/config/peer_Jemoeder
drwx------  2 420 420 4096 Mar 18 11:26 ./
drwxr-xr-x 10 420 420 4096 Jul 14 17:46 ../
-rw-------  1 420 420  345 Jul 14 17:46 peer_Jemoeder.conf
-rw-------  1 420 420 1257 Jul 14 17:46 peer_Jemoeder.png
-rw-------  1 420 420   45 Mar 18 11:26 presharedkey-peer_Jemoeder
-rw-------  1 420 420   45 Mar 18 11:26 privatekey-peer_Jemoeder
-rw-------  1 420 420   45 Mar 18 11:26 publickey-peer_Jemoeder

Basically you can copy over that peer_Jemoeder.conf to the other host’s /etc/wireguard/wg0.conf, run wg-quick up wg0 and it should work.

Ansible role for wireguard

Of course this means setting up wireguard as client on all my machines that didn’t have it yet, so I need an ansible role to easily add these on both my local server and all the machines I would normally ssh to. As one does these days you let the boilerplate be coughed up by an LLM, hit it a few times with a stick to be more concise, jailbreak it to have it stop arguing about all the humans that will be killed as a result of this conversation, and at some point your junior intern might have generated something that you probably could’ve done yourself in the same time. However, rubber ducking does have its merits and I find it enjoyable at times 🙂

Long story short, I now have a role like this:

├── Makefile
├── README.md
├── group_vars
│   └── all
├── inventory
│   └── testhost
├── roles
│   └── wireguard
│       ├── defaults
│       │   └── main.yml
│       ├── tasks
│       │   ├── check_add_peer.yml
│       │   ├── configure_peer.yml
│       │   ├── main.yml
│       ├── templates
│       │   └── peer_config.conf.j2
│       └── vars
│           └── main.yml
└── site.yml

The concept being that I want to be able to run this for a random host and have it add a local entry in the peers, and generate the config file on the other end. It looked like this:

# roles/wireguard/tasks/main.yml
---
- include_tasks: check_add_peer.yml
- include_tasks: configure_peer.yml
  when: peer_added | default(false)

# roles/wireguard/tasks/check_add_peer.yml
---
- name: Read WireGuard server .env file
  ansible.builtin.slurp:
    src: "{{ wireguard_server_env_file }}"
  register: env_file_content
  delegate_to: localhost

- name: Parse PEERS from .env file
  ansible.builtin.set_fact:
    current_peers: "{{ (env_file_content['content'] | b64decode | regex_search('PEERS=([^\n]+)', '\\1')) | first | split(',') }}"

- name: Check if the new peer exists
  ansible.builtin.set_fact:
    peer_exists: "{{ wireguard_peer_name in current_peers }}"

- name: Add new peer if not present
  ansible.builtin.lineinfile:
    path: "{{ wireguard_server_env_file }}"
    regexp: '^PEERS='
    line: "PEERS={{ (current_peers + [wireguard_peer_name]) | join(',') }}"
  when: not peer_exists
  delegate_to: localhost
  register: peer_added

- name: Restart WireGuard server container
  ansible.builtin.command:
    cmd: docker compose up -d
    chdir: /docker/wireguard-server
  when: peer_added.changed
  delegate_to: localhost

# roles/wireguard/tasks/configure_peer.yml
---
- name: Wait for peer configuration files to be created
  ansible.builtin.wait_for:
    path: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/peer_{{ wireguard_peer_name }}.conf"
    state: present
    timeout: 300
  delegate_to: localhost

- name: Read WireGuard server configuration
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_path }}"
  register: wg_server_config
  delegate_to: localhost

- name: Extract peer IP address
  ansible.builtin.set_fact:
    peer_ip: >-
      {{ (wg_server_config['content'] | b64decode | regex_findall('(?m)^# friendly_name=peer_' + wireguard_peer_name + '\n^PublicKey = .*\n^PresharedKey = .*\n^AllowedIPs = ([^/\n]+)') | first) }}

- name: Read WireGuard private key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/privatekey-peer_{{ wireguard_peer_name }}"
  register: private_key_content
  delegate_to: localhost

- name: Read WireGuard SERVER's public key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/server/publickey-server"
  register: public_key_content
  delegate_to: localhost

- name: Read WireGuard preshared key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/peer_{{ wireguard_peer_name }}/presharedkey-peer_{{ wireguard_peer_name }}"
  register: preshared_key_content
  delegate_to: localhost

# Figure out the name of this server
- name: Get Ansible control node hostname
  ansible.builtin.command: hostname -s
  register: ansible_control_hostname
  delegate_to: localhost
  run_once: true
  changed_when: false

- name: Set fact for Ansible control node hostname
  ansible.builtin.set_fact:
    ansible_control_short_hostname: "{{ ansible_control_hostname.stdout | lower }}"

- name: Generate WireGuard peer configuration
  ansible.builtin.template:
    src: peer_config.conf.j2
    dest: "/etc/wireguard/wg-{{ ansible_control_short_hostname }}.conf"
    owner: root
    group: root
    mode: '0600'
  vars:
    wireguard_private_key: "{{ private_key_content['content'] | b64decode | trim }}"
    wireguard_public_key: "{{ public_key_content['content'] | b64decode | trim }}"
    wireguard_preshared_key: "{{ preshared_key_content['content'] | b64decode | trim }}"
    wireguard_peer_ip: "{{ peer_ip }}"

# roles/wireguard/templates/peer_config.conf.j2
#############################
### {{ ansible_managed }} ###
#############################

[Interface]
# Name = {{ wireguard_peer_name  }}
Address = {{ wireguard_peer_ip }}
PrivateKey = {{ wireguard_private_key }}

[Peer]
# friendly_name=peer_{{ wireguard_peer_name }}
PublicKey = {{ wireguard_public_key }}
PresharedKey = {{ wireguard_preshared_key }}
Endpoint = {{ wireguard_peer_endpoint }}
AllowedIPs = {{ wireguard_peer_allowed_ips }}
PersistentKeepalive = 25

The vars files are boring enough:

# roles/wireguard/defaults/main.yml
---
# Defaults for wireguard
wireguard_peer_dns: 0
wireguard_peer_endpoint: "your.mother.com:51820"
wireguard_peer_allowed_ips: "192.168.123.0/24"

# roles/wireguard/vars/main.yml
wireguard_server_env_file: "/docker/wireguard-server/.env"
wireguard_server_config_dir: "/docker/wireguard-server/mounts/config"
wireguard_server_config_path: "{{ wireguard_server_config_dir }}/wg0.conf"

With a test inventory file we can now go ahead and see if it works for our new host.

# inventory/testhost
ungrouped:
  hosts:
    new.testhost.com:
      # Temp IP override while provisioning new host
      ansible_host: 123.123.123.123
      ansible_ssh_private_key_file: /home/ansible/.ssh/id_ecdsa
      ansible_user: ansible
      ansible_become: true
      wireguard_peer_name: TestHost

Result? It works! Or does it….

Ansible results

Of course this went back and forth with the LLM a few times, but it did well. The new peer was generated by the docker, the config was parsed and the template spit out to the new test host. This test host was running Centos 9 Stream, (don’t ask – Slackware and Arch weren’t options), but wireguard-tools were installed, the kernel module loaded, and we now had a /etc/wireguard/wg-jemoeder.conf (since my server is called jemoeder obviously). Nice. And it looked good too:

# /etc/wireguard/wg-jemoeder.conf
#############################
### Ansible managed: peer_config.conf.j2 modified on 2024-07-14 18:02:45 by root on jemoeder.example.com ###
#############################

[Interface]
# Name = TestHost
Address = 10.20.50.6
PrivateKey = APcRD9qFTJzM5pNNd4s4yVmeLO8er5R61oLb1DNmT0k=

[Peer]
# friendly_name=peer_TestHost
PublicKey = 0eGCaYbRJMxDBPlUVKdEw53ucmapD3rQ3udh9cg/oEo=
PresharedKey = gg+1LT2erQng12eELThRfuP0yKt1niAStl2eCWQjQ34=
Endpoint = your.mother.com:51820
AllowedIPs = 192.168.123.0/24
PersistentKeepalive = 25

Great! Time to start it up:

$ wg-quick up wg-jemoeder
[#] ip link add wg-jemoeder type wireguard
[#] wg setconf wg-jemoeder /dev/fd/63
[#] ip -4 address add 10.20.50.6 dev wg-jemoeder
[#] ip link set mtu 1420 up dev wg-jemoeder

$ wg show
interface: wg-jemoeder
  public key: 0eGCaYbRJMxDBPlUVKdEw53ucmapD3rQ3udh9cg/oEo=
  private key: (hidden)
  listening port: 44516

$

Uhhhh….. where is my peer?

Wireguard bug

So what do we see?

• Wireguard came up
• No errors returned
• No errors or warnings in dmesg
• wg-jemoeder interface is there with the correct IP
• No new routes
• No peers, not even with wg show dump or other commands

After jumping high and low, manually running wg set commands and variants, tcpdumping, turning on kernel module debugging and going absolutely crazy for a long time, troubleshooting with LLMs which provide the usual “have you tried turning it off and on again” and “maybe you’re special, try starting from scratch” and “have you checked your wg0 config file for syntax error”, running wg through strace and seeing no errors, and scouring the internet for similar problems, there was no solution in sight.

“Well, dear BenV, what was the outcome of the battle then, certain defeat?!”

Of course not. After raging for a while and tinkering with various bobs of the config, it finally struck me. Turns out the `PublicKey` that our ansible role picked up was indeed a public key…. just the wrong one – its own instead of the server’s key.

UGH.

# in ansible roles/wireguard/tasks/configure_peer.yml

- name: Read WireGuard SERVER's public key
  ansible.builtin.slurp:
    src: "{{ wireguard_server_config_dir }}/server/publickey-server"
  register: public_key_content
  delegate_to: localhost

This makes ansible read the correct public key (that the server uses) as opposed to the client’s own key, and after re-running the playbook it works like a charm.

Conclusion

Is this a bug? In my opinion it is, although I can see the confusion on the wireguard side of things where it matches it own keys and somehow deals with it, but as a user this is unacceptable behavior.

I’m defining a [Peer] block, not my own interface, so it should treat it as a foreign entity. If the key matches its own public key it should complain. Is it user error? Of course, but that doesn’t mean it shouldn’t help the user out.

Would this have happened without the use of LLMs as a junior? Probably not, but then again, maybe it would (copy paste has the same effects, the 3 keys would have been copy/paste snippets even when manually writing). That said, this is still on you, Claudippityard….. :p

Yay, another new linux kernel. Of course this broke the closed NVidia driver, I tried 331.38:

[ 110.917769] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)

The fix for this is easy, unpack the installer and then patch file kernel/nv-acpi.c hack out line 306:

// NV_ACPI_OS_WAIT_EVENTS_COMPLETE();

Run nvidia-installer and you should be good to go. Incredible that it always takes NVidia forever to fix these things, this bug has been known since at least 3.13-rc1. Oh well.

Next up, testing nftables. Looks like iptables will soon be obsolete 🙂
Here’s a nice howto to get you started – https://home.regit.org/netfilter-en/nftables-quick-howto/.
Update:
Might you be wondering (like me) where the heck the libnftables git repository went: they decided to rename the thing to libnftnl. You can find the repository here – https://git.netfilter.org/libnftnl/

Another thing: compiling nftables from git breaks on my Slackware64-current machine atm because 1. configure.ac still points to libnftables (instead of libnftl), I assume this will be fixed within the hour. 2: it tries to compile against libreadline without linking libncurses.
Fix for that is running configure like: LDFLAGS=-lncurses ./configure

So far I like what Valve has been doing with Steam on Linux. Obviously there’s always a lot to be improved, but they try and it works pretty well for me.
Since they don’t have a native 64 bit steam client yet and I don’t want to clobber my Slackware64 install with multilibs and other 32 bit garbage I still have my old 32 bit Slackware-current install mounted that I can chroot into and run 32 bit garbage.
Running steam for me looks a bit like:

benv@steammachine:~$ su -lc "chroot /slackware32"
Password: jemoeder
root@steammachine:~# su benv
benv@steammachine:~$ steam


A bit of a hassle but it works, and I need that 32 bit chroot for running windows garbage in wine32 anyway.

Lately I noticed a lot of games that have a linux version under Steam crashed. When running them a window pops up, disappears, and your last played date is set to “Today” in steam. Trying to run the game on the console shows something like this:

benv@steammachine:~/.local/share/Steam/SteamApps/common/Ittle Dew:0>./IttleDew.x86
Set current directory to /home/benv/.local/share/Steam/SteamApps/common/Ittle Dew
Found path: /home/benv/.local/share/Steam/SteamApps/common/Ittle Dew/IttleDew.x86
Mono path[0] = '/home/benv/.local/share/Steam/SteamApps/common/Ittle Dew/IttleDew_Data/Managed'
Mono path[1] = '/home/benv/.local/share/Steam/SteamApps/common/Ittle Dew/IttleDew_Data/Mono'
Mono config path = '/home/benv/.local/share/Steam/SteamApps/common/Ittle Dew/IttleDew_Data/Mono/etc'
Aborted (core dumped)

The reason was simple, but took me a lot of digging through strace etc to figure out. (gdb didn’t help since the included libmono that caused this segmentation fault was obviously stripped). At some point the program tries to read from /sys/devices/, which failed since I hadn’t bothered to mount /sys in my chroot. My mistake. (note that I did bother to mount /dev to get gamepad support etc).

Strace output:
[pid 16978] openat(AT_FDCWD, "/sys/devices/", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
segfault

Conclusion: make sure that Steam for Linux games have access to /sys/devices/ 😉

Good news everyone! Linux 3.8 has been released! Obviously I immediately fired up my compiled to upgrade from the by now ancient 3.6.8 kernel that I was running.
After rebooting my Slackware64 machine into the new kernel without a problem it was time to recompile the NVIDIA binary blob. You know, this piece of garbage. It doesn’t matter if you pick the latest official release version (seems to be 310.32 atm) or the beta that I picked, it won’t compile.
Running the installer will fail and leave you a /var/log/nvidia-installer.log that contains something like this:

root@machine:/# tail /var/log/nvidia-installer.log
/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c: In function ‘nv_kern_open’:
/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c:1521:30: warning: passing argument 2 of ‘request_irq’ from incompatible pointer type [enabled by default]
In file included from /tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv-linux.h:128:0,
from /tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c:13:
include/linux/interrupt.h:130:1: note: expected ‘irq_handler_t’ but argument is of type ‘enum irqreturn_t (*)(int, void *, struct pt_regs *)’
/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c:1525:17: error: implicit declaration of function ‘NV_TASKQUEUE_INIT’ [-Werror=implicit-function-declaration]
/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c:1537:25: warning: passing argument 2 of ‘request_irq’ from incompatible pointer type [enabled by default]
In file included from /tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv-linux.h:128:0,
from /tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.c:13:
include/linux/interrupt.h:130:1: note: expected ‘irq_handler_t’ but argument is of type ‘enum irqreturn_t (*)(int, void *, struct pt_regs *)’
cc1: some warnings being treated as errors
make[3]: *** [/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel/nv.o] Error 1
make[2]: *** [_module_/tmp/selfgz7139/NVIDIA-Linux-x86_64-313.18/kernel] Error 2
NVIDIA: left KBUILD.
nvidia.ko failed to build!
make[1]: *** [module] Error 1
make: *** [module] Error 2
-> Error.
ERROR: Unable to build the NVIDIA kernel module.

Fortunately here’s a little patch you can run to fix it. This assumes you have your linux 3.8 kernel sources symlinked in /usr/src/linux!

root@machine:/usr/src# wget -q ftp://download.nvidia.com/XFree86/Linux-x86_64/313.18/NVIDIA-Linux-x86_64-313.18.run
root@machine:/usr/src# bash NVIDIA-Linux-x86_64-313.18.run -x
root@machine:/usr/src# cd NVIDIA-Linux-x86_64-313.18
root@machine:/usr/src/NVIDIA-Linux-x86_64-313.18# wget -q http://notes.benv.junerules.com/wp-content/uploads/2013/02/nvidia-313.18-linux-3.8.patch
root@machine:/usr/src/NVIDIA-Linux-x86_64-313.18# patch -p1 < nvidia-313.18-linux-3.8.patch root@machine:/usr/src/NVIDIA-Linux-x86_64-313.18# ./nvidia-installer # now it should work
Works for me at least 😉

Yep, you read it right, they already released it. Not much new to report about the client though, they fixed a few things, but still no Slackware release, only Ubuntu. No problem there though, you can follow my post on how to get Steam for Linux running in a 32-bit Slackware chroot environment.

As you can see from the image above, they’re even having a celebration sale! An excellent time to pick up a few great games such as Faster Than Light — although you’re probably better off buying it from the Faster Than Light site so you’ll also get the DRM-free version 😉

Keep up the good work Steam!

Every now and then you find out that this huge disk you’ve been using — you know, the one that when you bought it you thought “How on earth am I ever going to fill this one up? My biggest game can fit on this disk 100 times!” — … isn’t as huge anymore. Or at least all the free space on it has disappeared and nagios is whining that your disk is full or about to explode.
Some background info: My fileserver here at home has 3 linux software raid arrays (raid-1 mirrors) on top of 4 physical disks. The first and also smallest array is used as root filesystem to boot from into Slackware linux. The second and third arrays are both big and simply for storage of games, music, series, etc.
When I created that first array a few years ago I figured “Hm, 20GB should be enough for a slackware install, right? Well, let’s just make it 50GB to be sure, we have plenty of space anyway on this huge disk“. Back then the ‘huge’ disks were 500GB. Meanwhile those 500GBs have been replaced with 1TB ones, but that array remained the same. Today I have a set of 1.5TB drives to replace the 1TB ones. Not a huge upgrade, but I didn’t have to buy these disks since they came from a server that had its drives upgraded as well. Anyhow, the 50GB partition managed to get filled with over 40GB of stuff that I can’t trash (mostly user home directories). I could move them to a different partition of course, but today we’re going to resize that partition to 100GB and put the rest in the storage partition.
Off-topic note: Do you also hate it when you’re typing in a browser and hit CRTL-w to delete your last word and realize you just closed your tab? I sure as heck do, good thing wordpress saves these drafts every now and then 🙂 (continue reading…)

More and more rumors were popping up about Steam coming to linux.
With the appearance of Windows 8 and their marketplace it’s not strange to see Steam trying to cover their asses.
If Microsoft is successful in convincing users to buy their garbage through the new M$ Marketplace Steam might lose a lot of customers.
Then again, people like me who already have a ton of stuff on steam will likely stick.
Also it’s quite likely that Steam will stay leaps ahead of the marketplace in features (cloud saves, achievements, ease of use, auto patching, etc).
Another possibly reason for Steam to go to linux is that they might be working on their own game console. If that thing works on linux that would be very interesting 🙂

Today Valve opened their Steam for Linux Beta Survey, you can find it here: http://www.valvesoftware.com/linuxsurvey.php
Seems like they’re trying to gather a batch of experienced linux users to try out their limited beta. In their blog post they mention 1000 users for the limited beta, so fill out the survey and cross your fingers 🙂
And if you ask me, nothing says experienced linux user like ‘Slackware linux!’ 🙂
Or if you’re on Pokemon OS you can simply wait until they open up the stuff for everyone, it can’t be long now 🙂

Some days after tinkering for a little bit you come to the realization that it might be better to stop doing anything with devices and just wait for the day to pass, because everything you touch breaks in the most spectacular ways. Of course this never stopped me from breaking even more, but I’m stupid like that.
Today is a day like that it seems. First our ADSL line at home received an upgrade to FTTH (aka a fiber connection), boosting our internet speed from a lousy 8Mbit down to 50Mbit down, and from less than 1Mbit upstream to 50Mbit upstream. (continue reading…)

So I still run Slackware Linux on pretty much all of my machines, which I often smile upon when I see another Ubuntu/Debian/Pokemon update break stuff (that I then get to fix, details). However, running bleeding edge Slackware-current also draws some blood every now and then. (continue reading…)

In an attempt to try out the latest DRBD version (8.4.1 atm) on a new Xen installation, I ran into some issues.
Initially I built the kernel with the included DRBD module, but I soon realized that this version was older than I wanted to play with (the included version was 8.3.11), so I grabbed the source and tried to build it myself. (continue reading…)