Ansible Architecture for Wolfy's Server Ecosystem

Configuration

[defaults]
inventory = ./inventories/prod/hosts.yml
log_path = /home/wolfy/log/ansible.log
host_key_checking = True
retry_files_enabled = False
stdout_callback = yaml

[privilege_escalation]
become = True
become_method = sudo
become_ask_pass = False

The current Ansible configuration is set to log internally and not attempt to retry upon failure (in order to prevent infinite retries on certain tasks since the proper safeties have not been put in place yet). Every server gets an ansible user added to it which has passwordless sudo permissions. The account is locked and has a secure password. It uses an ssh key for connecting to the servers.

Playbooks

apt-update-test.yml

---
- name: Update and upgrade wolfywebserver02
  hosts: wolfywebserver02
  become: true
  gather_facts: true

  tasks:
    - name: Update apt cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600

    - name: Upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist

    - name: Remove unneeded packages
      ansible.builtin.apt:
        autoremove: true

This script is currently setup to connect to wolfywebserver02 escalate to sudo then run sudo apt update then sudo apt upgrade then sudo apt autoremove.

update-prod-servers.yml

---
- name: Patch Prod Ubuntu servers and report to Discord
  hosts: prod_ubuntu_servers
  become: true
  gather_facts: true

  vars:
    discord_webhook_id: "1507643788326338630"
    discord_webhook_token: "TkbDnf6naR7KxhqLDIvGJyeesq9OTglxHrLHyy3d6_1WiyI-bbzUGDeVu1yCjZN7fCdd"

  tasks:
    - name: Update apt cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
      register: apt_cache_result

    - name: Upgrade installed packages
      ansible.builtin.apt:
        upgrade: dist
      register: apt_upgrade_result

    - name: Remove unneeded packages
      ansible.builtin.apt:
        autoremove: true
      register: apt_autoremove_result

    - name: Check if reboot is required
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_required_file

    - name: Reboot server if required
      ansible.builtin.reboot:
        pre_reboot_delay: 60
        post_reboot_delay: 60
        reboot_timeout: 600
        msg: "Rebooting in 60 seconds (intiated by Ansible)"
      when: reboot_required_file.stat.exists
      register: reboot_result

    - name: Build per-host summary
      ansible.builtin.set_fact:
        patch_summary: |
          Host: {{ inventory_hostname }}
            Apt Cache Updated: {{ 'Yes' if (apt_cache_result.changed | default(false)) else 'No' }}
            Packages Upgraded: {{ 'Yes' if (apt_upgrade_result.changed | default(false)) else 'No' }}
            Autoremove Changed: {{ 'Yes' if (apt_autoremove_result.changed | default(false)) else 'No' }}
            Reboot Required:    {{ 'Yes' if (reboot_required_file.stat.exists | default(false)) else 'No' }}
            Reboot Performed:   {{ 'Yes' if (reboot_result.changed | default(false)) else 'No' }}

    - name: Send Discord report
      community.general.discord:
        webhook_id: "{{ discord_webhook_id }}"
        webhook_token: "{{ discord_webhook_token }}"
        username: "Ansible"
        avatar_url: "https://opreviews.anime-pictures.net/49f/49f1117eec83be2bd5af12772a70d2e4_bp.avif"
        content: |
          ========================================
          Test Servers Update Report
          Generated: {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }}
          ========================================

          Total Hosts: {{ ansible_play_hosts_all | length }}

          {% for host in ansible_play_hosts_all | sort %}
          {{ hostvars[host].patch_summary | default('Host: ' ~ host ~ '\n  No summary generated\n') }}

          {% endfor %}
          ========================================
          Summary
          ========================================
          Hosts Requiring Reboot: {{
            ansible_play_hosts_all
            | select('in', ansible_play_hosts_all)
            | map('extract', hostvars)
            | selectattr('reboot_required_file.stat.exists', 'defined')
            | selectattr('reboot_required_file.stat.exists')
            | list
            | length
          }}

          Hosts Rebooted: {{
            ansible_play_hosts_all
            | select('in', ansible_play_hosts_all)
            | map('extract', hostvars)
            | selectattr('reboot_result.changed', 'defined')
            | selectattr('reboot_result.changed')
            | list
            | length
          }}
      delegate_to: localhost
      run_once: true
      become: false

This playbook is designed to connect all prod_ubuntu_servers hosts and run update, upgrade, autoremove, and restart if necessary then send out a webhook with details on how each step went on each server.

Run

update-prod-servers.sh

#!/bin/bash
set -euo pipefail

LOG_DIR="/home/wolfy/log/ansible-prod"
mkdir -p "$LOG_DIR"

TIMESTAMP="$(date '+%Y-%m-%d_%H-%M-%S')"
LOG_FILE="$LOG_DIR/ubuntu-prod-update-$TIMESTAMP.log"

exec >>"$LOG_FILE" 2>&1

echo "========================================"
echo "Prod patching run started: $(date -Is)"
echo "========================================"

cd /home/wolfy/ansible

/home/wolfy/ansible-env/bin/ansible-playbook playbooks/update-prod-servers.yml

echo
echo "========================================"
echo "Prod patching run completed: $(date -Is)"
echo "========================================"

This wrapper script is ran by update-prod-servers.service a systemd service which is triggered by update-prod-servers.timer a systemd timer. This is designed to trigger every Monday at 7:30 EST and update all production ubuntu servers using update-prod-servers.yml.

update-prod-servers.service

[Unit]
Description=Run weekly production patching playbook
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=wolfy
Group=wolfy
ExecStart=/home/wolfy/ansible/run/update-prod-servers.sh
WorkingDirectory=/home/wolfy/ansible

update-prod-servers.timer

[Unit]
Description=Weekly production patching timer

[Timer]
OnCalendar=Mon *-*-* 07:30:00
Persistent=true
Unit=update-prod-servers.service

[Install]
WantedBy=timers.target

Inventory

wolfywebserver02

• Hostname: wolfywebserver02
• OS: Ubuntu 24.04.4 LTS
• VM Host: wolfyvmserver
• Ansible Status: Ready
• Ansible Category: ubuntu_servers

This server is currently being used for running audiobookshelf so that my friends can access the server and listen to audiobooks on their phone or computer using an app or the webui. The friends are networked via Tailscale.

wolfyminecraftserver

• Hostname: wolfyminecraftserver
• OS: Ubuntu 24.04.4 LTS
• VM Host: wolfyvmserver
• Ansible Status: Ready
• Ansible Category: ubuntu_servers

This server is currently being used to host MoreModsReloaded server for Rockman and his friends. It is running Crafty Controller to manage the Minecraft Server and provide an admin web UI so that Rockman and a few admins can make changes to the server if they need to. Playit service is used to tunnel a port on the server to a playit cloud server that reverse proxies outside users to my server without exposing my IP. Tailscale is used to allow Rockman and Waluu to access the Crafty Controller web UI.

wolfydiscordbot01

• Hostname: wolfydiscordbot01
• OS: Ubuntu 24.04.4 LTS
• VM Host: wolfyvmserver
• Ansible Status: Ready
• Ansible Category: ubuntu_servers

TO DO

Notes

Setting up Ansible Access

Bootstrap Script

.bootstrap/ contains a script that can be used to setup ansible on new servers.

Debian/Ubuntu

1. On the control node generate a dedicated key

ssh-keygen -t ed25519 -f ~/.ssh/ansible-ed25519 -C "ansible-control"

2. Create the ansible user and give it sudo permissions

# For Password
openssl rand -base64 16

# Create account and give sudo permissions
sudo adduser ansible
sudo usermod -aG sudo ansible

3. Allow passwordless sudo for the ansible account

sudo visudo

Add:

ansible ALL=(ALL) NOPASSWD: ALL

4. On the control node copy the ssh key over to the target server

ssh-copy-id -i ~/.ssh/ansible-ed25519.pub ansible@server.tailnet.ts.net

Test:

ssh ansible@server.tailnet.ts.net

5. Lock the ansible account to prevent login attempts

sudo passwd -l ansible

To-Do