# Ansible Architecture for Wolfy's Server Ecosystem ## Configuration ```INI [defaults] inventory = ./inventories/prod/hosts.yml log_path = /home/wolfy/log/ansible.log host_key_checking = True retry_files_enabled = False stdout_callback = yaml [privilege_escalation] become = True become_method = sudo become_ask_pass = False ``` The current Ansible configuration is set to log internally and not attempt to retry upon failure (in order to prevent infinite retries on certain tasks since the proper safeties have not been put in place yet). Every server gets an `ansible` user added to it which has passwordless sudo permissions. The account is locked and has a secure password. It uses an ssh key for connecting to the servers. ## Playbooks ### apt-update-test.yml ```YAML --- - name: Update and upgrade wolfywebserver02 hosts: wolfywebserver02 become: true gather_facts: true tasks: - name: Update apt cache ansible.builtin.apt: update_cache: true cache_valid_time: 3600 - name: Upgrade installed packages ansible.builtin.apt: upgrade: dist - name: Remove unneeded packages ansible.builtin.apt: autoremove: true ``` This script is currently setup to connect to `wolfywebserver02` escalate to sudo then run `sudo apt update` then `sudo apt upgrade` then `sudo apt autoremove`. ### update-prod-servers.yml ```YAML --- - name: Patch Prod Ubuntu servers and report to Discord hosts: prod_ubuntu_servers become: true gather_facts: true vars: discord_webhook_id: "1507643788326338630" discord_webhook_token: "TkbDnf6naR7KxhqLDIvGJyeesq9OTglxHrLHyy3d6_1WiyI-bbzUGDeVu1yCjZN7fCdd" tasks: - name: Update apt cache ansible.builtin.apt: update_cache: true cache_valid_time: 3600 register: apt_cache_result - name: Upgrade installed packages ansible.builtin.apt: upgrade: dist register: apt_upgrade_result - name: Remove unneeded packages ansible.builtin.apt: autoremove: true register: apt_autoremove_result - name: Check if reboot is required ansible.builtin.stat: path: /var/run/reboot-required register: reboot_required_file - name: Reboot server if required ansible.builtin.reboot: pre_reboot_delay: 60 post_reboot_delay: 60 reboot_timeout: 600 msg: "Rebooting in 60 seconds (intiated by Ansible)" when: reboot_required_file.stat.exists register: reboot_result - name: Build per-host summary ansible.builtin.set_fact: patch_summary: | Host: {{ inventory_hostname }} Apt Cache Updated: {{ 'Yes' if (apt_cache_result.changed | default(false)) else 'No' }} Packages Upgraded: {{ 'Yes' if (apt_upgrade_result.changed | default(false)) else 'No' }} Autoremove Changed: {{ 'Yes' if (apt_autoremove_result.changed | default(false)) else 'No' }} Reboot Required: {{ 'Yes' if (reboot_required_file.stat.exists | default(false)) else 'No' }} Reboot Performed: {{ 'Yes' if (reboot_result.changed | default(false)) else 'No' }} - name: Send Discord report community.general.discord: webhook_id: "{{ discord_webhook_id }}" webhook_token: "{{ discord_webhook_token }}" username: "Ansible" avatar_url: "https://opreviews.anime-pictures.net/49f/49f1117eec83be2bd5af12772a70d2e4_bp.avif" content: | ======================================== Test Servers Update Report Generated: {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }} ======================================== Total Hosts: {{ ansible_play_hosts_all | length }} {% for host in ansible_play_hosts_all | sort %} {{ hostvars[host].patch_summary | default('Host: ' ~ host ~ '\n No summary generated\n') }} {% endfor %} ======================================== Summary ======================================== Hosts Requiring Reboot: {{ ansible_play_hosts_all | select('in', ansible_play_hosts_all) | map('extract', hostvars) | selectattr('reboot_required_file.stat.exists', 'defined') | selectattr('reboot_required_file.stat.exists') | list | length }} Hosts Rebooted: {{ ansible_play_hosts_all | select('in', ansible_play_hosts_all) | map('extract', hostvars) | selectattr('reboot_result.changed', 'defined') | selectattr('reboot_result.changed') | list | length }} delegate_to: localhost run_once: true become: false ``` This playbook is designed to connect all `prod_ubuntu_servers` hosts and run update, upgrade, autoremove, and restart if necessary then send out a webhook with details on how each step went on each server. ## Run ### update-prod-servers.sh ```bash #!/bin/bash set -euo pipefail LOG_DIR="/home/wolfy/log/ansible-prod" mkdir -p "$LOG_DIR" TIMESTAMP="$(date '+%Y-%m-%d_%H-%M-%S')" LOG_FILE="$LOG_DIR/ubuntu-prod-update-$TIMESTAMP.log" exec >>"$LOG_FILE" 2>&1 echo "========================================" echo "Prod patching run started: $(date -Is)" echo "========================================" cd /home/wolfy/ansible /home/wolfy/ansible-env/bin/ansible-playbook playbooks/update-prod-servers.yml echo echo "========================================" echo "Prod patching run completed: $(date -Is)" echo "========================================" ``` This wrapper script is ran by `update-prod-servers.service` a systemd service which is triggered by `update-prod-servers.timer` a systemd timer. This is designed to trigger every Monday at 7:30 EST and update all production ubuntu servers using `update-prod-servers.yml`. #### update-prod-servers.service ```ini [Unit] Description=Run weekly production patching playbook Wants=network-online.target After=network-online.target [Service] Type=oneshot User=wolfy Group=wolfy ExecStart=/home/wolfy/ansible/run/update-prod-servers.sh WorkingDirectory=/home/wolfy/ansible ``` #### update-prod-servers.timer ```ini [Unit] Description=Weekly production patching timer [Timer] OnCalendar=Mon *-*-* 07:30:00 Persistent=true Unit=update-prod-servers.service [Install] WantedBy=timers.target ``` ## Inventory ### wolfywebserver02 - Hostname: wolfywebserver02 - OS: Ubuntu 24.04.4 LTS - VM Host: wolfyvmserver - Ansible Status: Ready - Ansible Category: ubuntu_servers This server is currently being used for running audiobookshelf so that my friends can access the server and listen to audiobooks on their phone or computer using an app or the webui. The friends are networked via Tailscale. ### wolfyminecraftserver - Hostname: wolfyminecraftserver - OS: Ubuntu 24.04.4 LTS - VM Host: wolfyvmserver - Ansible Status: Ready - Ansible Category: ubuntu_servers This server is currently being used to host MoreModsReloaded server for Rockman and his friends. It is running Crafty Controller to manage the Minecraft Server and provide an admin web UI so that Rockman and a few admins can make changes to the server if they need to. Playit service is used to tunnel a port on the server to a playit cloud server that reverse proxies outside users to my server without exposing my IP. Tailscale is used to allow Rockman and Waluu to access the Crafty Controller web UI. ### wolfydiscordbot01 - Hostname: wolfydiscordbot01 - OS: Ubuntu 24.04.4 LTS - VM Host: wolfyvmserver - Ansible Status: Ready - Ansible Category: ubuntu_servers TO DO ## Notes ### Setting up Ansible Access #### Bootstrap Script `.bootstrap/` contains a script that can be used to setup ansible on new servers. #### Debian/Ubuntu 1. On the control node generate a dedicated key ```bash ssh-keygen -t ed25519 -f ~/.ssh/ansible-ed25519 -C "ansible-control" ``` 2. Create the ansible user and give it sudo permissions ```bash # For Password openssl rand -base64 16 # Create account and give sudo permissions sudo adduser ansible sudo usermod -aG sudo ansible ``` 3. Allow passwordless sudo for the ansible account ```bash sudo visudo ``` Add: ```ini ansible ALL=(ALL) NOPASSWD: ALL ``` 4. On the control node copy the ssh key over to the target server ```bash ssh-copy-id -i ~/.ssh/ansible-ed25519.pub ansible@server.tailnet.ts.net ``` Test: ```bash ssh ansible@server.tailnet.ts.net ``` 5. Lock the ansible account to prevent login attempts ```bash sudo passwd -l ansible ``` ### To-Do