302 lines
8.4 KiB
Markdown
302 lines
8.4 KiB
Markdown
# Ansible Architecture for Wolfy's Server Ecosystem
|
|
|
|
## Configuration
|
|
|
|
```INI
|
|
[defaults]
|
|
inventory = ./inventories/prod/hosts.yml
|
|
log_path = /home/wolfy/log/ansible.log
|
|
host_key_checking = True
|
|
retry_files_enabled = False
|
|
stdout_callback = yaml
|
|
|
|
[privilege_escalation]
|
|
become = True
|
|
become_method = sudo
|
|
become_ask_pass = False
|
|
```
|
|
|
|
The current Ansible configuration is set to log internally and not attempt to retry upon failure (in order to prevent infinite retries on certain tasks since the proper safeties have not been put in place yet). Every server gets an `ansible` user added to it which has passwordless sudo permissions. The account is locked and has a secure password. It uses an ssh key for connecting to the servers.
|
|
|
|
## Playbooks
|
|
|
|
### apt-update-test.yml
|
|
|
|
```YAML
|
|
---
|
|
- name: Update and upgrade wolfywebserver02
|
|
hosts: wolfywebserver02
|
|
become: true
|
|
gather_facts: true
|
|
|
|
tasks:
|
|
- name: Update apt cache
|
|
ansible.builtin.apt:
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
|
|
- name: Upgrade installed packages
|
|
ansible.builtin.apt:
|
|
upgrade: dist
|
|
|
|
- name: Remove unneeded packages
|
|
ansible.builtin.apt:
|
|
autoremove: true
|
|
```
|
|
|
|
This script is currently setup to connect to `wolfywebserver02` escalate to sudo then run `sudo apt update` then `sudo apt upgrade` then `sudo apt autoremove`.
|
|
|
|
### update-prod-servers.yml
|
|
|
|
```YAML
|
|
---
|
|
- name: Patch Prod Ubuntu servers and report to Discord
|
|
hosts: prod_ubuntu_servers
|
|
become: true
|
|
gather_facts: true
|
|
|
|
vars:
|
|
discord_webhook_id: "1507643788326338630"
|
|
discord_webhook_token: "TkbDnf6naR7KxhqLDIvGJyeesq9OTglxHrLHyy3d6_1WiyI-bbzUGDeVu1yCjZN7fCdd"
|
|
|
|
tasks:
|
|
- name: Update apt cache
|
|
ansible.builtin.apt:
|
|
update_cache: true
|
|
cache_valid_time: 3600
|
|
register: apt_cache_result
|
|
|
|
- name: Upgrade installed packages
|
|
ansible.builtin.apt:
|
|
upgrade: dist
|
|
register: apt_upgrade_result
|
|
|
|
- name: Remove unneeded packages
|
|
ansible.builtin.apt:
|
|
autoremove: true
|
|
register: apt_autoremove_result
|
|
|
|
- name: Check if reboot is required
|
|
ansible.builtin.stat:
|
|
path: /var/run/reboot-required
|
|
register: reboot_required_file
|
|
|
|
- name: Reboot server if required
|
|
ansible.builtin.reboot:
|
|
pre_reboot_delay: 60
|
|
post_reboot_delay: 60
|
|
reboot_timeout: 600
|
|
msg: "Rebooting in 60 seconds (intiated by Ansible)"
|
|
when: reboot_required_file.stat.exists
|
|
register: reboot_result
|
|
|
|
- name: Build per-host summary
|
|
ansible.builtin.set_fact:
|
|
patch_summary: |
|
|
Host: {{ inventory_hostname }}
|
|
Apt Cache Updated: {{ 'Yes' if (apt_cache_result.changed | default(false)) else 'No' }}
|
|
Packages Upgraded: {{ 'Yes' if (apt_upgrade_result.changed | default(false)) else 'No' }}
|
|
Autoremove Changed: {{ 'Yes' if (apt_autoremove_result.changed | default(false)) else 'No' }}
|
|
Reboot Required: {{ 'Yes' if (reboot_required_file.stat.exists | default(false)) else 'No' }}
|
|
Reboot Performed: {{ 'Yes' if (reboot_result.changed | default(false)) else 'No' }}
|
|
|
|
- name: Send Discord report
|
|
community.general.discord:
|
|
webhook_id: "{{ discord_webhook_id }}"
|
|
webhook_token: "{{ discord_webhook_token }}"
|
|
username: "Ansible"
|
|
avatar_url: "https://opreviews.anime-pictures.net/49f/49f1117eec83be2bd5af12772a70d2e4_bp.avif"
|
|
content: |
|
|
========================================
|
|
Test Servers Update Report
|
|
Generated: {{ lookup('pipe', 'date -u +%Y-%m-%dT%H:%M:%SZ') }}
|
|
========================================
|
|
|
|
Total Hosts: {{ ansible_play_hosts_all | length }}
|
|
|
|
{% for host in ansible_play_hosts_all | sort %}
|
|
{{ hostvars[host].patch_summary | default('Host: ' ~ host ~ '\n No summary generated\n') }}
|
|
|
|
{% endfor %}
|
|
========================================
|
|
Summary
|
|
========================================
|
|
Hosts Requiring Reboot: {{
|
|
ansible_play_hosts_all
|
|
| select('in', ansible_play_hosts_all)
|
|
| map('extract', hostvars)
|
|
| selectattr('reboot_required_file.stat.exists', 'defined')
|
|
| selectattr('reboot_required_file.stat.exists')
|
|
| list
|
|
| length
|
|
}}
|
|
|
|
Hosts Rebooted: {{
|
|
ansible_play_hosts_all
|
|
| select('in', ansible_play_hosts_all)
|
|
| map('extract', hostvars)
|
|
| selectattr('reboot_result.changed', 'defined')
|
|
| selectattr('reboot_result.changed')
|
|
| list
|
|
| length
|
|
}}
|
|
delegate_to: localhost
|
|
run_once: true
|
|
become: false
|
|
```
|
|
|
|
This playbook is designed to connect all `prod_ubuntu_servers` hosts and run update, upgrade, autoremove, and restart if necessary then send out a webhook with details on how each step went on each server.
|
|
|
|
## Run
|
|
|
|
### update-prod-servers.sh
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
LOG_DIR="/home/wolfy/log/ansible-prod"
|
|
mkdir -p "$LOG_DIR"
|
|
|
|
TIMESTAMP="$(date '+%Y-%m-%d_%H-%M-%S')"
|
|
LOG_FILE="$LOG_DIR/ubuntu-prod-update-$TIMESTAMP.log"
|
|
|
|
exec >>"$LOG_FILE" 2>&1
|
|
|
|
echo "========================================"
|
|
echo "Prod patching run started: $(date -Is)"
|
|
echo "========================================"
|
|
|
|
cd /home/wolfy/ansible
|
|
|
|
/home/wolfy/ansible-env/bin/ansible-playbook playbooks/update-prod-servers.yml
|
|
|
|
echo
|
|
echo "========================================"
|
|
echo "Prod patching run completed: $(date -Is)"
|
|
echo "========================================"
|
|
```
|
|
This wrapper script is ran by `update-prod-servers.service` a systemd service which is triggered by `update-prod-servers.timer` a systemd timer. This is designed to trigger every Monday at 7:30 EST and update all production ubuntu servers using `update-prod-servers.yml`.
|
|
|
|
#### update-prod-servers.service
|
|
```ini
|
|
[Unit]
|
|
Description=Run weekly production patching playbook
|
|
Wants=network-online.target
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
User=wolfy
|
|
Group=wolfy
|
|
ExecStart=/home/wolfy/ansible/run/update-prod-servers.sh
|
|
WorkingDirectory=/home/wolfy/ansible
|
|
```
|
|
#### update-prod-servers.timer
|
|
```ini
|
|
[Unit]
|
|
Description=Weekly production patching timer
|
|
|
|
[Timer]
|
|
OnCalendar=Mon *-*-* 07:30:00
|
|
Persistent=true
|
|
Unit=update-prod-servers.service
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|
|
```
|
|
|
|
## Inventory
|
|
|
|
### wolfywebserver02
|
|
|
|
- Hostname: wolfywebserver02
|
|
- OS: Ubuntu 24.04.4 LTS
|
|
- VM Host: wolfyvmserver
|
|
- Ansible Status: Ready
|
|
- Ansible Category: ubuntu_servers
|
|
|
|
This server is currently being used for running audiobookshelf so that my friends can access the server and listen to audiobooks on their phone or computer using an app or the webui. The friends are networked via Tailscale.
|
|
|
|
|
|
|
|
### wolfyminecraftserver
|
|
|
|
- Hostname: wolfyminecraftserver
|
|
- OS: Ubuntu 24.04.4 LTS
|
|
- VM Host: wolfyvmserver
|
|
- Ansible Status: Ready
|
|
- Ansible Category: ubuntu_servers
|
|
|
|
This server is currently being used to host MoreModsReloaded server for Rockman and his friends. It is running Crafty Controller to manage the Minecraft Server and provide an admin web UI so that Rockman and a few admins can make changes to the server if they need to. Playit service is used to tunnel a port on the server to a playit cloud server that reverse proxies outside users to my server without exposing my IP. Tailscale is used to allow Rockman and Waluu to access the Crafty Controller web UI.
|
|
|
|
### wolfydiscordbot01
|
|
|
|
- Hostname: wolfydiscordbot01
|
|
- OS: Ubuntu 24.04.4 LTS
|
|
- VM Host: wolfyvmserver
|
|
- Ansible Status: Ready
|
|
- Ansible Category: ubuntu_servers
|
|
|
|
TO DO
|
|
|
|
## Notes
|
|
|
|
### Setting up Ansible Access
|
|
|
|
#### Bootstrap Script
|
|
|
|
`.bootstrap/` contains a script that can be used to setup ansible on new servers.
|
|
|
|
#### Debian/Ubuntu
|
|
|
|
1. On the control node generate a dedicated key
|
|
|
|
```bash
|
|
ssh-keygen -t ed25519 -f ~/.ssh/ansible-ed25519 -C "ansible-control"
|
|
```
|
|
|
|
2. Create the ansible user and give it sudo permissions
|
|
|
|
```bash
|
|
# For Password
|
|
openssl rand -base64 16
|
|
|
|
# Create account and give sudo permissions
|
|
sudo adduser ansible
|
|
sudo usermod -aG sudo ansible
|
|
```
|
|
|
|
3. Allow passwordless sudo for the ansible account
|
|
|
|
```bash
|
|
sudo visudo
|
|
```
|
|
|
|
Add:
|
|
|
|
```ini
|
|
ansible ALL=(ALL) NOPASSWD: ALL
|
|
```
|
|
|
|
4. On the control node copy the ssh key over to the target server
|
|
|
|
```bash
|
|
ssh-copy-id -i ~/.ssh/ansible-ed25519.pub ansible@server.tailnet.ts.net
|
|
```
|
|
|
|
Test:
|
|
|
|
```bash
|
|
ssh ansible@server.tailnet.ts.net
|
|
```
|
|
|
|
5. Lock the ansible account to prevent login attempts
|
|
|
|
```bash
|
|
sudo passwd -l ansible
|
|
```
|
|
|
|
### To-Do
|
|
|