초기 구현: TailRescue Headscale ISO 프로젝트 정리

2026-06-01 19:46:05 +09:00
commit 6334bf0eb8
9 changed files with 480 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,31 @@
+# build outputs
+*.iso
+*.img
+*.qcow2
+*.raw
+*.squashfs
+*.packages
+*.files
+*.buildlog
+/rootfs/
+/chroot/
+/binary/
+/cache/
+/build/
+/dist/
+
+# secrets / local field config
+.env
+*.env
+rescue.env
+secret.env
+preauth*.txt
+*.key
+*.pem
+id_*
+!*.pub
+
+# temp
+*.log
+*.tmp
+.DS_Store
--- a/README.md
+++ b/README.md
@@ -0,0 +1,73 @@
+# TailRescue Headscale Live ISO
+
+Debian Live 기반 구조/백업용 rescue ISO 프로젝트입니다.
+
+목표:
+
+- Ventoy/iVentoy/Proxmox ISO 부팅
+- DHCP로 유선 네트워크 자동 연결
+- PharmQ Headscale에 Tailscale 자동 등록
+- 외부에서 Tailnet IP로 SSH 접속
+- Windows/산업용 PC 디스크 확인 및 NTFS read-only 마운트
+- rsync/rclone/restic/gddrescue로 안전한 백업 지원
+
+## 현재 검증 상태
+
+검증일: 2026-06-01
+
+- 빌드 호스트: `pve7`
+- 테스트 VM: `pve7` VMID `990`
+- Headscale login server: `https://head.pharmq.kr`
+- 자동등록 노드 예: `tailrescue-44a29acb-3hhi3pl0`
+- Tailnet SSH 검증: `ssh rescue@100.64.0.80` 성공
+- passwordless sudo 검증 성공
+- 디스크 인식 검증: `/dev/sda`, `/dev/sdb`
+- NTFS 도구 포함 확인: `ntfs-3g`, `ntfs-3g.probe`, `ntfsfix`
+
+## 저장소 정책
+
+Git에는 다음만 보관합니다.
+
+- live-build 설정 템플릿
+- 빌드/검증 스크립트
+- runbook/docs
+- Hermes skill
+
+Git에는 다음을 넣지 않습니다.
+
+- 완성 ISO (`*.iso`)
+- Headscale preauth key
+- password 원문
+- private SSH key
+- 빌드 chroot/cache/binary 산출물
+
+ISO는 Gitea Release attachment, 별도 artifact storage, 또는 `/root/tailrescue-dist` 같은 내부 보관소에 둡니다.
+
+## 빠른 사용
+
+```bash
+cp templates/rescue.env.example rescue.env
+# rescue.env에 현장용 preauth key/password/authorized key 설정
+./scripts/build-live-iso.sh
+./scripts/test-proxmox-vm.sh
+```
+
+## 현장 흐름
+
+1. 최신 ISO를 Ventoy USB에 복사
+2. 대상 PC에서 ISO 부팅
+3. Debian Live 메뉴에서 Enter
+4. 1~2분 대기
+5. Headscale에서 `tailrescue-*` 노드 IP 확인
+6. `ssh rescue@100.64.x.y`
+7. `rescue-status`, `list-disks`
+8. `sudo mount-ntfs-ro /dev/sdXN /mnt/windows`
+9. 백업 실행
+
+## 보안 원칙
+
+- 현장별 1회용/단기 Headscale preauth key 사용
+- password fallback은 Tailnet-only MVP용이며 현장마다 교체
+- 가능하면 public-key auth 우선
+- 원본 NTFS는 read-only 마운트 기본
+- 작업 후 ephemeral `tailrescue-*` 노드 정리
--- a/docs/runbook.md
+++ b/docs/runbook.md
@@ -0,0 +1,39 @@
+# TailRescue 운영 Runbook
+
+## 산출물 관리
+
+- Git: 소스/문서/스크립트/스킬만 저장
+- ISO: Gitea Release attachment 또는 내부 artifact path에 저장
+- 현재 PoC ISO hash: `3d7995cfdf58c62f6ee167458079a7eaa1d2a79ac56e5f019cab1ec856943ddd`
+
+## 빌드
+
+```bash
+./scripts/headscale-create-preauth.sh
+cp templates/rescue.env.example rescue.env
+cp templates/authorized_keys.example templates/authorized_keys
+# rescue.env에는 현장용 preauth key/password를 넣고, authorized_keys에는 공개키만 넣는다.
+./scripts/build-live-iso.sh
+```
+
+## 검증
+
+```bash
+cp /root/tailrescue-dist/$(cat /root/tailrescue-dist/latest.txt) /var/lib/vz/template/iso/tailrescue-headscale-test.iso
+./scripts/test-proxmox-vm.sh
+ssh rescue@100.64.x.y 'echo SSH_OK; sudo -n true; rescue-status; list-disks'
+```
+
+## 현장
+
+- Ventoy USB에 ISO 복사
+- 대상 PC에서 ISO 선택 후 Enter
+- Headscale node list에서 `tailrescue-*` 확인
+- `ssh rescue@100.64.x.y`
+- `sudo mount-ntfs-ro /dev/sdXN /mnt/windows`
+
+## 장애 대응
+
+- Headscale에 노드가 안 뜸: DHCP/NIC/firmware/케이블 확인, `ip -br a`, `journalctl -u tailrescue-firstboot`
+- SSH가 안 됨: `systemctl status ssh`, `/var/log/auth.log`, `id rescue`, `sudo passwd -S rescue`
+- 내장 NIC 미인식: Realtek RTL8153/RTL8156 또는 ASIX AX88179 USB LAN 동글 사용
--- a/scripts/build-live-iso.sh
+++ b/scripts/build-live-iso.sh
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+WORKDIR=${WORKDIR:-/root/tailrescue-live}
+OUTDIR=${OUTDIR:-/root/tailrescue-dist}
+ISO_NAME=${ISO_NAME:-tailrescue-headscale-$(date +%Y%m%d-%H%M).iso}
+DIST=${DIST:-bookworm}
+RESCUE_ENV=${RESCUE_ENV:-rescue.env}
+AUTHORIZED_KEYS=${AUTHORIZED_KEYS:-templates/authorized_keys}
+
+if [[ ! -f "$RESCUE_ENV" ]]; then
+  echo "missing $RESCUE_ENV; copy templates/rescue.env.example and fill field secrets" >&2
+  exit 2
+fi
+
+install -d "$WORKDIR" "$OUTDIR"
+rm -rf "$WORKDIR"
+mkdir -p "$WORKDIR"
+cd "$WORKDIR"
+
+lb config \
+  --distribution "$DIST" \
+  --archive-areas "main contrib non-free non-free-firmware" \
+  --binary-images iso-hybrid \
+  --bootappend-live "boot=live components hostname=tailrescue username=rescue"
+
+cat > config/package-lists/tailrescue.list.chroot <<"PKGS"
+systemd-sysv
+openssh-server
+sudo
+curl
+ca-certificates
+gnupg
+iptables
+nftables
+iproute2
+iputils-ping
+dnsutils
+net-tools
+isc-dhcp-client
+pciutils
+usbutils
+lshw
+parted
+gdisk
+fdisk
+ntfs-3g
+rsync
+rclone
+restic
+smartmontools
+gddrescue
+pv
+jq
+tmux
+vim-tiny
+less
+firmware-linux
+firmware-linux-free
+firmware-linux-nonfree
+firmware-misc-nonfree
+firmware-realtek
+firmware-atheros
+firmware-brcm80211
+firmware-bnx2
+firmware-bnx2x
+firmware-iwlwifi
+firmware-libertas
+PKGS
+
+mkdir -p config/includes.chroot/etc/apt/keyrings config/includes.chroot/etc/apt/sources.list.d
+curl -fsSL https://pkgs.tailscale.com/stable/debian/${DIST}.noarmor.gpg \
+  -o config/includes.chroot/etc/apt/keyrings/tailscale-archive-keyring.gpg
+cat > config/includes.chroot/etc/apt/sources.list.d/tailscale.list <<TAILSRC
+deb [signed-by=/etc/apt/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/debian ${DIST} main
+TAILSRC
+mkdir -p config/packages.chroot
+(cd config/packages.chroot && apt-get download tailscale)
+
+mkdir -p config/includes.chroot/etc/tailrescue
+cp "$OLDPWD/$RESCUE_ENV" config/includes.chroot/etc/tailrescue/rescue.env
+chmod 600 config/includes.chroot/etc/tailrescue/rescue.env
+
+mkdir -p config/includes.chroot/usr/local/bin config/includes.chroot/etc/systemd/system
+cat > config/includes.chroot/usr/local/bin/rescue-status <<"RS"
+#!/bin/bash
+set -euo pipefail
+echo "== TailRescue Status =="
+echo "Hostname: $(hostname)"
+echo "Date: $(date -Is)"
+echo "LAN IPs:"
+ip -br addr || true
+echo
+echo "Routes:"
+ip route || true
+echo
+echo "Tailscale:"
+tailscale status || true
+echo
+echo "Disks:"
+lsblk -o NAME,SIZE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL || true
+echo
+echo "SSH: ssh rescue@<tailscale-ip>"
+RS
+chmod +x config/includes.chroot/usr/local/bin/rescue-status
+
+cat > config/includes.chroot/usr/local/bin/list-disks <<"LD"
+#!/bin/bash
+exec lsblk -o NAME,TYPE,SIZE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL,SERIAL "$@"
+LD
+chmod +x config/includes.chroot/usr/local/bin/list-disks
+
+cat > config/includes.chroot/usr/local/bin/mount-ntfs-ro <<"MN"
+#!/bin/bash
+set -euo pipefail
+if [ "$#" -ne 2 ]; then echo "usage: mount-ntfs-ro /dev/sdXN /mnt/windows" >&2; exit 2; fi
+mkdir -p "$2"
+exec ntfs-3g -o ro,show_sys_files,streams_interface=windows "$1" "$2"
+MN
+chmod +x config/includes.chroot/usr/local/bin/mount-ntfs-ro
+
+cat > config/includes.chroot/usr/local/bin/tailrescue-firstboot <<"FB"
+#!/bin/bash
+set -u
+exec > >(tee -a /var/log/tailrescue-firstboot.log) 2>&1
+set -x
+ENV_FILE=/etc/tailrescue/rescue.env
+[ -f "$ENV_FILE" ] && . "$ENV_FILE"
+RESCUE_USER=${RESCUE_USER:-rescue}
+TAILSCALE_LOGIN_SERVER=${TAILSCALE_LOGIN_SERVER:-https://head.pharmq.kr}
+TAILSCALE_TAGS=${TAILSCALE_TAGS:-tag:rescue}
+mkdir -p /run/sshd
+systemctl enable --now ssh || systemctl enable --now sshd || service ssh restart || /usr/sbin/sshd || true
+systemctl enable --now tailscaled || service tailscaled start || true
+sleep 3
+if [ -n "${TAILSCALE_AUTHKEY:-}" ]; then
+  tailscale up --login-server "$TAILSCALE_LOGIN_SERVER" --authkey "$TAILSCALE_AUTHKEY" --hostname "tailrescue-$(cat /etc/machine-id | cut -c1-8)" --advertise-tags "$TAILSCALE_TAGS" || true
+fi
+rescue-status || true
+FB
+chmod +x config/includes.chroot/usr/local/bin/tailrescue-firstboot
+
+cat > config/includes.chroot/etc/systemd/system/tailrescue-firstboot.service <<"SVC"
+[Unit]
+Description=TailRescue first boot SSH and Headscale enrollment
+After=network-online.target tailscaled.service ssh.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/tailrescue-firstboot
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+SVC
+ln -sf /etc/systemd/system/tailrescue-firstboot.service config/includes.chroot/etc/systemd/system/multi-user.target.wants/tailrescue-firstboot.service 2>/dev/null || true
+
+mkdir -p config/hooks/normal
+cat > config/hooks/normal/0900-tailrescue-users.hook.chroot <<"HOOK"
+#!/bin/bash
+set -eux
+. /etc/tailrescue/rescue.env || true
+USER_NAME=${RESCUE_USER:-rescue}
+PASS_PLAIN=${RESCUE_PASSWORD:-}
+if ! id "$USER_NAME" >/dev/null 2>&1; then useradd -m -s /bin/bash "$USER_NAME"; fi
+if [ -n "$PASS_PLAIN" ]; then echo "$USER_NAME:$PASS_PLAIN" | chpasswd; fi
+usermod -aG sudo "$USER_NAME"
+echo "$USER_NAME ALL=(ALL) NOPASSWD:ALL" >/etc/sudoers.d/90-rescue
+chmod 0440 /etc/sudoers.d/90-rescue
+mkdir -p /home/$USER_NAME/.ssh /root/.ssh
+if [ -f /etc/tailrescue/authorized_keys ]; then
+  cp /etc/tailrescue/authorized_keys /home/$USER_NAME/.ssh/authorized_keys
+  cp /etc/tailrescue/authorized_keys /root/.ssh/authorized_keys
+fi
+chown -R $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh
+chmod 700 /home/$USER_NAME/.ssh /root/.ssh
+chmod 600 /home/$USER_NAME/.ssh/authorized_keys /root/.ssh/authorized_keys 2>/dev/null || true
+sed -i "s/^#\?PasswordAuthentication .*/PasswordAuthentication yes/" /etc/ssh/sshd_config || true
+sed -i "s/^#\?PermitRootLogin .*/PermitRootLogin prohibit-password/" /etc/ssh/sshd_config || true
+sed -i "s/^#\?PubkeyAuthentication .*/PubkeyAuthentication yes/" /etc/ssh/sshd_config || true
+grep -q "^AllowUsers " /etc/ssh/sshd_config && sed -i "s/^AllowUsers .*/AllowUsers $USER_NAME root/" /etc/ssh/sshd_config || echo "AllowUsers $USER_NAME root" >> /etc/ssh/sshd_config
+HOOK
+chmod +x config/hooks/normal/0900-tailrescue-users.hook.chroot
+
+if [[ -f "$OLDPWD/$AUTHORIZED_KEYS" ]]; then
+  cp "$OLDPWD/$AUTHORIZED_KEYS" config/includes.chroot/etc/tailrescue/authorized_keys
+elif [[ -f "$OLDPWD/templates/authorized_keys.example" ]]; then
+  cp "$OLDPWD/templates/authorized_keys.example" config/includes.chroot/etc/tailrescue/authorized_keys
+fi
+
+lb build
+cp -f live-image-amd64.hybrid.iso "$OUTDIR/$ISO_NAME"
+(cd "$OUTDIR" && sha256sum "$ISO_NAME" > SHA256SUMS && echo "$ISO_NAME" > latest.txt)
+echo "$OUTDIR/$ISO_NAME"
--- a/scripts/headscale-create-preauth.sh
+++ b/scripts/headscale-create-preauth.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+HEADSCALE_HOST=${HEADSCALE_HOST:-root@192.168.0.100}
+HEADSCALE_CONTAINER=${HEADSCALE_CONTAINER:-headscale}
+USER_ID=${USER_ID:-1}
+EXPIRATION=${EXPIRATION:-24h}
+TAGS=${TAGS:-tag:rescue}
+ssh "$HEADSCALE_HOST" "docker exec $HEADSCALE_CONTAINER headscale preauthkeys create -u $USER_ID --reusable --ephemeral --expiration $EXPIRATION --tags $TAGS -o json"
--- a/scripts/test-proxmox-vm.sh
+++ b/scripts/test-proxmox-vm.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+VMID=${VMID:-990}
+ISO=${ISO:-local:iso/tailrescue-headscale-test.iso}
+STORAGE=${STORAGE:-local-lvm}
+BRIDGE=${BRIDGE:-vmbr0}
+HEADSCALE_HOST=${HEADSCALE_HOST:-root@192.168.0.100}
+HEADSCALE_CONTAINER=${HEADSCALE_CONTAINER:-headscale}
+
+qm stop "$VMID" --skiplock 1 || true
+qm destroy "$VMID" --purge 1 --destroy-unreferenced-disks 1 || true
+qm create "$VMID" \
+  --name tailrescue-test \
+  --memory 2048 \
+  --cores 2 \
+  --net0 virtio,bridge="$BRIDGE" \
+  --ostype l26 \
+  --scsihw virtio-scsi-single \
+  --vga std \
+  --boot order=ide2 \
+  --ide2 "$ISO",media=cdrom \
+  --agent enabled=0
+qm set "$VMID" --scsi0 "$STORAGE":8
+qm start "$VMID"
+sleep 3
+qm sendkey "$VMID" ret || true
+sleep 75
+ssh "$HEADSCALE_HOST" "docker exec $HEADSCALE_CONTAINER headscale nodes list | grep -i tailrescue || true"
--- a/skills/devops/tailrescue-headscale-live-iso/SKILL.md
+++ b/skills/devops/tailrescue-headscale-live-iso/SKILL.md
@@ -0,0 +1,98 @@
+---
+name: tailrescue-headscale-live-iso
+description: Use when building, testing, or operating a Ventoy/iVentoy Debian Live rescue ISO that auto-enrolls into PharmQ Headscale/Tailscale for remote SSH and Windows/NTFS backup.
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [devops, live-iso, headscale, tailscale, ventoy, rescue, ntfs]
+    related_skills: [gitea-pat-askpass-push, nested-pve-auto-install-lab]
+---
+
+# TailRescue Headscale Live ISO
+
+## Overview
+
+This project builds a Debian Live rescue ISO for field PCs. It should boot from Ventoy/iVentoy/Proxmox, get DHCP on common Ethernet NICs, enroll into `https://head.pharmq.kr`, start OpenSSH, and expose disk/NTFS read-only backup helpers.
+
+## Current Verified Baseline
+
+- Build host: `pve7`
+- Test VM: `pve7` VMID `990`
+- Verified SSH over Tailnet: `rescue@100.64.0.80`
+- Verified commands: `rescue-status`, `list-disks`
+- Verified packages: `tailscale`, `openssh-server`, `ntfs-3g`, firmware packages
+- Verified ISO hash: `3d7995cfdf58c62f6ee167458079a7eaa1d2a79ac56e5f019cab1ec856943ddd`
+
+## Repository Policy
+
+Commit scripts, docs, templates, and this skill. Do not commit ISO files, `rescue.env`, preauth keys, passwords, private keys, live-build `chroot/`, `binary/`, `cache/`, or other build artifacts.
+
+## Build Flow
+
+1. Create a short-lived Headscale preauth key:
+
+```bash
+./scripts/headscale-create-preauth.sh
+```
+
+2. Create local secrets:
+
+```bash
+cp templates/rescue.env.example rescue.env
+cp templates/authorized_keys.example templates/authorized_keys
+```
+
+3. Fill `rescue.env` and `templates/authorized_keys` without committing them.
+4. Build:
+
+```bash
+./scripts/build-live-iso.sh
+```
+
+## Proxmox Test Flow
+
+```bash
+cp /root/tailrescue-dist/$(cat /root/tailrescue-dist/latest.txt) /var/lib/vz/template/iso/tailrescue-headscale-test.iso
+./scripts/test-proxmox-vm.sh
+ssh rescue@100.64.x.y 'echo SSH_OK; sudo -n true; rescue-status; list-disks'
+```
+
+## Field Flow
+
+1. Copy ISO to Ventoy USB or iVentoy ISO folder.
+2. Boot target PC and select the ISO.
+3. Press Enter at Debian Live menu if needed.
+4. Wait 1-2 minutes.
+5. Find `tailrescue-*` in Headscale.
+6. SSH to `rescue@100.64.x.y`.
+7. Run `list-disks`.
+8. Mount Windows partition read-only:
+
+```bash
+sudo mount-ntfs-ro /dev/sdXN /mnt/windows
+```
+
+## Ethernet Coverage
+
+The ISO includes Debian 12 kernel and broad firmware: `firmware-linux`, `firmware-linux-nonfree`, `firmware-misc-nonfree`, `firmware-realtek`, `firmware-atheros`, `firmware-brcm80211`, `firmware-bnx2`, `firmware-bnx2x`, `firmware-iwlwifi`, `firmware-libertas`. Carry USB Ethernet dongles such as Realtek RTL8153/RTL8156 or ASIX AX88179 for field fallback.
+
+## Common Pitfalls
+
+1. SSH may connect but auth can fail if the rescue user is only created in firstboot. Keep the chroot hook that creates `rescue`, sudoers, sshd config, and authorized keys at build time.
+2. Headscale duplicate ephemeral nodes can appear because live ISOs reuse machine IDs. Use the newest online Tailnet IP.
+3. Never write to real rescue disks by default. Mount NTFS read-only.
+4. Separate Ventoy/iVentoy boot problems from Linux NIC/firmware problems.
+5. Rotate preauth keys and fallback passwords per field build.
+
+## Verification Checklist
+
+- [ ] ISO boots in Proxmox VM.
+- [ ] Headscale node appears as `tailrescue-*`.
+- [ ] `ssh rescue@100.64.x.y` works with public key.
+- [ ] `sudo -n true` works.
+- [ ] `rescue-status` shows LAN and `tailscale0` IPs.
+- [ ] `list-disks` shows internal disks.
+- [ ] `ntfs-3g`, `ntfs-3g.probe`, and `ntfsfix` are present.
+- [ ] No secrets or ISO files are staged in Git.
--- a/templates/authorized_keys.example
+++ b/templates/authorized_keys.example
@@ -0,0 +1 @@
+# Put SSH public keys here, one per line. Do not put private keys here.
--- a/templates/rescue.env.example
+++ b/templates/rescue.env.example
@@ -0,0 +1,7 @@
+# Copy to rescue.env. Do not commit rescue.env.
+TAILSCALE_LOGIN_SERVER=https://head.pharmq.kr
+TAILSCALE_AUTHKEY=PASTE_FIELD_PREAUTH_KEY_HERE
+TAILSCALE_TAGS=tag:rescue
+RESCUE_USER=rescue
+RESCUE_PASSWORD=CHANGE_PER_FIELD_BUILD
+# Optional: paste one or more public keys into templates/authorized_keys before build.
				`@@ -0,0 +1 @@`
				`# Put SSH public keys here, one per line. Do not put private keys here.`