headscale/docker-dev/TESTING-CHECKLIST.md

# Testing Verification Checklist

This checklist ensures the Headscale Docker environment is working correctly. Follow these steps to verify your setup.

## ✅ Pre-Setup Verification

### System Requirements
- [ ] Docker installed and running
- [ ] Docker Compose installed
- [ ] Ports 8180, 9090, 50443 available on host
- [ ] At least 2GB free disk space
- [ ] Network subnet 10.99.0.0/24 not in use

### Port Conflicts Check
```bash
# Check if required ports are free
sudo lsof -i :8180 :9090 :50443
# Should return no results if ports are free
```

### Network Conflicts Check
```bash
# Check for existing Docker networks using similar subnets
docker network ls --format '{{.Name}}' | xargs -I {} sh -c 'echo "Network: {}"; docker network inspect {} 2>/dev/null | jq -r ".[0].IPAM.Config[0].Subnet // \"No subnet\""; echo' | grep -A1 "10.99"
# Should return no results
```

## ✅ Initial Setup Verification

### 1. Environment Startup
```bash
cd /path/to/headscale/docker-dev
make up
```

**Expected results:**
- [ ] All containers start without errors
- [ ] Headscale container shows "listening and serving" messages
- [ ] No port binding errors
- [ ] User 'testuser' created successfully
- [ ] Pre-auth keys generated and saved to .env

### 2. Container Status Check
```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

**Expected results:**
- [ ] `headscale-server` status: `Up X seconds`
- [ ] `tailscale-client1` status: `Up X seconds`  
- [ ] `tailscale-client2` status: `Up X seconds`
- [ ] `test-webserver` status: `Up X seconds`
- [ ] Port mappings visible: `0.0.0.0:8180->8080/tcp` etc.

### 3. Network Creation Check
```bash
docker network inspect headscale-dev_headscale-net
```

**Expected results:**
- [ ] Network exists with subnet `10.99.0.0/24`
- [ ] Gateway at `10.99.0.1`
- [ ] All 4 containers attached to network
- [ ] Each container has assigned IP in correct range

## ✅ Headscale Server Verification

### 1. Health Check
```bash
curl -s http://localhost:8180/health
```

**Expected result:**
- [ ] Returns: `{"status":"pass"}`

### 2. User Management
```bash
docker exec headscale-server headscale users list
```

**Expected results:**
- [ ] Shows user with ID 1
- [ ] Username: `testuser`
- [ ] Created timestamp present

### 3. Pre-auth Keys
```bash
docker exec headscale-server headscale preauthkeys list --user 1
```

**Expected results:**
- [ ] Shows 2 pre-auth keys
- [ ] Both keys marked as `reusable: true`
- [ ] Expiration set to 24h from creation
- [ ] Keys not yet used

### 4. Configuration Validation
```bash
docker exec headscale-server headscale configtest
```

**Expected result:**
- [ ] Configuration validates successfully (if command exists)
- [ ] Or server starts without configuration errors

## ✅ Tailscale Client Verification

### 1. Client Registration
```bash
docker exec headscale-server headscale nodes list
```

**Expected results:**
- [ ] Shows 2 nodes (client1, client2)
- [ ] Both nodes have status `online`
- [ ] IP addresses: `100.64.0.1` and `100.64.0.2`
- [ ] IPv6 addresses: `fd7a:115c:a1e0::1` and `fd7a:115c:a1e0::2`
- [ ] Both nodes associated with `testuser`
- [ ] No expired nodes

### 2. Client Status
```bash
docker exec tailscale-client1 tailscale status
docker exec tailscale-client2 tailscale status
```

**Expected results:**
- [ ] Client1 shows itself at `100.64.0.1`
- [ ] Client1 shows client2 at `100.64.0.2`
- [ ] Client2 shows itself at `100.64.0.2`
- [ ] Client2 shows client1 at `100.64.0.1`
- [ ] Both show status as logged in to `testuser`

### 3. Authentication Verification
```bash
cat .env | grep TS_AUTHKEY
```

**Expected results:**
- [ ] Two auth keys present: `TS_AUTHKEY_CLIENT1` and `TS_AUTHKEY_CLIENT2`
- [ ] Keys are non-empty 32-character hex strings
- [ ] Keys are different from each other

## ✅ Network Connectivity Testing

### 1. Docker Network Connectivity
```bash
# Test basic Docker networking
docker exec tailscale-client1 ping -c 3 headscale
docker exec tailscale-client1 ping -c 3 10.99.0.22
docker exec tailscale-client2 ping -c 3 10.99.0.21
```

**Expected results:**
- [ ] All pings successful with 0% packet loss
- [ ] Round trip times < 10ms (local network)
- [ ] DNS resolution working (headscale resolves to 10.99.0.10)

### 2. Tailscale Network Connectivity
```bash
# Test Tailscale VPN connectivity
docker exec tailscale-client1 tailscale ping client2
docker exec tailscale-client2 tailscale ping client1
```

**Expected results:**
- [ ] Both pings return successful "pong" messages
- [ ] Response shows Tailscale IP (100.64.0.x)
- [ ] Response shows underlying transport (via 10.99.0.x:port)
- [ ] Response time < 1s

### 3. IP-level Connectivity
```bash
# Test direct IP ping through Tailscale
docker exec tailscale-client1 ping -c 3 100.64.0.2
docker exec tailscale-client2 ping -c 3 100.64.0.1
```

**Expected results:**
- [ ] Pings successful through Tailscale tunnel
- [ ] 0% packet loss
- [ ] Consistent round trip times

## ✅ Application Layer Testing

### 1. Web Server Connectivity
```bash
# Test HTTP connectivity through Tailscale
docker exec tailscale-client1 curl -s http://webserver | grep -i "hello"
```

**Expected results:**
- [ ] Successfully retrieves web page
- [ ] HTML content contains expected text
- [ ] No connection errors

### 2. Make Target Testing
```bash
# Test automation commands
make status
make ping-test
```

**Expected results:**
- [ ] `make status` shows all nodes online
- [ ] `make ping-test` reports successful connectivity
- [ ] No error messages in output

## ✅ Security Verification

### 1. ACL Policy Check
```bash
docker exec headscale-server headscale policy get
```

**Expected results:**
- [ ] Policy loaded successfully
- [ ] Shows rule allowing testuser@headscale to communicate
- [ ] No policy parsing errors

### 2. Encryption Verification
```bash
# Check that traffic is encrypted (this is implicit in Tailscale)
docker exec tailscale-client1 tailscale status --json | jq '.Peer[] | {Name: .HostName, Online: .Online, LastSeen: .LastSeen}'
```

**Expected results:**
- [ ] Peers show as online
- [ ] Recent LastSeen timestamps
- [ ] Secure connections established

### 3. Noise Protocol Verification
```bash
docker exec headscale-server ls -la /var/lib/headscale/noise_private.key
```

**Expected results:**
- [ ] Noise private key file exists
- [ ] File has appropriate permissions
- [ ] Non-zero file size

## ✅ Performance Testing

### 1. Latency Test
```bash
# Test latency through Tailscale
docker exec tailscale-client1 sh -c 'for i in {1..10}; do tailscale ping client2; done'
```

**Expected results:**
- [ ] All pings successful
- [ ] Consistent low latency (< 1s for local setup)
- [ ] No timeout errors

### 2. Throughput Test (Optional)
```bash
# Basic throughput test using nc (if available)
docker exec tailscale-client2 nc -l 8888 > /dev/null &
docker exec tailscale-client1 sh -c 'yes | head -c 1M | nc 100.64.0.2 8888'
```

**Expected results:**
- [ ] Data transfer completes successfully
- [ ] No connection refused errors

## ✅ Log Analysis

### 1. Check for Errors
```bash
# Check all container logs for errors
docker logs headscale-server 2>&1 | grep -i error
docker logs tailscale-client1 2>&1 | grep -i error  
docker logs tailscale-client2 2>&1 | grep -i error
```

**Expected results:**
- [ ] No critical errors in Headscale logs
- [ ] No authentication failures
- [ ] No network connectivity errors
- [ ] Warning messages acceptable (non-blocking)

### 2. Successful Operations
```bash
# Look for success indicators
docker logs headscale-server 2>&1 | grep "listening and serving"
```

**Expected results:**
- [ ] Headscale shows "listening and serving" for all ports
- [ ] No startup failures
- [ ] Database operations successful

## ✅ Cleanup Verification

### 1. Controlled Shutdown
```bash
make down
```

**Expected results:**
- [ ] All containers stop gracefully
- [ ] No force-kill required
- [ ] Networks removed cleanly

### 2. Complete Cleanup
```bash
make clean
```

**Expected results:**
- [ ] All containers removed
- [ ] All volumes removed  
- [ ] Networks removed
- [ ] .env file cleaned up

## 🔧 Troubleshooting Failed Checks

If any checks fail, refer to:
- **TROUBLESHOOTING.md** - Common issues and solutions
- **Container logs** - `docker logs <container-name>`
- **Network inspection** - `docker network inspect <network-name>`
- **Headscale CLI** - `docker exec headscale-server headscale --help`

## 📊 Test Results Summary

Create a test report with:
- [ ] Test execution date/time
- [ ] All checklist items marked as pass/fail
- [ ] Any failures documented with error messages
- [ ] Environment details (Docker version, OS, etc.)
- [ ] Performance measurements if collected

---

**✅ All checks passed? Congratulations! Your Headscale environment is fully functional.**