headscale/docker-dev/TROUBLESHOOTING.md

# Troubleshooting Guide

This guide covers common issues encountered when setting up and running the Headscale Docker development environment.

## Configuration Issues

### ❌ "headscale now requires a new `noise.private_key_path` field"

**Symptom**: Headscale container fails to start with error about missing noise private key path.

**Cause**: Newer versions of Headscale require the Noise protocol configuration for Tailscale v2.

**Solution**: Add the noise configuration to `headscale-config.yaml`:
```yaml
noise:
  private_key_path: /var/lib/headscale/noise_private.key
```

### ❌ "no IPv4 or IPv6 prefix configured"

**Symptom**: Headscale fails with error about missing IP prefixes.

**Cause**: Configuration format changed from `ip_prefixes` to `prefixes` with `v4`/`v6` subfields.

**Solution**: Update the configuration format:
```yaml
# Old format (doesn't work)
ip_prefixes:
  - 100.64.0.0/16
  - fd7a:115c:a1e0::/48

# New format (works)
prefixes:
  v4: 100.64.0.0/10
  v6: fd7a:115c:a1e0::/48
  allocation: sequential
```

### ❌ "Username has to contain @, got: \"*\""

**Symptom**: ACL policy fails to parse with username format error.

**Cause**: Newer Headscale versions require usernames in email format.

**Solution**: Use proper username format in ACL:
```json
{
  "acls": [
    {
      "action": "accept",
      "src": ["testuser@headscale"],
      "dst": ["testuser@headscale:*"]
    }
  ]
}
```

### ❌ "type *v2.Group not supported"

**Symptom**: ACL fails with unsupported group type.

**Cause**: Some ACL features might not be supported in newer versions.

**Solution**: Simplify ACL policy to use direct user references instead of groups.

## Network Conflicts

### ❌ "Pool overlaps with other one on this address space"

**Symptom**: Docker Compose fails to create network.

**Cause**: The subnet conflicts with existing Docker networks.

**Solution**: 
1. Check existing networks: `docker network ls`
2. Choose a different subnet in docker-compose.yml:
```yaml
networks:
  headscale-net:
    driver: bridge
    ipam:
      config:
        - subnet: 10.99.0.0/24  # Use available subnet
          gateway: 10.99.0.1
```

### ❌ "Bind for 0.0.0.0:8080 failed: port is already allocated"

**Symptom**: Port conflict when starting Headscale.

**Cause**: Port 8080 is already in use by another service.

**Solution**: Map to a different host port:
```yaml
ports:
  - "8180:8080"  # Use 8180 on host instead of 8080
```

## Authentication Issues

### ❌ "invalid argument \"testuser\" for \"-u, --user\" flag"

**Symptom**: Pre-auth key creation fails with user argument error.

**Cause**: Newer Headscale uses user IDs instead of usernames.

**Solution**: 
1. Get user ID: `docker exec headscale-server headscale users list`
2. Use ID in commands: `headscale preauthkeys create --user 1`

### ❌ Clients not registering automatically

**Symptom**: Tailscale clients don't register with pre-auth keys.

**Troubleshooting**:
1. Check if auth keys are set in `.env`:
   ```bash
   cat .env
   ```
2. Verify Headscale is reachable:
   ```bash
   docker exec tailscale-client1 wget -O- http://headscale:8080/health
   ```
3. Check client logs:
   ```bash
   docker logs tailscale-client1
   ```

## Health Check Issues

### ❌ "unknown command \"health\" for \"headscale\""

**Symptom**: Health check fails because command doesn't exist.

**Cause**: The `headscale health` command doesn't exist in current versions.

**Solution**: Use HTTP health check instead:
```yaml
healthcheck:
  test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health"]
```

Or remove health check dependency:
```yaml
depends_on:
  - headscale  # Simple dependency without health check
```

## Connectivity Issues

### ❌ Clients can't ping each other

**Troubleshooting**:
1. Check if nodes are registered:
   ```bash
   docker exec headscale-server headscale nodes list
   ```
2. Verify Tailscale status on clients:
   ```bash
   docker exec tailscale-client1 tailscale status
   ```
3. Check ACL policy allows communication:
   ```bash
   docker exec headscale-server headscale policy get
   ```

### ❌ "dependency failed to start: container headscale-server is unhealthy"

**Symptom**: Clients won't start because Headscale health check fails.

**Solution**: Either fix the health check or remove the health dependency:
```yaml
depends_on:
  - headscale  # Remove health condition
```

## Container Issues

### ❌ Permission denied with /dev/net/tun

**Symptom**: Tailscale clients can't create TUN device.

**Solution**: Ensure proper capabilities and device access:
```yaml
cap_add:
  - NET_ADMIN
  - SYS_MODULE
volumes:
  - /dev/net/tun:/dev/net/tun
```

### ❌ Clients keep restarting

**Troubleshooting**:
1. Check client logs for specific errors
2. Verify Headscale is accessible
3. Ensure auth keys are valid
4. Check if TUN device is available

## Debugging Commands

### Check Service Status
```bash
# View all containers
docker ps

# Check specific service logs
docker logs headscale-server
docker logs tailscale-client1

# Inspect network configuration
docker network inspect headscale-dev_headscale-net
```

### Test Network Connectivity
```bash
# Test from client to Headscale
docker exec tailscale-client1 ping headscale

# Test Headscale API
curl http://localhost:8180/health

# Check Tailscale status
docker exec tailscale-client1 tailscale status
```

### Verify Configuration
```bash
# Check Headscale users
docker exec headscale-server headscale users list

# List registered nodes
docker exec headscale-server headscale nodes list

# View pre-auth keys
docker exec headscale-server headscale preauthkeys list --user 1
```

## Complete Reset

If everything is broken, start fresh:
```bash
# Stop and remove everything
make clean

# Remove any conflicting networks manually if needed
docker network prune

# Start from scratch
make up
```

## Getting Help

1. **Check logs first**: Most issues are visible in container logs
2. **Verify network connectivity**: Ensure Docker network is working
3. **Test step by step**: Start with Headscale, then add clients
4. **Use simple ACL**: Start with basic ACL and expand later
5. **Check Headscale documentation**: https://headscale.net/