headscale/docker-dev/TROUBLESHOOTING.md

Troubleshooting Guide

This guide covers common issues encountered when setting up and running the Headscale Docker development environment.

Configuration Issues

❌ "headscale now requires a new noise.private_key_path field"

Symptom: Headscale container fails to start with error about missing noise private key path.

Cause: Newer versions of Headscale require the Noise protocol configuration for Tailscale v2.

Solution: Add the noise configuration to headscale-config.yaml:

noise:
  private_key_path: /var/lib/headscale/noise_private.key

❌ "no IPv4 or IPv6 prefix configured"

Symptom: Headscale fails with error about missing IP prefixes.

Cause: Configuration format changed from ip_prefixes to prefixes with v4/v6 subfields.

Solution: Update the configuration format:

# Old format (doesn't work)
ip_prefixes:
  - 100.64.0.0/16
  - fd7a:115c:a1e0::/48

# New format (works)
prefixes:
  v4: 100.64.0.0/10
  v6: fd7a:115c:a1e0::/48
  allocation: sequential

❌ "Username has to contain @, got: "*""

Symptom: ACL policy fails to parse with username format error.

Cause: Newer Headscale versions require usernames in email format.

Solution: Use proper username format in ACL:

{
  "acls": [
    {
      "action": "accept",
      "src": ["testuser@headscale"],
      "dst": ["testuser@headscale:*"]
    }
  ]
}

❌ "type *v2.Group not supported"

Symptom: ACL fails with unsupported group type.

Cause: Some ACL features might not be supported in newer versions.

Solution: Simplify ACL policy to use direct user references instead of groups.

Network Conflicts

❌ "Pool overlaps with other one on this address space"

Symptom: Docker Compose fails to create network.

Cause: The subnet conflicts with existing Docker networks.

Solution:

1. Check existing networks: docker network ls
2. Choose a different subnet in docker-compose.yml:

networks:
  headscale-net:
    driver: bridge
    ipam:
      config:
        - subnet: 10.99.0.0/24  # Use available subnet
          gateway: 10.99.0.1

❌ "Bind for 0.0.0.0:8080 failed: port is already allocated"

Symptom: Port conflict when starting Headscale.

Cause: Port 8080 is already in use by another service.

Solution: Map to a different host port:

ports:
  - "8180:8080"  # Use 8180 on host instead of 8080

Authentication Issues

❌ "invalid argument "testuser" for "-u, --user" flag"

Symptom: Pre-auth key creation fails with user argument error.

Cause: Newer Headscale uses user IDs instead of usernames.

Solution:

1. Get user ID: docker exec headscale-server headscale users list
2. Use ID in commands: headscale preauthkeys create --user 1

❌ Clients not registering automatically

Symptom: Tailscale clients don't register with pre-auth keys.

Troubleshooting:

1. Check if auth keys are set in .env:

   cat .env
   

2. Verify Headscale is reachable:

   docker exec tailscale-client1 wget -O- http://headscale:8080/health
   

3. Check client logs:

   docker logs tailscale-client1
   

Health Check Issues

❌ "unknown command "health" for "headscale""

Symptom: Health check fails because command doesn't exist.

Cause: The headscale health command doesn't exist in current versions.

Solution: Use HTTP health check instead:

healthcheck:
  test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health"]

Or remove health check dependency:

depends_on:
  - headscale  # Simple dependency without health check

Connectivity Issues

❌ Clients can't ping each other

Troubleshooting:

1. Check if nodes are registered:

   docker exec headscale-server headscale nodes list
   

2. Verify Tailscale status on clients:

   docker exec tailscale-client1 tailscale status
   

3. Check ACL policy allows communication:

   docker exec headscale-server headscale policy get
   

❌ "dependency failed to start: container headscale-server is unhealthy"

Symptom: Clients won't start because Headscale health check fails.

Solution: Either fix the health check or remove the health dependency:

depends_on:
  - headscale  # Remove health condition

Container Issues

❌ Permission denied with /dev/net/tun

Symptom: Tailscale clients can't create TUN device.

Solution: Ensure proper capabilities and device access:

cap_add:
  - NET_ADMIN
  - SYS_MODULE
volumes:
  - /dev/net/tun:/dev/net/tun

❌ Clients keep restarting

Troubleshooting:

1. Check client logs for specific errors
2. Verify Headscale is accessible
3. Ensure auth keys are valid
4. Check if TUN device is available

Debugging Commands

Check Service Status

# View all containers
docker ps

# Check specific service logs
docker logs headscale-server
docker logs tailscale-client1

# Inspect network configuration
docker network inspect headscale-dev_headscale-net

Test Network Connectivity

# Test from client to Headscale
docker exec tailscale-client1 ping headscale

# Test Headscale API
curl http://localhost:8180/health

# Check Tailscale status
docker exec tailscale-client1 tailscale status

Verify Configuration

# Check Headscale users
docker exec headscale-server headscale users list

# List registered nodes
docker exec headscale-server headscale nodes list

# View pre-auth keys
docker exec headscale-server headscale preauthkeys list --user 1

Complete Reset

If everything is broken, start fresh:

# Stop and remove everything
make clean

# Remove any conflicting networks manually if needed
docker network prune

# Start from scratch
make up

Getting Help

1. Check logs first: Most issues are visible in container logs
2. Verify network connectivity: Ensure Docker network is working
3. Test step by step: Start with Headscale, then add clients
4. Use simple ACL: Start with basic ACL and expand later
5. Check Headscale documentation: https://headscale.net/