From 5248b5c0934bc6321f49ecb4adca0b3561de405e Mon Sep 17 00:00:00 2001 From: Ryan Malloy Date: Wed, 18 Feb 2026 10:19:20 -0700 Subject: [PATCH] Expand README with full configuration reference and usage docs Added: Why section, env vars table, merge strategy explanation, NORAD encoding table, regime classification, library usage examples, SQL output format, and development instructions. --- README.md | 183 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 153 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 10c8607..23d1b81 100644 --- a/README.md +++ b/README.md @@ -2,42 +2,89 @@ TLE catalog builder for [pg_orrery](https://pg-orrery.warehack.ing) — download, merge, and load satellite catalogs from Space-Track, CelesTrak, and SatNOGS into PostgreSQL. +## Why + +pg_orrery needs TLEs in a table. The satellite catalog has 30,000+ tracked objects spread across multiple data sources, each with different freshness. Getting them all merged and loaded into PostgreSQL is tedious: download from each source, handle different auth methods, deduplicate by epoch, format as SQL, pipe to psql. + +`pg-orrery-catalog` handles the whole pipeline. + ## Install ```bash -uv pip install pg-orrery-catalog -# or +# Run directly (no install needed) uvx pg-orrery-catalog --help -``` -For direct database loading: +# Or install +uv pip install pg-orrery-catalog -```bash +# For direct database loading (adds psycopg) uv pip install "pg-orrery-catalog[pg]" ``` -## Quick Start +Requires Python 3.11+. + +## Usage + +### Download TLE data ```bash -# Download TLE data from all sources -pg-orrery-catalog download +pg-orrery-catalog download # All configured sources +pg-orrery-catalog download --source celestrak # CelesTrak only (no auth) +pg-orrery-catalog download --source spacetrack # Space-Track (needs credentials) +pg-orrery-catalog download --force # Re-download even if cached +``` -# Build SQL and pipe to psql -pg-orrery-catalog build | psql -d mydb +Files are cached in `~/.cache/pg-orrery-catalog/` and reused for 24 hours. -# Or load directly (requires [pg] extra) -pg-orrery-catalog load --database-url postgresql:///mydb --create-index +### Build a merged catalog -# Export as 3LE file -pg-orrery-catalog build --format 3le -o merged.tle +```bash +pg-orrery-catalog build | psql -d mydb # SQL to stdout +pg-orrery-catalog build --table satellites -o catalog.sql # Save SQL file +pg-orrery-catalog build --format 3le -o merged.tle # 3LE text output +pg-orrery-catalog build --format json -o catalog.json # JSON with metadata +``` -# Check cache status +When the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means CelesTrak SupGP data (operator-provided, fresher epochs) automatically overrides stale Space-Track entries. + +### Build from specific files + +```bash +pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle +``` + +### Load directly into PostgreSQL + +```bash +pg-orrery-catalog load \ + --database-url postgresql:///mydb \ + --table satellites \ + --create-index +``` + +### Check status + +```bash pg-orrery-catalog info --cache ``` ## Configuration -Create `~/.config/pg-orrery-catalog/config.toml`: +Three layers (highest precedence first): CLI flags > environment variables > config file. + +### Environment variables + +| Variable | Purpose | +|----------|---------| +| `SPACETRACK_USER` | Space-Track login email | +| `SPACETRACK_PASSWORD` | Space-Track password | +| `SOCKS_PROXY` | SOCKS5 proxy for CelesTrak/SatNOGS (e.g. `localhost:1080`) | +| `DATABASE_URL` | PostgreSQL connection URL for `load` command | +| `PG_ORRERY_TABLE` | Default table name | + +### Config file + +`~/.config/pg-orrery-catalog/config.toml`: ```toml [spacetrack] @@ -55,25 +102,101 @@ table = "satellites" url = "postgresql://localhost/mydb" ``` -Environment variables (`SPACETRACK_USER`, `SPACETRACK_PASSWORD`, `SOCKS_PROXY`, `DATABASE_URL`) override config file values. - ## Sources -| Source | Auth | Coverage | -|--------|------|----------| -| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | -| [CelesTrak](https://celestrak.org) | None | Active sats + operator SupGP | -| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | +| Source | Auth | Coverage | Freshness | +|--------|------|----------|-----------| +| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days | +| [CelesTrak](https://celestrak.org) | None | Active sats + operator SupGP | Minutes to hours | +| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies | -When the same NORAD ID appears in multiple sources, the entry with the newest epoch wins — CelesTrak SupGP data (operator-provided, fresher epochs) overrides stale Space-Track entries. +CelesTrak's supplemental GP groups are configured via `supgp_groups`. These contain operator-submitted ephemerides (SpaceX for Starlink, OneWeb, Planet, Orbcomm) that are typically hours fresher than Space-Track's catalog. -## NORAD ID Decoding +## Merge strategy -Handles all four encoding schemes from Bill Gray's sat_code: -- **Traditional**: 5-digit (00001–99999) -- **Alpha-5**: letter + 4 digits (A0000–Z9999 → 100000–339999) -- **Super-5 case 3**: base-64 with uppercase last char (340000–906309663) -- **Super-5 case 4**: base-64 with non-digit 4th char (906309664+) +Sources are processed in order: Space-Track (base catalog), CelesTrak active, SatNOGS, then SupGP groups (freshest last). For each NORAD ID, the entry with the newest TLE epoch wins. + +Example merge output: + +``` +spacetrack_everything: 33053 objects (33053 new, 0 updated) +celestrak_active: 14376 objects (2 new, 0 updated) +satnogs_full: 1488 objects (121 new, 5 updated) +supgp_starlink: 9703 objects (77 new, 7398 updated) +Total: 33253 unique objects +Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123 +``` + +The 7,398 Starlink updates are fresher epochs from SpaceX overriding stale Space-Track entries. + +## NORAD ID decoding + +TLEs use a 5-character NORAD catalog field. With 100,000+ tracked objects, numeric encoding ran out of space. `pg-orrery-catalog` handles all four encoding cases, matching the C implementation in pg_orrery's vendored SGP4 library: + +| Case | Format | Range | Example | +|------|--------|-------|---------| +| Traditional | `ddddd` | 0 -- 99,999 | `25544` = 25,544 (ISS) | +| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 | +| Super-5 (3) | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 | +| Super-5 (4) | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 | + +Alpha-5 skips letters I and O (ambiguous with 1 and 0). Super-5 uses a base-64 alphabet: 0-9, A-Z, a-z, `+`, `-`. + +## SQL output format + +The generated SQL matches pg_orrery conventions: + +```sql +DROP TABLE IF EXISTS satellites; +CREATE TABLE IF NOT EXISTS satellites ( + id serial, + name text, + tle tle +); + +INSERT INTO satellites (name, tle) VALUES ('ISS (ZARYA)', E'1 25544U 98067A ...\n2 25544 51.6400 ...'); +``` + +## Orbital regime classification + +Objects are classified by mean motion (revolutions per day): + +| Regime | Mean Motion | Description | +|--------|-------------|-------------| +| LEO | > 11.25 rev/day | Low Earth Orbit (period < 128 min) | +| MEO | > 1.8 rev/day | Medium Earth Orbit (GPS, navigation) | +| GEO | > 0.9 rev/day | Near-synchronous (comms, weather) | +| HEO | <= 0.9 rev/day | Highly elliptical (Molniya, tundra, GTO) | + +## Library usage + +```python +from pg_orrery_catalog.tle import decode_norad, parse_3le_file +from pg_orrery_catalog.catalog import merge_sources +from pg_orrery_catalog.regime import regime_summary +from pg_orrery_catalog.output.sql import generate_sql + +# Parse and merge multiple files +merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"]) +print(f"{stats.total_unique} unique objects") + +# Regime breakdown +print(regime_summary(merged)) +# {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123} + +# Generate pg_orrery-compatible SQL +sql = generate_sql(merged, table="my_catalog") +``` + +## Development + +```bash +git clone git@git.supported.systems:warehack.ing/pg-orrery-catalog.git +cd pg-orrery-catalog +uv venv && uv pip install -e ".[dev]" +pytest tests/ -v # 58 tests +ruff check src/ tests/ # Lint +``` ## License