Expand README with full configuration reference and usage docs
Added: Why section, env vars table, merge strategy explanation, NORAD encoding table, regime classification, library usage examples, SQL output format, and development instructions.
This commit is contained in:
parent
1d36729bed
commit
5248b5c093
183
README.md
183
README.md
@ -2,42 +2,89 @@
|
|||||||
|
|
||||||
TLE catalog builder for [pg_orrery](https://pg-orrery.warehack.ing) — download, merge, and load satellite catalogs from Space-Track, CelesTrak, and SatNOGS into PostgreSQL.
|
TLE catalog builder for [pg_orrery](https://pg-orrery.warehack.ing) — download, merge, and load satellite catalogs from Space-Track, CelesTrak, and SatNOGS into PostgreSQL.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
pg_orrery needs TLEs in a table. The satellite catalog has 30,000+ tracked objects spread across multiple data sources, each with different freshness. Getting them all merged and loaded into PostgreSQL is tedious: download from each source, handle different auth methods, deduplicate by epoch, format as SQL, pipe to psql.
|
||||||
|
|
||||||
|
`pg-orrery-catalog` handles the whole pipeline.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv pip install pg-orrery-catalog
|
# Run directly (no install needed)
|
||||||
# or
|
|
||||||
uvx pg-orrery-catalog --help
|
uvx pg-orrery-catalog --help
|
||||||
```
|
|
||||||
|
|
||||||
For direct database loading:
|
# Or install
|
||||||
|
uv pip install pg-orrery-catalog
|
||||||
|
|
||||||
```bash
|
# For direct database loading (adds psycopg)
|
||||||
uv pip install "pg-orrery-catalog[pg]"
|
uv pip install "pg-orrery-catalog[pg]"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Quick Start
|
Requires Python 3.11+.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Download TLE data
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Download TLE data from all sources
|
pg-orrery-catalog download # All configured sources
|
||||||
pg-orrery-catalog download
|
pg-orrery-catalog download --source celestrak # CelesTrak only (no auth)
|
||||||
|
pg-orrery-catalog download --source spacetrack # Space-Track (needs credentials)
|
||||||
|
pg-orrery-catalog download --force # Re-download even if cached
|
||||||
|
```
|
||||||
|
|
||||||
# Build SQL and pipe to psql
|
Files are cached in `~/.cache/pg-orrery-catalog/` and reused for 24 hours.
|
||||||
pg-orrery-catalog build | psql -d mydb
|
|
||||||
|
|
||||||
# Or load directly (requires [pg] extra)
|
### Build a merged catalog
|
||||||
pg-orrery-catalog load --database-url postgresql:///mydb --create-index
|
|
||||||
|
|
||||||
# Export as 3LE file
|
```bash
|
||||||
pg-orrery-catalog build --format 3le -o merged.tle
|
pg-orrery-catalog build | psql -d mydb # SQL to stdout
|
||||||
|
pg-orrery-catalog build --table satellites -o catalog.sql # Save SQL file
|
||||||
|
pg-orrery-catalog build --format 3le -o merged.tle # 3LE text output
|
||||||
|
pg-orrery-catalog build --format json -o catalog.json # JSON with metadata
|
||||||
|
```
|
||||||
|
|
||||||
# Check cache status
|
When the same NORAD ID appears in multiple sources, the entry with the newest epoch wins. This means CelesTrak SupGP data (operator-provided, fresher epochs) automatically overrides stale Space-Track entries.
|
||||||
|
|
||||||
|
### Build from specific files
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog build /path/to/spacetrack.tle /path/to/celestrak.tle
|
||||||
|
```
|
||||||
|
|
||||||
|
### Load directly into PostgreSQL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pg-orrery-catalog load \
|
||||||
|
--database-url postgresql:///mydb \
|
||||||
|
--table satellites \
|
||||||
|
--create-index
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check status
|
||||||
|
|
||||||
|
```bash
|
||||||
pg-orrery-catalog info --cache
|
pg-orrery-catalog info --cache
|
||||||
```
|
```
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
Create `~/.config/pg-orrery-catalog/config.toml`:
|
Three layers (highest precedence first): CLI flags > environment variables > config file.
|
||||||
|
|
||||||
|
### Environment variables
|
||||||
|
|
||||||
|
| Variable | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| `SPACETRACK_USER` | Space-Track login email |
|
||||||
|
| `SPACETRACK_PASSWORD` | Space-Track password |
|
||||||
|
| `SOCKS_PROXY` | SOCKS5 proxy for CelesTrak/SatNOGS (e.g. `localhost:1080`) |
|
||||||
|
| `DATABASE_URL` | PostgreSQL connection URL for `load` command |
|
||||||
|
| `PG_ORRERY_TABLE` | Default table name |
|
||||||
|
|
||||||
|
### Config file
|
||||||
|
|
||||||
|
`~/.config/pg-orrery-catalog/config.toml`:
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
[spacetrack]
|
[spacetrack]
|
||||||
@ -55,25 +102,101 @@ table = "satellites"
|
|||||||
url = "postgresql://localhost/mydb"
|
url = "postgresql://localhost/mydb"
|
||||||
```
|
```
|
||||||
|
|
||||||
Environment variables (`SPACETRACK_USER`, `SPACETRACK_PASSWORD`, `SOCKS_PROXY`, `DATABASE_URL`) override config file values.
|
|
||||||
|
|
||||||
## Sources
|
## Sources
|
||||||
|
|
||||||
| Source | Auth | Coverage |
|
| Source | Auth | Coverage | Freshness |
|
||||||
|--------|------|----------|
|
|--------|------|----------|-----------|
|
||||||
| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) |
|
| [Space-Track](https://www.space-track.org) | Login required | Full catalog (~30k+ on-orbit) | Hours to days |
|
||||||
| [CelesTrak](https://celestrak.org) | None | Active sats + operator SupGP |
|
| [CelesTrak](https://celestrak.org) | None | Active sats + operator SupGP | Minutes to hours |
|
||||||
| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects |
|
| [SatNOGS](https://db.satnogs.org) | None | Community-tracked objects | Varies |
|
||||||
|
|
||||||
When the same NORAD ID appears in multiple sources, the entry with the newest epoch wins — CelesTrak SupGP data (operator-provided, fresher epochs) overrides stale Space-Track entries.
|
CelesTrak's supplemental GP groups are configured via `supgp_groups`. These contain operator-submitted ephemerides (SpaceX for Starlink, OneWeb, Planet, Orbcomm) that are typically hours fresher than Space-Track's catalog.
|
||||||
|
|
||||||
## NORAD ID Decoding
|
## Merge strategy
|
||||||
|
|
||||||
Handles all four encoding schemes from Bill Gray's sat_code:
|
Sources are processed in order: Space-Track (base catalog), CelesTrak active, SatNOGS, then SupGP groups (freshest last). For each NORAD ID, the entry with the newest TLE epoch wins.
|
||||||
- **Traditional**: 5-digit (00001–99999)
|
|
||||||
- **Alpha-5**: letter + 4 digits (A0000–Z9999 → 100000–339999)
|
Example merge output:
|
||||||
- **Super-5 case 3**: base-64 with uppercase last char (340000–906309663)
|
|
||||||
- **Super-5 case 4**: base-64 with non-digit 4th char (906309664+)
|
```
|
||||||
|
spacetrack_everything: 33053 objects (33053 new, 0 updated)
|
||||||
|
celestrak_active: 14376 objects (2 new, 0 updated)
|
||||||
|
satnogs_full: 1488 objects (121 new, 5 updated)
|
||||||
|
supgp_starlink: 9703 objects (77 new, 7398 updated)
|
||||||
|
Total: 33253 unique objects
|
||||||
|
Regimes: LEO: 31542, GEO: 1203, MEO: 385, HEO: 123
|
||||||
|
```
|
||||||
|
|
||||||
|
The 7,398 Starlink updates are fresher epochs from SpaceX overriding stale Space-Track entries.
|
||||||
|
|
||||||
|
## NORAD ID decoding
|
||||||
|
|
||||||
|
TLEs use a 5-character NORAD catalog field. With 100,000+ tracked objects, numeric encoding ran out of space. `pg-orrery-catalog` handles all four encoding cases, matching the C implementation in pg_orrery's vendored SGP4 library:
|
||||||
|
|
||||||
|
| Case | Format | Range | Example |
|
||||||
|
|------|--------|-------|---------|
|
||||||
|
| Traditional | `ddddd` | 0 -- 99,999 | `25544` = 25,544 (ISS) |
|
||||||
|
| Alpha-5 | `Ldddd` | 100,000 -- 339,999 | `T0002` = 270,002 |
|
||||||
|
| Super-5 (3) | `xxxxX` | 340,000 -- 906,309,663 | `0000A` = 340,000 |
|
||||||
|
| Super-5 (4) | `xxxXd` | 906,309,664+ | `000A0` = 906,309,664 |
|
||||||
|
|
||||||
|
Alpha-5 skips letters I and O (ambiguous with 1 and 0). Super-5 uses a base-64 alphabet: 0-9, A-Z, a-z, `+`, `-`.
|
||||||
|
|
||||||
|
## SQL output format
|
||||||
|
|
||||||
|
The generated SQL matches pg_orrery conventions:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
DROP TABLE IF EXISTS satellites;
|
||||||
|
CREATE TABLE IF NOT EXISTS satellites (
|
||||||
|
id serial,
|
||||||
|
name text,
|
||||||
|
tle tle
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT INTO satellites (name, tle) VALUES ('ISS (ZARYA)', E'1 25544U 98067A ...\n2 25544 51.6400 ...');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Orbital regime classification
|
||||||
|
|
||||||
|
Objects are classified by mean motion (revolutions per day):
|
||||||
|
|
||||||
|
| Regime | Mean Motion | Description |
|
||||||
|
|--------|-------------|-------------|
|
||||||
|
| LEO | > 11.25 rev/day | Low Earth Orbit (period < 128 min) |
|
||||||
|
| MEO | > 1.8 rev/day | Medium Earth Orbit (GPS, navigation) |
|
||||||
|
| GEO | > 0.9 rev/day | Near-synchronous (comms, weather) |
|
||||||
|
| HEO | <= 0.9 rev/day | Highly elliptical (Molniya, tundra, GTO) |
|
||||||
|
|
||||||
|
## Library usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pg_orrery_catalog.tle import decode_norad, parse_3le_file
|
||||||
|
from pg_orrery_catalog.catalog import merge_sources
|
||||||
|
from pg_orrery_catalog.regime import regime_summary
|
||||||
|
from pg_orrery_catalog.output.sql import generate_sql
|
||||||
|
|
||||||
|
# Parse and merge multiple files
|
||||||
|
merged, stats = merge_sources(["spacetrack.tle", "celestrak.tle"])
|
||||||
|
print(f"{stats.total_unique} unique objects")
|
||||||
|
|
||||||
|
# Regime breakdown
|
||||||
|
print(regime_summary(merged))
|
||||||
|
# {'LEO': 31542, 'MEO': 385, 'GEO': 1203, 'HEO': 123}
|
||||||
|
|
||||||
|
# Generate pg_orrery-compatible SQL
|
||||||
|
sql = generate_sql(merged, table="my_catalog")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone git@git.supported.systems:warehack.ing/pg-orrery-catalog.git
|
||||||
|
cd pg-orrery-catalog
|
||||||
|
uv venv && uv pip install -e ".[dev]"
|
||||||
|
pytest tests/ -v # 58 tests
|
||||||
|
ruff check src/ tests/ # Lint
|
||||||
|
```
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user