§1. Daily automatic ingestion — Daily (Skycop API, permanent path)

daily/ingest/skycop_unit_update_from_api.py · daily/ingest/skycop_coordinates_from_api.py · daily/ingest/retry_failed_fetches.py

Purpose: bring skycop_coordinates straight from the Skycop API into telemetry_bronze every day, no manual intervention, and keep the telemetry_bronze.skycop_unit catalog up to date with new units the platform reports. This is the automatic path today — it replaced the PostgreSQL daily ingestion on Sep 25 (see §2). 6 Jobs: the 4 ingestion ones, deployed with daily/ingest/deploy_daily_ingest.ps1, plus daily-silver/daily-gold (Sep 28), deployed with daily/materialize/deploy_daily_materialize.ps1:

You can also launch any of the 6 Jobs on demand — with its defaults or with custom parameters — from the Daily tab's Run section (a form that shows which parameters have a specific default and which are simply omitted).

Example — manual execution outside schedule

gcloud run jobs execute daily-coords-included --project=geoagp-data-bigquery --region=us-central1 --wait

Setup: SKYCOP_API_KEY and (optional) SKYCOP_API_BASE_URL in .env (default https://api.skycop.com.py) — without the key the scripts stop with "ERROR: definí SKYCOP_API_KEY".

Key parameters: of skycop_coordinates_from_api.py — --date-from/--date-to (YYYY-MM-DD, inclusive — without dates a fixed window is used: yesterday and the 2 days before, --lookback-days, default 3; no watermark), --fleet-name-list/--fleet-name-exclude-list, --unit-list, --mode {merge,append} (default merge, same unique_key as the pgsql pipeline — both sources converge without duplicating), --batch-size (default 5000), --dry-run, --no-watermark-write, --debug-level N (or env DEBUG_LEVEL, levels 1-3, cumulative).

Example — one month, one fleet

.venv\Scripts\python.exe daily\ingest\skycop_coordinates_from_api.py `
  --date-from 2026-04-01 --date-to 2026-04-30 --fleet-name-list "TRA-PAL-%"

Example — retry only what failed in the last 24h

.venv\Scripts\python.exe daily\ingest\retry_failed_fetches.py --dry-run --debug-level 1

No pagination in skycop_coordinates_from_api.py: it makes one request per (unit, day) — a long range with many units takes proportionally longer. On startup it cross-checks /external/devices and flags new units missing from skycop_unit, and units in skycop_unit the API no longer recognizes (those aren't queried). Every run (success or error) leaves a row in telemetry_ops.job_runs, same as the PostgreSQL pipeline.

From the dashboard: Daily tab — Run (execute any of the 4 Jobs on demand) + live list of Jobs/Scheduler + Cloud Scheduler execution log in reverse-chronological order.

Full doc: daily/docs/daily_EN.md

§2. Migration — historical backfill (PostgreSQL, transitional path)

migration/ingest/skycop_coordinates_from_pgsql.py

Historical purpose: until Sep 25, this was the automatic daily path (see §1 for the current one). Today it stays active only to finish the historical backfill of months that were only partially migrated — the Cloud Scheduler that fired it every day is paused, both Cloud Run Jobs stay deployed and in use by backfill_months.py:

Example — manual execution (one-off backfill, outside schedule)

gcloud run jobs execute ingest-coords-pgsql --project=geoagp-data-bigquery --region=us-central1 --wait

The script's parameters, one-off backfills of a specific range, and the month-by-month historical backfill toolkit (backfill_months.py and friends) all live in migration/README_EN.md and migration/docs/historical_backfill_EN.md.

From the dashboard: Migration tab — watermark/calendar status, Run (execute/backfill on demand), live list of Jobs/Scheduler and execution log.

Full doc: migration/docs/cloud_run_jobs_EN.md, docs/job_schedules_EN.md

§3. Querying data — query_bronze.py / query_silver.py / query_gold.py

util/query_bronze.py · util/query_silver.py · util/query_gold.py

Purpose: query each medallion layer from the command line without writing SQL by hand — the same parameter set across all 3 scripts.

Common parameters: --since/--until (inclusive), --fleet-name-list/--fleet-name-exclude-list, --unit-list, --farm/--plot (silver/gold), --count (default 100, 0 = no limit), --format {tsv,csv,json,table}, --where, --dry-run.

Example — Bronze, 8 rows for a day

.venv\Scripts\python.exe util\query_bronze.py `
  --since 2026-04-15 --until 2026-04-15 --fleet-name-list "TRA-PAL-%" --count 8 --format table

Example — Silver, one unit's track

.venv\Scripts\python.exe util\query_silver.py `
  --since 2026-04-15 --until 2026-04-15 --unit-list "348CB" --format table

Example — Gold, one unit's daily summary (no plot breakdown)

.venv\Scripts\python.exe util\query_gold.py --table gold_daily_unit_summary `
  --since 2026-04-01 --until 2026-04-30 --unit-list "419CB" --format table

Full doc: util/docs/utils_EN.md

§4. Generating reports — generate_kml_csv_from_bigquery.py

util/generate_kml_csv_from_bigquery.py

Purpose: the 3 track-report types (built on Gold's TVFs, computed on the fly, no prior backfill needed) — detailed point CSV, detailed segment CSV, summarized CSV, and KML for map visualization.

There's no "summarized KML": once segments are grouped, the track is no longer continuous, and drawing it as a single line would produce false jumps on the map — the command explicitly rejects that combination.

Key parameters: --date-from/--date-to, --farm/--plot, --unit-list, --contractor-list/--exclude-contractor-list, --fleet-name-list, --count (default 0 = no limit, unlike query_*.py), --format {csv,tsv,table,kml}, --out, --exclude-columns, --show-columns.

Example — detailed point CSV, to a file

.venv\Scripts\python.exe util\generate_kml_csv_from_bigquery.py points `
  --date-from 2026-04-15 --date-to 2026-04-15 --unit-list "348CB" --out puntos.csv

Example — detailed segment CSV, to a file

.venv\Scripts\python.exe util\generate_kml_csv_from_bigquery.py segments `
  --date-from 2026-04-15 --date-to 2026-04-15 --unit-list "348CB" --out segmentos.csv

Example — summarized CSV for a month, one fleet

.venv\Scripts\python.exe util\generate_kml_csv_from_bigquery.py segments --rollup summary `
  --date-from 2026-04-01 --date-to 2026-04-30 --fleet-name-list "TRA-PAL-%" --out resumen_abril.csv

Example — KML for one track

.venv\Scripts\python.exe util\generate_kml_csv_from_bigquery.py segments --format kml `
  --date-from 2026-04-15 --date-to 2026-04-15 --unit-list "348CB" --out recorrido_348CB.kml

KML has no tabular "sample data" of its own — it's the same underlying segment data as the detailed segment CSV above, rendered as map geometry instead of columns; use that example's Sample data output button to see the data.

Full doc: util/docs/utils_EN.md

§5. Web dashboard (this app)

Not a command-line task — the local panel you're using right now (web/dashboard/, service_start.bat/service_stop.bat), with the 7 tabs above: Dashboard (per-table counters), Configuration (sync stats), Inventory (GCP resources + storage size), Migration (PostgreSQL path: watermark status, Run, Jobs/Scheduler, execution log), Daily (Skycop API path: same, for its own 4 Jobs), Cost Balance (estimated monthly cost), and this HOWTO.

Full doc: web/dashboard/docs/Dashboard_architecture_EN.md