Operator handbook¶
Operate from durable state outward: check the control plane, inspect its external axes, and act only when Beampipe can prove what the scheduler and DALiuGE have done.
Begin a shift¶
beampipe doctorbeampipe statusbeampipe worker listscheduler status --profilebeampipe console/metrics + timelineThe daily workflow explains the role contract and normal path. The console is an operational lens, while PostgreSQL remains authoritative.
Triage by symptom¶
Separate admission pressure from worker capacity
Check ready jobs, stale claims, pool labels, per-profile concurrency, and automation caps before adding workers.
Reconcile before retrying
An SSH disconnect after sbatch is not proof of failure. Search by deterministic job identity and resolve the submission axis first.
Preserve scheduler and session facts independently
Probe translator and manager endpoints, inspect the persisted session identifier, and compare scheduler state before changing execution state.
Do not equate scheduler success with scientific success
Inspect the output-verification axis, immutable artifacts, run record, and provenance before deciding whether work is retryable.
Handbook map¶
| Operational need | Source of truth | Procedure |
|---|---|---|
| Watch active work | Execution ledger and external axes | Live console |
| Scale or drain workers | Claims, heartbeats, labels, pools | Workers and scheduling |
| Check translator, manager, SSH, or Slurm | Profile snapshot and adapter diagnostics | DALiuGE and Setonix |
| Explain a failure | Structured diagnostic and provenance | Recovery and cancellation |
| Tune alerts and dashboards | Prometheus metrics | Observability |
| Promote a release | Migration, backup, and preflight evidence | Production runbook |
| Rotate or restore | Database backup and external secret store | Upgrades, backups, and secrets |
Operator rule¶
Beampipe persists intent before I/O and fences claims so that process restarts and additional workers do not turn uncertainty into duplicate scientific work.