Skip to content

~ beampipe-core

Rust v2 control plane for archive-driven radio astronomy workflows. Register sources, discover archive metadata, build manifests, and submit DALiuGE pipelines to local or HPC backends.

Install API Reference

Quick start

Use the installed beampipe binary for operator examples. Docker uses the same entrypoint, so host and container commands share the same CLI surface.

docker compose up -d postgres
export DATABASE_URL=postgres://postgres:postgres@localhost:5432/beampipe
export BEAMPIPE_JWT_SECRET=change-me

beampipe migrate
beampipe admin create-user --username admin --password change-me --email admin@example.test
beampipe serve --worker false

Add workers from another shell:

export DATABASE_URL=postgres://postgres:postgres@localhost:5432/beampipe
export BEAMPIPE_JWT_SECRET=change-me
BEAMPIPE_WORKER_SCHEDULER_ENABLED=false beampipe worker

Control plane

+---------+   +-----------+   +----------+   +-----------+
| sources |-->| discovery |-->| ledger   |-->| backends  |
| config   |   | TAP rows  |   | manifest |   | DIM/Slurm |
+---------+   +-----------+   +----------+   +-----------+
     ^              |              |              |
     +--------- operator API / metrics -----------+

beampipe-core does not replace the science workflow. It coordinates the archive-facing and scheduler-facing work around that workflow: source registration, metadata discovery, manifest preparation, graph mutation, backend submission, state, provenance, metrics, and retryable jobs.

Operator map

[.] Getting started Install `beampipe`, run migrations, create an operator, and complete the first source workflow. [.] Operations Run API, scheduler, and workers; tune queues; watch metrics; and handle incidents. [.] Architecture Understand control-plane state, discovery/execution lifecycle, and deployment profile selection. [.] Project configs Define archive queries, transforms, manifests, DALiuGE Graphs, automation, and WASM hooks.

Next steps

  1. Install and bootstrap from Installation.
  2. Run one source through First run.
  3. Review environment variables in Configuration.
  4. Move from mock work to real backends with Deployment profiles.