Discovery and execution

The lifecycle begins with a stable project/source identity and ends only when control state, external observations, artifacts, and provenance explain the outcome.

End-to-end flow

01registersource identity

-->

02discoverarchive metadata

-->

03preparemanifest + graph

-->

04submitscheduler intent

-->

05reconcileSlurm + DALiuGE

-->

06verifyoutputs + outcome

Discovery

Discovery is a repeatable metadata preparation pipeline, not an execution side effect.

Step	Project control	Durable result
Build query	discovery.queries and source template variables	Rendered archive request context
Enrich	discovery.enrichments	Additional archive rows
Prepare fields	prepare_metadata.field_map and typed transforms	Normalized metadata
Derive flags	prepare_metadata.discovery_flags	Admission-relevant facts
Sign	prepare_metadata.signature	Stable change digest

Exclude volatile archive fields such as access URLs, sizes, and timestamps from the signature when they should not trigger new scientific work.

Admission and preparation

Admission checks automation policy, source readiness, queue pressure, and per-profile concurrency before creating runnable work. Preparation then pins the project revision and deployment-profile snapshot, renders the manifest, applies graph patches, and records checksums.

configrevision + profilesnapshot + manifestsha256 + graphsha256

Submission and reconciliation

Submission intent is durable before I/O. Slurm job identity and DALiuGE session identity are persisted as soon as they are known. Pollers then update scheduler and DALiuGE axes independently; the reducer derives the next safe control action.

Observation	Meaning
Scheduler running, DALiuGE building	Normal startup progression
Scheduler succeeded, DALiuGE running	Inconsistent; investigate rather than complete
SSH disconnected after sbatch	Submission uncertain; search before retry
DALiuGE finished, outputs unverified	Continue output verification
Outputs verified	Eligible for successful terminal outcome

Operator reading order

For a stalled execution, inspect:

1. Readiness and admission diagnostics.
2. Execution control phase and external axes.
3. Worker claim and heartbeat.
4. Scheduler job and DALiuGE session identifiers.
5. Immutable artifacts and run record.
6. Provenance timeline and metrics around the transition.

The execution state model defines exact values and retry gates. Deployment profiles define the backend behavior pinned to each execution.