Process management

Some work doesn’t fit a single request/response: a long-running job that fans out into many sub-jobs and is only done when all of them finish. chenile-process-management orchestrates exactly this — the classic splitter / aggregator (map-reduce) pattern — on Chenile’s state machine. The flow is declared once as data; the work itself is done by pluggable workers that run outside the state machine and report back by raising events.

2.1.31 substantially revamps this module: unified process configuration, predecessor-driven chaining, trigger lineage, an always-on durable outbox (shared with the trigger framework), a management API and React dashboard, and a JGen blueprint that generates typed multi-level process applications. See the 2.1.31 release notes for migration steps.

At a glance

A trigger raises a ProcessCreate event that creates a root Process. The state machine drives it by type: a leaf executes directly; a non-leaf splits into children, waits for all of them, and aggregates — until it reaches a terminal state, PROCESSED or PROCESSED_WITH_ERRORS. On completion it emits a ProcessCompletedEvent, from which configured successor processes are created, so processes compose in sequence. Everything — process state, definitions, outbox, receipts and trigger idempotency — lives in your application’s own database.

Modules

The domain model

A Process carries the fields that drive orchestration: leaf, processType, parentId, predecessorId, triggerId (correlation to the activation that created it), the transient subProcesses used during a split, splitCompleted, the counters numSubProcesses / numCompletedSubProcesses, serialized input and output, an errors collection, the inherited tenant, and the current STM state.

The state machine

            ┌── leaf ──────────▶ EXECUTING ── doneSuccessfully ──▶ PROCESSED
[create] ─▶ isThisLeafNode?          │  (statusUpdate)  └ doneWithErrors ─▶ PROCESSED_WITH_ERRORS
            └── non-leaf ─▶ SPLIT_PENDING ── splitPartiallyDone / splitDone ─▶ SUB_PROCESSES_PENDING
                                  └ splitDoneWithErrors ─▶ PROCESSED_WITH_ERRORS
SUB_PROCESSES_PENDING ── child done / splitDone ─▶ areAllSubProcessesDone?
        (numSubProcesses == numCompletedSubProcesses && splitCompleted) ─▶ AGGREGATION_PENDING
AGGREGATION_PENDING ── aggregationDone ─▶ errors? ─▶ PROCESSED | PROCESSED_WITH_ERRORS
                    └ aggregationDoneWithErrors ─▶ PROCESSED_WITH_ERRORS

A new process starts at isThisLeafNode. There is no parked or dormant state and no separate activation event — processes are created when triggered or when their predecessor completes (2.1.31 removed the old dormant/activate pair). The guard on areAllSubProcessesDone requires the split to be finalized and every child to have reported, which makes streaming/partial splits safe: children may finish before the split is declared complete without the parent deciding prematurely that it is done.

End-to-end lifecycle

Trigger (Quartz crontab, HTTP, or any adapter)
  → TriggerService.trigger(TriggerInput)            idempotent on (triggerId, eventName)
  → EventProcessor.handleEvent("ProcessCreate", ProcessDto, headers)
  → ProcessManagerImpl.create(ProcessDto)            root created, trigger correlation kept
  → state machine entered → SPLITTER / EXECUTOR / AGGREGATOR worker launched
  → workers raise events (splitDone, subProcessDoneSuccessfully, aggregationDone, …)
  → children created, parents signalled, state advances
  → PROCESSED / PROCESSED_WITH_ERRORS
  → ProcessCompleted emitted → successors created as new processes

HTTP and event surface

{ "processDefName": "feed", "args": { "file": "s3://…" }, "triggerId": "crontab:nightly:abc", "description": "Nightly feed" }

create requires processDefName, serializes args into input, and records triggerId from the payload or the x-chenile-trigger-id header. A direct POST /process always creates a new root — deduplicating repeated activations is the trigger layer’s job, not the process manager’s.

Workers and launch modes

PostSaveHook resolves the worker for the current state and hands a WorkerDto to the configured WorkerStarter:

Every worker type receives the same config map (copied, never mutated). Workers extend the process-delegate base classes, which handle payload typing, tenant propagation and error reporting.

Process definitions

A ProcessDef describes a processType: whether it is a leaf, its parentProcessType, its predecessor relationship (predecessorProcessType, predecessorArgs), and one config map shared by its splitter, executor and aggregator. (2.1.31 replaced splitterConfig / executorConfig / aggregatorConfig with config, and configured successors with predecessor declarations.)

Definitions resolve through IProcessConfigurator.findByName, from one of two sources:

• chenile.process.configurator=json (default) — a classpath JSON resource (defs.json).
• chenile.process.configurator=database — DatabaseProcessConfigurator reads the process_definition table (process_type primary key, definition JSON text). Definitions are cached in a HashMap with explicit invalidation after committed changes; the management API edits them and refreshes the cache.

insert into process_definition (process_type, definition) values
('chunk', '{"leaf":true,"parentProcessType":"file","config":{"batchSize":"100"}}');

Completion and chaining

A terminal process emits a transport-neutral ProcessCompletedEvent (it extends ProcessDto) carrying processId, processType, tenantId, triggerId, state, input, output and successful. The process manager subscribes to completion ingress: every definition whose predecessorProcessType matches the completed type is started as a new process, with predecessorArgs choosing what it receives — INPUT, OUTPUT or BOTH (default). Processes retain trigger, parent and predecessor lineage, so an execution can be traced end to end from the stored IDs and each process’s terminal state and state-entry time — there is no chenile_process_completion_event snapshot table. Chaining is part of the EMIT_COMPLETED command’s transaction; delivery to arbitrary application subscribers stays best-effort after commit.

Multi-tenancy

The x-chenile-tenant-id header is persisted as the inherited BaseJpaEntity.tenant, propagated to sub-processes and into the worker execution context, and carried on completion events — so chained processes and subscribers run under the same tenant. (2.1.31 also fixes tenant preservation through JPA flush.)

Reliable failure reporting

Since 2.1.30, worker payload failures are reported to the process daemon rather than logged while the worker reports success — a malformed payload, incompatible known-field type, missing payload type or invocation failure completes the work item as an error, so a run can’t sit pending with no error record. Unknown JSON properties are tolerated for additive producer evolution.

Configuration reference

Related guides

• Durable outbox — atomic consequences, receipts, fencing, dead-letter and replay.
• Administration API, dashboard & server — crons, manual triggers, definitions, process trees and traces.
• Trigger framework — idempotent activation and execution history.
• Process management vs. Temporal — when to choose which.
• JGen process-management blueprint — generate a typed, multi-level process application.

Durable trigger acceptance

POST /trigger acknowledges HTTP 202 only after committing a DISPATCH_TRIGGER command. Its payload contains only triggerId, the outbox ID. An optional idempotencyKey scopes submission retries by tenant and event. GET /trigger/{id} reads PENDING, RUNNING, DONE or DEAD in the caller tenant. The worker invokes EventProcessor later; invocation output is logged, not returned. That outbox ID becomes the x-chenile-trigger-id and process correlation. There is no trigger_log table. Cron identity, timing and x-headers are generic triggerContext JSON. Local subscriber effects and command acknowledgement commit together; external effects still need receiver-side idempotency.

Crontab API and service split

crontab-api contains the shared JPA model and service/reader contracts. crontab-service contains implementations, persistence, validation, Quartz integration, and the Chenile controller. Replace the old crontab-trigger Maven dependency with crontab-service (or only crontab-api for contract consumers). Public Java packages and database schema are unchanged.

The controller exposes POST/GET /crontab and GET/DELETE /crontab/{id} through the Chenile highway. Every operation delegates to process(...); the service owns validation, tenant isolation, and transactions. Successful responses use standard Chenile envelopes with HTTP 200; invalid input returns 400 and absent/foreign-tenant records return 404. Quartz changes run after commit. See the crontab service README for request examples and BDD coverage.

Standalone process-definition REST API

The ProcessDefinitionService contract is in process-api; implementation and Chenile controller are in process-service. POST /process-definitions creates a definition (201), PUT /process-definitions/{processType} replaces it (200), and GET on those routes lists/retrieves definitions (200). Responses use the standard Chenile envelope and ProcessDef objects. The service is available independently of the optional management dashboard; existing management endpoints delegate to the same implementation and retain their response format.

Definitions remain global. Database mode permits transactional writes and clears the local runtime cache only after commit; JSON mode supports reads and rejects writes with 409. Validation errors return 400, missing definitions return 404, and duplicate creation returns 409. Body/path identities must agree. Parent and predecessor cycles are rejected while forward references are allowed for incremental graph entry. There is no delete endpoint, cross-replica cache invalidation, or worker code generation at runtime. Hosts apply their administrative authorization policy. Cucumber tests exercise the highway, real persistence, global visibility, errors, and cache/rollback behavior.