Automation: triggers and subscribers

Overview

Automation lets a company admin run background work when business records change, with no custom code.

A trigger watches one entity type for one kind of change: added, changed, deleted, or a command executed. A trigger on a change can watch chosen properties, and a trigger can have a condition the record must meet.
A subscriber is the work that runs. One subscriber can be attached to many triggers.
Each run is logged as a subscriber run: Scheduled, Running, then Succeeded or Failed, with its duration and reason.

Only committed changes fire. The user's save returns at once, and the subscribers run in the background.

Registration

using Benevia.Core.Automation.DependencyInjection;

services.AddCoreAutomation();

This registers the four entities (Trigger, TriggerSubscription, Subscriber, SubscriberRun), their logic and endpoints, the change detection, the background worker that takes the runs from each tenant's database, and the poll that finds the runs other servers wrote or left behind.

The poll checks every tenant's run log at startup and then every 30 seconds. It also checks whether the tenant's triggers changed, so a trigger change saved on one server reaches every server within one interval. Change the interval with the options. An interval under a millisecond, such as zero or Timeout.InfiniteTimeSpan, or over 49 days turns the poll off, which suits a test host that drives runs itself. With the poll off, a server takes runs only from the tenants its own saves wake it for, and a trigger change reaches only the server that saved it:

services.AddCoreAutomation(options => options.PollInterval = TimeSpan.FromMinutes(1));

How a change becomes a run

A save or a finished Modify command reaches the package through the save observer and method listener hooks.
The package matches the change against the tenant's active triggers, which each server caches per tenant. A trigger change reaches the server that saved it at once, and every other server at its next poll.
After the change commits, it writes one Scheduled run per active subscriber, with the firing the run starts from, and wakes its server's worker. A record that does not meet the trigger's condition gets no run, so the run log lists only the changes a subscriber runs for. Inside a workflow, this waits until the workflow commits. A command run in a workflow step is kept with the workflow and fires at the workflow's next commit, on any server, even one that writes nothing because later steps undid every change; one that changed nothing fires at once.
The background worker runs up to four subscribers at a time. Each time one of its slots is free, it claims the oldest waiting run from a tenant's database, and the tenants with waiting runs take turns, a run each, so one tenant's backlog does not hold up another's. Each run starts from its row, with the subscriber's settings as they are then, and runs in its own tenant scope, as the system user, with a timeout. A run whose trigger was deleted, whose trigger or subscriber was turned off, or whose subscriber was removed from the trigger while it waited fails without running. So does a run whose change no longer matches the trigger: its entity type, change type, watched properties or command changed. So does a run in a copy of the database, as Copied databases describes.
A server claims a run before it starts it, and a claim skips the runs another server's claim holds, so servers sharing a database, as during a deploy or when scaled out, share its waiting runs and never run the same one. Every server polls the run logs: it takes its share of the Scheduled runs another server wrote or left behind, and it marks Failed a run left Running by a server that stopped, once its timeout and a minute passed, with a reason naming the restart.
A run that fails for a temporary reason is tried again, up to the subscriber's RetryAttempts (5 by default, 0 to 10, 0 turns retries off). See Retries.

A trigger never watches the automation entities themselves, a virtual entity type, an abstract entity type, or an entity type that has subclasses: records are saved as their own type, so such a trigger would miss them. Its watched properties and command must exist on the entity type, spelled exactly.

Conditions

A trigger's Condition is an OData $filter expression on its entity type, such as Amount gt 1000 and CustomerGuid ne null. A run is written only when its record meets it. Empty means every change runs.

It is checked right after the change commits, as the runs are written, with one query: does the record exist and meet the filter? For a record that does not, no run is written. Outside a workflow this query is part of the save call, one per trigger with a condition that the change matches; inside a workflow it runs after the response.
It reads the record in the database right after the commit. That is almost always the values the change saved, which a webhook for an added or changed record sends. A change saved a moment later is read instead, so in that short window a run can be missed, or sent with values that no longer meet the condition. A command's values are read the same way. A later edit of the record does not cancel a run already written.
Query filters do not apply. A record a query filter hides, such as a template record, fires as it does on a trigger without a condition.
It is checked once. Neither the run's start nor a retry checks it again.
It runs as the system, with no user's permissions. Only admins edit triggers.
It must be valid for the entity type when it is saved. A condition that breaks later, such as one that names a property removed in an upgrade, writes its runs as Failed with "The trigger's condition could not be checked. The details are in the host log." Such a run does not send.
"Record deleted" has no condition: the record is gone when its runs are written. Changing a trigger's change type to "Record deleted" clears its condition.
Only stored values can be used. The query runs in the database, so a virtual property cannot be in a condition.

To fire on "Status changed to Shipped", watch Status and add the condition Status eq 'Shipped'. There is no "changed from": a trigger keeps no old values.

ConditionRules holds the settings page's rows the condition was built from, as JSON. The server does not read it.

Inside a workflow

Most edit pages save through a workflow, so the timing matters. Take a webhook on a trigger that watches the Customer of a Sales Order:

The user edits the page. Each edit is a workflow step: it is staged in the workflow and never reaches the database, so triggers do nothing. A command run in a step, such as a Post button, is kept with the workflow and fires at the commit in step 2, even when the command saves its own changes. A command that changed nothing in a workflow with no other changes fires at once, because nothing about it can be discarded.
The user saves, which commits the workflow. After the business logic, the save observer asks the cached triggers whether any of them watches a Sales Order change at all; watched properties are not compared yet. If one does, it keeps in memory the columns whose value really changed (CustomerGuid), a snapshot of every column of the record with its new values, and who saved it and when. Nothing is written yet.
The workflow writes its changes, then commits its transaction. AfterCommit runs after the write but before the commit, while the transaction can still roll back, so the package only keeps the change and waits for the response to end.
After the response, if the workflow really committed, the change is matched against the triggers, watched properties included (a reference's …Guid column counts as the reference). Each active subscriber of a matching trigger gets a Scheduled run and its firing, written with one save, and only then is the worker woken. A failed or discarded commit writes nothing.
The worker claims the run, reads it and its firing from the database, checks the trigger, the subscriber and the subscription, sends the webhook, records the outcome, and deletes the firing.

A save outside a workflow follows the same steps, except that step 4 runs inside SaveChanges, before the response.

Retries

A failure is temporary when the run timed out, when its server was shut down while it ran, or when the subscriber returned SubscriberOutcome.TemporaryFailure. A webhook returns one for HTTP 408, 429 and 5xx (except 501 and 505), for a connection refused, reset or dropped, also during the TLS handshake, and for a DNS server that did not answer. A run left Running by a server that crashed counts as temporary too: the poll that closes it writes its next attempt. Any other failure is not retried, such as a refused TLS certificate, an address that does not exist or a blocked address.

Each retry is a new Scheduled run in the run log. Attempt counts the tries, starting at 1, and every retry's IdempotencyKey holds the first run's id, so it groups the attempts. A retry's ScheduledAt is the time it may start. The waits grow: 5 seconds, 10 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, and then 1 hour for each later retry; 10 retries wait about 4 hours in all. A retry waiting when its server stops is started by a poll after the restart.

No retry is written when its trigger was deleted or turned off, or its subscriber turned off or removed from the trigger. A retry already waiting fails without running in those cases too. It also fails when its change is more than 6 hours old. A retry keeps the change's hidden and encrypted values, and must not send them long after the fact. Turn the subscriber off to stop its retries at once.

A failed run may have done part of its work, and a timed-out webhook may have reached the destination. So every webhook request sends an Idempotency-Key header with the first run's id, the same for every attempt, and a receiver can use it to drop a request it already got. Set the header under Headers to send another value. A custom subscriber gets the key as TriggerFiring.IdempotencyKey.

Access

No responsibility grants the automation entities. Only users with full access, or the Administrator role, can read or change them. A user with no roles has full access by default, so give every other user a role. Add a responsibility in the host if other users need the entities.

A user with full access can delete runs, one at a time or by deleting their subscriber. The run log is not a tamper-proof audit trail.

Subscriber types

Each value of SubscriberType has one ITriggerSubscriber implementation. SubscriberType is a Core enum, so a new type is added in Core: add the value and register its implementation in AddCoreAutomation().

A host can replace the implementation of an existing type. Register its own after AddCoreAutomation(); the last registration of a type is the one that runs:

services.AddCoreAutomation();
services.AddScoped<ITriggerSubscriber, MyApplicationLogSubscriber>();

A subscriber gets the TriggerFiring: the change (Event, with the acting user and the depth of the run that caused it), the Trigger and the Subscriber. It runs in a service scope of its own: what it leaves unsaved when it fails is never saved. If it throws, its run shows only the exception type, and the host log gets the exception.

Event.ChangedProperties and Event.Snapshot are not filtered by permissions. They include hidden and encrypted columns, but not binary ones. A subscriber that sends data out of the process must pick the properties it sends. Each snapshot value keeps the type of its property, even for a run that starts after a restart. The firing is kept in a table no API exposes, and is deleted when its run finishes; a retry gets a copy of it.

Webhooks

A subscriber with Type = Webhook calls an external system. Its fields: RequestMethod (GET, POST, PUT, PATCH, DELETE), Url, PayloadType (Json or Text), Headers (one Name: value per line), PayloadTemplate, Authentication (None, Basic, Bearer), Username, and Secret. A webhook must call an https address that names its host, not an IP address. The secret is encrypted with the tenant key as soon as it is set, so not even a workflow's staged state holds it in plaintext; one whose ciphertext would not fit its column is refused. Secret is hidden from clients: they set it with the SetSecret method and never read it, not even encrypted, and only the server decrypts it, to send the request. An encrypted value other than the record's own is refused, since one key decrypts every record of the tenant. SecretHint holds the first four characters of a secret of 12 or more characters, so a page can show which token is saved, and **** for a shorter one; clearing the secret clears the hint. A new Url clears a saved secret, also while Authentication is None, so it is typed again for the new address; a secret typed in the same edit stays. That is why Secret and SecretHint stay visible without sign-in. A refused save names the field to fix, and a webhook that a save would now refuse, such as one saved with http, does not run.

Url, Headers and PayloadTemplate can hold placeholders: {{Record.<Name>}}, {{Trigger.Name}}, {{Trigger.ChangeType}}, {{Trigger.EntityType}}, {{Trigger.EntityId}}, {{Trigger.Command}}, {{User.Id}}, {{User.Name}} and {{Trigger.Timestamp}}. {{Trigger.ChangeType}} is RecordAdded, RecordChanged, RecordDeleted or CommandExecuted. A saved reference gives its id under both {{Record.Customer}} and {{Record.CustomerGuid}}, and its own saved properties under {{Record.Customer.Name}}. A collection the record loads whole gives its rows under {{Record.Lines}}, as a JSON array of objects with each row's saved properties as strings, which a body inserts as it is; a paged collection is not offered. The record's virtual (calculated) properties are offered under {{Record.<Name>}} too. A virtual reference gives only its id, under its own name.

Related values and virtual properties are read when the webhook runs, not at the change. They show the records as they are then, and a deleted record has none. So one request can mix two moments. If the record changes again before the run, a virtual total can already include the later change, while the saved values do not. A virtual property is worked out only when the request uses it. It is worked out in the background, with no signed-in user. So a value that depends on the user or on permissions can differ from what a page shows, or it can fail the run.

Webhooks cannot send binary values, because a change keeps none. A placeholder that names a saved binary property of the changed record fails the run. A binary value that is read when the webhook runs does not send its content. Such a value comes from a virtual property, the record of a command, or a related record. Values are escaped for where they go: JSON string content in a Json body, URL encoding in the address, no line breaks in a header. The address's scheme, host and port cannot hold a placeholder, so a record's value never chooses where the request goes. An unknown placeholder is refused when the subscriber is saved; {{Record.<Name>}} is checked when the subscriber names its entity type. A placeholder that still has no value when the request is built fails the run with the placeholder's name, rather than send an empty field.

A 2xx response is a success, and anything else a failure. The run's description is HTTP <code> <reason> for a success and HTTP <code> <reason>: <first 1,000 characters of the body> for a failure, an HTML page kept as its text without styles and scripts, with the password or token replaced by *** wherever the server put it, even where the cut leaves only its start; a timeout fails with the runner's timeout message. A run the server cut off, because it stopped or restarted, says the request may already have reached the destination. A temporary failure is retried.

Send test (POST /api/Subscriber(<id>)/SendTest?$select=Succeeded,Description) sends the request with sample values and a new Idempotency-Key, and returns a WebhookTestResult: whether it succeeded, and the description a run would log. It writes no run. An editor can test an edit before saving it, so Send test first checks the webhook as a save does; when a rule is broken it sends nothing and returns the save's message as the failure.

A webhook cannot call an address inside the hosting network: loopback, private and shared ranges, link-local and the cloud host addresses (metadata services, Azure's host agent), multicast, and IPv6 addresses that carry one of these are refused at connect time, and redirects are not followed. A host can replace the handler of the ServiceCollectionExtensions.WebhookClientName client, as the tests do.

Copied databases

A tenant database copied to another environment, such as a test or staging copy, runs no subscriber. Its runs never call the customer's systems. Its triggers still fire and its run log records each run, so the copy shows what would have run. Every run fails as it starts, with the reason that the subscriber is not set up for this database. Runs that were waiting when the database was copied fail the same way, and no request leaves. Send test still sends, since an admin asks for it.

Each TriggerSubscription carries a token: its trigger and subscriber, encrypted with the tenant key each time the subscription is saved. A run starts only when the token decrypts with the key of the server that runs it, and names its trigger and subscriber. The key comes from the tenant's configuration, not from the database, so a copy run with another environment's configuration cannot read the token. Clients never read or set it.

Give each environment its own tenant keys. A copy run with the key of its source runs every subscriber.
Turn a subscriber on in a copy on purpose. Remove it from the trigger, attach it again and save. This seals the subscription with the copy's key. Saving only the trigger leaves its subscribers off.
Attach older subscriptions again. No upgrade adds a token. A subscription saved before this version has none, so its runs fail the same way in every environment, production too, until it is attached again.

Limits

A run left waiting by a restart waits for a server's poll. A run still waiting when its server stops stays Scheduled, and so does one its server claimed but had not started yet. A running server starts it at its next poll. A run its server was running when it stopped fails and is retried. One left Running by a server that crashed fails once its deadline passed, at a poll, and is retried from there. A tenant whose features leave automation out runs nothing; once automation is back, a run it left waiting fails instead of sending the old change.
One slow endpoint can hold the worker slots. Every tenant shares four worker slots. A webhook that hangs holds one per attempt: up to about 55 minutes per change at the longest settings. Keep timeouts short.
A crash right after a save can lose its runs. The runs are written just after the change commits, so a failed run write never fails the user's save. A crash between the two loses that change's runs, and a run write that fails is only logged.
A firing is kept up to 256 KB. A change to a record too wide to keep fails its run with a reason. Binary values are not kept, so a subscriber never sees them in Event.Snapshot.
No loop guard. A subscriber that changes a record its own trigger watches fires that trigger again, without end. Each run records its depth, but no limit is enforced yet.
Runs have no order. Two changes to one record can reach a subscriber in either order, and a retry can arrive hours after a newer change. A subscriber that sends changes out must not depend on their order; a webhook can send {{Trigger.Timestamp}} so its receiver drops a change older than the one it has.
A trigger change takes up to one poll to reach the other servers. Several servers can serve the same databases, during a deploy and when scaled out. A server that did not save a trigger, subscription or subscriber change matches saves against its old triggers until its next poll, 30 seconds by default. A save it matches in that window does not fire for a trigger, subscriber or condition the change added or turned on, and nothing recovers that run later. A run it fires on conditions the trigger no longer has fails as it starts, and sends nothing. With the poll off, only the server that saved a change sees it.
A copy with its source's key runs its subscribers. A copy is told apart only by its tenant key, so one run with the key of its source environment runs every subscriber. See Copied databases.
A subscriber must honour its cancellation token. The runner abandons a subscriber at its timeout and records the run. One that ignores its token keeps running after that and can use its scope after it is disposed (beneviasoftware/erp#3725). It can also still be running when its retry starts, so it handles the same change twice at once.
The run log only grows. Every firing a condition does not remove adds a row, and nothing deletes old runs.
Saves without a tenant fire nothing. A save outside a request and outside the package's own worker has no tenant, so it fires no trigger. A trigger change it saves reaches the servers at their next poll. A subscription it saves gets no token, so its runs fail until it is saved again for a tenant, such as by attaching it again.
Subscribers run as the system. A subscriber's writes skip user permissions, and no owner is recorded on a trigger or a subscriber.
Feature id. The entities' feature id comes from their namespace, Benevia.Core.Automation.Model, shortened by the host's Features:NamespaceTrimRoots setting. Use that id to exclude automation for a tenant. See Feature management.