Skip to content

process-protocol.md

Latest commit

 

History

History
2089 lines (1689 loc) · 113 KB

process-protocol.md

File metadata and controls

2089 lines (1689 loc) · 113 KB
Raw

Plugin Process Protocol

Purpose: Document Ze's plugin communication protocol and process lifecycle.

Overview

Ze plugins communicate with the engine via newline-framed YANG RPCs over a single bidirectional connection. All messages use the wire format #<id> <verb> [<json>]\n.

  • Events (engine to plugin): BGP events delivered via deliver-event or deliver-batch RPCs
  • Commands (plugin to engine): Route updates, command dispatch, event emission via engine RPCs
  • Callbacks (engine to plugin): Config verification, command execution, OPEN validation

The protocol is the same for all invocation modes (internal goroutine, external subprocess). Internal plugins get a performance optimization via DirectBridge after startup.

Wire Format

Every message is a single newline-terminated line:

Message type Format Example
Request #<id> <method> [<json>]\n #1 ze-plugin-engine:ready {"subscribe":{...}}
Success #<id> ok [<json>]\n #1 ok {"peers-affected":2}
Error #<id> error [<json>]\n #1 error {"code":"error","message":"..."}

A command answer has a second form, and every peer reads it and writes it. The engine answers a plugin's dispatch-command and dispatch-command-args with a head, zero or more records, and a terminator. The plugin answers the engine's execute-command the same way. Every other request keeps the single line above.

Each answer line is #<id> <kind> <positional fields>, where the kind is top, row, bad, end or nay. An answer line carries no verb and no key name. "Command Execution" below names each word and decodes a real answer field by field. ipc_protocol.md, "Answer Protocol", carries the same grammar with the buffering threshold and the failure cases beside it.

  • <id> is a monotonically increasing uint64 correlation ID
  • Methods use YANG-style <module>:<rpc-name> naming
  • JSON payloads are optional (omitted when empty or null)
  • Error payloads carry code and message fields

Framing: FrameReader uses bufio.Scanner with newline splitting. FrameWriter appends \n to each message. Maximum message size is 16 MB.

Multiplexing: MuxConn wraps a Conn to support concurrent RPCs on a single connection. A background reader goroutine routes responses (verb ok/error) to waiting CallRPC callers by #<id>, and pushes inbound requests (verb is a method name) to the Requests() channel.

5-Stage Startup Protocol

Ze uses a synchronized 5-stage startup protocol with barriers between stages. Within each dependency tier, all plugins must complete each stage before any can proceed to the next. Tiers are sequenced by dependency order (see Tier-Ordered Startup below).

+---------------------------------------------------------------------------+
|                          STARTUP TIMELINE                                 |
+---------------------------------------------------------------------------+
|                                                                           |
|    Plugin A               Coordinator              Plugin B               |
|    --------               -----------              --------               |
|                                                                           |
|    STAGE 1: REGISTRATION                                                  |
|    #1 declare-registration  |                 #1 declare-registration     |
|    {...} ------------------>|                 {...} ------------------>    |
|    <-- #1 ok                |                 <-- #1 ok                   |
|                             |                                             |
|                  BARRIER (all plugins complete Stage 1)                   |
|                                                                           |
|    STAGE 2: CONFIG DELIVERY                                               |
|    <-- #1 configure {...}   |                 <-- #1 configure {...}      |
|    #1 ok ------------------>|                 #1 ok ------------------>   |
|                             |                                             |
|                  BARRIER (all plugins complete Stage 2)                   |
|                                                                           |
|    STAGE 3: CAPABILITY DECLARATION                                        |
|    #2 declare-capabilities  |                 #2 declare-capabilities     |
|    {...} ------------------>|                 {...} ------------------>    |
|    <-- #2 ok                |                 <-- #2 ok                   |
|                             |                                             |
|                  BARRIER (all plugins complete Stage 3)                   |
|                                                                           |
|    STAGE 4: REGISTRY SHARING                                              |
|    <-- #2 share-registry    |                 <-- #2 share-registry       |
|    {...} ------------------>|                 {...} ------------------>    |
|    #2 ok                    |                 #2 ok                       |
|                             |                                             |
|                  BARRIER (all plugins complete Stage 4)                   |
|                                                                           |
|    STAGE 5: READY                                                         |
|    #3 ready {...} --------->|                 #3 ready {...} --------->   |
|    <-- #3 ok                |                 <-- #3 ok                   |
|                             |                                             |
|                  BARRIER (all plugins ready)                              |
|                             |                                             |
|    [BGP peers start]        |                 [BGP peers start]           |
|                                                                           |
+---------------------------------------------------------------------------+

Barrier Semantics:

  • Each plugin signals stage completion via StageComplete(pluginID, stage)
  • Coordinator waits until ALL plugins complete the current stage
  • Only then does coordinator advance to next stage
  • All waiting plugins unblock simultaneously

Stage RPCs:

Stage Direction RPC Method Input Type
1. Registration Plugin to Engine ze-plugin-engine:declare-registration DeclareRegistrationInput
2. Config Engine to Plugin ze-plugin-callback:configure ConfigureInput
3. Capability Plugin to Engine ze-plugin-engine:declare-capabilities DeclareCapabilitiesInput
4. Registry Engine to Plugin ze-plugin-callback:share-registry ShareRegistryInput
5. Ready Plugin to Engine ze-plugin-engine:ready ReadyInput
Post Engine to Plugin ze-plugin-callback:post-startup (empty)

A Stage 2 error response is a REFUSAL, and a dead transport is not. A plugin that cannot take the configuration it was sent answers #<id> error with the reason. The engine reads that response, stops the plugin, and stops ze itself when the plugin's registration carries FatalOnConfigError: a mistyped address family gets an operator a refusal with the reason, not a running router silently missing the feature that plugin owns.

A failure to DELIVER the configuration produces no response at all. A closed connection, a timeout and a canceled context each fail the configure call without the plugin having said anything, so none of them is a refusal and none of them stops ze, whatever the registration asks for. The engine tells the two apart by what comes back: a refusal parses into an *rpc.RPCCallError, which only a response line can produce.

Stage 3 names no wire shape. DeclareCapabilitiesInput carries the BGP capabilities the plugin injects into OPEN and nothing else. A command answer has one encoding on every connection, so a plugin READS the record answer form of dispatch-command and dispatch-command-args, and it WRITES that form for execute-command, without asking for either.

A list of two or more entries arrives with its order beside it. A config section body is a JSON object, and a YANG list inside it is an object keyed by the list key. A JSON object states no order, so the order the operator wrote is delivered as a sibling key: the list name with an @ prefix, holding the entry keys as a JSON array of strings. A YANG node name never starts with @, so the sidecar can never collide with a declared sibling.

{"bgp":{"policy":{"prefix-list":{"ORDERED":{
  "@entry":["10.0.0.0/8","0.0.0.0/0"],
  "entry":{"0.0.0.0/0":{"action":"accept"},
           "10.0.0.0/8":{"action":"reject"}}}}}}}

The sidecar sits beside the list it orders, at whatever depth that list is. Here the operator declared one prefix-list, so prefix-list has one entry and no sidecar, and entry has two, so @entry rides beside it inside the named list. A second prefix-list would give prefix-list a sidecar of its own.

A list of one entry carries no sidecar, because one entry has one order. A plugin that does not read the order ignores the key. A plugin that needs the order reads it with configorder.Entries (internal/core/configorder), which refuses a list of two or more entries that arrives with no order rather than substituting a sorted one.

Timeout: Each stage has a 5-second timeout (configurable via stage-timeout in plugin config). If any plugin fails to complete a stage, startup aborts for all plugins.

Post-Startup Callback. Stages 1-5 run per-phase. Plugins load across up to five phases (config-path auto-load, explicit, family, event-type, send-type) in serial order, so the dispatcher command registry only contains every plugin's commands after the last phase completes. The engine sends the post-startup callback to every running plugin after signalStartupComplete has frozen both the plugin registry and the dispatcher command registry; at that point cross- plugin DispatchCommand is guaranteed to resolve. Plugins register a handler via the SDK OnAllPluginsReady method. Delivery is best-effort (fire-and- forget, bounded timeout, per-plugin goroutine) so a single slow or broken plugin cannot delay notification to the rest.

Why Barriers:

  • Ensures all plugins register commands before any receive config
  • Ensures all capabilities declared before registry shared
  • Prevents race conditions in multi-plugin configurations
  • Guarantees consistent state before BGP peers start

Shared stage-driver. Both startup callers drive these five stages through a single implementation, runStartupHandshake, rather than each hard-coding the wire sequence. The driver owns the wire choreography -- reading each plugin-initiated request, validating its method string, responding, and sending the two engine-initiated callbacks -- while the caller-specific effects between stages are injected through a startupSink. The engine's engineStartupSink performs the full registration set (registry, families, capabilities, commands, subscriptions, bridge dispatch, reactor signaling) and synchronizes each tier through the StartupCoordinator barrier; the hub's hubStartupSink harvests the plugin's declared commands and schema, delivers nil config and nil registry, and runs with no barrier. A protocol change (a new method string, a reordered stage, an added validation) therefore touches one place.

Filter Declaration (Stage 1):

Plugins may include a filters list in their declare-registration to offer named route filters. Each entry declares a filter name, direction (import/export/both), requested attributes, NLRI/raw payload needs, failure mode, and optional overrides of default filters.

Field Type Description
filters[].name string Filter name (referenced in config as <plugin>:<name>)
filters[].direction enum import, export, or both
filters[].attributes list Attribute names to receive (e.g., as-path, community)
filters[].nlri bool Include parsed NLRI text; defaults to true
filters[].raw bool Include raw UPDATE body hex for wire-format filters
filters[].on-error enum reject (fail-closed) or accept (fail-open)
filters[].overrides list Default filters this filter replaces (e.g., rfc:no-self-as)

Config references filters as <plugin>:<filter> in filter { import [...] export [...] }. The runtime uses the declaration to choose raw payload delivery, validate modify deltas against declared attributes, and pick fail-open or fail-closed behavior on filter RPC errors.

Doctor Check Declaration (Stage 1):

Plugins may include a doctor-checks list in their declare-registration to provide runtime health checks. Each entry declares a check name, phase, ordering, and the diagnostic codes it may emit. The engine stores declarations in PluginRegistration and invokes them via the doctor-check callback when show doctor runs.

Field Type Description
doctor-checks[].name string Check name (kebab-case, 1-128 chars)
doctor-checks[].phase enum pre-config, missing-config, or post-config
doctor-checks[].order int Ordering within phase (0-9999, default 0)
doctor-checks[].dependencies list Other check names that must run first
doctor-checks[].platforms list Platform filter (empty defaults to any)
doctor-checks[].codes list Diagnostic codes (1-16, must start with doctor-)

Offline ze doctor does not invoke plugin doctor checks (plugins are runtime processes). Plugin checks cover runtime health; Go-registered checks cover pre-start readiness.

Enricher Declaration (Stage 1):

Plugins may include an enrichers list in their declare-registration to provide show command enrichment. Each entry declares a command path and unique key. The server registers a proxy enricher via show.Register() for each declaration. At show time, the proxy serializes the base map, calls enrich-show with a 2s timeout, and merges the response into the base map. Proxy enrichers are cleaned up via show.Unregister() when the plugin process exits.

Field Type Description
enrichers[].command string Show command path (e.g., show subscriber detail)
enrichers[].key string Unique enricher key within command (kebab-case, 1-128 chars)

Session-Ready Declaration (Stage 1):

A plugin CAN set signals-session-ready in its declare-registration to state that its routes belong to a peer's INITIAL routing update and that it dispatches request peer <addr> plugin session ready once they are out. The engine then holds that peer's End-of-RIB for the report, so the marker means what RFC 4724 Section 4 says it means.

Field Type Description
signals-session-ready bool The plugin reports when its share of a peer's initial routing update is out

The declaration is VOLUNTARY (owner directive, 2026-09-02) and it is the only route an external plugin has to it: an external process appears in no compile-time registry, so a plugin that leaves the field false is never waited for and owes no report. Declaring is a claim about WHEN this plugin's routes belong, not about what it may send.

Three facts have to hold before a peer waits for the process, so declaring alone does not arm the wait. The peer grants send [ update ] or send [ raw ], the plugin declares, and the peer grants receive [ state ]. The last one is the peer telling the process its session started, which is what the report answers: a process the peer never tells cannot push into that session's initial update.

Failure-Policy Declaration (Stage 1):

A plugin CAN set failure-policy in its declare-registration to say what its own failure means. The engine applies it when the plugin's process ends, and when the plugin fails a startup stage after it has declared.

Value On plugin failure May the plugin be restarted?
restart The engine starts the plugin again Yes
ignore The engine logs and carries on without it No
fatal The daemon stops No

The declaration is VOLUNTARY. A plugin that omits the field is read as ignore, which is what the engine did before the field existed, so silence is never read as consent to a restart. A value that is none of the three fails the whole registration: the engine does not guess.

One value answers two questions, because the engine starts a plugin again for exactly one reason. restart is also the plugin's statement that it MAY be started again, and the other two are its statement that it must not be.

The respawn leaf of a plugin { external <name> } block is the operator's request inside that declaration. It can ask for less and never for more. With no leaf the declaration decides. With respawn false a plugin that would have been started again is left stopped. With respawn true against a plugin that declares it must not be restarted, the daemon stops at startup and the error names the plugin, the policy it declared and the leaf.

A restart is bounded: RespawnLimit restarts in RespawnWindow, and MaxTotalRespawns over the life of the daemon. Past either bound the plugin is disabled, the plugin-down warning is raised, and the daemon carries on, which is the ignore outcome.

fatal is open to any plugin, whether ze ships it or an operator wrote it (owner directive, 2026-09-06). Configuring a plugin is accepting its terms.

Pipe Alias Declaration (Stage 1):

A plugin CAN include a pipes list in its declare-registration to name a CLI pipe alias for one of its own commands. An alias is the word an operator types after the pipe character, and it stands for an operator chain. The daemon resolves it, because the daemon runs the chain.

Field Type Description
pipes[].command string Command path the alias sits on. MUST be one of this plugin's own declared commands
pipes[].name string The word an operator types after the pipe character (kebab-case, 1-64 chars)
pipes[].description string The line completion and command help show beside the name
pipes[].expansion string The operator chain the name stands for, as an operator would type it

validatePipeDecls reads the shape and the ownership before any conversion. registerPluginPipes then writes the accepted set into the alias registry under startupRegistrationMu, between the registry row and the runtime families. One bad entry refuses the whole list and fails the plugin's startup, with a message naming the plugin, the command path and the alias name. UnregisterPluginAliases takes the declaration back when the plugin stops or its startup is rolled back, which is what lets a stopped plugin start again.

A pipe alias SELECTS and re-sequences an answer. It renames nothing, sums nothing and counts nothing, so a command that wants one MUST emit the aggregate fields beside the detail rows. The full contract, both collision rules and the derived inheritance barrier are in docs/architecture/api/commands.md.

Three holders refuse a declared alias name: a built-in pipe operator that carries the name, a pipe filter on an OVERLAPPING command path that carries it, and an alias on the EXACT same command path that carries it. The two populations differ because the two resolution rules differ: a filter wins its whole subtree, and a longer alias path deliberately shadows a shorter one.

A declared alias resolves over the SSH exec channel and in the daemon-hosted interactive session, because the daemon expands the chain. It does not resolve in ze cli with no command argument. That process runs its own copy of the interactive model and expands the chain before it sends anything, and no plugin alias is registered there, so an operator reads pipe error: unknown pipe operator: <name> and Tab offers the name nowhere. cliClient.StreamMonitor has the same gap. The repair is a channel that carries the daemon's alias table to the client, and it is not built.

Help Text Declaration (Stage 1):

Each entry of the commands list carries the command's two help texts. They are two declarations, and neither is derived from the other.

Field Type Description
commands[].description string The one-line SUMMARY. Every surface that shows the command on one line reads it: a completion candidate, a list row, a table cell. Maximum 256 bytes, no control character
commands[].description string The LONG explanation the command's own help page prints, under the summary. Maximum 4096 bytes, newlines kept, every other control character refused

The key is description and NOT help, because help already names the summary in a Completion row on this same boundary. One spelling, one meaning.

An absent description is what every plugin written before the key existed sends. It renders as summary-present and explanation-absent, and it MUST NOT render as a blank summary. The engine carries the two texts in two maps (PluginRegistration.CommandShortHelp, PluginRegistration.CommandDescription) and writes them into RegisteredCommand.ShortHelp and RegisteredCommand.Description. VisibleCommandEntries then hands both to command.MergeCommandPaths, which fills each field of the command tree on its own. This package and the command package both spell them ShortHelp and Description.

validateHelpDecls reads both texts where validateShapeDecls reads the shapes, before any conversion. The summary is written into the tab-separated shell-completion format and into the one-line terminal candidate, so a newline, a tab or an ESC in it breaks the format for every row that follows and writes an ANSI sequence to the operator's terminal. The explanation is a paragraph only the command's own help page prints, so it keeps its newlines. The alias description above is held to the same one-line rule.

command help "<name>" answers with both, under short-help and description, for a builtin and for a plugin command alike. A builtin's two texts come from its YANG node: PathToDescription and PathToHelp build the two maps loadBuiltinsWithAliases registers them from.

Answer Shape Declaration (Stage 1):

Each entry of the commands list carries three optional fields that say what the command's ANSWER holds. An absent field is an undeclared field, so a plugin that sends none keeps the behavior it had before these fields existed.

Field Type Description
commands[].shape string doc for one document or one value, map for rows that carry their own keys, tab for rows read against column names
commands[].columns []string The answer's keys, lowercase kebab-case, in the order a person reads them. Needs a shape that has rows. Maximum 64, each name 1 to 64 bytes
commands[].address-fields []string The keys whose value holds an IP address or a prefix. Needs a shape. Maximum 16, each name 1 to 64 bytes

validateShapeDecls reads the three fields before any conversion. It refuses four declarations:

  • an unknown spelling.
  • a field list with no shape.
  • a declaration on a blank command path.
  • a list or a name past its bound.

One bad entry refuses the whole list and fails the plugin's startup. The message names the command and the offending value, clamped to 64 bytes so a plugin cannot write an unbounded string into the daemon log.

registerPluginShapes then writes the accepted set into the shape, column and address-field registries under startupRegistrationMu, beside the alias write. UnregisterPluginShapes takes the declaration back when the plugin stops or its startup is rolled back. A path that an in-core package declared EMPTY returns to that empty declaration rather than to nothing.

A declared shape decides which pipe operators the command publishes, and which it refuses by name before dispatch. A declared address-field list admits | resolve and | origin and limits both transforms to those fields. The full contract is in docs/architecture/api/commands.md.

Query Mode: Stage 1 and No Stage 2

A process started with ZE_PLUGIN_MODE=declare is being interrogated rather than run. It writes the Stage 1 declare-registration message to STDOUT, in the same newline framing and under the same request id #1, and exits 0. There is no connection, no ok response to read, no barrier and no Stage 2, so none of the five stages above runs.

Live start Query mode
Carrier the hub connection the child's stdout
Message #1 ze-plugin-engine:declare-registration <json> the same line, byte for byte for the same declaration
After it Stage 2 configure, then the barrier the process exits 0
Hub variables read host, port, token, CA none

The declaration a query answers carries what the plugin states in its own source. Run derives one field from a Plugin's registered callbacks, wants-validate-open, so a plugin that wants that field in its query answer states it in the declaration it hands the query entry point.

Two routes write that line, and neither drives the handshake. For a plugin the ze binary carries, cli.Run answers from the looked-up registration and calls no plugin code. For a plugin binary ze does not carry, sdk.RunOrDeclare(declaration, activate) writes the line and never calls activate. A plugin that adopts neither route runs its live start under a query, and the reader records it as having sent nothing. Both routes write the line through rpc.WriteDeclaration, so the id, the method and the framing have one definition and cannot drift apart.

Tier-Ordered Startup

Plugins are grouped into dependency tiers before handshake begins. All processes are started at once (single ProcessManager), but the 5-stage handshake is sequenced tier by tier. Tier 0 completes its full handshake -- including command registration -- before tier 1 begins.

Tier computation (Kahn's algorithm / BFS layering):
  Tier 0: plugins with no dependencies      (e.g., bgp-adj-rib-in)
  Tier 1: plugins depending only on tier 0   (e.g., bgp-rs depends on bgp-adj-rib-in)
  Tier N: plugins whose deps are all in tiers < N

Per-tier coordinator: Each tier gets its own StartupCoordinator with tier-local indices. Processes in later tiers block naturally on net.Pipe write until the engine reads their declare-registration during their tier's turn.

+-------------------------------------------------+
|            TIER-ORDERED STARTUP                  |
+-------------------------------------------------+
|                                                  |
|  ProcessManager starts ALL processes             |
|         |                                        |
|         v                                        |
|  TopologicalTiers(names) -> [[rib], [rs]]        |
|         |                                        |
|         v                                        |
|  TIER 0: [bgp-adj-rib-in]                       |
|    Coordinator(1 plugin)                         |
|    5-stage handshake -> commands registered       |
|    procWg.Wait()                                 |
|         |                                        |
|         v                                        |
|  TIER 1: [bgp-rs]                                |
|    Coordinator(1 plugin)                         |
|    5-stage handshake -> can dispatch to rib      |
|    procWg.Wait()                                 |
|         |                                        |
|         v                                        |
|  ALL TIERS DONE                                  |
|    coordinator = nil                             |
|    Start async handlers for ALL processes        |
|                                                  |
+-------------------------------------------------+

Why tier ordering:

  • Prevents "unknown command" errors when dependent plugins dispatch to dependencies during or immediately after startup
  • Dependencies are registered in CommandRegistry after stage 5, so they must fully complete before dependents attempt dispatch-command RPCs

Plugins within the same tier still use the original barrier model (diagram above) -- they progress through all 5 stages together. Tier ordering only serializes across tiers, not within them.

Shutdown

Explicit subsystem shutdown sends ze-plugin-callback:bye with an optional reason:

#99 ze-plugin-callback:bye {"reason":"shutdown"}
#99 ok

After a successful bye, the SDK exits its callback loop and the engine closes the connection. OnBye has signature func(reason string) error on both socket and bridge transports. The SDK sends an error response and keeps the plugin running when cleanup fails.

A live configuration removal sends reason removed while the plugin's engine connection and dependencies are still available. The callback can withdraw installed state through engine RPCs. The server waits up to 500 ms for its acknowledgement. A callback refusal or timeout rejects removal and retains the dependencies; it does not close a live callback transport or report a successful reload. A pending callback keeps its original acknowledgement waiter until it completes or the daemon shuts down. Startup failures before the running stage receive no callback. A disconnect or callback panic instead requires replacement of the failed generation; the removal owner performs that recovery rather than leaving it to the ordinary crash policy.

Removal remains provisional until the whole reload is accepted. On rejection, including a callback that succeeds after timeout, recovery waits for the outer reload's completion and then acquires the existing transaction lock. It reads the current committed configuration, joins an exited or removed generation and restarts its source and needed dependencies, or redelivers configuration to a live source whose cleanup failed. The first recovery attempt runs without another reload. If restoration fails, its ownership record remains; the next explicit reload retries restoration before applying new configuration. Failed cleanup can be retried after recovery; an in-flight callback is never duplicated. Replacement startup receives the committed configuration, not the still-published candidate. Its failure remains a retryable recovery error; ordinary startup and runtime failures keep their declared failure policy.

A failed removal also compensates participants that already committed another change in the same reload, and restores reactor state, before returning the error. Rejecting an outer publication performs the same compensation under the transaction lock. An older scope cannot undo a newer successful reload, including one that accepted the same tree without a diff. Rejected nested scopes restore their predecessor's ownership, so both rejections return to the original tree. If compensation itself fails, the server retains the inverse transaction and blocks new reloads, including no-diff requests, until an explicit retry succeeds. It does not publish a restored tree or repeat completed inverse phases on failure.

When one removal also takes out dependencies, the server reverses the same hard- and optional-dependency tiers used at startup. Each producer must acknowledge successful cleanup before the server stops the next dependency tier. Failed auto-load rollback and orphan cleanup use this order too; a callback or ordering error stops the sequence and is returned to the caller.

Whole-daemon shutdown instead closes plugin connections without bye. Plugins must distinguish live removal from connection closure: subsystem owners retain their daemon-shutdown policy, including firewall rule retention.

Who stops the plugin server

A component does not stop a server it borrowed. Whoever CONSTRUCTS the plugin server stops it, and nobody else.

The hub constructs the one plugin server of a normal daemon and stops it at shutdown. A component that runs inside that server borrows the pointer: the BGP reactor gets it through registry.SetPluginServer -> registry.GetPluginServer -> Reactor.SetPluginServerAny, and its own engine runs as a plugin under the same server. A borrowed server is read, never stopped.

Stopping a borrowed server costs twice, and both costs were measured:

Occasion What a component stopping its host does
Daemon shutdown Server.Stop calls ProcessManager.Stop, which waits for every plugin engine. The engine that made the call cannot return until the call returns, so the stop is bounded only by pluginStopGrace plus the group wait: 3.520s per stop, with resources it installed may be left behind logged twice, on a daemon that had released everything
A reload that removes the component runBGPEngine returns when bgp is removed at reload, not only at shutdown, and its tail stops the reactor. An unguarded stop takes the hub's whole plugin server down, and every other plugin with it, while the daemon keeps running

A component that CONSTRUCTS its own server still stops it: a standalone reactor (Config.Standalone, the in-process runner of le chaos run) self-hosts, and its cleanup is that server's only stop. So the rule is a test of ownership, never of the call site.

Runtime RPCs

Engine to Plugin (Callbacks)

Method Input Response Description
deliver-event {"event":"<json>"} ok Single event
deliver-batch {"events":[...]} ok Batched events
execute-command ExecuteCommandInput a record answer, read as one value into ExecuteCommandOutput Command execution
config-verify ConfigVerifyInput ConfigVerifyOutput Validate candidate config
config-apply ConfigApplyInput ConfigApplyOutput Apply config changes
config-rollback {"transaction-id":"..."} ok Undo changes for a config transaction
config-operation-decompose ConfigOperationDecomposeInput ConfigOperationDecomposeOutput Decompose a config diff into operations
config-operation-verify ConfigOperationVerifyInput ConfigOperationVerifyOutput Validate one config operation
config-operation-apply ConfigOperationApplyInput ConfigOperationApplyOutput Apply one config operation
config-operation-rollback ConfigOperationRollbackInput ConfigOperationRollbackOutput Undo applied config operations
config-operation-commit ConfigOperationCommitInput ConfigOperationCommitOutput Commit a config operation transaction
post-startup None ok Report that all plugin startup phases are complete
validate-open ValidateOpenInput ValidateOpenOutput Validate an OPEN message
encode-nlri EncodeNLRIInput {"hex":"..."} Encode NLRI
decode-nlri DecodeNLRIInput {"json":<raw JSON>} Decode NLRI
decode-capability DecodeCapabilityInput {"json":<raw JSON>} Decode a capability
bye ByeInput ok or error Acknowledged cleanup; an error leaves the plugin running
filter-update FilterUpdateInput FilterUpdateOutput Route filter request
doctor-check DoctorCheckInput DoctorCheckOutput Doctor readiness check
enrich-show EnrichShowInput EnrichShowOutput Show command enrichment

All methods are prefixed with ze-plugin-callback:.

config-operation-*: The five operation callbacks carry a ConfigOperation whose payload is the ordering contract. Beside id, root, owner, type, target and params it carries three kebab-case keys: verb, which is create, destroy or modify; and produces and consumes, each a list of ResourceRef values naming what the operation makes available and what it needs another operation to have produced. The engine orders by verb plus the target's kind, and never compares type, which is the plugin's own label for the work.

An operation payload that carries no verb is refused at planning and the transaction aborts, naming the plugin, the root and the operation id.

None of the five has a default handler in the SDK, so a plugin that registered none answers "unknown method" to each. That is why a participant the planner produced no operation for is applied through config-apply rather than config-operation-apply.

doctor-check: Engine invokes a plugin's declared doctor check by name. Plugin runs the check and returns diagnostics (code, severity, message). Declared during Stage 1 via doctor-checks field in declare-registration. Only invoked at runtime via show doctor; offline ze doctor does not reach plugins.

enrich-show: Engine invokes a plugin's declared show enricher at show time. Plugin receives the command, key, mode ("detail" or "brief"), and base data map as JSON, returns enrichment data to merge into the base map. Declared during Stage 1 via enrichers field in declare-registration. Invoked at runtime when a show handler calls show.Enrich() (mode "detail") or show.EnrichBrief() (mode "brief"). 2s timeout prevents hung plugins from blocking show commands.

filter-update: Engine sends UPDATE attributes to a named filter. Plugin responds accept, reject, or modify (delta-only changed attributes). Includes filter name so the plugin can dispatch to the correct handler.

Plugin to Engine

Method Input Output Description
update-route UpdateRouteInput UpdateRouteOutput Inject route to peers
forward-cached ForwardCachedInput - Forward cached UPDATEs to destination peers
release-cached ReleaseCachedInput - Release cached UPDATEs and do not forward them
relay-stored-route RelayStoredRouteInput - Relay stored wire routes to one established peer
route-install RouteInstallInput RouteInstallOutput Insert a batch of computed routes into the engine Loc-RIB (forked route-installing plugin)
route-remove RouteRemoveInput RouteRemoveOutput Withdraw a batch of routes from the engine Loc-RIB (forked route-installing plugin)
route-metrics RouteMetricsInput RouteMetricsOutput Read recursive next-hop costs and the engine Loc-RIB revision
inject-wire-route InjectWireRouteInput - Inject a raw BGP UPDATE body into the RIB
batch-validate BatchValidateInput BatchValidateResult Apply a batch of RPKI validation decisions
resolve-dns ResolveDNSInput ResolveDNSOutput Resolve a name through the engine's single DNS resolver, so a plugin never builds a second one
state-get StateInput StateOutput Read an owned registered runtime key
state-put StateInput StateOutput Persist an owned key with durable acknowledgement
state-remove StateInput StateOutput Remove an owned key
state-list StateInput StateOutput Enumerate owned keys beneath a prefix
state-increment StateInput StateOutput Atomically advance a durable big-endian uint32 counter
dispatch-command DispatchCommandInput DispatchCommandOutput Inter-plugin command
dispatch-command-args DispatchCommandArgsInput DispatchCommandOutput Exact inter-plugin command with pre-tokenized args
emit-event EmitEventInput EmitEventOutput Push event to subscribers
subscribe-events SubscribeEventsInput - Subscribe to events
unsubscribe-events - - Unsubscribe from events
decode-nlri DecodeNLRIInput DecodeNLRIOutput Decode NLRI via registry
encode-nlri EncodeNLRIInput EncodeNLRIOutput Encode NLRI via registry
decode-mp-reach DecodeMPReachInput DecodeMPReachOutput Decode MP_REACH_NLRI
decode-mp-unreach DecodeMPUnreachInput DecodeMPUnreachOutput Decode MP_UNREACH_NLRI
decode-update DecodeUpdateInput DecodeUpdateOutput Decode full UPDATE

All methods are prefixed with ze-plugin-engine:.

Persistent runtime state

Plugins call StateGet, StatePut, StateRemove, StateList and StateIncrement on the SDK after their handshake, from OnStarted or a later runtime callback. Construction and OnConfigure cannot send these requests: the startup coordinator accepts only the stage methods until the handshake ends. An engine created by a reload completes the same state initialization before subscribing to interface events or opening a packet transport.

The daemon routes all five operations through engineOps on both the JSON connection and DirectBridge. It uses the lifetime-owned storage.Storage registered in statestore; a plugin never obtains that handle over the wire. The owning compiled plugin grants its registered key patterns with statestore.RegisterPluginKeys. Requests use the daemon's configured process name as identity. A request cannot choose another owner or grant itself access to credentials. Template parameters match one path segment, and a list returns only keys that the caller can read.

StateInput.key is bounded to 512 bytes. Values carry at most one MiB and use base64 in JSON. Lists return at most 4096 keys; excess fails rather than returning a partial result. ze-plugin-engine.yang models the same bounds. StateOutput.status explicitly names ok, absent, unavailable, corrupt, read-failed, or persist-failed. Only a successful mutation returns ok. StateGet turns absent into found=false with no error; failed outcomes become rpc.StateError. Existing in-daemon best-effort statestore.Put/Get/Remove callers keep their no-store behaviour, but the strict RPC path never treats false, nil as persistence.

StateIncrement holds the store write guard across read, increment and write, and waits for guard release before acknowledging the new value. An absent counter starts at one. Corrupt data and uint32 exhaustion fail without reset or wrap. OSPF uses this operation for its boot word, so separate instances cannot allocate the same sequence space.

The SDK honours caller cancellation before dispatch and when reading the result. A disk mutation already in progress is completed by the daemon. Cancellation or a lost reply leaves the caller without an acknowledgement, even if the mutation committed; callers cannot infer that an increment is safe to retry.

For existing raw-key consumers, Plugin.StateKeys(ctx) supplies the same SDK operations as ReadKey, WriteKey, RemoveKey and ListKeys. Each call has a five-second timeout beneath the supplied lifetime context. The caller retains that context's cancellation obligation, and the adapter has the same post-handshake restriction as the explicit state methods.

Forked route install (route-install / route-remove)

Route-installing plugins (OSPF, IS-IS) do not program the FIB directly: their SPF installers insert locrib.Path values into the process-wide Loc-RIB singleton (locrib.Default()), which sysrib arbitrates and fibkernel programs. In a FORKED (external) plugin subprocess, locrib.Default() returns nil (the singleton lives in the engine's address space), so those installers instead hold a routeinstall.Sink and ship each operation over route-install / route-remove. The engine applies the batch to its real Loc-RIB, where sysrib's OnChange programs the kernel exactly as for an in-process installer. Each entry carries the redistribute protocol name (not the numeric ProtocolID, which is per-process). The engine resolves it against its registered protocols and refuses an unknown name.

The engine stamps the configured administrative distance before Loc-RIB arbitration. For a BGP path, is-bgp and is-ebgp select the ibgp or ebgp configuration leaf without changing the canonical bgp route owner. Other protocols use their registered name. The producer's wire distance is the fallback only when the engine has no declaration for that class.

The process boundary preserves the primary next-hop device and on-link marker, each ECMP member's device and marker, the selected SRv6 service SID, and the BGP metric fields (is-bgp, aigp, aigp-present, metric-recursive). Invalid addresses reject the batch before any route is installed. These fields let recursive resolution and the FIB use the same path properties for a forked producer as for an in-process producer.

route-metrics returns one distance per requested address, in request order. Each distance distinguishes unresolved reachability from a resolved zero cost and reports whether a recursive BGP hop lacked AIGP. The revision is sampled before resolution, so a routing change during the batch invalidates the snapshot on the next poll. An empty address list requests the revision alone. Requests are limited to 4096 addresses; an invalid address rejects the whole batch. The SDK refuses a missing result or a short distance vector rather than treating either as a usable snapshot.

batch-validate carries ValidationDecision.MsgID as the received UPDATE generation. A zero explicitly selects the current route, or the next route when none is stored. A rejected route marked Ineligible remains stored for later validation; Accept and Ineligible cannot both be true. A successful call returns BatchValidateResult, including for an empty batch; an empty or null result is an error at the SDK boundary.

Event Delivery

A subscription states what a plugin CAN handle. The peer's configuration decides what it GETS. A peer-scoped event is delivered when both halves name it: the plugin subscribed to the type in that direction, and the peer's attach process <name> block grants it. A peer that attaches no block for a plugin feeds it nothing, whatever the plugin subscribed to.

Write no plugin that assumes its subscription is enough. Declare what the program can act on, and tell the operator which receive list the program needs. ze names each peer, process and event type the two halves disagree about, at plugin ready and after every config apply, so an operator can see the gap without reading the program.

An event that is not peer-scoped is untouched by this: the config filter is per peer.

Events that pass both halves are enqueued into a per-process channel. The delivery goroutine drains all available events into a batch and sends them in a single deliver-batch RPC, reducing syscalls and goroutine churn. Single events are delivered as a batch of 1.

Batched Delivery

#42 ze-plugin-callback:deliver-batch {"events":["<json-event-1>","<json-event-2>"]}
#42 ok

The SDK unpacks the batch and dispatches each event to the OnEvent handler individually.

Subscription Namespace

SubscribeEventsInput carries an optional namespace field. Empty (the default, and the wire-compatible value for every pre-existing caller) resolves to the namespace registered by the owning protocol component (bgp today). A non-empty value subscribes to another namespace (e.g. vpn-ipsec) at startup; an unregistered namespace is logged and the subscribe block is skipped rather than registered as a silently-dead NamespaceUnknown subscription. The SDK exposes this via SetStartupSubscriptionsIn(namespace, events, peers, format); SetStartupSubscriptions (namespace "") is unchanged.

A "*" event in a startup subscription expands at registration time into one subscription per registered event type of the namespace (no wildcard branch on the per-event match path).

Enveloped Delivery

SubscribeEventsInput.envelope (opt-in, default false; SDK SetEnvelope(true)) wraps each delivered event string in an EventEnvelope:

{"namespace":"vpn-ipsec","event":"sa-up","payload":<bare payload JSON>}

The envelope rides INSIDE the delivered event string, so it is transparent to both deliver-event and deliver-batch (both still carry a JSON string). It lets a plugin subscribed to several event types discriminate which one arrived even when two events share a payload type (e.g. sa-up vs sa-down). Without the opt-in, delivery is byte-identical to before: the bare payload. The engine renders the envelope at most once per emit and only when a matching subscriber opted in, preserving the lazy-marshal-once cost of the default path.

An Event Emitted Before the Subscribers Exist

A subscription is registered when its plugin handshakes, and plugins handshake in phases: config-path plugins load first, plugin { external ... } plugins after them. An event a config-path plugin emits during its own configure callback is therefore routed correctly and delivered to nobody. getMatching returns an empty set, deliverEvent returns 0, and no error is raised: an empty subscriber set and a filtered-out subscriber look the same.

A startup subscription does not close this window. It is registered at the subscribing plugin's ready handshake, which for a Phase 2 plugin is still after Phase 1 has run.

So a plugin that emits an event an external plugin must see MUST NOT start the work that emits it from its configure callback. It starts that work from OnAllPluginsReady, which the engine fans out after every plugin in every phase is running and both registries are frozen. BGP does this by starting its reactor at configure and its peers from coord.OnPostStartup. IKE does it by stashing the parsed configuration and reconciling peers from OnAllPluginsReady, which is why the first sa-up of a session now has a reader.

The handler MUST NOT wait on the activity it starts. sendPostStartupToAll is not ordered against peer startup, so a handler that blocks on peer activity deadlocks.

DirectBridge Optimization

For internal plugins with an active DirectBridge, deliverBatch() calls bridge.DeliverEvents(events) directly instead of conn.SendDeliverBatch(), bypassing RPC envelope construction, newline framing, and pipe I/O. The plugin's onEvent handler is called synchronously in the delivery goroutine.

Structured Event Delivery

When a plugin registers OnStructuredEvent, the engine delivers *rpc.StructuredEvent instead of formatted text strings. StructuredEvent carries pre-extracted peer metadata (PeerAddress, PeerAS, LocalAS, etc.) and a RawMessage pointer for wire message events. This eliminates the JSON round-trip: the engine skips text formatting, and the plugin reads data directly from AttrsWire (lazy per-attribute parsing) and WireUpdate (zero-copy section access) instead of calling ParseEvent.

For UPDATE events, RawMessage carries AttrsWire and WireUpdate with lazy accessors. For state events, StructuredEvent.State and StructuredEvent.Reason carry the data directly. For other wire messages (OPEN, NOTIFICATION, REFRESH), RawMessage.RawBytes contains the raw wire bytes.

StructuredEvent instances are pooled via GetStructuredEvent/PutStructuredEvent to eliminate per-event heap allocations on the hot path.

Plugins that register both OnStructuredEvent and OnEvent receive structured events via the former and text events via the latter. The delivery pipeline (deliverMixedBatch) routes each event to the appropriate handler based on whether Event or Output is set.

Text Event Formatting: the scratch discipline

A plugin without a structured handler receives text, and the engine builds that text on the delivery path. Four rules keep the path free of allocations, and the comments in internal/component/bgp/server/events.go name this section rather than restate them.

  1. No format strings. No fmt.Sprintf, no fmt.Fprintf, no fmt.Appendf. Reflection defeats escape analysis even when the output size is trivial. Use strconv.AppendUint, netip.Addr.AppendTo, hex.AppendEncode, and append(buf, "literal"...).
  2. No intermediate string lists. Never build a []string and strings.Join it. Write the element, write the separator byte, repeat. No strings.Builder, no strings.ReplaceAll, no strings.Replacer.
  3. One string(scratch) per named boundary. Every other code path stays on []byte. In the event path the boundary is the per-encoding format cache and the RPC payload; each is one conversion per distinct encoding, not one per subscriber.
  4. Scratch is stack-local to the outer caller. var scratchArr [N]byte lives on the goroutine stack of the event function that owns the loop. It is never a struct field, never a sync.Pool, and never per-peer. Output larger than N spills to the heap through append growth for that one call, which is correct and costs one allocation in the pathological case: the array size is chosen to cover the realistic maximum, not every input.

The formatters this path calls take the AppendXxx(buf []byte, ...) []byte shape, so they append into the caller's scratch and never allocate a string of their own.

Event Subscription

Plugins subscribe to events using either:

  1. Startup subscription (recommended): included in the ready RPC so the engine registers atomically before SignalAPIReady, avoiding the race between the reactor sending routes and the plugin subscribing.

  2. Runtime subscription: via subscribe-events RPC in OnStarted callback. Safe but has a small window where events could be missed.

Neither plugin subscription form nor the operator's request subscribe can widen what a peer grants. The operator command can add to the process's live capability within that grant, and the next config apply discards the addition.

A session that reaches Established before a plugin registers its subscription raises its state event into an empty list, and nothing replays it. This is why the startup form is recommended: it registers before SignalAPIReady, and ze starts its peers after every plugin tier.

Cross-Plugin DispatchCommand from Startup. A plugin whose startup logic must call DispatchCommand on a sibling plugin's command (e.g., bgp-rpki enabling the request bgp adj-rib-in enable-validation gate) MUST register the call via OnAllPluginsReady, not OnStarted. OnStarted fires after the plugin's own 5-stage handshake but potentially BEFORE other plugins in later startup phases are loaded, so the dispatcher may not yet know about the target command. OnAllPluginsReady fires via the event loop once the engine has frozen every registry after every phase completes, so the dispatch is guaranteed to resolve.

The Attribute Names in the Filter Text Protocol

FilterUpdateInput.Update is a space-separated list of <name> <value> pairs, followed by the nlri block. The engine renders one subject for the whole chain, so every filter on a peer reads the same text.

Thirteen attribute names can appear. An attribute the UPDATE does not carry produces no pair at all, so a plugin reads absence and never a default value. The order below is the order the engine emits.

Name Attribute Value shape Example
origin ORIGIN (RFC 4271 Section 5.1.1) One token: igp, egp or incomplete origin igp
as-path AS_PATH with AS4_PATH merged in (RFC 6793 Section 4.2.3) One AS number, or a bracketed list as-path [65001 65002]
next-hop NEXT_HOP (RFC 4271 Section 5.1.3) One address next-hop 10.0.0.1
med MULTI_EXIT_DISC (RFC 4271 Section 5.1.4) One unsigned decimal med 100
local-preference LOCAL_PREF (RFC 4271 Section 5.1.5) One unsigned decimal local-preference 150
atomic-aggregate ATOMIC_AGGREGATE (RFC 4271 Section 5.1.6) The bare token, with no value atomic-aggregate
aggregator AGGREGATOR (RFC 4271 Section 5.1.7) <asn>:<address> aggregator 65000:4.4.4.4
community COMMUNITIES (RFC 1997) One community, or a bracketed list. A well-known value renders as its name community [65000:100 no-export]
originator-id ORIGINATOR_ID (RFC 4456 Section 8) One address originator-id 3.3.3.3
cluster-list CLUSTER_LIST (RFC 4456 Section 8) Dotted decimal identifiers in wire order, separated by spaces, with no brackets cluster-list 1.1.1.1 2.2.2.2
extended-community EXTENDED COMMUNITIES (RFC 4360) Sixteen lowercase hexadecimal characters for each, or a bracketed list extended-community 0002fde800000064
aigp AIGP (RFC 7311) One unsigned decimal metric aigp 42
large-community LARGE COMMUNITIES (RFC 8092) <global>:<local1>:<local2>, or a bracketed list large-community 65000:1:2

FilterDecl.Attributes records the attributes a filter reads. It does not narrow this list: the subject is built once for the chain and carries every attribute the UPDATE holds.

The engine writes one warning and no pair when it holds an attribute no renderer names. The chain then runs on the subject it has. An operator sees the line under the ze.log.bgp.reactor.forward subsystem.

A plugin MUST look up a keyword and MUST NOT count tokens from the start of the string. Ze added origin, med, local-preference, atomic-aggregate and cluster-list to the subject on 2026-09-04. A plugin that reads a name is unaffected. A plugin that reads a position is not.

The AS Path in the Filter Text Protocol

The as-path pair of FilterUpdateInput.Update carries the AS path information, not the AS_PATH attribute as it arrived on the wire.

A peer that did not negotiate the four-octet AS capability sends an AS_PATH that holds AS_TRANS (23456) wherever a four-octet AS number belongs, and sends the real numbers in the AS4_PATH attribute (RFC 6793 Section 4.2.2). The engine reconstructs one path from the two before it renders the text (RFC 6793 Section 4.2.3), so every text-mode filter reads the AS numbers the route traversed on every session. There is no as4-path token: one attribute name carries one fact.

A filter that must see the AS_PATH and AS4_PATH attributes as encoded MUST declare raw=true and read the wire payload from FilterUpdateInput.Raw.

Non-CIDR Families in the Filter Text Protocol

The engine's filter text protocol (FilterUpdateInput.Update) inlines NLRI prefixes only for address families whose wire encoding is a plain CIDR prefix. These are the "CIDR-family" set: IPv4 and IPv6 for the SAFIs unicast, multicast, and mpls-label. Every other family -- EVPN, Flowspec, VPN, BGP-LS, MVPN, MUP, RTC, and any future family with a specialised NLRI encoding -- is classified non-CIDR.

For non-CIDR families the engine emits a marker block of the form nlri <family> <op> (for example nlri l2vpn/evpn add) with NO prefixes. The marker tells a text-mode filter plugin that the family is present in the update without forcing the engine to generate a family-specific text format.

A filter plugin that needs per-NLRI decisions on a non-CIDR family MUST declare raw=true in its FilterRegistration and parse the wire payload itself from FilterUpdateInput.Raw. A raw=false filter attached to a session carrying non-CIDR families sees only the marker block and is advisory for those families -- it cannot distinguish individual destinations within the family.

Family set Filter text protocol emits Filter requirement
CIDR (ipv4/ipv6 unicast / multicast / mpls-label) nlri <family> <op> <prefix>... (prefixes inline) raw=false works for text-mode per-prefix decisions
Non-CIDR (EVPN, Flowspec, VPN, BGP-LS, MVPN, MUP, RTC, ...) nlri <family> <op> (marker only) raw=true required for per-NLRI decisions; text-only is advisory

Subscription fields:

Field Type Description
events []string Event types (e.g., ["update","state"]); "*" expands to all event types of the namespace
peers []string Peer filter (e.g., ["*"] for all)
format string Format preference (e.g., "json")
encoding string "json" (default) or "text"
namespace string Event namespace; empty resolves to the protocol component's default (bgp), non-empty (e.g. vpn-ipsec) subscribes to another namespace (see Subscription Namespace)
envelope bool When true, deliveries are wrapped in an EventEnvelope (see Enveloped Delivery); default false = bare payload

Internal Plugin Invocation Modes

Ze plugins run as long-lived processes (goroutines for Go, subprocesses for external). Each plugin registers the families it handles at startup, then processes requests in a loop.

Architecture Overview

+--------------------------------------------------------------------------+
|                              ENGINE                                       |
|                                                                           |
|   +------------------------------------------------------------------+   |
|   |                        Family Registry                            |   |
|   |   ipv4/flowspec     -> flowspec plugin                            |   |
|   |   ipv6/flowspec     -> flowspec plugin                            |   |
|   |   ipv4/flowspec-vpn -> flowspec plugin                            |   |
|   |   ipv6/flowspec-vpn -> flowspec plugin                            |   |
|   +------------------------------------------------------------------+   |
|                                    |                                      |
|                        RPC (MuxConn / DirectBridge)                       |
|                                    |                                      |
|   +------------------------------------------------------------------+   |
|   |       FLOWSPEC PLUGIN (long-lived goroutine / process)            |   |
|   |                                                                    |   |
|   |  1. 5-stage startup (YANG RPCs)                                   |   |
|   |  2. Event loop (encode/decode callbacks)                          |   |
|   +------------------------------------------------------------------+   |
+--------------------------------------------------------------------------+

Automatic OPEN Capability Injection

Key design: When a plugin declares decode for a family, the engine automatically advertises that family in OPEN messages via Multiprotocol capability (Code 1).

Rationale:

  • If a plugin can decode a family, peers should be able to send it
  • No explicit capability declaration needed for Multiprotocol
  • Reduces protocol overhead and prevents duplicate capability issues

How it works:

Plugin Stage 1: declare-registration with family ipv4/flow mode=decode
                     |
Registry: families["ipv4/flow"] = "flowspec"
                     |
Session.sendOpen(): GetDecodeFamilies() -> ["ipv4/flow", ...]
                     |
OPEN: Multiprotocol(AFI=1, SAFI=133)

Override behavior: Config families completely override plugin families:

  • Config has family {} block: ONLY config families used, plugin families ignored
  • Config has NO family {} block: plugin decode families used

This is intentional: explicit config = full control. Plugin families provide defaults when config doesn't specify families.

Auto-loading plugins: When a family is configured but no plugin has claimed it, the engine automatically loads the internal plugin for that family (if one exists).

Five-phase plugin startup:

  1. Phase 1: Config-path plugins start first (for example, BGP, interface, and FIB infrastructure plugins)
  2. Phase 2: Explicit plugins from plugin { external ... } start after config-path infrastructure is available
  3. Phase 3: The engine checks which configured families are still unclaimed. Internal plugins are auto-loaded only for unclaimed families.
  4. Phase 4: The engine checks which custom event types are referenced in peer receive config but not produced by any running plugin. Producing plugins and their transitive dependencies are auto-loaded. For example, receive [ update-rpki ] auto-loads bgp-rpki-decorator and its dependency bgp-rpki.
  5. Phase 5: The engine checks which custom send types are referenced in peer send config but not enabled by any running plugin. Enabling plugins and their transitive dependencies are auto-loaded. For example, send [ enhanced-refresh ] auto-loads bgp-route-refresh.

Family auto-loading (Phase 3) is prevented when:

  1. An explicit plugin declares decode for the family (family-based check)
  2. --plugin <name> is passed on command line (prevents auto-load for that plugin)

The check is based on family claims, not plugin name. Plugin names are informational only.

Config Plugin Result
family { ipv4/flow; } None Auto-loads bgp-nlri-flowspec
family { ipv4/flow; } --plugin bgp-nlri-flowspec Uses explicit plugin (no auto-load)
family { ipv4/flow; } plugin { external my-traffic { declares ipv4/flow } } Uses config plugin (no auto-load, family claimed)
family { ipv4/foo; } None Startup fails (no plugin for family)

Event type auto-loading (Phase 3) triggers when a peer process has receive [ <custom-type> ] and no running plugin produces that event type. The producing plugin is found via registry.PluginForEventType() which matches against Registration.EventTypes. Dependencies are resolved transitively.

Config Plugin Result
receive [ update-rpki ] None Auto-loads bgp-rpki-decorator + dependency bgp-rpki
receive [ update-rpki ] plugin { external rpki-decorator { ... } } Uses explicit plugin (no auto-load)

Send type auto-loading (Phase 4) triggers when a peer process has send [ <custom-type> ] and no running plugin enables that send type. The enabling plugin is found via registry.PluginForSendType() which matches against Registration.SendTypes. Dependencies are resolved transitively.

Config Plugin Result
send [ enhanced-refresh ] None Auto-loads bgp-route-refresh
send [ enhanced-refresh ] plugin { external route-refresh { ... } } Uses explicit plugin (no auto-load)

Functional tests:

  • test/plugin/flowspec-open-capability.ci - auto-load for known family
  • test/plugin/family-no-plugin-failure.ci - failure for unknown family
  • test/plugin/explicit-plugin-precedence.ci - explicit --plugin prevents auto-load
  • test/plugin/explicit-plugin-config.ci - config plugin prevents auto-load
  • test/plugin/rpki-decorator-autoload.ci - auto-load for custom event type
  • test/parse/send-enhanced-refresh.ci - dynamic send type accepted in config
  • test/parse/send-unknown-rejected.ci - unregistered send type rejected

Ordering: Plugin families are sorted alphabetically for deterministic OPEN messages.

What plugins should NOT do:

  • Send declare-capabilities with Multiprotocol (Code 1) for their families
  • Assume plugin families will be used if config has a family {} block

What plugins SHOULD do:

  • Declare decode for all families they can parse (provides defaults)
  • Use declare-capabilities only for non-Multiprotocol capabilities (GR, hostname, etc.)

NLRI Routing via Engine RPCs

NLRI encode/decode requests are routed via the engine's plugin registry:

Direction RPC Method Input Output
Plugin to Engine ze-plugin-engine:encode-nlri {"family":"...","args":[...]} {"hex":"..."}
Plugin to Engine ze-plugin-engine:decode-nlri {"family":"...","hex":"...","add-path":<bool>} {"json":<raw JSON>}
Engine to Plugin ze-plugin-callback:encode-nlri {"family":"...","args":[...]} {"hex":"..."}
Engine to Plugin ze-plugin-callback:decode-nlri {"family":"...","hex":"...","add-path":<bool>} {"json":<raw JSON>}

How it works:

  1. Plugin calls EncodeNLRI/DecodeNLRI via engine RPC
  2. Engine looks up the family plugin via registry.LookupFamily()
  3. Engine sends callback to the appropriate family plugin
  4. Family plugin processes and returns result

For in-process plugins with DirectBridge, the RPC path is replaced by direct function calls, bypassing JSON marshaling and pipe I/O entirely.

Mode 1: In-Process (goroutine + net.Pipe + DirectBridge)

For Go plugins (ze.pluginname) -- runs in same process:

  1. startInternal() creates a single net.Pipe for bidirectional YANG RPC
  2. Creates a DirectBridge and wraps the plugin-side connection in BridgedConn
  3. Runner goroutine receives BridgedConn (implements net.Conn) transparently
  4. SDK discovers bridge via Bridger type assertion in NewWithConn()
  5. 5-stage startup runs over sockets (cold path, 5 round-trips total)
  6. After Stage 5: bridge activates for direct function calls (hot path)

Bridge activation sequence:

Step Side Action
1 Engine wireBridgeDispatch() registers DispatchRPC handler on bridge
2 Engine Sends Stage 5 OK response over pipe (last pipe message)
3 Engine If ReadyInput.Transport == "bridge": calls conn.SetBridge(bridge)
4 SDK Receives OK, registers DeliverEvents handler on bridge
5 SDK Calls bridge.SetReady() -- bridge now active
6 SDK Closes pipe (engineMux.Close()), enters bridgeEventLoop

The engine wires its handler (step 1) before sending OK (step 2), ensuring no race between SDK bridge activation and engine readiness. After bridge activation, the pipe is fully shut down -- the MuxConn readLoop exits, and all engine-to-plugin callbacks flow through bridge.CallbackCh().

Engine-side dispatch registry: all three transports for a plugin-to-engine RPC (the socket JSON path, the in-process Direct path, and the typed DirectBridge fast-path slot) derive from a single method registry -- one engineOp entry per operation carrying the rpc.Method* wire string, a handler receiving the request context and process, and an optional typed-slot descriptor. wireBridgeDispatch installs the typed slots by iterating the entries that declare a descriptor (not a hand-written Set* list), and dispatchPluginRPC / dispatchPluginRPCDirect resolve the method through the same table, so adding an operation touches one place and the JSON / Direct / bridge paths cannot drift. The rpc.Method* constants are shared with the SDK caller so the sent and dispatched method strings stay in lockstep.

Generic direct calls use DispatchRPC(ctx, method, params). The typed route and command slots also carry the SDK caller's context into CommandContext. For batch route announce and withdrawal, that context reaches write admission and the peer socket write and flush. Cancellation ends the operation itself, not just a goroutine waiting for an uncancelled writer.

Runtime hot path (after bridge activates):

Direction Socket path (before) Direct path (after)
Engine to Plugin events (text) RPC envelope -> newline frame -> net.Pipe.Write -> read -> unmarshal -> onEvent bridge.DeliverEvents(events) -> onEvent directly
Engine to Plugin events (structured) -- bridge.DeliverStructured([]any) -> onStructuredEvent with *StructuredEvent (no text formatting, no JSON parsing)
Plugin to Engine RPCs (generic) json.Marshal -> newline frame -> net.Pipe.Write -> read -> unmarshal -> dispatcher.Dispatch bridge.DispatchRPC(ctx, method, params) -> dispatcher.Dispatch directly
Plugin to Engine dispatch-command JSON marshal DispatchCommandInput -> RPC -> unmarshal -> dispatch bridge.DispatchCommand(ctx, command) -> dispatchCommand() directly (struct passthrough, no serialization)
Plugin to Engine dispatch-command-args JSON marshal DispatchCommandArgsInput -> RPC -> unmarshal -> exact command route bridge.DispatchCommandArgs(ctx, command, args, peer) -> dispatchCommandArgs() directly (Go args slice, no tokenizer)
Plugin to Engine emit-event JSON marshal EmitEventInput -> RPC -> unmarshal -> deliver bridge.EmitEvent(namespace, eventType, ...) -> deliverEvent() directly (Go strings, no JSON)
Engine to Plugin callbacks MuxConn RPC + 3-way select bridge.SendCallback() -> callback channel -> bridgeEventLoop 2-way select

Callback dispatch: Both event loops (pipe and bridge) dispatch through a generic callback registry (map[string]callbackHandler). Each On* method registers a typed wrapper in the map. Adding a new callback requires only one On* method -- zero changes to the dispatch or event loop code. See rules/plugin-design.md "SDK Is Generic".

Shutdown and callback failure: Process.Stop() cancels the context and calls bridge.CloseCallbacks() (guarded by sync.Once), closing callback channels. The bridgeEventLoop exits on channel close. SendCallback recovers from send-on-closed-channel panics and returns ErrBridgeClosed. If a DirectBridge callback panics, the SDK sends an ErrBridgeFailed-wrapped error to the waiting caller, marks callbacks failed, closes callback channels, and later SendCallback / ExecuteCommand calls fail fast.

Files:

File Purpose
pkg/plugin/rpc/bridge.go DirectBridge, BridgedConn, Bridger, BridgeCallback, SendCallback, CloseCallbacks
internal/component/plugin/server/dispatch_registry.go engineOp, engineOps (the unified plugin-to-engine method registry), serveEngineOpJSON, serveEngineOpDirect
pkg/plugin/sdk/sdk_callbacks.go initCallbackDefaults, On* wrappers, callbackHandler registry
pkg/plugin/sdk/sdk_dispatch.go eventLoop, bridgeEventLoop, getCallback -- generic dispatch
internal/component/plugin/process/process.go Bridge creation in startInternal(), bridge check in deliverBatch(), CloseCallbacks in Stop()
internal/component/plugin/ipc/rpc.go PluginConn.SetBridge(), CallRPC bridge routing
internal/component/plugin/server/startup.go Bridge transport activation after Stage 5 OK
pkg/plugin/sdk/sdk.go Bridge discovery, callEngineRaw() bridge path, SetReady(), pipe close

Mode 2: Subprocess (TLS connect-back)

For external plugins (Python, Rust, etc.) -- runs as separate process:

  1. Engine starts TLS listener from plugin { hub { server <name> { ip ...; port ...; secret ...; } } } config
  2. Engine forks /bin/sh -c <run> as the child, with env vars: ZE_PLUGIN_HUB_HOST, ZE_PLUGIN_HUB_PORT, ZE_PLUGIN_HUB_TOKEN (per-plugin unique token), ZE_PLUGIN_CA_PEM (the PEM certificate authority root that issued the listener's certificate), ZE_PLUGIN_NAME
  3. Child validates the engine certificate chain against that root and nothing else, then authenticates with #0 auth {"token":"...","name":"..."}
  4. Engine validates token matches the per-plugin token generated for that name (name binding prevents impersonation)
  5. Token is cleared from the child's OS environment after first read (Secret: true registration)
  6. Single bidirectional connection using MuxConn (responses routed by #<id>, requests via Requests() channel)
  7. No DirectBridge -- always uses newline-framed RPC over TLS
  8. Same 5-stage handshake over the same connection

The Go SDK registers ze.plugin.name with the transport environment keys, so SDK consumers can read the daemon-assigned identity without the plugin CLI. Compiled fixture observers use that identity when they call NewFromTLSEnv; the SDK constructor authenticates with the name its caller supplies. The plugin CLI keeps go-plugin as its fallback when no name is assigned.

The trust anchor is the issuer, not one certificate. TLSConfigWithRoot builds the client config from ZE_PLUGIN_CA_PEM alone. It fails closed: an empty root and an unparsable root each return an error and no config, so no child holds a config it cannot verify a peer with. A certificate the engine reissues still validates, because the anchor outlives it. The engine reissues on its next start, and also while it runs: the acceptor serves through plugin.ServingLeaf, which answers tls.Config.GetCertificate and mints a fresh leaf once two thirds of the current one's life is spent. A plugin that connects back on day two therefore meets a valid certificate.

Two dialers reach the acceptor, and both go through TLSConfigWithRoot: sdk.dialAndAuth, which a standalone plugin binary uses, and cli.connFromEnv, which the framework every in-tree system plugin registers through uses when the engine starts it out of process. Neither writes its token before the chain validates.

ZE_PLUGIN_HUB_HOST is the acceptor host, except that an unspecified listen address (0.0.0.0 or ::) is handed to the child as 127.0.0.1. The issued certificate carries no SAN for an unspecified address, and the plugin runs beside the engine, so the loopback is both reachable and verifiable.

Benefits of Long-Lived Design

Benefit Description
No per-request overhead Plugin starts once, handles many requests
Language agnostic Same protocol for Go/Python/Rust
Hot-swappable Restart plugin without engine restart
Testable Plugin protocol can be tested independently
Internal optimization In-process plugins bypass transport overhead via DirectBridge

Family Plugin NLRI System

Family plugins provide NLRI encoding/decoding for address families that require complex parsing (FlowSpec, EVPN, BGP-LS, VPN). This section details the complete protocol.

Family Registration (Stage 1)

Plugins declare which families they handle via the families field in DeclareRegistrationInput. Each declaration carries both the canonical name AND the RFC 4760 wire-format AFI/SAFI numbers, so the engine can register the family in internal/core/family/ (the cross-component family registry) at runtime alongside families registered by internal plugins at init.

{
  "families": [
    {"name": "ipv4/flow",     "mode": "encode", "afi": 1, "safi": 133},
    {"name": "ipv4/flow",     "mode": "decode", "afi": 1, "safi": 133},
    {"name": "ipv6/flow",     "mode": "encode", "afi": 2, "safi": 133},
    {"name": "ipv6/flow",     "mode": "decode", "afi": 2, "safi": 133},
    {"name": "ipv4/flow-vpn", "mode": "both",   "afi": 1, "safi": 134},
    {"name": "ipv6/flow-vpn", "mode": "both",   "afi": 2, "safi": 134}
  ]
}

FamilyDecl fields:

Field Values Description
name "ipv4/flow", "l2vpn/evpn", etc. Address family (afi/safi canonical form)
mode "encode", "decode", "both" Direction of conversion
afi RFC 4760 AFI number (e.g., 1 = IPv4, 2 = IPv6, 25 = L2VPN, 16388 = BGP-LS) Required for runtime registration
safi RFC 4760 SAFI number (e.g., 1 = unicast, 133 = FlowSpec) Required for runtime registration

Runtime family registration:

After the plugin completes Stage 1, the engine calls registerPluginFamilies, which validates the whole FamilyDecl batch and commits it through family.RegisterFamilyBatch. This is the runtime equivalent of internal plugins calling family.MustRegister at init. The server records the families actually added for that plugin and removes them if a later startup stage fails before the plugin reaches ready. After committed startup, Family.String() and family.LookupFamily() return the plugin's family. Re-registration with identical values is a no-op; conflicting AFI or SAFI names abort plugin startup.

Registry conflict detection:

  • Only ONE plugin can register for a family+mode combination
  • Conflict results in startup error

OPEN capability injection (decode mode):

  • Families declared with decode are automatically advertised in OPEN
  • Engine adds Multiprotocol capability (Code 1) for each decode family
  • No explicit declare-capabilities needed from plugins for Multiprotocol

Encode/Decode Protocol

Engine to Plugin (callback):

RPC Input Output
ze-plugin-callback:encode-nlri {"family":"ipv4/flow","args":["destination","10.0.0.0/24"]} {"hex":"0701180A0000"}
ze-plugin-callback:decode-nlri {"family":"ipv4/flow","hex":"0701180A0000"} {"json":{"destination-ipv4":...}}

Plugin to Engine (via registry):

RPC Input Output
ze-plugin-engine:encode-nlri {"family":"ipv4/flow","args":["destination","10.0.0.0/24"]} {"hex":"0701180A0000"}
ze-plugin-engine:decode-nlri {"family":"ipv4/flow","hex":"0701180A0000"} {"json":{"destination-ipv4":...}}

Additional Decode RPCs (Plugin to Engine)

These RPCs allow plugins to request full UPDATE or MP attribute decoding:

RPC Input Output
ze-plugin-engine:decode-mp-reach {"hex":"...","add-path":false} {"family":"...","next-hop":"...","nlri":[...]}
ze-plugin-engine:decode-mp-unreach {"hex":"...","add-path":false} {"family":"...","nlri":[...]}
ze-plugin-engine:decode-update {"hex":"...","add-path":false} {"json":"..."}

Error Handling

Error Type Response
Invalid family #<id> error {"code":"error","message":"unknown family: ipv4/unknown"}
Parse error (encode) #<id> error {"code":"error","message":"invalid prefix: 10.0.0/24"}
Cannot decode #<id> error {"code":"error","message":"..."}
Handler not registered #<id> error {"code":"error","message":"encode-nlri not supported"}

Files

File Purpose
internal/component/plugin/registration.go Family registry, conflict detection
internal/component/plugin/server/events.go NLRI routing
pkg/plugin/rpc/types.go RPC input/output types
pkg/plugin/sdk/sdk_engine.go SDK encode/decode methods
pkg/plugin/sdk/sdk_dispatch.go SDK encode/decode callback handlers

Command Execution

Plugins register commands in Stage 1 via the commands field of DeclareRegistrationInput. At runtime, the engine dispatches commands to plugins via execute-command:

Engine to Plugin:

#5 ze-plugin-callback:execute-command {"serial":"abc","command":"rib adjacent status","args":[],"peer":"*"}

Plugin to Engine: the answer is a head, its records and a terminator. Every plugin writes it. The frame is the same whatever the payload is, so a handler that built one value takes the same three lines as a handler that walked a table.

An answer line carries no verb and no key name. The field after the id is a three-byte word saying what the line IS, and every field after that is positional. The five words:

Word The line is Its fields, in order
top the head. It opens the answer and is always the first line for this id item type, envelope name, column names
row one record the command produced the payload
bad one record the command rejected. The walk goes on the payload
end the terminator. It ends the answer and is always the last line for this id records produced, rows rejected, message
nay the whole answer to a command text naming no command error code, message

The head's item type is a three-byte word too, and it says how the records read:

Word The records are
doc one document. The whole answer is that one value
map one map of names to values for each record
tab one positional row for each record, read against the head's column names

Two field shapes carry every value. A NUMBER is decimal digits closed by a space or by the end of the line. A TEXT is decimal digits, a colon, then that many BYTES. The count is a BYTE count, never a count of characters. A value holding multi-byte utf-8 is therefore sliced by the bytes that arrived. A text of zero bytes is written 0:, present and empty, so a line's field count never varies.

A command that RAN and failed states its reason on the TERMINATOR, which is the one line an answer states an outcome on. It is not an error response, because the RPC itself succeeded even when the command failed. The error verb is kept for a handler that returned a Go error, where the RPC itself did not succeed.

A handler that built one value writes it as the one record of a doc answer. The head states no outcome, and the record carries the value byte for byte:

#5 top doc 0: 0:
|  |   |   |  |
|  |   |   |  +----- column names, 0 BYTES, so the records are not positional
|  |   |   +-------- envelope name, 0 BYTES, so the document carries its own
|  |   +------------ item type doc: the whole answer is one document
|  +---------------- kind top: the head, always the first line
+------------------- correlation id 5, the digits a space closes

#5 row 26:{"running":true,"peers":1}
|  |   |  |
|  |   |  +----- those 26 bytes, the handler's value byte for byte
|  |   +-------- 26, the payload's BYTE count, then the colon every text carries
|  +------------ kind row: one record the command produced
+--------------- correlation id 5

#5 end 1 0 0:
|  |   | | |
|  |   | | +----- message, 0 BYTES, so the command stated none
|  |   | +------- 0 rows rejected
|  |   +--------- 1 record produced
|  +------------- kind end: the terminator, always the last line
+---------------- correlation id 5

A handler that answered with a plugin.Records walk of more than 256 rows writes one line for each row. A walk over a large table therefore never becomes one 16 MB line. A shorter walk collapses to the doc document above. Only the head differs from the three lines above, and it differs in two fields:

#5 top map 5:peers 0:
|  |   |   | |     |
|  |   |   | |     +----- column names, 0 BYTES, so the records are not positional
|  |   |   | +----------- those 5 bytes: peers, the key the records go under
|  |   |   +------------- 5, the envelope name's BYTE count, then its colon
|  |   +----------------- item type map: each record is one map of names to values
|  +--------------------- kind top: the head
+------------------------ correlation id 5

#5 row 44:{"address":"10.0.0.1","state":"established"}
|  |   |  |
|  |   |  +----- those 44 bytes, one row of the walk byte for byte
|  |   +-------- 44, the payload's BYTE count, then its colon
|  +------------ kind row: one record the walk produced
+--------------- correlation id 5

#5 end 1 0 0:

The terminator reads exactly as the one above it: one record produced, no row rejected, and no message.

Inter-Plugin Communication

Plugins can dispatch commands to other plugins via the external-compatible string API:

#4 ze-plugin-engine:dispatch-command {"command":"rib adjacent inbound show"}

Internal plugins that already know the exact registered command should use the typed args API. It carries the command name, pre-tokenized arguments, and optional peer selector separately, so runtime values are not split by the command tokenizer.

#4 ze-plugin-engine:dispatch-command-args {"command":"request bgp adj-rib-in replay","args":["peer key with spaces","0"],"peer":"*"}

Both APIs answer with a head, records, and a terminator. See ipc_protocol.md, "Answer Protocol".

A caller that reads that answer as one value gets DispatchCommandOutput. Its data field carries raw JSON (single-decode) and holds the document a bounded answer puts in its one record. On error the value uses a separate error field: {"status":"error","error":"message"}. The in-process DirectBridge carries that value directly, because it has no line to put a record on.

The string API routes by normal dispatcher parsing and remains the compatibility surface for CLI and external plugin callers. The typed args API routes by exact registered plugin command and sends the existing execute-command callback to the target plugin with args []string.

Config Reload Protocol

Config reload uses a two-phase verify/apply pattern:

Phase 1: Verify -- engine sends candidate config to all plugins for validation:

#10 ze-plugin-callback:config-verify {"sections":[{"root":"bgp","data":"{...}"}]}
#10 ok {"status":"ok"}

If any plugin rejects, the reload is aborted.

Phase 2: Apply -- engine sends config diffs to all plugins:

#11 ze-plugin-callback:config-apply {"sections":[{"root":"bgp","added":"{...}","removed":"","changed":""}]}
#11 ok {"status":"ok"}

ConfigDiffSection fields:

Field Type Description
root string Config root name
added string JSON-encoded added config
removed string JSON-encoded removed config
changed string JSON-encoded changed config

OPEN Validation

Plugins can validate incoming OPEN messages by registering OnValidateOpen. The engine sends both local and remote OPENs for inspection:

#7 ze-plugin-callback:validate-open {"peer":"192.168.1.1","local":{"asn":65001,"router-id":"1.1.1.1","hold-time":90,"capabilities":[...]},"remote":{"asn":65002,...}}
#7 ok {"accept":true}

peer is the peer's configured name. group is its enclosing group, and the field is omitted for a peer that stands alone:

#8 ze-plugin-callback:validate-open {"peer":"dyn-192.0.2.7","group":"ix","local":{...},"remote":{...}}

A peer created from a dynamic group's template has only that second identity. The engine builds such a peer when a connection arrives inside the group's range, and names it dyn-<addr>. Neither that name nor its address is in the config the plugin read. A plugin that keys per-peer policy on the config resolves that peer through group or resolves nothing for it.

To reject:

#7 ok {"accept":false,"notify-code":2,"notify-subcode":6,"reason":"unacceptable hold time"}

Plugin Transport

Internal Plugins (goroutine)

Transport: net.Pipe() for 5-stage startup, then DirectBridge for hot path. No network, no TLS, no auth. Fastest path.

External Plugins (TLS connect-back)

Transport: single TLS connection per plugin.

The child is started as /bin/sh -c <run>, because the run string is written for a shell and the engine appends no argv to it. The shell is therefore a runtime dependency of every external plugin. On a host that carries none, such as a gokrazy appliance image, the start names the absent shell rather than the plugin, and ze doctor reports the same fact before the daemon runs, under doctor-plugin-shell-missing. The child is started in a process group of its own, and the daemon's stop signals that whole group. The shell is the daemon's direct child, and a shell that does not exec-optimize the run string stays alive with the plugin as its own child, so a stop aimed at the direct child leaves the plugin running with nothing to talk to. The declaration query stops its child the same way, through the same function, so a plugin started two ways is stopped one way. On Linux the fork also carries Pdeathsig, which the kernel delivers to that direct child when ze dies. It reaches the plugin for a run string the shell exec-optimizes, and a run string the shell keeps a process for leaves the plugin a grandchild, which no death signal reaches.

  1. Engine reads plugin { hub { server <name> { ip ...; port ...; secret ...; } } } from config
  2. Engine starts TLS listener(s) (one per server entry), creates PluginAcceptor holding the certificate authority root that issued the served certificate
  3. Engine generates per-plugin token, forks child with ZE_PLUGIN_HUB_HOST, ZE_PLUGIN_HUB_PORT, ZE_PLUGIN_HUB_TOKEN (unique per plugin), ZE_PLUGIN_CA_PEM, ZE_PLUGIN_NAME env vars
  4. Child validates the engine chain against that root, connects via TLS, sends #0 auth {"token":"...","name":"..."}
  5. Engine authenticates: per-plugin token lookup by name (constant-time comparison), name binding enforced
  6. Token cleared from child OS environment after first read
  7. Single MuxConn handles bidirectional RPC (responses by #<id>, requests via Requests() channel)
  8. Standard 5-stage handshake proceeds over the same connection

Config

Key Type Default Description
plugin.hub.server named list -- TLS listener entries (keyed by name)
plugin.hub.server.<name>.ip string 127.0.0.1 Bind address
plugin.hub.server.<name>.port uint16 12700 Bind port
plugin.hub.server.<name>.secret string (required, min 32 chars) Auth token

Files

File Purpose
internal/component/plugin/ipc/tls.go TLS listener, auth, cert gen, PluginAcceptor
internal/component/plugin/ipc/rpc.go PluginConn with MuxConn
internal/component/plugin/process/process.go startInternal, startExternal, InitConns
pkg/plugin/sdk/sdk.go NewFromTLSEnv, NewWithConn

Plugin Examples

Capability-Only Plugin (e.g., GR)

The GR plugin only participates in startup -- it injects GR capabilities into OPEN messages. No event subscription needed because it doesn't need runtime events.

Ze Engine                              GR Plugin
----------                             ---------

STAGE 1: REGISTRATION
                       <--- #1 ze-plugin-engine:declare-registration
                            {"wants-config":["bgp"]}
#1 ok                  --->

STAGE 2: CONFIG DELIVERY
#1 ze-plugin-callback:configure
  {"sections":[{"root":"bgp","data":"{...}"}]}  --->
                       <--- #1 ok

STAGE 3: CAPABILITY DECLARATION
                       <--- #2 ze-plugin-engine:declare-capabilities
                            {"capabilities":[
                              {"code":64,"encoding":"hex","payload":"0078",
                               "peers":["192.168.1.1"]},
                              {"code":64,"encoding":"hex","payload":"005a",
                               "peers":["10.0.0.1"]}
                            ]}
#2 ok                  --->

STAGE 4: REGISTRY SHARING
#2 ze-plugin-callback:share-registry {"commands":[...]}  --->
                       <--- #2 ok

STAGE 5: READY
                       <--- #3 ze-plugin-engine:ready
#3 ok                  --->

=== BGP PEERS START - GR capability included in OPEN ===

RUNTIME: (waits for bye)
#99 ze-plugin-callback:bye {"reason":"shutdown"}  --->
                       <--- #99 ok
(plugin exits)

Capability hex format: Code 64 = Graceful Restart (RFC 4724). 0078 = restart-time 120 (0x78 = 120). 005a = restart-time 90.

Event-Driven Plugin (e.g., RIB)

The RIB plugin tracks routes and replays them on peer reconnect. Requires event subscription for runtime events.

Ze Engine                              RIB Plugin
----------                             ----------

STAGE 1: REGISTRATION
                       <--- #1 ze-plugin-engine:declare-registration
                            {"commands":[
                              {"name":"rib adjacent status"},
                              {"name":"rib adjacent inbound show"},
                              {"name":"rib adjacent outbound resend"}
                            ]}
#1 ok                  --->

STAGE 2: CONFIG DELIVERY
#1 ze-plugin-callback:configure {"sections":[]}  --->
                       <--- #1 ok

STAGE 3: CAPABILITY DECLARATION
                       <--- #2 ze-plugin-engine:declare-capabilities
                            {"capabilities":[]}
#2 ok                  --->

STAGE 4: REGISTRY SHARING
#2 ze-plugin-callback:share-registry {"commands":[...]}  --->
                       <--- #2 ok

STAGE 5: READY (with startup subscription)
                       <--- #3 ze-plugin-engine:ready
                            {"subscribe":{"events":["update","state","sent"],
                                          "peers":["*"],"format":"json"}}
#3 ok                  --->

=== BGP PEERS START ===

RUNTIME: Peer comes up
#42 ze-plugin-callback:deliver-batch
  {"events":["{\"type\":\"state\",\"peer\":\"192.168.1.1\",\"state\":\"up\"}"]}  --->
                       <--- #42 ok

RUNTIME: Route sent to peer
#43 ze-plugin-callback:deliver-batch
  {"events":["{\"type\":\"sent\",\"peer\":\"192.168.1.1\",...}"]}  --->
                       <--- #43 ok

RUNTIME: Command request
#44 ze-plugin-callback:execute-command
  {"serial":"abc","command":"rib adjacent status","args":[],"peer":"*"}  --->
                       <--- #44 top doc 0: 0:        head: doc, no envelope, no columns
                       <--- #44 row 26:{"running":true,"peers":1}   26 BYTES of payload
                       <--- #44 end 1 0 0:           1 produced, 0 rejected, no message

RUNTIME: Plugin sends route update to engine
                       <--- #4 ze-plugin-engine:update-route
                            {"peer-selector":"192.168.1.1",
                             "command":"update text nhop set 10.0.0.1 nlri ipv4/unicast add 10.0.1.0/24"}
#4 ok {"peers-affected":1,"routes-sent":1}  --->

SHUTDOWN
#99 ze-plugin-callback:bye {"reason":"shutdown"}  --->
                       <--- #99 ok
(plugin exits)

Capability Decode API

Plugins can provide capability decoding for ze bgp decode --plugin <name>.

This is a standalone mode separate from the 5-stage startup protocol.

Usage

# Decode OPEN message with plugin-provided capability decoding
ze bgp decode --plugin bgp-hostname --open FFFF...

Without plugin, unknown capabilities show raw hex:

{"code": 73, "name": "unknown", "raw": "0C6D792D686F73742D6E616D65..."}

With plugin, capabilities are decoded:

{"name": "fqdn", "hostname": "my-host-name", "domain": "my-domain-name.com"}

Protocol

Plugin is spawned with --decode flag and communicates via stdin/stdout.

Request Formats

Request Description
decode capability <code> <hex> JSON output (default)
decode json capability <code> <hex> JSON output (explicit)
decode text capability <code> <hex> Human-readable text output
decode nlri <family> <hex> JSON output (default)
decode json nlri <family> <hex> JSON output (explicit)
decode text nlri <family> <hex> Human-readable text output

Response Formats

Response Description
decoded json <json> JSON-formatted result
decoded text <text> Human-readable single-line text
decoded unknown Plugin cannot decode this input

Examples

Capability decode (JSON):

Direction Message
ze to plugin decode json capability 73 0C6D792D686F7374...
plugin to ze decoded json {"name":"fqdn","hostname":"my-host","domain":"dom.com"}

Capability decode (text):

Direction Message
ze to plugin decode text capability 73 0C6D792D686F7374...
plugin to ze decoded text fqdn my-host.dom.com

NLRI decode (text):

Direction Message
ze to plugin decode text nlri ipv4/flow 0501180a0000
plugin to ze decoded text destination 10.0.0.0/24

If plugin cannot decode:

Direction Message
plugin to ze decoded unknown

Plugin Implementation

Plugin entry point with --decode flag:

ze plugin bgp-hostname --decode

Plugin reads decode requests from stdin, writes responses to stdout, exits on EOF.

Files

File Purpose
internal/component/bgp/cli/decode_plugin.go Invokes plugin decode API
internal/component/bgp/cli/cmd_plugin.go ze plugin <name> --decode entrypoint
internal/component/bgp/plugins/hostname/hostname.go RunDecodeMode() - hostname capability
internal/component/bgp/plugins/nlri/flowspec/plugin.go RunFlowSpecDecode() - FlowSpec NLRI

Last Updated: 2026-03-22