Skip to content

docs: add user-facing examples for evidence source provenance #876

Description

@marcusburghardt

Problem

PR #840 added evidence source provenance (EvidenceMapping) support across complyctl's entire pipeline -- proto API, provider SDK, evaluator, and Markdown formatter. The implementation is thorough but all documentation is developer-oriented (CHANGELOG entry, AGENTS.md Recent Changes, OpenSpec artifacts). There are no user-facing or auditor-facing examples explaining what the feature looks like in practice or how to interpret the new output fields.

Users running complyctl scan will encounter new source: blocks in EvaluationLog YAML/JSON and source: <ref> @ <coord> lines in Markdown reports with no documentation explaining what they mean.

What's Missing

  1. No example of the EvaluationLog source: block -- users see a new YAML key in .complytime/scan/evaluation-log-*.yaml when providers populate evidence source, but nothing explains the fields or how to interpret them.

  2. No example of the Markdown report source line -- --format pretty now renders source: <ref> @ <coord> in evidence metadata but this isn't documented.

  3. No explanation of mapping-references -- the metadata.mapping-references block in EvaluationLog is now populated from policy metadata. Auditors have no reference for what these entries are or how they connect to evidence.source.reference-id.

  4. No field-level reference -- the five fields (reference-id, coordinate, entry-id, digest, remarks) are never explained from a user/auditor perspective.

Proposed Documentation

Location 1: docs/QUICK_START.md, Step 7 (Scan)

Add a "Understanding scan output" subsection after line 310 (Output written to ./.complytime/scan/.) with two annotated examples.

RHEL example (OpenSCAP provider checking SSH configuration):

# .complytime/scan/evaluation-log-<policy>-<timestamp>.yaml (excerpt)
evidence:
  - description: "SSH root login is disabled"
    relevant-evidence: "PermitRootLogin no"
    source:
      reference-id: "ref-sshd-config"        # resolves via mapping-references
      coordinate: "/etc/ssh/sshd_config.d/50-hardening.conf:3"  # file:line
      digest: "sha256:e4f8a9..."              # content hash at scan time

Kubernetes example (Ampel provider checking resource limits):

evidence:
  - description: "Resource limits are configured"
    relevant-evidence: "limits.cpu=500m, limits.memory=128Mi"
    source:
      reference-id: "ref-k8s-deployment"
      coordinate: "apps/v1/Deployment/my-app/spec/containers/0/resources"
      digest: "sha256:a1b2c3..."

Markdown report rendering (--format pretty):

source: ref-sshd-config @ /etc/ssh/sshd_config.d/50-hardening.conf:3

Field reference table:

Field Purpose Example
reference-id Links to a mapping-references entry in policy metadata ref-sshd-config
coordinate Location within the artifact (file path, API path, line number) /etc/ssh/sshd_config.d/50-hardening.conf:3
entry-id Gemara entry identifier (alternative to coordinate) entry-001
digest Content hash at scan time for drift detection sha256:e4f8a9...
remarks Free-text provider notes "active configuration"

Include a note: "Not all providers populate source fields. When a provider does not report provenance, the source: block is omitted from the EvaluationLog and the Markdown report shows evidence without a source line."

Location 2: docs/man/complyctl.md

This is optional / lower priority. If the man page gains an OUTPUT section in the future, evidence source field semantics should be included there and QUICK_START.md should link to it. No immediate action needed -- the QUICK_START addition is self-contained.

Implementation Notes

  • Files to modify: docs/QUICK_START.md only (single-file change)
  • Insertion point: After line 310 (Output written to ./.complytime/scan/.), before the exit codes callout block
  • No code changes required -- this is documentation only
  • The YAML examples above are illustrative -- they show what providers can emit. The actual field names match the go-gemara EvidenceMapping struct serialization (kebab-case YAML keys)
  • Rendering code reference: internal/output/markdown.go lines 283-287 (formatEvidenceMeta) and internal/output/evaluator.go (shadow structs with yaml: / json: tags)

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

documentationImprovements or additions to documentationllm_assistedFiled or drafted with LLM assistance

Type

No type

Fields

Priority

High

Effort

Low

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions