Evidence, Reproducibility and Auditability in Security Testing

This annex defines prescriptive rules to ensure that security tests produce verifiable evidence, allow independent reproduction and support audit - even when execution is highly automated.

The aim is to avoid three recurring failures in modern testing processes:

1. Confusing “signal” with “evidence” (e.g. a report or a score replaces observable execution).
2. Losing reproducibility (e.g. the same commit produces incomparable results because of variation in rules, environment or data).
3. Exposing critical assets (e.g. credentials, data, telemetry or build artefacts leave the perimeter without explicit control).

Master rule: a test only “counts” as evidence when it can be verified and reproduced, with authorship and decision attributed to a human role.

---

1. Separation between automated signal and human decision

Tools may:

• detect patterns;
• correlate results;
• prioritise findings;
• suggest fixes.

But they may not:

• declare “approved” or “compliant”;
• close findings without defined validation;
• authorise exceptions;
• accept residual risk.

Prescription

• Each security gate (pass/fail) must have an explicit human owner (e.g. Tech Lead, AppSec, QA Lead, Release Manager).
• “It passed the scanner” is not an accepted argument without:
  • evidence of execution (logs/artefacts),
  • execution conditions (config/versions),
  • and approval criteria (policy/rules).

---

2. Minimum evidence (what must exist for a result to be accepted)

2.1 Evidence of execution (mandatory)

For each relevant execution of security tests, there must exist, at a minimum:

• Identifier of the commit / build / artefact tested (hash, tag, digest).
• Timestamp and identified runner / executor.
• Effective configuration used (file, parameters, profile).
• Version of the test engine and dependent versions (e.g. images, rulepacks, signatures).
• Raw output preserved (full log, JSON/XML/HTML, artefacts).
• Summarised result (pass/fail, thresholds, severities) derived from the raw output.

Dashboards and PDFs are merely “presentation”. The evidence is the raw output + execution context.

2.2 Evidence of validation (where applicable)

Whenever there is triage, exception, suppression or “compensation”:

• Record of who validated and when (author/role).
• Objective justification (criterion + reference).
• Validity condition (TTL, next retest date, closure condition).
• Link to the backlog item / ticket / release decision.

---

3. Reproducibility: making results comparable over time

3.1 Control of variables (mandatory)

Test results are only comparable if the critical variables are controlled:

• Engine and rules version: rulepacks, signatures, policies.
• Execution environment: runner image, operating system, dependencies.
• Target tested: immutable build/digest/artefact.
• Configuration: profiles, exclusion lists, limits, thresholds.
• Test data and credentials: dataset and permissions (see Section 4).

3.2 Practical prescriptions

• Use version pinning (engine + rules) per pipeline and per project.
• Version test profiles and policies (in the repo).
• Preserve raw outputs with a defined minimum retention (e.g. per release).
• Ensure that each execution can be re-run with the “same input” (replay).

3.3 Intrinsically non-deterministic tests

Some tests may produce variation (e.g. fuzzing, DAST with timing, distributed targets).

In those cases:

• Preserve the seed and execution parameters when they exist.
• Repeat the execution (minimum N) to reduce false negatives.
• Treat the absence of a finding as a weak signal without additional coverage evidence.

---

4. Protection of critical assets in the testing process

Security tests frequently involve:

• credentials and tokens for authenticated tests;
• representative (or accidentally real) data;
• telemetry, logs and traffic containing secrets;
• build artefacts and code.

4.1 Mandatory prescriptions

• Real data: prohibited by default in tests (except with explicit approval and compensating controls).
• Test credentials:
  • must be segregated by environment,
  • minimal (least privilege),
  • with rotation and expiry.
• Egress and telemetry:
  • any sending of outputs outside the perimeter must be treated as a supply chain dependency;
  • require explicit approval, data classification and contractual/technical control.
• Logs:
  • apply masking rules;
  • ensure that preserved outputs do not expose secrets.

4.2 Test environments and targets

• DAST/IAST/fuzzing must use environments with:
  • network isolation,
  • impact limitation (rate limiting, quotas),
  • monitoring of changes and traffic.
• Offensive and intrusive tests must have:
  • an approved window,
  • an operational “kill switch”,
  • stop criteria.

---

5. Minimum evidence by type of test (operational table)

Type of test	Minimum evidence of execution	Minimum reproducibility	Critical note
SAST	commit + config + engine/rules version + raw output	rule pinning + comparable baseline	“0 findings” without a baseline is a weak signal
DAST	target + window + auth profile + output + execution logs	versioned config + scope policy	high risk of exfiltration and false negatives
IAST	build + instrumentation + config + logs + findings	instrumentation version + controlled environment	outputs may contain sensitive data
Fuzzing	target + parameters + seed (if applicable) + crashes + logs	seed + execution time + versioned corpus	absence of a crash ≠ absence of a bug
Pentesting / manual	scope + methodology + evidence (PoC) + report	execution artefacts and notes preserved	requires tracking of decision and fix

---

6. Exceptions, suppressions and risk acceptance (closing the cycle)

Suppressions, “false positives”, exceptions and risk acceptance are governance facts, not technical “opinions”.

Prescription

• No exception is valid without:
  • a human owner,
  • an objective justification,
  • a deadline (TTL),
  • a retest plan,
  • a link to the affected release/artefact.

---

7. Acceptance criteria for this annex

This annex is considered applied when, in the project:

1. Raw evidence is preserved for relevant executions.
2. Test profiles and policies are versioned and controlled.
3. Replay of an execution is possible (or the variation is controlled by repetition).
4. Credentials/data/telemetry are treated as critical assets with explicit controls.
5. Exception and “pass/fail” decisions have a human owner and traceability.

---