Structured and Centralised Logging
🌟 Objective
Ensure that all applications record events in a structured, secure, traceable and auditable manner, enabling anomaly detection, correlation between systems and support for incident response.
🧬 What structured logging is
Structured logging consists of the emission of events with normalised fields, in a machine-readable format (e.g. JSON), transported to a centralised system and with integrity and retention guarantees.
📌 The structure and consistency of logs are essential for detection, traceability and effective investigation.
🧱 Recommended event structure
Adopt a convention such as the Elastic Common Schema (ECS) or an in-house JSON-based format with common minimum fields.
| Field | Example | Notes |
|---|---|---|
timestamp | 2025-07-21T10:35:14Z | ISO 8601 in UTC |
level | INFO, ERROR, WARN, DEBUG | Standardised levels |
event.action | user.login, file.upload, api.call | Consistent taxonomy |
user.id | abcd-1234 | Internal user ID, if applicable |
src.ip | 192.168.1.20 | Source IP |
trace.id | xyz-9876 | Support for distributed tracing |
application | frontend-web, api-gateway, worker-job | Logical component of the application |
📦 Log transport and centralisation
- Use forwarding agents (e.g. Filebeat, Fluentbit, Vector)
- Ensure secure transport (TLS) and temporary persistence (buffers)
- Prefer continuous streaming (e.g. TCP, gRPC) over batch or UDP
- Separate logging from the main process (e.g. sidecar container, logging driver)
Typical destinations:
| Category | Examples |
|---|---|
| SIEMs | Splunk, Sentinel, QRadar |
| Observability | Elastic Stack, Loki, Datadog |
| Hybrid | OpenSearch, Graylog, Grafana Tempo |
🔒 Log security and integrity
| Control | Description |
|---|---|
| Protected retention (WORM) | Prevents alteration of logs after writing |
| Restricted access | Writing only via the application; reading audited |
| Signature or hash | Integrity verification per batch or file |
| Function isolation | Logging separated from the application (e.g. forwarder, sidecar, service) |
⚠️ Local-only logs are acceptable only for L1 applications - with periodic collection and documented justification.
📌 Minimum mandatory events
| Event type | Must contain… |
|---|---|
| Failed authentication | IP, user, timestamp, reason |
| Sensitive changes | Who, what, where, before and after |
| Internal exceptions | Stack trace (summarised), module, request ID |
| Upload / download | Name, size, user, method |
| External API call | Endpoint, status, response time |
✅ Good practices
- Do not record sensitive data (e.g. passwords, tokens), even encrypted
- Separate logs by functional context (e.g.
app.log,auth.log,db.log) - Validate the presence and format of logs in automated tests
- Include
request.idandtrace.idfor traceability of distributed requests - Emit logs at every security state transition
🧩 Link with detection and response
Structured logging serves as the basis for the detection mechanisms defined in other documents:
| Document | Relation to this topic |
|---|---|
03-alertas-eventos-criticos.md | Generation of alerts from events |
06-correlacao-anomalias.md | Correlation between events for detection |
08-matriz-controles-por-risco.md | Logging requirements per level L1–L3 |
🔐 A well-structured and centralised logging system is a prior requirement for effective detection, incident response, and compliance with frameworks such as SSDF, SLSA, ISO 27001.