Modern SIEM Architecture: From Raw Logs to Detection-Ready Telemetry

A SIEM is not primarily a search interface. It is a data engineering system whose output happens to be security decisions. The quality of those decisions depends on collection, transport, integrity, parsing, normalization, enrichment, retention and detection design.

The common onboarding mistake is to ask whether a product can send logs to the SIEM. The better question is what decision the SOC needs to make with those logs. A firewall feed might support suspicious outbound connections, VPN logs might support impossible-travel investigations, identity events might support MFA abuse, and cloud audit data might support privilege-escalation detection.

That changes the architecture. A source is valuable when its events contain enough context to answer a security question. Volume alone is not a quality metric.

2. A practical telemetry pipeline

Each stage has a different failure mode. Collection can lose events. Transport can reorder or truncate them. Buffers can exhaust disk. Parsers can silently drop fields. Normalization can destroy source-specific meaning. Storage can become too expensive. Detection can produce alerts that nobody can action.

Collection

Prefer the source's supported export mechanism and document protocol, port, authentication, certificate requirements, expected event rate and retry behavior. For syslog, document whether the sender uses UDP, TCP or TLS and whether RFC 3164 or RFC 5424 semantics are expected. For cloud sources, document API permissions, object prefixes, polling intervals and eventual consistency.

Transport and buffering

Separate transport from processing. A temporary parser outage should not automatically become permanent data loss. Where the source and risk justify it, use a durable queue or local disk buffer. Monitor queue depth, oldest-event age and dropped-event counters rather than merely checking that a process is running.

Parsing and normalization

Parsing should preserve the original event while producing stable fields used by detections. A useful normalized model might contain timestamp, source, destination, actor, action, outcome, asset, identity, network context and vendor-specific extensions. Never throw away fields simply because they are not needed by today's rule.

3. Data quality is a security control

A dashboard showing “10,000 events received” can be misleading. The meaningful questions are: what percentage arrived, how late were they, which fields were populated, did the event schema change, and can an investigator reconstruct the sequence of actions?

For high-value sources, define a telemetry service-level objective. For example: 99% of expected events arrive within five minutes; parsing success remains above 98%; mandatory fields have greater than 95% population; and source clock drift stays below a defined threshold.

source_events_expected = 100000
source_events_received = 98500
coverage = received / expected

# Investigate when coverage falls below the agreed SLO,
# not merely when the collector process stops.

4. Detection-ready enrichment

Enrichment should answer questions the raw event cannot. Identity-to-user mapping, asset criticality, business owner, cloud account, geo context, threat-intelligence reputation and vulnerability context can turn an ambiguous event into a useful detection signal. Keep enrichment provenance and timestamps so investigators know what was known at the time of analysis.

5. Retention should follow investigative value

Do not use a single retention policy for every source. Authentication, privileged activity, endpoint process telemetry and cloud control-plane events often deserve longer retention than high-volume network records. Design hot, warm and archive tiers around investigation frequency and legal requirements, while preserving the fields needed to reconstruct incidents.

6. Detection architecture

High-fidelity detections generally combine multiple weak signals. A single failed login is rarely interesting; a burst of failures followed by a successful login from an unusual network and a privileged action may be. The SIEM should support correlation without forcing every raw event into an alert.

Build detections with explicit inputs, logic, threshold, suppression, exceptions, severity, owner and validation evidence. Record why the rule exists and what telemetry loss would make it blind.

7. Cost control without blind spots

Cost optimization should remove redundancy rather than security context. First measure event volume by source and field. Then identify duplicate feeds, verbose debug events, low-value payloads and unnecessary retention. Sampling should be used carefully; indiscriminate sampling can destroy the very sequence needed for investigation.

8. Operational architecture

A production SIEM needs health telemetry of its own: ingestion rate, parser error rate, queue depth, storage utilization, event latency, authentication failures, API quota consumption and source freshness. These should be monitored independently of the data they protect.

What defenders should do now

1. Inventory every source and map it to a security decision.
2. Define a minimum telemetry contract for each high-value use case.
3. Measure expected versus received events, not just process uptime.
4. Preserve raw events while normalizing fields for cross-source detections.
5. Add identity, asset and business context before writing complex correlation.
6. Define retention by investigative value and regulatory requirement.
7. Monitor the SIEM pipeline as a production system with its own SLOs.
8. Review every high-severity detection with real incident or test evidence.

Conclusion

The strongest SIEM architectures are deliberately boring: predictable collection, durable transport, observable parsing, stable schemas, useful enrichment, measurable data quality and detections that have a documented reason to exist. The sophistication belongs in the engineering discipline, not in the number of dashboards.

Research basis: this guide reflects Cylatic's engineering methodology and is intentionally written as an original systems-design resource rather than a rewrite of a vendor article.