SIEM, Log Analysis and SOC Operations for Beginners
When something goes wrong in an organisation, the first question is always “what happened?” The answer is in the logs, if they exist, if they were kept and if someone can search them. A SIEM (Security Information and Event Management) system collects logs from across the organisation, normalises and correlates them and raises alerts. A SOC (Security Operations Centre) is the team and process that watches those alerts and responds. This lesson explains the full chain from log to alert to investigation, and takes you through a lab with free tools.
What Is a Log, and Why Does It Matter?
A log is a time-stamped record of an event: a login, a file access, a firewall decision, a web request, a process starting. Individually they are plain text; together they tell a story. Typical sources:
| Source | Examples of useful events |
|---|---|
| Operating systems | Logons, account changes, process creation, service installs (Windows Security log, Linux auth log, journald) |
| Network devices | Firewall allows and denies, VPN sessions, DHCP and DNS queries |
| Web and application servers | Requests, errors, authentication events |
| Identity systems | Directory sign-ins, MFA outcomes, privilege changes |
| Cloud platforms | API calls, configuration changes, access to storage (for example, audit trails) |
| Security tools | EDR, IDS alerts, email gateway, vulnerability scanners |
Logging fundamentals
- Synchronise time everywhere (NTP). Correlation is impossible when clocks disagree.
- Centralise logs away from the machines producing them, so attackers cannot simply delete them.
- Protect integrity with restricted write access and, where needed, hashing or write-once storage.
- Retain long enough. Many intrusions are discovered months after they began; retention rules should reflect legal and business needs (for example, CERT-In directions in India require certain logs to be maintained for 180 days within Indian jurisdiction).
- Do not log secrets such as passwords and full card numbers.
How a SIEM Works
- Collect: agents, syslog, APIs and connectors forward events.
- Parse and normalise: fields such as source IP, user and action are extracted into a common format.
- Enrich: add context, such as the asset’s owner, geolocation, threat intelligence match or user department.
- Correlate and detect: rules and analytics look for patterns across events and time.
- Alert and triage: analysts review prioritised alerts.
- Investigate and respond: search, pivot, contain and document; see Incident Response Process and Playbooks.
- Report and improve: measure, tune rules and close gaps.
Popular platforms include Splunk, Microsoft Sentinel, IBM QRadar, Elastic Security, Wazuh and Graylog. Open-source stacks such as Elastic, Wazuh and Graylog let you practise for free.
Writing Detections: Examples
| Detection idea | Logic | Why it matters |
|---|---|---|
| Brute-force attempt | 10 or more failed logons for one account from one source in 5 minutes | Password guessing |
| Password spraying | One source fails logon against many accounts, few tries each | Avoids per-account lockout |
| Success after many failures | A successful logon follows a burst of failures | Possible compromised account |
| Impossible travel | Same user signs in from two distant countries within a short time | Stolen credentials |
| New admin created | Account added to privileged group | Persistence or privilege escalation |
| Log cleared | Windows event 1102 or 104, or deleted Linux auth logs | Attackers hide their tracks |
| Unusual process chain | An office application starting a command interpreter | Common malware execution pattern |
| Large outbound transfer | Unusual volume to a new external destination | Possible data theft |
Good detections are documented, mapped to MITRE ATT&CK techniques, tested and reviewed regularly for false positives.
Practice Lab 1: Read Linux Authentication Logs
# Failed SSH logins (Debian/Ubuntu path; use journalctl on other systems)
sudo grep "Failed password" /var/log/auth.log | tail
# Count failures per source IP
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head
# Successful logins
sudo grep "Accepted" /var/log/auth.log | tail
# journald alternative
sudo journalctl -u ssh --since "1 hour ago" | grep -i "failed"
From your analyst VM, make three deliberate wrong password attempts against your lab target, then find them in the logs. Write the timestamp, source and account for each.
Practice Lab 2: Build a Mini-SIEM With Wazuh or Elastic
- Install Wazuh (an open-source security platform) following its official quickstart on a monitoring VM with at least 4 GB of RAM, or use the Elastic stack with Filebeat.
- Install the agent on one or two lab VMs and confirm events arrive in the dashboard.
- Generate test events: failed logins, a new user account, a modified file under a monitored directory.
- Find each event in the dashboard. Note which built-in rules fired and their severity levels.
- Write one custom rule that alerts on the creation of a new user, then trigger it again.
Practice Lab 3: Simple Log Analysis Script
Learn what a SIEM does under the hood by writing a small detector in Python:
import re, collections, datetime
pattern = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
failures = collections.defaultdict(list)
with open("/var/log/auth.log") as f:
for line in f:
m = pattern.search(line)
if m:
user, ip = m.groups()
failures[ip].append(user)
for ip, users in failures.items():
if len(users) >= 5:
distinct = len(set(users))
kind = "possible spraying" if distinct >= 3 else "possible brute force"
print(f"ALERT {kind}: {ip} made {len(users)} failed attempts against {distinct} account(s)")
Run it against your lab log after generating test failures. Then extend it to use time windows and to ignore your own analyst IP.
How a SOC Works
| Role or tier | Typical duties |
|---|---|
| Tier 1 analyst | Monitor alerts, triage, close false positives, escalate real incidents |
| Tier 2 analyst | Deeper investigation, containment guidance, threat research |
| Tier 3 / threat hunter | Proactive hunting, detection engineering, forensics support |
| SOC manager | Staffing, metrics, processes, stakeholder reporting |
| Detection engineer | Builds and tunes detection rules and integrations |
| Incident commander | Leads response for major incidents |
Alert triage method
- What fired? Read the rule, the data and the severity.
- Is it real? Check context: is the user travelling? Is the host a scanner? Is it a known test?
- How far does it go? Search for related events: same user, host, IP and time window.
- What is the impact? Which asset and data are involved?
- Decide: close as benign, escalate, or contain.
- Document the reasoning and evidence in the ticket.
SOC Metrics
- Mean time to detect (MTTD) and mean time to respond (MTTR).
- False positive rate per rule.
- Alert volume and the share closed without action.
- Coverage: percentage of important log sources onboarded and ATT&CK techniques covered by detections.
Common Mistakes
- Collecting everything but monitoring nothing.
- Flooding analysts with low-quality alerts, which leads to alert fatigue.
- Missing key sources such as identity and cloud audit logs.
- Letting clocks drift, which breaks correlation.
- Having no tested playbooks, so each alert is handled differently.
Practice Exercises
- List ten log sources you would onboard first for a 100-person company and explain your priorities.
- Write the logic (in plain English) for detecting password spraying.
- Explain why a failed login followed by a success from the same source is more interesting than either event alone.
- Draft an alert triage checklist for a Tier 1 analyst.
- Choose one MITRE ATT&CK technique and describe one log source and one detection that would reveal it.
Mentor tip: A detection nobody tested is a hope, not a control. For every rule you write, generate the event yourself in the lab and confirm the alert fires, then record how.
Educational use only: collect and analyse logs only on systems you own or are authorised to monitor, and respect privacy law.
- Logs are the evidence trail; centralise, time-synchronise, protect and retain them.
- A SIEM collects, normalises, enriches, correlates and alerts; a SOC acts on the alerts.
- Good detections are tested, documented and mapped to attacker techniques.
- Tune relentlessly to reduce false positives and alert fatigue.
- Practise with free tools such as Wazuh and Elastic in your lab.