SIEM, Log Analysis and SOC Operations for Beginners

When something goes wrong in an organisation, the first question is always “what happened?” The answer is in the logs, if they exist, if they were kept and if someone can search them. A SIEM (Security Information and Event Management) system collects logs from across the organisation, normalises and correlates them and raises alerts. A SOC (Security Operations Centre) is the team and process that watches those alerts and responds. This lesson explains the full chain from log to alert to investigation, and takes you through a lab with free tools.

What Is a Log, and Why Does It Matter?

A log is a time-stamped record of an event: a login, a file access, a firewall decision, a web request, a process starting. Individually they are plain text; together they tell a story. Typical sources:

SourceExamples of useful events
Operating systemsLogons, account changes, process creation, service installs (Windows Security log, Linux auth log, journald)
Network devicesFirewall allows and denies, VPN sessions, DHCP and DNS queries
Web and application serversRequests, errors, authentication events
Identity systemsDirectory sign-ins, MFA outcomes, privilege changes
Cloud platformsAPI calls, configuration changes, access to storage (for example, audit trails)
Security toolsEDR, IDS alerts, email gateway, vulnerability scanners

Logging fundamentals

  • Synchronise time everywhere (NTP). Correlation is impossible when clocks disagree.
  • Centralise logs away from the machines producing them, so attackers cannot simply delete them.
  • Protect integrity with restricted write access and, where needed, hashing or write-once storage.
  • Retain long enough. Many intrusions are discovered months after they began; retention rules should reflect legal and business needs (for example, CERT-In directions in India require certain logs to be maintained for 180 days within Indian jurisdiction).
  • Do not log secrets such as passwords and full card numbers.

How a SIEM Works

  1. Collect: agents, syslog, APIs and connectors forward events.
  2. Parse and normalise: fields such as source IP, user and action are extracted into a common format.
  3. Enrich: add context, such as the asset’s owner, geolocation, threat intelligence match or user department.
  4. Correlate and detect: rules and analytics look for patterns across events and time.
  5. Alert and triage: analysts review prioritised alerts.
  6. Investigate and respond: search, pivot, contain and document; see Incident Response Process and Playbooks.
  7. Report and improve: measure, tune rules and close gaps.

Popular platforms include Splunk, Microsoft Sentinel, IBM QRadar, Elastic Security, Wazuh and Graylog. Open-source stacks such as Elastic, Wazuh and Graylog let you practise for free.

Writing Detections: Examples

Detection ideaLogicWhy it matters
Brute-force attempt10 or more failed logons for one account from one source in 5 minutesPassword guessing
Password sprayingOne source fails logon against many accounts, few tries eachAvoids per-account lockout
Success after many failuresA successful logon follows a burst of failuresPossible compromised account
Impossible travelSame user signs in from two distant countries within a short timeStolen credentials
New admin createdAccount added to privileged groupPersistence or privilege escalation
Log clearedWindows event 1102 or 104, or deleted Linux auth logsAttackers hide their tracks
Unusual process chainAn office application starting a command interpreterCommon malware execution pattern
Large outbound transferUnusual volume to a new external destinationPossible data theft

Good detections are documented, mapped to MITRE ATT&CK techniques, tested and reviewed regularly for false positives.

Practice Lab 1: Read Linux Authentication Logs

# Failed SSH logins (Debian/Ubuntu path; use journalctl on other systems)
sudo grep "Failed password" /var/log/auth.log | tail

# Count failures per source IP
sudo grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr | head

# Successful logins
sudo grep "Accepted" /var/log/auth.log | tail

# journald alternative
sudo journalctl -u ssh --since "1 hour ago" | grep -i "failed"

From your analyst VM, make three deliberate wrong password attempts against your lab target, then find them in the logs. Write the timestamp, source and account for each.

Practice Lab 2: Build a Mini-SIEM With Wazuh or Elastic

  1. Install Wazuh (an open-source security platform) following its official quickstart on a monitoring VM with at least 4 GB of RAM, or use the Elastic stack with Filebeat.
  2. Install the agent on one or two lab VMs and confirm events arrive in the dashboard.
  3. Generate test events: failed logins, a new user account, a modified file under a monitored directory.
  4. Find each event in the dashboard. Note which built-in rules fired and their severity levels.
  5. Write one custom rule that alerts on the creation of a new user, then trigger it again.

Practice Lab 3: Simple Log Analysis Script

Learn what a SIEM does under the hood by writing a small detector in Python:

import re, collections, datetime

pattern = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\S+)")
failures = collections.defaultdict(list)

with open("/var/log/auth.log") as f:
    for line in f:
        m = pattern.search(line)
        if m:
            user, ip = m.groups()
            failures[ip].append(user)

for ip, users in failures.items():
    if len(users) >= 5:
        distinct = len(set(users))
        kind = "possible spraying" if distinct >= 3 else "possible brute force"
        print(f"ALERT {kind}: {ip} made {len(users)} failed attempts against {distinct} account(s)")

Run it against your lab log after generating test failures. Then extend it to use time windows and to ignore your own analyst IP.

How a SOC Works

Role or tierTypical duties
Tier 1 analystMonitor alerts, triage, close false positives, escalate real incidents
Tier 2 analystDeeper investigation, containment guidance, threat research
Tier 3 / threat hunterProactive hunting, detection engineering, forensics support
SOC managerStaffing, metrics, processes, stakeholder reporting
Detection engineerBuilds and tunes detection rules and integrations
Incident commanderLeads response for major incidents

Alert triage method

  1. What fired? Read the rule, the data and the severity.
  2. Is it real? Check context: is the user travelling? Is the host a scanner? Is it a known test?
  3. How far does it go? Search for related events: same user, host, IP and time window.
  4. What is the impact? Which asset and data are involved?
  5. Decide: close as benign, escalate, or contain.
  6. Document the reasoning and evidence in the ticket.

SOC Metrics

  • Mean time to detect (MTTD) and mean time to respond (MTTR).
  • False positive rate per rule.
  • Alert volume and the share closed without action.
  • Coverage: percentage of important log sources onboarded and ATT&CK techniques covered by detections.

Common Mistakes

  • Collecting everything but monitoring nothing.
  • Flooding analysts with low-quality alerts, which leads to alert fatigue.
  • Missing key sources such as identity and cloud audit logs.
  • Letting clocks drift, which breaks correlation.
  • Having no tested playbooks, so each alert is handled differently.

Practice Exercises

  1. List ten log sources you would onboard first for a 100-person company and explain your priorities.
  2. Write the logic (in plain English) for detecting password spraying.
  3. Explain why a failed login followed by a success from the same source is more interesting than either event alone.
  4. Draft an alert triage checklist for a Tier 1 analyst.
  5. Choose one MITRE ATT&CK technique and describe one log source and one detection that would reveal it.

Mentor tip: A detection nobody tested is a hope, not a control. For every rule you write, generate the event yourself in the lab and confirm the alert fires, then record how.

Educational use only: collect and analyse logs only on systems you own or are authorised to monitor, and respect privacy law.

Key takeaways
  • Logs are the evidence trail; centralise, time-synchronise, protect and retain them.
  • A SIEM collects, normalises, enriches, correlates and alerts; a SOC acts on the alerts.
  • Good detections are tested, documented and mapped to attacker techniques.
  • Tune relentlessly to reduce false positives and alert fatigue.
  • Practise with free tools such as Wazuh and Elastic in your lab.

Related Lessons