Author

Richard Malcolm

Date

February 25, 2025

Splunk: Learning SIEM and Incident Investigation

Recently, I had the opportunity to explore Splunk, a widely used Security Information and Event Management (SIEM) tool, through a hands-on security investigation. My objective was to analyze security logs, track attacker activity, and identify security threats within an e-commerce network.
  • Understand how SIEM tools collect, index, and analyze log data
  • Use Splunk's Search Processing Language (SPL) to perform investigations
  • Apply the Cyber Kill Chain to track an attacker's actions
  • Identify failed SSH login attempts and potential brute-force attacks

Understanding Incident Handling & Splunk’s Role

An incident in cybersecurity is any unauthorized access, malware infection, or system compromise. Organizations rely on SIEM tools like Splunk to detect, investigate, and respond to security incidents.

Using Splunk, I followed the Incident Handling Lifecycle, which includes:

  • Preparation – Setting up SIEM tools, defining policies, and monitoring logs
  • Detection & Analysis – Identifying suspicious activity and analyzing logs
  • Containment & Eradication – Isolating and eliminating threats
  • Post-Incident Review – Learning from incidents to improve security

Step-by-Step Investigation Using Splunk

1. Uploading and Searching Data in Splunk

Once I set up Splunk Cloud, I uploaded security logs from an e-commerce network and ran my first search to confirm the data was indexed:

index=main

This returned over 100,000 events, proving that Splunk successfully ingested the log data.

2. Identifying Failed SSH Login Attempts

To investigate failed SSH login attempts on the mail server (mailsv), I narrowed my search:

index=main host=mailsv fail* root

This search revealed over 100 failed SSH login attempts on the root account—potentially indicating a brute-force attack.

3. Extracting Attack Details Using Advanced Queries

Digging deeper, I analyzed HTTP traffic logs to identify login attempts on a Joomla-based admin panel:

index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70″ http_method=POST uri=”/joomla/administrator/index.php”

The logs showed a brute-force attack originating from IP 23.22.63.114, using automated login attempts.

4. Investigating Malicious File Uploads

To check if any malicious files were uploaded, I searched for executable files (.exe) sent to the server:

index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70″ *.exe

This led to the discovery of a suspicious file named 3791.exe, which was later executed on the compromised server.

5. Mapping the Attack to the Cyber Kill Chain

Using the Cyber Kill Chain framework, I mapped how the attacker operated:

  • Reconnaissance – IP 40.80.148.42 was scanning the network using Acunetix, a known vulnerability scanner.
  • Exploitation – A brute-force attack from 23.22.63.114 successfully accessed the admin panel.
  • Installation – The attacker uploaded 3791.exe, potentially a backdoor.
  • Command & Control – The compromised server initiated outbound traffic to malicious IPs.
  • Actions on Objectives – The attacker defaced the e-commerce website using an image file.
  • Weaponization – Found domains that look pretty similar to the WAYNE Enterprise site
  • Delivery – Found the virus details MirandaTateScreensaver.scr.exe - c99131e0169171935c5ac32615ed6261

Key Takeaways & Lessons Learned

  • SIEM tools like Splunk are essential for real-world security investigations. Without log analysis, detecting security incidents would be nearly impossible.
  • Search queries can make or break an investigation. By refining searches, using wildcards, regex, and field extractions, I was able to quickly identify attack patterns.
  • Cybersecurity is proactive. Understanding attack behaviors (e.g., brute-force attempts, lateral movement, and data exfiltration) allows analysts to anticipate threats before they escalate.

Final Thoughts: Why This Experience Was Valuable

This hands-on exercise transformed how I see cybersecurity. It wasn’t just about learning Splunk—it was about thinking like an analyst, tracking adversaries, and uncovering hidden threats within vast amounts of data.

I look forward to expanding my skills by exploring threat hunting, anomaly detection, and advanced SIEM configurations

Bonus: Additional Splunk Queries for Incident Response

  • Identify brute-force attempts on admin login pages

index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70″ http_method=POST uri=”/joomla/administrator/index.php” | table _time uri src_ip dest_ip form_data

  • Extract passwords used in a brute-force attack

index=botsv1 sourcetype=stream:http dest_ip=”192.168.250.70″ http_method=POST form_data=*username*passwd* | rex field=form_data “passwd=(?<creds>\w+)” | table src_ip creds

  • Detect outbound connections to attacker-controlled infrastructure

index=botsv1 src=192.168.250.70 sourcetype=suricata dest_ip=23.22.63.114

  • Find execution of suspicious files on a compromised server

index=botsv1 “3791.exe” sourcetype=”XmlWinEventLog” EventCode=1

Copyright © 2024 RICHARD MALCOLM