Splunk Hands-on Labs¶

Choose the route that fits your available time and computing resources. Use the Interactive Lab for a short guided investigation, or the Full Lab to build and operate the complete Splunk workflow in virtual machines.

Hands-on proof of concept

Build and test your Splunk lab

Build a two-machine lab, confirm that a recognisable event reaches Splunk, investigate failed logins, and save your findings in Splunk.

Start building

Self-hosted

Splunk Full Lab

Isolated server and client VMs required

How this differs from the videos

Use the linked installation guides alongside these exercises. The videos show the original PoC; addresses, software versions, menus, and field names may differ in your environment.

Use an isolated, authorised environment

Generate test traffic only against systems you own or are authorised to assess. Use harmless manual login failures instead of automated brute-force tooling if you are unsure about the scope or safety of your environment.

Before you begin

You need: an x86-64 computer with virtualisation enabled, administrator access, and temporary internet access for downloads. The example VMs below allocate 12 GB of RAM and 140 GB of virtual disk capacity in total. Allow additional memory and storage for the host OS, installation media, and snapshots. These are small-lab allocations, not vendor minimums or production sizing. Check the requirements for your chosen software versions before downloading.

How to use this lab

Complete one session at a time. Stop at a checkpoint if the expected result is missing; resolve the problem before continuing or adding another data source.

Recommended first

Beginner Core Lab¶

Use one Splunk Enterprise VM and one Ubuntu endpoint. Complete all three activities with this pair before adding another source.

Optional extension

Complete PoC¶

After the core checkpoint passes, add Windows, Sysmon, FortiGate, SC4S, or a second forwarder using the linked guided journeys.

Session 1Build and snapshot
Session 2Ingest and verify
Session 3Investigate and publish

Objective¶

You are preparing a small SOC demonstration. Build the core Splunk topology, onboard one endpoint, prove that its data is searchable, investigate repeated test-account login failures, and record the result in a saved Splunk object.

The beginner route consists of:

one Splunk Enterprise instance;
one Ubuntu endpoint running the Universal Forwarder; and
an isolated network connecting the systems.

Use the complete proof-of-concept lab setup only when you are ready to reproduce the optional FortiGate, SC4S, Windows, Sysmon, and Linux paths together.

01

Activity 1: Build the environment¶

Download checklist¶

Record the filename, version, architecture, checksum, licence condition, and download date for every item.

Required for the Beginner Core Lab¶

Item	Purpose	Authoritative source
VMware Workstation Pro, demonstrated	Hypervisor used for the screenshots; another x86-64 hypervisor is suitable if it provides isolated virtual networking and snapshots	Broadcom installation and download guidance
Ubuntu Server ISO	Operating system for the Splunk server and endpoint	Ubuntu releases
Splunk Enterprise and Linux Universal Forwarder	Central platform and endpoint collection	Splunk downloads

Evaluation and vendor access

Trial licences may be time-limited, and some downloads require registration or licence acceptance. Check the terms for each selected product before building the lab.

Create the virtual machines¶

VM	Suggested lab allocation	Demonstrated address
Splunk	4 vCPU, 8 GB RAM, 100 GB disk	`10.0.0.100`
Ubuntu endpoint	2 vCPU, 4 GB RAM, 40 GB disk	`10.0.0.200`

Enable hardware virtualisation and install VMware Workstation or an equivalent x86-64 hypervisor with isolated networking and snapshot support.
Create an isolated lab network. Provide NAT only when a machine needs authorised download or update access.
Create the Splunk and Ubuntu endpoint VMs, attach the ISO, and install the operating system.
Configure hostnames, static addresses, gateway, DNS, and time synchronisation. Adapt the entire address plan consistently if required.
Install guest tools where appropriate and apply the selected OS baseline.
Confirm the two VMs can reach each other and only the required lab and update services.
Take a clean base-os snapshot before installing security products.

Milestone 1Two machines onlineBoth hosts have recorded addresses, synchronised clocks, working lab connectivity, and a clean snapshot.

Optional PoC machines and downloads

Add these only after the Beginner Core Lab works:

Optional item	Purpose
Windows Server evaluation ISO	Windows endpoint; document substitutions from the demonstrated 2019 baseline
Sysmon	Additional Windows endpoint telemetry
FortiGate and SC4S	Network-log ingestion path
Kali	Optional administrative or controlled test workstation

Size each optional VM using the requirements for its selected version and workload. Add its allocation to the host budget before starting it.

Expected result The Splunk and Ubuntu endpoint VMs start with synchronised clocks, recorded addresses, working connectivity, and a recoverable `base-os` snapshot.

02

Activity 2: Ingest and verify¶

Follow the checkpoints in order. Keep this page open and return after each linked journey.

Splunk is onlineInstall Splunk Enterprise, open Splunk Web, and confirm the receiving service.
The endpoint is connectedInstall the Linux Universal Forwarder on the Ubuntu endpoint and confirm it reports to the Splunk server.
The destination is readyCreate or confirm the index and input used for the Ubuntu endpoint.
The first event arrivesGenerate the recognisable event below, then find it by host, text, and execution time.
The baseline is recoverableRecord the working state and take a working-splunk-core snapshot.

Before testing, confirm that the forwarder monitors the actual system and authentication logs on your endpoint, can read them, and sends them to unixlog (or your chosen index). A connected forwarder alone does not collect these logs. See the Splunk Unix and Linux input guidance.

On the Ubuntu endpoint, record the current time and generate a harmless event:

logger -t cdk-lab "splunk-ingestion-check"

Confirm the message exists locally before searching Splunk. Depending on your logging configuration, it may be in /var/log/syslog or the systemd journal (journalctl -t cdk-lab --since "5 minutes ago"). If it exists only in the journal, a file-monitor input will not collect it; configure a supported journal input or a syslog file destination and monitor that destination.

In Splunk Search, replace <your-ubuntu-host> with the host value used by your forwarder. Replace unixlog too if you configured a different index, then run:

index=unixlog host="<your-ubuntu-host>" "splunk-ingestion-check"
| table _time host source sourcetype _raw

Use a narrow time range around the command. Save the query and record the event time, host, source, sourcetype, index, and raw message.

Milestone 2First event foundYou generated one known event and traced it from the Ubuntu endpoint into a searchable Splunk record.

Extend the working core with another ingestion path

Choose only one extension at a time:

add a Windows Universal Forwarder and Sysmon using Endpoint forwarders;
collect network logs using one method from Network logs; or
reproduce the wider proof-of-concept topology.

Repeat the same source → transport → index → search checkpoint before adding another integration. Treat SC4S and the FortiGate application as alternatives, not simultaneous requirements.

Expected result One Universal Forwarder is connected, the controlled Ubuntu event reaches the intended index, its key fields are recorded, and the working core has a recovery snapshot.

03

Activity 3: Investigate and publish¶

Restore or clone the working-splunk-core snapshot from Activity 2. Confirm Splunk Web opens and rerun the ingestion-check search. Continue only when the known event is still searchable.

Investigation scenario¶

Generate a few failed logins with a test account on your lab endpoint. Determine which source and account appear most often, when the activity occurred, and whether the evidence is sufficient to escalate.

From the Splunk VM or another authorised lab host, attempt to connect to the Ubuntu endpoint with ssh <test-user>@10.0.0.200 and deliberately enter an incorrect password three times. This requires a running SSH server that accepts password authentication for that test account. Check the account's lockout policy first. If the endpoint uses key-only authentication, use existing authorised authentication test data instead. See Ubuntu's OpenSSH guide.
Record the test start and finish times. Do not use automated brute-force tooling for this exercise.
Search the narrow test window and confirm the raw events are present before aggregating them.
Confirm the authentication events are in unixlog. If you selected an extension, use its actual index instead. Identify the field names present in one raw event before aggregating them.
Confirm that source and account fields are extracted and populated. Substitute their actual names for src_ip and user below, then run the investigation:

index=unixlog ("login failed" OR "authentication failure" OR "Failed password")
| stats count earliest(_time) AS first_seen latest(_time) AS last_seen
  BY src_ip user
| convert ctime(first_seen) ctime(last_seen)
| sort - count

If either grouping field is missing, stop and correct the extraction or use the appropriate fields for your source. Do not interpret an empty aggregation as proof that no failures occurred. Record field substitutions in your notes.

Answer these questions:

Which source generated the most failures?
Which accounts were targeted?
Over what period did the activity occur?
What does the available evidence support?
What additional evidence would you need before treating it as an incident?

Expected result You can trace the test data into Splunk and use it to support a defensible finding rather than only listing raw events.

Present and validate¶

Choose one of the following:

save the search as a report;
create an alert with a threshold suitable for your lab; or
add a panel to a dashboard.

Give the object a clear title and description so another analyst can understand its purpose. Repeat the failed-login test and confirm the report or dashboard includes the new events in its time range. For an alert, meet its configured threshold and check that it triggers; a generic logger message will not match the failed-login search. Record any differences between your environment and the original PoC, then stop test activity and remove temporary inputs or accounts.

Milestone 3Finding publishedAnother analyst can open your saved object, understand the question it answers, and reproduce the result.

Expected result Another analyst can open your saved object, understand its purpose, and reproduce the result.

Full Lab evidence checklist¶

This checklist applies to the VM-based Full Lab. If you completed the Browser Lab, retain its downloaded evidence summary instead.

Do not aim to match every screenshot or complete every optional integration. Complete the challenge by collecting evidence that your selected route works:

The required core downloads with versions, filenames, checksums, sources, and licence constraints.
Core VM specifications, network, addresses, time synchronisation, and snapshots; include optional systems only if used.
A screenshot or note showing that Splunk Web is available.
A test event with its index, host, source, sourcetype, and timestamp.
The SPL used to investigate the authentication pattern.
A short written conclusion supported by the search results.
A saved report, tested alert, or dashboard panel.
Your selected route, field substitutions, and any differences from the original PoC.

0 of 8 recorded Mark each item when you have saved the evidence.

Clean up¶

Stop any test activity, disable temporary inputs or accounts that are no longer required, remove test credentials, and revert disposable VMs if appropriate. Retain your SPL, screenshots, and conclusions as your lab record.