Walkthrough: Building a Splunk Search (2024)

The following walkthroughs illustrate how to build a Splunk search from scratch.

Scenario: we want to identify users who launch a specific executable more often than n times in a given time range.

Splunk SPL Search

Let’s build the Splunk SPL search step by step.

Step 1

We start with all events from uberAgent’s index.

Note: to facilitate changing the index name used by uberAgent, all our dashboards make use of the uberAgent_index macro which contains the actual index name. The macro is defined in macros.conf of the uberAgent searchhead app.

index=`uberAgent_index`<!--NeedCopy-->

Step 2

We filter for the process startup sourcetype which contains one event per started process.

Note: The documentation of uberAgent’s sourcetypes and fields can be found here.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup<!--NeedCopy-->

Step 3

We ignore processes started by SYSTEM, LOCAL SERVICE and NETWORK SERVICE.

Note: The pseudo-users sys, lvc and nvc are defined in the lookup table systemusers.csv of the uberAgent searchhead app. They are auto-expanded to the proper user names SYSTEM, LOCAL SERVICE and NETWORK SERVICE in uberAgent’s data model.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc<!--NeedCopy-->

Step 4

We add a filter for the name of the process we are interested in, Winword.exe in this example.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe<!--NeedCopy-->

Step 5

We count the number of (start) events per user.

Note: The only purpose of adding the field ProcName to the stats command is to make it part of the results table, too.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe| stats count as Starts by ProcName ProcUser<!--NeedCopy-->

Step 6

We only keep users with more than five starts in the results list.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe| stats count as Starts by ProcName ProcUser| where Starts > 5<!--NeedCopy-->

Step 7

We rename fields to make them look nicer.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe| stats count as Starts by ProcName ProcUser| where Starts > 5| rename ProcUser as User ProcName as Process<!--NeedCopy-->

Step 8

We sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.

index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup ProcUser!=sys ProcUser!=lvc ProcUser!=nvc ProcName=Winword.exe| stats count as Starts by ProcName ProcUser| where Starts > 5| rename ProcUser as User ProcName as Process| sort 0 -Starts<!--NeedCopy-->

The Result

This is what the final Splunk SPL search for users with more than five starts of Winword.exe looks like. The screenshot below shows the search being run over the past 30 days. In practice, you would adjust the time range to any relevant time interval.

Accelerated Data Model Search

uberAgent comes with an accelerated data model. Searching an accelerated data model is a lot faster than searching the underlying index (by "a lot" we mean at least 50x), but requires a different search syntax based on the pivot or tstats commands. We are using pivot because of the easier syntax compared to tstats.

In this second example, we demonstrate how to search for starts of a "modern" UWP app, specifically the weather app that is part of Windows. Most UWP apps cannot be identified by process name - which is simply backgroundTaskHost.exe. Luckily uberAgent determines the real app name automatically.

Step 1

We start with a count of all process starts.

Note: The documentation for the Splunk pivot command can be found here.

Note: the macro uA_DM_Process_ProcessStartup resolves to the name of the data model containing the Process_ProcessStartup dataset. We use this technique to facilitate moving datasets between data models.

| pivot `uA_DM_Process_ProcessStartup` Process_ProcessStartup count(Process_ProcessStartup) as Starts<!--NeedCopy-->

Step 2

We filter for the weather app.

Note: An easy way to identify the name of the weather app is to dig around with a search like the following: index=`uberAgent_index` sourcetype=uberAgent:Process:ProcessStartup AppName=*Weather*

| pivot `uA_DM_Process_ProcessStartup` Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is "Microsoft.BingWeather"<!--NeedCopy-->

Step 3

We split by user so that we get a count of process starts per user (renaming the ProcUser field to User in the process).

| pivot `uA_DM_Process_ProcessStartup` Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is "Microsoft.BingWeather" splitrow ProcUser as User<!--NeedCopy-->

Step 4

We only keep users with more than five starts in the results list. We also sort the results so that the user with the highest number of starts is listed first. The 0 in the sort command ensures that the output is not truncated after the 10,000th result.

| pivot `uA_DM_Process_ProcessStartup` Process_ProcessStartup count(Process_ProcessStartup) as Starts filter AppName is "Microsoft.BingWeather" splitrow ProcUser as User| where Starts > 5| sort 0 -Starts<!--NeedCopy-->

Step 5

We add the application name as a row to the results table.

| pivot `uA_DM_Process_ProcessStartup` Process_ProcessStartup count(Process_ProcessStartup) as Starts latest(AppName) as Application filter AppName is "Microsoft.BingWeather" splitrow ProcUser as User| where Starts > 5| sort 0 -Starts| table Application User Starts<!--NeedCopy-->

The Result

The resulting output is very similar to the first example above: a table with the application, the users and the number of starts that can easily be exported to CSV or otherwise be processed further.