This can be useful for predicting future change, and allows you to more easily at a glance determine the current trend of your data, when it may not be clear.

Lets start with some base data.

In the above data, the goal is to display the percentage change week to week, as a easy-to-view trendline.

First- we need to calculate the percent change from one interval to the next. Thankfully- this is pretty easy with streamstats.

| streamstats current=f window=1 values(Value) as PValue
| eval PctChange = ((Value-PValue)/PValue)*100
| fields - PValue

For an explanation- “streamstats current=f window=1 values(Value) as PValue” will store the previous value as new variable, “PValue”.

We can then use the new variable to calculate the percentage change. This leaves us with the following data.

While, at this point, we could now enable an overlay for PctChange- it does not clearly display our trend. The goal, is for whoever looks at this chart, to be able to clearly see how the data is trending over time.

To better accomplish the goal, we are going to utilize the trendline command.

The documentation for Trendline can be found here: Trendline

For this example, we are going to do a simple moving average, for at least 4 periods.

| trendline sma4(PctChange) as PctChangeTrend

This should be adequate for smoothing out the results.

Lastly- I do recommend setting the intervals for your overlay to produce an easily distinguishable trend.

Here is my final query:

| inputlookup temp_1.csv
| timechart span=1w sum(value) as Value
| streamstats current=f window=1 values(Value) as PValue
| eval PctChange = ((Value-PValue)/PValue)*100
| trendline sma4(PctChange) as PctChangeTrend
| fields - PValue PctChange

This is an updated version of the original 8.0.1 test, located here. The reason for the update- Splunk reached out to me and provided me with a newly introduced method of ingesting metrics, as of version 8.0.

As a result, I implemented the new methods, and re-executed the tests, INCLUDING the original methods, along with the new methods as well.

How testing will be performed

For testing purposes, I will have four inputs, each pointing at their own separate index. Each of the inputs are configured with the same data collection, and interval.

1. Regular Perfmon as Events (Default for TA_Windows)
2. Regular Perfmon as Metrics
3. Perfmon MK as Events
4. Perfmon MK as Metrics MK (New Method)

For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.

# Regular Perfmon Data, stored in Events index.
[perfmon://LogicalDisk_Event]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 1
interval = 15
useEnglishOnly = true
index=perfmon_disk_events
showZeroValue=1

# Regular Perfmon Data, stored in Metrics index.
[perfmon://LogicalDisk_Metric]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 1
interval = 15
useEnglishOnly = true
index=perfmon_disk_metrics
showZeroValue=1
sourcetype=Perfmon_To_Metric

# Perfmon MK Data, Stored in Events index.
[perfmon://LogicalDisk_MK_Event]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 1
interval = 15
useEnglishOnly = true
index=perfmon_mk_disk_events
mode=multikv
showZeroValue=1

# Perfmon MK Data, Stored in Metrics Index.
[perfmon://LogicalDisk_MK_MVMetric]
counters = % Free Space;; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 1
interval = 15
mode=multikv
useEnglishOnly = true
index=perfmon_mk_disk_metrics_mk
showZeroValue=1
sourcetype=PerfmonMK_To_MetricMK_AUTO

#Convert Regular Perfmon Event, into a Metric
[Perfmon_To_Metric]
TRANSFORMS-_value = value
TRANSFORMS-metric_name = perfmon_metric_name
TRANSFORMS-instance = instance
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

#Convert Perfmon MK Event, into a multi-key Metric
[PerfmonMK_To_MetricMK_AUTO]
INDEXED_EXTRACTIONS = tsv
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = 1
category = Log To Metrics
pulldown_type  = 1
METRIC-SCHEMA-TRANSFORMS = metric-schema:PerfmonMK_To_MetricMK_AUTO
TRANSFORMS-perfmonmk = perfmonmk:PerfmonMK_To_MetricMK_AUTO

[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

[metric-schema:PerfmonMK_To_MetricMK_AUTO]
METRIC-SCHEMA-MEASURES = _ALLNUMS_

[perfmonmk:PerfmonMK_To_MetricMK_AUTO]
WRITE_META = 1
REGEX = collection=\"?(?<collection>[^\"\n]+)\"?\ncategory=\"?(?<category>[^\"\n]+)\"?\nobject=\"?(?<object>[^\"\n]+)\"?\n([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t\n([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t([^\t]+)\t\n
FORMAT = collection::"$1" category::"$2" object::"$3" "$4"::"$28" "$5"::"$29" "$6"::"$30" "$7"::"$31" "$8"::"$32" "$9"::"$33" "$10"::"$34" "$11"::"$35" "$12"::"$36" "$13"::"$37" "$14"::"$38" "$15"::"$39" "$16"::"$40" "$17"::"$41" "$18"::"$42" "$19"::"$43" "$20"::"$44" "$21"::"$45" "$22"::"$46" "$23"::"$47" "$24"::"$48" "$25"::"$49" "$26"::"$50" "$27"::"$51"
WRITE_META = true

# Regular Perfmon Data, Events Index.
[perfmon_disk_events]
coldPath = $SPLUNK_DB\$_index_name\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\$_index_name\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\$_index_name\thaweddb

# Regular Perfmon Data, Metrics Index.
[perfmon_disk_metrics]
coldPath = $SPLUNK_DB\$_index_name\colddb
datatype = metric
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\$_index_name\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\$_index_name\thaweddb

# Perfmon MK Data, Events Index.
[perfmon_mk_disk_events]
coldPath = $SPLUNK_DB\$_index_name\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\$_index_name\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\$_index_name\thaweddb

# Perfmon MK Data, Metrics Index.
[perfmon_mk_disk_metrics_mk]
coldPath = $SPLUNK_DB\$_index_name\colddb
enableDataIntegrityControl = 0
datatype = metric
enableTsidxReduction = 0
homePath = $SPLUNK_DB\$_index_name\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\$_index_name\thaweddb

Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am still trying to get ahold of a Ryzen….)

NO additional or 3rd party apps are installed. Testing was performed on a fresh install of Splunk, with only the above configuration files added.

The tests were started at 8:57am, and ended at 9:27am.

Data Collection Methods

Event Count

Count of events was obtained by recording the number displayed at http://localhost:8000/en-US/manager/search/data/indexes

Storage Usage

Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the indexes, and recording “Size on disk”

License Utilization

index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log"
| stats sum(b) as Size by idx
| eval Size= Size/1024

Performance Testing

Performance tests will be done with a specific query used for each index. Due to the limited amount of data (30 minutes, at a 15 second interval), there may not be enough data to do a “Production” test. Tests will be ran on the same timespan from 9am to 9:30am. An average of 5 query times will be recorded.

Here are the individual searches:

Test Results – 30 Minutes

PerfmonMK -> MetricsMK Statistics

Performance Results

index=perfmon_disk_events instance=”C:” counter=”% Disk Read Time” | timechart span=15s avg(Value)
This search has completed and has returned 121 results by scanning 109 events in 0.132 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.223 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.136 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.139 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.122 seconds

index=perfmon_mk_disk_events | timechart span=15s avg(%_Disk_Read_Time)
This search has completed and has returned 121 results by scanning 109 events in 0.161 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.159 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.149 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.142 seconds
This search has completed and has returned 121 results by scanning 109 events in 0.195 seconds

| mstats avg(_value) WHERE metric_name="LogicalDisk.%_Disk_Read_Time" AND index="perfmon_disk_metrics" span=15s
This search has completed and has returned 109 results by scanning 436 events in 0.079 seconds
This search has completed and has returned 109 results by scanning 436 events in 0.088 seconds
This search has completed and has returned 109 results by scanning 436 events in 0.081 seconds
This search has completed and has returned 109 results by scanning 436 events in 0.15 seconds
This search has completed and has returned 109 results by scanning 436 events in 0.087 seconds

| mstats avg(_value) WHERE metric_name=%_Disk_Read_Time AND index=perfmon_mk_disk_metrics_mk span=15s
This search has completed and has returned 109 results by scanning 109 events in 0.19 seconds
This search has completed and has returned 109 results by scanning 109 events in 0.076 seconds
This search has completed and has returned 109 results by scanning 109 events in 0.09 seconds
This search has completed and has returned 109 results by scanning 109 events in 0.158 seconds
This search has completed and has returned 109 results by scanning 109 events in 0.081 seconds

Disclaimer: 30 minutes of data is not enough data to do a real-world comparison test.

If you wanted an accurate test, I would recommend searching at least one month of data in an production system. These tests were performed on my local machine, and are subject to variances caused by other processes running in the background.

My conclusion:

Metrics are faster then events. I will not give a percentage here, because I do not feel enough data is present to create an accurate test of measuring performance.

Conclusions

In the original post, the method used to convert Perfmon MK events to metrics was a pretty old method introduced in the Splunk infrastructure app a few years back. After making the post, Splunk’s engineering team reached out to me providing a lot of technical insight and documentation into the Metrics MK format.

After converting my tests to utilize the metrics MK format, I am completely blown away at the reduction in Licensing, and disk. Compared to the PerfmonMK format I am using in production currently, I can save over 90% on licensing, and over 10% on storage consumption by switching to a MUCH faster format, which is easier for users to ingest.

If you are interested in converting your perfmon data to metrics, I am in the process of finishing up a python script which will automatically build out the props.conf and transforms.conf to do so, with no manual configuration adjustments required.

If you are interested in contributing to this project, please visit the github page here.

My two cents- If you are not in the process of converting your data to MetricsMK, You should be!!! I cannot express how much better the performance, license usage, and disk usage is compared to the out-of-the-box perfmon format.

Documentation

Metrics Overview: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/Overview

Using Multi-Value Metrics: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/GetMetricsInOther

Log to Metrics Overview: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/L2MOverview

Special Thanks

I have received a lot of assistance from the Splunk team to provide this article. As such, I would like to call out their assistance.

1. David Maislin @ Splunk has greatly assisted with issues related to Metrics, and has provided a lot of recommendations on putting together this content.
2. (More coming after I get their permission to post their names.)

Since Splunk has done many changes and improvements to how metrics are stored, and licensed, I felt it was time to run another comparison.

How testing will be performed

Most, if not all, of the test cases will be copied from the old tests.

For testing purposes, I will have three inputs, each pointing at their own separate index. Each of the inputs are configured exactly the same, with three variations.

1. Regular Perfmon data. (Default for windows TA)
2. Perfmon MK format.
3. Perfmon as metrics

For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.

Inputs.conf

[perfmon://LogicalDisk_Reg]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Regular
showZeroValue=1
[perfmon://LogicalDisk_MK]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_MK
mode=multikv
showZeroValue=1
[perfmon://LogicalDisk_Metric]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Metrics
showZeroValue=1

Transforms.conf

[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1

[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

Props.conf

[source::Perfmon:*Metric]
TRANSFORMS-_value = value
TRANSFORMS-metric_name = perfmon_metric_name
TRANSFORMS-instance = instance
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Indexes.conf

[disk_perfmon_regular]
coldPath = $SPLUNK_DB\disk_perfmon_regular\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_regular\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_regular\thaweddb

[disk_perfmon_mk]
coldPath = $SPLUNK_DB\disk_perfmon_mk\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_mk\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_mk\thaweddb

[disk_perfmon_metrics]
coldPath = $SPLUNK_DB\disk_perfmon_metrics\colddb
datatype = metric
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_metrics\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_metrics\thaweddb

Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am already trying to get ahold of a Ryzen….)

NO additional or 3rd party apps are installed.

If you would like to reproduce my results, you can do a fresh install of Splunk enterprise, and add the four configuration files listed above.

I added the configuration files, restarted Splunk, and took a lunch break.

When I returned, I disabled the inputs, and restarted Splunk, for a total of 25 minutes of testing.

Here are the results. The methods to obtain the data are below.

Event Count Query

Just a quick count of events to ensure we are fairly grading the results.

| tstats count WHERE index=disk* groupby index 
| union 
    [| mstats count where index=disk* metric_name=* groupby index
        ]

Storage Usage

Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the three indexes, and recording “Size on disk”

License Utilization

index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log"
| stats sum(b) as Size by idx
| eval Size= Size/1024

Test Results – 25 Minutes

Statistics

Further Testing

At this point, I re-enabled the inputs, restarted Splunk, and started the stopwatch and let it run for 45 more minutes.

I am curious to see the trend with more data. While, I am 100% certain Perfmon MK will be the hands-down winner in all of these tests, I am curious to know the longer term results….

Test Results – 1 Hour

Conclusions

I was under the impression the licensing of Metrics had been improved in Splunk 8… however- compared to the PerfmonMK format- there is additional room for improvement left.

While- I will still continue to utilize metrics for use-cases, mostly due to the ease of use… I would be cautious around converting your existing PerfmonMK data to Metrics.

If I apply the 785% increase in licensing to what I am collecting in my production environment, I would go from 3GB Daily, to 25GB Daily for my PerfmonMK traffic. While- this would only account for a ~1% increase in my daily licensing, it is still something to be aware of.

index=index_utilization_summary st=PerfmonMK*
| stats sum(bytes) as TotalMKLicense 
| eval Total_MK_GB = TotalMKLicense / 1024 / 1024 / 1024 
| eval Total_Metrics_GB = (TotalMKLicense*7.85) / 1024 / 1024 / 1024

* If I performed the above math incorrectly, please let me know! *

In my opinion, the additional speed, performance, and usability of metrics would likely outweigh the 1% impact to MY licensing. However, for customers licensed for 100-500GB, this impact would be far more considerable.

If you are currently using the regular Perfmon format, instead of PerfmonMK, I would recommend to considering changing your collections to instead use metrics, as it is only a 7% difference in licensing. I also anticipate the metric’s disk usage difference will also reduce as the indexes grow.

One issue that gets thrown around alot, is being able to restrict users to specific sourcetypes, in certain indexes.

While- this feature was possible in earlier versions, it was very clumsy, and could be difficult to maintain… and had potential security concerns due to its implementation.

With the release of Splunk 8.0.0– It appears significant improvements have been made to this process, which would make it much more feasible in a production environment.

With a bit of research- the new controls appear to be more optimized, and easier to maintain.

Here is the documentation for the configuration of this new functionality.

Caveats

1. These restrictions do not apply to metrics data.
2. This only applies to filtering by indexed fields

Pros

1. This means you can filter a role to specific sourcetypes / sources within an index, securely.

Lets test!

For my test- I created user “testuser”, with role “user-testuser”

Test 1: Filter out a specific sourcetype for all users, but, allow test-user access

My first task, is to set a restriction on the user’s role to deny access to index=_internal, sourcetype=mongod

NOT (index::_internal AND sourcetype::mongod)

As expected, my test-user can no longer see the mongod sourcetype.

Here is the generated search, pulled from the searchlog

| tstats prestats=t local=f summariesonly=f allow_old_summaries=f chunk_size=10000000 count where ((index=* OR index=_*) (NOT sourcetype::mongod OR NOT index::_internal)) groupby index sourcetype

Something which confuses me about the generated search, is this line: (NOT sourcetype::mongod OR NOT index::_internal). However- it does work as we configured it… oddly.

But- the expanded filtering search looks as expected

( ( ( index=* OR index=_* ) ( NOT sourcetype::mongod OR NOT index::_internal ) ) ) ( ( NOT ( sourcetype::mongod AND index::_internal ) ) )

So- next, lets see if we can override this rule using inheritance. On a hunch- I decided to see how it would appear in a search.

It appears to just append the SPL filters for each role, starting with the highest role first. As a result, it does not look like you will be able to override the SPL filter based on inheritance.

Summary

With the new improvements, you can more easily filter out subsets of your data on fields such as source, sourcetype, or other indexed fields. This functionality did exist before, but, a well-crafted search could bypass the filter.

The additional UI features will make it easier to add additional rules to this new feature. While this is not a completely new feature, the subtle improvements are well welcomed.

Since Splunk version 8.0 was dropped earlier today, I decided to upgrade my DEV environment to start playing with the latest additions and changes.

Links and References

Splunk 8.0.0 Release Notes

Before Upgrading – Notes

The problem

However- after running the upgrade, I quickly ran into this issue, found in web_service.log, preventing SplunkWeb from starting properly.

web_service.log

2019-10-22 10:45:25,928 INFO    [5daf2415ae7f3323106710] __init__:174 - Using default logging config file: /opt/splunk/etc/log.cfg
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk level=INFO
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver level=INFO
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.controllers level=INFO
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.appserver.lib level=WARN
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.pdfgen level=INFO
2019-10-22 10:45:25,935 INFO    [5daf2415ae7f3323106710] __init__:212 - Setting logger=splunk.archiver_restoration level=INFO
2019-10-22 10:45:26,259 ERROR   [5daf2415ae7f3323106710] root:769 - Unable to start splunkweb
2019-10-22 10:45:26,259 ERROR   [5daf2415ae7f3323106710] root:770 - No module named 'UserDict'
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in <module>
    from splunk.appserver.mrsparkle.controllers.top import TopController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in <module>
    from splunk.appserver.mrsparkle.controllers.admin import AdminController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 25, in <module>
    from splunk.appserver.mrsparkle.controllers.appinstall import AppInstallController
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/appinstall.py", line 22, in <module>
    from splunk.appserver.mrsparkle.lib import module
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 465, in <module>
    moduleMapper = ModuleMapper()
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 83, in __init__
    self.installedModules = self.getInstalledModules()
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 28, in helper
    return f(*a, **kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 448, in getInstalledModules
    mods = self.getModuleList(root)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 37, in helper
    return f(*a, **kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/module.py", line 223, in getModuleList
    mod = __import__(modname)
  File "/opt/splunk/etc/apps/search_activity/appserver/modules/D3DynamicTree/D3DynamicTree.py", line 5, in <module>
    import UserDict
ModuleNotFoundError: No module named 'UserDict'

So, the first troubleshooting step I will perform- is to reinstall the .rpm, just in case.

sudo rpm --reinstall ~/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86_64.rpm
splunk start
splunk status

Of course- that did not resolve the issue. The biggest issue which stands out in the logs, is “ModuleNotFoundError: No module named ‘UserDict'”

So- since I see references to the search_activity app, I removed the app and restarted Splunk.

Voila, Problem resolved. Perhaps somebody will find some use in this article.

Update- the search_activity app was updated for Splunk 8.0

FIrst of all- I will include a working example in this post, assuming you utilize the Maps+ Plugin. If you do not leverage this plugin- I strongly recommend you take a look. It is actively developed by one of Splunk’s employees, and is loaded with useful features.

Step 1. Create “Sample” data. If you intend on using this with your data, you may leave this step out.

| makeresults count=100 

Step 2. Calculate a “Percent” column.

| eventstats count as total 
| streamstats count as pos 
| eval pct = ROUND(pos/total*100,0) 

Step 3. Generate a dummy latitude and longitude based on the data. I just need to demonstrate a straight line…. If you are using your own data, leave this step out.

| eval latitude=pct * 0.10, longitude=0 

Step 4. Generate the “Color Curve”. This implementation starts at green, and ends with red. If you want to change the colors, tweak this line.

| eval C_r=IF(pct<50, 255, 510-5.10*pct), C_g=IF(pct<50, 5.1*pct, 255), C_b=0 

Step 5. Here is the condensed version of my original method, using foreach, instead of one eval per color channel.

| foreach C_* 
    [| eval <<FIELD>>= REPLACE(tostring(ROUND(<<FIELD>>, 0),"hex"), "0x", "") 
    | eval <<FIELD>>=substr("00", 0, max(2-len(<<FIELD>>), 0)).<<FIELD>>] 

Step 6. Generate a hex color code.

| eval circleColor="#".C_r.C_g.C_b

Step 7. Formatting. I am piping everything to a table out. markerType=”Circle” changes how the icons are formatted.

| eval markerType="circle" 
| table pct latitude longitude circleColor markerType

Here is the finished product:

Here is the full query I used for this example.

| makeresults count=100 
| eventstats count as total 
| streamstats count as pos 
| eval pct = ROUND(pos/total*100,0) 
| eval latitude=pct * 0.10, longitude=0 
| eval C_r=IF(pct<50, 255, 510-5.10*pct), C_g=IF(pct<50, 5.1*pct, 255), C_b=0 
| foreach C_* 
    [| eval <<FIELD>>= REPLACE(tostring(ROUND(<<FIELD>>, 0),"hex"), "0x", "") 
    | eval <<FIELD>>=substr("00", 0, max(2-len(<<FIELD>>), 0)).<<FIELD>>] 
| eval circleColor="#".C_r.C_g.C_b
| eval markerType="circle" 
| table pct latitude longitude circleColor markerType

So- I had a need to calculate a RGB value based on the percentage total progress. My specific need was to color-code a vehicles route on a map by the time.

Here is how I accomplished it.

First, we need to calculate the percentage complete. Your implementation may vary. However- this basic use case will calculate a percentage for most queries.

| eventstats count as total 
| streamstats count as pos 
| eval pct = ROUND(pos/total*100,0)

Next, I needed to calculate the RGB value. My implementation goes from red(0%) to green(100%).

If you wanted to tweak the colors, this would be the line to modify.

| eval r=IF(pct<50, 255, 510-5.10*pct), g=IF(pct<50, 5.1*pct, 255), b=0

Next, is this ugly block of commands, to convert the RGB values, into a single hex value. If you find a better way to manage this- please let me know.

## Round the values. Cannot convert decimals to hex.
| eval r=ROUND(r,0), g=ROUND(g,0), b=ROUND(b,0)
## Replace the 0x with empty.
| eval rH=REPLACE(tostring(r,"hex"), "0x", ""),gH=REPLACE(tostring(g,"hex"), "0x", ""), bH=REPLACE(tostring(b,"hex"), "0x", "")
## Ensure each "grouping" is two characters long.
| eval rH=substr("00", 0, max(2-len(rH), 0)).rH, gH=substr("00", 0, max(2-len(gH), 0)).gH, bH=substr("00", 0, max(2-len(bH), 0)).bH 
## Combine the hex segments
| eval circleColor="#".rH.gH.bH

Here is the color curve generated.

The below article has been superseded.

Please click here to view the newly updated article for Splunk 8.0.2

So, three months ago, I had a hunch that metrics would result in significantly higher license utilization.

My theory was posted to Reddit HERE:

Since… Splunk 7.2 added numerous improvements to how metrics are handled and searched,
Today, I am going to go through the process of testing this theory, and provide my results.

For testing purposes, I will have three inputs, each pointing at their own separate index. Each of the inputs are configured exactly the same, with three variations.

1. Regular Perfmon data.
2. Perfmon MK format.
3. Perfmon as metrics

For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.

Inputs.Conf

[perfmon://LogicalDisk_Reg]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Regular
showZeroValue=1

[perfmon://LogicalDisk_MK]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_MK
mode=multikv
showZeroValue=1

[perfmon://LogicalDisk_Metric]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Metrics
showZeroValue=1

Transforms.conf, unmodified from Splunk_TA_Infrastructure

########### Metrics ######################
[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1


########### Transforms for Windows ######################
[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

# Example: object=PhysicalDisk counter="%_Disk_Write_Time"
# Transform - metric_name::PhysicalDisk.%_Disk_Write_Time
[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true
Props.conf
INI:
[source::Perfmon:*Metric]
TRANSFORMS-_value = value
TRANSFORMS-metric_name = perfmon_metric_name
TRANSFORMS-instance = instance
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Testing was performed on a fresh install of Splunk enterprise 7.2, on my windows 10 workstation.

All inputs and indexes were enabled/created while Splunk was disabled, to prevent any of the methods from having skewed results.

While, the overall data collection interval was rather short, the data is conclusive to backup my original theory.

A few notes-

• For every single perfmonMK event collected, there are 92 separate metric/normal perfmon events collected.
• I did not collect enough data to properly assess the claimed performance benefits for using metrics, over events. This test was purely from the standpoint of license and disk utilization.
• Splunk did recently release an APP for browsing and displaying metrics. This app performs extremely well, and, in my opinion, makes it extremely easy for end users of the platform to consume metric data and create dashboards.
  • As a note- The app does also perform with normal, accelerated datasets very effectively. If you are like me, and have previously created a datasets for end users to consume performance data with, it works very well for that.

My list of CONs for metrics so far

• Ignoring the license utilization theory which will be evaluated below…
• One HUGE downside so far, is the inability to do automatic lookups to the data, or to enrich the data with more sources. From my understanding, the dimensions will need to be added during index time.
  • Good example: I have a lookup called “HostInformation” which, contains a lot of CMDB-related information. What applications the server belongs to, if its production or non-production, the OS version, etc. I have not found a meaningful method of utilizing this data WITH metrics, at least in the context of the metrics explorer.

After 10 minutes of testing, here are the results.

Total size on disk:
disk_perfmon_metrics — 0.14 MB
disk_perfmon_mk — 0.10 MB
disk_perfmon_regular — 0.20 MB

Perfmon MK is the clear winner, with metrics in 2nd place by a healthy margin.

License Utilization:

As expected, Perfmon:MK is a clear winner by a MASSIVE margin.

Here are the results from the Indexes viewer in Splunk:

(Sorry the picture is greyed out, I shut splunk down before collecting the test results)

My summary: I recommend to continue using perfmon:MK format, and wrapping the results into an accelerated dataset for consumption by the end users. The data model acceleration will greatly improve the performance, while the multikey format will significantly reduce the amount of license required.

At which time a few other improvements are made around metrics, I will evaluate if it is worth making the switch, at the expense of increased license utilization.

For me- License usage is important, because I collect hundreds of counters, from thousands of devices, at very regular intervals.