Note- this page is hidden from view due to inaccuracies discovered long after this article was published. The below method is flawed, and actually only records the first instance from each perfmon category collected.
This is an updated version of the original 8.0.1 test, located here. The reason for the update- Splunk reached out to me and provided me with a newly introduced method of ingesting metrics, as of version 8.0.
As a result, I implemented the new methods, and re-executed the tests, INCLUDING the original methods, along with the new methods as well.
How testing will be performed
For testing purposes, I will have four inputs, each pointing at their own separate index. Each of the inputs are configured with the same data collection, and interval.
- Regular Perfmon as Events (Default for TA_Windows)
- Regular Perfmon as Metrics
- Perfmon MK as Events
- Perfmon MK as Metrics MK (New Method)
For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.
Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am still trying to get ahold of a Ryzen….)
NO additional or 3rd party apps are installed. Testing was performed on a fresh install of Splunk, with only the above configuration files added.
The tests were started at 8:57am, and ended at 9:27am.
Data Collection Methods
Event Count
Count of events was obtained by recording the number displayed at http://localhost:8000/en-US/manager/search/data/indexes
Storage Usage
Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the indexes, and recording “Size on disk”
License Utilization
index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log" | stats sum(b) as Size by idx | eval Size= Size/1024
Performance Testing
Performance tests will be done with a specific query used for each index. Due to the limited amount of data (30 minutes, at a 15 second interval), there may not be enough data to do a “Production” test. Tests will be ran on the same timespan from 9am to 9:30am. An average of 5 query times will be recorded.
Here are the individual searches:
Test Results – 30 Minutes
| Index | Event Count | Disk Size | License Usage |
| perfmon -> events | 10,856 | 572 KB | 1,431 KB |
| perfmon -> metrics | 10,856 | 516 KB | 1,508 KB |
| perfmon_mk -> events | 118 | 292 KB | 173 KB |
| perfmon_mk -> metrics_mk | 118 | 256 KB | 16 KB |
PerfmonMK -> MetricsMK Statistics
| % License Decrease compared to Perfmon Events | 98% |
| % License Decrease compared to Perfmon MK | 90.7% |
| % Disk Usage Decrease compared to Perfmon Events | 55% |
| % Disk Usage Decrease compared to Perfmon MK | 12% |
Performance Results
| Index Name | Average Speed (Seconds) |
| perfmon_disk_events | 0.1504 |
| perfmon_mk_disk_events | 0.1612 |
| perfmon_disk_metrics | 0.097 |
| perfmon_mk_disk_metrics_mk | 0.119 |
Disclaimer: 30 minutes of data is not enough data to do a real-world comparison test.
If you wanted an accurate test, I would recommend searching at least one month of data in an production system. These tests were performed on my local machine, and are subject to variances caused by other processes running in the background.
My conclusion:
Metrics are faster then events. I will not give a percentage here, because I do not feel enough data is present to create an accurate test of measuring performance.
Conclusions
In the original post, the method used to convert Perfmon MK events to metrics was a pretty old method introduced in the Splunk infrastructure app a few years back. After making the post, Splunk’s engineering team reached out to me providing a lot of technical insight and documentation into the Metrics MK format.
After converting my tests to utilize the metrics MK format, I am completely blown away at the reduction in Licensing, and disk. Compared to the PerfmonMK format I am using in production currently, I can save over 90% on licensing, and over 10% on storage consumption by switching to a MUCH faster format, which is easier for users to ingest.
If you are interested in converting your perfmon data to metrics, I am in the process of finishing up a python script which will automatically build out the props.conf and transforms.conf to do so, with no manual configuration adjustments required.
If you are interested in contributing to this project, please visit the github page here.
My two cents- If you are not in the process of converting your data to MetricsMK, You should be!!! I cannot express how much better the performance, license usage, and disk usage is compared to the out-of-the-box perfmon format.
Documentation
Metrics Overview: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/Overview
Using Multi-Value Metrics: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/GetMetricsInOther
Log to Metrics Overview: https://docs.splunk.com/Documentation/Splunk/8.0.1/Metrics/L2MOverview
Special Thanks
I have received a lot of assistance from the Splunk team to provide this article. As such, I would like to call out their assistance.
- David Maislin @ Splunk has greatly assisted with issues related to Metrics, and has provided a lot of recommendations on putting together this content.
- (More coming after I get their permission to post their names.)
