So- over a year back, when metrics was a new concept to Splunk, I ran a licensing and storage comparison HERE.
Since Splunk has done many changes and improvements to how metrics are stored, and licensed, I felt it was time to run another comparison.
How testing will be performed
Most, if not all, of the test cases will be copied from the old tests.
For testing purposes, I will have three inputs, each pointing at their own separate index. Each of the inputs are configured exactly the same, with three variations.
- Regular Perfmon data. (Default for windows TA)
- Perfmon MK format.
- Perfmon as metrics
For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.
[expand title=”Click To View Configuration Files”]Inputs.conf
[perfmon://LogicalDisk_Reg] counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec object = LogicalDisk instances = * disabled = 0 interval = 15 useEnglishOnly = true index=Disk_PerfMon_Regular showZeroValue=1 [perfmon://LogicalDisk_MK] counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec object = LogicalDisk instances = * disabled = 0 interval = 15 useEnglishOnly = true index=Disk_PerfMon_MK mode=multikv showZeroValue=1 [perfmon://LogicalDisk_Metric] counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec object = LogicalDisk instances = * disabled = 0 interval = 15 useEnglishOnly = true index=Disk_PerfMon_Metrics showZeroValue=1
Transforms.conf
[metrics-hostoverride] DEST_KEY = MetaData:Host REGEX = host=(\S+) FORMAT = host::$1 [value] REGEX = .*Value=(\S+).* FORMAT = _value::$1 WRITE_META = true [perfmon_metric_name] REGEX = .*object=(\S+).*counter=(\S+).* FORMAT = metric_name::$1.$2 metric_type::$1 WRITE_META = true [instance] REGEX = .*instance=(\S+).* FORMAT = instance::$1 WRITE_META = true
Props.conf
[source::Perfmon:*Metric] TRANSFORMS-_value = value TRANSFORMS-metric_name = perfmon_metric_name TRANSFORMS-instance = instance SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g
Indexes.conf
[disk_perfmon_regular] coldPath = $SPLUNK_DB\disk_perfmon_regular\colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB\disk_perfmon_regular\db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB\disk_perfmon_regular\thaweddb [disk_perfmon_mk] coldPath = $SPLUNK_DB\disk_perfmon_mk\colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB\disk_perfmon_mk\db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB\disk_perfmon_mk\thaweddb [disk_perfmon_metrics] coldPath = $SPLUNK_DB\disk_perfmon_metrics\colddb datatype = metric enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB\disk_perfmon_metrics\db maxTotalDataSizeMB = 512000 thawedPath = $SPLUNK_DB\disk_perfmon_metrics\thaweddb[/expand]
Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am already trying to get ahold of a Ryzen….)
NO additional or 3rd party apps are installed.
If you would like to reproduce my results, you can do a fresh install of Splunk enterprise, and add the four configuration files listed above.
I added the configuration files, restarted Splunk, and took a lunch break.
When I returned, I disabled the inputs, and restarted Splunk, for a total of 25 minutes of testing.
Here are the results. The methods to obtain the data are below.
[expand title=”Click to View Data Collection Methods”]Event Count Query
Just a quick count of events to ensure we are fairly grading the results.
| tstats count WHERE index=disk* groupby index | union [| mstats count where index=disk* metric_name=* groupby index ]
Storage Usage
Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the three indexes, and recording “Size on disk”
License Utilization
index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log" | stats sum(b) as Size by idx | eval Size= Size/1024[/expand]
Test Results – 25 Minutes
Index | Event Count | Disk Size | License Usage |
disk_perfmon_mk | 100 | 248 KB | 140 KB |
disk_perfmon_regular | 9,200 | 492 KB | 1,160 KB |
disk_perfmon_metrics | 9,200 | 468 KB | 1,239 KB |
Statistics
% Licensing Difference MK Vs Metrics | 785% |
% Disk Difference MK Vs Metrics | 88.7% |
Further Testing
At this point, I re-enabled the inputs, restarted Splunk, and started the stopwatch and let it run for 45 more minutes.
I am curious to see the trend with more data. While, I am 100% certain Perfmon MK will be the hands-down winner in all of these tests, I am curious to know the longer term results….
Test Results – 1 Hour
Index | Event Count | Disk Size | License Usage |
disk_perfmon_mk | 279 | 672 KB | 403 KB |
disk_perfmon_regular | 25,668 | 724 KB | 3,340 KB |
disk_perfmon_metrics | 25,668 | 1,120 KB | 3,569 KB |
% Licensing Difference MK Vs Metrics | 785% |
% Disk Difference MK Vs Metrics | 66% |
% Licensing Difference Perfmon Vs Metrics | 7% |
% Disk Difference Perfmon Vs Metrics | 54% |
Conclusions
I was under the impression the licensing of Metrics had been improved in Splunk 8… however- compared to the PerfmonMK format- there is additional room for improvement left.
While- I will still continue to utilize metrics for use-cases, mostly due to the ease of use… I would be cautious around converting your existing PerfmonMK data to Metrics.
If I apply the 785% increase in licensing to what I am collecting in my production environment, I would go from 3GB Daily, to 25GB Daily for my PerfmonMK traffic. While- this would only account for a ~1% increase in my daily licensing, it is still something to be aware of.
index=index_utilization_summary st=PerfmonMK* | stats sum(bytes) as TotalMKLicense | eval Total_MK_GB = TotalMKLicense / 1024 / 1024 / 1024 | eval Total_Metrics_GB = (TotalMKLicense*7.85) / 1024 / 1024 / 1024 * If I performed the above math incorrectly, please let me know! *
In my opinion, the additional speed, performance, and usability of metrics would likely outweigh the 1% impact to MY licensing. However, for customers licensed for 100-500GB, this impact would be far more considerable.
If you are currently using the regular Perfmon format, instead of PerfmonMK, I would recommend to considering changing your collections to instead use metrics, as it is only a 7% difference in licensing. I also anticipate the metric’s disk usage difference will also reduce as the indexes grow.