So- over a year back, when metrics was a new concept to Splunk, I ran a licensing and storage comparison HERE.

Since Splunk has done many changes and improvements to how metrics are stored, and licensed, I felt it was time to run another comparison.

How testing will be performed

Most, if not all, of the test cases will be copied from the old tests.

For testing purposes, I will have three inputs, each pointing at their own separate index. Each of the inputs are configured exactly the same, with three variations.

  1. Regular Perfmon data. (Default for windows TA)
  2. Perfmon MK format.
  3. Perfmon as metrics

For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.

Click To View Configuration Files

Inputs.conf

[perfmon://LogicalDisk_Reg]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Regular
showZeroValue=1
[perfmon://LogicalDisk_MK]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_MK
mode=multikv
showZeroValue=1
[perfmon://LogicalDisk_Metric]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Metrics
showZeroValue=1

Transforms.conf

[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1

[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

Props.conf

[source::Perfmon:*Metric]
TRANSFORMS-_value = value
TRANSFORMS-metric_name = perfmon_metric_name
TRANSFORMS-instance = instance
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Indexes.conf

[disk_perfmon_regular]
coldPath = $SPLUNK_DB\disk_perfmon_regular\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_regular\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_regular\thaweddb

[disk_perfmon_mk]
coldPath = $SPLUNK_DB\disk_perfmon_mk\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_mk\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_mk\thaweddb

[disk_perfmon_metrics]
coldPath = $SPLUNK_DB\disk_perfmon_metrics\colddb
datatype = metric
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_metrics\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_metrics\thaweddb

Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am already trying to get ahold of a Ryzen….)

NO additional or 3rd party apps are installed.

If you would like to reproduce my results, you can do a fresh install of Splunk enterprise, and add the four configuration files listed above.

I added the configuration files, restarted Splunk, and took a lunch break.

When I returned, I disabled the inputs, and restarted Splunk, for a total of 25 minutes of testing.

Here are the results. The methods to obtain the data are below.

Click to View Data Collection Methods

Event Count Query

Just a quick count of events to ensure we are fairly grading the results.

| tstats count WHERE index=disk* groupby index 
| union 
    [| mstats count where index=disk* metric_name=* groupby index
        ]

Storage Usage

Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the three indexes, and recording “Size on disk”

License Utilization

index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log"
| stats sum(b) as Size by idx
| eval Size= Size/1024

Test Results – 25 Minutes

IndexEvent CountDisk SizeLicense Usage
disk_perfmon_mk 100248 KB140 KB
disk_perfmon_regular 9,200 492 KB1,160 KB
disk_perfmon_metrics 9,200 468 KB 1,239 KB

Statistics

% Licensing Difference MK Vs Metrics785%
% Disk Difference MK Vs Metrics88.7%

Further Testing

At this point, I re-enabled the inputs, restarted Splunk, and started the stopwatch and let it run for 45 more minutes.

I am curious to see the trend with more data. While, I am 100% certain Perfmon MK will be the hands-down winner in all of these tests, I am curious to know the longer term results….

Test Results – 1 Hour

IndexEvent CountDisk SizeLicense Usage
disk_perfmon_mk 279672 KB403 KB
disk_perfmon_regular 25,668724 KB 3,340 KB
disk_perfmon_metrics 25,668 1,120 KB3,569 KB
% Licensing Difference MK Vs Metrics785%
% Disk Difference MK Vs Metrics66%
% Licensing Difference Perfmon Vs Metrics 7%
% Disk Difference Perfmon Vs Metrics 54%

Conclusions

I was under the impression the licensing of Metrics had been improved in Splunk 8… however- compared to the PerfmonMK format- there is additional room for improvement left.

While- I will still continue to utilize metrics for use-cases, mostly due to the ease of use… I would be cautious around converting your existing PerfmonMK data to Metrics.

If I apply the 785% increase in licensing to what I am collecting in my production environment, I would go from 3GB Daily, to 25GB Daily for my PerfmonMK traffic. While- this would only account for a ~1% increase in my daily licensing, it is still something to be aware of.

index=index_utilization_summary st=PerfmonMK*
| stats sum(bytes) as TotalMKLicense 
| eval Total_MK_GB = TotalMKLicense / 1024 / 1024 / 1024 
| eval Total_Metrics_GB = (TotalMKLicense*7.85) / 1024 / 1024 / 1024

* If I performed the above math incorrectly, please let me know! *

In my opinion, the additional speed, performance, and usability of metrics would likely outweigh the 1% impact to MY licensing. However, for customers licensed for 100-500GB, this impact would be far more considerable.

If you are currently using the regular Perfmon format, instead of PerfmonMK, I would recommend to considering changing your collections to instead use metrics, as it is only a 7% difference in licensing. I also anticipate the metric’s disk usage difference will also reduce as the indexes grow.

Share this content
%d bloggers like this: