Splunk – 8.0.1 Metrics vs Events Licensing Comparison

So- over a year back, when metrics was a new concept to Splunk, I ran a licensing and storage comparison HERE.

Since Splunk has done many changes and improvements to how metrics are stored, and licensed, I felt it was time to run another comparison.

How testing will be performed

Most, if not all, of the test cases will be copied from the old tests.

For testing purposes, I will have three inputs, each pointing at their own separate index. Each of the inputs are configured exactly the same, with three variations.

1. Regular Perfmon data. (Default for windows TA)
2. Perfmon MK format.
3. Perfmon as metrics

For testing, I will be looking at the LogicalDisk perfmon, collecting data at a 15 second interval, with a very generous handful of metrics selected, to facilitate collecting a lot of data, rather quickly.

[expand title=”Click To View Configuration Files”]

Inputs.conf

[perfmon://LogicalDisk_Reg]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Regular
showZeroValue=1
[perfmon://LogicalDisk_MK]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_MK
mode=multikv
showZeroValue=1
[perfmon://LogicalDisk_Metric]
counters = % Free Space; Free Megabytes; Current Disk Queue Length; % Disk Time; Avg. Disk Queue Length; % Disk Read Time; Avg. Disk Read Queue Length; % Disk Write Time; Avg. Disk Write Queue Length; Avg. Disk sec/Transfer; Avg. Disk sec/Read; Avg. Disk sec/Write; Disk Transfers/sec; Disk Reads/sec; Disk Writes/sec; Disk Bytes/sec; Disk Read Bytes/sec; Disk Write Bytes/sec; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Read; Avg. Disk Bytes/Write; % Idle Time; Split IO/Sec
object = LogicalDisk
instances = *
disabled = 0
interval = 15
useEnglishOnly = true
index=Disk_PerfMon_Metrics
showZeroValue=1

Transforms.conf

[metrics-hostoverride]
DEST_KEY = MetaData:Host
REGEX = host=(\S+)
FORMAT = host::$1

[value]
REGEX = .*Value=(\S+).*
FORMAT = _value::$1
WRITE_META = true

[perfmon_metric_name]
REGEX = .*object=(\S+).*counter=(\S+).*
FORMAT = metric_name::$1.$2 metric_type::$1
WRITE_META = true

[instance]
REGEX = .*instance=(\S+).*
FORMAT = instance::$1
WRITE_META = true

Props.conf

[source::Perfmon:*Metric]
TRANSFORMS-_value = value
TRANSFORMS-metric_name = perfmon_metric_name
TRANSFORMS-instance = instance
SEDCMD-remove-whitespace = s/ /_/g s/\s/ /g

Indexes.conf

[disk_perfmon_regular]
coldPath = $SPLUNK_DB\disk_perfmon_regular\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_regular\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_regular\thaweddb

[disk_perfmon_mk]
coldPath = $SPLUNK_DB\disk_perfmon_mk\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_mk\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_mk\thaweddb

[disk_perfmon_metrics]
coldPath = $SPLUNK_DB\disk_perfmon_metrics\colddb
datatype = metric
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\disk_perfmon_metrics\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\disk_perfmon_metrics\thaweddb

[/expand]

Testing will be performed on a new install of Splunk enterprise 8.0.1, on my workstation. 32GB ram, xeon processor. (Don’t worry- I am already trying to get ahold of a Ryzen….)

NO additional or 3rd party apps are installed.

If you would like to reproduce my results, you can do a fresh install of Splunk enterprise, and add the four configuration files listed above.

I added the configuration files, restarted Splunk, and took a lunch break.

When I returned, I disabled the inputs, and restarted Splunk, for a total of 25 minutes of testing.

Here are the results. The methods to obtain the data are below.

[expand title=”Click to View Data Collection Methods”]

Event Count Query

Just a quick count of events to ensure we are fairly grading the results.

| tstats count WHERE index=disk* groupby index 
| union 
    [| mstats count where index=disk* metric_name=* groupby index
        ]

Storage Usage

Storage utilization was obtained in Windows explorer by manually going to C:\Program Files\Splunk\var\lib\splunk, right clicking the folders for each of the three indexes, and recording “Size on disk”

License Utilization

index=_internal source="C:\\Program Files\\Splunk\\var\\log\\splunk\\license_usage.log"
| stats sum(b) as Size by idx
| eval Size= Size/1024

[/expand]

Test Results – 25 Minutes

Index	Event Count	Disk Size	License Usage
disk_perfmon_mk	100	248 KB	140 KB
disk_perfmon_regular	9,200	492 KB	1,160 KB
disk_perfmon_metrics	9,200	468 KB	1,239 KB

Statistics

% Licensing Difference MK Vs Metrics	785%
% Disk Difference MK Vs Metrics	88.7%

Further Testing

At this point, I re-enabled the inputs, restarted Splunk, and started the stopwatch and let it run for 45 more minutes.

I am curious to see the trend with more data. While, I am 100% certain Perfmon MK will be the hands-down winner in all of these tests, I am curious to know the longer term results….

Test Results – 1 Hour

Index	Event Count	Disk Size	License Usage
disk_perfmon_mk	279	672 KB	403 KB
disk_perfmon_regular	25,668	724 KB	3,340 KB
disk_perfmon_metrics	25,668	1,120 KB	3,569 KB

% Licensing Difference MK Vs Metrics	785%
% Disk Difference MK Vs Metrics	66%
% Licensing Difference Perfmon Vs Metrics	7%
% Disk Difference Perfmon Vs Metrics	54%

Conclusions

I was under the impression the licensing of Metrics had been improved in Splunk 8… however- compared to the PerfmonMK format- there is additional room for improvement left.

While- I will still continue to utilize metrics for use-cases, mostly due to the ease of use… I would be cautious around converting your existing PerfmonMK data to Metrics.

If I apply the 785% increase in licensing to what I am collecting in my production environment, I would go from 3GB Daily, to 25GB Daily for my PerfmonMK traffic. While- this would only account for a ~1% increase in my daily licensing, it is still something to be aware of.

index=index_utilization_summary st=PerfmonMK*
| stats sum(bytes) as TotalMKLicense 
| eval Total_MK_GB = TotalMKLicense / 1024 / 1024 / 1024 
| eval Total_Metrics_GB = (TotalMKLicense*7.85) / 1024 / 1024 / 1024

* If I performed the above math incorrectly, please let me know! *

In my opinion, the additional speed, performance, and usability of metrics would likely outweigh the 1% impact to MY licensing. However, for customers licensed for 100-500GB, this impact would be far more considerable.

If you are currently using the regular Perfmon format, instead of PerfmonMK, I would recommend to considering changing your collections to instead use metrics, as it is only a 7% difference in licensing. I also anticipate the metric’s disk usage difference will also reduce as the indexes grow.