I have been trying to see exactly how fast I can push my existing hardware, without dumping ridiculous amounts of money on it.

I have tried many things to maximize the amount of performance, including even temporarily buying a pair of 100GBe nics a while back. (TLDR; lots of driver issues….)

Well, after months and months of tinkering, I have finally reached my goal of being able to easily saturate 40GBe ethernet.

If you wish to read more of the details which led me to this point, here are a few of the previous posts:

Backstory

The Original NAS – 500$ Mini Build

I started my current NAS journey, with a 500$ build, after being disappointed at the high prices of synology / drobo / qnap systems. I wanted to prove I could build a more capable system for less money. I did succeed.

This unit was eventually replaced due to the underpowered CPU.

Combined Gaming PC / Server

I ended up selling the above build, and for a few months, I turned my gaming PC into a combined gaming PC and server. Overall, it worked great. The only downside was the amount of heat being put into my bedroom. Oh, and limited number of PCIe lanes.

This was replaced by a Dell R720XD, which I picked up for 500$ shipped, with 128GB of ram included.

Upgrading my home network to 10/40G backbone

Sept of last year, I picked up a pair of brocade ICX-6610 10/40G switches. As well, this is the first article where my Dell R720XD was mentioned. I never did a formal write up on obtaining it. However, it replaced my Combined GamingPC/Server.

As apart of this project, I also ran 10GBe to my office.

Adding a server rack

Self explanatory, however, since having all of your hardware sitting on buckets and cardboard boxes is not the best thing to do, I invested in a rack… and made everything somewhat presentable.

Reducing power consumption – Retiring Brocade

Turns out, the brocade ICX-6610 produces more heat/noise then I was happy with. In this post, I retired the old switches. This saved a big chunk of power, but, also meant, I no longer had a 40GBe network core. Instead, I leveraged my opnsense firewall for providing a 10G core, although routed.

Running 40Gbit Fiber

Since, 10GBe just wasn’t cutting it for me, I finally got around to running 40Gbe fiber from my server closet to my office. This also added a Chelsio 40GBe NIC to my Gaming PC.

Since the 40GB was point to point, no switches, no routers, etc, This also improved performance significantly over the 10GBe routed iscsi from previous steps.

Adding “Bifurcation”

Since, my r720XD does not support bifurcation, I found a few PCIe cards to work as switches for my NVMes. This allowed me to run many more NVMe drives rather then only being able to leverage a single NVMe per PCIe-slot.

Infiniband

One day a few months back, I experimented with using Infiniband.

This involved custom compiling drivers for TrueNAS, and having to hack up a lot of bits and pieces you generally do not want to touch.

As well, the community forums are quite toxic to you doing anything that isn’t supported out of the box. But, I did manage to successfully get infiniband up and running. However, I was unable to perform any reasonable tests as I didn’t want to hack up the system anymore.

Further Reducing Power Consumption

Given electricity costs money, and power usage means heat production, I continued down the path of reducing the energy consumption of my home lab.

In this post, I reduced power consumption by ADDING new servers. Overall, I was able to reduce my R720XD’s power consumption by a pretty good chunk by moving services to the new SFF optiplex machines I had acquired.

Since, the server was no longer handling containers/VMs, I also removed its second CPU which actually helped performance out a good bit.

On the downside, I now only have 96G of ram instead of 128G.

Further Reducing Power Usage (Not yet published)

In the next post regarding reducing my power utilization, I enabled power management features within TrueNAS, to allow for greater savings. This knocked off ~20w of consumption, while not allowing the disks to spin down. As well, I swapped in a low-powered E5-2630L to further reduce power draw.

(Note, processor is still in the mail. Article will be completed AFTER this one has already been published)

Migrating from TrueNAS Scale, to TrueNAS Core

The final step to achieve the performance numbers I have been hunting for- I installed TrueNAS core, and rebuilt my configuration from scratch.

Out of the box, with no tuning whatsoever, it had no issues at all besting every performance metric from Scale.

The Benchmarks

I have split the benchmarks into two seperate sections. Those against my Flash array, and those against my pool of spinning rust.

I did post both 1G and 32G test sizes where possible.

I started running the benchmarks in 32G to hush the trolls who blame the results on cache/ram.

Against my 8-disk Z2 Array

The array, is a 8-disk Z2. LZ4 compression is enabled, without dedup. The drives are 8x USED 8TB Seagate Exos.

TrueNAS Scale

These benchmarks were performed before migrating to core.

iSCSI – 1G Test Size

SMB – 1G Test Size

TrueNAS Core

iSCSI – 1G Test Size

iSCSI – 32G Test Size

SMB – 1G Test Size

SMB – 32G Test Size

It is rather interesting to see the differences between 1G and 32G. The sharp decline in write speed/IOPs is due to the cache being saturated. However, large sequential writes are still working just fine.

Against My Flash Array

My flash array is a single mirrored VDEV, consisting of a pair of 1T Samsung 970 EVO NVMe. While, previously, I did add an additional stripe, I didn’t notice dramatic performance differences and ended up removing the extra vdev(Possible to do with openzfs).

Since, I did not take recent benchmarks of the flash array before migrating to core, I do not have any “before” results to compare with.

As well, I did not take any benchmarks over SMB. However, it is safe to assume the results will be faster than the SMB results.

TrueNAS Core – After Results

iSCSI – 1G Test Size

iSCSI – 32G Test Size

I will note, there is a tiny amount more room for improvement as my CPU is getting pegged pretty hard during write operations.

But, next week, I will be replacing my E5-2695v2 with a E5-2630L in attempts to further reduce my energy consumption. However, I do feel, I should still be able to obtain very reasonable benchmark results.

Next Steps?

Well, Other then replacing my CPU with a low-powered model which is going to hurt my performance slightly, I don’t really have any next steps.

I have been debating wanting to build out a CEPH cluster using a pile of SFF/MFF PCs, however, the budget for such a project is not in the cards for this year.

As well, I have toyed with the idea of deploying a disk array, but, this wouldn’t net me any performance improvements or reducings in power consumption. The array itself would use 30-60w without any disks. The only meaningful advantage would be to make it much easier to swap to a different piece of hardware. I have decided, that since the number of spinning spools directly correlates to the wattage draw, I don’t want to deploy a 16+ disk array. When zfs online expansion comes eventually, I will likely grow my existing array to 12 disks total.

For now, It is nearing time to focus my efforts on the 1,000hp turbocharged 4×4 suburban project! Because, everyone needs a 1,000hp surburban! (NOT an exaggeration on the horsepower numbers….)

Well, given my recent push to improve energy efficiency, I have reduced the average power utilization from my R720XD from 400-450w average, down to around 190w average. However, there is still more room to reduce usage.

Based on benchmarks I have seen, core should offer improved performance, which means less impact from the low-powered CPU I plan on putting into my server. As well, this should be a very easy migration from Scale to Core.

Benchmarks

Before doing anything, I ran a few simple, quick benchmarks across iScsi and SMB, to my Z2 array. I used the peak performance profile in CrystalBenchmark.

This isn’t designed to be a definitive test, but, to give me something quick to compare against.

TrueNAS Scale

iSCSI

SMB

TrueNAS Core (After migration)

iSCSI

SMB

As seen above, there are numerous performance improvements across the board. Given the sole purpose of my TrueNAS box at this point in time is storage, and storage alone, this makes the decision to move from Scale back to Core all worth it.

Steps

Step 1. Backups

The first and foremost step before doing anything to your storage system, is to ensure you have working backups. Assume, your migration will fail in a catastrophic manner, leaving all of your data unusable. So, make sure you have working backups.

Step 2. Configuration Backup

With a configuration backup, you can easily restore all of your configurations to a fresh install of TrueNAS, allowing you to be back up and online in minutes. Always keep a backup!

Step 3. Download the core ISO.

Download the desired release from HERE. I went with Core 13.0-RC1 assuming it would have the newer version of ZFS, minimizing potential pool compatibility issues.

Step 4. Mount / Burn the ISO.

If you do not have enterprise hardware with remote management capabilities, burn the disk to a CD/DVD/or ISO.

If you have iDrac / iLo, attach the image as virtual media.

If you are using idrac/ilo, set the next boot, to boot to your virtually mapped ISO.

Reboot your system.

Step 5.1 – Reboot, and find unexpected memory errors

During the process of rebooting to reinstall, I had one of my Dimms fail testing. As I have recently had a weird issue popping up seemingly related to memory…. I decided to go ahead and replace the A12 dimm.

After replacing stick A12, the errors/warnings went away.

Step 5.2 Install TrueNAS Core

After you get booted to the installer, you will want to select Install/Upgrade

After I was prompted for which drives to install TrueNAS, I selected my two 10K SAS drives where it was previously installed.

After which, I received a prompt letting me know everything would be erased.

Select your desired boot mode. The r720XD will boot either mode.

When asked to create a swap partition, I selected to not create swap (As the system has over 100G of ram…)

After finishing up the GUI-based options, a terminal appeared.

This step took a while to complete, likely due to the virtual media being accessed over the network.

After…. 10 minutes or so, Core was successfully reinstalled.

I disconnected the virtual media, and rebooted the system.

Step 6. Boot / Import Configuration

After a while, my system booted into core.

At this point, we are done using iDrac / Physical console for configuration.

Assuming your system was able to acquire a usable IP address (I have multiple NICs configured, a few of which gave out a DHCP address), navigate to the configuration page, and log in with the password you configured earlier during the installation.

From this point, I would enjoy telling you to upload your previous configuration backup, however, my scale backup was deemed “too new” for core to import. This isn’t a big problem. Lets rebuild a new configuration.

I will start by importing my pools from scale.

After, a long time, the pool from scale was able to successfully import.

I continued importing my other pools.

After this, I went ahead and configured my 10/40GBe interfaces. I had no issues getting the 40GBe ConnectX-3 NIC up and running.

From this point, I went through each configuration section and slowly reconfigured everything… Emails, Network, DNS, SSH, NTP, etc…

After configuring everything, I rebooted the host. It rebooted and came back online without any issues or side-effects.

Step 7. Reconfiguring iSCSI

In the previous step, I went ahead and configured most of iSCSI. However, I could not select the existing zvols as extends.

To knock out some initial testing, I went ahead and deleted the old 50G testing zvol from my flash array, and used the wizard to go ahead and setup iSCSI.

After attaching/formatting the disk in windows, I went ahead and ran a test.

Granted, this test is against my Flash array, however, I will note the write performance is MUCH better then it was during the previous tests conducted when I replaced my core switch.

As well, I have performed no tuning, yet.

Results?

Overall, the migration was not too bad.

On a scale of 1-10, 10 being the hardest, I would give this a 6.

It would have saved many hours of reconfiguration If I were able to import my configuration from Scale. Since I was not able to, I had to go and manually reconfigured all users/groups/shares/jobs/etc. This was a pretty time consuming process.

I will note, my existing pools imported without a hitch, and I was able to get all of my iscsi mounts reattached after a few minor issues.

Getting SMB working again, took quite a bit more effort then I was expecting, however.

Summary

1. Was able to get infiniband working
2. IPoIB performance is utter garbage.
3. I didn’t want to hack up TrueNAS more to support RDMA
4. Gave up, went back to my Chelsio 40GBe adaptor.

Step 1. Install a few packages.

By default, TrueNAS Scale has apt “disabled”. Please see THIS POST on how to re-enable it.

Once you have… re-enabled the ability to install packages, AND added the debian repos, we need to install a few packages to assist.

At this point, you should be able to run ibstat and get a status. If you don’t have a status, make sure your card is installed. I am using a ConnectX-3 (Non-pro)

To note, I have one port set as infiniband, and the other port set as ethernet.

root@truenas:~# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x506b4b03007bfc50
        System image GUID: 0x506b4b03007bfc53
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x506b4b03007bfc51
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x526b4bfffe7bfc52
                Link layer: Ethernet

How to change configuration parameters

# First- get the PCI address of your card.
root@truenas:~# lspci | grep Mell
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

# In this case- 03:00.0 is my address.

# Show current configuration
root@truenas:~# mstconfig -d 03:00.0 query

Device #1:
----------

Device type:    ConnectX3
Device:         03:00.0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        IB(1)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)


# Set port mode to infiniband
mstconfig -d 03:00.0 set LINK_TYPE_P1=1

# Set port mode to ethernet
mstconfig -d 03:00.0 set LINK_TYPE_P1=2

# You can also set multiple properties in the same command
mstconfig -d 03:00.0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1

# Disable boot rom, if you desire. I don't have plans on using it.
mstconfig -d 03:00.0 set LINK_TYPE_P1=1 BOOT_OPTION_ROM_EN_P1=0 BOOT_OPTION_ROM_EN_P2=0

Step 2. Ping across the network

This guide ASSUMES you already have a working physical link established between two infiniband cards.

In my case, I have a windows client, and a TrueNAS server.

Next, lets lists connected hosts

root@truenas:~# ibhosts
ibwarn: [2618545] get_abi_version: can't read ABI version from /sys/class/infiniband_mad/abi_version (No such file or directory): is ib_umad module loaded?
ibwarn: [2618545] mad_rpc_open_port: can't open UMAD port ((null):0)
../libibnetdisc/ibnetdisc.c:802; can't open MAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed

Seems… a kernel module is missing. Lets add it, and try again.

root@truenas:~# modprobe ib_umad
# I also went ahead and loaded the infiniband IP over InfiniBand module.
root@truenas:~# modprobe ib_ipoib
root@truenas:~# ibhosts
Ca      : 0x0002c90300a29060 ports 2 "XTREMEOWNAGE-PC"
Ca      : 0x506b4b03007bfc50 ports 2 "MT25408 ConnectX Mellanox Technologies"

Well, look at that. The two PCs can see each other over infiniband.

The next step, was to allocate an ip address for the infiniband adapter.

ibp3s0: flags=4098<BROADCAST,MULTICAST>  mtu 4092
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As we can see, it doesn’t have an address. I provisioned the IP address through the TrueNAS WebGUI.

After saving the changes, applying, etc…. I was able to ping the local IP, but, not the remote IP.

root@truenas:~# ping 10.100.255.1
PING 10.100.255.1 (10.100.255.1) 56(84) bytes of data.
64 bytes from 10.100.255.1: icmp_seq=1 ttl=64 time=0.052 ms
^C
--- 10.100.255.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.052/0.052/0.052/0.000 ms
root@truenas:~# ping 10.100.255.2
PING 10.100.255.2 (10.100.255.2) 56(84) bytes of data.
^C

Doing a bit of troubleshooting, I ran ibstat on both ends.

and, determined the link was stuck in status “Initializing”

After a bit of research, I determined a subnet manager needs to exist, for a point to point link.

So…

root@truenas:~# apt-get install opensm
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  opensm
...

At this point, I decided to do a reboot, to ensure all of the modules are being properly loaded at boot.

Or… to see if…. all of the above work has been reset.

After rebooting, all of the configuration and packages were still in place, However, the kernel modules were not loaded back in. So… I reloaded the modules manually, and added a config to load them at boot.

root@truenas:~# modprobe ib_ipoib ib_umad ib_uverbs rdma_ucm
root@truenas:~# nano /etc/modules-load.d/infiniband.conf
# File Contents:
ib_umad
ib_ipoib
ib_uverbs
rdma_ucm

The next step, was to get opensm up and running.

root@truenas[/etc/modules-load.d]# systemctl status opensm
● opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers
     Loaded: loaded (/lib/systemd/system/opensm.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sat 2022-03-26 13:26:55 CDT; 8min ago
       Docs: man:opensm(8)

Mar 26 13:26:55 truenas.local.xtremeownage.com systemd[1]: Condition check resulted in Starts the OpenSM InfiniBand fabric Subnet Managers being skipped.


## I assume, it failed to start due to the missing kernel modules. Lets start it.

root@truenas[/etc/modules-load.d]# systemctl start opensm
root@truenas[/etc/modules-load.d]# systemctl status opensm
● opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers
     Loaded: loaded (/lib/systemd/system/opensm.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2022-03-26 13:35:38 CDT; 1s ago
       Docs: man:opensm(8)
    Process: 296934 ExecCondition=/bin/sh -c if test "$PORTS" = NONE; then echo "opensm is disabled via PORTS=NONE."; exit 1; fi (code=exited, status=0/SUCCESS)
    Process: 296935 ExecStart=/bin/sh -c if test "$PORTS" = ALL; then PORTS=$(/usr/sbin/ibstat -p); if test -z "$PORTS"; then echo "No InfiniBand ports found."; exi>
   Main PID: 296935 (code=exited, status=0/SUCCESS)

Mar 26 13:35:38 truenas.local.xtremeownage.com systemd[1]: Starting Starts the OpenSM InfiniBand fabric Subnet Managers...
Mar 26 13:35:38 truenas.local.xtremeownage.com sh[296935]: Starting opensm on following ports: 0x506b4b03007bfc51
Mar 26 13:35:38 truenas.local.xtremeownage.com sh[296935]: 0x526b4bfffe7bfc52
Mar 26 13:35:38 truenas.local.xtremeownage.com systemd[1]: Finished Starts the OpenSM InfiniBand fabric Subnet Managers.


## Much better.

At this point, I checked the stats for the infiniband nics.

root@truenas[/etc/modules-load.d]# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x506b4b03007bfc50
        System image GUID: 0x506b4b03007bfc53
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251486a
                Port GUID: 0x506b4b03007bfc51
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x526b4bfffe7bfc52
                Link layer: Ethernet


## Boom, good to go on the truenas side.

BADPROMPT:ibstat
CA 'ibv_device0'
        CA type:
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 0x0
        Node GUID: 0x0002c90300a29060
        System image GUID: 0x0002c90300a29063
    Port 1:
        State: Down
        Physical state: Disabled
        Rate: 1
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x0202c9fffea29061
        Link layer: Ethernet
        Transport: RoCE v1.25
    Port 2:
        State: Active
        Physical state: LinkUp
        Rate: 40
        Real rate: 32.00 (QDR)
        Base lid: 2
        LMC: 0
        SM lid: 1
        Capability mask: 0x90580000
        Port GUID: 0x0002c90300a29062
        Link layer: IB
        Transport: IB

Excellent, the link is active on both sides!

Since, the module was not active when the machine was rebooted, the interface was down. I used iconfig to bring it up.

root@truenas[/etc/modules-load.d]# ifconfig ibp3s0 up
root@truenas[/etc/modules-load.d]# ifconfig ibp3s0
ibp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4b03:7b:fc51  prefixlen 64  scopeid 0x20<link>
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 2  bytes 212 (212.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37  bytes 11759 (11.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@truenas[/etc/modules-load.d]# ping 10.100.255.1
PING 10.100.255.1 (10.100.255.1) 56(84) bytes of data.
64 bytes from 10.100.255.1: icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from 10.100.255.1: icmp_seq=2 ttl=64 time=0.049 ms
^C
--- 10.100.255.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.049/0.052/0.056/0.003 ms

We can now ping over IP. Lets do a speed test.

## Since, TrueNAS doesn't ship with iperf2 (Does include iperf3, however, getting speed results above 10G isn't consistent using iperf3, due to its single-threaded nature....
## So, I prefer iperf2 for testing over 10G.
root@truenas[/etc/modules-load.d]# apt-get install iperf
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  iperf
0 upgraded, 1 newly installed, 0 to remove and 105 not upgraded.
...
root@truenas[/etc/modules-load.d]# iperf -v
iperf version 2.0.14a (2 October 2020) pthreads
root@truenas[/etc/modules-load.d]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------


## on the windows side.
BADPROMPT:iperf.exe -c 10.100.255.1
------------------------------------------------------------
Client connecting to 10.100.255.1, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 10.100.255.2 port 23277 connected with 10.100.255.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.69 GBytes  6.60 Gbits/sec

BADPROMPT:iperf.exe -c 10.100.255.1 -P 10
------------------------------------------------------------
Client connecting to 10.100.255.1, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ 12] local 10.100.255.2 port 23297 connected with 10.100.255.1 port 5001
[ 10] local 10.100.255.2 port 23295 connected with 10.100.255.1 port 5001
[ 11] local 10.100.255.2 port 23296 connected with 10.100.255.1 port 5001
[  9] local 10.100.255.2 port 23294 connected with 10.100.255.1 port 5001
[  7] local 10.100.255.2 port 23292 connected with 10.100.255.1 port 5001
[  6] local 10.100.255.2 port 23291 connected with 10.100.255.1 port 5001
[  8] local 10.100.255.2 port 23293 connected with 10.100.255.1 port 5001
[  5] local 10.100.255.2 port 23290 connected with 10.100.255.1 port 5001
[  4] local 10.100.255.2 port 23289 connected with 10.100.255.1 port 5001
[  3] local 10.100.255.2 port 23288 connected with 10.100.255.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0-10.0 sec  1.07 GBytes   916 Mbits/sec
[ 10]  0.0-10.0 sec   899 MBytes   753 Mbits/sec
[  6]  0.0-10.0 sec  1.01 GBytes   870 Mbits/sec
[  8]  0.0-10.0 sec   220 MBytes   185 Mbits/sec
[  5]  0.0-10.0 sec   561 MBytes   470 Mbits/sec
[  9]  0.0-10.1 sec   695 MBytes   578 Mbits/sec
[ 11]  0.0-10.1 sec   674 MBytes   557 Mbits/sec
[  7]  0.0-10.2 sec   346 MBytes   285 Mbits/sec
[  4]  0.0-10.2 sec   253 MBytes   208 Mbits/sec
[  3]  0.0-10.2 sec   681 MBytes   560 Mbits/sec
[SUM]  0.0-10.2 sec  6.31 GBytes  5.31 Gbits/sec

Well, that was extremely disappointing!

Troubleshooting poor performance

The first step- is always to check MTU. I found the server was configured for 8,000, however, my client only supported 4092 max.

 root@truenas# ifconfig ibp3s0 mtu 4092 up

Just, to confirm this wasn’t an iperf issue, I decided to do a quick iscsi performance test.

Only, 5Gbits….. far short of the expected 40/56Gbits.

Well… lets doublecheck MTU.

root@truenas[/etc/modules-load.d]# ifconfig ibp3s0 mtu 4092
root@truenas[/etc/modules-load.d]# ifconfig ibp3s0
ibp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4b03:7b:fc51  prefixlen 64  scopeid 0x20<link>
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 23373377  bytes 47020734796 (43.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6652816  bytes 6411049427 (5.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

## Appears the card on my server only supports 2044 MTU... this might explain the issues.

So… I set the MTU on both ends to 2044.

Sadly, this did not help performance.

Conclusion.

I DID successfully manage to get infiniband up and running.

I DID learn, IPoIB (IP over InfiniBand) is actually quite slow. To realize huge performance gains over using the NIC in ETH mode, you would need to leverage RDMA/RCOE.

Since, I have hacked up my TrueNAS install enough for the day, I decided to switch everything back to ethernet mode, and call it a day. I can easily get 40G out of ethernet mode.

How to switch back to ethernet mode.

If you recall earlier, I left one interface in ETH mode…. I am just going to switch the physical adaptor, and move the ip address over… so, I don’t need to reboot my PC.

ifconfig ibp3s0 0.0.0.0 0.0.0.0
ifconfig enp3s0d1 10.100.255.1/30 mtu 9000 up
root@truenas[/etc/modules-load.d]# ifconfig enp3s0d1
enp3s0d1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4bff:fe7b:fc52  prefixlen 64  scopeid 0x20<link>

## On the windows side, I also physically swapped the adaptors over.

root@truenas[/etc/modules-load.d]# ping 10.100.255.2
PING 10.100.255.2 (10.100.255.2) 56(84) bytes of data.
64 bytes from 10.100.255.2: icmp_seq=1 ttl=128 time=0.482 ms
64 bytes from 10.100.255.2: icmp_seq=2 ttl=128 time=0.162 ms

## Connection is good to.... Lets test performance.
[SUM] 0.0000-10.0208 sec  10.7 GBytes  9.15 Gbits/sec

## Appears my client is only connected at 10G.... interesting.

root@truenas[/etc/modules-load.d]# ethtool enp3s0d1
Settings for enp3s0d1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

## Wonder what happens if I uh, FORCE it to 40g.

root@truenas[/etc/modules-load.d]# ethtool -s enp3s0d1 speed 40000 duplex full autoneg off

## Turns out, the connection on the windows side dropped.... I am starting to think this particular ConnectX-3 does not support 40G. 
## I know the adaptor on the server side supports 40G, because, its the same exact adaptor i used to benchmark 40G in a previous post!

I found documentation on my particular NIC here: https://downloads.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_pedge_srvr_ethnt_nic/mellanox-adapters_users-guide_en-us.pdf

Based on the specifications, it “should” support 40G ethernet mode…

Edit- Turns out, it supported 40G infiniband, but, only 10GBe ethernet.

Yup. Oh well. Putting the chelsio NIC back in my PC. It had no problems doing 40G.

A while back, I did a 10/40G Network Upgrade. This involved adding a new Brocade ICX6610 core switch, capable of handling line speed routing and forwarding, across all ports.

You can see the previous state of my rack with pictures and details in my Rack em Up post.

While, this switch is an absolute unstoppable monster, it comes with two drawbacks. It makes a lot of noise, and it uses a lot of energy. My calculations place its energy consumption around 200w, which is nearly what my massive server consumes.

So, the plan….. is to completely replace it, WITHOUT degradation of services.

I plan on doing this by replacing the ICX6610 completely, and moving all routing up to my firewall, where previously, my ICX6610 handled routing between various subnets.

Getting Started

New Core Switch

Sometime earlier this year, I ran out of POE ports and acquired a Zyxel GS1900-24EP 24 port gigabit POE switch. While, there is nothing fantastic about this switch, it can do vlans, and management POE devices. It does exactly what I ask of it. The key piece here, it is silent, and has low power consumption.

The downside, it has no 10G or faster uplink ports.

I decided to reinstall a old quad port gigabit NIC into my Opnsense firewall. I then, ran a 4-link lagg from the firewall to the Zyxel.

Afterwards, I moved my POE cameras, APs, Livingroom Switch (powered via poe), iDrac, and other gigabit loads to the zyxel. I pointed the vlans in opnsense from the old 10G lagg, to the new lagg connected to the zyxel.

After a bit of configuration, I had all of my security cameras, IOT devices, etc working as expected.

How to maintain 10G connectivity from my PC, to my NAS

One of the big reasons I upgraded to 10/40G this year, was to ensure I had a very fast iSCSI connection to my NAS, allowing me to use storage from my server. I use this to host my secondary steam library for less often played games, and other bulk-storage.

However, without a 10/40G switch into place, I decided to leverage the two 10G ports already on my opnsense firewall. I dedicated one of the 10G ports to my TrueNAS server, and I dedicated the other 10G port to my bedroom/office.

While, this is far less then the 40G of dedicated bandwidth my server had before, I don’t imagine this should be a bottleneck very often… if ever. One of these days, I still plan on running a dedicated 40G link just for iSCSI.

Initial Benchmarks

So, after moving everything over, and setting up a new network for my office on opnsense, I decided to run benchmarks to determine the level of performance…. Since, my traffic now has to be routed, instead of switched.

Initial Network Benchmark

I have grown to enjoy using openspeedtest for doing quick, simple network benchmarks.

Without doing any configuration, the network benchmark isn’t horrible. I am getting 70%-80% of the expected download, with nearly 100% of the expected upload.

Initial ISCSI benchmark

Around 1,200MB/s is the maximum speed obtainable over a 10G connection. However, I am getting far below what is possible. The write speed is literally a quarter of what the network is capable of.

While, my main ISCSI mount is only a 8-disk Z2 array, these results are… quite low.

Adjust MTU

During the process of moving around networks, I did finally adjust the maximum MTU on my server to 9,000. It was set at 1,500 before due to a mixture of devices on LAN segment which did not support jumbo frames. However, since my bedroom/office now has a dedicated lan segment, I am able to fully leverage jumbo frames without additional impact.

So, 9,000 MTU, here I come.

After adjusting the MTU size, the first test was to ensure the jumbo packets are able to successfully navigate to my NAS without fragmenting. This can be accomplished with a simple ping comment, and the do-not-fragment flag set.

XtremeOwnage:ping 10.100.4.24 -f -l 8955

Pinging 10.100.4.24 with 8955 bytes of data:
Reply from 10.100.4.24: bytes=8955 time<1ms TTL=63
Reply from 10.100.4.24: bytes=8955 time<1ms TTL=63
Reply from 10.100.4.24: bytes=8955 time<1ms TTL=63

Ping statistics for 10.100.4.24:
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C

Network test with Jumbo Frames

ISCSI Benchmark after Jumbo Frames

MUCH better! I am at least getting full read speeds now.

My write speeds, are actually likely correct, given this is a single 8-disk Z2 Array, without SLOG enabled. As well, this is not a clean room test. My array is running its normal production load. If you are reading this website, it is hosted on my server, and the same storage I am benchmarking right now.

Will SLOG Help Performance?

Lets find out!

Sometime in the last month or two, I replaced my boot pool with a pair of 600G 15k SAS drives. This freed up a extra NVMe drive in my pool.

I do not at ALL recommend only using a single drive, ESPECIALLY, a consumer-grade SSD for your SLOG. If this device fails, you will have potential data corruption issues!!!

I am only going to use it for the duration of this test.

First, make sure you are forcing sync writes. You will want to set Sync=Always for the ZVOL you are testing with. (Or, your entire array if you plan on keeping an SLOG device.) However- I am only interested in testing to see if there is a noticeable performance difference.

I decided it may be useful to perform another benchmark at this time, with the only change being to force sync=always.

As I stated before, it is extremely recommended to never use a single device for your SLOG. I had to ignore the errors and force the configuration.

Post SLOG Benchmark

Since, the SLOG device is now apart of my pool, lets once again run a benchmark to see how performance is.

Well, performance is better then it was with sync=always, but, less then it was before we touched it.

Lets… remove sync=always, and see what the outcome is.

This backs up my previous opinion that a SLOG is useful situationally. At this point, I went ahead and removed the SLOG device from the pool.

To note, this is not a perfectly fair test of a SLOG. A 970 evo, while a fantastic NVMe for your home PC, is NOT a suitable device for a SLOG. Ideally, a Intel Optane 900P would be leveraged. These are much more suitable due to vastly improved durability, and reduced latency.

Testing Against NVMe.

Just as a final test to ensure I am getting the expected ISCSI throughput being routed through my firewall…. I went ahead and setup a 50G iSCSI mount on my mirrored NVMe array.

Interesting. The write speeds are still half of what I expected.

Enabling File/Folder Compression on Windows

I assumed perhaps windows File/Folder compression may have a difference. So, I preformatted the drive and enabled compression.

However, this actually reduced performance. Next, I went to TrueNAS, and disabled compression there. This was as simple as editing the zvol, and setting compression to none. After doing this, I also re-formatted the drive from the windows box, and disabled File/Folder Compression.

Read speeds are back to normal. However, no improvements on write speeds.

Final iSCSI results.

At this point, I decided to call it quits for now.

Worst-case scenario, the sequential read on my 8-Disk Z2 array is still twice as fast as a SATA SSD, and achieving the same read speeds. I decided to perform a full benchmark to determine the random speed of my spinning array as well.

Compare this to the budget gaming PC I built a while back, which used SATA SSDs.

Benchmark Size Disclaimer

I personally ran most of the below benchmarks at 2G size. Because, I personally do not care to wait 10/20/30 minutes for benchmarks to complete. To demonstrate the difference in benchmark results with various sample sizes…. I performed various benchmarks at increasing sizes.

Here are benchmarks performed at the end of my testing, of various sizes, against my 8-disk Z2 array.

As noted in the above benchmarks, write speed remains extremely consistent. Read speed goes down as sample size is increased. To note, my server has 128G of ram. For real world use-cases, there isn’t a lot of content that will not be fully cached in ram.

Power Consumption Differences.

Using Iotawatt, which tracks the power consumption of many of the various circuits in my home…. I am able to see the exact difference in power consumption used by my server rack.

Based on the data provided, It is safe to say, my rack is consuming on average, 100-150 watts less then it did before.

510-520watts was the average minimum consumption before the change. After the change, 400 watts appears to be the new minimum.

While, the total reduction was not AS much as I would have hoped,

(100 watts * 24 hours * 365 days) = 876kwh of consumption per year.

876kwh * $0.08c/kwh = 70$ less spent on electricity per year. Or… around 6$ a month total savings.

To note, this does NOT include how much it costs to remove the extra heat from the house. That number should be around 25% of the total cost. So, I will ballpark and say, replacing the brocade with my preexisting zyxel saved me 100$ a year.

I could have potentially reduced that number more by using smaller, more efficient switches. However, my unifi switch does not have enough POE ports to satisfy my POE needs.

Firewall Impact

Since, all of my 10G traffic is now being routed through the firewall, and passing through ACLs, it was also important to document the impact to the firewall’s performance.

So, I captured the dashboard while performing a iSCSI test. The first large spike was the READ test, followed by the WRITE test. As noted, the firewall’s CPU utilization remains…. basically idle.

I believe it is safe to say, my firewall is hardly affected by this large volume of traffic.

Closing Notes

I was able to successfully reduce my yearly electric bill by nearly 100$.

I actually IMPROVED my 10G performance with this network change. As the network was setup before, I was not able to safely increase the MTU size, without causing impact to other devices not capable of handling the increased MTU size.

I do plan on investing further to attempt to determine why my iSCSI write speeds are lower than expected as well. However, for now, It is faster then it was before.

Often, I see a bunch of posts where someone is having a tough time considering between Unraid, or TrueNAS. so, I am going to write a quick summary based on my experiences, and recommendations.

This article is basically copied from my comment on reddit.

Unraid

+ You add disks, you gain storage, simple as that.

+ Its method of write caching. You add NVMe to a dedicated flash pool, and all of your writes will funnel through the NVMe cache. This is actually a really nice feature. With Truenas, you write directly to your array.

– Performance. Unless your data lives on your flash cache, you are limited to the performance of a single drive. This was my biggest driver for moving from* unraid. Single spool performance is not that great.

+ Resources. You don’t need many resources to run a decently performant unraid install.

+ UI. Unraid has a FANTASTIC UI. Very easy to use and navigate.

+ Power Usage. Only the drive which contains the file you are using needs to be spun up. If the file is on flash, you don’t have to spin up any drives.

+ Tiered Storage. Unraid’s flash/array system is FAR from perfect. However, it does give you a very rudimentary system of tiered storage.

TrueNAS

Adding storage in the CURRENT state, basically means adding another vdev. (I aim for 8-10 disk vdevs) You CAN replace drives one by one, to increase capacity, but, unless you are replacing old 2TB disks with 18T disks, its generally not worth doing this. In the future, online expansion will be added removing this CON.

+ Performance. If you have a 8 disk VDEV, your data is read from, and written to all 8 disks at once. If you have plenty of ram for ARC, reads are generally lightning fast.

– Resources. If you run TrueNAS, RAM RAM and then add MORE RAM. Ram is everything. Also- use ECC ram.

– UI. Scale is still in beta, bugs are being worked out. But, Its UI still doesn’t compare to how easy it is to use Unraid.

– Power Usage. When you read or write something, your entire arrays has to spin up.

– Tiered Storage. Non existent in truenas.

Which is better?

Neither. Each one fits into a unique niche.

For me to recommend one or the other, I would need to know your specific needs.

If you are not highly technical, and you just want a NAS that just “works”, You cannot beat unraid. From its simplistic UI, up to its storage model of being able to just add disks to expand, its easy. You want a docker container? Click the docker tab and add it. Within 30 seconds, you have a new container up and running. You want a VM? Click the VM tab. Easy enough.

Want to add a share? Three more clicks and you are off to the races.

If you are a technical user, who doesn’t mind tinkering, I personally love TrueNAS SCALE

But, in the current state, I would say its better for power users and more technical individuals.

Which one is faster?

Also- neither.

For spinning disk array performance, TrueNAS is hard to beat. Unraid falls pretty flat on its face in this category, only using a single spindle. However, Unraid has the ability to cache reads and writes with NVMe storage, which greatly minimizes this issue.

Note- before somebody mentions the L2ARC from ZFS, it is not really comparable in this situation. It only caches reads, not writes, and it is VERY SITUATIONAL. In most cases, it will hurt your performance rather then helping unless you have a very diverse workload, and LOTS of ram!!!!

What is my personal choice?

Simple. I like TrueNAS. Why?

1. Performance. The unraid single spool bottleneck was becoming a pain in my ass.
2. Reliability. If you give truenas the proper hardware, its EXTREMELY reliable. I have a lot of faith in ZFS for maintaining my data integrity. I never actually got a chance to test how well Unraid can recover from a failed drive, however, I have experienced many drive failures with TrueNAS. The last failure, was one of my 8TB drives dying. Access to my data was never lost during the failure. It was degraded in performance, however, but, the missing drive was actually rebuilt from one of my array’s hot spares in ~24h with zero loss to my data.
3. ZFS > BTRFS IMO. BTRFS raid levels are not reliable as said by the developer. Don’t use BTRFS Raid5/6.
4. Power. I feel like TrueNAS gives me MUCH more control over everything to a point where it can be dangerous.

Seriously, both are good platforms to choose from.