My blog, is not going away- I am just moving to other software

Why?

I posted on the new site, HERE surrounding my reasons to migrate away from wordpress to a new static site design.

To summarize the post- Security, Privacy, and Performance.

Security

WordPress nearly always has a new vulnerability, in either its core, or via a 3rd party plugin needed to add desired functionality. By shifting to a completely static design, there is basically nothing which can be exploited.

As well, since the new site is completely static, with no dynamic content whatsoever, It can be hosted by a service such as… Cloudflare pages, github pages, etc.

Privacy

While- I try my best to keep a hand on the data of my users, in the event this site is PWNED, I cannot promise, nor guarantee the privacy of my users. As such, in the new site, I have full control over exactly what data goes where.

Performance

While I have done numerous optimizations to make this blog very responsible and performant- It will never be able to compete with a fully cached/static site.

Where

New posts will be done at https://static.xtremeownage.com/

New post notifications

For those who rely on wordpress to send notifications of new content, The only option I currently have for you is to setup a RSS feeder, and keep an eye on the NEW RSS FEED.

If you wish to self-host a feeder, FreshRSS is amazing.

If you wish to use a cloud-service, Feedly.IO works well.

Outlook also has built-in support for RSS feeds.

When

I have already started posting new content at the static site months ago. At this particular point in time, I have already migrated over a few handfuls of content… and will continue migrating content as time goes on.

For content which has already been migrated, you will be redirected automatically to the new location.

You don’t.

However, there are many precautions you can take when running a public WordPress site, which will reduce the chances of you being PWNED, or, at least, greatly mitigate the damages.

WordPress itself, has frequent security issues. This site itself, isn’t immune to them. However, over the course of the last decade running this site, it has so far, only managed to be pwned once…. and that was due to a plugin which introduced a vulnerability. The impact was limited, due to precautions I will discuss below… and it took me all of 10 minutes to get everything back up and running.

Use a dedicated DMZ.

A DMZ, is typically a subnet reserved for “Untrusted services”.

It is best to treat any publicly available service on your home network, as thought it WILL be exploited, and attackers WILL be able to gain full control over it.

Step one, is to run your public services as either a dedicated server, or a dedicated VM. You want to limit this server’s network access to only the bare minimal required for its operation.

1. Internet Access – Your server will need to be able to access the internet.
2. DNS – To limit exposure of potential vulnerabilities on your local DNS server, I would recommend using a public DNS server, instead of your internal server….. Just in the event this box gets PWNED, and your internal DNS server has a potential vulnerability.
3. NTP – Again- I recommend using public NTP services, rather then your internal services, in the event you have a vulnerability in your NTP server.
4. Allow NOTHING else. This server should be able to ping the internet, but, have NO access to any of your internal services.

I personally, leverage vlans to control my DMZ network. The switch enforces, only properly tagged traffic is accepted on the port. Even better, would be to step it up and use a dedicated switch for your vlan.

Keep everything up to date.

If you intend on exposing services to the internet, you NEED to keep everything up to date.

This means, you need to keep your firewall up to date. You need to keep your server up to date, and, you need to keep your wordpress up to date.

For me, this blog you are currently reading, is hosted on a linux server, inside of my DMZ. Since, I don’t log into my server daily to check for updates, I setup AUTOMATIC security update installations.

The above step, also includes updates to your php libraries.

Your WordPress itself, also has updates. I enabled automatic platform security updates to keep it up to date.

Most importantly- your WordPress PLUGINs are THE most common attack vector for WordPress. I cannot say I would recommend automatic updates here, but, I would ensure you keep a very close eye on them.

TLDR; You need to keep everything up to date for security patches.

Limit Attack Surface

Some endpoints, such as /wp-admin, have no functionality which a general user should have access to. I would recommend blocking access to any requests hitting /wp-admin, and locking this behind a trusted IP / SSO. For an example on how this would look, visit https://xtremeownage.com/wp-admin

I personally, leverage cloudflare’s tools for providing a layer of SSO on top of this.

From your PHP configuration, I also recommend disabling php info, and headers which may give out version information. Knowing the particular version of software running on a server, makes it much easier to find relevant vulnerabilities for that particular piece of software.

Use Secure/Unique Passwords

If say, you use a common and simple password such as “hunter2” for all of your internal services.

Lets say, an attacker managers to get access to the local server running your blog. Lets assume they managed to find a way to escape your DMZ, such as… you neglecting to block a certain service, neglecting to apply your security updates, etc….

Now, they have access to your network. When they do determine you use the credentials admin // hunter2 for all of your internal services…. they have just obtained access to everything.

My recommend here, is to use a password manager, and use unique/strong passwords for all of your internal accounts. There are both paid, free, and self-hosted services you can use for managing credentials.

I would personally, recommend either paying for such a service, or hosting your own…. assuming you can keep it secure.

Have Backups

The MOST IMPORTANT thing you can do when hosting a public service, always assume it will be pwned, and always assume everything on the box will become a complete loss.

To mitigate this risk, have backups.

In my case, assuming the server hosting this blog becomes a total loss, I have the capability of rolling back to a previous snapshot (which are taken daily), or pulling a backup out of cold storage.

Anything worth having, is worth backing up.

I am not going to go in depth on how to make proper backups- but, having a folder on the same server, named “backups”, isn’t going to cut it… If someone can gain access to your server, and delete your backups, your “backups” aren’t going to help you.

Ideally, you want your backups stored offsite. There are services such as backblaze which can assist you with this.

Use Cloudflare Argo / Access

I personally, recommend cloudflare argo tunnel.

It allows you to expose a service on your local network, WITHOUT opening a port on your firewall. It does this, by doing a reverse proxy from your network, back into cloudflare’s network.

There are a few advantages to doing this.

1. It hides your actual IP Address

While- your actual IP might not seem like a huge thing to give out- I personally don’t want you to know roughly where I live. Even knowing the general vicinity of where you live, causes a small attack surface… such as being able to figure out your zip-code, ISP, etc.

While- that information by itself, usually isn’t an issue at all….. if you are ever targeted by a dedicated “hacker”, you want to disclose as little information as possible.

Another thing to consider- your security questions you provide on many websites, generally have a question of, where did you grow up? What school did you attend, etc.

Assuming, you may live around the area you grew up- by giving out your public IP address, you just gave away a few potential pieces of information.

Also- don’t assume you can KEEP your ip address hidden. There are occasionally vulnerabilities/exploits which will disclose this information.

The second advantage- you can add an additional layer of authentication around “sensitive” endpoints, such as your admin settings.

Lastly, I personally use cloudflare to block a few countries…. China, Russia, Belarus, for example… Because I receive a TON of vulnerability scans from those countries. Also- this particular website does not cater to those countries.

“Security” plugins

There are many “security” plugins you can acquire for wordpress, which will help mitigate some risk. Jetpack, for example, has its own security plugins. Cleartalk does a great job of blocking spam and a few other potential security issues.

I do advise you to look around…. the cheapest option here, is not always the best.

Also, these plugins aren’t going to be the be-all/end-all solution, as they DO rely on WordPress itself… which might have an unmitigated vulnerability.

Fail2Ban

I enjoy the use of fail2ban for detecting vulnerability scanning attempts in my logs. Detecting scan attempts is quite easy. Look at the 404 not found records being logged on your server.

I receive THOUSANDS of such attempts every single day on this site. Using fail2ban to temporarily block the IP greatly reduces the chance of a valid vulnerability being located.

Why temporarily? Hackers/Scammers frequently rotate to new IPs. Chances are- the one scanning now, won’t be in use in a few weeks.

If you have a sensor of humor, you can get very creative with a reverse proxy / .htaccess / nginx configuration.

Example, https://xtremeownage.com/backup.gz

Security Event Notifications

I personally, recommend a solution for syslog traffic. My favorite solution, happens to be Splunk.

You can also use logstash/influx/other service.

The service you choose isn’t the important part. The important part, is having notifications for when something “unexpected” happens.

Generally- there shouldn’t be any logins to your servers, unless you are logging into them to do maintenance. As such, setup a notification for new/unexpected logins.

One of the most important steps here, is to just ensure you have a way to be aware something is wrong.

The most important tip

The most important thing to remember- you CANNOT perfectly secure your public facing WordPress site. There are always new vulnerabilities occurring. Your best line of defense, is to PLAN on it being PWNED, and to have steps in place to mitigate the amount of risk / reduce the amount of time to recover.

As well, ensure you are logging requests to a location where the attacker has a much, much harder time of finding / removing the logs.

TLDR;

1. Keep everything up to date.
2. Treat your DMZ Server hosting wordpress, like it has an attacker already on it. Keep it isolated from EVERYTHING else.
3. Keep backups of everything.
4. Maintain logs and paper trails.

Why am I posting this article?

So you don’t end up like THIS GUY.

For over a year, I have been wanting to experiment with a wall-mounted tablet for viewing and controlling my home automation.

This week, I caught Amazon’s Fire HD10 tablets on sale for 60$ a piece, and decided to pick up a couple.

Parts / Subscriptions Required

A Tablet

For this article, I will be using a Fire HD 10 Tablet. You can pick them up as low as 50/60$ (for a refurbished one)

I will note, I have previously tried to get everything up and running on older tablets, but, I had pretty bad luck due to their web browsers being pretty far out of date…. and thus, not working too well with home assistant.

Why a fire tablet? Simple. You can pick them up quite cheap.

(Optional) A Wall Mount

This item is optional. However, I wanted a nice looking wall mount to hold the tablet. I personally went with THIS OPTION for ~ 50$ If you have a 3d printer, you can print your own case from THINGVERSE.

I will note, while I did order this mount… as of the time of writing this article, I leveraged a piece of 3m double sided tape to do the job…. since this mount is a week or two out in the mail.

(Optional) A Smart Plug

Completely optional. however, if you want to automate charging the battery, you will need a smart plug.

For this…. you can use anything that integrates with home assistant.

KASA Plugs – These are still local control as far as I know. I still have 5 or 10 of them around my house.

There are also plugs using z-wave, zigbee, etc. Up to your personal preference if you choose to do this, and what plug you choose to do it with.

Fully Kiosk

While you don’t HAVE to buy this subscription- It costs a one-time payment of around 7$ USD per device to unlock all of the features. Trust me- some of the additional features are worth having.

To note, you can fully try everything without having this license. Please feel free to experiment before buying this software. The only limitation without an active license, will be a large watermark across the screen reminding you to acquire a license.

To also note, this post is not affiliated or sponsored by Fully Kiosk in any way whatsoever.

HACs Integrations/Front End Modules Used

Fully Kiosk

This integration handles getting fully kiosk fully integrated with home assistant.

Mushroom Cards / lovelace-mushroom

These cards are pretty to look at, and are extremely suitable for a touchscreen interface, such as this tablet.

Kiosk-Mode

This card can hide side bars, menu bars, and more allowing you to more easily build suitable dashboards for mobile devices.

Slider Button Card

Thanks for the tip by r/filtertuple.

This card makes for a great sliding control for my HVAC.

Getting Started

Step 1. Acquire a tablet.

Acquire the tablet. Charge the tablet.

Configure the tablet.

UPDATE the tablet.

Once the tablet is charged, up to date, and configured, move onto the next step.

Step 2. Enable Development Mode on the Tablet

We will need to enable development mode in order to install unsigned apks (fully-kiosk).

Go to Settings, Device Options, About Fire Tablet.

Once you are here, click on the serial number field about 5 or 10 times. You will see a prompt appear at the bottom of the screen.

Go back one menu, back to device options. You should now see Developer Options.

Go into developer options.

Turn it off (at the top).

Scroll down slightly to the debugging section. Enable USB Debugging.

Installation & Configuration of Fully Kiosk

Install Fully Kiosk Browser

On the tablet, visit this page: https://www.fully-kiosk.com/en/#download-box

You can also google, “Fully Kiosk Browser” to get to this location.

You want to download, “Fully Kiosk Browser”. Click on GET APK File.

After you download this, you will receive a prompt regarding unsigned device. You will need to grant the silk browser the ability to install unknown apps. Afterwards, the app should install. Open it.

Network Configuration (Optional)

I personally want to leverage this tablet on a dedicated IOT network, where it will have no network access.

For my home network, I have a dedicated SSID for this network. I configured the device to connect to this new SSID, and gave it a static IP. The static IP is a bit important, as it ensures your device retains the same IP when home assistant tries to talk to it.

Blocking… the device from the internet, also keeps amazon from sneaking in unwanted updates… or advertisements.

Note, if you purchase a license, the device will need internet access initially to fetch the license. You can disable internet access once it is activated.

Configure Fully Kiosk Browser

The first step, was to set the default URL. I set this to https://hass.mydomain.com/

Next, I created a new non-administrative user account within home assistant, with local sign-in only, and used this new account to sign in on the tablet, remember to click the button, “Keep me signed in”

At this point, you should be able to see your home assistant dashboards on the fire tablet without issues.

BUT, we are not done yet…

Enable Fully Kiosk Remote Access (PLUS Only)

Note, Plus only, means you need to have a paid license for these features.

Swipe your screen from the left, which should reveal the fully kiosk menu.

Click Settings.

Go to Remote Administration (PLUS)

Enable Remote Administration. Set a password. Note the remote administration URL.

After which, back out of the settings screen, and you will be presented with a bunch of permissions prompts. Accept all of the permissions, and grant fully kiosk all of the other special required permissions.

From a computer on your network, browse the the remote administration URL from earlier.

From here, we can continue the process of configuring the app, without needing to interact directly with the tablet.

Enable Start on boot (PLUS Only)

Under Device Management, there is a checkbox for Launch on boot. You will want to select this.

Set Power Settings

Under Settings -> Power Settings

Note- You will want to initially set the wakeup/sleep schedules via the app instead of the remote UI.

I scheduled the device to wake up at 7:30am every day, and go to sleep at 10:30pm every night. This should correspond with times I will actually be awake.

Next, I enabled the battery warning. I set this value to 20%.

Kiosk Settings (PLUS Only)

Under Settings -> Kiosk Mode

I enabled Kiosk mode. This will force a pin to exist kiosk mode. Note, I had to configure this via the app itself. I setup the exit gesture to require 5 taps on the screen.

After saving this, I needed to grant additional notifications permissions to fully Kiosk.

I also disabled all of the buttons, notifications, calls. I left the camera active for the motion detection feature.

Configure Motion Detection (PLUS Only)

Under Settings -> Motion Detection

I enabled motion detection, and enabled turning on the screen when motion is detected.

I also enabled, “Turn screen off in darkness”. If the lights are out, there is no need for this tablet to be active.

Configure “Other Settings”

Under Settings -> Other Settings.

I enabled Daily Usage Stats, Environmental Sensors, and MQTT.

For MQTT, I slightly modified the topics to include the prefix of kiosk.

After saving the settings, the device instantly appeared in my broker. The devie does send motion events via MQTT as well.

Add Fully Kiosk To Home Assistant

Install Fully Kiosk Browser Integration via HACs

Self explanatory- But, install the Fully Kiosk Browser Integration via HACs. Restart home assistant afterwards.

Add The Integration

After home assistant has been added, Navigate to Settings -> Devices & Services -> Integrations.

Click Add Integration

After adding in the static IP you were told to add earlier, and your remote administration password, the integration will be added.

With the integration, you have the ability to manage the tablet from home assistant. You can start apps change the browser… turn it on/off, reboot it… whatever you wish!

As a bonus, you can also transmit text to speech through the tablet.

At this point, I modified a few of the scenes for my office, to turn the tablet on while my “Working” scene was active, and turn it off when my “Away” scene is active.

You can adjust the screen brightness for each scene.

Charging Automation

Some people will tell you to add a smart plug, to maintain the battery level between 20 -> 80% or so.

If, you so choose to do this, Fully Kiosk does expose the battery percentage via the integration. You can easily toggle a smart plug to control the charging this way.

If you choose to do this, I would set an automation to turn on the smart plug when it is below 20-40%, and leave it on until the device is > 80%, at which point I would disconnect the charger.

I personally, will not be doing this at this time. This device will remain plugged in forever, while sitting on my wall. While this can potentially cause battery degradation, to re-clarify, it will always be plugged in.

Create Tablet-Specific Dashboards

So, my primary dashboard isn’t exactly suited for the lower resolution of this tablet.. It was developed with a 4k screen in mind. As well, I want this tablet tailed more specifically for my office.

So, I will be taking the relevant elements from various dashboards to create a panel tailored specifically to the area where this tablet will be located.

I will bring attention to the kiosk-mode addon which is available via HACs. It can hide headers, sidebars, and more for your tablet user.

For my mostly-finished product-

While, this, doesn’t look very… appealing on the web, Capturing a screenshot from a fire tablet is a bit more challenging than expected… especially, with a background video playing.

For an actual picture-

I will note, the space is almost perfectly filled up. I have instant navigation to sections of home assistant where I may need to go via the buttons on the left. I have access to all of the relevant scenes for this room on the bottom left.

The top-center panel will display any alerts I may need to be aware of. Batteries running low, devices not working. Fridge/Freezer getting too hot. Home battery running low.. Fire alarms going off… etc.

Under that, I have controls for this room’s lighting and HVAC systems. To the right, I have my home security. I have alarm control, as well as instant access to the primary security cameras on the exterior of the house.

Overall, this dashboard will likely grow and be expanded in the future. However, for now, it has all of the basic needs I would leverage while I am in the office.

For layout, the above dashboard uses…

1. Single panel dashboard view, with a horizontal card with 3 columns being the only panel.
2. Each column, contains a verticle stack.
3. The buttons on the left, leverages the grid card.
4. The lamp and alarm control leverages lovelace-mushroom.

My only big complaint right now- the thermostat card takes up a lot of screen real-estate for its given function.

7/13/22 – This issue was resolved by the slider button card, which gives a nice slider for setting the temp.

Kiosk-specific automations

Restart when “waking”

I noticed occasionally when I got up in the morning, the tablet would be displaying a white screen. After some investigation, refreshing the page would not resolve the issue. However, restarting the browser would. So…. I made an automation to do this.

Quite simply, when the screen turns on, it restarts the browser.

alias: Kiosk - Restart Browser On Wakeup
description: Kiosk likes to wake up with a blank/white screen.
trigger:
  - platform: device
    type: turned_on
    device_id: 8c55dffb0aedf92bcc0c63f9cbc6f9df
    entity_id: light.bedroom_kiosk_screen
    domain: light
condition: []
action:
  - device_id: 8c55dffb0aedf92bcc0c63f9cbc6f9df
    domain: button
    entity_id: button.bedroom_kiosk_restart_browser
    type: press
mode: single

This turned out to be a simple, yet stupid solution for fixing the issue…. and it does work.

Announce Alarmo State

A simple idea, but, I wanted the kiosk to announce the state of my alarm when it changes.

IE, when you set the alarm to “Armed Home”, the kiosk will say, “Alarm Armed Home.”

Very simple automation, but, was very effective for the task at hand.

alias: Kiosk - Announce Alarm State
description: ''
trigger:
  - platform: state
    entity_id:
      - alarm_control_panel.alarmo
condition: []
action:
  - service: tts.cloud_say
    data:
      entity_id: media_player.bedroom_kiosk_media_player
      message: >-
        Alarm {{ states('alarm_control_panel.alarmo') | regex_replace(find='_',
        replace=' ', ignorecase=False) }}
mode: single

Display NVR Popup when motion is detected

When people are detected outside of my house, I wanted to automatically announce via the kiosk, and display a live video feed of the cameras.

For this, I used the binary_sensors provided by the frigate integration for when a person is detected. When detected, I leverage the TTS service to announce motion, and then tell fully kiosk to show the blue iris footage.

After 30 seconds, it reverts back to the primary dashboard.

alias: Kiosk - NVR Popup
description: ''
trigger:
  - type: motion
    platform: device
    device_id: 3b6e1cf99252fc44fbd4ca2159ca609d
    entity_id: binary_sensor.front_door_person_motion
    domain: binary_sensor
  - type: motion
    platform: device
    device_id: ef15f114880855207dfefc401dfda68c
    entity_id: binary_sensor.front_driveway_person_motion
    domain: binary_sensor
  - type: motion
    platform: device
    device_id: ec6af8f8edaa09c3c46d4d9dec430ee1
    entity_id: binary_sensor.mailbox_person_motion
    domain: binary_sensor
condition: []
action:
  - service: tts.cloud_say
    data:
      entity_id: media_player.bedroom_kiosk_media_player
      message: Front Motion Detected
  - service: fullykiosk.load_url
    data:
      entity_id: media_player.bedroom_kiosk_media_player
      url: https://nvr.xtremeownage.com/ui3.htm?group=Front
  - delay:
      hours: 0
      minutes: 0
      seconds: 30
      milliseconds: 0
  - device_id: 8c55dffb0aedf92bcc0c63f9cbc6f9df
    domain: button
    entity_id: button.bedroom_kiosk_load_start_url
    type: press
mode: single

During the LAST POST, I reduced the average power consumption of my r720xd running truenas from 400 watts, down to around 200 watts running. This was a pretty big reduction. Most of the reduction was caused by moving my loads to newer, slightly more efficient servers to run docker. The remaining reduction was made possible by removing the 2nd CPU.

For this post, I am going to try and further reduce the power consumption of my server, without butchering performance.

Baseline

Based on iDrac’s reported power consumption, this server averages around 215 watts running normally.

So, for the baseline, We are going to go with 215 watts as the target number to beat.

TrueNAS – Configure Drive Power Management

At this point, I do believe a big chunk of the overall power consumption is from my HDDs themselves. However, I do not want them to spindown, risking increased wear.

However, there are some options we can tweak to potentially reduce power consumption, without spinning down the disks. It appears, by default on scale, power management features are disabled.

So, I am going to specify level 128, which should offer the best power consumption, without causing the drives to spin down.

I went through all 13 spinning disks in my system, and adjusted the power consumption to 128.

I do have a couple(2) other drives which are not currently in use on my system, for any reason. I set those drives to level 1, with standby enabled.

After adjusting the power options, off the bat, it appeared to have reduced overall consumption by 20 watts (195w, down from 215w). However- I need to leave the server alone for a few hours at least to establish a proper baseline.

A few hours later…

It is safe to say, we saved around 10-15 watts by enabling power management features. Not a dramatic amount, but, every little bit helps.

Results – 10-15 watts saved.

Installing TrueNAS Core

I replaced Scale with Core for performance benefits. However, this actually had the downside of increasing the average power utilization. But, it did improve performance pretty significantly.

These performance benefits will come in very handy when I replace the processor.

Result – Negated previous step. But, improved performance.

r720XD – Swapping E5-2695v2 with E5-2630L v2

Browsing the r720XD’s owner manual, and doing a few comparisons on CpuBenchmark.net, I found I could swap my E5-2695v2 out for a E5-2630L v2 and cut my CPU’s TDP in half.

There were other options, such as the E5-2650L v2, which offers increased cores, but, lower clock speed.

Since my TrueNAS server has been delegated to storage-only, I decided to prioritize the single-threaded performance.

Results – MABYE 10 watts saved

After swapping the CPU, I gave it a bit to start calculating a new average. Overall, I can see a reduced trend… but, the difference is not very dramatic.

The data below was collected from my Iotawatt. If you focus on the 400 watt line, you can see before swapping the CPU, the circuit spent the majority of the time slightly above it. After swapping the CPU, you will notice the circuit spends the majority of the time right under it, slightly. Do note, circuit, refers to the entire circuit powering my rack, and is not specific to just the server.

Overall, there is a reduction. But, it is not very significant. I believe I would see more reductions by removing the 40G networking card.

Conclusions / Next Steps

At this point, I don’t think there is much more room for optimizing the performance of this system, other then removing NICs, and replacing 8G dimms, with 32G dimms, as the dimms themselves, do consume energy.

Personally, the next step I take reducing energy, will be to retire the R720XD, and replace it with a Ryzen 5 3600, which I have sitting in my closet. The only thing missing, is a decent, rack-mountable case.

I did, however, through the various optimizations, get the r720XD to idle around 170-200 watts, with the main array always spinning.

I have been trying to see exactly how fast I can push my existing hardware, without dumping ridiculous amounts of money on it.

I have tried many things to maximize the amount of performance, including even temporarily buying a pair of 100GBe nics a while back. (TLDR; lots of driver issues….)

Well, after months and months of tinkering, I have finally reached my goal of being able to easily saturate 40GBe ethernet.

If you wish to read more of the details which led me to this point, here are a few of the previous posts:

Backstory

The Original NAS – 500$ Mini Build

I started my current NAS journey, with a 500$ build, after being disappointed at the high prices of synology / drobo / qnap systems. I wanted to prove I could build a more capable system for less money. I did succeed.

This unit was eventually replaced due to the underpowered CPU.

Combined Gaming PC / Server

I ended up selling the above build, and for a few months, I turned my gaming PC into a combined gaming PC and server. Overall, it worked great. The only downside was the amount of heat being put into my bedroom. Oh, and limited number of PCIe lanes.

This was replaced by a Dell R720XD, which I picked up for 500$ shipped, with 128GB of ram included.

Upgrading my home network to 10/40G backbone

Sept of last year, I picked up a pair of brocade ICX-6610 10/40G switches. As well, this is the first article where my Dell R720XD was mentioned. I never did a formal write up on obtaining it. However, it replaced my Combined GamingPC/Server.

As apart of this project, I also ran 10GBe to my office.

Adding a server rack

Self explanatory, however, since having all of your hardware sitting on buckets and cardboard boxes is not the best thing to do, I invested in a rack… and made everything somewhat presentable.

Reducing power consumption – Retiring Brocade

Turns out, the brocade ICX-6610 produces more heat/noise then I was happy with. In this post, I retired the old switches. This saved a big chunk of power, but, also meant, I no longer had a 40GBe network core. Instead, I leveraged my opnsense firewall for providing a 10G core, although routed.

Running 40Gbit Fiber

Since, 10GBe just wasn’t cutting it for me, I finally got around to running 40Gbe fiber from my server closet to my office. This also added a Chelsio 40GBe NIC to my Gaming PC.

Since the 40GB was point to point, no switches, no routers, etc, This also improved performance significantly over the 10GBe routed iscsi from previous steps.

Adding “Bifurcation”

Since, my r720XD does not support bifurcation, I found a few PCIe cards to work as switches for my NVMes. This allowed me to run many more NVMe drives rather then only being able to leverage a single NVMe per PCIe-slot.

Infiniband

One day a few months back, I experimented with using Infiniband.

This involved custom compiling drivers for TrueNAS, and having to hack up a lot of bits and pieces you generally do not want to touch.

As well, the community forums are quite toxic to you doing anything that isn’t supported out of the box. But, I did manage to successfully get infiniband up and running. However, I was unable to perform any reasonable tests as I didn’t want to hack up the system anymore.

Further Reducing Power Consumption

Given electricity costs money, and power usage means heat production, I continued down the path of reducing the energy consumption of my home lab.

In this post, I reduced power consumption by ADDING new servers. Overall, I was able to reduce my R720XD’s power consumption by a pretty good chunk by moving services to the new SFF optiplex machines I had acquired.

Since, the server was no longer handling containers/VMs, I also removed its second CPU which actually helped performance out a good bit.

On the downside, I now only have 96G of ram instead of 128G.

Further Reducing Power Usage (Not yet published)

In the next post regarding reducing my power utilization, I enabled power management features within TrueNAS, to allow for greater savings. This knocked off ~20w of consumption, while not allowing the disks to spin down. As well, I swapped in a low-powered E5-2630L to further reduce power draw.

(Note, processor is still in the mail. Article will be completed AFTER this one has already been published)

Migrating from TrueNAS Scale, to TrueNAS Core

The final step to achieve the performance numbers I have been hunting for- I installed TrueNAS core, and rebuilt my configuration from scratch.

Out of the box, with no tuning whatsoever, it had no issues at all besting every performance metric from Scale.

The Benchmarks

I have split the benchmarks into two seperate sections. Those against my Flash array, and those against my pool of spinning rust.

I did post both 1G and 32G test sizes where possible.

I started running the benchmarks in 32G to hush the trolls who blame the results on cache/ram.

Against my 8-disk Z2 Array

The array, is a 8-disk Z2. LZ4 compression is enabled, without dedup. The drives are 8x USED 8TB Seagate Exos.

TrueNAS Scale

These benchmarks were performed before migrating to core.

iSCSI – 1G Test Size

SMB – 1G Test Size

TrueNAS Core

iSCSI – 1G Test Size

iSCSI – 32G Test Size

SMB – 1G Test Size

SMB – 32G Test Size

It is rather interesting to see the differences between 1G and 32G. The sharp decline in write speed/IOPs is due to the cache being saturated. However, large sequential writes are still working just fine.

Against My Flash Array

My flash array is a single mirrored VDEV, consisting of a pair of 1T Samsung 970 EVO NVMe. While, previously, I did add an additional stripe, I didn’t notice dramatic performance differences and ended up removing the extra vdev(Possible to do with openzfs).

Since, I did not take recent benchmarks of the flash array before migrating to core, I do not have any “before” results to compare with.

As well, I did not take any benchmarks over SMB. However, it is safe to assume the results will be faster than the SMB results.

TrueNAS Core – After Results

iSCSI – 1G Test Size

iSCSI – 32G Test Size

I will note, there is a tiny amount more room for improvement as my CPU is getting pegged pretty hard during write operations.

But, next week, I will be replacing my E5-2695v2 with a E5-2630L in attempts to further reduce my energy consumption. However, I do feel, I should still be able to obtain very reasonable benchmark results.

Next Steps?

Well, Other then replacing my CPU with a low-powered model which is going to hurt my performance slightly, I don’t really have any next steps.

I have been debating wanting to build out a CEPH cluster using a pile of SFF/MFF PCs, however, the budget for such a project is not in the cards for this year.

As well, I have toyed with the idea of deploying a disk array, but, this wouldn’t net me any performance improvements or reducings in power consumption. The array itself would use 30-60w without any disks. The only meaningful advantage would be to make it much easier to swap to a different piece of hardware. I have decided, that since the number of spinning spools directly correlates to the wattage draw, I don’t want to deploy a 16+ disk array. When zfs online expansion comes eventually, I will likely grow my existing array to 12 disks total.

For now, It is nearing time to focus my efforts on the 1,000hp turbocharged 4×4 suburban project! Because, everyone needs a 1,000hp surburban! (NOT an exaggeration on the horsepower numbers….)

Well, given my recent push to improve energy efficiency, I have reduced the average power utilization from my R720XD from 400-450w average, down to around 190w average. However, there is still more room to reduce usage.

Based on benchmarks I have seen, core should offer improved performance, which means less impact from the low-powered CPU I plan on putting into my server. As well, this should be a very easy migration from Scale to Core.

Benchmarks

Before doing anything, I ran a few simple, quick benchmarks across iScsi and SMB, to my Z2 array. I used the peak performance profile in CrystalBenchmark.

This isn’t designed to be a definitive test, but, to give me something quick to compare against.

TrueNAS Scale

iSCSI

SMB

TrueNAS Core (After migration)

iSCSI

SMB

As seen above, there are numerous performance improvements across the board. Given the sole purpose of my TrueNAS box at this point in time is storage, and storage alone, this makes the decision to move from Scale back to Core all worth it.

Steps

Step 1. Backups

The first and foremost step before doing anything to your storage system, is to ensure you have working backups. Assume, your migration will fail in a catastrophic manner, leaving all of your data unusable. So, make sure you have working backups.

Step 2. Configuration Backup

With a configuration backup, you can easily restore all of your configurations to a fresh install of TrueNAS, allowing you to be back up and online in minutes. Always keep a backup!

Step 3. Download the core ISO.

Download the desired release from HERE. I went with Core 13.0-RC1 assuming it would have the newer version of ZFS, minimizing potential pool compatibility issues.

Step 4. Mount / Burn the ISO.

If you do not have enterprise hardware with remote management capabilities, burn the disk to a CD/DVD/or ISO.

If you have iDrac / iLo, attach the image as virtual media.

If you are using idrac/ilo, set the next boot, to boot to your virtually mapped ISO.

Reboot your system.

Step 5.1 – Reboot, and find unexpected memory errors

During the process of rebooting to reinstall, I had one of my Dimms fail testing. As I have recently had a weird issue popping up seemingly related to memory…. I decided to go ahead and replace the A12 dimm.

After replacing stick A12, the errors/warnings went away.

Step 5.2 Install TrueNAS Core

After you get booted to the installer, you will want to select Install/Upgrade

After I was prompted for which drives to install TrueNAS, I selected my two 10K SAS drives where it was previously installed.

After which, I received a prompt letting me know everything would be erased.

Select your desired boot mode. The r720XD will boot either mode.

When asked to create a swap partition, I selected to not create swap (As the system has over 100G of ram…)

After finishing up the GUI-based options, a terminal appeared.

This step took a while to complete, likely due to the virtual media being accessed over the network.

After…. 10 minutes or so, Core was successfully reinstalled.

I disconnected the virtual media, and rebooted the system.

Step 6. Boot / Import Configuration

After a while, my system booted into core.

At this point, we are done using iDrac / Physical console for configuration.

Assuming your system was able to acquire a usable IP address (I have multiple NICs configured, a few of which gave out a DHCP address), navigate to the configuration page, and log in with the password you configured earlier during the installation.

From this point, I would enjoy telling you to upload your previous configuration backup, however, my scale backup was deemed “too new” for core to import. This isn’t a big problem. Lets rebuild a new configuration.

I will start by importing my pools from scale.

After, a long time, the pool from scale was able to successfully import.

I continued importing my other pools.

After this, I went ahead and configured my 10/40GBe interfaces. I had no issues getting the 40GBe ConnectX-3 NIC up and running.

From this point, I went through each configuration section and slowly reconfigured everything… Emails, Network, DNS, SSH, NTP, etc…

After configuring everything, I rebooted the host. It rebooted and came back online without any issues or side-effects.

Step 7. Reconfiguring iSCSI

In the previous step, I went ahead and configured most of iSCSI. However, I could not select the existing zvols as extends.

To knock out some initial testing, I went ahead and deleted the old 50G testing zvol from my flash array, and used the wizard to go ahead and setup iSCSI.

After attaching/formatting the disk in windows, I went ahead and ran a test.

Granted, this test is against my Flash array, however, I will note the write performance is MUCH better then it was during the previous tests conducted when I replaced my core switch.

As well, I have performed no tuning, yet.

Results?

Overall, the migration was not too bad.

On a scale of 1-10, 10 being the hardest, I would give this a 6.

It would have saved many hours of reconfiguration If I were able to import my configuration from Scale. Since I was not able to, I had to go and manually reconfigured all users/groups/shares/jobs/etc. This was a pretty time consuming process.

I will note, my existing pools imported without a hitch, and I was able to get all of my iscsi mounts reattached after a few minor issues.

Getting SMB working again, took quite a bit more effort then I was expecting, however.

Introduction

My electric bill is expensive. I need to reduce it. I don’t enjoy every month seeing my home lab is the main cause for my high energy bill.

So, I wanted to reduce this usage a bit, without affecting performance.

Note- Affiliate advertising links are utilized on this article.

Step 1. Adding more servers

Sounds counter-productive, right?

Well, currently, I have everything hosted on my ancient dell R720XD. Its not the most efficient thing in the world. The plan was to take my primary services and host them on a newer, more modern machine so that I can perhaps schedule my r720XD to shutdown at times.

Hardware

For hardware, I decided to pickup a pair of Dell SFF Machines.

7040 SFF – i5-6500 + 8G of ram – 100$

EBay Link

I plan on using this machine to host my Blue Iris instance. The i5 offers rudimentary hardware encoding, and 8G of ram is plenty.

I added a 8T WD Red HDD for bulk NVR footage, and a 500G WD Black for “new” footage.

I purposely chose the WD Black, because it is cheap. The NVMe tied to the new storage will be constantly written to, and will eventually burn out.

For the OS drive, I found new 128G NVMe drives for 15$ each. I used a 12$ adaptor to install it into one of the PCIe slots.

7060 SFF – i7-8700 + 16G – 400$

EBay Link

For my container host, I chose a more modern device with an i7-8700. This offers 6c and 12t, with a good clock speed. As well, the quick sync is quite effective. This will be very handy for plex transcoding.

The 7060 does offer a lot more features over the 7040, including a few extra ports.

For additional hardware, I installed…

1. (Amazon) Samsung 980 evo – 99$
   1. This will host /var, where all of my containers and their associated data will be stored.
2. (Amazon) KIOXIA 128G NVMe – 14$
   1. This will be the primary OS drive. This is the same NVMe I also used for the NVR box.
3. (EBay) Mellanox ConnectX-3 Dual Port 40GBe – 50$
   1. Point to point between my TrueNAS server, and my container host. Hardly more expensive then running 10G….

Migrating my container setup.

I am going to skip over the boring details of migrating my containers, however, everything was stored in docker-compose files, and this essentially was just a matter of moving the compose files over, and copying data where needed, and then updating the DNS record to point my load balancer at the new IP.

I will note, for plex, nextcloud, and other storage-heavy containers, I leveraged NFS from within my docker-compose files to connect back to the primary NAS.

Power Savings – 15 watts

While, 15 watts does not sound like a dramatic improvement in power consumption, Remember, this is the savings after ADDING another server. I did not shutdown any of my hardware. This is solely savings from migrating the loads from the r720XD over to the new 7060SFF.

Migrating my NVR

To migrate the NVR, I simply installed windows server on the new device, installed Blue Iris, and imported my old configuration. I then configured new footage to be stored on the 500G WD Black, and configured long-term storage on the 8T drive.

And, then, I shutdown the NVR VM which had been running on TrueNAS.

Power Savings = -10w

Well, we didn’t actually save energy here, but, rather, used 10 more watts. However, remember, the purpose here was to migrate loads away from my TrueNAS server.

Step 2. Removing additional hardware from the r720XD

Since, all of the containers and VMs at this point have been moved away from the r720XD, I wanted to remove one of the CPUs to save on energy.

The server had dual E5-2695v2 processors, and 128G of DDR-3 1333mhz RAM.

I removed CPU 2, and removed all of its ram, using it to fill any empty slots from CPU 1.

After doing so, I had only a small pile of hardware left over…. 32G of ram, and a processor.

Power Savings – 80w.

Based on initial data after removing the extra CPU, it appeared the power consumption had been reduced by over 80 watts.

Overall reduction – ~100w = 5$ per month.

Keep in mind, at this point, I can safely shutdown my R720XD, which will reduce the overall consumption DRAMATICALLY. This system was using an average of around 300-400w 24/7 when I started this project.

Now, it is using 200-250w average, which is nearly a reduction of half. Some of the energy savings were lost when I used the i5-6500 for my NVR box, however, it is still a net reduction.

Total Monthly Savings = A few bucks, maybe.

Lets assume I shut the r720XD down at nighttime, and fire it back up at 7am for a total of 10 hours of downtime per day. I can do this, because all of the important containers and workloads were migrated to the new SFF machines.

This machine now uses ~220w average. ((220 * 24 hours * 30 days) / 1,000 watts ) * 0.08 (cents / kwh) = 13$ of energy per month.

It WAS using 30$ per month in energy. However, I think the majority of this load is/was coming from its NVR duties.

Unexpected performance gains.

Right before I deprovisioned the hardware in my r720XD, I had ran benchmarks against my spinning array over iScsi.

However, I noticed a improvement of benchmark performance after removing the 2nd CPU.

Performance shot up quite a bit…. in some scenarios, it was quite dramatic.

Comparing the performance of my NAS, versus a locally installed 970 evo….. the performance gap is getting closer.

While, the local drive can push out quite a few more IOPs, in terms of bandwidth, the remote storage isn’t doing bad at all. Regarding writes, the penalty is understandable since it is writing to a mirrored pool.

Summary

While, I actually failed to reduce power consumption by any meaningful amount, I did succeed in a few things.

I can do maintenance on my primary storage server without impacting critical services.

My NVR has dedicated hardware.

My container services are much faster with the i7 processor.

My iSCSI/SMB traffic is actually quite a bit faster.

I no longer have to hack up my TrueNAS to run docker containers.

Next Steps?

Well, I have actually been in the process of designing and installing a solar system for over a year now. It may come to be a thing soon, hopefully….

Introduction

Let me start off by saying, I like icicles on the wall while I sleep. As well, my bedroom also doubles as my home office.

During the summer time, with a few computers running, it gets pretty toasty back here.

In the winter time, being this room is on the far end of the house, it gets pretty chilly back here.

So, I decided to invest into a mini split… and integrate this into home assistant.

How much did it cost / products

Contractor Quotes

So, first of all, let me tell you how much I was quoted to have a 3/4 ton (9,000btu) mini split installed.

Contractor #1-

I was quoted 2,800$ to install a 3/4 ton gree mini split. This did not include running the electrical work. Electrical was 500$ extra.

Contractor #2

I was quoted 3,000$ to install a 3/4 ton Daikin mini split. This also did not include the electrical work. Electrical was 450$ extra.

My Prices

After having my mouth drop at the 3,500$ total price tags, I decided do it myself. Here are the products I purchased:

Pioneer WYS 3/4 Ton – 23 Seer (Amazon)

For the cost of only a touch over 900$ shipped, for a 23 seer unit, seemed like a pretty acceptable deal to me. I went the pioneer route after hearing good luck from a co-worker who tied these units into home assistant.

Charging Port Adaptor – 5/16″ -> 1/4″ (Amazon)

This is needed to connect a standard set of vacuum gauges up to the unit.

60 Amp AC Disconnect Box (Pioneer)

Sorry, no amazon link. But, for 14.99$.. the price was good. I don’t think I could have beaten this price at my local hardware store.

6ft Plastic Flexible Whip – 10AWG (Pioneer)

I went ahead and ordered the whip from pioneer as well.

Total So Far – 970$

I ordered all four of the above items directly from pioneer. However, the amazon prices linked above have the same cost at this time.

But- wait, there are a few more items required.

Pioneer Wi-Fi module for WYS Systems.

This part is CRUCIAL for integration into home assistant. Without it, you will need to rely on integrating the device using IR senders…… and personally, I want two way communication, in real time.

100ft 10/2 Wire

I actually didn’t end up needing to use this at all. In the future, I will run an outlet to my welder using it. I will explain more below.

Pioneer Foldable Condenser Bracket – 2 piece

If its built by the company making the mini split, I would assume it is more then capable of holding their unit! So far, it is up to the task. Also, I accidentally ordered two of these. Oops.

Total Price: ~1,500$

Do note, I also spent around 200$ extra at the local hardware store, on a small section of 2″ PVC, as well as a full length 3/4″ PVC… and a few small odds and ends, including a masonry drill bit, and a few concrete anchors.

Other Potential Units:

• Gree 1 Ton 16 Seer
• Daikin 3/4 ton 17 Seer

Installation

Sorry- I did not take any pictures during this process…. But, I will detail the steps required. You can skip down to (INSERT LINK HERE) for how to connect this unit into home assistant.

1. First, we went inside the house and located a nice spot in the center of my bedroom wall.
2. The unit requires 6-8 inches of clearance on the top.
3. Next, we mounted the “plate” on the wall, in the center of the wall, with the top 10 inches from the ceiling to allow for proper top clearance.
4. After that, we followed the instructions and used a 2″ hole bit to drill a hole through the interior sheetrock.. Followed by a LONG masonry bit to drill all the way through the outside brick wall.
5. Next, was the hardest part of this entire process. Drilling a 2″ hole through the brick. After tracing a 2″ circle using the section of 2″ PVC, we started using a small masonry bit to drill holes around the perimeter of the traced outline.
6. I then utilized my air chisel to carefully knock a hole through the brick.
7. After this was completed, we had a hole which fit the 2″ PVC Perfectly.
8. I used a knife and screwdriver to poke and extend a 2″ hole through the insulation.
9. The 2″ PVC was cut to the length from the interior wall, to the exterior wall. IE, it fits flush on either side.
10. The lineset was then ran through the 2″ pipe. After this, the pipe containing the line set was inserted into the new 2″ hole going from outside of the house, to the interior.
11. MAKE SURE TO RUN THE DAMN CONTROL CABLE THROUGH THE PIPE TOO!!!!!!!! Its a lot harder to run AFTER you put everything in the wall, and bolt everything together.. Ask me how I know… /s
12. At this point, the interior unit was hinged from the top of the mount, while we connected the line set, and routed the control wires.
13. After connecting the line set and control wires, the interior unit “snaps” to the mounting plate. You are now done inside of the house.
14. Back outside, I used spray foam to fill any remaining gaps from the 2″ hole we created. I then used masonry clay to fill the gaps directly around the 2″ pipe in the brick.
15. Next, the outdoor mounting brackets were mounted to the exterior brick, using concrete anchors… after careful measurement.
16. We mounted the outdoor unit to the mounting brackets using the included rubber pads.
17. The line set was carefully bent in a few key locations, to route into the side of the outdoor unit, in a somewhat pretty way.
18. The line set was then connected to the exterior unit, along with the control wires.
19. The drain line was inserted into the piece of 3/4″ pvc from earlier, which was mounted vertically to the wall to drain near the ground… and secured to the brick using clamps and more anchors.
20. The final piece of work remaining was to run electrical.
21. After looking through my breaker box, I realized I had an unused 220v circuit going to my oven. As my oven is natural gas, and there was not even an outlet behind it, I decided to steal this cable instead of running the new 10/2 I had acquired.
22. I drilled a 1″ hole through the gabled end of my house, and placed a 1ft section of 1″ pipe through this new hole. Where the pipe exits the gabled end of my house, it has a 90 degree turn so that it runs down vertical to the disconnect box. The cable runs through this piece of PVC. Caulking was used to seal up around the new PVC.
23. At the bottom of the 1″ PVC containing the romex, it has a 90 degree joint, screwing directly into the side of the disconnect box.
24. I did swap the original 60amp breaker out for a much more suitable 20 amp breaker.
25. After this was completed, and the whip was installed between the disconnect box, and the outdoor unit…. it was time to vacuum out the air.
26. This step is absolutely crucial to your unit. Borrow or acquire a vacuum pump, and a set of gauges. You need to vacuum all of the air out of the line set. Once you have a vacuum pulled, you need to wait a bit…… and ensure it holds the vacuum.
27. Once you are sure it is not leaking, you can slowly open up the valves on the outdoor unit, allowing the refrigerant to enter the lines. Use a spray bottle filled with soapy water to ENSURE you don’t have any leaks. You don’t want this stuff to leak.
28. After that step is done, you are all done!

Overall, the installation took us a full day to complete.

Connecting to Home Assistant

WARNING!!!!!!

As of May 2023, I no longer recommend the below method.

Instead, there is a much cheaper, AND more reliable option.

https://static.xtremeownage.com/blog/2023/pioneer-mini-split-esphome/

However, for historical reasons- the original method is still outlined below:

This part was actually pretty easy.

First- make sure you installed the wifi unit into the indoor unit. There is a special slot where it goes in.

Next, download the “Nethome Plus” app on your phone.

Open the app, and add the unit. You will need to scan the QR code which was included with the wireless module.

After this, you will need to connect the unit to your wifi. If you are like me, I keep all of my IOT stuff on a dedicated subnet, with no internet or outbound access at all. NTP is the only allowed traffic for my IOT devices. However, my home assistant can access everything on the IOT subnet.

If you use a IOT subnet, you will need a temporary firewall rule while connecting the unit via the app. Once the unit is on the wifi, you can disable/delete this rule.

Next, in home assistant, visit HACS, and add the “Midea Smart Aircon” integration.

Now, we will need a machine with python to obtain a few piece of details. Please read these INSTRUCTIONS. you will need the token and key.

I used this command:

# Create a venv
python -m venv ac_stuff

# Enter the venv
source ac_stuff/bin/activate

# Install "MSmart"
pip3 install msmart

# Discover the device. Put your device's IP in here. I allocated a static IP for my unit in my router/firewall.
midea-discover --ip 10.20.10.5

You will then see output... you will need this for later.

Next, we need to modify Home Assistant’s configuration.yaml

climate:
  - platform: midea_ac
    host: 10.100.3.5
    id: 43980465114375           ## Put your ID here.
    token: PUT_YOUR_TOKEN_HERE!!!!
    k1: PUT YOUR K1 / KEY HERE!!!
    prompt_tone: false           ## Set this to disable beeps every time you change something.
    temp_step: 1                 ## Optional, but, I want 1 degree temp increments instead of the default 0.5

After this is completed, restart home assistant, and you should have new climate device.

At this point, I took the liberty of renaming the device, and customizing it.

At this point, you are basically done. If you add a climate/thermostat card, you should be able to fully interact with the device.

However, I decided to use the scheduler-card to add a few schedules.

This was extremely easy to do, and very intuitive. I highly recommend this addon. You can install it via hacs.

If you are interested in how to set this up, please read THE TUTORIAL I CREATED

Overall opinons

Overall, I am extremely happy with this unit.

One thing I will note, the original 3,000$ price tags aren’t quite as bad as I originally expected though.

If you figure, I spent 1,500$ in materials. Lets say, they can get the same materials for 1,200$, which is very possible. Then, lets assume you have two people on site for 10 hours to install this unit… (The helper I had, is a professional HVAC installer…) Lets say, the lead is 60$/hr, and the helper makes 20$/hr. 600+200 in labor = 800$. So, 2,000$ total.

In my case, it was extremely helpful having someone help me who has done this before. After doing one of these units, I would feel pretty comfy doing another one by myself though. But, you can also watch youtube videos on how to install these.

Regarding the units performance, its extremely quiet, and it works very well. While working, it keeps the temp very comfy, and at night time, It kept it at 56F last night. Nice!

Regarding power usage, this unit is quite good. Since, putting it in, it has averaged around 16cents of power consumption per day…. that includes cooling the bedroom to around 60 from 8pm, to 6am, and then heating to around 70-72 from 6am – 8pm. You will notice, once it has “cooled” the room, the unit uses basically no energy.

Power Consumption:

Bedroom Temp:

Yesterday’s power consumption.

Regarding the automation side, this unit is COMPLETELY local control, and I have it on my IOT subnet, which has zero access to anything. The automation works in real time, without any delay… and the schedules are fantastically easy to setup and configure. Would highly recommend.

FAQs

Q: Why didn’t you get a Mitsubishi or Fujisto? They are the best

A: I don’t doubt they are made with a much higher quality. However, I cannot buy their units without going through a contractor. I purposely got the pioneer unit, because it was cheap, and extremely DIY friendly…. and I was able to order it right off the internet and have it shipped to my front door.

Summary

1. Was able to get infiniband working
2. IPoIB performance is utter garbage.
3. I didn’t want to hack up TrueNAS more to support RDMA
4. Gave up, went back to my Chelsio 40GBe adaptor.

Step 1. Install a few packages.

By default, TrueNAS Scale has apt “disabled”. Please see THIS POST on how to re-enable it.

Once you have… re-enabled the ability to install packages, AND added the debian repos, we need to install a few packages to assist.

At this point, you should be able to run ibstat and get a status. If you don’t have a status, make sure your card is installed. I am using a ConnectX-3 (Non-pro)

To note, I have one port set as infiniband, and the other port set as ethernet.

root@truenas:~# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x506b4b03007bfc50
        System image GUID: 0x506b4b03007bfc53
        Port 1:
                State: Initializing
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02514868
                Port GUID: 0x506b4b03007bfc51
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x526b4bfffe7bfc52
                Link layer: Ethernet

How to change configuration parameters

# First- get the PCI address of your card.
root@truenas:~# lspci | grep Mell
03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

# In this case- 03:00.0 is my address.

# Show current configuration
root@truenas:~# mstconfig -d 03:00.0 query

Device #1:
----------

Device type:    ConnectX3
Device:         03:00.0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        IB(1)
         LINK_TYPE_P2                        ETH(2)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)


# Set port mode to infiniband
mstconfig -d 03:00.0 set LINK_TYPE_P1=1

# Set port mode to ethernet
mstconfig -d 03:00.0 set LINK_TYPE_P1=2

# You can also set multiple properties in the same command
mstconfig -d 03:00.0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1

# Disable boot rom, if you desire. I don't have plans on using it.
mstconfig -d 03:00.0 set LINK_TYPE_P1=1 BOOT_OPTION_ROM_EN_P1=0 BOOT_OPTION_ROM_EN_P2=0

Step 2. Ping across the network

This guide ASSUMES you already have a working physical link established between two infiniband cards.

In my case, I have a windows client, and a TrueNAS server.

Next, lets lists connected hosts

root@truenas:~# ibhosts
ibwarn: [2618545] get_abi_version: can't read ABI version from /sys/class/infiniband_mad/abi_version (No such file or directory): is ib_umad module loaded?
ibwarn: [2618545] mad_rpc_open_port: can't open UMAD port ((null):0)
../libibnetdisc/ibnetdisc.c:802; can't open MAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed

Seems… a kernel module is missing. Lets add it, and try again.

root@truenas:~# modprobe ib_umad
# I also went ahead and loaded the infiniband IP over InfiniBand module.
root@truenas:~# modprobe ib_ipoib
root@truenas:~# ibhosts
Ca      : 0x0002c90300a29060 ports 2 "XTREMEOWNAGE-PC"
Ca      : 0x506b4b03007bfc50 ports 2 "MT25408 ConnectX Mellanox Technologies"

Well, look at that. The two PCs can see each other over infiniband.

The next step, was to allocate an ip address for the infiniband adapter.

ibp3s0: flags=4098<BROADCAST,MULTICAST>  mtu 4092
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As we can see, it doesn’t have an address. I provisioned the IP address through the TrueNAS WebGUI.

After saving the changes, applying, etc…. I was able to ping the local IP, but, not the remote IP.

root@truenas:~# ping 10.100.255.1
PING 10.100.255.1 (10.100.255.1) 56(84) bytes of data.
64 bytes from 10.100.255.1: icmp_seq=1 ttl=64 time=0.052 ms
^C
--- 10.100.255.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.052/0.052/0.052/0.000 ms
root@truenas:~# ping 10.100.255.2
PING 10.100.255.2 (10.100.255.2) 56(84) bytes of data.
^C

Doing a bit of troubleshooting, I ran ibstat on both ends.

and, determined the link was stuck in status “Initializing”

After a bit of research, I determined a subnet manager needs to exist, for a point to point link.

So…

root@truenas:~# apt-get install opensm
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  opensm
...

At this point, I decided to do a reboot, to ensure all of the modules are being properly loaded at boot.

Or… to see if…. all of the above work has been reset.

After rebooting, all of the configuration and packages were still in place, However, the kernel modules were not loaded back in. So… I reloaded the modules manually, and added a config to load them at boot.

root@truenas:~# modprobe ib_ipoib ib_umad ib_uverbs rdma_ucm
root@truenas:~# nano /etc/modules-load.d/infiniband.conf
# File Contents:
ib_umad
ib_ipoib
ib_uverbs
rdma_ucm

The next step, was to get opensm up and running.

root@truenas[/etc/modules-load.d]# systemctl status opensm
● opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers
     Loaded: loaded (/lib/systemd/system/opensm.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sat 2022-03-26 13:26:55 CDT; 8min ago
       Docs: man:opensm(8)

Mar 26 13:26:55 truenas.local.xtremeownage.com systemd[1]: Condition check resulted in Starts the OpenSM InfiniBand fabric Subnet Managers being skipped.


## I assume, it failed to start due to the missing kernel modules. Lets start it.

root@truenas[/etc/modules-load.d]# systemctl start opensm
root@truenas[/etc/modules-load.d]# systemctl status opensm
● opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers
     Loaded: loaded (/lib/systemd/system/opensm.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2022-03-26 13:35:38 CDT; 1s ago
       Docs: man:opensm(8)
    Process: 296934 ExecCondition=/bin/sh -c if test "$PORTS" = NONE; then echo "opensm is disabled via PORTS=NONE."; exit 1; fi (code=exited, status=0/SUCCESS)
    Process: 296935 ExecStart=/bin/sh -c if test "$PORTS" = ALL; then PORTS=$(/usr/sbin/ibstat -p); if test -z "$PORTS"; then echo "No InfiniBand ports found."; exi>
   Main PID: 296935 (code=exited, status=0/SUCCESS)

Mar 26 13:35:38 truenas.local.xtremeownage.com systemd[1]: Starting Starts the OpenSM InfiniBand fabric Subnet Managers...
Mar 26 13:35:38 truenas.local.xtremeownage.com sh[296935]: Starting opensm on following ports: 0x506b4b03007bfc51
Mar 26 13:35:38 truenas.local.xtremeownage.com sh[296935]: 0x526b4bfffe7bfc52
Mar 26 13:35:38 truenas.local.xtremeownage.com systemd[1]: Finished Starts the OpenSM InfiniBand fabric Subnet Managers.


## Much better.

At this point, I checked the stats for the infiniband nics.

root@truenas[/etc/modules-load.d]# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 1
        Node GUID: 0x506b4b03007bfc50
        System image GUID: 0x506b4b03007bfc53
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251486a
                Port GUID: 0x506b4b03007bfc51
                Link layer: InfiniBand
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x526b4bfffe7bfc52
                Link layer: Ethernet


## Boom, good to go on the truenas side.

BADPROMPT:ibstat
CA 'ibv_device0'
        CA type:
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 0x0
        Node GUID: 0x0002c90300a29060
        System image GUID: 0x0002c90300a29063
    Port 1:
        State: Down
        Physical state: Disabled
        Rate: 1
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x80500000
        Port GUID: 0x0202c9fffea29061
        Link layer: Ethernet
        Transport: RoCE v1.25
    Port 2:
        State: Active
        Physical state: LinkUp
        Rate: 40
        Real rate: 32.00 (QDR)
        Base lid: 2
        LMC: 0
        SM lid: 1
        Capability mask: 0x90580000
        Port GUID: 0x0002c90300a29062
        Link layer: IB
        Transport: IB

Excellent, the link is active on both sides!

Since, the module was not active when the machine was rebooted, the interface was down. I used iconfig to bring it up.

root@truenas[/etc/modules-load.d]# ifconfig ibp3s0 up
root@truenas[/etc/modules-load.d]# ifconfig ibp3s0
ibp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4b03:7b:fc51  prefixlen 64  scopeid 0x20<link>
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 2  bytes 212 (212.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37  bytes 11759 (11.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@truenas[/etc/modules-load.d]# ping 10.100.255.1
PING 10.100.255.1 (10.100.255.1) 56(84) bytes of data.
64 bytes from 10.100.255.1: icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from 10.100.255.1: icmp_seq=2 ttl=64 time=0.049 ms
^C
--- 10.100.255.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.049/0.052/0.056/0.003 ms

We can now ping over IP. Lets do a speed test.

## Since, TrueNAS doesn't ship with iperf2 (Does include iperf3, however, getting speed results above 10G isn't consistent using iperf3, due to its single-threaded nature....
## So, I prefer iperf2 for testing over 10G.
root@truenas[/etc/modules-load.d]# apt-get install iperf
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  iperf
0 upgraded, 1 newly installed, 0 to remove and 105 not upgraded.
...
root@truenas[/etc/modules-load.d]# iperf -v
iperf version 2.0.14a (2 October 2020) pthreads
root@truenas[/etc/modules-load.d]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------


## on the windows side.
BADPROMPT:iperf.exe -c 10.100.255.1
------------------------------------------------------------
Client connecting to 10.100.255.1, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[  3] local 10.100.255.2 port 23277 connected with 10.100.255.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.69 GBytes  6.60 Gbits/sec

BADPROMPT:iperf.exe -c 10.100.255.1 -P 10
------------------------------------------------------------
Client connecting to 10.100.255.1, TCP port 5001
TCP window size:  208 KByte (default)
------------------------------------------------------------
[ 12] local 10.100.255.2 port 23297 connected with 10.100.255.1 port 5001
[ 10] local 10.100.255.2 port 23295 connected with 10.100.255.1 port 5001
[ 11] local 10.100.255.2 port 23296 connected with 10.100.255.1 port 5001
[  9] local 10.100.255.2 port 23294 connected with 10.100.255.1 port 5001
[  7] local 10.100.255.2 port 23292 connected with 10.100.255.1 port 5001
[  6] local 10.100.255.2 port 23291 connected with 10.100.255.1 port 5001
[  8] local 10.100.255.2 port 23293 connected with 10.100.255.1 port 5001
[  5] local 10.100.255.2 port 23290 connected with 10.100.255.1 port 5001
[  4] local 10.100.255.2 port 23289 connected with 10.100.255.1 port 5001
[  3] local 10.100.255.2 port 23288 connected with 10.100.255.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0-10.0 sec  1.07 GBytes   916 Mbits/sec
[ 10]  0.0-10.0 sec   899 MBytes   753 Mbits/sec
[  6]  0.0-10.0 sec  1.01 GBytes   870 Mbits/sec
[  8]  0.0-10.0 sec   220 MBytes   185 Mbits/sec
[  5]  0.0-10.0 sec   561 MBytes   470 Mbits/sec
[  9]  0.0-10.1 sec   695 MBytes   578 Mbits/sec
[ 11]  0.0-10.1 sec   674 MBytes   557 Mbits/sec
[  7]  0.0-10.2 sec   346 MBytes   285 Mbits/sec
[  4]  0.0-10.2 sec   253 MBytes   208 Mbits/sec
[  3]  0.0-10.2 sec   681 MBytes   560 Mbits/sec
[SUM]  0.0-10.2 sec  6.31 GBytes  5.31 Gbits/sec

Well, that was extremely disappointing!

Troubleshooting poor performance

The first step- is always to check MTU. I found the server was configured for 8,000, however, my client only supported 4092 max.

 root@truenas# ifconfig ibp3s0 mtu 4092 up

Just, to confirm this wasn’t an iperf issue, I decided to do a quick iscsi performance test.

Only, 5Gbits….. far short of the expected 40/56Gbits.

Well… lets doublecheck MTU.

root@truenas[/etc/modules-load.d]# ifconfig ibp3s0 mtu 4092
root@truenas[/etc/modules-load.d]# ifconfig ibp3s0
ibp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4b03:7b:fc51  prefixlen 64  scopeid 0x20<link>
        unspec 80-00-02-28-FE-80-00-00-00-00-00-00-00-00-00-00  txqueuelen 256  (UNSPEC)
        RX packets 23373377  bytes 47020734796 (43.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6652816  bytes 6411049427 (5.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

## Appears the card on my server only supports 2044 MTU... this might explain the issues.

So… I set the MTU on both ends to 2044.

Sadly, this did not help performance.

Conclusion.

I DID successfully manage to get infiniband up and running.

I DID learn, IPoIB (IP over InfiniBand) is actually quite slow. To realize huge performance gains over using the NIC in ETH mode, you would need to leverage RDMA/RCOE.

Since, I have hacked up my TrueNAS install enough for the day, I decided to switch everything back to ethernet mode, and call it a day. I can easily get 40G out of ethernet mode.

How to switch back to ethernet mode.

If you recall earlier, I left one interface in ETH mode…. I am just going to switch the physical adaptor, and move the ip address over… so, I don’t need to reboot my PC.

ifconfig ibp3s0 0.0.0.0 0.0.0.0
ifconfig enp3s0d1 10.100.255.1/30 mtu 9000 up
root@truenas[/etc/modules-load.d]# ifconfig enp3s0d1
enp3s0d1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.100.255.1  netmask 255.255.255.252  broadcast 10.100.255.3
        inet6 fe80::526b:4bff:fe7b:fc52  prefixlen 64  scopeid 0x20<link>

## On the windows side, I also physically swapped the adaptors over.

root@truenas[/etc/modules-load.d]# ping 10.100.255.2
PING 10.100.255.2 (10.100.255.2) 56(84) bytes of data.
64 bytes from 10.100.255.2: icmp_seq=1 ttl=128 time=0.482 ms
64 bytes from 10.100.255.2: icmp_seq=2 ttl=128 time=0.162 ms

## Connection is good to.... Lets test performance.
[SUM] 0.0000-10.0208 sec  10.7 GBytes  9.15 Gbits/sec

## Appears my client is only connected at 10G.... interesting.

root@truenas[/etc/modules-load.d]# ethtool enp3s0d1
Settings for enp3s0d1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseKX4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

## Wonder what happens if I uh, FORCE it to 40g.

root@truenas[/etc/modules-load.d]# ethtool -s enp3s0d1 speed 40000 duplex full autoneg off

## Turns out, the connection on the windows side dropped.... I am starting to think this particular ConnectX-3 does not support 40G. 
## I know the adaptor on the server side supports 40G, because, its the same exact adaptor i used to benchmark 40G in a previous post!

I found documentation on my particular NIC here: https://downloads.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_pedge_srvr_ethnt_nic/mellanox-adapters_users-guide_en-us.pdf

Based on the specifications, it “should” support 40G ethernet mode…

Edit- Turns out, it supported 40G infiniband, but, only 10GBe ethernet.

Yup. Oh well. Putting the chelsio NIC back in my PC. It had no problems doing 40G.

Multiple times per day, I will be browsing reddit, and see…. MANY posts from users trying to achieve the lowest possible energy consumption regardless of hardware cost.

While, I have made previous posts about this before, I have decided to put all of the math involved front and center to assist users with making their own choices.

Note- Affiliate advertising links are utilized on this article.

ROI Calculator

Examples

Example #1, My R720XD versus a newer server

Here is a simple calculator, to calculate the amount of energy it will cost you per year to run a given piece of hardware, provided its energy consumption, and then, compare those numbers with a more efficient piece of hardware.

The calculator is pre-filled with my data. On the left, represents my Dell R720XD Server, with its 300w average power consumption. On the right, estimates a rough cost for me to build a new server with the required features and performance of my workloads. (The actual number would likely be far higher, as I typically USE 70-100GB of RAM… and RAM is not cheap.)

Results? Used Enterprise Server Wins, By Far.

As you can see in the above example- I would not achieve a ROI for at least 10 years. Yes- my electric bill would be 100$ cheaper per year, however, I still wouldn’t break even for a full decade.

Given hardware usually has a finite lifecycle, I would say anything with a ROI > 7 years is certainly not worth the additional effort or cost.

Example #2 – My R720XD versus a newer server in Germany

This is exactly the same as the first example, with one exception. Instead of using MY local price per kWh, I am going to use the national average cost for germany, which at this time, appears to be roughly 32c / kWh.

Results? Much more expensive, but, efficient option wins by a large margin.

Despite being identical to the first example, except in terms of cost/kwh- The more efficient option wins by FAR here.

Example #3 – Used Corporate SFF vs Intel NUC

In this example, I will be comparing a used HP Z240 versus a brand-new intel NUC. I specifically use this example, because I see LOTS of people opting to get a NUC instead of a used piece of enterprise gear.

HP z240

Ebay: HP Z240 – 8GB DDR4 Ram – i5-6500

As of writing this article, the average cost of one of these shipped, is around 120-140$ USD.

Intel NUC

Amazon: NUC7PJYHN2 (Quad-Core Pentium Silver J5005, 8GB DDR4 RAM,

Cost as of the time of writing this, was 400$.

Hardware Comparison

CPU Benchmark

Off the bat, the older z240’s I5-6500 offers nearly double the performance as the “low power” celeron used in the NUC. Both machines offer 8G of DDR4 ram. Lets ignore the hard drive for now, both options can be had with NVMe storage for basically the same price.

Power Consumption

I DO have multiple HP Z240 desktops laying around my house. One is used for my opnsense firewall, another is used for a gaming PC in my livingroom.

I have personally measured my firewall’s power consumption under normal load, at 25w of consumption. This is a normal HP z240, with a i5-6500, 8G of ram, NVMe, and a 10G NIC. For worst-case scenario, lets assume it uses 40w on average which is WELL above what it actually uses.

I do NOT have a intel NUC laying around. However, its Documentation states < 15 watts under idle/sleep conditions. Lets assume best-case scenario, it uses 5 watts typical. This is likely to be well below its actual usage.

Results? SFF Wins.

Keep in mind, I nearly doubled my ACTUAL usage of my HP z240. I also gave the NUC a unrealistic average power consumption of 5 watts. As well, this was one of the cheapest NUCs you could spec out. Its CPU offers HALF of the performance as the older i5-6500 released in Q3 of 2015.

Effectively, this NUC machine will never reach ROI…. because you are going to throw it in the garbage can in a few months due to its underpowered processor!

Conclusion

Conclusion of this post- Do the math.

If you don’t like to do math? Fine. I provided a calculator above to do the math for you.

Just enter the values. If the ROI is >= 7 Years, I would say go with option 1. If the ROI is < 7 years, Go with option number two!

Obviously, I have relatively cheap electricity(Ignoring the solar panels going up soon as well). What works for me, will not work for someone living in a high-cost area such as germany where the average cost / kWh is 4 times greater then what I would spend.

So- in conclusion…. DO YOUR MATH!!!