Aaron Weiss

freenas

Using Raspberry Pi For PiHole Ad Blocking and Network UPS Tool Monitoring

by

When I first built my FreeNAS machine, I bought an UPS device to give it backup power in case of a power outage. But my primary computer also had a UPS device.

Recently, in order to minimize the amount of equipment that I have, I decided I only wanted to have one UPS. After searching for solutions, I was pointed to Network UPS Tools (NUT), which is open source software that can manage UPS units and the machines they power. This way I can have my primary computer and NAS connected to the same UPS unit, and control them both during certain power events.

Before this, I also started to run PiHole on a VM with my NAS, but I quickly found two issues. First, if the VM halted for some reason, my entire network would too. Also, if I needed to restart my NAS for any reason, the network would also go down.

This finally gave me two solid reasons and goals to justify the purchase a Raspberry Pi. Luckily, the Raspberry Pi 4B 2GB model had just become the primary model at $35, so I knew I’d have plenty of performance to run these two applications.

Since I was not only looking to minimize my homelab’s physical footprint, I also wanted to keep cords and cables to a minimum. So I also decided to get a Power-Over-Ethernet Switch that would allow me to power my Raspberry Pis with just an ethernet connection. I also wanted to reduce the amount of Ethernet wire lengths to make cable management easier, so I purchased a Cat 6 crimper to create my own ethernet cable lengths.

Finally, this new RaspberryPi would allow me to run cron jobs that control scripts on remote machines from the Pi as well such as  my Digital Ocean Snapshot script.

So to sum up my goals:

  • Run NUT server to monitor my NAS and Windows 10 machine
  • Run PiHole
  • Execute cron jobs

Network UPS Tools Equipment

Below are all the items I purchased for this project.

ItemQuantityPriceStore
Raspberry Pi 4 2GB2$70PiShop
Rasberry Pi Case1$5PiShop
POE Hat2$41.90PiShop
16GB SD Card3$17.97Amazon
High Pi Case1$9.95PiShop
SD to USB Card Reader1$9.99Amazon
Total$154.81

The above doesn’t include shipping or the cost of a mini HDMI cable as I already had one lying around. There’s three SD cards in this list to have one as a backup.

First Raspberry Pi

The Pi and its accessories are incredibly small and easy to put together. The POE switch worked perfectly. I was able to install Raspbian on a SD card and start up the Pi with no issues with the Raspberry Pi Foundation’s Imager Tool. Once I got the basic settings just so, including assigning a static IP address. I disconnected the HDMI and connected via VNC instead, using VNC Viewer.

I typically name my devices after robots from film history. My FreeNAS is named Maria after the female protagonist in Metropolis, and I’ve maintained that naming convention for many of the Virtual Machines, networking equipment, and now my Pi. I named it Worker11811 after the character whose job was to maintain the Moloch machine.

Installing PiHole

This is an easy step, especially since I had already been running it in a VM. You simply run this code:

curl -sSL https://install.pi-hole.net | bash

That is the basic install and runs your through a few configuration options such as choosing your DNS host (I chose Cloudflare), your static IP address, etc.

PiHole Teleporter
Backing up a PiHole and moving it to another installation is easy with the PiHole Teleporter. Click to enlarge.

Since I already had custom rules and settings from my VM, I used PiHole’s Teleporter to download the settings from the original VM, and then restore it to the newly installed PiHole.

I updated my Router’s DNS settings to point to 192.168.1.3, and boom, PiHole was working as expected.

Installing NUT

The Pi will act as both the NUT server and client server as the Pi will act as both. First, I connected my UPS to my Pi and ran lsusb to ensure it could see the UPS.

pi@worker11811-pi:~ $ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0764:0501 Cyber Power System, Inc. CP1500 AVR UPS
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

No issues there. It sees my CyberPower UPS.

Installing NUT isn’t difficult on Raspbian as there is a Debian package available, and found a few excellent instructions like this one from Medium:

apt-get update && apt-get install nut nut-client nut-server

Next, I added the following lines to the sudo nano /etc/nut/ups.conf file. I found my USB driver for the UPS using this Hardware Compatibility List. It was the same driver that my FreeNAS was using, so I knew it worked well.

[ups]
driver = usbhid-ups
port = auto
desc = "CyberPower 1350PFLCD"

Next, I created users for each of the clients that will be connecting to this NUT server by editing my /etc/nut/upsd.users file:

[admin]password = ******
actions = SET
instcmds = ALL

[upsmon_worker11811]
password  = ******
upsmon master

[upsmon_maria]
password = *******
upsmon slave

[upsmon_johnny5]
password = ******
upsmon slave

As you can see, there are 4 users. The Admin, worker11811, maria, and Johnny5 (which is my primary Windows 10 machine).

Then I started the nut-client and drivers:

systemctl restart nut-client.service
upsdrvctl stop
upsdrvctl start

Next, I checked to see if I can see the UPS device via the NUT

pi@worker11811-pi:~ $ upsc ups@192.168.1.3
Init SSL without certificate database
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 20
battery.mfr.date: CPS
battery.runtime: 1470
battery.runtime.low: 600
battery.type: PbAcid
battery.voltage: 24.0
battery.voltage.nominal: 24
device.mfr: CP1350PFCLCD
device.model: CRCA102*AF1
device.serial: CPS
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: CyberPower HID 0.4
driver.version.internal: 0.41
input.transfer.high: 139
input.transfer.low: 88
input.voltage: 123.0
input.voltage.nominal: 120
output.voltage: 139.0
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.delay.start: 30
ups.load: 22
ups.mfr: CP1350PFCLCD
ups.model: CRCA102*AF1
ups.productid: 0501
ups.realpower.nominal: 810
ups.serial: CPS
ups.status: OL
ups.test.result: Done and warning
ups.timer.shutdown: -60
ups.timer.start: -60
ups.vendorid: 0764

Additionally, you can run upslog which will check the UPS’s status every 30 seconds:

pi@worker11811-pi:/usr/local/bin $ upslog
Network UPS Tools upslog 2.7.4
logging status of ups@localhost to - (30s intervals)
Init SSL without certificate database
20200905 115423 100 123.0 20 [OL] NA NA
20200905 115453 100 123.0 23 [OL] NA NA
20200905 115505 100 123.0 23 [OL] NA NA
Signal 2: exiting

Lastly, we can check to see if NUT sees the same machine as a client:

pi@worker11811-pi:~ $ upsc -c ups
Init SSL without certificate database
127.0.0.1

The 127.0.0.1 stands for the home machine which is my worker11811 Raspberry Pi. We’ll be using the same command a few more times later.

Updating FreeNAS UPS Settings to become a NUT Client

Now that NUT is working, I had to update the FreeNAS settings. FreeNAS comes with NUT already, making this super easy to update. I made the following changes:

  • Switched UPS Mode from Master to Slave (I hate this terminology)
  • Added the Remote Host IP to 192.168.1.3 which is my Pi’s internal network address
  • Added the Monitor Username and Password
FreeNAS UPS Settings
Click to enlarge.

After saving, I immediately noticed that my FreeNAS recognized the connection to my NUT Server on my Pi via the shell.

To double-check, I ran the same upsc ups@192.168.1.3 command on the FreeNAS shell, and received the same results when I initially ran it on the Pi. Of course, we’ll run the same client command as before:

pi@worker11811-pi:~ $ upsc -c ups
Init SSL without certificate database
192.168.1.9
127.0.0.1

Now it sees my FreeNAS IP address so I’m in good shape here.

Adding Windows 10 as a NUT Client

This is where the real headaches occurred for me at first. Windows support is severely lacking. I was only able to find an ancient winNUT Google Code repository and a MSI installer directly on the NUT site. After attempting to get both to work and nearly a week of frustration, I uninstalled both.

I stumbled upon this Github repo called WinNut-Client which has a super-simple interface to setup your Windows machine connection to your NUT server. After downloading the MSI file, installing, and setting up the config I had my Windows machine capturing the UPS data and was able to setup the settings to choose when and how I want this machine to shutdown. There’s even a hibernate option which I prefer over a full shutdown which I prefer if I’m away from my computer.

WinNUT Client Options

You can also choose how the service starts at the machine’s startup, how often it looks for updates, etc. What I didn’t see is the client appear on my NUT Server, so I opened an Issue on the Github repo to get clarification.

WinNut Client Status
Click to enlarge.

Monitoring the UPS with NUT

NUT comes with several CGI programs that can be viewed in a browser. However, I had issues attempting to log into http://192.168.1.3/cgi-bin/nut/upsstats.cgi. After some time, I realized that PiHole uses Litehttp web server that already uses port 80. Wherease, NUT uses Apache. I updated /etc/apache2/ports.conf and changed Listen 80 to Listen 8080 and restarted Apache. Going to http://192.168.1.3:8080/cgi-bin/nut/upsstats.cgi brought up a result that reminded me of the internet’s nascent days:

NUT Monitoring Page
Example of the Network UPS Tools Stats Page. Click to enlarge.
UPS Stats Chart
Example of the individual UPS stats. Click to enlarge.

Regardless, my UPS’s stats were easily revealed to me.

Issues

Temperatures

The 4th generation Pis run rather hot, more so with the POE hat, which even comes with a fan that sits right on top of the CPU. When inside the official case, the temperatures were reaching the high 70s in Celsius. This was quite worrying. I ended up purchasing the HighPi case which is meant to provide extra room for taller hats. I figured this would be an excellent way to hopefully get more air to the Pi. This ended up reducing the temperatures, but not to where I really prefer.

I was able to find instructions to configure the Pi to manage the POE fan speeds to keep the fan from being on all the time. Otherwise, there’s a high-pitched whine. Of course, as I was testing this, I had to restart the Pi each time. So there’s a balance between the speed of the fan and the temperatures that I feel are appropriate. The goal is generally to keep the CPU below 80 degrees as that is when the CPU begins to throttle performance.

Network downtime with reboots

The first issue I encountered was that if I need to reboot my Pi when I was working on it or installing generally updates, it would take down my network for about a minute. This was essentially the same problem I experienced prior to this project. So, I really hadn’t fixed this problem.

Second Raspberry Pi

The downtime issues is where I realized that I’d prefer to have a second PiHole machine in order to have 100% uptime. Therefore, I purchased another Pi and POE hat, but chose to remain with the official Pi case I already had. To continue the Metropolis naming scheme, I named this one Rotwang after the primary antagonist.

I ran through the same processes as before, except I chose not to run a graphic interface with this Pi, using just SSH to manage the device. After I installed PiHole, and ran the Transporter process to have mirror settings through the web interface.

I had found this Reddit post of someone sharing their script on how to keep two PiHoles synced. I don’t believe it works that well, but essentially, if one Pi goes down or is rebooted, I still have a network connection.

While writing this post, I recently stumbled upon this post about sycning two PiHoles, that might be a better option than what I have now. I expect to try it out sometime in the future. (EDIT/UPDATE [Sept. 7th, 2020]: This script works as described and is super easy to install and run.)

Backing up the Pis & Data Recovery

One thing I learned quickly is that the Pis don’t have a power off button. So if they lose power abruptly without powering down, it could corrupt the SD Card data.

I found this super-simple Pi backup script on GitHub, and it’s been working like a charm. I have these backups sent to their respective folders on my FreeNAS, so they are always in a safe place.

Even recently this has been helpful. In attempting to make some changes to one of the Pis, it wouldn’t reboot. Instead of getting mad, I was able to write the backup image to the backup SD card and reboot with no issues.

Conclusion

Entering into the Raspberry Pi universe has been fun and I see why these little guys are so popular. Anytime I start to find myself relying on a Virtual Machine in my FreeNAS machine, I start to consider whether or not it’s better to run that application on one of these Pis or add an additional one.

I still worry about the temperatures for both Pis, which are running in the low 70 degree Celcius range. After removing both Pis from their cases for a few days and found them to operate within the 50 degree range, which is much more reasonable to me. I am considering a Cloudlet case such as this one  where I can have all the Pis setup in an array with plenty of airflow and the ability to run a POE switch near by.

Worker11811 Temperature
Rotwang Pi Temperatures

Overall, I’m super happy with this setup, and I recognize that I certainly over-engineered this entirely. I spent over $150 just to have one less UPS near my gear. Honestly, I could have just added a Serial Connect from the UPS to the NAS and maintained the USB connection to my Windows machine. I’ve learned a lot and I’m happy with what I’ve accomplished.

How FreeNAS and WP-CLI Grew My Interest in Linux and Automation

by

Last year, I built a FreeNAS server. Initially, it was only meant as a means to store my computer backups and house my music and videos.

However, to do it right, meant I needed to perform commands in the shell, mostly to test the hard drives before I began to store files on them. I found an excellent resource, but I didn’t know what any of commands meant. I executed them and waited until they were done.

The same was for Bash scripts to automate system configuration backups, reports, and notifications.

It was when I stumbled across a some YouTube videos on how to run an Ubuntu Server to host your own websites did I finally test the Virtual Machine waters FreeNAS offered. I installed  Ubuntu 18.04 Server LTS on a VM, and learned a little at a time. The idea that I could learn a new operating system without buying another computer floored me.

Setting Goals

With VMs, CLI, and some basic web server understanding under my belt, I was ready to take a leap and move aaronweiss.me to a Digital Ocean server, but with the following goals:

  1. Separate WordPress Environments:
    • Development (DEV): Any new plugins, theme enhancements, or other changes that would affect the WordPress installation or how the software worked would be developed and tested on WordPress installation.  Plugin, theme, and core updates would also be completed and tested on this server.
    • Quality Assurance (QA): This environment was meant to test any changes made in the DEV environment as if it were a functional website. No changes would be made to this environment except common WordPress functions such as adding and managing posts and pages.
    • Production (PROD): This would the live website visible to the public. Like QA, major changes would not be made on this environment.
  2. Automated Deployment Scripts: Deploy changes from DEV to QA and then QA to PROD
  3. Maintenance Scripts: Create a script to check for security vulnerabilities, cleanup temporary files, backup site, optimize database, and compress images on all three environments.

The above goals meant I could successfully, host, develop, and maintain my website using a secure approach with lots of ways to quickly get up to speed if something were to happen.

Additional Achievements Unlocked

Once I achieved these goals, I was hooked on what else I could do. My next set of goals were:

  • Create an automated Digital Ocean snapshot script. Digital Ocean has a backup options, but only does so once per week. That didn’t fly with me, so I wrote DOCTL Remote Snapshots as a way to have some control of how often and how many snapshots would be created.
  • Learn GIT – I’ve had some Git knowledge through Microsoft Team Foundation Server at work. However, it was time to really learn Git. I combined this with my DOCTL Remote Snapshot script and now have a published repository.

Next Up:

  • Create a website monitoring script. I don’t need server up time, I need to know website up time. I want to know that my website can fully load and perform its basic tasks throughout the day.
  • Build a Raspberry Pi and install:
    • PiHole. PiHole is an free, open source ad blocker.
    • NUT (Network UPS Tool). The goal of this is a script to monitor two computers from Raspberry Pi and shut them down gracefully using one Uninterruptible Power Supply. I currently have two UPSs, one for my primary computer and one for my FreeNAS. The primary one can handle up to 850 watts which is enough to cover all my devices, but only has one UPS port to monitor the primary device. Ideally, NUT will allow monitoring over Ethernet and can handle the shutdown of both machines.
    • Additionally, these two programs also feed my yearning to want to build and learn Raspberry Pi.

These are some short-term goals that I think are obtainable for the future.

FreeNAS: A Hero’s Journey

by

Building and maintaining a FreeNAS server in my home has been a rewarding experience that continues to provide me with value and an avenue to grow personally and professionally.

It’s an experience generally started with loss, some stability, neglect, some more loss, stress, and then eventually some stability and growth.

Backstory

I bought my Netgear ReadyNAS NV+ in 2010 after I suffered my very first HDD failure where I lost hours of video footage. Luckily, it was archival video, nothing urgent. The scenario left me yearning to never let it happen again.

I learned all about the available redundancy options for storage and backup, and set my sights on Netgear’s X-RAID feature with their ReadyNAS platform, allowing for 1 of 4 hard drive failures. I thought that would be enough for my budget at the time.

Around 2017, I received a notice that the machine would no longer be supported with software updates. Great. It was bound to happen. On a tight personal budget, I looked for new options but never committed to anything.

Inciting Incident

A late-Summer 2018 Florida thunderstorm sent my dog into a static-electricity induced panic attack, including heavy panting and a neediness where she was attached to my leg with imaginary heavy-duty Velcro.

Despite the many UPS units around my home, I failed to connect the coaxial cable that carried my broadband internet connection to the UPS. A lightning strike hit a nearby coaxial terminal that fed surrounding apartment buildings their internet. The surge carried itself to my modem which let out a internal thunderclap that also sprinkled its way through two additional WiFi routers, which were also destroyed in the surge.

The last in the chain was my ReadyNAS which was hit too. I was able to power on the device again, and after a file system check, it booted. But no network connectivity. The NIC was fried.

Call to Adventure

After the disappointment faded, action needed to take place. I needed to:

  1. Get access to my data.
  2. Find and implement a new solution. I narrowed my options to:
    1. Buying a more modern ReadyNAS device
    2. Building a DIY NAS using FreeNAS or other similar software

I had found similar instances of users’ NIC ports getting hit by surges and recoveries possible on the same device and model. It turns out that ReadyNAS’s X-RAID solution is just a Linux-based RAID with some bells and whistles, and there were many ways to retrieve this information. However, this would have required the need to build a separate machine that could connect to all 4 of my drives. Therefore, building a new machine would be the eventual course of action.

Crossing the First Threshold

It turned out that one could purchase a used ReadyNAS of the same model. Finding a version that matched my version was difficult, and often times I waited too long to bid before an auction was purchased from under me. After a few weeks I finally found a compatible version on eBay.

The moment I was able to re-install all my drives, boot, and see my data appear was a million pounds of stress off my shoulders. My data was safe, but I need a new solution to backup my main computer as soon as I could.

While I did have a great experience with ReadyNAS and Netgear products in general, I felt that I needed personal and professional growth, and FreeNAS would be my choice. I’ve built many computers in my life, but this would be the first that was specific to being a server and maintain it.

Land of Adventure

After nearly a month of research and deciding on a budget and components, I chose the following:

Spiritual Death and Rebirth

Once all the components came, installing them was tricky because the BitFenix MicroATX Case is a strange case not mean for a NAS, but I made it work anyway. I’ve considered moving my personal machine components to it, but that hasn’t happened. This would be one of the first lessons in this journey.

Road of Trials

Testing

Thanks to the plethora of FreeNAS information, I had plenty of tools ready to test the components before moving forward.

My testing was predominately done using Ultimate Boot CD, which included CPU and memory testing tools. Memtest passed as well as CPU burn in. I finally noticed that one of the not HDDs passing my test. RMA!!!!

A new drive arrived, but wasn’t being discovered at all. Even by my personal computer. That’s two White Label HDDs RMAed. I contacted the company, they had no problem with the RMA and even explained they’d test a HDD before it’s sent to me. I guess that shows you the profit margin on these White Label HDDs.

Installation

Installing FreeNAS is super simple. You really don’t need to know any command line knowledge at all. When you get to the screen that says “you want FreeNAS”, you just hit the “hell yes” button, and within 5 minutes or so, your FreeNAS is installed, and displays a few options for additional administration and an IP address to visit.

Refusal of the Return

After the testing and installation, I began to configure the system how I expected to use it. Creating Windows shares, labeling drives within the system, and one of the most rewarding aspects: finding and customizing shell scripts to perform testing and reports.

I was able to find scripts that can email SMART reports, UPS reports, and security reports to start. Learning shell scripting is something that I’ve been able to carry on to other server operations which would come later. One of the most important scripts creates a daily configuration file so that in case something happens to the OS or hardware, I’ll be able to get back up and running on a new system and maintain my configurations and pool.

The Return

With everything setup, I could finally transfer my data from one NAS to another. Since there’s about a 10 year difference between architecture, there was also about about a 10x difference in speed. The ReadyNAS could sustain a transfer rate of less than 30MBs where as the FreeNAS can get to just about 100MBs. At the <30MB rate, it would be a day and half of transfers.

Once everything was said and done, I was able to begin scheduling various backups and file syncs to bring myself back to a fully redundant workflow.

Freedom to Live

Thanks to several built-in plugins, I’ve been able to enjoy services such as Plex to watch videos on my FreeNAS from my Amazon Firestick.

I finally decommissioned my ReadyNAS, and sold both the working and non-working NV+ devices on eBay for some scratch money, and had the old hard drives destroyed.

The process was long, but fulfilling beyond just having my data redundant. I’ve been taken advantage of Virtual Machines and learning Linux in general, which is allowing me to grow further.

You Need a Backup and Disaster Recovery Plan

by

You’ll never realize that a backup and and disaster recovery plan will help you sleep better at night if you run a website, even if you never have to recover your website.

Recently, two hosting platforms and their users suffered missteps.

a2 Hosting, a shared hosting provider I’ve been using since 2013, has had their Windows servers shut down for over a week as the company suffers from a ransomware attack. Additionally, the available backups the company has for customers appear to be over 2 months old.

DigitalOcean mistook a user’s script as a crypto-mining operation, and shut down a startup’s servers.

I’m fortunate to not be affected as my a2 Hosting account is Linux-based, and my DigitalOcean VPS is a low-profile risk. However, this is devastating for these company and their users. I’m sure there are terms of service policies that cover these hosting companies for situations like these to a certain aspect.

There’s a much to learn from these situations, and this is a good time to reflect on having plans for your website in situations like these.

Restorable backup plan

There’s no excuse not to have a backup plan and infrastructure for your computer, websites, and any important data. Here are some of the backups I have set in my digital life:

  • For my main computer, I have a full weekly backup with daily incremental backups, that are then synced to my FreeNAS box, which are also synced to a Backblaze B2 Bucket.
  • For my FreeNAS server, I have a backup of the config file that is backed up to Dropbox, Backblaze, and mirrored on a second USB drive.
  • For my websites, my entire cPanel host instance is backed up each week, and then downloaded to my FreeNAS server. The individual websites have backups with BackupBuddy which have weekly and daily schedules relative to their respective performance, which are then synced with Dropbox. Some sites also backup to Amazon S3.

As you can see, take my data very seriously. Some data has 3 or 4 destinations. I’m ready to launch a new computer image, FreeNAS box, or return my entire cPanel instance or individual website back from the dead.

In fact, I recently had a botched release of improvements to this very website that went poorly. I was able to bring the site back up in less than 30 minutes because I have the infrastructure and documentation in place to recover.

Disaster Recovery Plan and Exercises

Just having a back up isn’t enough. Knowing how to recover those backups is an important aspect too.

In the case of a2 Hosting, had someone had recent backups of their website, they could have found a new service, restored their backups, and changed their DNS to the new service. After DNS propagation, a website could return to full operations within 24 hours at the latest.

At my day job, I’ve participated in Disaster Recovery Exercises where I help validate whether or not applications I use can perform critical tasks after the recovery begins. It’s a boring exercise, but I now see how important it really is.

My recommendation is to have a test environment that is nearly identical to your site’s live environment, do something to make it no longer work, and then restore the site from a backup. You might even want to try and see if you can also find a new vendor and restore your site to that vendor. Having that knowledge will help you sleep better at night.

Planning for the Future

Despite this situation, I’m still sticking with a2 Hosting and DigitalOcean for the immediate future. a2 Hosting has been a great partner, and I have only a few support tickets with them since I started. I know if I was on the other side of the table, I’d be furious. Companies make mistakes, and no company is infallible.

The moral of this story is: a company as large as these two companies are, they should have had their customer’s data backed up to a separate location (although customers should be responsible for their own data) and had a plan in place to return their service to functionality at a quicker pace.

You don’t have to be a2 Hosting or DigitalOcean, or their users. You now have the knowledge to be better.