Aaron Weiss

freenas

FreeNAS: A Hero’s Journey

by

Building and maintaining a FreeNAS server in my home has been a rewarding experience that continues to provide me with value and an avenue to grow personally and professionally.It's an experience generally started with loss, some stability, neglect, some more loss, stress, and then eventually some stability and growth.

Backstory

I bought my Netgear ReadyNAS NV+ in 2010 after I suffered my very first HDD failure where I lost hours of video footage. Luckily, it was archival video, nothing urgent. The scenario left me yearning to never let it happen again.I learned all about the available redundancy options for storage and backup, and set my sights on Netgear's X-RAID feature with their ReadyNAS platform, allowing for 1 of 4 hard drive failures. I thought that would be enough for my budget at the time.Around 2017, I received a notice that the machine would no longer be supported with software updates. Great. It was bound to happen. On a tight personal budget, I looked for new options but never committed to anything.

Inciting Incident

A late-Summer 2018 Florida thunderstorm sent my dog into a static-electricity induced panic attack, including heavy panting and a neediness where she was attached to my leg with imaginary heavy-duty Velcro.Despite the many UPS units around my home, I failed to connect the coaxial cable that carried my broadband internet connection to the UPS. A lightning strike hit a nearby coaxial terminal that fed surrounding apartment buildings their internet. The surge carried itself to my modem which let out a internal thunderclap that also sprinkled its way through two additional WiFi routers, which were also destroyed in the surge.The last in the chain was my ReadyNAS which was hit too. I was able to power on the device again, and after a file system check, it booted. But no network connectivity. The NIC was fried.

Call to Adventure

After the disappointment faded, action needed to take place. I needed to:
  1. Get access to my data.
  2. Find and implement a new solution. I narrowed my options to:
    1. Buying a more modern ReadyNAS device
    2. Building a DIY NAS using FreeNAS or other similar software
I had found similar instances of users' NIC ports getting hit by surges and recoveries possible on the same device and model. It turns out that ReadyNAS's X-RAID solution is just a Linux-based RAID with some bells and whistles, and there were many ways to retrieve this information. However, this would have required the need to build a separate machine that could connect to all 4 of my drives. Therefore, building a new machine would be the eventual course of action.

Crossing the First Threshold

It turned out that one could purchase a used ReadyNAS of the same model. Finding a version that matched my version was difficult, and often times I waited too long to bid before an auction was purchased from under me. After a few weeks I finally found a compatible version on eBay.The moment I was able to re-install all my drives, boot, and see my data appear was a million pounds of stress off my shoulders. My data was safe, but I need a new solution to backup my main computer as soon as I could.While I did have a great experience with ReadyNAS and Netgear products in general, I felt that I needed personal and professional growth, and FreeNAS would be my choice. I've built many computers in my life, but this would be the first that was specific to being a server and maintain it.

Land of Adventure

After nearly a month of research and deciding on a budget and components, I chose the following:

Spiritual Death and Rebirth

Once all the components came, installing them was tricky because the BitFenix MicroATX Case is a strange case not mean for a NAS, but I made it work anyway. I've considered moving my personal machine components to it, but that hasn't happened. This would be one of the first lessons in this journey.

Road of Trials

Testing

Thanks to the plethora of FreeNAS information, I had plenty of tools ready to test the components before moving forward.My testing was predominately done using Ultimate Boot CD, which included CPU and memory testing tools. Memtest passed as well as CPU burn in. I finally noticed that one of the not HDDs passing my test. RMA!!!!A new drive arrived, but wasn't being discovered at all. Even by my personal computer. That's two White Label HDDs RMAed. I contacted the company, they had no problem with the RMA and even explained they'd test a HDD before it's sent to me. I guess that shows you the profit margin on these White Label HDDs.

Installation

Installing FreeNAS is super simple. You really don't need to know any command line knowledge at all. When you get to the screen that says "you want FreeNAS", you just hit the "hell yes" button, and within 5 minutes or so, your FreeNAS is installed, and displays a few options for additional administration and an IP address to visit.

Refusal of the Return

After the testing and installation, I began to configure the system how I expected to use it. Creating Windows shares, labeling drives within the system, and one of the most rewarding aspects: finding and customizing shell scripts to perform testing and reports.I was able to find scripts that can email SMART reports, UPS reports, and security reports to start. Learning shell scripting is something that I've been able to carry on to other server operations which would come later. One of the most important scripts creates a daily configuration file so that in case something happens to the OS or hardware, I'll be able to get back up and running on a new system and maintain my configurations and pool.

The Return

With everything setup, I could finally transfer my data from one NAS to another. Since there's about a 10 year difference between architecture, there was also about about a 10x difference in speed. The ReadyNAS could sustain a transfer rate of less than 30MBs where as the FreeNAS can get to just about 100MBs. At the <30MB rate, it would be a day and half of transfers.Once everything was said and done, I was able to begin scheduling various backups and file syncs to bring myself back to a fully redundant workflow.

Freedom to Live

Thanks to several built-in plugins, I've been able to enjoy services such as Plex to watch videos on my FreeNAS from my Amazon Firestick.I finally decommissioned my ReadyNAS, and sold both the working and non-working NV+ devices on eBay for some scratch money, and had the old hard drives destroyed.The process was long, but fulfilling beyond just having my data redundant. I've been taken advantage of Virtual Machines and learning Linux in general, which is allowing me to grow further.

You Need a Backup and Disaster Recovery Plan

by

You'll never realize that a backup and and disaster recovery plan will help you sleep better at night if you run a website, even if you never have to recover your website.Recently, two hosting platforms and their users suffered missteps.a2 Hosting, a shared hosting provider I've been using since 2013, has had their Windows servers shut down for over a week as the company suffers from a ransomware attack. Additionally, the available backups the company has for customers appear to be over 2 months old.DigitalOcean mistook a user's script as a crypto-mining operation, and shut down a startup's servers.I'm fortunate to not be affected as my a2 Hosting account is Linux-based, and my DigitalOcean VPS is a low-profile risk. However, this is devastating for these company and their users. I'm sure there are terms of service policies that cover these hosting companies for situations like these to a certain aspect.There's a much to learn from these situations, and this is a good time to reflect on having plans for your website in situations like these.

Restorable backup plan

There's no excuse not to have a backup plan and infrastructure for your computer, websites, and any important data. Here are some of the backups I have set in my digital life:
  • For my main computer, I have a full weekly backup with daily incremental backups, that are then synced to my FreeNAS box, which are also synced to a Backblaze B2 Bucket.
  • For my FreeNAS server, I have a backup of the config file that is backed up to Dropbox, Backblaze, and mirrored on a second USB drive.
  • For my websites, my entire cPanel host instance is backed up each week, and then downloaded to my FreeNAS server. The individual websites have backups with BackupBuddy which have weekly and daily schedules relative to their respective performance, which are then synced with Dropbox. Some sites also backup to Amazon S3.
As you can see, take my data very seriously. Some data has 3 or 4 destinations. I'm ready to launch a new computer image, FreeNAS box, or return my entire cPanel instance or individual website back from the dead.In fact, I recently had a botched release of improvements to this very website that went poorly. I was able to bring the site back up in less than 30 minutes because I have the infrastructure and documentation in place to recover.

Disaster Recovery Plan and Exercises

Just having a back up isn't enough. Knowing how to recover those backups is an important aspect too.In the case of a2 Hosting, had someone had recent backups of their website, they could have found a new service, restored their backups, and changed their DNS to the new service. After DNS propagation, a website could return to full operations within 24 hours at the latest.At my day job, I've participated in Disaster Recovery Exercises where I help validate whether or not applications I use can perform critical tasks after the recovery begins. It's a boring exercise, but I now see how important it really is.My recommendation is to have a test environment that is nearly identical to your site's live environment, do something to make it no longer work, and then restore the site from a backup. You might even want to try and see if you can also find a new vendor and restore your site to that vendor. Having that knowledge will help you sleep better at night.

Planning for the Future

Despite this situation, I'm still sticking with a2 Hosting and DigitalOcean for the immediate future. a2 Hosting has been a great partner, and I have only a few support tickets with them since I started. I know if I was on the other side of the table, I'd be furious. Companies make mistakes, and no company is infallible.The moral of this story is: a company as large as these two companies are, they should have had their customer's data backed up to a separate location (although customers should be responsible for their own data) and had a plan in place to return their service to functionality at a quicker pace.You don't have to be a2 Hosting or DigitalOcean, or their users. You now have the knowledge to be better.