Aaron Weiss

TrueNAS

TrueNAS Virtual Machine with Ubuntu 18.04 Halting with Signal 11

by

For about one week, I had an Ubuntu 18.04 installation that I’ve had running for years as TrueNAS virtual machine starting to fail after a few hours of running.

This VM not working is not the end of the world, but I had been using this as a utility knife for all sorts of things, and I didn’t want to chuck it to the curb just yet.

I was seeing the following in my TrueNAS Log:

Mar 13 11:47:56 maria kernel: pid 1552 (bhyve), jid 0, uid 0: exited on signal 11
Mar 13 11:47:58 maria kernel: vnet1: link state changed to DOWN

I had to force a power down and then restart the VM. Sometimes the VM wouldn’t start at all.

After finding an ancient article about a similar issue, someone had mentioned that when they switched their virtual machine’s NIC device from Intel e82858 (e1000) to VirtIO, that was the solution.

Changing the network interface device in a TrueNAS virtual machine running Ubuntu 18.04
Changing the network interface device in a TrueNAS virtual machine running Ubuntu 18.04

After 48 hours, my TrueNAS Virtual Machine was running stable.

It’s difficult to say why this began. I last updated TrueNAS to 12.08-U8 on February 2nd, 2022. I first noticed the TrueNAS VM wasn’t running when I was checking my internet speed graph in Grafana starting around March 3rd. However, I have unattended upgrades for this virtual machine set, so it’s possible there was an update to the packages that caused a conflict with Intel e82858 (e1000).

Perhaps this will be helpful to you.

FreeNAS: A Hero’s Journey

by

Building and maintaining a FreeNAS server in my home has been a rewarding experience that continues to provide me with value and an avenue to grow personally and professionally.

It’s an experience generally started with loss, some stability, neglect, some more loss, stress, and then eventually some stability and growth.

Backstory

I bought my Netgear ReadyNAS NV+ in 2010 after I suffered my very first HDD failure where I lost hours of video footage. Luckily, it was archival video, nothing urgent. The scenario left me yearning to never let it happen again.

I learned all about the available redundancy options for storage and backup, and set my sights on Netgear’s X-RAID feature with their ReadyNAS platform, allowing for 1 of 4 hard drive failures. I thought that would be enough for my budget at the time.

Around 2017, I received a notice that the machine would no longer be supported with software updates. Great. It was bound to happen. On a tight personal budget, I looked for new options but never committed to anything.

Inciting Incident

A late-Summer 2018 Florida thunderstorm sent my dog into a static-electricity induced panic attack, including heavy panting and a neediness where she was attached to my leg with imaginary heavy-duty Velcro.

Despite the many UPS units around my home, I failed to connect the coaxial cable that carried my broadband internet connection to the UPS. A lightning strike hit a nearby coaxial terminal that fed surrounding apartment buildings their internet. The surge carried itself to my modem which let out a internal thunderclap that also sprinkled its way through two additional WiFi routers, which were also destroyed in the surge.

The last in the chain was my ReadyNAS which was hit too. I was able to power on the device again, and after a file system check, it booted. But no network connectivity. The NIC was fried.

Call to Adventure

After the disappointment faded, action needed to take place. I needed to:

  1. Get access to my data.
  2. Find and implement a new solution. I narrowed my options to:
    1. Buying a more modern ReadyNAS device
    2. Building a DIY NAS using FreeNAS or other similar software

I had found similar instances of users’ NIC ports getting hit by surges and recoveries possible on the same device and model. It turns out that ReadyNAS’s X-RAID solution is just a Linux-based RAID with some bells and whistles, and there were many ways to retrieve this information. However, this would have required the need to build a separate machine that could connect to all 4 of my drives. Therefore, building a new machine would be the eventual course of action.

Crossing the First Threshold

It turned out that one could purchase a used ReadyNAS of the same model. Finding a version that matched my version was difficult, and often times I waited too long to bid before an auction was purchased from under me. After a few weeks I finally found a compatible version on eBay.

The moment I was able to re-install all my drives, boot, and see my data appear was a million pounds of stress off my shoulders. My data was safe, but I need a new solution to backup my main computer as soon as I could.

While I did have a great experience with ReadyNAS and Netgear products in general, I felt that I needed personal and professional growth, and FreeNAS would be my choice. I’ve built many computers in my life, but this would be the first that was specific to being a server and maintain it.

Land of Adventure

After nearly a month of research and deciding on a budget and components, I chose the following:

Spiritual Death and Rebirth

Once all the components came, installing them was tricky because the BitFenix MicroATX Case is a strange case not mean for a NAS, but I made it work anyway. I’ve considered moving my personal machine components to it, but that hasn’t happened. This would be one of the first lessons in this journey.

Road of Trials

Testing

Thanks to the plethora of FreeNAS information, I had plenty of tools ready to test the components before moving forward.

My testing was predominately done using Ultimate Boot CD, which included CPU and memory testing tools. Memtest passed as well as CPU burn in. I finally noticed that one of the not HDDs passing my test. RMA!!!!

A new drive arrived, but wasn’t being discovered at all. Even by my personal computer. That’s two White Label HDDs RMAed. I contacted the company, they had no problem with the RMA and even explained they’d test a HDD before it’s sent to me. I guess that shows you the profit margin on these White Label HDDs.

Installation

Installing FreeNAS is super simple. You really don’t need to know any command line knowledge at all. When you get to the screen that says “you want FreeNAS”, you just hit the “hell yes” button, and within 5 minutes or so, your FreeNAS is installed, and displays a few options for additional administration and an IP address to visit.

Refusal of the Return

After the testing and installation, I began to configure the system how I expected to use it. Creating Windows shares, labeling drives within the system, and one of the most rewarding aspects: finding and customizing shell scripts to perform testing and reports.

I was able to find scripts that can email SMART reports, UPS reports, and security reports to start. Learning shell scripting is something that I’ve been able to carry on to other server operations which would come later. One of the most important scripts creates a daily configuration file so that in case something happens to the OS or hardware, I’ll be able to get back up and running on a new system and maintain my configurations and pool.

The Return

With everything setup, I could finally transfer my data from one NAS to another. Since there’s about a 10 year difference between architecture, there was also about about a 10x difference in speed. The ReadyNAS could sustain a transfer rate of less than 30MBs where as the FreeNAS can get to just about 100MBs. At the <30MB rate, it would be a day and half of transfers.

Once everything was said and done, I was able to begin scheduling various backups and file syncs to bring myself back to a fully redundant workflow.

Freedom to Live

Thanks to several built-in plugins, I’ve been able to enjoy services such as Plex to watch videos on my FreeNAS from my Amazon Firestick.

I finally decommissioned my ReadyNAS, and sold both the working and non-working NV+ devices on eBay for some scratch money, and had the old hard drives destroyed.

The process was long, but fulfilling beyond just having my data redundant. I’ve been taken advantage of Virtual Machines and learning Linux in general, which is allowing me to grow further.