Best products from r/freenas

Don't worry about a ZIL/Slog or L2Arc device. I have ~100TB across a couple FreeNAS servers. One is a racked dual xeon hexacore 24-bay enclosure, and the other is a standalone little xeon quadcore 8-bay enclosure. I use Plex on both, haven't utilized anything beyond the spinning rust, and it works just fine. IF you run into issues you think can be resolved by it, then look into it. For now, don't worry about it.

64gb is plenty.

Mirroring m.2 drives for boot is a bit overkill. I'm booting from a single USB drive in my big racked NAS, and from a single m.2 in my little desk NAS. Boot from a cheap SSD or USB drive, then save your config somewhere safe. If the drive ever fails, you simply replace, reinstall, upload config. The OS loads into memory once each time you reboot (once every month or three). Mirror the m.2 drives and use them for your Plex jail if you want it to load metadata quickly. I'd even go with smaller, cheaper drives.

Transcoding doesn't need flash memory. Your CPU will be the bottleneck. Don't forget that even UHD Blu-rays can be streamed from a single HDD... If you have modern devices, you shouldn't even need transcoding within the home. I direct stream HEVC HDR blurays from my e3 Xeon over Wifi to my Sony smart tv without any issues. h.264 can be hardware-decoded by everything, and even h.265/hevc hardware decoding is pretty common. I wouldn't expect transcoding to be necessary excep when bandwidth is limited by streaming outside the home, or when streaming to old devices.

Try 1gbe before upgrading. I understand wanting 10gbit -- but wait until it's necessary. Even a 50mbps bluray direct-stream will only use ~5-10% of your 1gb link. Once you NEED 10gbit, it's likely that the prices on switches will be lower. I have 10gbit between my desktop and my little NAS just to make transfers speedy.

You will have a little under 50TB usable with that configuration. See here: https://wintelguy.com/zfs-calc.pl

What's going on with the harddrives? Would the Element drives be good for a NAS? I prefer HGST above WD, and WD above Seagate, but I would give these a chance at the current prices:

10TB, $259

https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST10000NM0086/dp/B01LXXV880?psc=1&SubscriptionId=AKIAJA5VHS6TIVNX3PRQ&tag=synack-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B01LXXV880

https://www.amazon.com/Seagate-Enterprise-Capacity-Cache-Internal-ST10000NM0016/dp/B01DSRHFOK?SubscriptionId=AKIAJA5VHS6TIVNX3PRQ&tag=synack-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B01DSRHFOK

12TB, $325

https://www.amazon.com/Seagate-Exos-Internal-Drive-Enterprise/dp/B0759Q9FXZ?SubscriptionId=AKIAJA5VHS6TIVNX3PRQ&tag=synack-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B0759Q9FXZ

14TB, $388

https://www.amazon.com/Western-Digital-14TB-Ultrastar-HC530/dp/B07KPL474H?psc=1&SubscriptionId=AKIAJA5VHS6TIVNX3PRQ&tag=synack-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B07KPL474H

Jeez, I can't imagine how cramped a 12bay midtower would be to work on. Not a midtower, but this is what I use: https://smile.amazon.com/gp/product/B008O2HOI2 [CM Storm Stryker - Full Tower SGC-5000W-KWN1] Works great. I ended up putting https://smile.amazon.com/gp/product/B00DGZ42SM Rosewill 4 x 3.5-Inch Hot-swap cage to help w/the hotswap portion. Works great. Good luck with whatever you decide!

My original setup went like this:

I have an IBM m1015 I picked up off of eBay and flashed to IT mode so that the drives are passed straight through and no RAID functionality of the card is used.

One of the two ports from it was connected via an SFF-8087 SAS cable to a double SFF-8087 to SFF-8088 adapter, but you can go single as well.

Then an SFF-8088 SAS Cable ran from the adapter to an external case with an HP SAS Expander in it. There are many, many, other models of expanders but this is what I went with because it has the external SAS SFF-8088 port on it (it's not the only one that has this feature, but it's what I went with, and it makes it so every enclosure doesn't need its own motherboard).

All the ports off the expander then went to the backplanes in the external case and connected the drives to the IBM m1015 through the expander.

Since the 4224 did not have a motherboard at the time I used this thing that's usually used for crypto currency mining to provide power to the expander because that's all it needs from its PCIe interface (many of those other models of expander have a Molex connector in it for power and no PCIe interface at all).

As for daisy chaining SAS enclosures, it can be done, but I haven't gotten that far in my storage adventures yet. I do know that there are enclosures that support it and if I were to expand from where I'm at now it's probably the route I would go. Rolling my own was fun and all but sometimes you just want to plug and play.

I'll concur with A_watcher that eSATA enclosures are crap. Or at least the one I've used was as well.

I think that address the questions that were in the OP as well as posed to others that responded.

I'm ridiculously far from an expert, and my setup has changed a lot since the first way it was set up, but I think it was pretty close and answered those questions.

*Many the guys over at /r/datahoarder are much more knowledgeable in this arena than I am and are another source of information when it comes to storage.

Yes, FreeNAS-11.2-U5, was also working on U4.

I literally just plugged it in and plugged a drive in to it and it worked. I also have switched drives between the controller and the motherboard and that works too without breaking my pools. Support for the Marvell 9215 chipset was apparently added in FreeBSD 9.2... Not sure why you would be having trouble, does the card work in a different PC?

Oh fuck my bad, I Just double checked and THIS is the one I got... Slightly different but super close image wise. The one I got has the Marvel 9215, the one OP linked has Marvel 9235.. But based on a quick google it looks like the 9235 was added to be supported in FreeBSD 9 as well...

This ASRock C2550D4I board is in the same ballpark pricewise and has lots of nice things (12x SATA, IPMI, dual wan) and is only slightly less powerful than the 2750. This is a board I use.

I run 6 jails, one of which is Plex, and it does a nice job. It isn't overly powerful, but does everything I ask of it (serving 4 computers, 2 phones, and 2 iPads) while gently sipping power, which is pretty awesome for something that is on 24/7.

Op got me curios and we can into the same thing:

>You could use a PCIE expander card like this Asus Hyper M.2 x16 Card Expansion NV Me M.2 Drives and Speed up to 128Gbps Components https://www.amazon.com/dp/B0753JTJTG/ref=cm_sw_r_cp_apa_aLQ-BbAQ4JE2Y

I wanted to point out that this adapter is designed to be used with Intel VROC on X299 Series Motherboard models. Here are the ASUS supported motherboards:

• ROG RAMPAGE VI EXTREME
  
• ROG RAMPAGE VI APEX
  
• ROG STRIX X299-XE GAMING
  
• ROG STRIX X299-E GAMING
  
• PRIME X299-DELUXE
  
• PRIME X299-A
  
• TUF X299 MARK 1
  
• TUF X299 MARK 2
  
  EDIT: From the Amazon link, a great comment found in a review about his adapter. Things to consider with the motherboard & CPU:
  
  >This cards has a lot of variables that need to be meet before it will work.
  
  >1. 16 pci express lanes is a lot. Most CPU's will not have enough for this, and two GPU's for example. As that is 48 pci lanes needed for 2 GPU's and one of these cards fully loaded. So lets say you buy a super expensive I9 Intel Processor. Well that CPU only has 44 pci lanes. Meaning at max you can only get 3 of the M.2 cards to work in this device while still having two GPU's running all 16 of there lanes.
  
  >2. Mother boards are deceptive. While they may on average have 4 slots that will fit a x16 card. They do not all have 16 CPU express lanes assigned to them. Most often only the first slot (the one closest to the CPU socket), and the third slot actually have 16 lanes. The other two slots will only have 8. Even on high end boards. Meaning that only two of the M.2 cards will be recognized when plugged in there. So check your motherboard manuals first.
  
  >3. Intel requires you to also buy a VROC key. This is not required on AMD. I have seen some people blame that on this card.
  
  >4. Sense the speed of this card is heavily dependent on the slot you put it in, and the CPU running the system. Is why it says you can get up to 128Gbps. Realistically though... Unless you really know what you are doing you won't hit those speeds. But it will still be faster then what your used too. Most SSD's are limited by the sata cable speed, and other things. Which won't be anywhere as fast as m.2 drives raided.
  
  >5. This tech is still new. So not all motherboards support it. Actually most don't. While the card itself doesn't need any drivers to run. The MB and CPU both need to support NVMe raid. Or your better off getting a single m.2 card.
  
  OP's desired use though sound different from you and myself might be thinking of. He wants a small NAS but has not stated his intended use yet. Where-as this sparked an interest in how fast could one possibly get a NAS to transfer data. I think in this setup one would saturate their network long before R/W speeds of their arrays, lmao.

Have you looked into how well that Avaton will do for Plex encoding? Its not exactly the most powerful thing out there... You can get a Xeon E31241 and a SuperMicro X10SLL-F (that's what I have) for $300 and $150, which is less total than your chosen motherboard/cpu combo.

Edit: Also trade out that memory for some server memory with ECC support. I used Crucial but a lot of people seem to like Kingston RAM.

CPU

Motherboard

RAM

Edit 2: I just noticed that whole Canadian thing. Maybe its not as expensive as I realized....but do look into how much transcoding with Plex you want to do and what the Avaton can do and you might want to spend a bit more and go with a more powerful chip like the stuff I suggested.

If you save while making it work now, I would suggest the WD Red line. I bought 4x 4TB WD Reds to form a Raid-Z2 for my FreeNAS build.

WD Red 2TB NAS Hard Disk Drive - 5400 RPM Class SATA 6 Gb/s 64MB Cache 3.5 Inch - WD20EFRX https://www.amazon.com/dp/B008JJLZ7G/ref=cm_sw_r_cp_apa_i_0cl5AbFYGQG0M

I haven't played with it yet, but I got a Mikrotik CRS305-1G-4S+IN. https://www.amazon.com/gp/product/B07LFKGP1L/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1

Cheapest >2 port SFP+ switch I've seen.

And yeah, UBNT is great, but the cheapest switch they have with SFP+ is 500 bucks. Which is an awesome switch, but total overkill for most applications.

If it's just a data drive and you're not looking to do anything super fancy with it. These work great.

However, be advised that, because the card is PCI-E x1, if you were to actually plug in 4 hard drives or SSDs, you're gonna run smack against bandwidth limitations if you start trying to hammer I/O on the drives connected to the card all at once.

I have this exact card as well as the 2-port PCI-E x2 slot version in use and they work very well for supplementing on-board headers when you're a few short.

I would not attempt to use these cards to run HDDs/SSDs that were going to be datastores for VMs nor as the HBA for something like FreeNAS. If your goal is along those lines, you'd be much better off looking for an HBA like the 9207-8i. You can get those way, way cheap on ebay, and then you just need the correct cables for 'em.

You could use a PCIE expander card like this Asus Hyper M.2 x16 Card Expansion NV Me M.2 Drives and Speed up to 128Gbps Components https://www.amazon.com/dp/B0753JTJTG/ref=cm_sw_r_cp_apa_aLQ-BbAQ4JE2Y

Find a motherboard and chipset that has a lot of PCIE lanes and supports bifurcation (so those 4xNVMe drives can split up the shared PCIE slot).

I've also thought about doing this just because.....well because it would just be fun to see how fast you could make it. Lolz

RAM should be fine. 32GB is plenty to run reliably. I have several systems with 64GB of RAM that have 365TB useable. Not sure about those HDDs. I haven't done much research on SMR drives in a FreeNAS system. Perhaps somebody else can tell you more about that. As far as a boot drive goes, grab yourself 2 16GB Cruzer Fit USB drives and use those for a mirrored boot drive. Real easy to do. That mobo only has 2 USB ports total, so I'd snag something like this:
http://www.amazon.com/Motherboard-Female-Header-Adapter-Cable/dp/B015F6QXKO/ref=sr_1_2?s=electronics&ie=UTF8&qid=1457296058&sr=1-2&keywords=usb+header+dual
Attach it to the header inside, and leave your boot drives in the box, which frees up your limited USBs on the back.

I built a box very similar to yours but with the ASrock C2750, and 6TB WD Reds. One thing to look out for is that your mobo has 6 Intel SATA ports and 6 Marvell SATA ports. Do not use the Marvell ports for your set up, or your customer is going to have nothing but trouble. Marvell controllers are notoriously bad in FreeNAS set ups, and although you may initially see all of your disks, you best believe those suckers randomly stop being able to see them. I read that folks have had success only using them in Mirror set ups, where 1 disk of the mirror is on the Intel side and the other disk is on the Marvell side.

For a case maybe a node 804? Or a Silverstone ds380? If you can't find a micro atx or mini itx motherboard with enough sata ports you'll need a controller in IT mode. The mini itx is smaller but you'll be limited on memory ports and sata ports most likely. I suggest building it with pcpartpicker to ensure compatibility. You can also filter for various qualities like number of sata ports or number of 3.5 inch hard drives things like that.

I'm going to second going for an actual server here. I generally hate HP, but I like their servers. Personally, I have the HP Micro Gen8 server: https://www.amazon.com/HP-ProLiant-MicroServer-Server-783958S01/dp/B00LGJ9NHK and I love it.


The latest Freenas isn't compatible with it though (FreeBSD boot issue), so I'm stuck on one version lower than current. Hopefully next major version the issue will get fixed...

You're welcome!
Link to the case on Norco's website, take special note of the PSU depth max length
Here are a few parts that you will need if you do decide to get that case.
FlexATX PSU The ones on amazon at the time were too long to fit in the case, i had to return the one i bought from them and ended up getting this one.
Reverse Breakout SATA Cable
2x 4PIN MOLEX Extension cables This is to plug into the HDD backplane

Thanks. Sounds like a good plan. Note that "replacement... card" won't help you if it's the same model number. It's not a fault in a bad HBA card, but a limitation on it, so every single SI-PEX40064 card is going to be a bottleneck on the drives connected to it.

Here is a FreeNAS Forum post about choosing an HBA/RAID card. This is generally considered the best card for FreeNAS when it's in IT mode, which comes from that forum post.

Obviously it's much more expensive than your current one, but you get what you pay for, unfortunately, when it comes to RAID/HBA cards and bandwidth/features.

One important note: That "8GB is the minimum" comment is only relevant for when you are using ZFS on FreeNAS. UFS does not need more than 4GB in most cases, but 8GB is the recommended minimum for ZFS.

>If it's just a data drive and you're not looking to do anything super fancy with it. These work great.

I had a card with the same chipset and it was utter shite. Seemed to work fine, but I got all kinds of SMART errors (related to data transfer, not the drives themselces), drives would spontaneously be kicked from their vdev array, drive commands would fail, etc.

I recommend never using PCI SATA expanders. You can get an LSI HBA for the same price plus rock solid performance and better speed.

>-I'm using 2 mirrored Corsair M510 480gb NVME M.2 drives as the main boot drive. This will also be where the metadata for my Plex server will be stored, so I wanted it to be very snappy/responsive.

I could be wrong, but I don't think you can use the boot pool for storing VM or Jail data. You'd be better off just buying a cheap low capacity MLC SSD like this one for just your FreeNAS installation. https://www.amazon.ca/Silicon-Power-Endurance-2-5-Inch-SP060GBSS3S60S25AE/dp/B01M2UUACN/ref=sr_1_21?keywords=64gb+MLC+SSD&qid=1569235568&sr=8-21

You're better off using your NVME M.2 drives as a separate mirror pool for all your VM/Plugins/Jails and Plex transcoding. You won't need to use a RAM disk.

Then i would just opt for a board like this, or something similar. https://www.amazon.com/gp/product/B00F0YROSC/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

The Avoton chip is plenty powerful for what you are going to be doing and will save you a ton on power & thermals.

I use one of these: IO Crest 4 Port SATA III PCI-e 2.0 x1 Controller Card Marvell Non-Raid with Low Profile Bracket SI-PEX40064 https://www.amazon.com/dp/B00AZ9T3OU/ref=cm_sw_r_cp_apa_.8j.zbB9E0901

It works well, but I don't do tons of read/write as I only is mine for backups. I would recommend to get hba if you're going to use it for a lot...going to be better for u in the long run.

What about this one? Seems to have decent reviews.

MBD-A1SAI-2750F-O

It's a bit more than the ASRock C2550D4I, but it's 8-core, and only around $90 CAD more. Might be worth spending a little extra if the quality is supposedly better.

I may experiment with Plex a little, and those extra cores might be handy.

This is much better. Dual Marvell 9215 https://www.amazon.com/Crest-Non-RAID-Controller-Supports-FreeNAS/dp/B07NFRXQHC/ref=sr_1_3?keywords=I%2FO+Crest+8+Port+SATA+III&qid=1563899507&s=electronics&sr=1-3

I/O Crest 8 Port SATA III Non-RAID PCI-e x4 Controller Card Supports FreeNAS and ZFS RAID - Includes Mini SAS to SATA Break Out Cables

I am literally looking to build almost the exact same system that you have listed, and I trying to decide between the C2750 vs the C2550. I do want to run a Plex jail and do some transcoding, so I am leaning towards the 2750. But the quad-core is $150 cheaper from Amazon here in the US.

On the other hand, with the hard drives being the largest cost, adding an extra $150 doesn't seem to be that big of a deal.

In any case, looking at Amazon UK it looks like a seller in Germany has the C2550 for 240 Euros. Not sure how that compares to what you are looking to pay for the 2750.

I would probably just buy 2TB WD Reds for $1.50 more -> https://www.amazon.com/Red-2TB-Hard-Disk-Drive/dp/B008JJLZ7G/ref=sr_1_1?ie=UTF8&qid=1486573277&sr=8-1&keywords=wd+red+2tb

Similar to the DS380, but I've had the Silverstone CS380B for several months and I'm very pleased with it. I got it used from craigslist, but it's been a decent case so far. https://www.amazon.com/Silverstone-Technology-CS380B-Storage-Cases/dp/B01MA6NJE8

​

SFF8087 to 4x SATA.

Like these:
http://www.amazon.com/StarTech-com-SAS8087S450-Serial-Attached-Cable/dp/B004KDO22K/ref=sr_1_11?ie=UTF8&qid=1396212048&sr=8-11&keywords=sff+8087

http://www.amazon.com/Monoprice-Internal-SFF-8087-Forward-Breakout/dp/B005E2XTO8/ref=sr_1_4?ie=UTF8&qid=1396212048&sr=8-4&keywords=sff+8087

http://www.amazon.com/Discrete-SFF-8087-Reverse-breakout-cable/dp/B002MK7F0Y/ref=sr_1_7?ie=UTF8&qid=1396212048&sr=8-7&keywords=sff+8087

you can add internal usbs. Something like this is what u are looking for http://www.amazon.com/StarTech-Motherboard-4-Pin-Header-USBMBADAPT/dp/B000IV6S9S
Or
http://www.amazon.com/gp/aw/d/B015F6QXKO?psc=1

I only point it out because I have the same board (and freaking love it). Here's the RAM I've got, threw in 32GB for good measure

https://www.amazon.com/gp/product/B008EMA5VU/ref=oh_aui_detailpage_o04_s01?ie=UTF8&psc=1


When did this implementation take place? I hadn't heard of it. I bought mine right about a year ago.

edit holy shit, I got that ram for $90/pair, I can't believe it's almost doubled in a year

Like the other commenters have said, you don't want a RAID card, but either an HBA or SATA expansion card.

Here's a 4-port on on amazon.es

There's plenty of reviews of the same model on amazon.com saying it works out-of-the-box with FreeNAS.

Here is the HBA I use with FreeNAS.

LSI Logic SAS 9207-8i Storage Controller LSI00301 https://www.amazon.com/dp/B0085FT2JC/

You will also need these to connect drives to that card.
Cable Matters Internal Mini SAS to SATA Cable https://www.amazon.com/dp/B012BPLYJC/

I bought it about a year ago on german amazon. (There is now english translated version of german amazon.)

I bought it from a seller called 'Jacob Elektronik'. Also the price difference between the C2550D4I and C2750D4I isn't quite as large anymore but it's still over 100€.

You just need a processor with a lot of PCIe lanes and some of these

When you upgrade can I buy your old board and RAM?




Is it this RAM?




https://www.amazon.com/gp/aw/d/B008EMA5VU?psc=1&ref=yo\_pop\_mb\_pd\_title