Just curious if anyone has crunched the numbers on this. Looking to expand my farm but not sure what is the best long term solution for best bang for buck and expandability. I am relatively new to this and only have 2 external 12 Tb HDD but want to slowly expand to something around 300tb+.
Here are my Opinions on each setup so far, feel free to correct me as I am still new at this:
External HDD: easy setup, plug and plot but expensive price per TB. ($20/TB) New ($15-$20 Used)
Are there any limitations to this? max amount of External Drives supported?
SATA: Feel like this is a better long term solution but Price per TB can still feel expensive at current Chia Prices.($20/Tb) new ($15/Tb) Used
Does SATA use PCIe Lanes? Would I have to get a CPU with a lot of PCie Lanes available such as Threadripper or an old Server?
SAS: Can find cheap drives since usually used in data center, a lot of bulk available but can be confusing to understand SAS setups and you need additional components like SAS controller and Expansion cards ($25/Tb) new ($10-$15/Tb) Used
Let me know what solution as worked best for you and what benefits does each storage solution have?
Does SATA use PCIe Lanes? Would I have to get a CPU with a lot of PCie Lanes available such as Threadripper or an old Server?
SATA controllers or HBAs (host bus adapters) use PCIe lanes. Most consumer motherboards have four or six SATA ports onboard, integrated into the chipset; server motherboards frequently have 10 or more. You can add additional ports by installing additional controllers in your motherboard’s PCIe expansion slots, or you can use expanders which typically just need power. For example you could get a card that goes into a x8 slot and adds eight SATA ports.
can be confusing to understand SAS setups and you need additional components like SAS controller and Expansion cards
There is definitely a learning curve with SAS, but it can add a lot of flexibility to your setup. (It can also add a lot of resiliency, but that’s not really a requirement for Chia farming.) SAS HBAs are usually of a higher quality and easier to find, and most will work fine with both SAS and SATA drives. SAS expanders also tend to be of a higher quality, but they can get a little pricey. That being said, with a cheap LSI SAS2008-based HBA and couple expanders like the Intel RES2SV240 or HP 727253 you can very easily scale out to dozens or even hundreds of drives—assuming you can figure out power, cooling, etc.
SAS HBAs and expanders also give you the ability to have external ports for connecting disk shelves, which can themselves be chained together. Most disk shelves have SAS expanders built into the backplane, so you can run e.g. a single four-lane mini-SAS connection to a 24-bay disk shelf, and a second mini-SAS connection from that shelf to a second shelf, etc. There are limitations, of course, but that should give you some idea of what’s possible.
I can see the appeal of external hard drives and I initially started out with 2 myself. But remember that inside of just about every external hard drive is an internal hard drive that will work just fine outside of it’s shell. All my current 17 farming drives are sata connected, including those first 2 that I shucked with the plots on them, and it all worked fine. There is a limit to the number of usb devices you can connect per controller. But it’s usually very high, more than 100.
I ultimately switched to doing everything sata based for several reasons.
Personal prejudice from working as a computer technician. I felt that long-term, connected for weeks and months at a time sata would be more stable. That might not be true in all circumstances, but seemed right to me at the time.
Less cable mess and clutter. I now have 17 drives and they are all in a single computer chassis. The only cables connected to the outside of it all the time are power and ethernet. Usually I use tightvnc to remote connect to it, and only connect a monitor keyboard and mouse when I have to. With external drives right now I’d have an extra 34 cables hanging around. A usb and power for each.
I usually buy whatever drives I think are a good deal for that size. You go back 10 months or so, and that was often external drives for some reason. Now it seems to be internal drives. Sata gives you more flexibility to use either. Just shuck the external drives. If you settled on external, you’d have to purchase usb enclosures for your internal drives separately.
When I run out of space in my current chassis, I plan on adding a sas controller to it, and connecting it to another chassis that will just have a power supply, sas expanders and drives. But the drives I connect to the sas expanders will probably be sata.
Ultimately you could do a bit of everything at first, see what you like better, and go from there.
I personally am not a fan of external usb drives. Pros USB Drive
powers on external 12 volts psu, so no hassle with 5 volts power limits on psu side when expanding farm
easy to set up. Plug in power brick, plug in usb, put in shelf
often found cheaper than internal enterprice drives
can be removed from the case to obtain an internal sata drive
Cons USB Drive
Especially in the case, not designed for 24/7 use, may break down faster due to bad ventilation / heat dissipation
Im not a fan of 10’s of usb hubs
Cable mess (the octopus of death )
due to the mess, might be hard to expand
might get into electricity issues by daisychaining those external power plugs on 100 drives
Drives may not be the same quality than enterprice capacity drives
2 years warranty only
MAY (not tested) use more power with inefficient power plug + internal usb adapter pcb
Sata pros (enterprice drives)
5 years warranty
can build/stack shelves with ventilation or even get enclosures
relatively cheap hardware compared to sas (pcie adapter cards are available for up to 24 drives per pcie card)
easier initial set up than sas drives
can be used as well with enterprice gear (most hdd enclosures and sas expanders also support sata)
mostly easy to configure or almost plug and play
Sata cons
probably a lot of troubleshooting when scaling up (sata expansion cards of varying quality, 5v rail psu electricity limitations, sata connection not as reliable, …)
when scaling up, tends to become a DIY solution with a lot of tinkering
can as well become a lot of cabling to connect 100 sata & power cables as compared to sas
SAS Pros
When set up correctly, probably the most reliable solution (hot swap, no cabling with disk carriers, …)
a lot of relatively cheap enterprice hardware around.
SAS Cons
probably you all of a sudden find your self building a datacenter with server rack, disk enclosures, etc. etc.
enterprice hardware is heck of noisy (mostly 40mm fans pushing air like in a windtunnel)
Enterprice hardware is very heavy (good luck installing a server rack etc on your own)
the cost adds up. Buying new enterprice equipment is a joke of what you get fo your money compared to sata DIY. Even used, you will find your self needing lots of equipment etc.
may be hard to set up correctly (console etc)
please note that sas and sata are more or less interchangable. You can run Sata drives in enterprice equipment. But usually you cant easily run sas drives on DIY equipment. This is why I made the separation with sata drives - diy and sas - enterprice equipment.
Keep in mind, that used enterprice SAS drives are usually not very large. The capacity of each drive has no effect on electricity use so116tb drive is twice as energy efficient than 2 8 TB drives. Additionally having many small drives needs more expander cards also adding some cost.
I scraped some sata drives from usb cases which I got cheaply used. However, with hdd prices falling, I doubt its worth to save a little on those drives while possibly voiding the warranty. I had some failures with these drives already.
In my opinion external usb 3.0 connection is the best one. For example wd elements use nas drives. For multiple usb drives you can connect them to a PSU thus avoiding multiple adaptors. Each drive needs about one amper voltage.
Actually, I was just doing some testing with a USB power meter (maybe I should have done this before ) There is power going over the USB 5V during powerup. I’ve seen between 0.32 and 0.48A so far, for a few seconds. During farming operations it’s negligible, less than 0,1 A
this was with a WD elements 10TB
SAS, is superior is pretty much all ways. This is why datacenters use it.
Most importantly it scales very well. Secondly you can use both sata and sas drives, giving more option to find good second hand deals.
But if we are talking about 300TB farm, we are basically talking about 15-25 hard drives, you can go whichever way is most convenient/cheap for you.
There are plenty cheap consumer desktop cases that can fit 10 drives, and some have optional drive caddy’s that can expand that to 15 or 20. If your mainboard has 6-8 sata port, you just need to add in one 12-port sata expansion card and you’re done.
Want to expand? get a second one and set it up as harvester.
If you get some second hand mobo+cpu set, this could well be the cheapest option.
It also depends on your plans for plotting. Dual xeon servers are great for plotting, so if you are going to use one of those, you might as well stick with enterprise gear.
For the record, I use all three
I have a dual Xeon system with a HBA connected to the 8 hotswap bays in the Chassis.
8 Sata ports, connected to hdd’s outside the case in DIY rig.
6 external USB drives
still got 4 sata ports left.
Yes, I know about the theoretical maximums but I am at 20 drives so far. I wonder if the lookup times can still keep up and if this works in IT mode with the operating system.
In theory, Linux will just continue the mounting pattern: sda-sdz…sdaa…sdaaaa
Yes, it shouldn’t be an issue unless you’re maxing out the throughput of the SAS connection. A four-lane SAS-2 channel is 24Gbps so you’d have be doing some heavy I/O for there to be any impactful latency.
)
) There is power going over the USB 5V during powerup. I’ve seen between 0.32 and 0.48A so far, for a few seconds. During farming operations it’s negligible, less than 0,1 A