I want to know what is the maximum drives per device (per motherboard) you can achieve.
and in what connection type (sata, usb,… etc.) and OS.
Only actual data please. I’ve heard about 128 drives per device for many times, but have not seen an actual set up even once.
May I ask you to also share your total hardware cost of farming divided by number of drive. Example: $1000 PC and 20 drives, so it’s equal $50/drive.
This is the farming focused, so plotting hardware should not be included. Assume it’s at the stage where everything is plotted.
My current consumer set up. Combine plotter and farmer.
i7-11700 desktop = $55 per drive which is poor.
I have 80 SAS/SATA disks attached to one server, I will go up to 96 on this device before moving to any kind of multi-harvester model.
That server is running an EPYC 7551, on a H11SSL-I motherboard with a 16 port LSI SAS HBA in almost every PCIe slot (+ 16 SATA ports on the board). I could feasibly use SAS expanders to get 3x the number of SAS ports, but they’d have to be externally mounted and they’re not easy to get hold of for a good price any more.
I’ve using also s SAS setup with a LSI HBA 9207-4i4e, which has a SFF-8087 4 lane internal port, which holds 16x mixed SATA/SAS drives through a Supermicro backplane and a SFF-8088 4 lane external port, which is actually connected to a NetApp DS4246 (24x 3.5" Bays) and a NetApp DS2246 (24x 2.5" Bays). You can daisy chain multiple disk enclosures to get up to 256 drives though a single 4 lane sas host port.
The 256 drives limitation is the upper limit of my HBA card.
I think in theory I could do that from every HBA, i.e. daisy chain 256 drives * 6 HBAs = 1536 disks, I don’t think I’m going to go quite that big, though. I bought a load of HBA cards very early when the prices were low, I wish I had bought a load of expanders as well.
I do have a couple of 8 port → 32 port expander cards, but I’m going to use them on another board I think, I have some 8 port cards that don’t justify a PCIe slot on their own, but my experiments show you can run expander PCIe cards without any mobo, with just a powered riser (like the ones for GPU mining).
JBODs too expensive for my taste, I have like the guts of JBODs (backplanes, expander cards) but outside of any chassis. Still trawling ebay for bargains but I like to keep connection costs under 10GBP per disk
When I bought my SAS HBAs, they were 15 GBP each, when I bought the cables that connect the SAS HBAs to disks, they were about 15 GBP each, and each card needs 4 of them.
(15+(15*4))/16 = 4.6875 GBP
Power supply side, I use quite expensive power supplies, I could make this aspect cheaper, but currently including the cost of splitters, it costs about another 2.50GBP per disk.
So in total, it cost me about 7.20 GBP per disk.
This doesn’t include screws, strips of metal to mount the disks on, rack space, etc, or the server motherboard, I had most of that stuff already (the server motherboard isn’t essential - you could stick the same HBAs onto a couple of 2nd hand ebay office PCs).
These prices are no longer correct though, the cables I use have gone up to about 20 GBP each, and 8 port SAS cards are now about 30 GBP each, power supply costs are relatively unchanged, so if I did the same thing again with more disks, it would cost about 9.40 GBP a disk, hence looking at backplanes, expanders etc to see if I can get the costs down in future.
My Supermicro 3HE server with 16x 3.5" bays was already present at my home with the LSI 9207-4i4e.
If you have HW with a empty PCIe slot, you will need only a HBA, the 9207-4i4e costs about 50 till 70 € on ebay, the NetApp DS4246 with everything included was about 350 € and the NetApp DS2246 with everything included was about 90 €. Then I’ve bought a SFF-8088 to QSFP (HBA to NetApp) cable for about 40 € and a QSFP to QSFP (NetApp to NetApp) cable for 15 €.
LSI 9207-4i4e: 70 €
NetApp DS4243: 350 €
NetApp DS2246: 90 €
SFF-8088 to QSFP cable: 40 €
QSFP to QSFP cable: 15 €
This summs up to 565 € to get 24x 3.5" and 24x 2.5" HDDs running with good power supply and good cooling. To extend this setup, you I’ll need another 350€ + 15€ for 24x 3.5" HDDs and 90€ + 15€ for 24x 2.5" HDDs if the prices are the same as I bought my HW. The DS4243 was bought on may and the DS2246 was bought end of june.