Hi there, I came across 9 SYBA disk enclosures (the 8 bay ones). This one: 8 Bay 2.5 Inch and 3.5 Inch SATA Hard Drive External USB 3.0 Hard Drive Enclosure
However I’m now wondering how I can connect them to my HP Z840 with PCI-E cards and USB 3 hubs. Has anyone got a recommendation? No matter what I do, I can’t seem to get more than 16 drives visualized in Ubuntu. I don’t care about the speed issues for farming, but I simply can’t see all the drives… driving me nuts… thanks!
Not all USB controllers are built the same.
I have two of those 8-bay Syba enclosures connected to a sub $100 mini PC, running Windows 10 Pro, and they max out at those two. When I tried connecting a 3rd Syba enclosure, nothing happened.
So I purchased another sub $100 mini PC, and Syba enclosure #3 works fine.
Any combination of those Syba enclosures work on any of my mini PCs, as long as I keep it at two.
I did the above, before compression. My mini PCs are incapable of handling compressed plots.
But they exhibit the same “quantity two – max” as you have run in to.
Note that USB supports a finite number of connections, and the hubs, themselves, eat into that finite number. And the Syba units are also glorified hubs.
I do have other USB controllers in other PCs that handle more than 16 drives, including using hubs. So it is a USB controller thing – as far as I can tell.
I had the same problem with USB lane limit. I ended up adding more USB controller into motherboard. It was messy.
My solution is using cheap old SAS array. Neat and simple. $30 can buy a good HBA card. That SYBA thing is $200 and it is $25 per hard drive housing.
Above thing (posted one year ago), STL3, can now go for less than $150 complete. That is $10 per hard drive housing. Used SAS hard drive is also getting cheaper and cheaper.
It is. There is a practical limit of around 96-128 USB ‘Endpoints’ per USB controller, depending on controller specification and implementation. This is a USB specification and design limit and cannot be circumvented.
A USB Endpoint is a set of resources that support the connection of a USB device (mouse, keyboard, camera, HDD etc). Most HDD’s on USB 3.x require at least 3 endpoints. Some require 5 endpoints.
All other devices on USB also require the use of endpoints. Root hubs built into the USB controllers require endpoints too. Given the overhead of a controller and it’s root hub(s) requiring some endpoints, and given that some internal devices of a PC hang off USB without you even knowing about it in some cases, and given that a HDD can use at least 3, sometimes 5, and those hard drives that have an additional USB port or two on the front so you can plug other stuff in (sounds great) use even more endpoints, it doesn’t take long to run out of endpoint resources on the USB controller and thus problems are encountered.
The solutions are to plan carefully and/or add more USB controllers on the PCIe bus;
Some final points - anyone trying to get many USB HDD’s to work for Chia Farming (or anything else for that matter) needs to understand the terminology;
Note that many of the external multi-bay USB docks/bays you see (like IcyBox ones etc) are really little more than a box with a built in PSU and USB hub and provide a convenient way to attach drives to a USB controller. I don’t know how many endpoints these suck up as I don’t have any. Maybe someone who does can advise ?
If you have spare PCIe slots in your motherboard, you can install additional USB controller cards. But don’t fall into a trap where adding a cheap card with 6x ports on it makes you think you can add another 6x USB hubs and another 8x HDD’s per port on the USB hub. It doesn’t work like that. You need additional USB controllers (often referred to as host controllers’) to support additional USB devices.
I have used these Startech PEXUSB3S44V USB host controller cards to good effect on my farm:
https://www.amazon.co.uk/dp/B00HJZEA2S?ref=ppx_yo2ov_dt_b_product_details&th=1
These have 4x controllers onboard - in theory supporting typically upwards of 400+ USB endpoints on this card alone. These are in addition to the endpoints supported by the USB host controller on the PC motherboard.
I have also used these USB hubs to good effect with amazing reliability with large numbers of USB HDD’s:
https://www.amazon.co.uk/gp/product/B07KHRLSTT/ref=ox_sc_act_title_6?smid=A1E3IQLEA4G42D&psc=1
Two of my farming PC’s, which each have their onboard USB host controllers, also have 2x Startech PEXUSB3S44V USB host controller cards linked above installed. To each of these cards’ ports I have one of the 16-port Sabrent USB hubs attached, and to each port on the hub, a USB HDD.
So I have 9x 16-port USB hubs on one PC, with various sized HDD’s, and with a few USB HDD’s on the PC’s onboard USB ports too - totaling around 150 HDD’s on one PC. I forget the total endpoint count but it’s less than (or around) 100 per USB host controller. No issues whatsoever.
I have an identical setup on the second PC. This gives me over 300 HDD’s attached to just two PC’s and provides over 3.5PB for my farm, using variously sized USB HDD’s.
So it can work, but requires planning and careful choice of hardware.
Lots of good info.
By the way, I am using a StarTech card with 6 or 7 (can’t remember) 3.x ports.
I do not know how many of those ports are using their own controller.
But I have dozens of drives plugged in, and all is well.
I also have that Sabrent 16 port hub, which gave me problems – so I stopped using it.
After adding 4 or 5 drives to it, it seemed to work fine – that is, until I tried to copy plots between two drives connected to that Sabrent hub. I was constantly getting hangs, and could not restart the copy job, without power-cycling the hub. And then it would hang, again if I tried a plot transfer.
No matter which two drives I tried doing a plot transfer with, it would hang. Not always on the first transfer. But if not on the first plot transfer, then on the next one.
So I did not trust it, even for the small amount of harvesting data, as I was worried that after X bytes of data, it might hang. Now it sits in a box.
Perhaps mine was a lemon?
Yes, could have been a bad device I guess, it happens. Did you have it powered by it’s own PSU ?
All of my USB drives and all of my USB hubs are powered by a separate 12V PSU - I don’t use the individual power bricks they came with but use a common 12V rail from a dedicated, quality (industrial) PSU (well, actually one per PC hosting these drives…).
On the other hand, I haven’t really done much in the way of bulk transfers using the current setup - most if not all drives were filled with separate plotter PC’s in the OG days, and the drives then moved and added to their current resting place when full.
If/when I decide to re-plot I may have to move the drives to fill them, putting them back later.
Yes.
17, 18, 19, 20.
Thanks, however I need it to be quiet and I had 2 15drive EMC JBODs that were making a hell of a noise… Also, I have 72 SATA hard drives and the 9 SYBA housings, so a bit stuck with that…
Wow, wow, wow. Thanks for taking the time to spill this out in so much detail. The fact that you actually got this to work is encouraging to me. Understanding the importance of a number of controllers, let me share my setup with you, maybe it’s really down to the fact that my USB PCI-e cards are on the lower end:
Per UNRAID, this is how it looks with regards to the controllers:
* [8086:8d31] 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
* Bus 003 Device 001 Port 3-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 003 Device 002 Port 3-13 ID 0451:8142 Texas Instruments, Inc. TUSB8041 4-Port Hub
* Bus 003 Device 003 Port 3-13.1 ID ffff:5678 USB Disk 2.0
* Bus 004 Device 001 Port 4-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 004 Device 002 Port 4-4 ID 0451:8140 Texas Instruments, Inc. TUSB8041 4-Port Hub
* [8086:8d2d] 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
* Bus 001 Device 001 Port 1-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 001 Device 002 Port 1-1 ID 8087:800a Intel Corp. Hub
* [8086:8d26] 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
* Bus 002 Device 001 Port 2-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 002 Device 002 Port 2-1 ID 8087:8002 Intel Corp. 8 channel internal hub
* [1912:0014] 02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
* Bus 005 Device 001 Port 5-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 005 Device 002 Port 5-1 ID 045b:0209 Hitachi, Ltd
* Bus 006 Device 001 Port 6-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 006 Device 002 Port 6-1 ID 045b:0210 Hitachi, Ltd
* [1b73:1100] 03:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
* Bus 007 Device 001 Port 7-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 007 Device 002 Port 7-4 ID 1d5c:5011 Fresco Logic USB2.0 Hub
* Bus 008 Device 001 Port 8-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 008 Device 002 Port 8-4 ID 1d5c:5001 Fresco Logic USB3.0 Hub
* [1b73:1100] 06:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
* Bus 011 Device 001 Port 11-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 011 Device 002 Port 11-4 ID 1d5c:5011 Fresco Logic USB2.0 Hub
* Bus 012 Device 001 Port 12-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 012 Device 002 Port 12-4 ID 1d5c:5001 Fresco Logic USB3.0 Hub
Or in a more compact format:
* Bus 001 Device 001 Port 1-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 001 Device 002 Port 1-1 ID 8087:800a Intel Corp. Hub
* Bus 002 Device 001 Port 2-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 002 Device 002 Port 2-1 ID 8087:8002 Intel Corp. 8 channel internal hub
* Bus 003 Device 001 Port 3-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 003 Device 002 Port 3-13 ID 0451:8142 Texas Instruments, Inc. TUSB8041 4-Port Hub
* Bus 003 Device 003 Port 3-13.1 ID ffff:5678 USB Disk 2.0
* Bus 004 Device 001 Port 4-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 004 Device 002 Port 4-4 ID 0451:8140 Texas Instruments, Inc. TUSB8041 4-Port Hub
* Bus 005 Device 001 Port 5-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 005 Device 002 Port 5-1 ID 045b:0209 Hitachi, Ltd
* Bus 006 Device 001 Port 6-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 006 Device 002 Port 6-1 ID 045b:0210 Hitachi, Ltd
* Bus 007 Device 001 Port 7-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 007 Device 002 Port 7-4 ID 1d5c:5011 Fresco Logic USB2.0 Hub
* Bus 008 Device 001 Port 8-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 008 Device 002 Port 8-4 ID 1d5c:5001 Fresco Logic USB3.0 Hub
* Bus 011 Device 001 Port 11-0 ID 1d6b:0002 Linux Foundation 2.0 root hub
* Bus 011 Device 002 Port 11-4 ID 1d5c:5011 Fresco Logic USB2.0 Hub
* Bus 012 Device 001 Port 12-0 ID 1d6b:0003 Linux Foundation 3.0 root hub
* Bus 012 Device 002 Port 12-4 ID 1d5c:5001 Fresco Logic USB3.0 Hub
However, per HUB, I seem to be limited to 16 hard drives (i.e. 2 syba enclosures). Each drive is recognized individually, so it’s not a problem of the drive IDs not being passed through.
When I run lsusb with 24 drives attached, I only get 16 recognized. This doesn’t make sense to me…
root@super:~# lsusb
Bus 002 Device 002: ID 8087:8002 Intel Corp. 8 channel internal hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 012 Device 002: ID 1d5c:5001 Fresco Logic USB3.0 Hub
Bus 012 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 011 Device 002: ID 1d5c:5011 Fresco Logic USB2.0 Hub
Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:800a Intel Corp. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 0451:8140 Texas Instruments, Inc. TUSB8041 4-Port Hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID ffff:5678 USB Disk 2.0
Bus 003 Device 002: ID 0451:8142 Texas Instruments, Inc. TUSB8041 4-Port Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 002: ID 1d5c:5001 Fresco Logic USB3.0 Hub
Bus 008 Device 013: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 012: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 011: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 009: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 005: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 010: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 008: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 007: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 006: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 004: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 003: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 024: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 025: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 022: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 020: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 018: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 016: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 023: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 021: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 019: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 017: ID 2ce5:0014 InX8 Inc [AKiTiO] Mass Storage [NT2 U31C]
Bus 008 Device 015: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 014: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 002: ID 1d5c:5011 Fresco Logic USB2.0 Hub
Bus 007 Device 005: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 004: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 003: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 016: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 008: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 007: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 006: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 002: ID 045b:0210 Hitachi, Ltd
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 002: ID 045b:0209 Hitachi, Ltd
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Would you have an ideas what else I could do? I’m contemplating ordering the Startech USB controllers you referened, as maybe my noname devices can’t handle more than 2 enclosures…
So, it looks like you have 3x PCIe addon USB controllers as well as the 1x Intel controller which is onboard the HP Z840 motherboard (it looks like it could be interpreted as 3x Intel host controllers from the output dump but I suspect it is in fact only one ? Any way you can confirm from the spec of the motherboard in that Z840 ?).
In theory, that’s 4x controllers in total. If we assume as a basic start point your controllers support the typical 96 endpoints each, and if each drive was a USB attached drive using 3 endpoints each, that’s 72 drives x 3 endpoints = 216 endpoints required (bare minimum). On the face of it, that would work with those 4x controllers, but you have 9x 8-bay SYBA USB drive chassis’ to house the 72 drives (9x8 = 72).
I suspect, as mentioned above, that each of the SYBA chassis has it’s own internal USB hub with 8x USB–>SATA interfaces, which will likely use 2 endpoints each. Depending on the internal layout of the SYBA USB hubs, it may be grouped as 2x 4-port USB hubs, which are connected internally to a root hub, each of which will use 2x endpoints at least.
If the SYBA units have some kind of USB port(s) on the rear or front to connect additional USB devices (do they ?), these will likely also take 2 endpoints (even if nothing is connected to them) as they will effectively be a hub too.
Anyway, I would guess, unless you can confirm otherwise through spec/data sheets, that each SYBA unit will require the following endpoints as a minimum:
2 for USB root hub = 2
2 for each of the sub USB hubs (usually in banks of 4 ports) = 4
3 (possibly 5) for each of the 8x bays = 24 or 40
So potentially 46 endpoints when populated with drives. When you add this to the endpoints used internally by a USB controller (for it’s root hub(s)) then only one SYBA unit might be possible per controller, if the controller has a practical limit of 64 endpoints. Even if a controller has a limit of 96 endpoints (which is typical) then you will no doubt not get 2x SYBA units per controller as it could breach the 96 endpoints.
Again, as I said above, it depends on the capabilities of the controller(s). And all of the above example calculations will be even worse if the drives, when added, are taking 5x endpoints each.
Question. Is there any way you can ‘see’ the SYBA units connected, even if there are no drives in them ? I am not a Linux expert so can’t advise on what specifically to look for. Are the SYBA units hot plugable ? That is, can you insert drives to the SYBA whilst it is powered on ? What I would do is add one drive at at time and see if you can spot the ‘break’ point.
Also, see if you can find detailed specs on the USB side of everything - motherboard, controller cards, SYBA unit and it’s (internal) hub(s) etc.
But it sounds like 1x SYBA unit breaches some endpoint limitation to me, so the only way to get 9x working would be to use 9x controllers, which is effectively what I have on my systems (2x quad controller PCIe cards and 1x onboard controller).
Or - also mentioned above - you could try an older driver to force the SYBA’s into USB2.x mode (eHCI mode). Try to find an older driver from the same manufacturer, it’s possible and likely the card will be hardware compatible with older systems/drivers anyway. This will save some endpoints per controller when you start adding devices as the system will then just think they’re all USB2 rather than USB3.
I’m running 7 of those right now with no issues at all on Windows 11 Pro.
I have 45 drives connected via usb, including 6 x 5 bays.
Using only 2 x 8 or 10 port hubs and the backplane ports, works fine.
Is your USB connection USB3.0?
Hi and thanks again for taking the time to help! The SYBA drives don’t have extra USB ports, but your maths does make sense as I can get at most 2 of the SYBS enclosures to get recognized per Controller. I ordererd 2 of the Startech cards you mentioned in your first reply, so presumably I can attach 1 SYBA unit to each USB port, i.e get 8 enclosures connected? I have a nother 4 enclouses (2 Orico 5bays and 2 Terramaster 5bays), which I’ll also try and connect, but the SYBA drives are my priority.
I wasn’t sure I could go all USB 2.0 as I was afraid about the read/write speeds. All my plots are compressed at maximum level so I can imagine that there might be some more traffic between the full node and the HDD than with uncompressed plots (just assuming).
Either way, I think I’m on the right track and if anything, this has taught me a lot of things I didn’t know about USB… Will keep everyone updated once I get the PCI-E cards!
Cheers and once again thanks for your help, Dan!
You’re welcome, I hope it works out!
Just for info, those Startech cards I mentioned above are also based on the Renesas IC, they appear as Renesas USB 3.0 eXtensible xHCI Compliant Host Controller on my systems.
Yes, worked with my usb 2 hub before it died also
Hey, just wanted to let you know it worked out. Bought two of the startech cards and used one of my old ones and voila… all 72 drives connected, beautifully mounted and running like a charm…
A big thanks from my side, I was close to giving up and selling off the whole USB setup on ebay, until you wrote you had it working… Now I do, too! Cheers!!!
Great news - enjoy!!
I’m a bit late w/more advice, however each of the Startech PEXUSB3S44V USB host controller’s 4 ports can support 2 Syba 8-bay units each. Plug a cheap 4 port USB 3.0 hub on each port, and attach two 8-bays to the lil hub. A 3rd will fail, but still that’s eight per card total or 64 drives. If you have an additional x4 PCI-E slot, I’d imagine you can just do it all over again. With just 2 cards, a single PC can supports 2.56PBs worth of 20tb drives, theoretically. Easy peasy and mine work just fine, never had any issues.
You can save the MB ports for other things, or connect as many more drives as they will support thru USB 3.0 hubs (varies).