JBOD Setup Mistake

General Chia Discussion
1

Well, my JBOD noob level has presented itself. I have extensive experience in all sorts of RAID in very large enterprise environments. But in that field, you just don’t play with JBOD. I understood the concept, but never implemented it. So when I setup my first 2 NAS systems with it, I setup one JBOD storage pool with all the drives in it and then one volume spanning all drives in that pool. I assumed that the JBOD setting meant the system would see they are separate drives and treat them that way. NOPE. I had this nagging feeling about the setup (screaming SOMETHING IS WRONG). Fortunately I had a 3rd NAS that is not in service yet. So, I tested it. Set it up the same way, dropped some files on the share and pulled a drive. CRASH. Data gone. Well, that wasn’t what I was expecting but what I feared. So, I have setup that NAS (has 8 drives) with 8 separate storage pools, each pool has a volume and each volume has a share. Now I can map to each share and know that it is an actual physical drive. Now if a drive dies, I lose that drive, nothing else. The issue now is that I have to move all of my plots from the misconfigured devices to this new one so I can rebuild those.

Is my new approach correct? Or did I just do something wrong in the original configuration that would have made that setup work properly? Hopefully what I am doing now is right and this is just a warning for everyone to do it right.

2 Likes
2

Yeah, unfortunately JBOD means “present one giant volume” :frowning:

I learned this early on when messing with the terra-master NAS’es. You can kinda approximate this setup if you do one storage pool per drive (as you described), but the naive JBOD is “one giant volume, if anything happens to any drive in the volume, it all goes poof”.

Not a super useful setup, but then, Chia is kind of an unusual setup itself…

3

i think what you are looking for is a Union of independent file systems,
have a look at unionfs/mergerfs/aufs etc

image
image846×700 22.7 KB

if one drive dies, only the files which been on that drive are gone, all drives are accessible through one folder, u can set policies so that the next file always lands on that one with most free space, or have them fill up on at a time, its ur choice

2 Likes
4

But here is the confusion. Everyone on here was preaching to use JBOD saying that if you lose a drive, you will only lose the data on that drive. Then just replot the drive. But that isn’t the case if you set it up as one volume. If you lose a drive, you lose the entire volume. Which is stupid. I would never put over 100 TB on a setup that would completely crash if one drive died. That is a month of plotting. But, since it is a good idea to separate your plots into folders anyway (for harvesting speed and just organization), I think the way I just setup makes sense. One drive = one folder.

5

Jbod just means that they appear seperately for the operating system, and are not controlled by the raid controller infront of the system which would hide the fact that these are xyz amount of drives

If u make a single volume spanning across all drives(or do a software raid) ofc you get that issue :sweat_smile:
Simple volumes are not that smart

6

Having a single large volume, with multiple points of failure is silly. Multiple drives, multiple volumes is fine.

You might say, hey, I have an airplane with 2 engines, so clearly it is more reliable than a 1 engine plane.

Wrong. It’s less reliable. Now you have two points of failure. The airplane can probably still land on 1 engine, but the chance overall of failure increases.

And in the case of 1 large volume, when 1 drive goes, it all goes. There’s no “landing” at that point.

7

Unless u span the volumes in a raid and loose some of the size for redundancy so that 1 or more drives can fail without loosing any data(depending on the config), and it’s still just one volume

8

This is on a Synology NAS. You grab all the drives and classify them as a JBOD storage pool. But then you have to create a volume for that pool. If you put all 8 in one pool you can only create one volume for that pool. That is where the problem shows up.

But if you create 8 individual pools, you can create a volume for each and then you get what most people think of when you say JBOD.

I don’t understand why you can setup an 8 drive “JBOD” storage pool when it does not treat them as individual drives. It is basically doing RAID 0. Which is not what JBOD is.

9

No reason to have redundancy. It’s just lost income. If a drive fails later on, fine, you still had the income in the short term.

10

I’m using this for my plot destination drives

11

Ok, first off I have to congratulate you on having the forethought to TEST the NAS by pulling a drive before filling it full of plots - I wouldn’t have done that :slight_smile:

Second, “JBOD” is a generic term that can mean different things to different people. It’s not IEEE defined anywhere.

So in Synology’s interface, it sounds like they use it to mean “stupid, unmanagaged storage”

The first red flag was when you selected multiple drives to add to a single volume - I’ll bet dollars to donuts it just created a giant RAID 0 array for you.

Third, there are “JBOD” style setups that simply expose all the drives to the host machine as if they were attached directly to SATA ports - this is done with an “HBA Card” as opposed to a “RAID Card” - I blogged in painful detail about setting mine up here:
https://rkalla.me/linux-ubuntu-asus-z590-lsi-9201-16e-hba-card-and-how-the-hell-you-get-it-all-working-together/

This passthrough is so transparent that it’s actually sorta hard to tell which drives are connected to the local host machine and which are in the external enclosure.

You can get a 24 drive enclosure for a “JBOD” setup for like $500 because all it has is a PSU to power the drives and then just simple connectors to the HBA card.

On the other hand if you buy a 24 drive enclosure for a RAID setup, it’s probably more like $1500+ because that RAID card on there is a significant piece of technology.

Because of how consumer oriented Synology is/has been, I’m GUESSING they are trying to abstract out this painful nuance for anyone using their devices and just trying to give you what they assume you want: simple/big storage.

2 Likes
12

You can also get a JBOD by presenting each disk separate as RAID 0 and create a volume per each disk.
If a disk dies, it only affects that one volume.

13

The chance of experiencing an engine failure. The chance of the aircraft failing to stay in the air increases. So its more reliable if your point of reference is don’t kill me!

14

My understanding of “JBOD” is that it stands for “Just a Bunch Of Disks”.
And in a RAID setup, all of the JBOD drives show up as a single drive to the OS.

With a RAID 0, that, too, shows up to the OS as a single drive. But it differs, significantly, from a JBOD RAID.

With JBOD, you can mix and jumble together any and all drives, and they will act as one drive (sort of). When you copy files to the JBOD array, the disks will fill up one-by-one. But if one drive fails, you lose the entire array. Perhaps there is recovery software that can recover files on the non failed drives, as long as the files did not cross drives (as in, the file starts at the end of one drive, and ends at the start of the next drive).

With RAID 0, when you copy files to the drives, the files get split up evenly (mostly, depending on the size of the files). So that 100GB plot file will get distributed in equal size chunks to all of the drives in the RAID.

The benefit is that if you have, for example, 10 drives in the array, then all 10 are writing the 100GB file at the same time, resulting in it taking the time for a 10GB file to be written (because 10GB is being written to each of the 10 drives, simultaneously).

But just as with JBOD, if one drive of a RAID 0 fails, you are toast. And recovery will be nearly impossible, because your plots will never be on a single drive. They will all be split evenly across all of the drives.

RAID 5 is the same as RAID 0, except that one of the drives acts as a parity drive. This gives the RAID fault tolerance. You can have 1 drive failure (any one drive in the RAID 5 can fail), and you will keep humming along. The RAID 5 will effectively become a RAID 0. If another drive fails, you are toast.

So you go out and purchase a replacement for the failed drive. You install it, and you tell your RAID controller to rebuild the RAID. It will do this in the background, and you can use your computer while it does its magic (however, heavy usage will delay the completion of the recovery).

Upon completion of the recovery of rebuilding the RAID, you will return to having a RAID 5, and will once again be fault tolerant.

The downside of a RAID 5 is:

  1. You do not get the full storage of all of the drives. If you have, for example, 5 (1TB) drives in your RAID 5 array, you will see only 4TB of storage.
  2. It will be slightly slower than a RAID 0. But for Chia, that is a non issue.

For Chia, never use JBOD or RAID 0. One disk failure takes down your world.

The only reason I can see for JBOD is if you have a weird assortment of different disks that will not get along in a RAID 5 (for a RAID 5, it is best to use identical drives). JBOD will probably not care what mix of drives you use, because it simply stacks their capacity and uses no fancy calculations to keep them synced with each other.

Cheers!

15

The term JBOD just defines a “collection of just a bunch of disks”. Not more, not less.

Does it matter If have a drive failure and loose lik 80x plots? for me it does not. I just replot.
If each disk is presented as its own volume/virtual disk, you just loose that disk on a drive failure.

Thats why I go for each physical disk as its own volume. I prefer massive amounts of storage than parity/redunancy in this specific use-case.

16

JBOD is dangerous, and not really needed. You can declare 1 storage pool & volume per physical disk and add them all on the plots directories so farm will find them all.

With such Synology NAS setup, i run CHIA headless as a docker container only to farm plots on the NAS.
Gui is installed on my Windows host so i can see what’s going on with the farm.

Each disk is a separate storage pool + volume.
Docker config file is set as follows, example is with 2 volumes, but can be extended to infinite.

docker run -d --name CHIA
–net host
-v /volume1/docker/chia/root:/root/.chia
-v /volume1/docker/chia/chia.key:/root/chia.key
-v /volume1/docker/chia/python_keyring/:/root/.local/share/python_keyring/
-v /volume3/PLOTS_1:/plots
-v /volume2/PLOTS_2:/plots2 \

remark: 3 lines below not needed when - net host is used, but i let it just for info

-p 8444:8444 \
-p 8555:8555
-p 55400:55400 \

-e keys="/root/chia.key"
-e TZ=“Europe/Zurich”

TZ variable not handled though for the time being…

On the GUI, i setup plots directories as follow and voila, all fine.

2021-05-27_114736

I was planning to write a full tutorial for such setup (Win 10 Gui only, Docker headless on syno, + separate mini PC ubuntu plotting & dropping plots directly to NAS. but did not have time to do it yet.

1 Like
17

This only works if you are running a plus model of Synology (like a DS1817+). The regular models do not support any VM environment, including docker. But I have found an earlier version of the DSM software that apparently did allow a workaround, I’m going to test that once my plots are moved.

18

yes and no. Docker can be installed on non + models, also with non intel CPU, but this will require workarounds.

Edit: VM is not docker. VM can be created / used with virtual machine manager (and such package can be installed manually when not available officially :slight_smile: )

Docker is a different package, this is containerization , not virtual machines. What is your model ?

image
image1101×1065 134 KB

19

I use mergerfs for the output directory - as it say, lose 1 disk = lose just plots from that disk - but ALSO it will not split a plot so you end up with residual at the end of each disk. Mine are small so i can fit 27 plots + just over 50GB left - if i join 2 volumes together i can get 55 (1 extra plot) + very little waste

20

That setting up the HBA card blog was a great read!

1 Like