Linux is killing bladebit when trying to move plots

Chia Tools & Scripts
1

So I’ve made myself some bash scripts to use bladebit and rclone so I can simultaneously create plots - store them in my NVME - move to HDDs
I’m still testing it so right now I have 5 scripts (will consolidate when I’m confident it will work)
1: makeplots with bladebit to fill up the nvme drive (4 directories within it)
2-5: Using some bash logic I’ll move existing plots and new plots over (1 by 1) to HDDs

To preface, I previously was doing this exact process in windows with Bladebit & RoboCopy on this same machine, but want to move to Ubuntu for admin reasons.

The problem I think I’m having is that scripts 2-5 are using up close to 250GiB of cache/buff space. So when Bladebit starts spinning up it has to reallocate - and the linux OOM Killer (out of memory killer) kills bladebit and one of rclone scripts at random. Unfortunately this takes a long time to happen and I am unsure how to even tell how the bash scripts are dying as the terminal windows are completely killed by the time I see that something has happened.
Surely this has to do with being less experienced with linux but from what I’ve been able to read it sounds a lot like OOM Killer.
I have a 512GB Ram machine and bladebit + scripts 2-5 can run fine on their own, but I’m unable to run them all simultaneously…
Does anyone know of a way I could pre-allocate and reserve free ram for just one of the scripts or prevent the rclone scripts from eating up so much buff space?
Actual RAM and Swap usage of scripts 2-5 and the rest of my system never exceed 6GiB so I should always have enough available.
I’ve seen others clearing buff space before they call bladebit, but from what I’ve read this is ill advised when you have other processes using it actively.
I wanted to see if anyone else had solutions that they were using to move plots in the background while a bladebit script chugs away.

2

In the previous post I mentioned rclone - I tried both rclone and rsync (rsync would be optimal in this case I believe) and have switched back to rsync.

I also just went ahead and tried clearing my buff space - and it seemed to help (runs longer now) but eventually meets the same end.

I think ultimately the issue here is that I can’t reserve free ram for bladebit and it’s dying on me. Has anyone done this / had to do this? and how was it done?

3

Why have 4 different scripts moving plot files and eating up overhead? Is your plotter plotting so fast that you need to move 4 plots at the same time? I think I’d just run one script and let it move one file at a time.

1 Like
4

Nail on the head…

I’m slower than my transfer speed to sata disks - in my initial program on my windows OS I could outpace bladebit if I did 3 or more simultaneous transfers, so with my 4bay SFF-8088 chassis I went for 4

May also be good to note that my transfer speeds start dying when bladebit is running. I’m not quite sure why this is happening as I’m leaving enough ram and threads available to do the transfers.

Also fully aware that I could just take the 20% hit to productivity and move these one at a time in the background but I would like to get this thing running for learning purposes - I’m sure there is a way.

1 Like
5

Gotcha. I never had that problem. My best plot speed is about 29 minutes. :joy: