Disclaimer - I haven’t written Bash in 15 years, this probably needs some work so feedback is appreciated.
What is this?
If you are plotting on Linux (Ubuntu 20.04 Server in my case) and have a setup that is similar to mine, this script will allow you to run your plotter at full speed and not worry about it filling up the finaldir holding directory OR bring your system to it’s IO-knees by trying to move multiple plots to the same spinning disk at the same time.
Environment
- More or less a rack setup.
- Linux / bladebit / madmax
- 1x plotter/harvester in the rack (Poweredge 820 in my case)
- Plotting at a rate that is faster than a single disk can accept the plots, SO you are plotting to a temporary space like an SSD or RAID-0 setup.
- Mount the disks 1 at a time under
/mnt/[UUID]naming scheme. - Multiple JBOD/disk shelves in the rack that provide many different potential targets for files to get written to simultaneously without saturating the HBA bus.
Benefits
If you try and do this by hand (like I was) or with a simpler script, you’ll notice a few things that become a show stopper pretty quickly so this script will do the following for you:
- Move up to N
.plotfiles at a time BUT ONLY to separate drives - your limiting factor here will be your HBA bandwidth limitation, JBOD bandwidth limitation or backplane bandwidth limitation if you are using a RAID setup on them. - Never move more than 1 file
.plotfile to the same spinning drive at a time. - Don’t let the harvester (Chia or flexfarmer) ‘see’ the plot until it’s done moving, to avoid them detecting an incomplete file and marking it invalid. This allows you to keep the harvester running and not needing it to be restarted over time to pickup the changes - it will pickup the changes naturally as each plot is moved.
- Don’t move a
.plotto a target drive once it’s filled up and doesn’t have room.
Disclaimer #2 - This is NOT a drop-in to any system, I have hard coded paths in here for /mnt locations where I find my disks. You will ALMOST CERTAINLY need to change aspects of this script. But at least it’s a starting place and I hope it helps someone.
Setup
I run this as a cron job that executes every 5 mins - this ensures that plots don’t back up AND if multiple .plot files are found, they can be moved (to different targets) simultaneously.
Here’s the crontab -e entry for script:
*/5 * * * * /home/myuser/chia/move-completed-plots.sh
Bash Script
#!/bin/bash
DRIVES=(
"d32083f3-926f-482b-9d8c-9376ef99ba3e"
"a07f239e-378d-45e9-8d05-da62fc613137"
"dcb82701-f970-4a95-9ffa-b5e30672d1d4")
cd /mnt/chia-tmp/
for file in *.plot
do
echo "Found plot [$file], processing..."
# Determine destination for plot
idx=$(($RANDOM % ${#DRIVES[@]}))
drive=${DRIVES[$idx]}
dest_dir="/mnt/$drive"
wip=false
echo -e "\tSelected destination [$dest_dir], checking for other writes in progress..."
# Scan all files in the directory and make sure no others are being moved to the same drive
for checkFile in *
do
if [[ $checkFile == *"$drive" ]]
then
# Found a file with the suffix equal to the drive we are trying to target so there is already a write heading to that drive
wip=true
fi
done
if $wip
then
# Skip trying to move this file to the target drive, it already has a move in progress
echo -e "\t\tSKIPPING, will retry later, write in progress found for this block device."
continue
fi
# Check free disk space on target drive
free_space=`df --output=avail -B 1 "$dest_dir" | tail -n1`
if [[ $free_space -lt 115000000000 ]]; then
echo -e "\t\tSKIPPING, not enough disk space on target block device [$dest_dir]"
continue
fi
# FILE IS OK TO MOVE
tmp_file="$file".MOVING_TO-$drive
echo -e "\tCONTINUING, temporarily changing filename for move [$tmp_file]..."
# First, change the suffix so the file is no longer picked up by the script
mv "$file" "$tmp_file"
# Second, perform the move operation
mv "$tmp_file" "$dest_dir"
# Third, once the move is over, rename the file back to its original name.
mv "$dest_dir/$tmp_file" "$dest_dir/$file"
echo -e "\tCOMPLETE, file moved to [$dest_dir/$file]"
done
Tips
-
DRIVES- the list of UUIDs I mount my drives under/mnt- I think this can be optimized out by just addressing the drives directly under/dev/drives/by-uuid/[UUID]- I just haven’t made that change yet. -
/mnt/chia-tmpis my NVME-based RAID-0 setup where I plot to. - Plot byte size of
115000000000is my arbitrary “round up” of the 108xxxxxxx byte size that XFS was listing my existing plots at. I added some slop in there. - I execute this as a cron job so many parallel moves can be executed at the same time - if you wrap this entire script in a
while true, you could just run it from the command line BUT then all moves will be sequential instead of parallel.
