Bash script to help move completed plots out of finaldir and onto N target mount points

Disclaimer - I haven’t written Bash in 15 years, this probably needs some work so feedback is appreciated.

What is this?
If you are plotting on Linux (Ubuntu 20.04 Server in my case) and have a setup that is similar to mine, this script will allow you to run your plotter at full speed and not worry about it filling up the finaldir holding directory OR bring your system to it’s IO-knees by trying to move multiple plots to the same spinning disk at the same time.

Environment

  • More or less a rack setup.
  • Linux / bladebit / madmax
  • 1x plotter/harvester in the rack (Poweredge 820 in my case)
  • Plotting at a rate that is faster than a single disk can accept the plots, SO you are plotting to a temporary space like an SSD or RAID-0 setup.
  • Mount the disks 1 at a time under /mnt/[UUID] naming scheme.
  • Multiple JBOD/disk shelves in the rack that provide many different potential targets for files to get written to simultaneously without saturating the HBA bus.

Benefits
If you try and do this by hand (like I was) or with a simpler script, you’ll notice a few things that become a show stopper pretty quickly so this script will do the following for you:

  • Move up to N .plot files at a time BUT ONLY to separate drives - your limiting factor here will be your HBA bandwidth limitation, JBOD bandwidth limitation or backplane bandwidth limitation if you are using a RAID setup on them.
  • Never move more than 1 file .plot file to the same spinning drive at a time.
  • Don’t let the harvester (Chia or flexfarmer) ‘see’ the plot until it’s done moving, to avoid them detecting an incomplete file and marking it invalid. This allows you to keep the harvester running and not needing it to be restarted over time to pickup the changes - it will pickup the changes naturally as each plot is moved.
  • Don’t move a .plot to a target drive once it’s filled up and doesn’t have room.

Disclaimer #2 - This is NOT a drop-in to any system, I have hard coded paths in here for /mnt locations where I find my disks. You will ALMOST CERTAINLY need to change aspects of this script. But at least it’s a starting place and I hope it helps someone.

Setup
I run this as a cron job that executes every 5 mins - this ensures that plots don’t back up AND if multiple .plot files are found, they can be moved (to different targets) simultaneously.

Here’s the crontab -e entry for script:

*/5 *  *   *   *     /home/myuser/chia/move-completed-plots.sh

Bash Script

#!/bin/bash

DRIVES=(
        "d32083f3-926f-482b-9d8c-9376ef99ba3e"
        "a07f239e-378d-45e9-8d05-da62fc613137"
        "dcb82701-f970-4a95-9ffa-b5e30672d1d4")

cd /mnt/chia-tmp/

for file in *.plot
do
        echo "Found plot [$file], processing..."

        # Determine destination for plot
        idx=$(($RANDOM % ${#DRIVES[@]}))
        drive=${DRIVES[$idx]}
        dest_dir="/mnt/$drive"
        wip=false

        echo -e "\tSelected destination [$dest_dir], checking for other writes in progress..."

        # Scan all files in the directory and make sure no others are being moved to the same drive
        for checkFile in *
        do
                if [[ $checkFile == *"$drive" ]]
                then
                        # Found a file with the suffix equal to the drive we are trying to target so there is already a write heading to that drive
                        wip=true
                fi
        done

        if $wip
        then
                # Skip trying to move this file to the target drive, it already has a move in progress
                echo -e "\t\tSKIPPING, will retry later, write in progress found for this block device."
                continue
        fi

        # Check free disk space on target drive
        free_space=`df --output=avail -B 1 "$dest_dir" | tail -n1`
        if [[ $free_space -lt 115000000000 ]]; then
                echo -e "\t\tSKIPPING, not enough disk space on target block device [$dest_dir]"
                continue
        fi

        # FILE IS OK TO MOVE
        tmp_file="$file".MOVING_TO-$drive
        echo -e "\tCONTINUING, temporarily changing filename for move [$tmp_file]..."

        # First, change the suffix so the file is no longer picked up by the script
        mv "$file" "$tmp_file"

        # Second, perform the move operation
        mv "$tmp_file" "$dest_dir"

        # Third, once the move is over, rename the file back to its original name.
        mv "$dest_dir/$tmp_file" "$dest_dir/$file"

        echo -e "\tCOMPLETE, file moved to [$dest_dir/$file]"
done

Tips

  • DRIVES - the list of UUIDs I mount my drives under /mnt - I think this can be optimized out by just addressing the drives directly under /dev/drives/by-uuid/[UUID] - I just haven’t made that change yet.
  • /mnt/chia-tmp is my NVME-based RAID-0 setup where I plot to.
  • Plot byte size of 115000000000 is my arbitrary “round up” of the 108xxxxxxx byte size that XFS was listing my existing plots at. I added some slop in there.
  • I execute this as a cron job so many parallel moves can be executed at the same time - if you wrap this entire script in a while true, you could just run it from the command line BUT then all moves will be sequential instead of parallel.
2 Likes

I have a bit different script - I’m using Mad Max as a plotter with option - dest dir.
So my script checks free space on dest dir. Than starts madmax with specific number of plots.
Than continue with next destination.

Here is the sample:

PLOTSIZE=108900000000
DIRSIZE=$(df --output=avail -B 1 "/chia/dest" |tail -n 1)
USABLESIZE=$(( (DIRSIZE / PLOTSIZE) ))
sudo /home/chpt/chia-plotter/build/chia_plot -n $USABLESIZE  -r 8 -t /chia/tmpdir/ -2 /chia/tmpdir2/ -d /chia/dest/ -f XXXXXXXXXXXXXXXXXXXXXX
1 Like

If my target drives were fast enough to pickup plots quickly, I would totally use this - unfortunately all my destinations are SATA drives and writing a plot file out can take 10mins - I don’t want to block the plotter (bladebit) and want it to just work as fast as it can because it pegs every core @ 100% of my 80-core Poweredge almost the entire time it’s running.

Thanks for posting your script though so we can learn some bash tricks from each other! :slight_smile:

I would still look at cron as just a different way of using sleep command, and starting mv in a slightly different way (through multiple instances, what can be done in the main script by splitting it into two parts and pushing that mv process to background).

First, I would modify your mv to be three stage process (e.g., move-it.sh):

  1. mv your plot to a parallel folder to tmp (thus your tmp plot folder is clean)
    1.1. from /mnt/bbit/tmp
    1.2. to /mnt/bbit/moving-dstX
  2. mv your plot from that parallel plot to a parallel plot folder in the destination drive (thus, your harvester doesn’t see this plot)
    2.1.from /mnt/bbit/moving-dstX
    2.2. to /mnt/dstX/moving
  3. move your plot to the final destination
    3.1. from /mnt/dstX/moving
    3.2 to /mnt/dstX/plots

Of course, you could also rename that file to something like *plot.rkalla-dstX. Although, as indicated above, I would make as many of those bbit moving folders, as you have destinations. This way, you could easily check whether a file is being moved to a given dst folder, and select the next folder in your script.

What you gain here is that the minute the new move-it.sh will pick up that plot (in #1), your tmp folder doesn’t need to be checked for plots that are being moved. Also, until the last mv (#3) finishes, your harvester doesn’t see that plot. (Both operations (#1 and #2) are instantaneous, as that is just file handle change.

You should start that move-it.sh to run in the background, i.e., it will not block your main script. Once you call that script, you can do your standard sleep command, and loop around. This would be more or less equivalent to running multiple instances of your current script from cron. My take is that it would be a cleaner solution, as your single instance could do better to select which drive is the next, or how to fill your drives (e.g., just two at a time, so you can swap those, while script is pushing to the next two, etc.).