how do you do that sometimes to move the finished plots?
Earlier I had written a small script myself, which moves the one finished plot and then deletes everything it ends on .plot.
THIS is no longer possible, because now the creation is faster than the copy.
I would have to rewrite my script now and delete specifically the one plot. But that is also kind of stupid.
Probably there is already something ready(?)
Hint: I don’t copy via LAN but distribute on e.g. 3 HDD’s.
Linux Ubuntu
#! /usr/bin/bash
while : ## Starte Endlosschleife
do
if test -e /mnt/NVME/*.plot #Prüfen, ob fertige *plot-Datein vorhanden sind
then
echo "=====================================================================================================";
echo "========================== START: $(date) ======================================"
echo "=====================================================================================================";
duf
rsync --progress /mnt/NVME/*.plot /mnt/usb-hdd3/
rm /mnt/NVME/*.plot
duf
date
echo "=====================================================================================================";
echo "========================== ENDE: $(date) ========================================"
echo "=====================================================================================================";
echo /n
echo /n
fi
sleep 5 ## Warte 5 sekunden und prüfe erneut ob Datei da ist
done
My Linux skills are minimal, but I think I have a solution – if my syntax is correct.
After “if test -e …”, the next line might be the solution you are seeking:
plotfile=$(ls -1 *plot | head -1)
The above should put the first instance of a file ending in plot into a variable named plotfile.
You can then reference the above variable for your rsync command and your rm command.
I suggest that you test the exit code from “rsync”, to ensure that the file copy was successful.
It would be frustrating if the file copy failed, and yet the script went on to delete the plot.
If your using Linux there’s an app called Plow (which I’ve never used), depending on how fast your plotting 3 drives may be quick enough.
I wrote my own program for Windows, which deletes a plot if there is not enough space and then moves the plot, as this was running across 10G network it all worked well to multiple drives.
My 2 cents: When copying a plot into a live farm, *.plot > *.temp.
After copy op has completed, rename the target back to *.plot.
My rationale: (May or may not be real risks if the OS locks the target during the copy)
Reason being: for ten or so minutes you avoid having a plot file in your farm directory that is partially copied. This avoids “invalid plot” designation for which the only recourse is to delete the plot. Also, there used to be a risk of database corruption which I attributed to partially copied plots confounding the database read/write routines.
logfile=“/var/log/plot_move.log” # Set to a valid file path to enable logging
rsync_options=“–preallocate --remove-source-files --skip-compress plot --progress”
if [ “$#” -ne 2 ]; then
echo -e “$(timestamp)${RED}Error: Invalid number of arguments.${RESET}”
echo -e “$(timestamp)${RED}Usage: $0 <watch_dir> <dest_dir>${RESET}”
exit 1
fi
echo -e “$(timestamp)${CYAN}Monitoring directory: ${YELLOW}$watch_dir${RESET}”
echo -e “$(timestamp)${CYAN}Files will be transferred to: ${YELLOW}$dest_dir${RESET}”
_move_plot(){
local plot
plot=“$1”
echo -e “$(timestamp)${GREEN}New plot detected: ${MAGENTA}$plot${RESET}”
echo -e “$(timestamp)${GREEN}Initiating transfer to: ${YELLOW}$dest_dir${RESET}”
if [ -n "$logfile" ]; then
rsync $rsync_options --log-file="$logfile" "$plot" "$dest_dir"/
else
rsync $rsync_options "$plot" "$dest_dir"/
fi
echo -e "$(timestamp)${GREEN}Transfer completed.${RESET}"
}
cleanup() {
echo -e “$(timestamp)${RED}Terminating script due to SIGINT…${RESET}”
kill – -$$
}
trap cleanup SIGINT
loop(){
local filename
while true; do
filename=$(ls -tr $watch_dir/*.plot 2> /dev/null | head -1)
if [[ $filename != *.plot ]]; then
sleep 5
continue
fi
_move_plot $filename
done
}
loop
@lord_icon might want to consider copying the $plotfile to a sub-directory, or some other directory on the same partition to the harvesting directory. That way, Chia will not see it (config.yaml will not be looking there).
Upon a successful rsync copy, his script could issue “mv” on the $plotfile and it will put it in place in a split second.
Can’t say I’ve ever had DB corruption through copying plots, DB isn’t really related to the plots in that way.
Also if you get an invalid plot due to it currently being copied, it will self correct next time it scans I believe, or simply restart Chia, absolutely no need to delete the plot.
My script renames them .mov01, .mov02 etc as I have multiple simultaneous moves to different drives, but previously I used to just copy them to the drives as .plot and not worry about the invalid warnings - when a drive was full I’d always run a plot check to find any invalid plots.
Unless you have recursive folder scanning on, then it may well scan the folder.
Why would I want to use Rsync on Windows? I just use my own VB.Net program to move the plots, all done automatically in a nice user interface, and it deletes old plots to make space for compressed plots.
If you’re on Linux then using Plow might be a good idea, I believe that does the same.
Simple really, I know what I’m doing with Windows, used it for getting on for 30 years. My server runs Windows, it just works better together.
When I replotted to NFT plots I did use Linux, as it was significantly faster than Windows, so I made the effort and did manage, but it was still quite involved having never used Linux before, and I still had issues. I also needed to put the drives in the plotter, and just fill the empty drives - I couldn’t get the Linux/Windows networking to work quite right.
With GPU plotting my workstation can run two GPU’s, in Windows my 3080 produces a plot every 5 minutes, my P4 every 10 minutes, for the size of my farm generating 32TB of plots a day is plenty quick enough. My own software deleted plots to make space and moved plots to multiple drives across 10G networking, so it was really set and forget, just leave it churning away.
I did try and get Linux cuda plotting, but just couldn’t get the driver to install, only reason I tried was because I was curious as to how much faster it would be.
I did try Gigahorse plots on Windows and that was just over 3 minutes IIRC.
The really simple answer is, its what works for me.
I don’t have the time, or the memory retention these days, and I have absolutely no need to, so let’s just leave it there and perhaps you’ll accept not everyone wants to or needs to use Linux!
Last night I built a Dell 720xd with 1080 GPU. Dump drive is a 12x4TB=48TB machanical HDD RAID 0. It did 5.5 minute a plot. Write to dump drive was fast. Plan to hook up a SAS external array to transfer plots. I know nothing about script so I will have move plot manually.
Looks like linux is faster, considering 1080 GPU is a quite old one.
If I find some spare time I may just see if I can get Linux working, it’s occurred to me that I can run my plot mover on my Windows Server, as that could see shared drives on the Linux box.
Last time I could not get Linux to see/access the drives on Windows, but it worked the other way around.
