Off load your destination to nvme for quicker queues

c0rnflake · May 1, 2021, 9:08pm

When I started plotting I was sending all my complete plots to my NAS which took what seemed like forever. It was around 20 minutes.

Anyhow, while plotting is sitting at 100% and moving your completed .plot file to your destaination this is halting the next plot in that queue from starting.

I changed my final destaintation folder to my boot nvme and the final .plot move takes around 3 minutes.

I then use a batch file to move the .plot file to my NAS.

This allows the next plot in the queue to start a lot earlier.

If you’re plotting 10 plots a day, staggered in multiple queue the idle time at 100% starts to add up.

Here’s the batch file i’m using (move_plots.bat):

@echo off
:start
move /Y "C:\Temp Plot\*.plot" "\\192.168.88.200\Plots\"
PING localhost -n 900 >NUL
goto start

And here’s the vbs file i’m using to hide the cmd.exe from the taskbar in Windows 10.

Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & "C:\Temp Plot\move_plots.bat" & Chr(34), 0
Set WshShell = Nothing

JustinLloyd · May 1, 2021, 9:45pm

github.com

JustinLloyd/chia-utilities/blob/main/migrate_plots_to_nas.ps1

#Requires -Version 7.0

# Description: This file copies plots from the plotter to the NAS
$Config=Import-PowershellDataFile -Path .\config.psd1 -ErrorAction Stop
$IntermediatePath = $Config.IntermediatePath
$HoldingPaths = $Config.HoldingPaths

$PlotFilesToMove = Get-ChildItem -Path $HoldingPaths -Filter '*.plot' -ErrorAction Ignore
# TODO only permit a single instance of this script to run
echo $PlotFilesToMove
$FileIndex = 0

echo "Found $($PlotFilesToMove.Count) plot files that need to be migrated."
foreach ($PlotFile in $PlotFilesToMove)
{
    $FileIndex = $FileIndex + 1
    if (Test-Path -Path (Join-Path $IntermediatePath $PlotFile.Name) -PathType Leaf)
    {
        $msg = 'Skipping file (' + $FileIndex + ' of ' + $PlotFilesToMove.Count + ') - "' + $PlotFile.Name + '" already exists on "' + $IntermediatePath + '"' 
        echo $msg

This file has been truncated. show original

Configuration for the PS script is in Config.psd1. There is also a companion .vbs script on that repo. Run the .vbs from Task Scheduler a couple of times per day.

codinghorror · May 2, 2021, 1:39am

Isn’t this sort of what the temp and temp2 paths are for in the chia plot command? I almost always use the same path for both, but…

-t [tmp dir]: Define the temporary directory for plot creation. This is where Plotting Phase 1 (Forward Propagation) and Phase 2 (Backpropagation) both occur. The -t dir requires the largest working space: normally about 4 times the size of the final plot.

-2 [tmp dir 2]: Define a secondary temporary directory for plot creation. This is where Plotting Phase 3 (Compression) and Phase 4 (Checkpoints) occur. Depending on your OS, -2 might default to either -t or -d. Therefore, if either -t or -d are running low on space, it’s recommended to set -2 manually. The -2 dir requires an equal amount of working space as the final size of the plot.

When you use two different temp drives some of the phases will transfer the temp files from temp 1 to temp 2. When you use 1 temp drive, that single drive will have to read and write during phase 3 and 4 (I think those are the correct phases). With 2 temp drives you get quicker times since one drive can read, the other can write. Plus I think it reduces the total space needed for each temp drive, useful for parallel plots.

But yes in general, you have to factor “time to write the final plot to disk” into your calculations and prevent logjams…

69chargr · May 3, 2021, 4:42pm

So question on this…

I am going to be setting up a 2nd NVMe drive to use as my 2nd temp drive as well as final drive which should speed up my queue. Then I’d like to use your move_plots.bat to see the *.plot file and move it to my USB drive.

Do I need to set it to run periodically through task manager or does it continually check for *.plot as long as it is running?

I have zero experience running a bat to do this kind of work so any help is appreciated.

c0rnflake · May 3, 2021, 4:57pm

You simply need to run the vbs file. The batch file includes a PING localhost -n 900 >NUL, the 900 in this PING is around 15 minutes and the batch loops again. So basicall the batch file will loop every 15 minutes automatically.

c0rnflake · May 3, 2021, 11:29pm

I’ve updated the move_plots.bat

I noticed that plots were being caught as invalid/couldn’t be opened while they were being moved. They aren’t but the plots weren’t being removed from that list either once they were successfully moved and added to the farming.

Anyhow, below is an updated batch script that will rename all the plots, move them and then rename them back once completely moved and should stop them from being picked up as invalid.

@echo off
:start
for /r "C:\Temp Plot\" %%G in (*.plot) do ren "%%~G" *.plottmp
move /Y "C:\Temp Plot\*.plottmp" "\\192.168.88.200\Plots\"
for /r "\\192.168.88.200\Plots\" %%G in (*.plottmp) do ren "%%~G" *.plot
PING localhost -n 900 >NUL
goto start

69chargr · May 3, 2021, 11:56pm

Ok…

So if I want to move from my 2nd temp drive (Y:) to the final USB drive (E:\ChiaPlot) this should work?

@echo off
:start
for /r "Y:\" %%G in (*.plot) do ren "%%~G" *.plottmp
move /Y "Y:\*.plottmp" "E:\ChiaPlot\"
for /r "E:\ChiaPlot\" %%G in (*.plottmp) do ren "%%~G" *.plot
PING localhost -n 900 >NUL
goto start

Thanks for this!

69chargr · May 5, 2021, 3:56am

Just a follow up…

Works like a charm! I did bump down the PING to 90. I also set my plots to ignore the final directory so I don’t confuse the harvester. Picks up the plot within a minute or so after being moved. This helps a lot because I don’t have a whole lot of space on my second temp drive and I need to move the completed plots off right away.

Thank you for this tool!

69chargr · June 2, 2021, 2:10pm

Now how do I do this in Linux?

#!/bin/bash
for i in ~/[MYTEMP]/*.plot
do
mv “$i” ~/[MYFINAL]
done

Save the code to a text file called moveplot.sh

Open Terminal and do: chmod +x ~/Desktop/moveplot.sh

To run it in Terminal: sh ~/Desktop/moveplot.sh

That would do the move but not the rename. I wonder if the farmer gets confused the same way in Linux without the rename?

c0rnflake · June 2, 2021, 4:32pm

I’m guessing something similar to this. I’d also run the script through cron rather than manually.

#!/bin/bash
for i in “/[MYTEMP]/*.plot”
do
mv “$i” “/[MYFINAL]/${i/%.plot/.plottmp}”
done
for i in “/[MYFINAL]/*.plottmp”
do
mv “$i” “/[MYFINAL]/${i/%.plottmp/.plot}”
done

Yae · June 2, 2021, 4:38pm

I’m not sure the boot drive it the best idea for this. It probably has some nice things (for a boot drive) running there such as journaling, indexing, AV etc which you most certainly do not want on a staging drive.

69chargr · June 2, 2021, 6:29pm

Thank you! Will try it! I’m hoping over the course of a day it gives a few more plots when running Swar’s.

alphatio · September 9, 2021, 7:03am

what is the difference between

PING localhost -n 900 >NUL

and

timeout /t 900

I cannot find a way to display output using move so I change to robocopy
Still wait for testing this edited script below

@echo off
:loop
for /r "**source**" %%G in (*.plot) do ren "%%~G" *.plottmp
robocopy **source** **destiny** /mov *.plottmp /A-:SH
for /r "**destiny**" %%G in (*.plottmp) do ren "%%~G" *.plot
timeout /t 60
goto loop