Any solution on windows to erase old plots progressively?

pj_tham · June 7, 2023, 6:31am

Hi,

I have a list of “old uncompressed plots folders” corresponding to a list of drives where I would like to automatically delete the oldest files to keep a certain amount free, as to keep chia_plot_sink putting new compressed plots on it…

My current solution is not good.
If anyone has a solution out of the box, I would be thankful.
I asked for an evolution of chia-plot-sink on its github, and I hope @madMAx43v3r will find it useful to ease me plotting

Ronski · June 7, 2023, 9:13am

I’d be interested in something for Windows, currently trying to write something myself, but that’s slow going.

I’m currently using the Chia Alpha, not sure if plot sink will work with that or not.

dinosauce · June 7, 2023, 12:14pm

What is your current solution?

Ronski · June 7, 2023, 12:30pm

Me or the OP?

Currently I just filling my temporary drive, then use Tera Copy to move them simultaneously to three different drives. Just want to check everything works and get plot transfer sorted out.

It’s writing my own app that’s taking time. I really want C9 plots, which Bladebit doesn’t do yet anyway, so at least I’m in no major rush.

dinosauce · June 7, 2023, 12:41pm

Actually the OP’s but thanks for sharing yours as well.

What is the entire workflow from plotting to temp drive to plot directory? I’m guessing you and the OP are wanting to also slowly replace the existing non compressed plots? Something like, before copying a compressed plot to an existing plot directory, delete an appropriate number of non compressed plots so the new one will fit?

drhicom · June 7, 2023, 1:45pm

To make this as easy as possible, when I had to replot, I took the drive from my farmer formatted the disk and installed on my plotter connected power and sata cable.
I ran…
,
In one CMD window run below

You can expand this to more drives.

chia_plot_sink – H:\mmxk32c\ I:\mmxk32c\

In another CMD windows run below to make MMX K32C7 plots

cuda_plot_k32rev2d -n 448 -x 11337 -M 128 -S 4 -C 7 -t f:\nvme\ -d @localhost -p pppppp -f ffffff
,
,
Worked great you adjust for how many drives connected on plotter. I fill 4 18tb disks in 2 1/2 days.

Ronski · June 7, 2023, 4:46pm

That works with GigaHorse plotter, but I doubt it works with Chia Bladebit compressed plots.

Ronski · June 7, 2023, 4:54pm

Workflow.

Generate plots to fast temporary storage.

In my case as I have 10G networking have up to 5 destination drives.
Monitor temporary storage for completed plot files.
If there is a free destination drive (no ongoing copy) rename the plot file to .MoveX where X is the destination number. Check if there is enough space on drive X, if not delete an old plot and start the file move going, if its full remove it from the list, once the move completes rename it back to a plot file.
You’d cycle through the drives starting parallel copies to each drive as plots became available, checking and making space on each drive, until they were all full.

Ultimately I’d like my program to calculate how many plots were required, perhaps even the correct ratio of plots sizes to fully fill the drive.

vasiliinorris · June 7, 2023, 6:10pm

Ask ChatGPT.
I’ve used it so many times to optimize my Chia workflow that I actually can’t live without it anymore.

WolfGT · June 7, 2023, 6:56pm

I have a Power Automate script that deletes old plots files as needed. Keeping a set number of drives with free space for new plots. If anyone is interested, I can post it up.

pj_tham · June 8, 2023, 5:09am

@WolfGT I definitely am!!!

I agree with @drhicom, but it works for plotting on empty drives…

So feel free to provide your solution!

aurelius · June 8, 2023, 6:33am

Just chiming in to say, I (1pb) simply erase in big batches manually every day while i check on the plotting. Costs less time and im not big enough to justify making a script, and the missed earnings are pretty low.

But if there was a solution provided by the community, I would definitely use it.

pj_tham · June 8, 2023, 8:52am

I don’t agree with you @aurelius…

If someone is smarter than me and can simplify my life, I take it!
I have enough trouble in my life to struggle to free some time!

aurelius · June 8, 2023, 9:04am

I agree with you. Do we really disagree ? I think you misread my message.

Relevant xckd

pj_tham · June 8, 2023, 12:03pm

I don’t fully agree with you?

seymour.krelborn · June 8, 2023, 1:26pm

aurelius · June 9, 2023, 11:29am

I’m very confused at this point to be honest lol

WolfGT · June 9, 2023, 2:07pm

Here ya go. This is a copy of the script straight out of Power Automate. To use it, just copy the text, open a new flow in Power Automate, right click in the Main window in the middle and paste the code. It will build the visual representations of the script. Then just read down through the notes entered into the code and make the changes to make it work for you. There are only 6 things to edit to customize it.

Healthcheck.io setup (If you want it to check-in to a CHECK on Healthcheck.io. Put the ping URL here so you get notifications if it ever hangs. If you don’t want this, just disable the line.
Replot before date (the system will consider any plot with a creation date older than this date eligible for being deleted. Ensure you enter the date in the format YYYYMMDD.)
New plot file size (since there are different compressed plot sizes, once you determine what compression size you are going to be using, enter the plot size here. The system will use this to determine if there is enough free space on a drive for a new plot. Note: You can always just leave it the way it is if you don’t want to bother. The size in there is the size of an uncompressed plot.)
Number of drives you want with free space (for the MadMax chia-plot-sink script, you want multiple destinations ready for plots so it can run multiple copy streams at one time. I have this set to 7 for me. It normally only has 4 or 5 running at a time on mine. But I give it a couple more just to be ahead. I have a 10G network.)
Text file containing all your plot drive locations (create a text file with your plot drive locations in it, one location per line. The script will pull that information into a list. This is much easier than trying to hard code the locations into the script.)
Log file name/location (give the script a place to log each plot it deletes. I have it setup to create a new log each day. This can be updated or the line just disabled if you don’t care to log this info.)

Then once it is all ready, just fire it up and it will delete one plot from however many drives you set it to keep free space on. (Don’t forget to save your flow) It will swing around and check each drive for plots that can be deleted (plots older than your specified date) and maintain a list of destination drives that it can work with. It will log each plot it deletes to a daily log file.

LABEL TheBeginning
# The goal of the script: Constantly make sure there are at least N drives (MinimumTargetDrives) with free space for a plot .... if there are drives containing old plots that can be deleted.
# You can setup a HealthChecks.io account and add your ping address here. That way if this script ever dies or hangs, you will get a notification.
Web.DownloadFromWeb.Download Url: $'''https://hc-ping.com/********-****-****-****-************''' ConnectionTimeout: 30 FollowRedirection: True ClearCookies: False UserAgent: $'''Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.21) Gecko/20100312 Firefox/3.6''' Encoding: Web.Encoding.AutoDetect AcceptUntrustedCertificates: False WebPageText=> WebPageText
# Set initial variables
# Give the script a date. This will tell the system to look for plots older than this date. Those plots can be deleted.
SET ReplotBefore TO 20220101
# Depending on what plot compression you are using, enter the final plot size here so the system can check to see if there is room for a new plot on each drive. This is GiB.
SET NewPlotSize TO 106.3
# This calculation is to convert the above GiB plot size to bytes. Do not modify.
SET NewPlotSize TO NewPlotSize * 1073741824
# Give the script a number of drives you want available for writing. Since the chia-plot-sink can and should write to multiple drives at a time, you want many drives ready to recieve new plots.
SET MinimumTargetDrives TO 7
# Do some magic conversions to the variables
Text.ToNumber Text: ReplotBefore Number=> ReplotBefore
Text.ToNumber Text: NewPlotSize Number=> NewPlotSize
Text.ToNumber Text: MinimumTargetDrives Number=> MinimumTargetDrives
# Create the two main drive lists. ReplotableDrivesList is dynamically updated by the script. This list will contain all drives that still have old plots that can be deleted. The TargetDrives list is a list of all plot drives. To make this list easy to populate, it reads the contents of a text file. The text file should list all plot drives one per line.
Variables.CreateNewList List=> ReplotableDrivesList
File.ReadTextFromFile.ReadTextAsList File: $'''C:\\Users\\greg\\Documents\\Chia_Replot\\target_drives.txt''' Encoding: File.TextFileEncoding.UTF8 Contents=> TargetDrives
# There is only one more thing below this point that needs updated manually. Look at line 89. That line creates a log file showing what files are deleted. Either give it a location to write to or disable the line.
# Check all drives to see how many drives have free space for plots
Variables.CreateNewList List=> DrivesWithFreeSpace
LOOP FOREACH Drive IN TargetDrives
    Scripting.RunDOSCommand.RunDOSCommand DOSCommandOrApplication: $'''dir %Drive%''' StandardOutput=> CommandOutput StandardError=> CommandErrorOutput ExitCode=> CommandExitCode
    IF Contains(CommandOutput, $'''bytes free''', False) THEN
        Text.Reverse Text: CommandOutput ReversedText=> CommandOutput
        Text.ParseText.ParseForFirstOccurrence Text: CommandOutput TextToFind: $'''eerf setyb''' StartingPosition: 0 IgnoreCase: False OccurrencePosition=> Position2
        Text.GetSubtext.GetSubtextFrom Text: CommandOutput CharacterPosition: Position2 Subtext=> CommandOutput
        Text.SplitText.Split Text: CommandOutput StandardDelimiter: Text.StandardDelimiter.Space DelimiterTimes: 1 Result=> CommandOutputList
        SET FreeSpace TO CommandOutputList[2]
        Text.Reverse Text: FreeSpace ReversedText=> FreeSpace
        Text.Trim Text: FreeSpace TrimOption: Text.TrimOption.Both TrimmedText=> FreeSpace
        Text.ToNumber Text: FreeSpace Number=> FreeSpace
        IF FreeSpace >= NewPlotSize THEN
            Variables.AddItemToList Item: Drive List: DrivesWithFreeSpace
        END
    ELSE
        Variables.AddItemToList Item: Drive List: DrivesWithFreeSpace
    END
END
DISABLE EXIT Code: 0
# Check to see if there are already enough drives with free space.
IF DrivesWithFreeSpace.Count < MinimumTargetDrives THEN
    # There are not enough drives with free space. Look for drives with old plots.
    # Update the TargetDrives list by removing the drives that already have free space
    Variables.SubtractLists FirstList: TargetDrives SecondList: DrivesWithFreeSpace OutputList=> TargetDrives
    Variables.ShuffleList List: TargetDrives
    SET ReplotableDrivesFound TO 0
    LOOP FOREACH Drive IN TargetDrives
        SET DriveHasOldPlots TO 0
        # Count how many plot files are in this target drive that were created before the replot date.
        Folder.GetFiles Folder: Drive FileFilter: $'''*.plot''' IncludeSubfolders: False FailOnAccessDenied: True SortBy1: Folder.SortBy.NoSort SortDescending1: False SortBy2: Folder.SortBy.NoSort SortDescending2: False SortBy3: Folder.SortBy.NoSort SortDescending3: False Files=> PlotFiles
        LOOP FOREACH ThisPlot IN PlotFiles
            Text.SplitText.SplitWithDelimiter Text: ThisPlot.Name CustomDelimiter: $'''-''' IsRegEx: False Result=> PlotNameList
            SET PlotCreated TO $'''%PlotNameList[2]%%PlotNameList[3]%%PlotNameList[4]%'''
            Text.ToNumber Text: PlotCreated Number=> PlotCreated
            IF PlotCreated < ReplotBefore THEN
                SET DriveHasOldPlots TO 1
                GOTO PlotsAnalyzedContinue
            END
        END
        LABEL PlotsAnalyzedContinue
        IF DriveHasOldPlots = 1 THEN
            Variables.AddItemToList Item: Drive List: ReplotableDrivesList
            Variables.IncreaseVariable Value: ReplotableDrivesFound IncrementValue: 1
        END
        IF (ReplotableDrivesFound + DrivesWithFreeSpace.Count) = MinimumTargetDrives THEN
            GOTO FoundEnoughDrives
        END
    END
END
LABEL FoundEnoughDrives
# Check to see if it has identified drives that need space cleared.
IF ReplotableDrivesList.Count > 0 THEN
    # Remove one of the old plots from each drive in the list.
    LOOP FOREACH Drive IN ReplotableDrivesList
        Folder.GetFiles Folder: Drive FileFilter: $'''*.plot''' IncludeSubfolders: False FailOnAccessDenied: True SortBy1: Folder.SortBy.NoSort SortDescending1: False SortBy2: Folder.SortBy.NoSort SortDescending2: False SortBy3: Folder.SortBy.NoSort SortDescending3: False Files=> PlotFiles
        LOOP FOREACH ThisPlot IN PlotFiles
            Text.SplitText.SplitWithDelimiter Text: ThisPlot.Name CustomDelimiter: $'''-''' IsRegEx: False Result=> PlotNameList
            SET PlotCreated TO $'''%PlotNameList[2]%%PlotNameList[3]%%PlotNameList[4]%'''
            Text.ToNumber Text: PlotCreated Number=> PlotCreated
            IF PlotCreated < ReplotBefore THEN
                DateTime.GetCurrentDateTime.Local DateTimeFormat: DateTime.DateTimeFormat.DateAndTime CurrentDateTime=> CurrentDateTime
                DateTime.GetCurrentDateTime.Local DateTimeFormat: DateTime.DateTimeFormat.DateOnly CurrentDateTime=> CurrentDate
                Text.SplitText.Split Text: CurrentDate StandardDelimiter: Text.StandardDelimiter.Space DelimiterTimes: 1 Result=> CurrentDateList
                Text.SplitText.SplitWithDelimiter Text: CurrentDateList[0] CustomDelimiter: $'''/''' IsRegEx: False Result=> CurrentDateList
                SET CurrentDate TO $'''%CurrentDateList[0]%-%CurrentDateList[1]%-%CurrentDateList[2]%'''
                File.WriteText File: $'''C:\\Users\\greg\\Documents\\Chia_Replot\\Logs\\%CurrentDate%-PlotDeletionLog.txt''' TextToWrite: $'''%CurrentDateTime% - Deleted plot - %ThisPlot.FullName%)''' AppendNewLine: True IfFileExists: File.IfFileExists.Append Encoding: File.FileEncoding.DefaultEncoding
                File.Delete Files: ThisPlot.FullName
                GOTO JumpToNextDrive
            END
        END
        LABEL JumpToNextDrive
    END
END
WAIT 60
GOTO TheBeginning
LABEL TheEnd
EXIT Code: 0

Ronski · June 9, 2023, 6:08pm

I did try one of your previous scripts, but loads of things were broken when pasting it in, likely due to different versions of power automate.

WolfGT · June 9, 2023, 7:43pm

Hard to tell. If you have issues, you can always post up and I can see if I can help. Power Automate is a free piece of software. I don’t see a reason why you wouldn’t be running the latest version.