New script to move finished plots

Since many scripts did not meet my requirements, I had to write my own script.

Background: I still have many 100GB plot files.
And also c05 and c07 files.
The new target should be c30 everywhere.
Later also: When the new plot format comes, it should also be usable.

Procedure: Only one file per RUNWORK is deleted at a time to free up 50GB.

My requirements were:

  • Delete as many files (plots) as is absolutely necessary to free up e.g. 50 GB.
    => min_free_space_gb = 50
  • Only delete files that meet my requirements:
    => to_delete_patterns = [“plot-k32-c07”, “plot-k32-c05”, “plot-k32-202”] # Patterns for files to be deleted.
  • guarantee that a file type will not be deleted
    => not_to_delete_pattern = “plot-k32-c30” # Pattern for files not to be deleted
  • To be on the safe side = do not perform a delete until the test run is confident:
    => testrun = 1 # 1 for test mode, 0 for real deletion
  • Repeat it continuously or check it periodically:
    => wait_time_minutes = 1 # Waiting time in minutes between runs

A “mover” is still required. I now use “Plow” here.

The script therefore only checks the remaining free storage space. If the storage space falls below 50 GB, a file is deleted at random so that at least 50 GB is free again. C30 plots are excluded from the deletion. If the drive is full, this is displayed accordingly. All specifications are then fulfilled. Theoretically, you could then remove the share for the drive and then delete the drive from “directories = [”.

Note: I am from Germany. Script parts are in German. However, you are welcome to modify or adapt the script as you wish.

As always. At your own risk. I do not give any guarantee or anything :wink:

Here are a few pictures of the test run and productive use

Testmodus:

Round 1. productive use

Note: now set to 1min. Unnecessary. 30 min to 60min are more useful later

And here is the code for viewing:

sudo python3 delete_files.py

import os
import time
from pathlib import Path
from rich.console import Console
from rich.table import Table
import datetime

# Konfiguration
testlauf = 0  # 1 für Testmodus, 0 für echten Löschvorgang
wait_time_minutes = 1  # Wartezeit in Minuten zwischen den Durchläufen
min_free_space_gb = 50  # Mindestmenge an freiem Speicherplatz in GB
to_delete_patterns = ["plot-k32-c07", "plot-k32-c05", "plot-k32-202"]  # Muster für zu löschende Dateien
not_to_delete_pattern = "plot-k32-c30"  # Muster für nicht zu löschende Dateien
directories = [
    "/mnt/01", "/mnt/02", "/mnt/03", "/mnt/04", "/mnt/05", "/mnt/06", "/mnt/07", "/mnt/08",
    "/mnt/09", "/mnt/10", "/mnt/11", "/mnt/12", "/mnt/13", "/mnt/14", "/mnt/15", "/mnt/16",
    "/mnt/17", "/mnt/18", "/mnt/19", "/mnt/20", "/mnt/21", "/mnt/22", "/mnt/23", "/mnt/24",
    "/mnt/25", "/mnt/26", "/mnt/27", "/mnt/28", "/mnt/29", "/mnt/30", "/mnt/31", "/mnt/32",
    "/mnt/97", "/mnt/98", "/mnt/99", "/mnt/100"
]

def get_free_space_gb(folder):
    """Gibt den freien Speicherplatz des Verzeichnisses in Gigabyte zurĂĽck."""
    st = os.statvfs(folder)
    free_space_bytes = st.f_bavail * st.f_frsize
    free_space_gb = free_space_bytes / 1024 / 1024 / 1024
    return free_space_gb

def can_delete_any_file(directory):
    """Überprüft, ob eine löschbare Datei im Verzeichnis vorhanden ist."""
    for root, dirs, files in os.walk(directory):
        for file in files:
            if any(file.startswith(pattern) for pattern in to_delete_patterns) and not file.startswith(not_to_delete_pattern):
                return True
    return False

def delete_file(directory, console):
    """Löscht eine Datei aus dem Verzeichnis basierend auf den gegebenen Kriterien."""
    deleted = False
    for root, dirs, files in os.walk(directory):
        for file in files:
            if any(file.startswith(pattern) for pattern in to_delete_patterns) and not file.startswith(not_to_delete_pattern):
                file_path = Path(root) / file
                current_time = datetime.datetime.now().strftime("%d.%m.%Y %H:%M")
                if testlauf == 1:
                    console.print(f"[white]{current_time}[/white] [TESTMODUS] Würde Datei löschen: {file_path}", style="yellow")
                else:
                    os.remove(file_path)
                    console.print(f"[white]{current_time}[/white] Datei gelöscht: {file_path}", style="green")
                deleted = True
                break
        if deleted:
            break
    return deleted

def main():
    console = Console()

    while True:
        for directory in directories:
            if get_free_space_gb(directory) < min_free_space_gb and can_delete_any_file(directory):
                delete_file(directory, console)

        # Erstelle die untergeordneten Tabellen fĂĽr die Ausgabe
        table_needs_action = Table(show_header=True, header_style="bold magenta")
        table_needs_action.add_column("Verzeichnis", style="dim", width=12)
        table_needs_action.add_column("Free Space(GB)", justify="right")
        table_needs_action.add_column("Status", justify="left")

        table_fulfilled = Table(show_header=True, header_style="bold cyan")
        table_fulfilled.add_column("Verzeichnis", style="dim", width=12)
        table_fulfilled.add_column("Free Space(GB)", justify="right")
        table_fulfilled.add_column("Status", justify="left")

        for directory in directories:
            free_space = get_free_space_gb(directory)
            if free_space < min_free_space_gb and can_delete_any_file(directory):
                table_needs_action.add_row(directory, f"{free_space:.2f}", "[bold red]Möglicherweise benötigt mehr Platz")
            elif free_space >= min_free_space_gb:
                table_needs_action.add_row(directory, f"{free_space:.2f}", "[bold green]GenĂĽgend Speicherplatz")
            else:
                table_fulfilled.add_row(directory, f"{free_space:.2f}", "[bold cyan]Die Vorgaben sind erfĂĽllt.")

        # Erstelle die ĂĽbergeordnete Tabelle und fĂĽge die untergeordneten Tabellen hinzu
        main_table = Table(box=None)
        main_table.add_column("Aktive Laufwerke", justify="left")
        main_table.add_column("ErfĂĽllte Laufwerke", justify="left")
        main_table.add_row(table_needs_action, table_fulfilled)

        console.print(main_table)

        next_run_time = (datetime.datetime.now() + datetime.timedelta(minutes=wait_time_minutes)).strftime("%d.%m.%Y %H:%M")
        console.print(f"Warte {wait_time_minutes} Minute(n), bevor der nächste Durchlauf beginnt... [white]{next_run_time}[/white]", style="bold blue")
        time.sleep(wait_time_minutes * 60)

if __name__ == "__main__":
    main()
1 Like

This works also:
.
.
:RESTART
TITLE Move Plots from Drive1 to Drive2
echo.
ROBOCOPY “E:\plots” “P:\plotsc29” “plot*.plot” /MOV /Z /R:999 /W:5
echo.
timeout 10 /nobreak
GOTO RESTART

pause

:top: # Mindestmenge an freiem Speicherplatz in GB / würde ich die Beschreibung noch anpassen das sich das auf jede HDD auswirkt → aber sonst top

Good! But, the HDD can then be fragmented, but how to delete files continuously from the beginning of the disk, i.e. from the middle?

@Hootch: Erledigt :wink: + noch n bissel mehr

@tom_fm:
A VERY good idea indeed.
Not possible in this constellation.
After all, they are network drives. You don’t have admin access to the disk. Only SMB or something.

You would have to write a 2nd script here, which then runs on the harvester. This then deletes so that the plotter per se always has storage space.

But nobody wants that. Nothing else has to be on the harvester than the harvester + firewall etc.

But what you could do: Stop my script shortly before so that 2 files still have space. Then you can send over a defrag script that builds the data one after the other. It won’t be able to do this 100%… but it should be enough.

Ansoten had to open a new thread and ask how you can guarantee this 100% WITHOUT erasing the disk completely

I’ll put it on the farm, I don’t mind one script there. However, I don’t know how to read information about files in Python so that I can delete files on the lowest sectors? (maybe add code in C?) Thanks for the advice.

Nice tidy program.
I don’t understand how the number of GB and PB are of the same magnitude. Right?