Introducing Swar's Chia Plot Manager (Windows, Linux, probably Mac too)

[swar]

@Tydeno

Good idea! But its Python based so I’m out (like no experience there)
From what I can tell, both tools do roughly the same. But I can tell , Swars implementation is more advanced and got better test-coverage than my PowerShell based solution called “Ploto”.
My Module is intended for people that don’t want spend time hustling around with Python and WSL.

These are very good points. I can’t even begin to explain how many questions I’ve been asked about configuring YAML / Python / some other system related issue. Nice plot manager though

[swar]

@ashrot

Does this plot manager accurately shows plots that are started with the -n 2 command?
The plotman version I have only shows phases of the first plot only and for all subsequent plots being generated by the same plotter command just shows 4:0 in the phase status.

You will run into the same issue with my Plot Manager as well.

[swar]

@DougieC

@swar If i have, for example, 4 destination drives and all 4 are in the same job. What happens when 1 of these drives becomes full? Does it skip to the next drive or will the plotting hang?

It will hang plotting. Unless you’re on the development branch because I have added that functionality. Still testing it out locally before I make it live.

[yamen]

PIP command did not work during install instructions. Guessing this is something that available on linux machines but not windows.

[DougieC]

My man. Live this weekend? I’m away for work for 2 weeks on Monday, free space check before launching new plot would be fantastic but even discard full final when full would be great.

When i get back i’ll gladly help test new dev on a spare machine if you need it.

[Eysteinh]

Got the hang of Swar’s Chia Plot manager now after some silly mistakes that caused me 2 days of headaces (no git clone just downloaded 0.0.0b etc) and have to say the manager is great! Good enough that I might tip Swar in the future once I get my hands on more Chia Thank you for giving this to the community!

[swar]

@DougieC

My man. Live this weekend? I’m away for work for 2 weeks on Monday, free space check before launching new plot would be fantastic but even discard full final when full would be great.

Should be live this weekend / when I have time. BUT feel free you try it out now. I’ve been using it myself

[swar]

@Eysteinh

Got the hang of Swar’s Chia Plot manager now after some silly mistakes that caused me 2 days of headaces (no git clone just downloaded 0.0.0b etc) and have to say the manager is great! Good enough that I might tip Swar in the future once I get my hands on more Chia Thank you for giving this to the community!

Of course, good luck plotting!

[ashrot]

OHH! that is very unfortunate! Any plan on addin this functionality?

[cryptn00b]

I am not tech savvy and don’t know coding at all, was just about to go full blown debian and learn linux from scratch for plotting/farming machines until I found this, I hope I can understand how to implement it, if I’m inside your target userbase at all

[jadatmag]

When this was about to happen I manually switched my full usb HDD for an empty one and assigned the right drive letter in windows disk management so it would still align with the config yaml.

It worked!

[swar]

@yamen

PIP command did not work during install instructions. Guessing this is something that available on linux machines but not windows.

Nope this was originally made for Windows. Everything works for all three major operating systems.

@ashrot

OHH! that is very unfortunate! Any plan on addin this functionality?

Unfortunately no plan for it. That’s mainly because my plot manager does not use the -n flag.

@cryptn00b

I am not tech savvy and don’t know coding at all, was just about to go full blown debian and learn linux from scratch for plotting/farming machines until I found this, I hope I can understand how to implement it, if I’m inside your target userbase at all

I think the instructions are pretty straight forward, but take that with a grain of salt. I am a software engineer after all.

[racabu]

@swar

@variableresults

Overall, I think the next logical step for Plot Manager would be analysis of the plotting system’s specs, such as the Temp storage used, CPU, etc, to dynamically manage and optimize plotting. Or at least allow the user to enter in the type of hardware they have if it’s too hard to poll the system resources.

I think you should first create some sort of reporting that based on statistics collected from the machine, you can recommend changes to the configuration. So you just provide the info and the user changes.

[gumush]

Great works that helps us to manage plots. I see PID’s available in this chart but when i kill thatprocess to stop spesific plot with taskkill /F /PID pidnumber it respawn automaticaly. Is there any way yo kill spesific plotting process ?

[Voodoo]

There is another thing that would be really useful to help deal with problems.

I sometimes run into the problem that a plot get’s stuck for no apparent reason. But it can take a long time to spot this and kill it off.

So if the (push) notification could give a warning about this would be really useful.
Something like, if log file of plot A doesn’t grow in size for x minutes->send message your plot might be stuck

[WolfGT]

For running parallel jobs on Windows, this is a requirement. I have found that if you do it in the GUI or with a script, after a couple days, all the jobs end up running in the same phases at the same time. Due to the heavy I/O of some of the phases, they all sync up over time. With Swar’s, it keep them in check. Making sure they don’t all stack up and also taking advantage of open resources when available. Big fan so far.

[vanlong]

Could you make new notification method via telegram bot and SMTP in the future, I think that will be greated !

[swar]

Release v.0.1.0 · swar/Swar-Chia-Plot-Manager

v.0.1.0

The first feature packed improvement of the plot manager.

New Features

• Adding exclude_final_directory as an option in the config.yaml. (#35)
• Skipping manager.log as a file and renaming to debug.log. (#38)
• Added destination directory skipping when a drive is full using skip_full_destinations at the job level. It will calculate size of total running plots and the predicted size of the new plot prior to making that judgement. (#36, #193)
• Added global setting for max_for_phase_1.
• Added global setting for minimum_minutes_between_jobs. (#380, #468)
• Added list support for temporary directories. This will cycle through all temporary directories in the order that they are listed for a job. (#150, #153, #182)
• Added CPU affinity support on the job level. This allows you to select and dedicate specific threads to your jobs. (#134, #281)
• Added process priority levels on the job level. This allows you to set the priority levels to whatever you choose. Some people want low priority, while others want higher priorities. (#282)
• Added an option to delay a job by a set number of minutes. If you started manager and there is a stagger for the job, it will use the initial delay only if it is longer than the stagger. (#283)
• Added an option in manager.py to spit out a single instance of the view using the status argument as well as json format of the jobs. (#300, #374)
• Added support for Telegram notifications. (#316, #364)
• Added support for IFTTT webhooks. (#425, #471)
• Added support for instrumentation using Prometheus (#87, #196)

Changes

• Switching notification imports to a separate requirements file and turning them into lazy imports. (#159, 196)
• Reworked the Drives Table in the view to include associated jobs. This includes minor tweaks to the display to remove ambiguity such as renaming plots to “#”. (#191, #368, #406)
• Adding basic checks that will break and have more detailed error messaging to assist in end-user interaction. Also, I was tired of getting the same repeat questions over and over again.
• Adding more psutil error handling.
• Jobs are now unique based on temporary directories.

Bug Fixes

• Fixed a bug where max_plots was not working properly. It was counting running plots when you restarted manager. Now it will only count new plots kicked off.
• Fixed a bug in elapsed_time column where elapsed days greater than 24 hours were resulting in calculations being off by a day. (#190)
• Skipping processes that result in an AccessDenied error when finding manager processes. (#147)
• Fixed a bug where psutil going stale on Linux users was not allowing the script to restart on its own. (#197)
• Fixed a bug where NFS drives weren’t being identified. (#284)
• Removed the hardcoded next log check date in the view.
• Fixed a bug where NoSuchProcess error pops up when viewing opened files.

[swar]

To everyone with feature requests, I’m sorry for not replying to you individually but I have been swamped with requests / questions / emails and everything else. I did not expect this to get as big as it currently has and I certainly do not have time to continue to add more and more features and functionality.

It is hard enough to balance my time between family and friends, work, and my masters program, then it gets even worse adding in open source projects into the mix.

I appreciate the love that this is getting, but I am just one person trying to keep up with everything.

[f1gm3nt]

@swar

skip_full_destinations

This is why I just switched from plotman to using this