Anyone run the "chia plots check" command?

So you have a lot of plots… have you run the chia plots check command on them to make sure they’re all valid and OK?

https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#check

I’m gonna run this on all my plots and report back. First I did

chia plots show

Which looked good; showed all the paths of my plots:

Directories where plots are being searched for:
Note that subdirectories must be added manually
Add with 'chia plots add -d [dir]' and remove with 'chia plots remove -d [dir]' Scan and check plots with 'chia plots check'

E:\chia-plots
F:\chia-plots

So we know that to check a plot it has to be somewhere in the plot paths, first!

Once you’ve established that your plot paths are good, to check any individual plot(s) you can use -g and any portion of the filename and it will check everything that matches, including the drive path, like so:

.\chia plots check -n 30 -g E:\chia-plots\plot-k32-2021-05-05-18-18-1f3bc58e47abf9755fa3f2311f59296f869e031a6c8d7ee8bb185c2a991f0c44.plot

INFO    Starting to test each plot with 30 challenges each
INFO 
INFO    Testing plot E:\chia-plots\plot-k32-2021-05-05-18-18-1f3bc58e47abf9755fa3f2311f59296f869e031a6c8d7ee8bb185c2a991f0c44.plot k=32
INFO    Pool public key: INFO b59e5c3f507029ac4bbff11e00aac05588effc49d210273080ee17f0fcbdf09c70501ce9aa2686411d00ed3bc8d77814
INFO    Farmer public key: 8d577c6df158f02314668bdb591de52a0ddc1b04c45f99d81ffaba5ccb0b27dc4fd5918725d9df94cef414971a753db5
INFO    
INFO    Proofs 38 / 30, 1.2667
INFO    
INFO    Summary
INFO     Found 1 valid plots, total size 0.09904 TiB
INFO    1 plots of size 32

and here’s what it looks like for a bad plot:

ERROR    Failed to open file D:\plot-k32-2021-05-05-16-40-c958b1d08bc76eb0b393e84a429c4cf93075214e07f61b463af9132ac75ddc3f.plot. Invalid plot header magic Traceback (most recent call last):
  File "chia\plotting\plot_tools.py", line 189, in process_file
ValueError: Invalid plot header magic

INFO     Loaded a total of 0 plots of size 0.0 TiB, in 0.7966477870941162 seconds
INFO
INFO
INFO     Summary
INFO     Found 0 valid plots, total size 0.00000 TiB
WARNING  1 invalid plots found:

To check all your plots, you can use

chia plots check -n 5 -l

and piped the output to a file so I can see what’s up! The docs indicate -n 5 is a very basic check, the absolute minimum… but that’s all I want, I might have corrupted plots or plots that didn’t copy correctly:

Running the command with -n 10 or -n 20 is good for a very minor check, but won’t actually give you much information about if the plots are actually high-quality or not.

-l checks for duplicates, I definitely want to identify duplicate plots which I may have from all my copying of stuff around so that’s a benefit all by itself…

6 Likes

The debug log will actually complain if there is a duplicate plot file (I’ve had a couple).

Ah yes I am on Windows so I’m looking in

C:\Users\user\.chia\mainnet\log\debug.log

Indeed I see a bunch of dupes being reported:

2021-04-23T17:39:10.307 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-10-14-58-c5a7d7b57078099b1a28a1a194f20540420026ec522433eea7c8a17c9e618535(1).plot, not adding it.
2021-04-23T17:39:10.588 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-16-22-30-b08201cff5a8f41516fdef96a51ec7ca93a21b789057067ffee6fc961774cb38(1).plot, not adding it.
2021-04-23T17:39:11.198 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-17-03-23-72833505e515ef0a1f5cba970959a7eecdd747505556c0a846773d4ec139883e(1).plot, not adding it.
2021-04-23T17:39:11.213 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-10-06-45-7aa5fa5c7fd40ae18613c37c2aae83149daefa1689482f0d3ecfeb574df35116(1).plot, not adding it.
2021-04-23T17:39:11.244 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-10-15-26-b2e2ca5070bc7fce00b8a73e20bafc5a9794f2c2d9b50ec54774a74df3431dd5.plot, not adding it.
2021-04-23T17:39:11.260 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.97\public\chia-plots-1\plot-k32-2021-04-15-16-11-f171ccdc7d01eb536d647f337558837e2c38e46940ba739c178d808b2c8b38e0(1).plot, not adding it.
2021-04-23T17:39:13.854 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.223\public\chia-plots-1\plot-k32-2021-03-30-00-13-67db759c36afedb74e92e37b197a3f7b1678936a7935d10aab9095e53c40a743.plot, not adding it.
2021-04-23T17:39:14.010 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.223\public\chia-plots-1\plot-k32-2021-04-14-05-01-1f537c2de700762b9d5ecaf8d56b5814a511983a985ba7800029c05635516181.plot, not adding it.
2021-04-23T17:39:14.135 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.223\public\chia-plots-1\plot-k32-2021-03-30-18-36-b30c670b9e1a895c55c59af9846f547280bcf3a79c05bde70f8dc4aad69eaeea(1).plot, not adding it.
2021-04-23T17:39:14.182 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.223\public\chia-plots-1\plot-k32-2021-04-14-04-43-5b0b7d7e274074c7476ac9c0461eaa063dffa84becfae9ed33504b456e5ebfc9.plot, not adding it.
2021-04-23T17:39:14.479 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.223\public\chia-plots-1\plot-k32-2021-03-28-21-48-33332bb6f0722139ade814f2bd7fe3d8747f00c778684e9492c9e2c0f7639dca.plot, not adding it.
2021-04-23T17:39:14.698 harvester chia.plotting.plot_tools: WARNING  Have multiple copies of the plot \\192.168.1.222\public\chia-plots-1\plot-k32-2021-04-06-19-40-aad47674376adb019e8bc03ad534f64bffc19fcc0846febb111f82ab82dbe7ca.plot, not adding it.

I opened a dedicated topic since people should be checking their debug.log regularly!

Running the command with -n 10 or -n 20 is good for a very minor check, but won’t actually give you much information about if the plots are actually high-quality or not.

And then later in that doc it says:

In theory , a plot with a ratio >> 1 would be more likely to win challenges on the blockchain. Likewise, a plot with a ratio << 1 would be less likely to win. However, in practice, this isn’t actually going to be noticeable. Therefore, don’t worry if your plot check ratios are less than 1, unless they’re significantly less than 1 for many -n .

So when that says high-quality does it literally mean some plots are better quality than others with an increased chance of winning? The way I read that last sentence implies it’s possible to have low-quality plots too.

I would love a detailed explanation because I only have 40TB of space and it’s going to be full by the end of the weekend. If I can continue plotting and replace low quality plots with high quality plots I would consider it even for marginal gains.

However, it also says (emphasis mine):

If the ratio is >1, your plot was relatively lucky for this run of challenges.

I’d like to know if / how the set of challenges changes because I’d isolate a single plot and run a check on it every time the challenge changes (if it changes) to how the results differ.

This shouldn’t really concern you unless your ratio is <0.70

Disclaimer: I don’t know if this paragraph is accurate. If it’s consistently 0.7, that would make re-plotting worthwhile at some point wouldn’t it? If it’s consistent, and the expectation is 1.0 per plot, that means you can gain .3 by replacing that low quality plot. Of course you’re much better off making a new plot for the full gain of 1.0 if you have space, but once you’re out of space replotting a 0.7 plot to a 1.0 makes that low quality plot 43% (1/.7) better. Again, that’s making a ton of assumptions that might be wrong.

Does anyone around here have an understanding of how it works?

1 Like

Chiadog is a great tool for being alerted when things happen in the log file.

I went AFK for the weekend but get an email every time I win a proof or if any warnings show up in the log like number of plots decreasing. I don’t think it alerts on everything, but it’s definitely open for contributions. Email is in the dev branch but should be moving to main this week I think.

I ran check plots after discovering I had bad memory that was corrupting plot data. It was able to identify invalid plots right away.

1 Like

Do you mean with chiadog or with chia check plots? Would you be willing to explain that a bit more? I’d really like to do some verification of my plots once the disks are idling.

To check the plots after my memory issue I ran chia plots check in the cli. If you have an invalid plot it tells you as it does the “quality” check, basically saying the plot is invalid and can’t be checked for proofs.

I think I remember in keybase someone saying if you make the n large enough during check plots all plots will trend towards 1, so there’s no real need to prune “low quality” plots. I could be wrong. Definitly a micro-optimization to consider once you’re 100% at capacity.

In developer’s experience bad plots happen due to hardware.

chia plots check primarily was written early on to verify plotting software was working properly, and modified later to check for invalid plots. Invalid plots can occur when the bits on the plots are changed. All the invalid plots I have ever troubleshoot was due to hardware issues of one way or another; from failing hardware, heat related errors, or too high overclocked CPU or memory.

The way devs verified the plotting software was to ensure that the probability for finding a proof reached close to 1 when -n became large enough. Usually -n 1000 was used for testing the plotting software.

For end users using chia plots check, you should just worry about invalid plots found as the priority. If you encounter a very low probability plot (< 0.05 with default -n 30), re-run chia plots check -n 1000 -g [plot filename] to verify it’s not bad luck streak with random numbers.

If you do have a plot with low probability (< 0.4) after chia plots check -n 1000, please message me. I want to debug the problem.

FYI - I am the community developer who modified and maintains chia plots check

I too wondered about this seeing redditors complainig often, but keep in mind we are really daring randomness here shifting more data in few days than most computers see over their whole lifetime.

As to the quality reporting chia plots check does there are good mathematical reasons for probability to converge over large datasets, as well as optimizations in the proof checking process (see “quality string” in the paper). The command was created to verify the implementation. Checking for file consistency being an afterthought.

If you do have a plot with low probability (< 0.4) after chia plots check -n 1000 , please message me. I want to debug the problem.

The implementation paper expressedly mentions replotting as a microoptimization one could want. That’s actually an attack and shouldn’t do very much over a large dataset, but sure, why not.

Replotting Attack

Some plots will by random noise have greater per-byte effectiveness. The amount of this is proportional to the square root of the size of the plot with a small constant factor so the potential gains aren’t high. Furthermore, this requires evaluating the entire forward propagation phase. If a farmer has multiple plots, and wants to delete one of them, the least effective one can be reclaimed first.

https://www.chia.net/assets/Chia_Proof_of_Space_Construction_v1.1.pdf

6 Likes

Hey thanks for cross-posting here, it saves me time to explain chia plots check in this forum.

As I mentioned in reddit, If you do have a plot with low probability (< 0.4) after chia plots check -n 1000 , please message me. I want to debug the problem.

Thanks!

3 Likes

I didn’t check a ton of mine, but I ran -n 5, -n 10, etc. on a few disks and cherry picked some of the lower / higher ones for -n 100 checks. Even with only -n 100 most of them approached 1 as explained. The most extreme ones were at .9 with -n 100.

4 Likes

If you want to only find duplicate plots fast, I coded in chia plots check -n 0 -l will do the trick.

4 Likes

Why isn’t it possible to execute CLI commands with Linux? I can’t execute any. Am I in the wrong folder location?

Never mind. I found the folder.

Cool can you share the folder path for others who might look for this info later? That’d be helpful!

Wow thank you! This definitely worked; cleaned up a bunch of dupes! Appreciate that! Output looks like:

WARNING  67506dd6f81ea4ceb02eec05c1166d924d2094665a7c80bf8c7077a450b23aeb.plot found in multiple files:
        C:\mounts\hd31\plot-k32-2021-05-06-17-05-67506dd6f81ea4ceb02eec05c1166d924d2094665a7c80bf8c7077a450b23aeb.plot
        C:\mounts\hd32\plot-k32-2021-05-06-17-05-67506dd6f81ea4ceb02eec05c1166d924d2094665a7c80bf8c7077a450b23aeb.plot

It also warns you if it finds files with unusual extensions, I had a few *.plot.plot floating around. Weeding my garden… :seedling:

2 Likes

A post was split to a new topic: Plot check crashes with malloc(): invalid size (unsorted)

~/chia-blockchain/venv/bin

Hi,
I have found around 25 of 550 plots which were completely broken with this error output and deleted them:

    2021-05-29T13:23:03.230  chia.plotting.check_plots        : ERROR    <class 'RuntimeError'>: Src size is incorrect error in getting challenge qualities for plot L:\chia\plot-k32-2021-05-12-09-09-e9f1046be7d5d823561dc25e77833235128b86057e6e91aaa5ec49ae9f30a867.plot
    2021-05-29T13:23:03.231  chia.plotting.check_plots        : ERROR       Proofs 0 / 50, 0.0

But one of my plots is really strange:

    Starting to test each plot with 1000 challenges each

    2021-05-29T13:28:13.265  chia.plotting.check_plots        : INFO     Testing plot L:\chia\plot-k32-2021-05-04-02-19-5ad846f11b03fc0f7fbc31290662704d0878e47e59325b0f5e67516bce7f108d.plot k=32
    ...
    2021-05-29T13:28:13.836  chia.plotting.check_plots        : ERROR    <class 'AssertionError'>:  error in proving/verifying for plot L:\chia\plot-k32-2021-05-04-02-19-5ad846f11b03fc0f7fbc31290662704d0878e47e59325b0f5e67516bce7f108d.plot
    2021-05-29T13:28:14.239  chia.plotting.check_plots        : ERROR    <class 'AssertionError'>:  error in proving/verifying for plot L:\chia\plot-k32-2021-05-04-02-19-5ad846f11b03fc0f7fbc31290662704d0878e47e59325b0f5e67516bce7f108d.plot
    2021-05-29T13:28:14.581  chia.plotting.check_plots        : ERROR    <class 'AssertionError'>:  error in proving/verifying for plot L:\chia\plot-k32-2021-05-04-02-19-5ad846f11b03fc0f7fbc31290662704d0878e47e59325b0f5e67516bce7f108d.plot
    2021-05-29T13:28:14.581  chia.plotting.check_plots        : ERROR       Proofs 3 / 1000, 0.003
    2021-05-29T13:28:14.581  chia.plotting.check_plots        : INFO
    2021-05-29T13:28:14.582  chia.plotting.check_plots        : INFO
    2021-05-29T13:28:14.582  chia.plotting.check_plots        : INFO     Summary
    2021-05-29T13:28:14.582  chia.plotting.check_plots        : INFO     Found 0 valid plots, total size 0.00000 TiB
    2021-05-29T13:28:14.582  chia.plotting.check_plots        : WARNING  1 invalid plots found:

Do you mean sth. like this one? Or do you mean only plots which are flagged as OK but have < 0.4?

Those are probably fine, it’s only the ones where it says they are invalid that you need to worry about. In my experience if it says they are invalid, then they are!

You can run more -n 100 tests to verify if you are worried.

Thanks for posting this. 1 plot was in error, and 1 was incomplete! I am going to delete them and re-plot those.

Hey @eFishCent, thanks for asking to have us message you. I’ve gotten 7 our of 110 plots being invalid… Let me know what information you need to debug them…

I’m running this command:
.\chia.exe plots check -n 1000 --challenge-start 1350 -g "d16356ed162f40907bfa9624a91fb6984583ffd70f4cbd9fde341b7e012dbe8a"

These are the results:

2021-06-01T06:24:43.648  chia.plotting.plot_tools         : INFO     Loaded a total of 1 plots of size 0.09894170303050487 TiB, in 0.0624852180480957 seconds
2021-06-01T06:24:43.648  chia.plotting.check_plots        : INFO
2021-06-01T06:24:43.648  chia.plotting.check_plots        : INFO
2021-06-01T06:24:43.648  chia.plotting.check_plots        : INFO     Starting to test each plot with 1000 challenges each
2021-06-01T06:24:43.648  chia.plotting.check_plots        : INFO     Testing plot F:\Chia Plot Folder AB\plot-k32-2021-05-18-03-05-d16356ed162f40907bfa9624a91fb6984583ffd70f4cbd9fde341b7e012dbe8a.plot k=32
2021-06-01T06:24:43.648  chia.plotting.check_plots        : INFO        Pool public key: 96e7985913d77d2d106f62f2d9db985d8cba821ccd8ca170af4d34dbaf7ecc53eeeb0b8fc2cfe6a91a65df8596d2c076
2021-06-01T06:24:43.664  chia.plotting.check_plots        : INFO        Farmer public key: 8081105dc5a5a843eb670c5c272ce378d3f5228b978b464eb609638cadc71ba786fec7f8d9796549ce8bba0969576bcb
2021-06-01T06:24:43.664  chia.plotting.check_plots        : INFO        Local sk: <PrivateKey [redacted :zipper_mouth_face: ]>
2021-06-01T06:24:45.726  chia.plotting.check_plots        : ERROR    <class 'AssertionError'>:  error in proving/verifying for plot F:\Chia Plot Folder AB\plot-k32-2021-05-18-03-05-d16356ed162f40907bfa9624a91fb6984583ffd70f4cbd9fde341b7e012dbe8a.plot
2021-06-01T06:24:46.286  chia.plotting.check_plots        : ERROR       Proofs 4 / 1000, 0.004
2021-06-01T06:24:46.286  chia.plotting.check_plots        : INFO
2021-06-01T06:24:46.286  chia.plotting.check_plots        : INFO
2021-06-01T06:24:46.286  chia.plotting.check_plots        : INFO     Summary
2021-06-01T06:24:46.286  chia.plotting.check_plots        : INFO     Found 0 valid plots, total size 0.00000 TiB
2021-06-01T06:24:46.286  chia.plotting.check_plots        : WARNING  1 invalid plots found:
2021-06-01T06:24:46.286  chia.plotting.check_plots        : WARNING  F:\Chia Plot Folder AB\plot-k32-2021-05-18-03-05-d16356ed162f40907bfa9624a91fb6984583ffd70f4cbd9fde341b7e012dbe8a.plot

So that’s 1 of 7 bad plots. I’m running Chia 1.1.6 on Windows 10, i7-10700 CPU @ 2.90GHz, 32GB RAM, using 1TB WD NVMe SSD for Temp 1, 1TB SSD for Temp 2.

Before upgrading to Chia 1.1.6 I was just getting low results on the plots, but now that I’ve upgraded to 1.1.6 I get the “AssertionError” on all 7 plots…

Any guidance on what to do would be appreciated! I still have a few TB to fill, but in the next few days I’d love to either fix these plots or just delete and re-write them.

Thanks for everyone’s help!