Harvester falsely identifying duplicate plots

Greetings,

The GUI, and the “chia farm summary” command are under-reporting my plot count, by over 1,000 plots.

The debug log contains over 1,000 lines that read:
2021-09-21T09:09:48.051 harvester chia.plotting.manager : WARNING Have multiple copies of the plot (and the rest of the line simply spells out the name of the plot and its location)

The above repeats over 1,000 times (for different plots), and would seem to be the reason for Chia under-reporting my plot count.

I have zero duplicate plots.
I have manually searched, everywhere, for several of the supposedly duplicate filenames. No duplicates exist.

I also run NirSoft’s “SearchMyFiles.exe”, as it has a feature specific for identifying duplicate filenames. It found no duplicates.

And there is no way that anyone could possibly duplicate over 1,000 files, and not know that they did so.

Please also note that the so-called duplicates are all located on 8 of my USB drives. Each of the 8 drives has a mix of valid plots and supposedly duplicate plots.

However, if I set-up config.yaml to farm only one of those drives at a time, then all of the plots get loaded. But when I have config.yaml set to farm all of my drives, then Chia rejects over 1,000 plots.

Does the Chia program track what it had once identified as a duplicate, and retains that finding, even when there is no longer a duplicate filename? If so, how do I tell Chia to reset its findings?

I did not have this issue, when running version 1.1.7
But I eventually jumped to version 1.2.5 and that is when the false positives began.

I just installed 1.2.7 and crossed my fingers. But the problem persisits.

All help is welcome.

Thank you.

Have you tried removing all hdd from your list of places to search for plots, and then re adding 1 drive at a time till they are all added?

In your config.yaml, look for the “plot_directories:” section. Maybe for some reason you have duplicate folders there. If yes, just clean that section to have only unique ones.

If you want to disable those folders (for testing, etc.), just use "# " to comment those lines. If you still see problems, just comments all folders (folders only, not the title of that section), and start uncommenting them one at a time.

There is no need to restart your harvesters, as by default it will rescan those folders every 2 minutes. No need to restart UI as well, although it will be lagging a bit, but check “Last Attempted Proof” section, as it will pick up those plots as soon as harvester will find them.

Long time ago, I had problems with Chia keeping “bad” files around (when for instance copying a file directly to a plot folder) I think that it was fixed a month or so ago. However, that indicates that the “sticky” logic may be still lingering in other parts of the code (what I rather doubt, as QA should pick such changes right away).

However, as mentioned above, every 2 minutes or so the harvester checks config.yaml for what are the search folders at the moment, and just verifies that it sees the same folders/plots as last time, and only works with those diffs (used to show all new files, don’t remember whether there was a trace of removed files).

As far as that rescan section, they changed it recently, and AFAIK don’t propagate new entries to old config files during updates, so you may want to add those lines manually (it is a section now, not a single duration).

Yes, I updated my config.yaml file to check only 1 drive, and then removed that drive and updated config.yaml to check the next drive, etc,

In each case, Chia loaded 100% of the plots on each connected drive.

But when my config.yaml file points to all of my drives, then the debug log complains about over 1,000 duplicate plots, and Chia falls short by that number of plots in its listing and time to win estimate.

And I do not have over 1,000 duplicate plots. I have zero duplicate plots. I am 100% certain.

I have no duplicate entries.
This problem surfaced when I upgraded from version 1.1.7 to 1.2.5 (and remains with version 1.2.7).

I even added each USB drive into config.yaml, one-by-one, and verified that my plot count went up accordingly.

All went well, until I added my 50th (or thereabout) entry into config.yaml. Upon that 50th entry, my plot count plummeted, and the so-called duplicate plots were reported in config.yaml.

“…add those lines manually…”
Which lines?

“…(it is a section now, not a single duration)…”
I do not know what you mean. Would you please clarify?

Thank you.

In the old (pre v1.2.6 or 1.2.7) rescan timeout was controlled by

plot_loading_frequency_seconds: 120 // 120 seconds

With that new change, during the update, they don’t read the value you may have in your config file, but use a new section (that is not inserted in the old config.files):

plots_refresh_parameter
  interval_seconds = 120
  retry_invalid_seconds = 1200
  batch_size = 300
  batch_sleep_milliseconds = 1

Looks like you have a big load on your harvester, so they recommend to increase (by a lot, e.g., 10x) that “interval_seconds” field.

Since for you (or me) that section is not in config file, it is hard to guess what to do, unless you are pointed to it.

From what you described, you are hitting either some OS limit, or Chia code is confused with that number of folders, and goes bonkers.

By the way, it used to be that when that timeout happened, there were INFO entries in the log file about which folders are being scanned, and plots added. I don’t see that “being scanned” lines there anymore (maybe after updating to v1.2.6), and have not added new plots recently. You may want to disable/enable one folder, and check whether log file will have some interesting info.

Although, there may be one more possibility. If you have a lot of folders/plots (don’t know what “a lot” means), it may be that the scanning process is just too slow, and the code is starting to chase its own tail (e.g., getting new data, when the old is not being processed, and then being confused about missing that new data, …).

How many drives you have sitting on that harvester?

Update
Here is the link that talks about those new entries. It looks like it is still under development, i.e., there is no clear understanding how to handle a big amount of folders/plots, as such you may be affected by that bug.

And as that link indicates, that change was added in v1.2.5, so exactly when you started having those problems - new, badly tested code.

So, add that section, make that entry be like 30-60 minutes or so, and see whether it helps. If you are still adding new plots, maybe it is better to wait that 30-60 minutes to pick those plots up, than have the software start barfing again.

By the way, that rescan is needed only when you are adding new plots to the harvesters. Otherwise, I think you may even disable it, if there is such possibility (on startup, that harvester will rescan your folders, whether you have that timeout or not - at least i think so). So, setting it to that 30-60 minutes is really not that big of a deal.

Also, I don’t understand why that section is so rigid, or is there in the first place. When you (I mean, chia code - program) load those folders, you know what speed you are going with, so that refresh should be dynamic, based on the folder/plots count and loading times. It is just kicking the can toward the end-user to avoid coding a more flexible solution. In my opinion, there should be one entry there “scan: on/off” and a way to manually (UI) triggering that rescan. Also, there is a inotifywait (on Linux, but you can code similar thing on Windows as well)
tool that can be used to monitor folders for changes. That would let that rescan chore be handled by the tool, and have your code run only when the tool will notify you (so minimal code will be needed to do those rescans, if at all). However, that is a different story.

I found a better link in chia github sources. Just search for “plots_refresh_parameter” The other link that I put above is to a Chia fork (still based on the main Chia code, I think, so their explanations may apply to Chia as well).

1 Like

Thanks for the info. I think you have identified the issue.

Do you know if installing a previous version of Chia will work?
I am considering installing 1.2.4, and if that fixes my issue, that is fine with me.

But I do not want to attempt doing so, at the risk of making things worse.
And I think that I can live without the improvements from versions after 1.2.4.
Whatever those improvements are, they are not worth losing over 1,000 plots.

If you are unsure about back-revving Chia’s version, then I will post my question as a new thread.

Cheers!

How many plots do you have, what CPU, how much RAM, what OS? Maybe you could just install a VM on that box, and run a harvester that would take 50% of your drives from the current one. This way, you may be able to relieve your current harvester/farmer a bit (if the harvester code is the issue).

I am on v1.2.6, and it is more CPU demanding (not much). I upgraded, either from v1.2.2 or 1.2.3 directly to v1.2.6. Either v1.2.3/4 or v1.2.4/5 were rather bad, so if you want to go back, you need to check whether to go to v1.2.3 or 1.2.4. I really don’t recall it. But, if you google more that new entry, I think it was introduced in v1.2.4, so maybe v1.2.3 would be safer for you.

But, do ask that question on github Issues page, as you may get some support guy run it over by devs. We are all here (this forum) affected in different ways, so my experience may be really worthless for you.

Also, if you are not actively plotting, changing pools, etc., you may be good with v1.2.2 or v1.2.3.

Your idea of a virtual machine is interesting. But I do not want to install any software on my Chia rigs, unless absolutely necessary (Windows has a way of easily getting bloated and having issues).

Your suggestion, however, presents me with, perhaps, a similar solution.

I have 2 Chia machines. I create plots on both of them, via batch files that run:
chia.exe plots create blah blah blah

But I have only machine #1 running the GUI (and the GUI runs everything that needs to run – and that is the limit of my understanding – which does not include the nuances of what the GUI actually runs). I just know that it takes care of everything.

I have machine #1’s config.yaml set up to farm/harvest all plots on both machines. Machine #1 is able to access all of the plots on machine #2, via “map network drive”.

But this whole farmer and harvester topic confuses me, even after I have read postings that explain them.

But from what you wrote, it sounds like I could have machine #2 do its own farming / harvesting, via its own config.yaml file? Is that correct?

If yes, that would split the load, and might fix my issue.

But I have no idea how to set that up, such that both machines share the same wallet.

If you can walk me through setting that up, or provide a link that clearly explains the steps, I would appreciate it.

Thank you.