No rewards for 9863 plots for about a month

As I know this checking performance must be running on the long term storage rather than SSD which is the short term storage. is that right?

I’ve no “Found 1 Proof”. But I’ve got something, Looking up qualities, you mentioned.
But they were before I checked plots by myself.

2021-06-25T23:42:58.023 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_6/plot-k32-2021-06-06-12-49-db16bde…5d73.plot took: 14.262152910232544. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T23:42:58.024 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_6/plot-k32-2021-06-03-12-19-de885…f2b2.plot took: 14.263249397277832. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T23:42:58.024 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_6/plot-k32-2021-06-04-21-53-4a0cc…059c.plot took: 14.263813018798828. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T23:42:58.025 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_6/plot-k32-2021-06-06-15-16-3eebc…7e4ff32bd.plot took: 14.264305353164673. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T23:42:58.025 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_6/plot-k32-2021-06-04-13-46-b4b27…4ac78.plot took: 14.264768123626709. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T23:42:58.025 harvester chia.harvester.harvester: WARNING Looking up qualities on /storage_7/plot-k32-2021-06-10-10-33-54cc9…55b267.plot took: 14.264850854873657. This should be below 5 seconds to minimize risk of losing rewards.

Ok, this can be an issue… I’ll lay out in tech terms what the harvester does… then why the 14+ seconds is a lot… maybe not enough to disqualify you…

First… the role of the harvester:

“”"
The harvester receives a new signage point from the farmer, this happens at the start of each slot.
The harvester does a few things:
1. The harvester applies the plot filter for each of the plots, to select the proportion which are eligible
for this signage point and challenge.
2. The harvester gets the qualities for each plot. This is approximately 7 reads per plot which qualifies.
Note that each plot may have 0, 1, 2, etc qualities for that challenge: but on average it will have 1.
3. Checks the required_iters for each quality and the given signage point, to see which are eligible for
inclusion (required_iters < sp_interval_iters).
4. Looks up the full proof of space in the plot for each quality, approximately 64 reads per quality
5. Returns the proof of space to the farmer
“”"

So in previous messages we talked about the Farmer doing these things… but like you had before (with 25 harvesters) you can split off these tasks to harvesters that in turn communicate with the 1 Farmer who is connected to the blockchain.

The warning in the log tells us that step 2, checking the quality of a plot that passed the plot filter + challenge was found took 14+ seconds.

It still needs to perform extracting the Proof if quality is OK, and still needs to then send the Proof to the Farmer…
The Farmer then still needs to submit the Proof and if this Proof is not verified by the blockchain before the time window of 30 seconds after the challenge was issued… it cannot participate in winning the block (if it also conforms to difficulty requirement).

The quality lookup is not very read intensive… if this takes 14+seconds… you have an issue with read speed of the disks…

Are the disk in a RAID setup where there are slow disks in there that slow everything down?

Finally… 14+ seconds is too much and deserves a warning… but it could still perform the other steps and the farmer may receive the Proof in time… we don’t know at this point because:
checking the quality of the plot is just a step in generating a Proof… it could be none of these events actually generated a Proof, because the quality score was too low…

4 Likes

One thing just came to mind… can you check your systems clock (as in time) and the Farmers timestamps to make sure they are not lagging behind with real time?

If your system is creating Proofs right now that are from challenges in the past instead of the current challenge then those Proofs will be disqualified.

Hi,

As you mentioned the systems clock, for example in Windows 10 PRO, each time I synced the time, after a certain a mount of time, I see I have to sync again. How to fix this issue?

I don’t know how to keep the system clock synced always like this:

I have the same problem. More than 6k plots and not winning a block for more than 3 weeks. Main Farmer/Harvester M1 Mac Mini 5k plots + harvesters running Ubuntu Server 20.0.4

TLDR; it seems this is a bug in the Chia code. (Does anyone having >5k plots win recently?)

I spent a lot of time debugging over the last few weeks.

  1. Looking up qualities:
    I used to have this issue. But changing plot_loading_frequency_seconds: 1264000 fixed it.
    Question: does checking the plot cause the looking up issue or it’s a bug in the software? I’d say it’s a bug in the software. Why?
  • running chia plots check while farming/harvesting doesn’t create looking up qualities issue
  • running chia start harvester -r causes the looking qualities issue in several disks.
  • Disk speed test shows high throughput and milliseconds seek time.
    (HDD sleeping is disabled)
  1. Many plots pass filter every 8 seconds

  2. Network connections are all. Ping time in a few miliseconds.

  3. Fullnode has been always in sync.

I won 4 times when the number of plots was small. But not winning for more than 3 weeks when the # of plots is more than 5k

Appreciate any suggestion.

PS: I used this code to check frequently, never found a lucky day.

import subprocess as sp
import os
import time
from datetime import datetime

"""
looking for this:

07:39:12.978 full_node chia.full_node.full_node: INFO     🍀 ️Farmed unfinished_block
"""

def main(chialogs):
    # chialogs = "~/.chia/mainnet/log/debug.log"
    for n in range(8):
        log_file = chialogs
        if n != 0:
            log_file += f".{n}"

        print(f"{datetime.now()}: Checking for luck in: {log_file}")
        os.system(f"cat {log_file} | grep Farmed | grep unfinished_block")


if __name__ == '__main__':
    main("~/.chia/mainnet/log/debug.log")

This is interesting…
chia plots check does the check in serial; i.e. one plot at a time

While the harvester has multiple plots passing filter and does the lookups in parallel.
via asyncio — Asynchronous I/O — Python 3.9.5 documentation
because the lookup function is a blocking function.

What kind of setup do you have for the disks? are they in a RAID config?
Having reads going on from multiple disks could be the source of the issue you see.
with 5000 plots you’d have ~10 plots passing filter and being read from at the same time… more likely those are spread over multiple disks.

If you know how to build from source, you can uncomment this line:
self.harvester.log.debug(f"Looking up qualities on {filename} took: {time_taken}")

place the line above the “pass” in the code and remove the ‘#’ at the front to uncomment.
in chia-blockchain/harvester_api.py at main · Chia-Network/chia-blockchain · GitHub

This will record the time the lookup takes for each plot that passes filter.
You can try to isolate the issue by removing plot locations from the harvester and then slowly add them back in to see when the lookup times start to spike… go from there

The disks are just a normal USB HDD not in RAID config.

The USB standard has a maximum of 127 endpoints so normally one couldn’t connect more than 30-50 USB HDD (each HDD has 2-3 endpoints and hubs have a few endpoints).
But the number 127 applies to one USB root hub. To connect more USB HDDs, just add more USB root hubs to the thunderbolt4 ports on M1 Mac. The USB hubs that connect to TB4 are root hubs. They are similar to the root hub inside the Mac Mini, which also uses the same popular chip: FL1100LX

Great point. I’ll check it later to figure out the root cause.
It does seem something’s wrong with the parallelism.

I tried chiapos==1.0.3 with chia-blockchain==1.0.7, still had the same problem. PR#239 improves lookup time but it doesn’t seem to help. Add parallel reads to GetFullProof by marcoabreu · Pull Request #239 · Chia-Network/chiapos · GitHub

I’m super-not-trying-ta-be-all-mean-and-shit here but reading all of this (and many other posts) makes me think that the issue, and no one is all that unique in it, here is that most folks are probably screwing with shit way too much.

Meaning- I’d be willing to bet many of us haven’t completed a block at one time or another because we had a node down.

…because WE COULDN’T STOP FUCKING WITH IT.

If you know you’re running good code, know your plots are good, have decent (sub 5sec) proof times, LET IT BE!

3 Likes

@brianP

I use the same system as you. I have about 90 8TB usb drives over 6k plots.

what do you mean by look up qualities?

I generally have 8-20 pass filter and it takes .6 to 1.3 seconds on average. Is there something else I should be looking for?

I last won 2 weeks ago(maybe less than 5k at the time) but prior to that is was 4 weeks. So far I am WAY below the number I should be getting(currently every 10 days).

Thanks

It’s the time for looking up the proof-of-space. By far, It’s the most famous problem that people have. And I have been wasting too much time in fixing it :grin:.
You can run this command to check if you had the problem: cat ~/.chia/mainnet/log/debug.log | grep qualities

Here you see what I had this morning in the log files.

Mac-mini chiadog % cat ~/.chia/mainnet/log/debug.log | grep qualities
2021-06-25T10:07:17.947 harvester chia.harvester.harvester: WARNING  Looking up qualities on /Volumes/SeaI2/plots/plot-k32-2021-05-30-01-44-af6203600420f2290542e72bc1a68ce400f958b205788ebf5eeec109a0791353.plot took: 14.535951852798462. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.947 harvester chia.harvester.harvester: WARNING  Looking up qualities on /Volumes/SeaF2/plots/plot-k32-2021-06-06-04-58-fd24c8b70d2fb6519daf27adbea4ea5049e11d4c7ebf99fb241e0dbe2742d2b3.plot took: 14.536158800125122. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.949 harvester chia.harvester.harvester: WARNING  Looking up qualities on /Volumes/SeaB/plots/plot-k32-2021-04-30-00-59-74db5788b30dda720efc2e4c6fb16446e796090c48c36e9f9dc38936738ddd67.plot took: 14.538320779800415. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.981 harvester chia.harvester.harvester: WARNING  Looking up qualities on /Volumes/SeaH/plots/plot-k32-2021-05-14-18-28-04c71c1200f2a07d09e5bb7bc7a8872e97a240936224d4370cf74c8f9700eeb1.plot took: 14.570097923278809. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:18.233 harvester chia.harvester.harvester: WARNING  Looking up qualities on /Volumes/SeaJ1/plots/plot-k32-2021-05-24-19-57-e287d2684348755e0d9e1abbe58fd255d3ec27801b0505d286ea306f78135ac4.plot took: 14.822555780410767. This should be below 5 seconds to minimize risk of losing rewards.

The time looks surprisingly similar between different HDDs. It seems like a bug that could be fixed

I get these as well. I thought it was just a plot taking to long to pass the filter…

When I noticed it, it seems like it always came after a plot reload so did not think anything of it…

So is there a fix for this?

How many occurrences of lookup quality issues exist in your logs?

Out of 609,515 proof attempts in my recent logs, I have 151 occurrences of warnings on lookup qualities. .025% I am not overly concerned about that.

Leaving for the weekend but when I come back ill write a quick script to check. Never really worried about it until now but .25% does not seem to be anything to worry about.

The only thing that might concern me is what EXACTLY does that mean IE is it just the lookup on passing the filter or is it all the ones that went to the next step. If its the later I would be concerned as it is such a low percentage that move to the next step…

1 Like

Join HPool my friend. I was against it for some time, then I got tired of waiting multiple weeks for a block reward.

And redo 6k plots…pass:) Ill join a pool when I lose to much ground on the netspace

1 Like

Completely with you there. My top end is about 180 plots/day but the energy cost to do that is not so cheap. Just farming ~5K plots for now isn’t very costly.

What? You don’t have to replot a single plot to move to hpool. You point your solo plots to hpool and only run their harvester.

It’s super fast and rewards are guaranteed.

With 9k plots you’re getting 2.5-3XCH every couple weeks.

I think Littlegrand and I are referring more to the mid-term, at least I am. I’m also not willing to fuss with compromising keys.

…and more to the point, I’m earning more than 2.5-3XCH in that timeframe with 5K plots. So :man_shrugging:t4: