Hi,
As you mentioned the systems clock, for example in Windows 10 PRO, each time I synced the time, after a certain a mount of time, I see I have to sync again. How to fix this issue?
I don’t know how to keep the system clock synced always like this:
I have the same problem. More than 6k plots and not winning a block for more than 3 weeks. Main Farmer/Harvester M1 Mac Mini 5k plots + harvesters running Ubuntu Server 20.0.4
TLDR; it seems this is a bug in the Chia code. (Does anyone having >5k plots win recently?)
I spent a lot of time debugging over the last few weeks.
plot_loading_frequency_seconds: 1264000 fixed it.chia plots check while farming/harvesting doesn’t create looking up qualities issuechia start harvester -r causes the looking qualities issue in several disks.Many plots pass filter every 8 seconds
Network connections are all. Ping time in a few miliseconds.
Fullnode has been always in sync.
I won 4 times when the number of plots was small. But not winning for more than 3 weeks when the # of plots is more than 5k
Appreciate any suggestion.
PS: I used this code to check frequently, never found a lucky day.
import subprocess as sp
import os
import time
from datetime import datetime
"""
looking for this:
07:39:12.978 full_node chia.full_node.full_node: INFO 🍀 ️Farmed unfinished_block
"""
def main(chialogs):
# chialogs = "~/.chia/mainnet/log/debug.log"
for n in range(8):
log_file = chialogs
if n != 0:
log_file += f".{n}"
print(f"{datetime.now()}: Checking for luck in: {log_file}")
os.system(f"cat {log_file} | grep Farmed | grep unfinished_block")
if __name__ == '__main__':
main("~/.chia/mainnet/log/debug.log")
This is interesting…
chia plots check does the check in serial; i.e. one plot at a time
While the harvester has multiple plots passing filter and does the lookups in parallel.
via asyncio — Asynchronous I/O — Python 3.12.3 documentation
because the lookup function is a blocking function.
What kind of setup do you have for the disks? are they in a RAID config?
Having reads going on from multiple disks could be the source of the issue you see.
with 5000 plots you’d have ~10 plots passing filter and being read from at the same time… more likely those are spread over multiple disks.
If you know how to build from source, you can uncomment this line:
self.harvester.log.debug(f"Looking up qualities on {filename} took: {time_taken}")
place the line above the “pass” in the code and remove the ‘#’ at the front to uncomment.
in chia-blockchain/chia/harvester/harvester_api.py at main · Chia-Network/chia-blockchain · GitHub
This will record the time the lookup takes for each plot that passes filter.
You can try to isolate the issue by removing plot locations from the harvester and then slowly add them back in to see when the lookup times start to spike… go from there
The disks are just a normal USB HDD not in RAID config.
The USB standard has a maximum of 127 endpoints so normally one couldn’t connect more than 30-50 USB HDD (each HDD has 2-3 endpoints and hubs have a few endpoints).
But the number 127 applies to one USB root hub. To connect more USB HDDs, just add more USB root hubs to the thunderbolt4 ports on M1 Mac. The USB hubs that connect to TB4 are root hubs. They are similar to the root hub inside the Mac Mini, which also uses the same popular chip: FL1100LX
Great point. I’ll check it later to figure out the root cause.
It does seem something’s wrong with the parallelism.
I tried chiapos==1.0.3 with chia-blockchain==1.0.7, still had the same problem. PR#239 improves lookup time but it doesn’t seem to help. Add parallel reads to GetFullProof by marcoabreu · Pull Request #239 · Chia-Network/chiapos · GitHub
I’m super-not-trying-ta-be-all-mean-and-shit here but reading all of this (and many other posts) makes me think that the issue, and no one is all that unique in it, here is that most folks are probably screwing with shit way too much.
Meaning- I’d be willing to bet many of us haven’t completed a block at one time or another because we had a node down.
…because WE COULDN’T STOP FUCKING WITH IT.
If you know you’re running good code, know your plots are good, have decent (sub 5sec) proof times, LET IT BE!
I use the same system as you. I have about 90 8TB usb drives over 6k plots.
what do you mean by look up qualities?
I generally have 8-20 pass filter and it takes .6 to 1.3 seconds on average. Is there something else I should be looking for?
I last won 2 weeks ago(maybe less than 5k at the time) but prior to that is was 4 weeks. So far I am WAY below the number I should be getting(currently every 10 days).
Thanks
It’s the time for looking up the proof-of-space. By far, It’s the most famous problem that people have. And I have been wasting too much time in fixing it 
.
You can run this command to check if you had the problem: cat ~/.chia/mainnet/log/debug.log | grep qualities
Here you see what I had this morning in the log files.
Mac-mini chiadog % cat ~/.chia/mainnet/log/debug.log | grep qualities
2021-06-25T10:07:17.947 harvester chia.harvester.harvester: WARNING Looking up qualities on /Volumes/SeaI2/plots/plot-k32-2021-05-30-01-44-af6203600420f2290542e72bc1a68ce400f958b205788ebf5eeec109a0791353.plot took: 14.535951852798462. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.947 harvester chia.harvester.harvester: WARNING Looking up qualities on /Volumes/SeaF2/plots/plot-k32-2021-06-06-04-58-fd24c8b70d2fb6519daf27adbea4ea5049e11d4c7ebf99fb241e0dbe2742d2b3.plot took: 14.536158800125122. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.949 harvester chia.harvester.harvester: WARNING Looking up qualities on /Volumes/SeaB/plots/plot-k32-2021-04-30-00-59-74db5788b30dda720efc2e4c6fb16446e796090c48c36e9f9dc38936738ddd67.plot took: 14.538320779800415. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:17.981 harvester chia.harvester.harvester: WARNING Looking up qualities on /Volumes/SeaH/plots/plot-k32-2021-05-14-18-28-04c71c1200f2a07d09e5bb7bc7a8872e97a240936224d4370cf74c8f9700eeb1.plot took: 14.570097923278809. This should be below 5 seconds to minimize risk of losing rewards.
2021-06-25T10:07:18.233 harvester chia.harvester.harvester: WARNING Looking up qualities on /Volumes/SeaJ1/plots/plot-k32-2021-05-24-19-57-e287d2684348755e0d9e1abbe58fd255d3ec27801b0505d286ea306f78135ac4.plot took: 14.822555780410767. This should be below 5 seconds to minimize risk of losing rewards.
The time looks surprisingly similar between different HDDs. It seems like a bug that could be fixed
I get these as well. I thought it was just a plot taking to long to pass the filter…
When I noticed it, it seems like it always came after a plot reload so did not think anything of it…
So is there a fix for this?
How many occurrences of lookup quality issues exist in your logs?
Out of 609,515 proof attempts in my recent logs, I have 151 occurrences of warnings on lookup qualities. .025% I am not overly concerned about that.
Leaving for the weekend but when I come back ill write a quick script to check. Never really worried about it until now but .25% does not seem to be anything to worry about.
The only thing that might concern me is what EXACTLY does that mean IE is it just the lookup on passing the filter or is it all the ones that went to the next step. If its the later I would be concerned as it is such a low percentage that move to the next step…
Join HPool my friend. I was against it for some time, then I got tired of waiting multiple weeks for a block reward.
And redo 6k plots…pass:) Ill join a pool when I lose to much ground on the netspace
Completely with you there. My top end is about 180 plots/day but the energy cost to do that is not so cheap. Just farming ~5K plots for now isn’t very costly.
What? You don’t have to replot a single plot to move to hpool. You point your solo plots to hpool and only run their harvester.
It’s super fast and rewards are guaranteed.
With 9k plots you’re getting 2.5-3XCH every couple weeks.
I think Littlegrand and I are referring more to the mid-term, at least I am. I’m also not willing to fuss with compromising keys.
…and more to the point, I’m earning more than 2.5-3XCH in that timeframe with 5K plots. So 
Changing the config to stop automatic plots loading fixed the problem for me.
plot_loading_frequency_seconds: 1264000
Cool. Been doing this since the very early days before mainnet. I have consistent rewards now on mainnet for the most part. My longest dry-spell was 17 days.
Interesting. I’ve been thinking of bumping that but statistically the concerning items in my logs are so low % that I haven’t been too worried. But now, of course I have to see if it also fixes my occasional > 1 sec proof times. Here we go again. 
@danarbraz You’re winning with more than 5k plots. So it seems the problem of not winning is due to something else, not the looking up quality? What could it be?