Currently Chia scan our plots every few seconds. Won’t it be better if they scan after few minutes instead of seconds to improve power efficiency and life of HDDs?
chia generate a block and reward in a few second, so need scan plot, it cant change.
As @douyacai said, there is a challenge about every 10 seconds (or less). But I don’t think the system actually scans through every plot for every challenge. I’m pretty sure it just checks the database to see if any of them pass the filter. Then if there are some that pass, the system checks to make sure the plot file is there.
Also, the farmer (or harvester, not sure which) does scan the drives that belong to it to see if all the plots are still there and if there are any new ones. It does that on an interval. I think 5 minutes or something like that.
But even if it did check the plots every time. That is no big deal. Drives are made to store and deliver data. It is no different than a corporate setup where the server is handing out data to 100’s of users. This is light work. Don’t get caught up in the “Chia destroys hard drives” BS that is out there. That is all it is, pure BS from people that just don’t know what they are talking about.
Chia scans just a small subset of plots every 10s which is good and bad at the same time as depending on each drive size they may have plots scanned every few minutes or longer. This creates a problems as typically the drive will start to spin down in to a deep standby mode after ~1-2min which parks the head. To extend the life of the HDD/spinners you want to keep them spinning and do not let it stop and start often as that is putting strain on the drive and will make it fail at som point as you are putting much more tear and ware on the mechanical components. I’ve seen drives which fail under warranty end period and drives that keep going after many years pass the warranty in a 24/7 system.
I think with almost every 2 seconds seeks by Chia, drives can never go to sleep/park state.
As it was already stated, chia doesn’t check every plot every 10 seconds. The 1:512 filter is being used, so if you have just one plot per disk, it will be scanned every 10s * 512 - 5120 seconds. You can do the math based on the number of plots you have per drive. Still, that is an average, so sometimes your drive will get a bit more hits, sometimes less (thus can still go to sleep).
You can check your logs for “Looking up” pattern (by harvester), as those lines will represent lookups that are longer than 5 secs.
Also, it looks that chia doesn’t parallelize lookups, so the more plots you have per harvester, the longer average lookups you will have.
Can anyone share the workload on the drives when it comes to pooling vs. self-pooling? I recall reading something but I’ve been on the sidelines for quite some time.
I’m pretty sure there is no difference (in drive activity). My understanding is that the farmer/harvester still does the same work, the node just reports back to the pool about the partial proofs so the pool can keep track of the farm size. The only other difference is on the reward end. The 1.75 XCH is given to the pool wallet and the .25 is still given to the farmer. As far as I know, that is it. If I’m wrong, I would love to know.
The difference comes with difficulty assigned to your farm (that also may depend on your pool). The lower the difficulty, the more scanning harvester needs to do.
However, looking at it from HD’s point of view, even with the lowest difficulty the amount of work a HD needs to be doing is basically zilch, so not worth dwelling on it.
There is very little work done on the HDDs themself as I think challenge is checked agains an index first before going down to the actually plot. In my case with >3k plots I had to create a method for keeping the drives spinning to prevent the frequent spin downs due to infrequent utilization which is likely the case with most farms.
Yes, there is very little needs to he done every 10 seconds but there is a sudden mechanical move by HDD heads on millions of drives world wide every 10 seconds. drives may still last for couple of years though but I wonder why it has to be done every 10 seconds. Same number of tokens can be distributed every few minutes instead.
Faster blocks, faster network.
Interesting. I’m going to read up on this matter of drives spinning down and the consequences of that. I have a bit to learn in this regard. I listen to any and all advice and insight.
The plot manager has a cache that stores the 64 reads needed to fetch a proof of space on disk. As far as I know, today, the software only does additional reads for plots that pass the filter and need to generate a proof.
So I think there has been a lot of work done in order to minimize reads necessary. And reads wear a drive far less than writes, esp with on-disk cache, and OS kernel disk cache. More RAM in a harvester is probably a good thing, cause the OS may cache certain disk reads in addition to what the Chia software does.
For me, I never considered scanning plots much of a drag on the drive’s life span. The bigger thing to worry about is your operating environment. Heat will increase the likelihood of failure and can potentially reduce the lifespan a lot more.
I would think that neither disk nor OS caching helps with Chia. The first is in MB, the second in low GB, where drives are in TB.
Also, assuming that we are talking about 20 TB drive that holds 200 k32 plots, a plot (just one) will need to be checked every ~25 seconds. And this is not the check, but as you stated the chia manager cache hit. Depending on difficulty, we can assume that the drive will need to be hit say every 10 (pick a number) such challenges, what implies a disk read every ~250 seconds. Whatever the number we picked to activate those reads that is zilch load on the stepper motor, neither much load on the main board. At the same time, the main spindle doesn’t do anything special as it stays in ACTIVE mode. So, this is really just a rabbit hole to dig in.
So, it may be that some of those clicks are not really from challenges being processed, but rather due to chia folder scans (defaults to 2 mins, I think). If a harvester is not actively collecting new plots, maybe the best option is to change that scan time to once per hour or day, and if needed manually force scans.
It also may be that the disk internal checks are kicking in more often than chia reads.
True. I don’t recall seeing how much reading is required to submit an entire proof, so wasn’t sure if maybe you could get lucky and gain some additional cache for the benefit of the process. I’m also not sure if the reads are consistent for every challenge, so there could be that too.
Unfortunately for me, I have yet to be able to “kick back” and enjoy watching my farm hum along, looking at average disk IO. I’m still in the process of getting it to that sweet spot
I think since 1.3.5 there have been a number of improvements to the plot manager and cache as well. 1.3.6 may get skipped as there’s a 1.4.0 release getting ready. I run some stuff from the main branch and update every day or few, and what few details I catch, many look extremely worthwhile.
When the plot is being (physically) scanned, let say that 1 MB of reads needs to be done (not really consecutive reads, as plot is 100 GB, so this is where clicking happens). One plot has basically billions of hashes, and we assumed 200 plots per drive. How lucky the reading process would need to be to hit the same challenge, and still fit into 500 MB disk cache? On the other hand, OS cache needs to handle all drives, so say100 TB worth of hashes.
I think what is being assumed is that OS can run with just few GB of RAM, but this is where all the memory trashing (using swap) is happening. So, adding a bit more RAM, say to 16-32 GB, fixes that problem on the harvester (no bc db). I don’t think any further gains will be there by adding RAM.
Not sure what you wanted to say here. Maybe this is about:
...harvester:INFO: X plots were eligible for farming 5ae79ac76f... Found Y proofs
X means how many plots chia mem manager had to check (as passed 512 filter), and Y will be related to difficulty, so actually physically scanned plots (AFAIK). Although, maybe physical scans are somewhere in between X and Y.
Kind of the kicker comes when a pool is lowering the difficulty (e.g., Flex), as more “worthless” proofs are discovered per each challenge. It looks to me that this part is not really a well written code, as chia is not parallelizing scans across different drives (also should not parallelize if plots are on the same drive), and this leads to longer lookup times when multiple proofs are found.
My understanding (right now) is that when those lookups are in the range of few seconds, it implies that either harvester is overloaded, or difficulty is just too low. This may lead to people chasing after OS level imaginary problems, where the source is just bad code.