I guess, this is the answer.
Although, as with everything else, whatever we use, there are limits. Some limits are hard, some soft. Harvester has soft limits (i.e., slow degradation).
What I noticed is that the averages are potentially a factor of how many plots are “eligible for farming”"
harvester chia.harvester.harvester: INFO X plots were eligible for farming
When that number goes up, the averages go up as well, and there are more outliers. If those averages go close to 1 sec (just a guess), it may be that harvester has just too many plots, and maybe it is a time to split it.
As far as those outliers, seeing a one / two over 5 secs I would consider harmless, although if those are present, some kind of averages monitoring should be used. (I share read only my log folders, and use chiaDog to monitor those - nothing installed on Chia boxes)
On the other hand, what I am fighting right now (after switching USB connectors) are outliers in 15 and 50 sec ranges. My feeling is that the 15 secs are when heads are being parked (IDLE_B state), and 50 secs when drives hit STANDBY power state. I used smartctl to keep my drives in IDLE/IDLE_A state, but with those new USB connectors, it looks like from time to time they tend to drop too low in power states. Yesterday, I added another script that writes to drives, and flushes file buffers, and that one drive is not going down anymore. Just added it to a couple of more drives, will see tomorrow.
It may be that this is related not so much to a bug with chia software, but rather a bad lookup handling (although, I am just guessing here). Let’s assume that there are a couple of lookups needed to be done on that drive. If we assume that those two lookups belong to one plot, most likely the software will do the first lookup, let it finish, and proceed to the second lookup on that plot. If, on the other hand, those lookup needs to be done on two plots, then potentially chia will parallelize those lookups, and HDs don’t really like to read from two places at the same time, so there will be a performance hit. Assuming that this is the case, that would justify having bigger plots. Again, I am not sure about it, just guessing.
Sorry, I didn’t get it right. the lookup times / averages go up with the second number on that log line:
X plots were eligible for farming 8492598b17... Found Y proofs
That Y number is really the one that reflects those times, and when it is more than 1, and potentially coming from the same drive, those lookup times become outliers. The X is also telling, but it is easier to see outliers looking at that Y number.