Plot size vs harvester performance

Chia Farming & Harvesting
1

Given the same finite amount of storage space, will larger plots be less burdensome on the harvester’s efforts than smaller plots?

For example, let’s say you have 5,000 K32 plots.
You could, instead, store approximately 2,500 K33 plots, or…
…approximately 1,250 K34 plots, etc.

Since a single K34 plot gives you just about the same estimated time to win as four K32 plots, would the larger plot sizes, and the resulting fewer number of plots, help your harvester respond to challenges any faster? Would your harvester be any more effecient?

2

One could figure out, if you can get 165 k32 plots on a 18TB disk (16.3tb space) with 41.2GB free, how many K34 can you get on that same drive?

3

Yes! You heard it here.

4

What do you mean by “more efficient?” Do you monitor your lookup times? Do you know your averages, maxes? Potentially (not too sure), if you have bigger plots, there might be less HD hits / lookups (although I don’t understand why this is being told). However, if your averages are not hurting, it really doesn’t matter. I believe you have a script that writes to all HDs every minute or so, and most likely that script will hit your drives more than lookups.

Also, if you are running solo, your lookups are much less frequent than if pooling. Also, some pools have excessive difficulty rates (e.g., Flex) that are causing lookup times to go a bit out of whack.

5

Whatever the harvester’s job entails.

We all know, overall, the purpose of the harvester.
But I have no deeper understanding of its specific, line-by-line duties that its code has it undertake.

If the harvester can get the job done faster, because it is dealing with ½ or ¼ of the number of plots, then that can only help. Or does the harvester now have to spend more time with each plot, due to the plots being larger?

If the harvester is already efficient enough, such that tweaking it (or giving it less to do) will be of no value, then this thread is moot. That is what I would like to ascertain. Do larger plots give the harvester less to do, over a fix amount of space vs smaller plots utilizing the same amount of space?

By the way, I do have “over 5 second” lookup times showing up in my logs. Frankly, I do not know what is causing them. But as of last week, their occurrence in my logs has fallen from many to very few.

It is why I thought that perhaps my harvester was pushed past some limit. But now that I have been replacing K32 plots with K34 plots (now having far fewer plots), my 5-second warnings have mostly stopped happening.

I had not taken time to address the problem, because I was not sure that those warnings were resulting in losses. My wins have been on-par with my estimated time to win… even slightly ahead of the curve.

Would I have won even more if I had no 5-second warnings in my logs? Doubtful (just a guess).

I have wondered if my 5-second warnings were due to the number of plots my harvester was dealing with? It would be more comforting if I had none of those warnings. And it seems that now that I have fewer plots, I have fewer warnings.

6

I guess, this is the answer.

Although, as with everything else, whatever we use, there are limits. Some limits are hard, some soft. Harvester has soft limits (i.e., slow degradation).

What I noticed is that the averages are potentially a factor of how many plots are “eligible for farming”"

harvester chia.harvester.harvester: INFO X plots were eligible for farming

When that number goes up, the averages go up as well, and there are more outliers. If those averages go close to 1 sec (just a guess), it may be that harvester has just too many plots, and maybe it is a time to split it.

As far as those outliers, seeing a one / two over 5 secs I would consider harmless, although if those are present, some kind of averages monitoring should be used. (I share read only my log folders, and use chiaDog to monitor those - nothing installed on Chia boxes)

On the other hand, what I am fighting right now (after switching USB connectors) are outliers in 15 and 50 sec ranges. My feeling is that the 15 secs are when heads are being parked (IDLE_B state), and 50 secs when drives hit STANDBY power state. I used smartctl to keep my drives in IDLE/IDLE_A state, but with those new USB connectors, it looks like from time to time they tend to drop too low in power states. Yesterday, I added another script that writes to drives, and flushes file buffers, and that one drive is not going down anymore. Just added it to a couple of more drives, will see tomorrow.

It may be that this is related not so much to a bug with chia software, but rather a bad lookup handling (although, I am just guessing here). Let’s assume that there are a couple of lookups needed to be done on that drive. If we assume that those two lookups belong to one plot, most likely the software will do the first lookup, let it finish, and proceed to the second lookup on that plot. If, on the other hand, those lookup needs to be done on two plots, then potentially chia will parallelize those lookups, and HDs don’t really like to read from two places at the same time, so there will be a performance hit. Assuming that this is the case, that would justify having bigger plots. Again, I am not sure about it, just guessing.

UPDATE
Sorry, I didn’t get it right. the lookup times / averages go up with the second number on that log line:

X plots were eligible for farming 8492598b17... Found Y proofs

That Y number is really the one that reflects those times, and when it is more than 1, and potentially coming from the same drive, those lookup times become outliers. The X is also telling, but it is easier to see outliers looking at that Y number.

7

Another comment on this topic… Not Chia, but from MMX blockchain FAQ, “For a large farm, it is better to plot higher k size plot to minimize the look-up time for finding proofs at every block height.”

1 Like
8

I think the answer here is maybe, maybe not.
If a proof is submitted too late you lose your chance to win, but not every lookup is also a winning proof so it might not matter. Also, I believe (from memory) that you actually have something like 30 seconds to submit the proof and the 5 sec is just there as a warning to notify you before it becomes a real problem. I guess the 30 sec is the total time to submit the proof to the chain and there is probably more to that than just the lookup time, hence the target of 5 sec for the lookup. --don’t trust me on any of this :sweat_smile:

Personally, I don’t want any of those lookup time warning is my log, It’s a sign that something is slower than it should be imo.

I’ve been running for a week now with my new system (i3-6100, windows 10, 240 TiB on 22 hdd’s, 6 USB, 8 Sas, 8 Sata) Have not seen a single lookup time warning since I removed a faulty USB disk from the mix.

2 Likes