Refresh_batch every 2 minutes

Find out, that my server doing refresh_batch routine every 2 minutes.
Never see that before.
Is that because a new version of the software?

> 2021-11-25T18:22:24.123 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4588, duration: 0.02 seconds
> 2021-11-25T18:22:24.154 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4288, duration: 0.03 seconds
> 2021-11-25T18:22:24.183 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3988, duration: 0.03 seconds
> 2021-11-25T18:22:24.212 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3688, duration: 0.02 seconds
> 2021-11-25T18:22:24.243 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3388, duration: 0.02 seconds
> 2021-11-25T18:22:24.274 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3088, duration: 0.02 seconds
> 2021-11-25T18:22:24.315 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2788, duration: 0.03 seconds
> 2021-11-25T18:22:24.352 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2488, duration: 0.02 seconds
> 2021-11-25T18:22:24.383 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2188, duration: 0.02 seconds
> 2021-11-25T18:22:24.415 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1888, duration: 0.02 seconds
> 2021-11-25T18:22:24.451 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1588, duration: 0.03 seconds
> 2021-11-25T18:22:24.483 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1288, duration: 0.03 seconds
> 2021-11-25T18:22:24.524 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 988, duration: 0.04 seconds
> 2021-11-25T18:22:24.554 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 688, duration: 0.02 seconds
> 2021-11-25T18:22:24.586 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 388, duration: 0.02 seconds
> 2021-11-25T18:22:24.620 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 88, duration: 0.02 seconds
> 2021-11-25T18:22:24.649 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 88, remaining 0, duration: 0.02 seconds
> 2021-11-25T18:22:24.650 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 4888, remaining 0, duration: 0.41 seconds
> 2021-11-25T18:22:30.974 harvester chia.harvester.harvester: INFO     12 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.88443 s. Total 4886 plots
> 2021-11-25T18:22:41.758 harvester chia.harvester.harvester: INFO     6 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 2.49848 s. Total 4886 plots
> 2021-11-25T18:22:50.508 harvester chia.harvester.harvester: INFO     7 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.79332 s. Total 4886 plots
> 2021-11-25T18:22:58.603 harvester chia.harvester.harvester: INFO     13 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.56574 s. Total 4886 plots
> 2021-11-25T18:23:08.144 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 2.11998 s. Total 4886 plots
> 2021-11-25T18:23:16.624 harvester chia.harvester.harvester: INFO     15 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.61348 s. Total 4886 plots
> 2021-11-25T18:23:24.969 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.36788 s. Total 4886 plots
> 2021-11-25T18:23:39.761 harvester chia.harvester.harvester: INFO     7 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 2.02416 s. Total 4886 plots
> 2021-11-25T18:23:45.182 harvester chia.harvester.harvester: INFO     6 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.71882 s. Total 4886 plots
> 2021-11-25T18:23:55.175 harvester chia.harvester.harvester: INFO     10 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 2.64176 s. Total 4886 plots
> 2021-11-25T18:24:03.043 harvester chia.harvester.harvester: INFO     14 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.82913 s. Total 4886 plots
> 2021-11-25T18:24:12.129 harvester chia.harvester.harvester: INFO     7 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.76229 s. Total 4886 plots
> 2021-11-25T18:24:21.263 harvester chia.harvester.harvester: INFO     9 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.86075 s. Total 4886 plots
> 2021-11-25T18:24:26.514 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 4888, duration: 0.00 seconds
> 2021-11-25T18:24:27.321 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4588, duration: 0.02 seconds
> 2021-11-25T18:24:27.358 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4288, duration: 0.02 seconds
> 2021-11-25T18:24:27.390 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3988, duration: 0.02 seconds
> 2021-11-25T18:24:27.421 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3688, duration: 0.02 seconds
> 2021-11-25T18:24:27.453 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3388, duration: 0.02 seconds
> 2021-11-25T18:24:27.485 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3088, duration: 0.02 seconds
> 2021-11-25T18:24:27.521 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2788, duration: 0.03 seconds
> 2021-11-25T18:24:27.553 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2488, duration: 0.03 seconds
> 2021-11-25T18:24:27.584 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2188, duration: 0.03 seconds
> 2021-11-25T18:24:27.617 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1888, duration: 0.03 seconds
> 2021-11-25T18:24:27.656 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1588, duration: 0.02 seconds
> 2021-11-25T18:24:27.686 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1288, duration: 0.02 seconds
> 2021-11-25T18:24:27.717 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 988, duration: 0.02 seconds
> 2021-11-25T18:24:27.762 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 688, duration: 0.04 seconds
> 2021-11-25T18:24:27.795 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 388, duration: 0.02 seconds
> 2021-11-25T18:24:27.828 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 88, duration: 0.02 seconds
> 2021-11-25T18:24:27.900 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 88, remaining 0, duration: 0.06 seconds
> 2021-11-25T18:24:27.902 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 4888, remaining 0, duration: 0.44 seconds
> 2021-11-25T18:24:31.736 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.01826 s. Total 4886 plots
> 2021-11-25T18:24:40.630 harvester chia.harvester.harvester: INFO     15 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.32721 s. Total 4886 plots
> 2021-11-25T18:24:48.808 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 0.88477 s. Total 4886 plots
> 2021-11-25T18:24:58.107 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.09369 s. Total 4886 plots
> 2021-11-25T18:25:06.461 harvester chia.harvester.harvester: INFO     6 plots were eligible for farming 8d74ad7d10... Found 0 proofs. Time: 1.15220 s. Total 4886 plots
> 2021-11-25T18:25:16.000 harvester chia.harvester.harvester: INFO     10 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 1.71158 s. Total 4886 plots
> 2021-11-25T18:25:27.809 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.19041 s. Total 4886 plots
> 2021-11-25T18:25:36.774 harvester chia.harvester.harvester: INFO     9 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 1.97896 s. Total 4886 plots
> 2021-11-25T18:25:45.742 harvester chia.harvester.harvester: INFO     8 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.60462 s. Total 4886 plots
> 2021-11-25T18:25:55.502 harvester chia.harvester.harvester: INFO     10 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.54659 s. Total 4886 plots
> 2021-11-25T18:26:06.602 harvester chia.harvester.harvester: INFO     11 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.75271 s. Total 4886 plots
> 2021-11-25T18:26:15.918 harvester chia.harvester.harvester: INFO     6 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.06146 s. Total 4886 plots
> 2021-11-25T18:26:26.009 harvester chia.harvester.harvester: INFO     11 plots were eligible for farming 4f11b16dec... Found 0 proofs. Time: 2.59028 s. Total 4886 plots
> 2021-11-25T18:26:29.609 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 4888, duration: 0.00 seconds
> 2021-11-25T18:26:30.301 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4588, duration: 0.02 seconds
> 2021-11-25T18:26:30.333 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 4288, duration: 0.02 seconds
> 2021-11-25T18:26:30.364 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3988, duration: 0.02 seconds
> 2021-11-25T18:26:30.398 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3688, duration: 0.02 seconds
> 2021-11-25T18:26:30.429 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3388, duration: 0.02 seconds
> 2021-11-25T18:26:30.460 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 3088, duration: 0.02 seconds
> 2021-11-25T18:26:30.498 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2788, duration: 0.03 seconds
> 2021-11-25T18:26:30.527 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2488, duration: 0.03 seconds
> 2021-11-25T18:26:30.559 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 2188, duration: 0.03 seconds
> 2021-11-25T18:26:30.590 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1888, duration: 0.03 seconds
> 2021-11-25T18:26:30.658 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1588, duration: 0.06 seconds
> 2021-11-25T18:26:30.695 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 1288, duration: 0.02 seconds
> 2021-11-25T18:26:30.738 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 988, duration: 0.03 seconds
> 2021-11-25T18:26:30.775 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 688, duration: 0.02 seconds
> 2021-11-25T18:26:30.807 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 388, duration: 0.02 seconds
> 2021-11-25T18:26:30.837 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 88, duration: 0.02 seconds
> 2021-11-25T18:26:30.862 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 88, remaining 0, duration: 0.01 seconds
> 2021-11-25T18:26:30.866 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 4888, remaining 0, duration: 0.42 seconds

I see the same behaviour, not sure if it’s changed recently. My guess is it’s control in config.yaml by this line:

plot_loading_frequency_seconds: 120

1 Like

just didn’t see that in previous versions
hm… should I increase that? or left it as it is? That’s a question :grin:

1 Like

The whole process takes less than half a second even with your 4888 plots. If it ain’t broke, don’t fix it?

2 Likes

looks like wise solution.
Great!
Thanks.

1 Like

My understanding is that it is an obsolete entry (replaced with four or so new params). It was already discussed in this forum before, so you may want to search for the new syntax.

You can make that value longer (say 10 mins or an hour, or …). The only side effect is that when you add a new plot (e.g., you are still plotting), it will take max that long to have it be discovered. Otherwise, it makes the job of harvester a bit easier (to make it longer).

I mean, when you will use UI to “refresh plots” that entry will be of course ignored, and the scanning process will kick in immediately.

1 Like

Ah, thank you. I really was guessing, although I stand by the if it ain’t broke part :smile:

plots_refresh_parameter
  interval_seconds = 120
  retry_invalid_seconds = 1200
  batch_size = 300
  batch_sleep_milliseconds = 1

Here’s a post from someone who knows what they are talking about:

2 Likes

Sure, but understand that the original line you have there is just ornamental right now. When chia introduced that change, they decided to not write new lines into config.yaml (on updates), and just run by the new default (those six new lines that are just missing). If you remove your config, you will see those new entries in the file, though (or if you do reinstall with killing that config file).

And again, whether that is 120 sec or 3600 sec, neither value is considered to be broken. One gives you faster folder refresh rates, the other less disk traffic. Although, it does matter when you have a big farm (sorry, don’t know what big mean, maybe 1PB, maybe 10PB - really don’t know). For those big farms your refresh times may be longer than that refresh time causing basically a constant disk load.

1 Like