My harvester memory leak

douyacai · December 1, 2022, 5:02am

my harvester in docker now have memory leak, previous created container have no leak, does some have same problem?

Bones · December 1, 2022, 5:51am

My main node had / has one. Clearing some space on the drive and a trim helped a bit.

drhicom · December 1, 2022, 1:10pm

Bones, Whats the size of that C:? And what other stuff you have installed that you really don’t need?

Bones · December 1, 2022, 2:08pm

500gb, we discussed this before in pm / dm, just win 10 ( not disabked any windows features ).
Only chia and old version of farmr installed.
Mem usage still creeps up, starts at about 7 or 8 gb,its currently at 14 to 16gb.
But since freeing up space on c, ive only had 1 slow day of responses ( only 4 over 5 s ) , before at 16gb used it would be most days having slow reponses.

guy.davis · December 1, 2022, 5:59pm

Hmm, maybe an issue with the official Chia docker? I’m not seeing memory increase in Machinaris containers run as harvesters (with the Chia 1.6.1 CLI bundled):

That said, harvester memory usage definitely seems to scale with plot count, due to in-memory caching the Chia CLI does. Hope this helps!

douyacai · December 2, 2022, 9:49pm

i found the reason, when use bladebit with system have a large block size, the plot file will get a large c2 table, when harvester load the plot file, the c2 table will load in memory, it will take so many memory if plots count many.

Fuzeguy · December 2, 2022, 10:55pm

What block size makes a problem? Why is a leak, if it loads and unloads, sounds like shortage of memory, not a leak??

douyacai · December 2, 2022, 11:02pm

github.com/Chia-Network/chiapos

Short-circuit C2 table loading at first encounter of lesser C2 entry

Chia-Network:main ← Chia-Network:c2-read-short-circuit

opened 07:36PM - 15 Sep 22 UTC

harold-b

+12 -1

Short-circuits C2 table loading at the first encounter of an C2 table entry whic…h is lesser than the previous C2 table entry. This prevents `DiskProver` from loading dead-space into memory when reading C2. As there's no specification for the layout of tables in the plot file, but simply a table address in the plot header, plotters may place tables at addresses not immediately following the previous table. Bladebit v1 does this since it writes with direct-io enabled and it does not make any effort to ensure the tables appear immediately after another if the previous table size is unaligned. Therefore the may be empty space between tables given the block size of the filesystem of the directory used to export the plot. See [Bladebit issue #153](https://github.com/Chia-Network/bladebit/issues/153) and [chiapos issue #13084](https://github.com/Chia-Network/chia-blockchain/issues/13084#issuecomment-1248002955) When said filesystem has a large block size, the empty space between tables will end up being quite large, and `DiskProver` will load the empty space following the C2 entries. This prevents that from happening. Because C2 entries are sorted in ascending order, we always expect to load a C2 entry of greater value, therefore if a lesser value is encountered it either means this is a corrupted/bad plot file or potentially there might be empty space, in which case it is safe to stop loading the table.

new chia harvester version , and bladebit 2.0 have no this issues