Advice sought: How plot filters work in the Chia blockchain

As part of my upcoming bachelor’s thesis, which I will submit in about five months, I would like to draw on your combined expertise and experience.
My studies in computer science have led me to a deep engagement with the topic of cryptocurrencies, focusing specifically on Bitcoin and similar digital currencies. So far, I have completed 40 pages of my work, drawing valuable support from the Bitcoin forum.
However, direct inquiries on my part led to misunderstandings and, in some cases, unwanted queries, such as about the necessity of my research or the mistaken assumption that I was planning a Bitcoin hack.

For another significant section of my work, I plan to cover the cryptocurrency CHIA, which presents an interesting alternative in terms of energy consumption.
After an intensive two-week introductory phase and the creation of two plots, I face the challenge of gathering enough material for about 30-40 pages.
So far, I have written about ten pages on the CHIA consensus mechanism, the plotting process, and GPU plotting.

I am currently dealing with the “lottery process” and need more detailed input from you.
So, the situation is 1024 K-32 plots. Uncompressed. If I have understood correctly, we are currently at a plot filter of 512. This will be reduced to 256 soon.
Then my research becomes vague. From 512 plots, on average, one plot should pass the filter. Based on my example of 1024, that would be an average of 2 files (plots) per round (and there are 64 of them).

Since I have already achieved a few hits with my 2 files in the pool, 512 does not really seem to hold up as a reference value. I guess the difficulty level also plays a role. Since I am at 1, I seem not really to be dependent on the 512-filter?

And then I read that a plot must start with 9x zero. But that seems to be a book with seven seals. You really can’t find an explanation here. When I display the header of such a plot file, none of my two files starts with 0. So, for the last two days, I have been exploring the source code trying to find some answers.

It obviously starts with caching. That is, when starting up Chia, C:\Users\CHIA.chia\mainnet\cache\plot_manager_v2.dat is written.
This file contains all the basic plot data. The file is serialized. Export in JSON format then shows:

{
    "/Z/plot-k32-2024-02-10-08-30-bb966065d955d27fb<*****>1e4a447b49d47.plot": {
        "size": 32,
        "memo": "a1f59b2906225c0729890a836efff586<*****>65315f5f9bad52ce9fbcac54657d1ef543053b351d47b8058d918999",
        "id": "bb966065d955d27fb<*****>1e4a447b49d47"
    },

So, unspectacular. I am still trying to figure out what the memo is supposed to represent. Terribly long. Maybe someone here knows more.
This cache file is apparently what the system works against. It is read again and again. So, I assume that the plot file is only checked at startup and then a hash value is calculated from it. If it’s a compressed plot, then it probably has to be completed first.

Then the CHIA system issues a challenge ID that remains the same for 64 rounds/index. I haven’t figured out exactly yet, but this challenge ID is probably matched against the cache file. If, by some calculation, 9 times 0 comes out, the plot is considered for selection.
Because then I see in the monitoring logs a high access on the file that passed. Apparently, the plot is checked again (probably really for existence) and obviously also content-wise. Several MB were read.

If it was successful, milliseconds later I had a “plot-Submitting” in the log with the values:

2024-02-14 19:31:34,879 - mein_neues_log - DEBUG - {
    "payload": {
        "launcher_id": "0xb902b64b<*****>5abe446947932aaaa3a8a",
        "authentication_token": 26<*****>26,
        "proof_of_space": {
            "challenge": "0x59e4a42ed33<*****>8650ec1da4ceca764991c2",
            "pool_public_key": null,
            "pool_contract_puzzle_hash": "0xa3f59b2<*****>e73c4d5",
            "plot_public_key": "0xbb966065d955d27fb<*****>1e4a447b49d47",
            "size": 32,
            "proof": "0xe60b6b39029115e3168a77<*****>d7e9fc20e5cb96313f777fc50dc734"
        },
        "sp_hash": "0x2d3b<*****>bf7c692034046f38dac73723e",
        "end_of_sub_slot": false,
        "harvester_id": "0x2ee4090bab82<*****>c9d1deb4ed4cde"
    },
    "aggregate_signature": "0x77b2844d3e511161e1f46<*****>1a16bbdf38a76ea845fe325aa"
}

So far so good… I can already make a rough plan up to here.
Just the thing with the plot filter. I can’t really get my head around how the pre-selection comes about here.
Even reverse engineering hasn’t given me any answers so far. The code is just too large and complex.

I BELIEVE that the answer can be found in the Harvester_api.py.
chia-blockchain/chia/harvester/harvester_api.py at main · Chia-Network/chia-blockchain · GitHub (async def new_signage_point_harvester)

But I can’t find anything here with a 512 filter.
That was the knowledge I found out in the last few days. Maybe someone knows the whole thing better and can provide assistance.

Thank you very much

A little hint:

I come from Germany and sometimes had to use a translator. It could be that not every word has been translated the way I wanted to convey it.

It would be helpful if you could form simple sentences. Then there would probably be fewer misunderstandings. Thanks again

Hi Mike,

I assume you have read the docs?

If you need more info, its perhaps better to interact with CNI via their discord, I am sure they are able to help you.