In this post, I am going to mainly discuss the plotter and harvester I developed, why the Chia community is in a mess and the new plot format proposal from the Chia team is on the wrong track again, why at least a third of the prefarm should be used to set up an open contribution fund, and my conditions on disclosing my new plot format that is energy-efficient and economically infeasible to do any compression by computation.
My work on Chia plotter and harvester
When it became clear early last year that the Chia team was not going to make the harvester of the new compressed plots more efficient, I decided to do it myself. After diving into it, I quickly realized that the rabbit hole of compression has no bottom, especially after NoSSD releasing the ~50 GiB format which presumably saves 2 bits of each x integer, because below that, computational costs will be inversely proportional to plot size (saving one bit for each x would require ~32GiB and saving one bit for a pair would require only ~16GiB) and therefore proportional to farming reward. So, if a farmer with a total capital of 1 spends c on computation and 1 - c on hard drives, his farming reward will be proportional to
c * (1 - c),
which is maximized when c = 0.5. So, it would be optimal for him to spend half on computation. The benefit of computation can be even higher because other compression techniques would be possible because he is no longer constrained by the requirement to save any of the back pointer tables. So, it is likely that almost half of the farming costs would eventually simply be electricity burnt, making Chia no greener than Bitcoin, given the complexity and additional management costs for farming Chia. The narrative of Chia being green was already a dead man walking.
So, it became clear to me that a hard fork would be required and I spent a lot of time thinking about how to achieve that. However, at that time, the Chia team was still reluctant to do a hard fork. So, I went on to develop a 32 GiB compression format to show a proof of concept that the compression size has no bottom. Just when I finished the plotter that worked as expected, Dr. Plotter came out and proved my point. It could already lead to the kill of the green narrative of Chia. It only takes some dedication of someone to push out even smaller plots to make it apparent to everyone.
Eventually, Bram realized how bad things were going and was talking about a hard fork and a new plot format. Unfortunately, he does not seem to have grasped what was really going wrong and was proposing a plot format far from what I consider optimal. So, I went on to further develop the harvester, but not for the 32GiB plot, because it would be irrelevant for the new plot format I have conceived. Compared to the official Bladebit software, What I have achieved (see GitHub - yjiangnan/Bladebit2: An improved version of the Chia Bladebit plotter and harvester ), is a roughly 9X reduction in computational cost for farming compression level 7 with almost all of the dropped proofs (about 5%) recovered, and added further support to compression levels up to 10, which is manageable for a RTX 3070 GPU for a farm with a few Petabytes of space. It is possible to make it more efficient by things such as moving the quality filter into the GPU code and deduplication of entries in the first compressed table, but I haven’t done that yet.
Additionally, I have been developing an in-memory-only plotter that requires less than 128GiB RAM and 3GiB VRAM to make it cheaper for plotting the new format. The code now can compile but does not work yet, but the memories have been mapped out and should work. By the way, reading through the official Bladebit code is painful. Tracing a single variable through almost 20 different names in non-transparent functions is no fun and exhausting, which also likely prevented the Chia team from improving it. I spent a lot of time simplifying and clarifying the code.
Why we are in a mess and the new plot format from the Chia team should not be the way to go
Many people had faith in Chia, which had an innovative technology, a good narrative, and an amazing team, until NoSSD came out to show that the plots could be compressed by computation and failed a negotiation with the Chia team. It was a devastating blow to people’s faith in Chia when the Chia team also failed the negotiation with Mad Max, who openly expressed frustration with the negotiation with Chia. This should raise a red flag for investors since it may indicate the inability of the Chia team to assimilate significant community inputs. What is worse, the official plotter could only produce a plot ~10% larger, and the official harvester dropped a significant amount of proofs, but still used more computational resources, and no further improvement was expected.
Gradually, the negative effects of compression by computation started to bite in. Many farmers are unprofitable and have to shutdown, likely selling their XCH in the process. Additionally, electricity and other costs of farming started to rise, farmers would have to sell more XCH to cover the running costs, which further suppressed the price of XCH, resulting in more XCH required to be sold. This became a vicious cycle. The halving initially boosted the price a bit. But quickly, with the reduction of rewards, only farmers with extreme compressions could survive and most others could no longer cover their running costs. This means that the surviving farmers are using more computational costs, need to sell more of XCH to cover costs, further grinding down XCH price.
To add to the assault, the Chia team hastily decided to increase the plot filter to double everyone’s computational costs in an attempt to discourage people from using high compression. However, the idea of adding to people’s electricity usage to prevent people from using more electricity in computation is just madness, further increasing the selling pressure of XCH from farmers due to higher electricity bill. If we revisit the above simple formula for farmers’ optimal investment in computation, we can clearly see that it is not related to the plot filter, meaning that farmers would optimize to increase computation regardless of the plot filter. The filter could have some small marginal influence on farmer behavior if most farmers only uses moderate compression, but this is no longer the case. Additionally, it would further drive out farmers with the official compressed plots due to its computational inefficiencies, and further lead to the centralization of the network into those closed source nodes. Therefore, it would do a lot of harm without much benefit.
Similarly, we can see why the new official plot format will not solve the most important problems. Based on their published talks, they are again planning to increase the costs of farmers to achieve their presumed resistance to compressions, such as by increasing the number of tables, the complexity of some functions, and costs of plotting. But it would still be possible for compression, just not by a degree of above 50%. However, even if they have successfully developed an algorithm resistant to above 50% compression, a 20% compression would still be very bad for the green narrative of Chia, since there is still a room of 20% for pure computation. As stated above, Chia is more complicated and require more management costs, which in reality also requires resources and energy. So, the greenness that Chia brings would be small. Additionally, their assumed technical resistance to compression could be broken again by new technologies. In fact, we potentially already have a prototype of such new technologies, that is the unified memory of the Apple chip, with a huge memory (128GiB) and slightly larger shared memory per block (64KB) than NVIDIA GPU. Since the primary bottleneck of Chia plotting and farming is the transfer of memory between GPU and RAM, a unified memory architecture will significantly increases the efficiency of compression and decompression. Therefore, we may eventually end up with a few closed source software, again.
Ideally, the new plot format will need to satisfy three criteria:
-
it does not add to the costs of farmers (can be plotted completely in memory with <128GiB RAM and <3GiB VRAM, and be farmed on a low cost hardware),
-
has a standard file size for everyone and is very energy efficient,
-
and it would be economically infeasible to do any compression by computation from the standard plot.
Fortunately, this is possible and I have already conceived one. The Chia team may as well come up with one given the fact that I have already pointed out the direction to go, if they can think outside of their own box.
Why we need to set up an open contribution fund using the prefarm
Ever since the emergence of NoSSD, the Chia team seem to have difficulties in integrating community contributions. To be fair, NoSSD suddenly coming out demanding hundreds of BTC as payment was a bad move, because that would drain the critical cash reserves the Chia team need to get through the bear market. However, the continued failure to negotiate with Mad Max made it clear that the Chia team had problems as well. The fundamental reason for these failures is that there is simply no mechanism established to integrate community contributions. In other words, the tokenomics of Chia was poorly designed, and no rewards have been reserved for contributions from the community. So, there is no protocol both sides could follow to achieve a fair process in the eyes of all. One may point out there is some grants for community contributions. However, these grants are tiny, and tiny grants should and could only buy tiny contributions. Let me put it into perspective. As a crypto quant trader and fund manager, I suffered a loss of millions of dollars of actual and opportunity cost simply because I spent too much time on this Chia project. That is the sacrifice I had to make to get things done here. For other talented contributors, their time could cost even more. Those tiny grants are simply unattractive for any talented contributor to make serious efforts to improve Chia.
On the other hand, the Chia team have been frequently justifying their holding of the large prefarm based on their contributions to the project. However, when it comes to community contribution, the same argument seems to have never come up in their minds. Without the principle of “same contribution, same rewards”, the Chia project would unlikely receive much contribution from a broader community.
Why should the rewards for community contribution come from the prefarm? Well, as we can see now, if they come from farmers, we will end up with a bunch of closed source software, fragmented community, and suffer a heavy blow to investor confidence. However, if they come from the prefarm, rules can be set up, and contributors, the Chia team, and the community can be better coordinated, resulting in a healthy community and ecosystem.
Why is it important to set up a fund for community contribution? Isn’t holding all the prefarm in their own pocket most beneficial for the Chia team? Well, there is a famous Chinese proverb saying that “the sea is big because it is capable of receiving from all rivers”. For Chia to grow, it also requires the Chia team to integrate the contributions from all community contributors. Holding all the prefarm in their own pocket is never going to achieve that. To put into the perspective of investors, it is well known that the most important thing in investing is to invest in the people. There is no doubt that the Chia team are highly talented. However, there is no team on Earth that are comparable to the global talents that are capable to contribute to a project with diverse backgrounds and expertise. Setting up an open contribution fund for potential contributors is simply a very cost-effective way to get all potential global talents on board to deal with future challenges, since the team do not need to pay upfront to hire these diverse people. This will change Chia from a project possible to succeed to a project impossible to fail. That is a huge difference and an enormous boost for investor confidence. The boost in XCH price will more than offset the portion of the prefarm the Chia team need to give up.
So, how much of the prefarm should be used to set up the open contribution fund? To formally answer this question, let us first consider a simple model in which a set of N equally important participants cooperate to make and sell a product with price p. Participant i shares wi from the profit. So, we have
w1 + w2 + … + wn = 1.
Additionally, it is reasonable to assume that the larger wi is, the more participant i will contribute. Critically, the final price p will be proportional to the product (instead of sum) of their contributions. This may not be so intuitive, but is a phenomenon widely seen in biology and economics. One only needs to Google something like “log normal distribution in economy and finance” to see how general this is. Log normal distribution is essentially a distribution in which each contributor has a multiplicative effect to a final output, and price usually follows log normal distribution. So, choosing appropriate unit for p, we have
p = w1 w2 … wn.
So, p is maximized (maximizing the benefit of the whole community) when wi = 1/N, and wi p is maximized (maximizing participant i 's personal gain) when wi = 2/(N+1). So, there is some conflict of interest here. If we consider that the contributions from community and the Chia team are equally important in determining p, then the Chia team will maximize the benefit for the whole Chia community if they allocate half of the prefarm to set up the open contribution fund, and will maximize their personal gains if they only allocate 1/3 of the prefarm for the fund and keep 2/3 in their own pocket. However, such selfish allocation could show a poor public image of the team, erode investor confidence, and have an influence beyond what this simple model assumes. So, they may as well maximize their personal gain by closely assign with the benefit for all. That is to allocate half of the prefarm to the fund. Additionally, if we also consider future investors from the stock market after IPO as an equally important group of contributors and a portion of the prefarm should be reserved for them, allocating 1/3 for the fund will still be optimal for the community.
How should the open contribution fund be spent? By definition, it should be spent on rewarding contributions from the community and should never be paid to any member of the Chia team or anyone closely related. Otherwise, corruption could arise and defy the whole purpose of the fund. A board may be formed from significant XCH holders to decide how much someone should be rewarded by the fund. Core members of the Chia team may participate in the board, and previous fund receivers may also participate. Board members and their relatives should no longer receive any reward from the fund, so they can stand on a ground with no conflict of interests and act purely for the benefit of the community.
How much should a contribution be rewarded? Since it is an open contribution fund, we obviously cannot predetermine how many contributions there will be and we do not want this fund to ever run out. One way to do this is to follow the Bitcoin reward model for miners that halves the reward by time. We can cap the reward for any contribution at 1/3 of the remaining fund. A higher proportion may reduce the reward for future contributors, while lower proportion may underutilize the fund and fail to incentivize recent contributors enough. Although early contributors are rewarded more XCH, the price of it is generally lower, as the XCH price is expected to increase over time if the community continue to build. So, it is fair even in the long run.
What kind of contributions can be rewarded by the fund? Well, this should not be predefined and all contributions should be welcome. However, the majority of the fund should focus on contributions that are difficult to replace, that save the project from crises, and that raise the project to new levels of prosperity. So, the contributions should be judged more on the value of the insight than on the hours of work or lines of code, and should not overlap with the grants and salaries the Chia team already covered. For example, for the new plot format, ideally, I should have disclosed it a long time ago so that the community could avoid costly going down the wrong path. However, without a clear rule for rewarding such insightful ideas, I simply could not do that. I had to spend a lot of time trying to prove its value, make it more believable (e.g., the efficient harvester) without disclosing it, and waited for bad things to play out to allow others to appreciate its value. But deep down, I know this is not optimal for the community. With the new fund, what I hope to achieve is to allow innovative ideas to open up to the community as early as possible without affecting the reward, and further development should be covered by grants and salaries of the Chia company.
Setting up the fund also helps to solve the problem of the concentration of XCH in the hands of the Chia team, which, as the majority XCH holder, is a single party investors would primarily rely on to achieve their expectation of profit for their investment of money on XCH. This makes the argument that XCH is not a security much weaker. However, the open contribution fund, if allocated with a significant portion of the prefarm, will make it clear that a much larger unspecific community (including farmers) will be relied on to realize the expectation of profit, making XCH remote from being a security, clearing doubt from investors, and giving the Chia team more freedom to use the prefarm, especially in a way aligned with community interests.
My conditions on open sourcing my plotter, harvester, and disclosing my new plot format
Given the uncertainties of the negotiations with the Chia team and previous failures in the community, I hope people could understand my caution. I propose a multi-step process to build trust and transparency, which could serve as a positive public relations opportunity for the Chia team and allow the community to witness the progress. Hopefully, the process could go smoothly, things would be done right, and all the future potential contributors could see this as an example to contribute to the project without hesitation. So, I will refuse private chat and closed door discussions. I hope to make everything open.
1. Initial engagement
First, I propose that the Chia team compensate for the time-consuming work I have done related to the plotter and harvester, equivalent to two years of the salary and benefits of a top engineer at the Chia company, as USDT sending to Ethereum ERC20 address 0x4479498b26f70a185e4af855acb9a8d806d072e5 . I do not have a specific number for this compensation, as long as it meets my stated standard. As I said, this should be a PR event for Chia and future contributors will see it as a reference to decide their dedication for contributing.
In return, after the payment, I will release a harvester that uses about 9X less computation for c7 plot. It will not be open source yet, but will be completely free. The plotter and harvester are an extension of the official Bladebit software, so are compatible with the official plots, farmer and full node, to allow the blockchain to be secured by the open source official full node. Compression levels up to 12 also work, but computationally less efficient due to flaws in the official format. A slightly different format is required to solve these inefficiencies. I will also release future improvement for free. This will disable my future attempt to monetize the software, so it shows my sincerity.
2. The open contribution fund
The Chia team will have to use at least 1/3 of the prefarm to set up an open contribution fund, and send the initial third of the fund to multisig address xch1p8f7gpjs04seqs0tgqatdfvuk9xjttz9nu0k9x6w2j9t34uzmznqn602uw as the reward for my following contributions:
-
My efforts to push the Chia team to establish this fund for the long-term prosperity of the Chia ecosystem. I consider this my most important contribution, although the Chia team as a group of a computer engineers may not appreciate so. All my previous arguments may seem simple once I spelled them out, but it took me a decade of work as a biostatistician and as a theoretical biologist solving some of the hardest game theory problems in evolutionary biology to arrive to such insight.
-
My proposal of a new plot format that is highly energy efficient, standardized for everyone (less management cost), and any compression by computation will be economically infeasible. I consider this my second most important contribution, and the Chia team may understand its importance as well.
-
My previous work on the plotter and harvester, including the source code, some interesting techniques, and much clearer code base compared to the official Bladebit code, that are helpful to implement the new plot format with minimal hardware requirement. I consider this my third most important contribution, although the most time-consuming one.
I strongly believe that establishing this fund is essential and a must for the long-term prosperity of the Chia ecosystem. While I have expectations for my reward, I am open to discussions to ensure the arrangement is justifiable and beneficial for all parties involved. For example, if the Chia team believe all the rewards related to getting pass this compression crisis should not exceed a third of the fund, I can agree to allocate some of the fund to reward NoSSD, Mad Max and Dr. Plotter that may still have contributions unsettled with the Chia team. But my reward should be no less than 1/4 for the initial fund, and such arrangement should be well justified, with a good mathematical model and a good target function to maximize to make it more objective. After receiving the reward, I can also serve on the board of the fund and no longer receive reward from it. I spent a decade prioritizing the pursuit of truth instead of economical gains, I worked a lot on game theory in an open ecosystem, I deeply understand market dynamics, and I have managed a 20PB Chia farm. So, I can provide some unique insights and contributions as a board member of the fund for the Chia community.
As a bonus, the Chia team could consider visa sponsorship until getting a US green card for contributors wanting to come to the US.
Given the complexity of this process, especially because of the majority of the prefarm being locked up, it may be helpful to break this step into finer steps. For example, the Chia team could pay me the prefarm that has already been unlocked, if any, in exchange for me to open source the plotter so that the team can immediately work on making it to work with low hardware requirement, which is very helpful for quickly implementing the new plot format.
For the part of the prefarm I control, what I can promise is that the vast majority of it will never be dumped into the market for cashing out. Instead, fund will be raised to combine with it to perform an automatic (therefore without insider information) liquidity-making quant trading with moderate but guaranteed long-term price-adjusted profitability. Only the profit will be taken out. This will give more stability for the price of XCH, and form a guaranteed bottom that the price of XCH is impossible to stay below to reduce risk for investors. The exact details and relevant mathematical proofs will be disclosed when the fund is set up. The part of the prefarm allocated to the Chia team may as well be used similarly to benefit all parties involved.
3. Timelord support for Chia forks
Although initially maybe seen as competition for Chia, now it is clear that Chia forks are very helpful for maintaining the green narrative of Chia, because any compression by computation will be economically costly to farm Chia forks and therefore has less to gain. So, allowing Chia forks to use the new ASIC timelord is a necessary step Chia needs to take. Otherwise, new Chia forks may opt to add reward for timelords to attract the ASIC timelords away from Chia, creating a new mess.
4. Freedom of speech
A prosperous ecosystem requires a huge number of people with diverse backgrounds and points of views. Freedom of speech is a must and should be one of the highest values the Chia team need to hold. Unfortunately, this is not the case. I have personally been kicked out of Keybase by the Chia team three times for arbitrary reasons. Such incidences significantly suppressed my willingness to communicate with the team. If the Chia team can only tolerate yes-men always praising them, then all they are building is just an artificial lake on top of a mountain. Yes, that could be high tech and green and people always need to look up to admire it, but it will never grow beyond the top of the mountain, not to mention becoming as big as the sea. During my academic years, I consider the most important thing I learnt is to be rational and cool when being criticized, which is very difficult but extremely important for both personal growth and prosperity of the scientific community. Responding to criticism by removing people from the discussion is clearly not the most constructive approach. People in the US may have freedom of speech for too long to truly appreciate its importance. But it is extremely important; I can testify from where I live.
So, I would very much like to see a change in the management of the communication channels hosted by the Chia team to show more respect for the freedom of speech. While it may be necessary to remove spamming accounts, decisions for removing members should involve at least two persons, and the person making the final decision should deeply appreciate the importance of the freedom of speech and act conservatively, and should not be the person who first proposed the action to avoid any impulsive decision.
5. What happens if the negotiation fails
I have tried my best to make things reasonable for the negotiation to succeed. However, in case it fails, I will still provide a path for disclosing my new plot format given its importance for a broader community. One way to do this is to get as many people from the community as possible on board to create a new cryptocurrency, with the tokenomics done right based on my best understanding of game theory and community input, and disclose my new plot format as soon as the new cryptocurrency reaches a target market cap, say $100 million, when the stake is too high to use a closed source software.
One the other hand, the Chia team may put in more resource to come up with a similar plot format given the fact that I have already pointed out that it can be done. However, as I have said, this is a PR event for the Chia team. It is never about whether they are capable of solving a technical problem in a defined scope, which is relatively simple if they can think out of the box. The real question is whether they are capable of strongly encouraging and fully integrating community contributions to solve the endless open problems they would face in the future if they really want to build something globally significant. So, even if they could solve this technical problem, solving it by themselves would generally be considered as stealing ideas without respecting originality, therefore would not be good for maintaining a positive public image, attracting contributors, and bolstering investor confidence. Additionally, if they delay too much on rolling out this new format with high energy efficiency, the negative effects of compression by computation will further bite into the price of XCH. So, I hope this would not be the path they decide to go down.
Conclusion
The birth of Chia captivated broad interests from the crypto community, primarily due to its innovation of proof of space. It is energy efficient, has a green narrative, and could be more secure and decentralized than Bitcoin. Despite the Chia team’s emphasis and business plan based on their new secure language Chialisp, the market is unlikely to be very excited by this, because there are already other secure smart contract languages out there, such as Haskell of Cardano and Scrypto of Radix. Additionally, it is unclear how such a business model could in any way help the price of XCH. Although the Chia team’s plan for IPO and plan to follow the lead of SEC might have also given some initial vibe, its long delay and the change of the political landscape (especially the failures of SEC in lawsuits, the new crypto laws passed in the US, the pass of BTC and ETH ETFs, and President Trump’s pro-crypto stance), make the importance of this path less clear, especially if the IPO would result in the eventual selling of the prefarm into the open market, either directly or indirectly, to satisfy IPO investors. The low price of XCH would also lead to further delay of the IPO because it results in a low valuation of the Chia company.
So, saving and strengthening proof of space and its green narrative is the most important thing the Chia team and the community need to do now. It is also likely one of the most important contribution the community can make in recent years. So, giving out the highest possible reward for it from the open contribution fund from the prefarm is justifiable.
The huge prefarm has always been contentious in the community, at least Bitcoin does not have it. Initially, the Chia team promised never to sell the prefarm, but instead planned to simply put it on their balance sheet as an asset to raise fund in a traditional way. If this can be done and the Chia team want to explore this unprecedented path, fine. However, this promise has been broken and obviously is a flawed idea, because investors are not stupid and the prefarm must be able to sell to really count as an asset. Additionally, raising fund by using XCH as a collateral could also easily lead to liquidation due to huge XCH price fluctuation. Therefore, it is necessary to rethink how the prefarm should be used. Having a prefarm by itself is not a problem. Ideally, all contributions should be fairly rewarded for the project to continuously grow. A project without a prefarm would most likely die off since a bunch of farmers spinning some disks clearly are not going to solve the much broader problems for a cryptocurrency to succeed. The same logic can also apply to the Chia team holding the huge prefarm all by themselves. For Chia to succeed, it will need countless talents from all around the world to solve unforeseeable and open challenges in the way. The current closed structure of the prefarm is clearly not helpful.
By setting up an open contribution fund with a significant portion of the prefarm, we can guarantee the continuous incoming of global talents to tackle any problem we will encounter in the future, and show to the world that we are a much better organized community up for any challenge. Did I mention that the additive contributions have multiplicative effects on the price? Then, we can really build a project that has huge growth potential.
Finally, let me re-quote (sorry I really like it) the Chinese proverb that “the sea is big because it is capable of receiving from all rivers”. I hope that the Chia team will choose to become the sea that stands below every river, instead of a lake high above on top of a mountain.
