I cannot find information about the .plot file format, are there any ressources available beside the official chia github?
I’m trying to understand why it isn’t possible to simply split a .plot file in half for harvesting on two HDD with a bit of space left. I don’t want to plot k33, and I also don’t want to setup a raid or JBOD. I’ve read that some corrupted .plot files can still provide valid solutions, so I’m looking to learn more about the actual file format.
Thanks for reading!
Not an expert myself.
I read that the big plots are intentional.
Apparently the plot is not just a collection of a few billion hashes, it also only works as one big unit and cannot be split into parts. I think that the data in the file is somehow cryptographically linked so that a proof would not work with only part of the file.
The idea is (apparently) to secure the network against ASICs and software-emulations.
In principle it is always possible to emulate a plot with software instead of actually having it on disk. That would simply mean that this emulation-software does the same as the plotter, only in real time.
But that emulator can not just emulate a few hashes, it would have to emulate the whole plot. And that would take about as long as actually creating a plot, much longer than the few seconds the network waits for a proof.
So (apparently) the only way to secure against ASICs and software-emulators is to make the plots big enough. That (apparently) is also the reason why chia has announced that the present plot-size (100gb) will remain usable for a long time unless (so I understand) someone develops an ASIC. If that happens the minimum plot-size will be increased and the ASIC will hopefully become a very hot paperweight.
Thanks, that makes complete sense. I’m wondering if there’s a way to fill drives other than splitting the files between many disks (jbod/raid). Corrupted plots sometimes work maybe because the whole still makes sense but some of the stored calculation is incorrect. If there was a way to simply split a file in two parts and tell the system to look here+there, that would help fill drives to their maximum without juggling with k33 or risking data loss with a disk failure.
I thought of generating a few bigger plots.
So far have not actually tried that.
So with for example 90gb of space left on one drive a single plot of size 190gb might work and fill it.
As I said: I have not tried if the plotsize can be increased in arbitrary steps.
The size is fixed with the size of K, there are calculator to optimize for example here : https://plot-plan.chia.foxypool.io/
Let’s see however if someone here has the knowledge to explain how the plot file works so we can achieve better density another way. There are a lot of smart people on here 
Just a note: The structure of a Burst/Signum plot appears to be more regular compared to a Chia plot and can be plotted much faster using a CPU, so it is possible that Signum might be unable to stop ASICs from completely taking over that cryptocurrency in the future (unless a Signum hard fork happens to prevent ASIC implementations).
Yes there are ways to split plots over disks, the ASIC resistance of Chia does not come from plots being somehow ‘unsplittable’, it comes from the amount of information needed to resolve some arbitrary challenge being larger than what could be feasibly implemented in silicon.
A RAID-0 array is technically ‘splitting a plot across disks’, and is probably the most approachable way to make use of the full space of your disks, but the tradeoff there is that a RAID-0 array is (usually) lost completely if one of the disks in it is lost. There are plenty of other filesystem-level ways too, e.g. Windows storage spaces, LVM volume groups, etc.
A filesystem is ultimately just a low-level interface, on Linux for e.g. you can use this framework to put a filesystem interface over whatever you like:
Thanks for your reply. I’d prefer avoiding raid configuration/lvm/jbod to avoid replotting.
Do you think it’s technically feasible to split the .plot files if the harvester is told where to look for over multiple disks for a whole file? Or if the file is split in a certain way, with the regular harvester software?
Do you think it’s technically feasible to split the .plot files if the harvester is told where to look for over multiple disks for a whole file? Or if the file is split in a certain way, with the regular harvester software?
It’s technically possible. Ultimately the chia application is going to make some system call to get needed data for different parts of the plot, you could provide your own replacement implementation that looks for different parts of the plot on different disks, there might be an even higher level in the chia application where you can plug in your own implementation of seeking parts of the plot file.
If it was me looking to do it, I’d prefer something at the filesystem level, because you don’t have to think about any chia-specific details, and the work has probably already been done for you. You wouldn’t need to replot to move to RAID, but you may need somewhere to hold the plots temporarily, and that might not be feasible.
Here’s a post on this forum where someone tries to solve the same problem on Windows, i.e. making a virtual volume out of the unused parts of already-plotted drives:
That’s really nice, I didn’t know you could do a volume like that. Thanks for sharing.
I’m concerned about disk failure where I would need to replot an entire array if one disk failed (and raid with redundancy goes against the space optimization). There’s also the issue of adding and removing drives which complicates the matter. Ultimately when plotting can be done for cheaper this will become less of an issue and raid would be easier for sure.
I’m concerned about disk failure where I would need to replot an entire array if one disk failed (and raid with redundancy goes against the space optimization). There’s also the issue of adding and removing drives which complicates the matter. Ultimately when plotting can be done for cheaper this will become less of an issue and raid would be easier for sure.
Yeah, same, I haven’t done RAID for Chia, just using individual disk mounts and accepting there is a bit of dead space at the end - the amount of time and possible maintenance overhead to do anything different to what I’m doing currently doesn’t justify it IMO.
Automagically doing it within the plotting/harvesting software would be most efficient !
My brother has used disk manager to shrink all the partitions, and then created a drive out of all the left over space, haven’t got around to trying myself yet.
https://www.chia.net/assets/Chia_Proof_of_Space_Construction_v1.1.pdf (2020-Jul-31)
There is also an older PDF from 2019: https://www.chia.net/assets/proof_of_space.pdf
Conceptual structure of a plot file is also depicted in Chia Network Consensus Explained - Manuals+
Thank you very much, I will look into it
Just be aware that once you activate windows storage-spaces and connect disks to it, it can be very difficult to change things, or remove a disk from the setup etc… I have had a pair of SSDs which I wanted to make into one larger drive, then wanted to revert back to using them as single drives, removing the storage-space was quite difficult.
I’ve reverted drives quite easily in the past by simply destroying the content. What was the problem with your storage-space array ?
I formatted a drive or changed a partition or something, and then windows partition manager locked up whenever either of the drives were attached. In the end I think I low-level formatted them and they started working again. I have 13 x 76gb, 24 x 38gb and 24 x 19gb portions of free space across my drives which is 3.88GB, but the idea windows might manage some sort of distributed storage system over this many drives fills me with terror.
That’s why I was looking at ways to simply split the plot file across drives, much more simple!
