Plot *re-packs* Compression

i think this could happen, has anyone attempt any form of heavy cryptographic compression on a plot? or drive? device…
are there any results of winning with said compressed plot?
i get its mostly random data…
i wonder tho… there’s always room for compression.
i get there is a couple “attack” that may be similar…
thats not what im talking about.
iv personally seen a fully featured game go from over 100gb to 30gb…
why not a plot.

1 Like

They’re already working on it. Bram said something like a 5% reduction.

I hope you can play nice this time around.

2 Likes

Hey buddy
Right he does briefly mentioned it…
with a strange look in his eye.
He mentioned the possibilities of further than 5%

But let’s say 5%
1 petabytes makes 50tb

Compression may be intensive on hardware.
But lead to a further efficient farm…

Inside a plot file I see ALOT of empty spaces… what if I simply went into every plot and deleted every empty space :joy:

Or realistically

laying multiple sets of numbers/symbols/letters/spaces
on top of each other.
Creating a secondary alphabet.
That is read as a whole. By a file interpreter
In simple terms
going from English to German if the characters are shorter/smaller. the same data remained. The same point was made. Just one was signicantly less characters…

Does a harvester go thru every symbol of a plot?

Has anyone tried creating their own “.plot”
just full of random data.
and named a accordingly.

Or modified a plot at all?
When I opened one manually my computer crashed.

This just all came to me. Sorry if I seem over eager.

I don’t think a harvester has time to read individually. a 100gb file x500….

I’m prolly just a mad man:
but gonna give it a real go because my bordem and obsession with chia.
Risk loosing every plot I have I suppose,
Prolly why not many would actually attempt.

I’m here to help and play nice and :thought_balloon: openly.
Just Lamborghini dreamin.

1 Like

man, whatever pills you are taking, they are not green :slight_smile:

3 Likes

The “empty space” is not “empty space” its information that describes the layout of the file. Like the index in a book. Sure, you could delete it, but then if you want to find every page a certain word is on, you would need to read the entire book instead of just looking at the index. This is why checking for a proof only takes a few kb of reads in a 100gb file. If you can read 10% of all your plots in under 10 seconds, then sure, you can delete the index (assuming you also update to code to scan the entire file). This area of computer science, and communication is well understood, Its called “information theory”. We even know how to calculate the maximum amount of compression possible for any piece of data. Check out the movie “the bit player”, it’s about the father of Information theory (and my personal hero) Claude Shannon. I think it’s on amazon.

2 Likes

Haha best one ever! That’s so funny!

1 Like

You can compress right now with your favorite compressing algorithm:)

Serious do you know how long this game from 50gb to 100gb needs to decompress? U want to do that in under 30seconds?

The whole reason chia works as it does is to make any proof of work irrelevant. And yes using computing power to give yourself an advantage is in the end profit of work.

Compression with a minimum overhead on the decompression side will come in the near future as bram has stated in the last ama.

2 Likes

I agree 100%. About Proof of work. Very good point.
But
Taking the “time* to compres a plot~
Assign a little hash stored in log to be read by the harvester only. And only unpacked when it’s 100% necessary when proof is required. Which isnt very often for most people…
than to preform the work of unpacking…

I suppose you may be right
But work was also spent crafting the plot.
Why not a couple more hours just to compress…
Why not include a log on a ssd for the harvester to read
And let my disks spin down
unless needed to be called on?
Maybe No way to secure it?
Somthing like that
could potentially cut chias power down to near nothing globally
Especially for hyper scaled farms.
And still be a great a coin.
With zero footprint.
As other coins are claiming.

Someone needs to offer me a job.

I can only see how
Somthing like this would only further Decentralize and secure.

More plots out there = better for everyone.
Better utilization of space
More efficient time and space
And more
people taking their time to think up the future… they will be wealthiest in the end I believe.

Future

Than what happens when we all move to qbits… and a single plot could be represented with a few bits. This will be Before the move to k33 I believe.
Cause Quantum will deff be a thing in 10 years,
when chias the last crypto standing.

From my understanding there is only one reason really that the Chia team is working on compression. They want to make sure that everyones plot files have the same value. If a big farmer invests into shrinking their files let’s say by 10% using their own proprietary compression they would have 10% more plots for the same investment as everyone else. Chia Inc is basically just trying to level the playground by making sure everyones files are already compressed to the maximum level possible so that no one can have this kind of advantage over other farmers.

The idea that shrinking your files with a compression program provided by Chia Inc. would increase your chances to win because you can have more plots doesn’t make much sense though since pretty much everyone will use it which means everyone will have more of those lottery tickets and since the number of wins per day are pretty much fixed they will win just as many times as they did without compression.

2 Likes

Dr Jones being Dr Jones.

Nurse, medication!!!

2 Likes

It already does this. Each plot has a 1 in 512 chance of passing the plot filter. When a plot does pass this one in 512 chance, it is then (paritally at least) read to see if there are proofs within it. As this sequence of reads takes place, more data is read until a point when either a proof is not found or a proof is found. This whole process takes place every 9-10 seconds (at each and every challenge, assuming your full node is in sync and your connection to the blockchain is good). If you add in mega-compression using fancy algorythms to this process there would not be enough time between challenges to uncompress plots without a massize amount of work. So pointless.

Then bring in a caching question - if you were able to uncompress plots on the fly fast enough, where are you going to store the uncompressed version (or parts of the uncrompessed version) ? RAM ? Temporary fast disk space (SSD/NVME) ?

On the subject of compression, I think 10% plots size footprint reduction will be good going… The plotting process already squeezes down a large amount of non-repeating data into a file of roughly 101.4 GiB. Making that footprint smaller and maintaining performance will not be easy without large amounts of work in realtime. We don’t want to go down the rabbit hole of ‘proof of space and time and work’ with Chia…

So the ‘10% compression’ in my view should not be considered as compression in the traditional sense at all, but just further optimisation of the current plot algorythm to make a file use a slightly smaller footprint but contain the same (or similar) amount of potential proofs.

3 Likes

Further research has led me to
hadoop and hdfs…
compressing at the block level.
Leaving plot file in tact.
Say 1 plot. Broken into 1/4. Split among a few servers. High level of Compression than
replicated.
stored across a few servers…

My research suggests as much as 75% compression capabilities with a single plot.

Also iv read it can compress-decompress on the fly and actually improve seek times with disks

It does all that

But I’m still quite new to this tool, maybe somone with experience can tell me why this wouldn’t work?

This assumes that the content you are storing can actually be compressed. plot files are already dense, Bram Cohen recons they can reduce the plot size max.3%, so my guess is that you won’t see your 75% compression on plot files. You can test this very easily, try to zip a plot file and check the difference in size :wink:

4 Likes

Let me know how your research and testing goes Dr Jones :rofl:

1 Like

This is far far beyond a standard compression. Such as zip.
This would leave .plot extension.
slicing the file up and distributed to other storage servers.
I think similarly to Kubernetes and ceph fs. But more like zfs with its fault tolerance
Even more interesting this applies to Remote or local storage servers…

This is enterprise tech intended for use when hyperscaling.

Combining thousands of servers computation power.
All for just the file system To run.
Wouldn’t even need harvesters just a main node with nsf would probably work.

Again this is all theory at this point don’t know how to implement harvester exactly lol where to put it I mean.
But the technology is there.

Interesting reading here.
But look into compression with hdfs.

Hdfs:

This is not “Far beyond standard compression” This is exactly standard compression. You could combine a trillion servers, it does not change the realities of information theory. Data has a maximum “entropy” where it can not be compressed any further. This has been proven decades ago by Claud Shannon. We know how to measure information entropy, we can calculate the minimum number of bits required to store a piece of information. We know the smallest size a plot can possibly be before we lose information. It’s fundamental law built into the universe, like the speed of light.

It’s possible there is “useless” information in a plot file, but no compression algorithm that is not PoS aware can possibly figure that out. (Not because we are not smart enough, But because it is not possible do to the laws of physics)

EDIT: People are teasing you because you are basically proposing free energy, or faster than light travel, or “atoms are mostly empty space so I bet I can walk through this wall”. Men who stare at goats stuff. Information theory is complex, relatively new, and not generally taught, so it’s ok not to know it. But for anybody who does know even a tiny bit of information theory understands that “My research suggests as much as 75% compression” is not in the realm of sci-fi its in the really of fantasy.

This is your fatal flaw. There is NOT always room for compression. Think of it this way. If you compress a file from 2 bits to 1 bit. How can you decompress it? Because 2 bits have 4 possible values (00, 01, 10, 11) But 1 bit only has 2 values (0, 1) So if you have a 1 bit compress file, it therefor must represent more than one decompressed file. But which one? you would need another bit to tell you. Hence 2 bits can not be compressed into 1 bit.

Compression is all about the probability of a specific value appearing next. The more probable a value is to occur, the fewer bits we can use to encode it. But an unlikely value therefore must consume more bits than it would normally require. If all values have equal probability to appear, no compression is possible. Plots are unpredictable by design.

I can do much better than that: I have an algo that can compress a detailed, high-res, 24-bit color photo of either Anna-Sophie Mutter playing Beethoven’s with the Berlin Philharmonics OR of a beautiful sunset over Coron Bay in the Philippines, down to a mere 1 bit. That’s right, one measly bit!

And it’s completely lossless too, as long as it’s one of those two photos :wink:

1 Like

I really appreciate the responses. Saves me a tone of time and headache. I love probing this place for answers. Chia has attracted a lot of very smart people.
When we’re both rich from chia.

I owe you a schmoke and a pancake,
I’m a -bit of a dreamer if u can’t tell.

I will say tho
The people who change computing forever.
Often called crazy,
Never knew what they were doing.
And They all Did something “different” than other people normally do… often “stupid” things, mistakes.
Ideas
thought to be impossible or just a form of magic.
Yet all there names are eternal at the ends of almost all software we know today.

Typical compression. Is just So…. 1990’s

Theory’s are just peoples constraints on reality that help guid us to a different answer. Because we couldn’t come up with the the answer and moved on. Objective truths.
Water is wet because we say it is.
Information theory will be studied forever.
With never ending breakthroughs.

*I’m sure your all aware. *
Right now there’s some interesting stuff happening. And with it new unexplored lands.
Let’s get some facts straight about the bleeding edge.

  1. Processors that wield light instead of electricity are all ready here. Meaning many many things but relevant to this discussion-
    Data can change states while in process.
    Such as a simple prism…say…(Zipped and unzipped) at light speed.

More astonishing
That path of light (data) can be refracted, split and multiplied. With simple mirrors.and prisms.
AT THE SPEED OF LIGHT.
Do you have any idea the real world throughput of light at a sub atomic lvl? Me niether.
But I’m working on that.

There’s an even a os that will support it.
The Long lost, never dead distro ~~

-Open indiana-

  1. Quantum computing is happening literally right now. Successfully… enough said on that.

  2. Physical computers today are made of literally rare earth metals , crystals, copper and electricity… basically
    (Magic)
    In
    100 years computers will be nothing more than a programmed crystal. Needing a source of light to run.

  3. Time crystals are real. Google has them now.

  4. Qubits held in storage is a thing sense 2013 when my father helped do it.

But we can keep talking like we are in the 90’s I gues… no using crude electricity…. But why

I’ll break it down.

If we look at two bits, they can take the following values:

0, 0
0, 1
1, 0
1, 1

Two qubits take all those values at once.

63 qubits contain as much classical bits as an Exabyte of data.

Forget about a k33.
Computing is changing altogether.

I’m ranting now I’ll stop. When I have my result I’ll update.

But

Look for Dr.jones in that list of eternal names in 50 years.

Sorry, but everything you said here is gibberish.

It is true 63 qubits contain store much classical bits as an Exabyte of data. But! It’s only possible to read 1 bit back out due to Holevo’s bound. “write once read never” storage is useless.

2 Likes