Is your plotting hanging randomly on Linux? Then read this

Which version do you run, the plotters command was introduced only in v1.2.11.

I am running version 1.2.9.

I read that the newer versions were rushed out, and not clean.
Also, there is the mandated password “feature”, that I do not want to deal with.

But thanks for clearing up my madmax question.
I always assumed it was some 3rd party add-on or replacement for Chia. Knowing that it “is” Chia, I am comfortable with using it, if I upgrade.

I just can’t deal with anything potentially going wrong with what I have working. Together with not having too many more plots to create, I will be holding off on making any changes.

Thank you.

I’m using the classic “chia plots create”, but also tried “chia plotters chiapos” (available in 1.2.11) and same results with the SWAP mem leak. I have not tried the new build-in MadMax (available in 1.2.11) as I’ve always ran a standalone build of MadMax on this PC with Pop!_OS/Ubuntu 21.04 without any problems. As for the buffers (-b) I’ve allocated way more than should be needed (46000MiB) when using 64 buckets and I did this because I’ve noticed that some steps allocate only half of that so wanted to be on the safe side. Buffer allocation does not appear to be a problem as I do get an out of mem error if I set this too low from my testing. With the above buffers+buckets and SWAP turned off the plotting of a K34 completes without a problem. I should mention that I assign 6 threads (-r) to the plotter which appear to be only used when writing to the tmp storage while reading is using only 1 regardless of how many threads you assign. You can confirm this by monitoring the plotter threads in htop by adding the IO_READ_RATE & IO_WRITE_RATE columns. You may be correct and this may be unique to Ubuntu 21.04 “better” mem management, usually I try to stick with LTS release but because I wanted to use this machine for gaming as well I went with the cutting edge version. I will try reverting Chia to previous version to see if I hit the same problem as I did come across at least one other forum thread mentioning some plotting issues in 1.2.11.

If anyone is interested in manually tweaking buffers & buckets during plotting I found this very informative article a while back which explains everything very well.

You are correct, I feel the same way and 1.2.10/11 feels slightly buggy to me. For example I’m still not able to get Chia CLI to read the new passphrase from a file so I had to disable this new functionality on my farmer/harvester which I created a service for in Ubuntu so that it auto starts on boot. Was going to start a new thread for this but did not get around to it and it was not on top of my priority list as my farmer has other protections in place and farms everything to a cold wallet anyway :wink:

The additional plotters like Madmax available in 1.2.11 are 3rd party and were included due to popular demand with permission from their developers. You do not have to use them but they are now available in Chia as a lot of ppl used them anyway.

1 Like

Thanks for the clarification.

I trust that, since the developers include the 3rd party Madmax code with the general installation, the developers reviewed the code to ensure compatibility and the absence of “mnemonics” phoning home.

As promised I plotted another k.34 and observed memory/swap usage during the phases, here are my findings.

For the record, Ubuntu 20.04 LTS, i7 8c/16t cpu, 128GB RAM, 2TB NVME as temp storage. Only running chia harvester and system monitor ‘steady state’ memory usage is appr. 2GiB, swap is 980MiB. I have chia version 1.2.11 installed.

Starting chia plotters madmax -k 34 -n 1 -r 12 -t -d -c -f with appropriate parameters memory usage is around 20GiB for phases 1 and 2, swap around 1.2MiB.
In phase 3 console output gives 2 subphases for the 7 tables to be processed.
During subphase 1 memory is around 20GiB, climbing to around 40GiB during subphase 2 for all tables (i really only observed tables 4 to 7 but assume similar behaviour for 1 to 3…). Swap doesn’t move a muscle here.
So in phase 3 it’s memory usage/release/usage/release/… (7 times)
If the memory release is not properly working (OS driven I think) swap usage could be explained and causing hanging further processing?!?

Maybe not so easy given your dual boot config, but I’m curious whether switching to Ubuntu 20.04 (or PoP!OS 20.04 if you want the nicer desktop) will solve the issue for you.

Now for some other mystery, my first k.34 plot on the same machine, same cli command took some 5.5 hours. This second one took a little under 4!!! Faster times for all phases.
Did they implement machine learning/AI also?

Hmm my tests were done using old fashion slow plotter and not Madmax as I though it needed much bigger RAM temp storage than the 110G I’ve been using with K32 so will have to give this a try. When using stand alone Madmax during K32 plotting I did see some little SWAP utilization but assumed it was expected due to other things I had running like Firefox. The SWAP just blew up when I tried plotting K34 with the legacy plotter.

What are you using for tmp1 & 2 when plotting K34 with Madmax? Is 110G RAM drive enough and it’s just using more of tmp on NVME?

No, madmax for K.33 and K.34 requires much more than 110GB, so both temp and temp2 on my NVME.
If you ommit the -2 parameter temp2 defaults to the location given by -t, at least the stand alone madmax does.
To be sure I use

chia plotters madmax -k 34 -n 1 -r 12 -t /mnt/chiaplots/ -2 /mnt/chiaplots/ -d /mnt/chiaplots/temp1/ -c xch1cxwykg5c7rqaxh2ne604r2hv93xxxxxxxxxxxxxxxxxxxxxxx -f b4db259690463b0f7849b039cbe0c9e3fbcdb8b9a32bb4e99703ac3581615082ayyyyyyyyyyyyyyyyyyyyy
Also my destination directory was on the NVME for this test.

Suppose you could use the new plotters madmax first on your current linux install. If this works out ok could you come back with plot times, just curious how my i7 is doing against your 5950x;-)

O yes, a single K.34 adds some 11TB to the TBW’s on your temp NVME or SSD!

That’s good news, I did not think of trying the new madmax K33/34 plotting with NVME tmp as I just assumed it needed a RAM drive. Will definitely give this a try once my legacy plotter is done with current K34 plot which took only 100286.848 seconds last time around (using -r 6 -u 64 -b 57000). Will be interesting to see what I can get for K34 as standalone madmax would take ~21min for K32 on this system.

24.685,7143 seconds I would guess, based on my 36 minutes k.32 and close to 4 hours for k.34.
Give or take a few :wink: But lots and lots faster than 100286.848 for sure! Those AMD cores are gonna want some air…

Actually I’m surprised how cold these Zen3 cores are. I’m coming from i7-3930K which was my main daily workhorse till I replaced it with this 5950X. I’m water cooling it with a custom loop along my GTX1080 that I used with the previous Intel build and this AMD CPU is staying at low 60c at full load! I considered getting TR but did not want another space heater lol

I may want to come back to my 24k seconds estimate for your madmax k.34 plot… Guessing you have a Gen4 NVME drive installed which will be considerably faster than my Gen3 Firecuda it could well be below my initial estimate. Let us know!
(You need over 1TB of free space for the temp directory but that speaks)

I’m using Gen3 CS3030 2T NVMe for temp and not sure it will be <1h for K34 when not using a RAM drive but we will find out.

My calculations were of terribly before, but since my 8core/16threads does close to 4 hours I would expect a 16c/32t 5950x at roughly the same clockspeed combined with gen3 nvme somewhere between 2 and 3 hours for k.34. We’ll see;-)

I just finished testing few runs of K33 and K34 with the build-in MadMax and the results are below. The CPU was snoozing at ~50% avg load so the limiting factor here is the tmp space for sure. I did fstrim in between each test and left the SWAP on during 1st run only while disabling it for remaining tests. During fist run SWAP filled to ~1.5G during P1 and stayed there for remainder of plotting. I used the Gen3 CS3030 2T NVMe with Ext4 (no tweaks) for tmp. This is still A LOT faster than the classic plotter :slightly_smiling_face:

Run 1: -k 33 -r 16 -u 256 (swapon); 5804.86 sec (96.7477 min)
Run 2: -k 33 -r 16 -u 256; 5693.85 sec (94.8975 min)
Run 3: -k 33 -r 32 -u 256; 6917.06 sec (115.284 min)
Run 4: -k 33 -r 16 -u 128; 5898.85 sec (98.3142 min)
Run 5: -k 33 -r 16 -v 128; 6014.67 sec (100.244 min)
Run 6: -k 34 -r 16 -u 256; 13574.3 sec (226.238 min)

Nice :+1:

And no hanging at stage 3!

Indeed it must be the NVME as the bottle neck, my i7-9700 is just slighty slower (14524s for k.34) in my setup but no match for the 5950x.

I looked at that brand/model you mentioned earlier and almost clicked on the buy button because of its very attractive price but checked some more to find out it’s not the best choice for plotting chia. Sustained writes are not it’s strong point…

Yea the sustained writes are not the best on this drive for Chia which they don’t tell you on the spec sheet and I think my Samsung Evo 970 Plus 2T and Sabrent Rocket 2T (SB-ROCKET-2TB) are better on sustained writes. I got the CS3030 after I created a lot of plots with my Samsung and Sabrent drives already and because of much higher TBW which PNY quietly changed since I purchased 2 of these drives! I specifically looked at the TBW & MTBF on their spec sheet before purchase :unamused:. I will stay away from PNY in the future just because they did this.

FYI, I had reliability issues with Sabrent drive. I had to RMA the first one because I found out that a large block of last cells was bad. This came up when I filled the drive past 90% and started to get errors and system hangs. SMART on the drive was not trigged :pensive: by this issue and the way I confirmed that a block of cells was bad was with a special tool that ran a check on cells via the onboard controller. Their RMA was easy and quick though but don’t trust SMART report on their drives.

Have you considered setting up a raid-0 with the samsung and sabrent drives?
I understand you’re finished with large scale plotting but want to optimize disk usage with some k.34’s / k33’s, also for fun? Don’t know the term OCD to be honest…
Using raid-0 could be part of that fun;-)
Both drives will share the writes so maybe there is enough live left in them to plot the higher k’s for you somewhat faster again.

OCD = obsessive-compulsive disorder :grinning_face_with_smiling_eyes:

Funny that you mentioned RAID. I originally used Samsung for OS + plotting and Sabrent was dedicated to plotting. I left Samsung with Win OS on it so was not planning on using this for plotting anymore especially because I switched to plotting on RAM when MadMax came out, but before MadMax I got 2 of these CS3030s and this adapter which allowed me to set them up in software RAID0 and the performance was decent but other drives with better sustained writes would likely do better. Also, to use this adapter your motherboard has to support bifurcation. With 128G of RAM I no longer had the need to use multiple NVMe drives for plotting so I repurposed them.

Also, the reason I wanted 2 same drives for RAID is because drive space between models/manufacturers is different:
Sabrent: 2048408248320 bytes
Samsung: 2000398934016 bytes
PNY: 2000398934016 bytes