2nd Temporary Directory Experiments

Chia Plotting
61

Good point, I stand corrected :innocent:

62

This works. Empty temp files are created early on -2, but only fleshed out end P3 on -2 eventually reaching 101GB/plot. P4 adds some essential info to full sized -2 files, renames them and your done. Just need to move somewhere, somehow.

1 Like
63
64

So for what you are doing here, I have a few questions:

  1. How many plots same time using this method? Would doing more plots at same time have an issue at all?

  2. P3/4 are same times as using SSDs, so you save the copy time. But what effect does it have for your P1/P2? The idea here is that does your method allow faster parallel plot times by doing this method because of easing up the SSDs from P3/P4 and copy.

  3. We have 8x SSDs and 8x sata HDDs, so would consider to pair 1 SSD (p1/p2) to 1 HDD (p3/p4). Then we would need to know if this gains us speed and how many per SSD plots could we run at once and not lose time, ideally we would run 4 per and roughly see 2 per setup in P1/P2 and 2 in P3/P4.

Appreciate your experiment and would love to hear further your results. Any feedback related to our system, also would like to hear. Using Threadripper 3970x and 256gb ram 3600 mhz, so CPU and Ram is more than enough to handle 32 plots in parallel across the entire system.

65

  1. I’m running 12 plots at a time on 3900x, 64GB, 3x 1TB WD Black sn750, raid0

  2. Yes that is the idea, I’m seeing slightly faster plot times compared to just plotting everything on the ssd’s. But not by much, about 15 minutes in phase 1 and 10 in phase 2. Phase 3 seems to take slightly longer, 5-10 minutes.

  3. I don’t think my setup would work for you. Because I am only running 12 plots with six hdd, 230 minute stagger. So I make sure that only one plot at a time is writing to a hdd.

One problem here is that I don’t have a big enough data set yet and that in both runs there where problems like plots freezing up, skewing the results a bit.
Also the setting for both runs where different. When I was plotting only on the temp drive I had 12 plots running with 35 minutes stagger, max 6 phase 1.
Now I am running 6 jobs(one for each hdd) of 2 plots each with stagger 230 minutes. So I have six plots starting almost at the same time and then 230 minutes later another 6.

But from what I see, it looks like this method could be beneficial if your temp space is constrained. If your ssd’s are not being overloaded(much) than I don’t think you will see any result, or even worse times.
In my case I only have 2.72 GB of temp space, running 12 plots, so a bit tight. As ssd’s slow down when the fill up, in my case this setup looks to help to take the load of the temp drives.

I will leave this running for a while to get consistent results. And then maybe I will run again with the same que setting plotting only to temp drives/ That should get solid data, but ti’s going to take a while.

66

The 980 Pro should perform better than the 970 Evo/Evo Plus at the same capacity and plots. As the 980 Pro are faster at random read/write and at sustained write. The difference shouldn’t be that big, though. This should be the case regardless of PCIE3.0 or PCIE4.0 on the motherboard. The 980 Pro is a faster drive.

The 970 PRO, however, should potentially beat the 980 Pro, because it uses a different kind of NAND-flash, and can keep a higher sustained write speed.

But, plotting is kinda both sequential and sustained, so difficult to know. I have bought two 2TB 980 Pros, to replace one MP600, and a 970 Evo Plus that will be put in another rig. The MP600 have not been performing as expected. It has a high TBW, but the sustained write speed is really poor compared to the Samsung NVMe drives. So I expect a performance boost, and much more plots per day.

67

Each plot takes roughly 270 GB of temp space while being created - running 12 in parallel will set you up to trying to create 3.24GB of temp files at some point during peak workload - your RAID 0 array is right around 3 GB (or a bit less due to overhead) - when that happens ALL the threads stop and start polling the drive for free space every 5mins before continuing… I really think you might want to drop this to 11 (or 10 to have some breathing room).

1 Like
68

290 GB in my experience

69

I got the ‘270’ from the log files, “temp space used: 269.53gb” or whatever it said.

Are you getting 290 from your log files or did you monitor the actual drive usage during a plot?

I’m wondering if your number is more accurate and the one in the log file is more ‘estimate’.

70

I think you are mixing up GiB with GB

The log files are in GiB

269.53 GiB == 289.41 GB

In the old days people spoke of KB, MB, GB etc and the binary base was implied.

  • 1 GB = 1024 MB, 1 MB = 1024 KB, 1 KB = 1024 Bytes .

Then commercial storage vendors got involved and they started using a decimal base to inflate their storage capacity claims. So nowadays

  • 1GB = 1000 MB, 1 MB = 1000 KB, 1 KB = 1000 Bytes,

and the ‘old’ (imho only sensible) way of measuring is now GiB, MiB, KiB etc.

Chia is reporting in the sensible GiB, your SSD vendor is using the commercial GB

1 Like
71

I did not know this - thank you!

72

Latest update on my two plotters. I added a second NVME to both of them and gained absolutely nothing. I guess it is just a matter of CPU at this point. I built these as budget systems and I have hit the limit. No additional memory or drive capacity is going to get me anything. 24-25 plots per day is the max for that processor. I have tried every imaginable combination of settings to try to squeeze out more but to no avail.

Good news. I have a line on an older HP Dual xeon server. Checking on the specs now. Trying to bump up my plotting power for the launch of pools coming soon.

2 Likes
73

Great analyzis, thanks. Could you explain how you got those pivot tables with plots copy time for 2 days? Is this from Swar? Because i can’t find a command for this.

74

It is PSChiaPlotter. A very useful set of powershell scripts.

75

I’m also seeing that it is mostly CPU/system bottleneck. If you have enough nvme drives, there are only very specific use-cases like creating a copy script to other destinations but most of that is also possible to setup with plotmanager. If you don’t have enough nvme, you can just add it as temp drive.

I’m running all my plots with hdd’s as second temp now. I’m seeing 0 slowdown compared to running fully on the nvme ,as only one plot at a time is writing to a hdd. It also save the copy time.
(hdd’s are Seagate Exos 12TB)

So speed wise, not so useful from what I’ve seen. But I recon I’m doing about 500GB less writes per plots on my temp drives. running 30 plots a day that means 15TB less wear on my nvme per day.
That should make those poor sn750’s last a bit longer :sweat_smile:

1 Like
76

So speed wise, not so useful from what I’ve seen. But I recon I’m doing about 500GB less writes per plots on my temp drives. running 30 plots a day that means 15TB less wear on my nvme per day.
That should make those poor sn750’s last a bit longer

It’s about 80GB per 1 plot economy.

1 Like
77

oh wow, that a hell of a lot whole lot less then I though. How did you measure this, produce those logs?
I completely missed that topic.

I just guestimated it by the fact that I saw an average of 40MB/s for 3 hours…so not too accurate.

78

sudo smartctl -a /dev/nvme0n1 | awk '/Data Units Written/{gsub(",","",$4); print $4*512/1024}' OFMT="%3.0f" and run it by cron every minute :sunglasses: (you need smartmontools to be installed).
Or you can run it before and after and just see the difference (in KB).
It’s a part of capturing script. I improved it and post it soon.

1 Like
79

Shouldn’t it be at least 106GB though? I mean the final file has that size, or is that my too simpel logic again?

80

It should be, but SMART says another. I’ll check it again soon.