Nice work!!! big Thanks!
Can You share your Swar config file You used for both machines?
1 Like
Awesome! Thanks for taking the time.
So overall you saw a max allocation of 236 GB on temp1 drive, is this correct?
For a single plot, running by itself, Yes. I haven’t really paid attention since I started the parallel plots but current with 8 in parallel, it runs between 1.1TB and 1.4TB on temp1 and between 150GB and 300GB on temp2.
1 Like
Here is my Swar’s config file. I cleaned it up so it isn’t so large in the forum. I am unsure if it would work like this so just pay attention to the settings. I wouldn’t try to copy/paste it. YAML files are really particular about format.
chia_location: C:\Users\greg\AppData\Local\chia-blockchain\app-1.1.5\resources\app.asar.unpacked\daemon\chia.exe
manager:
detailed logging. Warning: INFO will write a lot of information.
check_interval: 60
log_level: ERROR
log:
folder_path: C:\Users\greg\.chia\mainnet\plotter
view:
check_interval: 60
datetime_format: "%Y-%m-%d %H:%M:%S"
include_seconds_for_phase: false
include_drive_info: true
include_cpu: true
include_ram: true
include_plot_stats: true
notifications:
# DISCORD
notify_discord: false
discord_webhook_url: https://discord.com/api/webhooks/0000000000000000/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# PLAY AUDIO SOUND
notify_sound: false
song: audio.mp3
# PUSHOVER PUSH SERVICE
notify_pushover: false
pushover_user_key: xx
pushover_api_key: xx
# TWILIO
notify_twilio: false
twilio_account_sid: xxxxx
twilio_auth_token: xxxxx
twilio_from_phone: +1234657890
twilio_to_phone: +1234657890
progress:
phase1_line_end: 801
phase2_line_end: 834
phase3_line_end: 2474
phase4_line_end: 2620
phase1_weight: 33.4
phase2_weight: 20.43
phase3_weight: 42.29
phase4_weight: 3.88
global:
max_concurrent: 8
jobs:
- name: gobaby
max_plots: 100
farmer_public_key:
pool_public_key:
temporary_directory: D:\
temporary2_directory: E:\
destination_directory: Z:\
size: 32
bitfield: true
threads: 4
buckets: 128
memory_buffer: 4000
max_concurrent: 8
max_concurrent_with_start_early: 9
stagger_minutes: 30
max_for_phase_1: 3
concurrency_start_early_phase: 4
concurrency_start_early_phase_delay: 0
temporary2_destination_sync: false
1 Like
And I just noticed that it did 24 yesterday. So it is teetering between 23 and 24 per day on the system with the extra drive and 19 on the system without.
1 Like
I’ve been wondering about that as well. I’ll be following this experiment closely.
Thanks for sharing.
Well, the experiment is over for now. Because the next step would be to bump the 2nd temp drive to an NVME. But after seeing how little I/O the current SATA SSD is getting, I don’t see the benefit. So later today, I am modifying those machines to dual NVME systems with a SATA SSD as the second temp dir. That will completely take these to another level and move out of this direct comparison experiment.
But, if you guys are interested in seeing the result of going dual NVME (RAID 0) with 2nd temp, I’ll post up when I get them dialed in.
I will be removing the 970 Evo from one and putting it in the other to make it a dual NVME. Then I will be putting two 980 Pro’s in that one. Both will have SATA SSD’s as the 2nd temp. No memory upgrades, no CPU changes. Just going to push that as hard as I can.
3 Likes
if temp1 is nvme and temp2 is nvme there is no performance gain. even with 36 plots going in parallel I have ZERO I/O wait from start to completion, so there is zero gain by using an nvme drive. Perhaps if temp1 was a SLOW drive like mechanical or even sata ssd (non-nvme), then maybe you’d get some plotting improvement speed by using an nvme drive as your temp2 and dst, but otherwise no and afaik every type of configuration I’ve tried using temp2 only slowed the plots down.
1 Like
I agree with you about using an NVME as the 2nd temp. Not going to do it. But as for the second temp not helping, this entire thread says and proves otherwise. But as I mentioned, this would not work for everyone. I think the reason it helped me is the version of NVME I have. It is not the fastest. It gets saturated and slows down. The 2nd drive helped a lot.
2 Likes
As I can understand, first temp dir is used to make a lots of files, what would be combined to one file in second temp dir and then copied to dst.
So using “slow ssd” as secondary will save TWB for NVME drive, I think. And yes, thanks @WolfGT, it would be lowering saturation.
And here is a question: if destination drive will be used as “-2” dir? If temp file moved by “mv” to destination on same drive without modifying it will incredibly fast. Did anybody tested this?
Did not test that because my final destination is a NAS.
This is really, really excellent stuff. Lots of great data.
Can I propose a third test to put beside the two cases you already have?
What I mean is that you start with system A (baseline) and then you add a 2TB SATA drive… but what’s the optimal way to employ that new drive?
A) …as a -2 Temp Space to enhance your current processes, (like you did) or
B) …as another distinct -t temp space with it’s own parallel processes
I know SATA the can’t keep up with NVME, but if it can squeeze out more than 6 per day, then you might exceed the 23-24 you’re getting with the combination. This assumes you have the RAM and threads to feed the extra processes, (I do, I have lots of unused RAM and threads)
1 Like
Just for note, temporary files a not always 250GB/polt. If make time shift and use another drive as -2 you can make more ))) But it little bit of risky.
But good news. If there are no free temp space, plotting is not crashing, but trying to resume with 5 minutes delay, so you can kill one plot and free more space for others
@WolfGT I see you are using a NAS to store your plots. If I am correct phase 5 is moving the plot to the final destination.
In my case it takes about 30 minutes for phase 5. While phase 5 is not finished new plots will not start.
So I was wondering, could it be beneficial to set a folder on this 2nd temp as final destination in order to fishing phase 5 faster? Then will need to have a script to move the files automatically to the NAS, but at least a next plot has started already.
Is my thinking correct? Could some time gained this way?
Great experiment. 
How much max space is used per plot on the 2nd temp drive? Only 101.4GiB?
I have two 2TB NVMe temp-drives doing 8 plots in parallell each (16 total), and I’m thinking about getting a Sata SSD for 2nd temp. But I don’t know if I should get a 1TB or 2TB drive?
Final destination is 16TB USB drive.
1 Like
@WolfGT Really appreciate you posting all this and the experiments you have been doing.
We have 2 systems, 1 with 2x NVMEs and another threadripper 3970x with 8x NVMEs (Samsung 980 pros 2T). Use the 2x NVME system for testing stuff related to this and the threadripper system with 8x NVMEs for non stop plotting.
As with everyone, going faster is what we are always trying to tweak. I am actually going to test on the testing system using 1 NVME as -1 Temp and the other NVME -2 Temp with the same load we use and see if there are any gains.
But here is what I am thinking though for the main system with 8x ssds, what if used 6 drives for -1 Temp(P1/P2) and 2 of the drives for -2 Temp, so have a drive ratio of 3:1 and see if further gains are made? Any thoughts about this? Or even adding 2x SATA ssds to the system and trying 4 NVMes as -1 Temp into 1 SATA ssd as -2 Temp?
Love this testing!
In reality, there would be no benefit. If phase 5 is only taking up network bandwidth and read traffic on that temp drive you would have the same result if you just added another plot to the parallel set (which is what I have).
The way to think about this
-
(Your way) That plot is done and sitting in the way and needs to be moved. The custom script/program fires up and begins to move the file. That script uses network bandwidth and read from the drive.
-
(The way it is) That plot does exactly the same thing as your way but it does it in phase 5.
Either way, if the system can handle another plot, just add it to the parallel stream.
Just under 107GB per plot max. It only jumps to that right before it starts to copy off. So each plot only sits at that size during phase 5.
1 Like