2 x e5-2697v2; 384GB DDR3; TEMP DIR RAMDISK =>slow plotting; one plot more than 12h

Hello,

Create one plot take about 12h, if I use RAMDISK as TEMP dir. If I used SAS 15k it takes about 1h longer.
Is there possibility to speed it UP? Can you hlep me to optimize setup {number of threads; RAM; parallel plotting, delay}? I can use 4 x 600GB SAS 15k disks as TEMP dir too.

I saw that this system shoul be able to generate 40plots per day. I am now on less than 10 { 2 x 5 jobs 12h [1xRAM disk; 4x SAS; for each 6CPUs+16GB RAM ]}

Thank you,

BR

@DigitalSpaceport has done many benchmark video’s on youtube with dual E5-26xx processors.
Here’s one with your E5-2697’s in combination with just 110GB ramdisk and some raid0-'ed ssd-s.
Around 30 minute plottimes I saw at a quick glance.
Maybe you can get an idea of his approach and transfer it to your setup.

https://www.youtube.com/watch?v=eSP8sLgGmis

1 Like

HP Z840 Workstation
Dual, 2x Intel Xeon E5-2699 V4 (22C/44T, 2.2/2.4GHz)
128GB RAM DDR4 ECC
Windows 10 Pro
Samsung Evo 970 Plus 2TB NVMe PCIe M.2 SSD

chia_plot.exe -n 110 -r 32 -u 256 -v 128 -t “SSD” -2 “RAMDisk” -d “HDD” -c *** -f ***

2200-2400s

Issue is NUMA so I only use 1CPU.
If I had 256GB RAM, I would run parallel MadMax processes each bound to a specific CPU.

Damn. Something is going on here. Your times should be much better with dual e5-2699 v4’s!!! I have dual e5-2690 v2’s, and with a 110 GiB ramdisk, I’m putting up 26 min plot times! With 2 ramdisks and 2 madmax instances, I’m getting like a plot every 15 or 16 minutes I think.

What RAM do you have?

128GB RAM DDR4 ECC 2400MHz.

I’m using WIN10 too.

Anyhow, I’m OK with my timings, it is the OP who has issues.

I am on old DDR3. I think it’s low voltage 1333 that I have clocked up to 1600 MHz. 256 GB.

It is not the RAM issue, it is Win problem. In order to use both of your CPUs, you need to run Linux. Win just cannot handle the second CPU, and all timings go down the drain.

I have nearly finished plotting so am happy with my plotting time. I plan to sell the plotter soon. I have my 360Tib finished within weeks.

Windows suits me fine.

OP wanted help.

@xkredr59 thank you for help; in video is one thing I do not understand, e5-2697v2 have 12 cores so total both cpus have 48 threads so how he can use 64threads? or threads in chia are not mapped to real threads on system?

I forgot mention that I am running on fedora35, where I have BUG and not able to install Mad Max.

I will try today Ubuntu and test:

  1. two parallel jobs: 20threads; 60GB ram; 1st temp 2xSAS 600GB 15k RAID0 2nd temp 110GB RAMDISK
  2. one job: 40threads; 110GB ram; only one temp RAMDISK 250GB

Are you using for plotting and farming same system?

Thank you,

Jan

No, I use a separate plotter (Intel i7, 128GB RAM, 2TB NVME) and write plots to my farm over the network.

Why don’t you install the stand alone madmax plotter next to the chia software used for farming?
That way you can do all on Fedora.
There’s a separate install instructions for CentOS7, not sure but I think that’s compatible with Fedora.

Only reason to install chia with the madmax plotter integrated on Ubuntu would be if you want K >32 because stand alone madmax does not support k>32 (currently at least).

Please check https://github.com/madMAx43v3r/chia-plotter for further info and the CentOS install instructions, worth a try i think…

Copied from the madmax github…

git clone https://github.com/madMAx43v3r/chia-plotter.git
cd chia-plotter

git submodule update --init
sudo yum install epel-release -y
sudo yum install cmake3 libsodium libsodium-static -y
ln /usr/bin/cmake3 /usr/bin/cmake
# Install a package with repository for your system:
# On CentOS, install package centos-release-scl available in CentOS repository:
sudo yum install centos-release-scl -y
# Install the collection:
sudo yum install devtoolset-7 -y
# Start using software collections:
scl enable devtoolset-7 bash
./make_devel.sh
./build/chia_plot --help

I guess you can use more ‘chia’ threads than real cores/threads available but the OS will work it’s head off swapping processes over available cores/threads. That can’t be efficient. I would for a first try use the number of physical cores. When temp storage is fast enough to keep cores busy that max efficiency.
If cores are waiting for disk a lot you could consider higher chia threads to work ahead on calculations. There’s an optimum very system specific but more than threads on the system I don’t think is wise.

@xkredr59

So you mean that I will plotting on CENTOS7 and farming on FEDORA35? If I am right plotter dont need internet so can I have both on VMs on same physical system and only farming VM will have connection to internet?

Thank you,

Jan

Fedora and CentOS are both based on Red Hat Enterprise Linux.
So there is a good posibility you can install the CentOS labeled madmax on Fedora as well, just next to the chia installation.
That would be my first try, if it fails you can indeed install a hypervisor and use a dedicated CentOS VM for plotting, but then you could also use Ubuntu VM.
Chia itself has one installer for both Fedora/CentOS/RHEL, because they are so look-alike on the packages needed, could well be the same for madmax.

Aaaah now I know what you mean. Thank you for help and ideas I will go play :sunglasses:

Thank you,

Jan

Have fun!
I will be a nice solution when working.
Run chia client farming hdd’s mounted on for instance /mnt/chia/plots1,2,3,…
At the same time and on the same system run madmax stand alone plotting to the same directories, filling your hdd’s.
That worked fine for me :grinning:

Ok first TEST done and I am happy:

on fedora server 35 running centos7 KVM:

./chia_plot -n 1 -r 38 -u 256 -t {250GB RAMDISK} -d {600GB SAS 15k }

Total plot creation time was 1942.55 sec (32.3759 min)

Will try two parallel jobs with TEMP MIX of RAMDISK and one SAS disk; maybe try delay ~15 minutes.

:+1: nice!

No experience with dual processor systems and parallel madmax instances but should be fun experimenting for best plots/day config.

How much would you want for the plotter? I’d be interested possibly. Load that bitch up with 512GB of RAM and let it eat!

Run 2 or 3 plotters at the same time,
mount -t tmpfs -o size=110G tmpfs /mnt/ram1
mount -t tmpfs -o size=110G tmpfs /mnt/ram2
mount -t tmpfs -o size=110G tmpfs /mnt/ram3
-r 20 -u 512 -v 128 -t /mnt/nvme -2 /mnt/ram1 -d hdd

Yes, that looks very impressive, indeed! Try running MM, and we can compare notes :slight_smile: By the way, I didn’t say that Windows doesn’t run on two processors, but that MM will just not work.

In my case, running on a single CPU, I was initially getting around 50 mins plots (under Windows). When I added the second CPU, I killed that run after few hours, as it was getting nowhere. That was on two different motherboards (one SuperMicro, one Dell). After switching to Linux, the first start of two MM instances in parallel gave me 40 mins / plot (basically 20 mins plots per box). @RobbieL811 also used numactl, to further tine tune his box.