Hello,
Create one plot take about 12h, if I use RAMDISK as TEMP dir. If I used SAS 15k it takes about 1h longer.
Is there possibility to speed it UP? Can you hlep me to optimize setup {number of threads; RAM; parallel plotting, delay}? I can use 4 x 600GB SAS 15k disks as TEMP dir too.
I saw that this system shoul be able to generate 40plots per day. I am now on less than 10 { 2 x 5 jobs 12h [1xRAM disk; 4x SAS; for each 6CPUs+16GB RAM ]}
Thank you,
BR
@DigitalSpaceport has done many benchmark videoās on youtube with dual E5-26xx processors.
Hereās one with your E5-2697ās in combination with just 110GB ramdisk and some raid0-'ed ssd-s.
Around 30 minute plottimes I saw at a quick glance.
Maybe you can get an idea of his approach and transfer it to your setup.
HP Z840 Workstation
Dual, 2x Intel Xeon E5-2699 V4 (22C/44T, 2.2/2.4GHz)
128GB RAM DDR4 ECC
Windows 10 Pro
Samsung Evo 970 Plus 2TB NVMe PCIe M.2 SSD
chia_plot.exe -n 110 -r 32 -u 256 -v 128 -t āSSDā -2 āRAMDiskā -d āHDDā -c *** -f ***
2200-2400s
Issue is NUMA so I only use 1CPU.
If I had 256GB RAM, I would run parallel MadMax processes each bound to a specific CPU.
Damn. Something is going on here. Your times should be much better with dual e5-2699 v4ās!!! I have dual e5-2690 v2ās, and with a 110 GiB ramdisk, Iām putting up 26 min plot times! With 2 ramdisks and 2 madmax instances, Iām getting like a plot every 15 or 16 minutes I think.
What RAM do you have?
128GB RAM DDR4 ECC 2400MHz.
Iām using WIN10 too.
Anyhow, Iām OK with my timings, it is the OP who has issues.
I am on old DDR3. I think itās low voltage 1333 that I have clocked up to 1600 MHz. 256 GB.
It is not the RAM issue, it is Win problem. In order to use both of your CPUs, you need to run Linux. Win just cannot handle the second CPU, and all timings go down the drain.
I have nearly finished plotting so am happy with my plotting time. I plan to sell the plotter soon. I have my 360Tib finished within weeks.
Windows suits me fine.
OP wanted help.
@xkredr59 thank you for help; in video is one thing I do not understand, e5-2697v2 have 12 cores so total both cpus have 48 threads so how he can use 64threads? or threads in chia are not mapped to real threads on system?
I forgot mention that I am running on fedora35, where I have BUG and not able to install Mad Max.
I will try today Ubuntu and test:
Are you using for plotting and farming same system?
Thank you,
Jan
No, I use a separate plotter (Intel i7, 128GB RAM, 2TB NVME) and write plots to my farm over the network.
Why donāt you install the stand alone madmax plotter next to the chia software used for farming?
That way you can do all on Fedora.
Thereās a separate install instructions for CentOS7, not sure but I think thatās compatible with Fedora.
Only reason to install chia with the madmax plotter integrated on Ubuntu would be if you want K >32 because stand alone madmax does not support k>32 (currently at least).
Please check https://github.com/madMAx43v3r/chia-plotter for further info and the CentOS install instructions, worth a try i thinkā¦
Copied from the madmax githubā¦
git clone https://github.com/madMAx43v3r/chia-plotter.git
cd chia-plotter
git submodule update --init
sudo yum install epel-release -y
sudo yum install cmake3 libsodium libsodium-static -y
ln /usr/bin/cmake3 /usr/bin/cmake
# Install a package with repository for your system:
# On CentOS, install package centos-release-scl available in CentOS repository:
sudo yum install centos-release-scl -y
# Install the collection:
sudo yum install devtoolset-7 -y
# Start using software collections:
scl enable devtoolset-7 bash
./make_devel.sh
./build/chia_plot --help
I guess you can use more āchiaā threads than real cores/threads available but the OS will work itās head off swapping processes over available cores/threads. That canāt be efficient. I would for a first try use the number of physical cores. When temp storage is fast enough to keep cores busy that max efficiency.
If cores are waiting for disk a lot you could consider higher chia threads to work ahead on calculations. Thereās an optimum very system specific but more than threads on the system I donāt think is wise.
So you mean that I will plotting on CENTOS7 and farming on FEDORA35? If I am right plotter dont need internet so can I have both on VMs on same physical system and only farming VM will have connection to internet?
Thank you,
Jan
Fedora and CentOS are both based on Red Hat Enterprise Linux.
So there is a good posibility you can install the CentOS labeled madmax on Fedora as well, just next to the chia installation.
That would be my first try, if it fails you can indeed install a hypervisor and use a dedicated CentOS VM for plotting, but then you could also use Ubuntu VM.
Chia itself has one installer for both Fedora/CentOS/RHEL, because they are so look-alike on the packages needed, could well be the same for madmax.
Aaaah now I know what you mean. Thank you for help and ideas I will go play 
Thank you,
Jan
Have fun!
I will be a nice solution when working.
Run chia client farming hddās mounted on for instance /mnt/chia/plots1,2,3,ā¦
At the same time and on the same system run madmax stand alone plotting to the same directories, filling your hddās.
That worked fine for me 
Ok first TEST done and I am happy:
on fedora server 35 running centos7 KVM:
./chia_plot -n 1 -r 38 -u 256 -t {250GB RAMDISK} -d {600GB SAS 15k }
Total plot creation time was 1942.55 sec (32.3759 min)
Will try two parallel jobs with TEMP MIX of RAMDISK and one SAS disk; maybe try delay ~15 minutes.
nice!
No experience with dual processor systems and parallel madmax instances but should be fun experimenting for best plots/day config.
How much would you want for the plotter? Iād be interested possibly. Load that bitch up with 512GB of RAM and let it eat!
Run 2 or 3 plotters at the same time,
mount -t tmpfs -o size=110G tmpfs /mnt/ram1
mount -t tmpfs -o size=110G tmpfs /mnt/ram2
mount -t tmpfs -o size=110G tmpfs /mnt/ram3
-r 20 -u 512 -v 128 -t /mnt/nvme -2 /mnt/ram1 -d hdd
Yes, that looks very impressive, indeed! Try running MM, and we can compare notes 
By the way, I didnāt say that Windows doesnāt run on two processors, but that MM will just not work.
In my case, running on a single CPU, I was initially getting around 50 mins plots (under Windows). When I added the second CPU, I killed that run after few hours, as it was getting nowhere. That was on two different motherboards (one SuperMicro, one Dell). After switching to Linux, the first start of two MM instances in parallel gave me 40 mins / plot (basically 20 mins plots per box). @RobbieL811 also used numactl, to further tine tune his box.
