I start plotting compressed plots, and I have some problems
My system
Dell Precision Rack 7910
512 Gb ddr4 2133
2x Xeon 12 cores.
Dell SAS SSD 1.9 TB
RTX 3070 (GPU0)
Quadro K6000 (GPU1)
K33 C8
1- Time to plot, each plot finishes close to 6 min, but the copy time is too long… I need to move manually plot by plot for no stopping.
How I can use multiple destinations (different HDDs)?
2- How I can configure the GPU1 to farm?
I’m in the process of setting up a Dell R720XD to farm compressed plots similar to what you’re doing. I’m waiting on the server to arrive this Monday. Once I have it filled with Plots, I’m going to setup a NetApp DS4246 to put more plots on, which will be a bottleneck like you’re experiencing. The way to fix this, is to use a high speed switch between them with QFSP cables to transfer data around 40Gbps. Then you won’t have this bottleneck I suspect. This guy has a bunch of videos about his CHIA farm & he’s who I’m learning how to setup my farm. Yours sounds like watching his videos would help you:
Multiple Destinations
Multiple destinations can be specified by repeating -d:
cuda_plot_kxx -d /mnt/hdd0/ -d /mnt/hdd1/ -d /mnt/hdd2 …
The same can be done with remote destinations:
cuda_plot_kxx -d @HOST0 -d @HOST1 -d @HOST2 …
It’s also possible to mix local HDDs and remote destinations.
At most one copy is done per HDD, while remote destinations are load-balanced.
It’s possible to unmount and remount new drives onto the same mount points, the plotter will automatically recover and resume operation.
Max also has a little helper program called “Plot sink” that does the job of moving plots to alternating directories / network locations automatically (don’t know how it works though).
Use plotsink, a sufficiently large temp storage drive, and many destination drives so that slow write speed on the destination drives is balanced by a large number of destinations.
I got.
Chia k33 next-gen CUDA plotter - e161e4b
Plot Format: mmx-v2.4
Network Port: 8444 [chia]
No. GPUs: 1
No. Streams: 4
Final Destination: /mnt/7EBEB5C71C8626B7/
Final Destination: /mnt/7EBEB5C71C8626B7/
Final Destination: /mnt/494FAD81601B6257/
Final Destination: /mnt/4D92E78F689EC0DF/
Final Destination: /mnt/250E05FD6E177C57/
Final Destination: /mnt/0E0B6ED90D461856/
Final Destination: /mnt/6D2856250D233986/
Final Destination: /mnt/423B62572E34F4FD/
Final Destination: /mnt/370B33CC7F229AD1/
Thanks
I understand your point of view. It’s great, but I believe you don’t need to spend a lot of money on IT stuff, needs to be simple.
I use a Dell Compellent with 84 drives, and it is connected by a simple mini SAS card in IT mode. I don’t use RAID, for chia this can be a disaster if a drive fails.
I know there are limitations to the transfer rate of the controller, but after setting the multi destinations are working fine.
I used the server only to create plots, to farm I use i3. Now with the compressed plots, I will take a look if I need to replace the i3. But using a server to farm is insane and will overkill your electricity(only if you have a solar panel to be free of costs).
But thanks for the help.
I tried direct setting the destinations at the plot command line and it’s working great.
My system is perfect now. The plot and copy never stopped. The temp folder has 10 plots, 100% of the time between the plot and copy.
Does someone know how to set the GPU1 to farm??
Digital Spaceport uses a hypervisor, Proxmox (best thing since sliced bread). He then uses GPU passthrough to pass the GPUs through to VMs. Each GPU can then be assigned its own workload. The disadvantage of this approach is a loss of GPU performance (speed). Although, I think we are too time obsessed and should be measuring Watt Hours per plot rather than seconds per plot. I have my GPU, an RTX3060 with 12 GB of VRAM passed through to a VM. I plot and farm at the same time because I have sufficient VRAM to do that. My plotting speed is slow because I only have 8 x 16TB disks to plot to. The VM I am using to plot and farm has 187GB Ram assigned to it and I use my own script to move plots to their final destination.
Some other approaches might be to use vgpu (only works with server cards) to assign the Quadro to a LXC container on Proxmox. You could also look at the Nvidia container toolkit.
no matter how fast you are, you always end up with an average 150-220MBs writing speed to final storage. It is about ~10min if I recall. If you must buffer plots, your plot price will skyrocket.
I do <0.05$/plot, which is about 2 months to repay the plot, and 6 years to repay the HDD if you can get one for 15$/TB (>18TB drives). It is an extraordinary dividend rate over 11% annually - very few real companies pay something like that.
If you guys plot fast for > 0.1$/plot, how do you make money? Are you one of those fools that predict XCH prices back to 1500$? As those poor players in BTC that still refuse reality, and dream about 250k per BTC.
XCH price will drop to a level where only most efficient will make money. Make your choice - you can either be popular with fast plots, or wealthy.
