Nossd plotting mayhem

Chia Plotting
1

Hi there,

My first post here :slight_smile:
I just wanted to share my experience with plotting using ‘nossd’

  • using RTX A2000 6GB I am getting 2min20sec-2min-30sec on average. I also have RTX A2000 12GB and it behaves exactly the same as the 6GB variant same time for plotting.
  • using RTX A4000 (which is the bigger brother of A2000) I am getting 5minutes(!!!) so twice as slow. This is using a graphics card which is supposed to be 50% faster than A2000, and consumes twice more power.
  • using RTX 4060 8GB I am getting around 3min10sec. Still slower than the A2000, while it is supposed to be a faster card… And consumes more power as well.
  • using RTX 4000 8GB I am getting 4min30 sec for C15. Something’s wrong, as I am getting better times with this card, than A4000 which should be faster…

These tests were performed on a Dell NX3230 server (which is similar to R730XD with 12 x 3.5" bays). And yes, I connected the power cable for the GPUs which requires it. The A2000 doesn’t require external power (it draws only 70W from the PCI-E slot directly). The A4000 requires 140W (and uses a 6-pin power cable) and the RTX 4060 requires around 110W (and uses a 8-pin power cable).

Note: I also own Supermicro and HP servers and Dell P720 worksation, HP Z4/Z6 G4 workstation and the numbers are similar to those, with some small differences.

2nd Note: I am using ramdisk as destination and a modified plow script to move the plots off to the hard drives from the ramdisk. So disk I/O is not an issue in any of those scenarios.

Any idea why this weird behavior? I can add screenshots/pictures for those interested in.

BTW, for farming, the numbers look much different, pretty much in line with the expectations (at least, judging from the tests that the nossd script performs).
I can try posting numbers for farming as well, for anyone interested in this.

Best regards,

1 Like
2

a rtx 4060 has a 8x pcie link
a a2000 has a 16x link

look at nvtop and see if the a4000 is running at a weird link speed or is overheating badly.

1 Like
3

a rtx 4060 has a 8x pcie link a a2000 has a 16x link

Irrelevant. The 4060 uses 8x pcie link because it doesn’t need more. And it should still deliver more performance than the A2000 according to all online tests.
And I placed the cards in the exact same PCIE slot, when testing.

look at nvtop and see if the a4000 is running at a weird link speed or is overheating badly.

Maybe, but I really doubt. I have 4 pieces A4000 and around 9 pieces A2000, and I’ve tested them on multiple systems: HP and Lenovo workstations, Supermicro and HP servers, and got very similar results, within error margin. It’s not very likely they use wrong links everywhere and all A4000 cards are overheating…

So I think it’s rather some software issue with the nossd client binary.

I’m curious if others can post their results on creating plots with nossd, what times they got and which systems they used?

4

1060 6GB = 4min 128GB RAM, ssd destination

2 Likes
5

I just tested a Quadro RTX 4000 and got 4min30sec for plotting C15 using ramdisk.
Something’s wrong, as I am getting better times with this card, than A4000 which should be faster…

6

Im sorry you seem to be writting as if you know what you are talking about.

The plotting algo is usually bandwidth starved not core or memory latency / memory bandwidth starved.

Focus on this point before trying to make any further comments. You are wrong.

bUt tHe 4060 dOeSnT nEed a 16x liNk
maybe for gaming or some POW coin. However look at your core, its not pinned 100% of the time. You are wrong.

7

You could be right. The manual for that storage server says “PCI Express Generation 3”

OP you can check this by installing and running nvtop during a plot
You also dropped ramdisk into the mix. There are known issues with plotting to ramdisk so try a raid0 or nvme ssd destination.
For even more confusion it is going to be dual cpu, I can smell it!

8

quadro rtx is pcie 3.0

that sounds about right

pcie 4 8x = pcie 3 16x

9

For the sake of science! I have moved my GPU to x8 slot

NoSSDChiaPool version 2.2
Plotting GPU:
#1 "NVIDIA GeForce GTX 1060 6GB" (ID:A623FDC02936FFB511F7DDC101138415), 4.322GiB of 5.929GiB
Directories summary:
--- "/nvme/plots" ---
(869.22 GiB available)
Total plots: 0
Mineable plots: 0
08:04:18 Allocating memory...
08:04:38 107.96GiB of memory allocated
08:04:38 Start plotting "/nvme/plots/6240...A220.fpt"
08:04:39 Plotting, 0%, 0s elapsed
08:04:48 Plotting, 2%, 9s elapsed, 6m 17s remaining
08:04:58 Plotting, 6%, 19s elapsed, 4m 40s remaining
08:05:08 Plotting, 11%, 29s elapsed, 4m 6s remaining
08:05:18 Plotting, 16%, 39s elapsed, 3m 32s remaining
08:05:28 Plotting, 21%, 49s elapsed, 3m 10s remaining
08:05:38 Plotting, 25%, 59s elapsed, 2m 58s remaining
08:05:48 Plotting, 30%, 1m 9s elapsed, 2m 42s remaining
08:05:58 Plotting, 35%, 1m 19s elapsed, 2m 30s remaining
08:06:08 Plotting, 39%, 1m 29s elapsed, 2m 18s remaining
08:06:18 Plotting, 44%, 1m 39s elapsed, 2m 6s remaining
08:06:28 Plotting, 48%, 1m 49s elapsed, 1m 57s remaining
08:06:38 Plotting, 53%, 1m 59s elapsed, 1m 47s remaining
08:06:48 Plotting, 57%, 2m 9s elapsed, 1m 36s remaining
08:06:58 Plotting, 62%, 2m 19s elapsed, 1m 25s remaining
08:07:08 Plotting, 66%, 2m 29s elapsed, 1m 15s remaining
08:07:18 Plotting, 70%, 2m 39s elapsed, 1m 7s remaining
08:07:28 Plotting, 74%, 2m 49s elapsed, 58s remaining
08:07:38 Plotting, 78%, 2m 59s elapsed, 50s remaining
08:07:48 Plotting, 82%, 3m 9s elapsed, 41s remaining
08:07:58 Plotting, 86%, 3m 19s elapsed, 33s remaining
08:08:08 Plotting, 89%, 3m 29s elapsed, 26s remaining
08:08:18 Plotting, 92%, 3m 39s elapsed, 18s remaining
08:08:28 Plotting, 95%, 3m 49s elapsed, 11s remaining
08:08:37 Plotting, 98%, 3m 58s elapsed, 6s remaining
08:08:47 Plotting, 99%, 4m 8s elapsed, 3s remaining
08:08:57 Plotting, 99%, 4m 18s elapsed, 1s remaining
08:09:07 Plot done "/nvme/plots/6240...A220.fpt"
08:09:08 Deallocating memory...
08:09:15 Memory deallocated

Note : At the end you can see my SSD presumably can’t keep up with the output. The “remaining” is going down by less than 10s per 10s. Maybe this is the lack of bandwidth moving the plot from ram through the gpu to disk, showing?

This is PCIe 3 x8 (physical x8 slot)

10

This is PCIe 2 x4 (physical x8 but nvtop shows x4 throughout)

08:32:54 Plotting, 0%, 3s elapsed, 16m 22s remaining
08:33:04 Plotting, 1%, 13s elapsed, 18m 4s remaining
08:33:14 Plotting, 2%, 23s elapsed, 18m 11s remaining
08:33:24 Plotting, 4%, 33s elapsed, 15m 6s remaining
08:33:34 Plotting, 5%, 43s elapsed, 13m 43s remaining
08:33:44 Plotting, 6%, 53s elapsed, 12m 52s remaining
08:33:54 Plotting, 8%, 1m 3s elapsed, 12m 17s remaining
08:34:04 Plotting, 9%, 1m 13s elapsed, 11m 59s remaining
08:34:14 Plotting, 11%, 1m 23s elapsed, 11m 6s remaining
08:34:24 Plotting, 13%, 1m 33s elapsed, 10m 29s remaining
08:34:34 Plotting, 15%, 1m 43s elapsed, 9m 59s remaining
08:34:44 Plotting, 17%, 1m 53s elapsed, 9m 33s remaining
08:34:54 Plotting, 18%, 2m 3s elapsed, 9m 10s remaining
08:35:04 Plotting, 20%, 2m 13s elapsed, 8m 49s remaining
08:35:14 Plotting, 22%, 2m 23s elapsed, 8m 34s remaining
08:35:24 Plotting, 24%, 2m 33s elapsed, 8m 16s remaining
08:35:34 Plotting, 25%, 2m 43s elapsed, 8m 1s remaining
08:35:44 Plotting, 27%, 2m 53s elapsed, 7m 38s remaining
08:35:54 Plotting, 30%, 3m 3s elapsed, 7m 17s remaining
08:36:04 Plotting, 32%, 3m 13s elapsed, 6m 57s remaining
08:36:14 Plotting, 34%, 3m 23s elapsed, 6m 41s remaining
08:36:24 Plotting, 36%, 3m 33s elapsed, 6m 24s remaining
08:36:34 Plotting, 38%, 3m 43s elapsed, 6m 7s remaining
08:36:44 Plotting, 40%, 3m 53s elapsed, 5m 51s remaining
08:36:54 Plotting, 42%, 4m 3s elapsed, 5m 38s remaining
08:37:04 Plotting, 44%, 4m 13s elapsed, 5m 22s remaining
08:37:14 Plotting, 46%, 4m 23s elapsed, 5m 11s remaining
08:37:24 Plotting, 48%, 4m 33s elapsed, 4m 57s remaining
08:37:34 Plotting, 50%, 4m 43s elapsed, 4m 45s remaining
08:37:44 Plotting, 52%, 4m 53s elapsed, 4m 31s remaining
08:37:54 Plotting, 54%, 5m 3s elapsed, 4m 18s remaining
08:38:04 Plotting, 56%, 5m 13s elapsed, 4m 7s remaining
08:38:14 Plotting, 58%, 5m 23s elapsed, 3m 54s remaining
08:38:24 Plotting, 60%, 5m 33s elapsed, 3m 43s remaining
08:38:34 Plotting, 62%, 5m 43s elapsed, 3m 32s remaining
08:38:44 Plotting, 64%, 5m 53s elapsed, 3m 20s remaining
08:38:54 Plotting, 66%, 6m 3s elapsed, 3m 7s remaining
08:39:04 Plotting, 68%, 6m 13s elapsed, 2m 58s remaining
08:39:14 Plotting, 70%, 6m 23s elapsed, 2m 48s remaining
08:39:24 Plotting, 71%, 6m 33s elapsed, 2m 39s remaining
08:39:34 Plotting, 73%, 6m 43s elapsed, 2m 29s remaining
08:39:44 Plotting, 75%, 6m 53s elapsed, 2m 19s remaining
08:39:54 Plotting, 77%, 7m 3s elapsed, 2m 9s remaining
08:40:04 Plotting, 78%, 7m 13s elapsed, 1m 59s remaining
08:40:14 Plotting, 80%, 7m 23s elapsed, 1m 49s remaining
08:40:24 Plotting, 82%, 7m 33s elapsed, 1m 39s remaining
08:40:34 Plotting, 84%, 7m 43s elapsed, 1m 30s remaining
08:40:44 Plotting, 86%, 7m 53s elapsed, 1m 20s remaining
08:40:54 Plotting, 87%, 8m 3s elapsed, 1m 13s remaining
08:41:04 Plotting, 88%, 8m 13s elapsed, 1m 6s remaining
08:41:14 Plotting, 90%, 8m 23s elapsed, 58s remaining
08:41:24 Plotting, 91%, 8m 33s elapsed, 51s remaining
08:41:34 Plotting, 92%, 8m 43s elapsed, 43s remaining
08:41:44 Plotting, 94%, 8m 53s elapsed, 36s remaining
08:41:54 Plotting, 95%, 9m 3s elapsed, 28s remaining
08:42:04 Plotting, 97%, 9m 13s elapsed, 20s remaining
08:42:14 Plotting, 97%, 9m 23s elapsed, 15s remaining
08:42:24 Plotting, 98%, 9m 33s elapsed, 11s remaining
08:42:34 Plotting, 99%, 9m 43s elapsed, 7s remaining
08:42:44 Plotting, 99%, 9m 53s elapsed, 3s remaining

Writing out the plot at the end does seem to take much longer

1 Like
11

Finally, here is PCIe 3 x16 and I’ve found a BIOS setting that says “PCIe performance mode”

09:19:26 Plotting, 0%, 0s elapsed
09:19:34 Plotting, 3%, 8s elapsed, 4m 12s remaining
09:19:44 Plotting, 8%, 18s elapsed, 3m 44s remaining
09:19:54 Plotting, 13%, 28s elapsed, 3m 17s remaining
^C09:20:03 Stop requested. The program will quit after completing the current plot. Press Ctrl+C again to exit.
09:20:03 Plotting, 18%, 37s elapsed, 2m 55s remaining
09:20:13 Plotting, 23%, 47s elapsed, 2m 39s remaining
09:20:23 Plotting, 28%, 57s elapsed, 2m 27s remaining
09:20:33 Plotting, 34%, 1m 7s elapsed, 2m 12s remaining
09:20:43 Plotting, 39%, 1m 17s elapsed, 2m 0s remaining
09:20:53 Plotting, 45%, 1m 27s elapsed, 1m 48s remaining
09:21:03 Plotting, 50%, 1m 37s elapsed, 1m 37s remaining
09:21:13 Plotting, 56%, 1m 47s elapsed, 1m 26s remaining
09:21:23 Plotting, 61%, 1m 57s elapsed, 1m 15s remaining
09:21:33 Plotting, 66%, 2m 7s elapsed, 1m 5s remaining
09:21:43 Plotting, 71%, 2m 17s elapsed, 57s remaining
09:21:53 Plotting, 75%, 2m 27s elapsed, 50s remaining
09:22:03 Plotting, 79%, 2m 37s elapsed, 42s remaining
09:22:13 Plotting, 83%, 2m 47s elapsed, 34s remaining
09:22:23 Plotting, 87%, 2m 57s elapsed, 27s remaining
09:22:33 Plotting, 90%, 3m 7s elapsed, 20s remaining
09:22:43 Plotting, 94%, 3m 17s elapsed, 13s remaining
09:22:53 Plotting, 97%, 3m 27s elapsed, 7s remaining
09:23:03 Plotting, 98%, 3m 37s elapsed, 4s remaining
09:23:13 Plotting, 99%, 3m 47s elapsed, 2s remaining
09:23:23 Plot done "/nvme/plots/D98F...8BF1.fpt"

Hey, look at that! I’ve shaved ~20 seconds off my usual plot time of 4m8

1 Like