Ordered new Plotter Build - $5168 (UPDATED)

Chia Plotting
122

UPDATE

Wanted to keep everyone posted on a few things and lessons learned - for those asking “What’s the status/how did the build go?!” you actually didn’t miss ANYTHING. I’ve been sitting on my hands for the last 8 days waiting for parts (and dying inside every day).

Here are some stream of thought updates…

  • I purchased a 12-bay SATA JBOD from PC Pitstop in US - old school website, but I love the hardware. It was about $700 with an upgraded power supply. I filled it with the drives I took a picture of above (12x 14TB) but haven’t powered it on yet.
  • The website recommended an LSI 9201-16e as the host HBA card for use with that device (it has 3x outbound SAS connectors on the back and the LSI has 4 inbound) - so I found that card on eBay and bought it. I have it, but haven’t tried to use it yet.
  • I was waiting about 10 days for an 11700k that FedEx decided to finally deliver - that was infuriating. I physically got that in my hands last night and into the build and the build powered back on.

FINDING #1 - For those following along, the WHOLE POINT of replacing my 10800 with the 11700k was to get an 11th Gen Intel CPU which would enable 2 things on the ASUS TUF Gaming Z590-PLUS motherboard I had:

  1. M2_1 slot near the CPU.
  2. PCIe 4.0 speeds for the PCI_x16 slot near the CPU (for the eventual Hyper RAID card).

Well the shit-show that is the Intel platform + 590 chipset didn’t make this the win-win I thought it would.

Since the ASUS Hyper RAID card is backordered until the end-of-time, I figured I would:

  1. Install new CPU
  2. Put Samsung 860 EVO in M2_1 as a boot drive.
  3. Put 2x Samsung 980 Pro’s in the M2_2 and M2_3 slots, then RAID them as the PLOTTING drive.
  4. Knowingly sacrifice SATA Drive #5 and SATA Drive #6 because they share bandwidth with the M2_2 and M2_3 slots - you can only use 1 or the other, not both.
  5. Pray to Satan than the ASUS Hyper card show up, move the RAID setup to the card, free up the SATA Disks 5 and 6 again and go on my merry way.

Well that’s not what happened at all… filling all the M2 slots (1, 2 and 3) in the motherboard, even with an 11th Gen processor, causes the M2_1 slot to… not be seen/disabled/whatever-the-fuck… I can’t use it.

So now I’m losing M2_1 and SATA_5 and SATA_6… awesome, I love Intel. (It’s about this time that @codinghorror comment about why Chia farmers like AMD builds really hit home)

So I said f-it, I pulled all the M2 drives out, put a Samsung 980 in the M2_1 slot and now the machine boots and sees all the hard drives (because M2_2 and M2_3 are unoccupied)

I installed Ubuntu, spent hours fighting with the resolution it was detecting because of my KVM switch and then started plotting about 40 mins ago.

So right now, I am plotting to my boot drive - not ideal - but I just want things to be moving forwrad at this point in time.

I’ll keep filling the SATA drives and then after this first plot is done, I’ll get the JBOD connected and make sure that all works.

If the ASUS Hyper card ever ships, I’ll update the thread with reconfiguring the setup in the machine… but my take away right now is that if I had to do this over again, I would:

  1. Potentially look at BUYING plots - there are a lot of latent plotters out there just farming because their storage is full.
  2. Do an AMD build so I wouldn’t have to fight with all this PCI lane limitaitons.

I do like the iGPU on the Intel chips - that made the build very very easy and lots of airflow in the case… so it’s hard to say that the AMD is a slam-dunk win, but messing with all the drive visibility issues has cost me 2 weeks.

2 Likes
123

Well that sucks :sweat_smile:

Yeah those hyper cards seem to be made of solid unobtanium these days, tried to orde twice already, wouldn’t hold my breath for that one

I’m not to familiar with details of the pcie limitations of Intel on the m.2 slots, but the PCIe x16/x8 slots should have full bandwidth, right?
Wouldn’t it be better to get 2x “normal” m.2 risers for the 980’s and just do software Raid?
That way It shouldn’t conflict with the Sata ports

124

I think you only start running into these limitations when you fill out the board with storage - so normal people likely wouldn’t ever notice this.

YES! You nailed it, this is the next thing I’m going to try - get 2 of those and see if I can get the boot drive offloaded into 1 of those and then just have 2 temp-plotting 980 drives working and moving plots off to storage.

1 Like
125

Be sure to get ones with heatsink, 980 can run pretty hot

126

Ok so I went looking for Riser cards and found this and just ordered it - it’ll get here tomorrow and I’ll keep folks posted.

If I can get the 860 setup in M2_1 as boot, and the 2x 980’s in this card, I’m happy.

2 Likes
127

I learned this as well just yesterday, I bought an inexpensive 10$ PCIe x1 to NVME riser for both my old i3 6100 and my current i7 7700 only to find out that on both systems the installation of the PCIe riser broke both systems, internal GPUs and PCI devices got disabled, can’t even do a clean reinstall of windows while the NVMEs are plugged in, some have said that I should’ve used a PCIe x16 riser instead of the x1, I’m looking for that one currently, to try and salvage the current available hardware for plotting at least, since in the future most of the investments will be on the farming HDDs instead of the plotting system.

Does AMD motherboards not have this kind of problem? I wonder why, if that’s the case I might go AMD instead of intel, currently eyeing a 10900 system since I already have 2x1TB samsung 970 evo plus just lying around because of late shipment, I found some WD SN750 2x1TB locally instead.

Thing is, right now I’m kind of reaching the consensus (maybe because of personal bias towards intel) that going high-frequency CPU is the key, even though the 10th Gen 10900 only got 10 cores, I was going to do 11900 but the Gen4 2TB NVME prices kept me at bay, so what I can stomach to buy is the 10th gen system instead.

But, if I can’t find an Intel motherboard that can support 3xM.2 NVME SSD then I might as well go AMD… this remains a problem that haven’t got an answer.

I hope you can get the ASUS Hyper Card soon, my other option is look for one of those or similar if I can’t find a decent enough intel motherboard that can support 3xM.2 NVMEs.

Good luck, and thank you for your update, feeling your frustration there, but all is good, once the plotting fiasco is done and we’re beginning to fill the drives, there’s the harvester 30s trouble to be had, automoving the plots through a fast enough ethernet (my dream is 10 GbE), and then backing up the full node if it ever goes down for whatever reason (internet failure, power failure, my dog suddenly decide to play with it…), not to mention getting drives at a low enough price…

much more to come and so far however much time it has consumed, this has become my new job for now until I got all the automation dialed in. :sweat_smile:

128

If you ever do… don’t use a download manager that utilizes multipart file segmentation (like internet download manager or folx), recombining a plot file takes ages… I turned off the multithreaded download on internet download manager and even though download speed has lowered, the wait on recombining is nonexistent

1 Like
129

@codinghorror mentioned it - I think they’ve always had more PCI rails or something? Not entirely sure what the arch differences are.

The closest I’ve ever seen but not confirmed are the Gigabyte boards (I checked ASUS, ASRock and MSI - all mention some form of disablement when m2_2 and m2_3 are used) - Gigabyte has a series of Workstation/Pro boards that do NOT mention disablement… there was 1 other fellow in this thread looking at building with it and I was wondering how they ended up doing.

I ordered that one I linked above - never heard of the brand, but if it works it works. I’ll just leave the ASUS Hyper on indefinite backorder and see if it shows up some day :slight_smile:

Thanks my man - I agree, getting over the hump is the payoff… I see the light! :slight_smile:

130

Not so sure why you are that insisting on getting M.2 for your boot drive. It really would not make any difference for a Chia machine.

3 Likes
131

Yes AMD has more pcie lanes available, and also supports gen 4 since 3000 series already.
(but as storage_jm pointed out, you usually don’t need gen 4 for plotting. I guess the hyper card may end up using it, but i doubt it.)

I’m using a AMD system with 2 m.2 slots on the mainboard, and am using 0 of them.
Cheap Sata ssd as boot drive, my 2 plotting dives are each in Icy box 3.0X4 risers with a nice heatsink.

Guess my point is, don’t get an expensive motherboard just cause it has more m.2 slots. A decent pcie X4 riser costs 20-25 bucks and comes with the added bonus of a good heatsink.
Just don’t be like me and mistake X1 slots for X4 :sweat_smile:

3 Likes
132

Because all my SATA ports are full with farming/storage drives - the only remaining storage connections I have left are M.2.

133

Quick Update for anyone bored on a Friday night… I JUST figured out why my Samsung 860 drive was NEVER working in the M2_1 slot on this mobo… it’s an M.2 SATA drive, not an M.2 NVME drive.

Something I had only ever passively read about but never understood until this frigging moment.

Thank god for Amazon returns and just overnighted a Samsung 980 250gb (non-Pro) that is M.2 NVME - should be here in the morning along with that competitor card to the ASUS Hyper I ordered so I should have the build booting and plotting God willing.

1 Like
134

Just read through this whole thread and had to join to give my experience. Right now I am getting roughly 2.4 TB plotted a day.

I am currently running
I-9-10850K with a Kraken X53 (Temps stay in the mids 60s for the CPU running around 75to 90%)
32 gb 3200 ram (I am upgrading to 64)
1Tb Firecuda
1 TD WD Black
RX 6800 XT (mining away for eth it does throw off some heat)

Overall I have temps pretty steady in the mid 60s for both the CPU and GPU. I have not overclocked the system. If mining with a graphic card and I could see cpu temps becoming an issue just air cooling. Without the added heat from the GPU I think air cooling should be fine. The way my system is currently setup the two NVME drive dump to 2 internal 7200 rpm hhds and I transfer that over to wherever it is staying. It takes roughly 8 minutes after plotting is done to transfer and clear to be able to restart. I run 6 staggered plots at a time and the WD Black plots take roughly 5 hours 30 minutes and the Firecuda plots take roughly 6 hours 30 minutes.

I have been plotting for about a week now and already have a few things I am upgrading. First I am switching from a Z590 motherboard with only 2 NVME slots to a Z490 board I already have that has 3 NVME slots. I am upgrading my ram from 32gb to 64 gb. I also ordered a 1 TB SSD and already had a second 1 TB WD Black so I will be able to run 10 plots at a time instead of 6. 3 per 1 tb drive with 1 plot running on the host drive. If I can get 3TB a day plotted on this machine I will be pretty happy.

On a separate note I ordered a 1tb ssd for my old gaming pc that I am going to convert to a NAS. I have 100 ish tb total to plot so we will see how it works out. I am very curious to see how your 11th gen build turns out. I was thinking putting together an 11600k build to utilize the spare parts I have sitting around and stuff a bunch of storage in it.

1 Like
135

Thanks for adding your story here!

Do you manually manage this from the GUI front end or using scripts?

I think you are fine here if all your SATA ports aren’t filled up, but if they are, read through my pain again - trying to max out every storage connection in a 10th gen (and even an 11th gen) Intel setup results in you being forced to give up at least 2 of the drives which either nullified M2_2 and M2_3 or SATA_5 and SATA_6 connections.

That makes two of us - my m2 NVME boot drive AND PCIe RAID riser arrive tomorrow - so I’m going to get this thing running if it kills me and I’ll keep the thread posted.

My goal with this entire build is to figure out and finalize what “the best build for Chia plotting is”… farming is different, whoever has drives wins :slight_smile:

1 Like
136

The first two days I was manually doing everything from the GUI and keeping track of how long everything was taking and messing around with different settings to see how it impacted plot times. Now I have the plots queued every 6 hours on the 980 Black and every 7 hours on the firecuda from the GUI and everything seems to be running smoothly.

I did not originally build this with chia plotting in mind. This was supposed to be my gaming/video editing pc. I was mining with the GPU when I was not editing or gaming. I watched Der Bauer’s threadripper build for chia and I became interested in the process. I started plotting and decided to invest in some extra storage space. Now I am pretty hooked.

There is a couple of reasons why I am switching from the Z590 to the Z490. The biggest one is since I am using a 10th gen chip the Hyper m.2 drive does not work leaving me only two ultra m.2 3X4 slots and the rest would be sata drives. The Z490 board allows for me to have 3 ultra m.2 3X4 slots and only 1 sata ssd.

The other reason is the Z490 board is eventually going to be setup for a custom waterloop and has much better heat sinks for the M.2 drives, ram, and will allow me to overclock the cpu. I already had all this stuff anyway from having to buy a combo in order to get the GPU so I figure why not build out the system and see what it can do. I don’t plan on overclocking until I get my storage plotted out but once that is done I am going to see how far I can push it.

2 Likes
137

this is exactly what I did, thinking any PCIe to NVME riser would work, I bought a riser with PCIe x1 interface

When plugged in disabled all PCIe devices on my system, the onboard ethernet and also the onboard GPU, waited for 30 mins and hoping there will be a boot cycle, nothing… I plan to get one that’s PCIe x16 at least… some ppl have said to me that a PCIe x4 interface has worked for them, while those using the PCIe x1 interface has failed miserably

138

I had the same thing happen to me. i turned on high performance mode in bios. turns out it was sucking power from the gpu and gpu wouldnt turn on. after that everything worked fine.

139

I understand but adding more sata ports through a PCIe card is ultra cheap, like 6 ports for 50$

140

PCIe x1 is no good, that will only get you about Sata speed, X4 is required to get full speed for the m.2 slot. M.2 on the mainboard also uses x4 bus.

X16 is only needed for cards like the hyper card, that has 4 m.2 slots and uses bifurcation on the motherboard (if the motherboard supports that) to create 4 individual pcie x4 out of the x16 slot so that each m.2 can run at x4

Still very weird that it would cause that much trouble though

1 Like
141

AnandTech did an overview of different z590 boards on offer last January. You can look there for an initial survey of the different board features.

590
590854×600 28.9 KB

2 Likes