@ianj Just to go back over a previous comments i finally got things set up and going (dual 2670 v2, 96gb Ram (1866MHZ), and 8x 10k 900GB SAS drives (for now)). I understand how to use SWAR to plot in parallel 1 job per drive/plot but I’m really having issues with the times.
I noticed most people are getting around the 12ish hr mark with a similar set up but im running 14-16hr/plot.
This is a small sample of what I’ve tried: (plot size k32, ram, #of threads, phase 1,2,3,4, & total plot time.
Im not sure what I’m doing wrong or what I’ve got set up wrong here but it seems like something is adding a couple hours to my times. Any idea what might be slowing this down slightly?
As we discussed previously I could easily ramp up the parallel plots as long as I can stick one per drive, but I want to understand what I’m doing wrong here before committing all my drives.
Actually i have almost filled my disks and those remaining 3.5" 3TB SAS drives have now been subsumed into my farm collection (i have 10x900GB 2.5" not yet deployed) and those log files are no longer around - it may actually have been 12-13h. You should understand that I was was producing 100-120 plots per day so a variation in 2-3 a day was not a priority
Saying that i will definitely publish the results with the 10K 2.5" drives when i deploy them
I was leaving them on the the temporary drive and rsync them off asynchronously - that would probably make some difference as i was only dragging them off the temporary drives fast enough to stop them overfilling, so the next plot did get to start quickly
I also used Ubuntu and a single E5-2678v3 (10 unlocked cores)
So I was looking around for old servers to setup as hdd plotters
But I’m just seeing double from all the different specs and prices, so wanted to see if you guys think this is a good deal for 1100,- euro incl 21%vat (mind you everything here is bit more expensive anyway compared to US) The 48TB hdds make it kinda look interesting
Specs:
2x E5-2680 v2
128GB RAM
12x 4TB SAS 6Gb HDD
1x ADAPTEC - ASR-78165 24-port controller
2x 1000W Platinum SQ PSU
also included (dont need it i think)
2x Intel Ethernet I350-T4 4-port-Gb NIC Network card
Not so high end CPU but that would only cost £80-120 to upgrade to E5-2650v2 or E5-2670 v2 (and be able to sell the E5-2650 to cover some of cost)
Not so much memory but they use Registered DDR3 with shitloads of slots and 8GB modules are pretty cheap
SFF so no 3.5" but ok for a set of 2.5" 10k SAS for long term plotting - the full height DL360p might have more space to play with with NVME slots and 2U are usually are quieter than 1U
That is a bit of a generalism - i’d like to see a source of that
One of the best sources for info on drive reliability is from 2017 Hard Drive Failure Rates - What the Numbers Tell Us Backblaze. They are a cloud backup company and they have literally thousands of hard drives and they collect stats on which drives fail. If you look at the 2017 report, you’ll see that generally 1% of 3tb drives fail per year. 4tb drives still seem to have some quality problems.
3 TB hard drives as reliable as others.
An early set of 3 TB drive from Western Digital had excessive failure issues in some applications due to aggressive head parking, but the modern ones are fine. There’s even a firmware fix for the older, affected ones: WD Software
Early 3 TB drives got a bad rap since they were the first drives to cross the 2.1 TB barrier (Explaining the 3 TB hard drive barrier). As long as you have a modern OS and BIOS, it should be a non-issue.
There’s no reason not to use the 3 TB drives. It’s a no-brainer for stand alone data storage. For a boot drive, just take the extra steps to make sure your OS and BIOS can cope.
Going by size of the hard disk rather than brand, the general trend is that the bigger the drive, the less likely it is to fail. 8TB drives had a failure rate of 1.6%, with 6TB models on 1.76%, rising to 2.06% for 4TB hard disks.
The one exception to the rule is that 3TB drives are actually the least likely to fail with a death rate of just 1.4% over the course of 2016.
Granted, there isn’t a massive difference between 1.4% and 2.06%, but it’s still worth bearing in mind that 3TB drives seem to be the most reliable overall, followed closely by 8TB models.
I definitely prefer some additional information than a blank “never use”
The main accepted downside of 3TB is more power/heat, more connections for a given capacity
hi @ianj thanks for the response i understand that you were doing a large number of plots, hoping maybe something I’m doing stands out as being completely wrong…or causing the 15-17hr plot times. Compared with someone else and theyre getting better plot times on an older xeon with slower ram w/ 10k drives.
Appreciate you publishing the results when you get them. I’m also ignoring the copy times here as just looking and single plot times vs total daily for now until i figure out what is going on.
In the meantime here is the output for 3 sets of runs I did (all 20 min delays in each run). Did one run with 3390mb ram and different number of threads, then 5gb ram and 4 threads, then finally 10gb ram and 4 threads…which ended up with the longest times somehow???
I’ve made sure the spectre/meltdown patches arent on, write caching is disabled for the disks (default), the bios is set to max performance, and the Turbo is definitely working (>5 threads its 2.90ghz…and its sitting there the entire time).
Also just ran crystaldiskmark on the drives and they seem to be operating properly.
Are you running under Windows or Linux - i have never run my system under windows - it is Ubuntu all the way. Rumor has it that Linux might be about 10% faster
Running under windows server 2019, and I’ve taken the apparent “up to 10%” faster in Linux into consideration, as its extremely slow so not sure its windows vs Linux.
Also i ran a test last night with a space M.2 NVME I had laying around, its connected via pcie 3.0x4 (confirmed in crystaldiskmark) and the plot times are exactly the same as the SAS drives…this is so weird.
This is the latest run all 3 threads, and 6gb ram…you can see #1&2 are the nvme drive and the plotting times are the exact same times. What could be causing something like this? Could it be a RAM issue, i’ll check to make sure they’re installed in the right slots but from what I recall theyr’e in A1,2,3 & B1,2,3.
You are right to check memory - in my experience (outside of plotting) you can lose 5-15% of performance running dual channel and 10-25% running single channel. I don’t know your layout but as long as it is running triple channel you should be fine. Your memory is fast enough - in fact my experience is that drops in speed are far less severe than drops in channels in use - I have run E5-26* series at 3x1333 (10600R) with not a lot of loss
CPU-Z might help with that
Depending on the NVME it might not be that fast - i have heard that some cannot sustain throughput and collapse down to sub HDD speeds under a sustained write load - my Sabrent can sustain 900MB but i have heard that some can collapse to very low levels. A fast HDD can exceed the performance of as slow SSD - at least until you try to run plots in parallel
Still looks like it’s not that bad a price when you consider the resell value, but it leads me to better options with less investment. Most of all I have to stop comparing to Dutch prices, they are crazy here. (a single used 2670v2 “refurbished” is around 150,- a 4TB sas 2.0 drive, 70,- per piece)
Anyway, first step is completed, just snapped up 2x e5-2680v2 for 30,- each.
Also I’m going for a system in a Supermicro 4U tower chassis [SC743TQ-865-SQ] just because of the proximity to my living room and I realized that rackservers tend to make quite a bit of sound
Now “all” I need is some sas drives and a suitable HBA … to be continued… aiming to stay under 500,- total this time.
I don;t really need that much extra plotting power anymore, but I want to have somethin that can plot without wearing down nvme’s and the as soon as I’m done replotting for pools, I can sell of my main plotter and keep this one to do some plotting on any good storage deals I might come across
Oh wow those are steep performance hits for running the wrong kind of memory.
Actually so the r720 with the 2670v2 only does dual channel or quad channel from what Im reading…and guess this is confusing me now. If i understoond what you said…it’s that using the wrong channels is worse then using a slower ram. So i guess I should ask if im using the channels correctly. But if im using putting the ram in pairs thats dual channel right…even if i have 6 sticks as theyre still in pairs??
Does that mean I shouldnt use 6x 16gb as it will use 3 channels only? From this manual, im using slots A1+B1, A2+B2, A3+B3…should I take out the A3/B3 or add 2 more stick to A4/B4?
Manual starting on page 45 - examples for memory installation with 2 processors https://www.dell.com/support/manuals/en-us/poweredge-r720/720720xdom-v3
I dont see any examples where they used 6 sticks…i mean its registering properly but am i getting that massive performance hit because its not optimally placed?
Also im doing a test where I take out 2 stick so just using 2 channels and will try plot 1nvme and 1SAS drive.
MY BAD - the E5-26 goes to 4 channel - the triple channel/figures were from the prior generation of X56xx processors
I don’t have specific figures for E5-26* but i reckon 3 & 4 channel will be ok and 1/2 channel pretty poor. It depends on the workload - some algos (especially CPU based mining) are designed to consume memory - to break the L3 cache - they suffer more than general processing. I reckon chia is nearer to general processing but as i said i dont have figures for E5-26* and different usages
Some servers allow you to stack the RAM how you like and you pay the consequences - some are strict - if I could choose I would make sure:
The processors are BALANCED - ie not 75% of memory in 1 and 25% in the other
Try and use all channels before adding more banks
If I had 8 RAM modules i would use A1, A2, A3, A4, B1, B2, B3, B4
If I had 6 RAM modules i would use A1, A2, A3, B1, B2, B3
I don’t know if the 720 allows 3 channels - in my experience many servers do but i haven’t had a 720 so i can’t be sure - the fussiest i have played with were IBM
I’ve got a Dell R720 with 2x 2650 V3 (8 core/16 threads per cpu).
I got a deal £1/GB for 256Gb Hynix PC3L-12800R 16x 16GB modules. Configured it into two even banks added to the ram I had already (8x8Gb) with the largest modules in the lower numbered slots and now I have 320GB ram.
Tried a 270GB ram drive as temp drive for plotting with 4 threads and it still took 16 hours.
Since then I’ve been plotting 7 threads in parallel to 7x SAS drives 4 threads per plot and it’s been taking about 16 hours to do 7 plots. I can upgrade processors to 2670 v3s for not much money and a 50% speed boost, but I’m not processor limited with 7 plots.
I have no intention of using consumer grade SSDs or NVMEs to damage that seems against the spirit of the thing, none of this activity is substantially destroying the equipment.
The Dell Rx20 series with the dual Xeon are fantastic servers for the price. But then again, they are servers, so they would be better suited for NAS duties and farming.
Thank you for this, I had 6 RAM modules installed exactly as you said initially but poor performance and havent been able to isolate any of the other components. They way you explained it allowed me to get my head around this a little more and do some proper searching!
According to a Dell rep when using 2 CPU’s on an r720 the 3 channel configuration (i.e 6 DIMMS split into A1-3/B1-3) is not “recommended” it will work and not throw error but performance wont be optimal. 2 channel (4 DIMMS) works…but 4 channel across both CPUS (8 DIMMS - A1-4/B1-4) is optimal.
So I removed 2 DIMMS (as I dont have 2 extra to add at this point) to just test the theory and ran a test on one SAS drive and the (old/spare) NVME i had laying around and result! (Integal is the NVME, and other is the 10k SAS drive)
Previously both the NVME and the SAS drive were both doing 5.5-7hr Phase one times, just removing the two DIMMS to use 2 channels (vs 3) SEEMS to have helped. Phase 1 times down to 3hr20m, and 4hr20m! But this is just one plot.
I’ve got 2 DIMMS on order for tomorrow so I can test again across 4 channels (8 DIMMS) as the DELL rep mentioned that 4 channels is the optimal configuration even over 2.
What I’ve done now is run a follow up test with 8x SAS drives to see if I can do better then the 17hr plot times (with 7hr phase 1 times) with the 6 DIMMS (across 3 channels),
I dont want to celebrate too soon…but it seems to be a start to solving this. I just cant believe that going 3 channels even when the sticks are paired correctly across the CPU’s can result in such a drastic performance hit?
This thread is encouraging me to try out plotting on a dual-X5670 rig that I’ve been meaning to set up as secondary harvester, basically free plots since it can plot to the initially empty disks that will be going into it.
I am wondering if my 32GB RAM dual Xeon (can’t remember the processor other than I got the highest end one; shows how many times I use it) Mac pro 2012 (I’ll install Windows or Ubuntu to an SSD I have lying around) can plot relative to it’s power consumption. I tried to plot with a clean install of Mac OS and it stalled at 1% and I gave up after several days of plotting just chugging along.
I buy, upgrade and sell workstations for “fun” - T5600, T5610, Z800 - that sorta range - and i would, on principle, NEVER sell without the full set of channels configured. I would get them in sometimes with 1/2 channel populated and performance was awful - popping a few RAM sticks often made far more difference than a SIGNIFICANT CPU upgrade
Be careful about dropping to the X56xx range - for general processing they might be fine but for chia you would be looking at a SIGNIFICANT hit vs an E5-2670 - i only really sold them with large memory configurations (48-96GB)
I would think PASSMARK results offer a pretty good indication of what they would achieve - its a more practical benchmark than some pure CPU measures like CPU Info - you can get results for most CPU and will see that the X5670 is notably below the E5-2670
