Not honestly expecting much from my X5670s - they fit the definition of ‘stuff I had lying around’, though. Will post results here, mostly for curiosity.
my x5690s are tip top.cant even strain them even with 25 plots a day. and with the 5600’s hyper-threading is really great. loads of features not commonly known. t710 holds 2 x5690s. total 24 core at 3.7 turbo. have another r 410 plotting with 2 low energy Xeon L5640. again 24 cores in total.
1 Like
When I got this server it was from a proper refurbishing business that sells to other businesses, i really am quite annoyed now that they knowingly sold it to me like this. But i suppose you said it, on principle you would never do that…some people are just scummy. Guess they tried to just offload whatever on to me.
Well its plotting away now with the 4 sticks on 2 channels (vs 6 sticks on 3 channels) doing 8 plots on the SAS drives. The time seems to have gone back up to being quite high for phase one, doing 1/2 plots it was a huge improvement but now its drastically dropped back down for 8. I’ll let these run through and feedback after phase 1’s have completed.
Do you think going up to the recommended 4 channels (8 sticks) will make a significant difference, even if im not using all 64GB ram right now? I still cant believe how much quicker it was dropping down to 2 channels - really hope that there’s some more performance that can be squeezed out going to the recommended 4 channels. Also right now im using a 20 min stagger…should I increase this to an hour, would that make a difference?
What would you recommend for a good Memory benchmark? Not looking to stress test but get a good indication of the performance…i’d like to run it in 2, 3 & 4 channels to give the performance difference a tangible number.
With zero effort to optimize, and only 8GB of installed memory on the dual X5670, using 7.2KRPM SAS disks as /tmp dirs, I can get 4 plots in 24 hours (2 parallel plots) - considering it only has 4x2GB memory in it out of 12 possible slots, I’m quite happy with that.
I can get ECC DDR3 for < 1.5 USD a GB (for 2GB sticks), and considering this is my budget build I’m not going higher than that. Got another 8 2GB sticks on order, going to try filling the slots and see what I can get. I could move in some 10KRPM SAS disks for free, too.
ECC unbuffered are quite expensive - ECC (registered) are far less and 4GB modules are dirt cheap
I cannot quote you on performance for my 10K SAS drives - i have almost filled my disk collection - in particular my plotters with 10K drives are now full so i will not be plotting more until poolable plots arrive
Below are my P1-P4 and Total times for a single plot running in parallel with 15 other plots, so 16 total parallel plots. Stagger about 40 mins and run all day for 36 total a day.
T5610 precision workstation upgraded
dual Xeon 2690 v2
128GB ECC
Dual 1.6TB Samsung Enterprise NVME U.2 drives
Time for phase 1 = 13375.259 seconds. CPU (158.360%) Wed Jun 9 09:21:15 2021
Time for phase 2 = 6519.859 seconds. CPU (83.570%) Wed Jun 9 11:09:55 2021
Time for phase 3 = 14910.020 seconds. CPU (83.310%) Wed Jun 9 15:18:25 2021
Time for phase 4 = 1093.257 seconds. CPU (82.550%) Wed Jun 9 15:36:38 2021
Total time = 35898.397 seconds. CPU (111.300%) Wed Jun 9 15:36:38 2021
Hi guys! I’ve been tweaking with my configs but can’t seen to get my plot times under an average of 9 hours and 15 minutes. Maybe it is the best my hardware can do, but I wanted to see if anyone had any advice. Here is what I have/am currently doing:
Dell Precision T7910
2x E5-2680v3 2.5GHz 12C (24 total cores)
128GB Ram
3x Samsung 970 Evo Plus 2TB NVME (using Dell Quad adapter, and I have the a small boot NVME in the final slot). But I do plan on getting a 4th Samsung 970 soon, and then move the boot drive elsewhere.
Currently saving finished plots directly to 7.2K SAS drives.
I’ve been using Harry Plotter as I like a GUI interface and running latest Ubuntu. I edited systemctl fstrim to trim the drives hourly and I think its running right (I’m not the most comfortable using CL!). My current setup nets me between 45 to 48 plots a day.
I am running 18 concurrent plot jobs with a 25 minute stagger, with 9 max in Stage 1 (but usually by the time the 9th kicks off, the first is almost or already in stage 2). I’m using 4 threads per plot and have the memory set to 5000. I also stagger the plot jobs according to the NVME drive, so NVME 1 goes first, then NVME 2, then NVME 3, and it repeats for all 18 plot jobs.
This seems to be the best setup I’ve tried and I can’t seem to get the time down or plots up using anything else I can think of. Am I at my limit or is there anything else I should try? Thanks in advance!
Just an update - I was reading how someone else mentioned not to run more than 5 plots on a single 2TB NVME drive to not over saturate it. So I am trying something different now.
15 concurrent plots (instead of 18)
6 threads - 6000 Mib ram each plot
30 minute stagger
6 plots max in phase 1
I’ll report back how it goes!
I had my dual Xeon E5-2680 v3 spitting out 55 plots per day with swar and parallel plots across 4x 2TB NVMe drives. I have since swapped to MadMax and am doing ~20 minute plots, thus hitting 71-72 plots per day whilst only using 1 of my NVMe drives for temp1 and 110GB ramdisk for temp2.
2 Likes
Wow, those are insane numbers. I glanced at MaxMax before but now have to really look into it. How many threads are you giving to MaxMax, and does your machine have 128GB of Ram (because I noticed in the Github it said the machine needs min 128GB GiB ram, but not sure if it was a typo and he meant 128GB). Are you running through something like PlotNG or just through MadMax? Thanks!
Yeah for sure check it out. I give it 24 “threads” as it’s recommended to specify physical cores. My system has 128GB of ram and has no issues with the 110GB ramdisk when using 256 bucket; I run it natively without a plot manager.
Managed to sort out the speed issues I was having earlier…and as soon as I did that and optimized for parallel plots MadMax started making noise.
Now im re-optimizing for MadMax but from what I see I’ll easily double my plot count per day and not have to worry about losing plots partway through the day for any reason as I’ll just have one going at a time vs 12.
Still running through tests with various numbers of drives in RAID 0 (along with bucket size and a ramdisk) but I think I should be able to hit well under 30 min plot times easily without using an NVME.
I’ve actually just sold all my nvme’s (was going to purchase more smaller ones) but I think there’s no need for them as I have a ton of 10k/15k sas drives around and MadMax completely negates the need for an NVME. Also need to consider the space to store the plots if doing 45-50/day lol, the increase in speed with an nvme is probably not worth it for me.
Just found this thread… My $.02. Ivy bridge for me is kind of the starting point of worthwhile in terms of performance per dollar, because of the low price of DDR3 as well as how low the prices of the old chips are. I’ve got one Westmere (Nehalem die shrink) and it’s OK since it was cheap, but much worse in terms of speed per watt than the ivy bridge. Totally worth it to rebuild as ivy bridge to reduce power bill if I end up plotting more (I’m done for now).
I have a dual E5-2690v2 with 256GB of ram which was formerly getting about 52 plots per day using about 30 spinning disks as temp drives (it’s in a 36 bay chassis), but that’s a lot of effort. Switching to the madmax plotter, I’m able to get a single plot using ram disk to plot in less than 25 minutes. It’s only a bit more (perhaps 56-58 plots per day) but far simpler and easier to manage. But the performance is definitely there on the Ivy bridge and later, then it’s just a matter of the upfront cost tradeoff (cheaper for older) vs the power bill (higher for older).
1 Like
You are right about power, my R515 dual opteron (with 12 disks) uses 370W mostly just farming and plotting 4/day. Yet my R720 dual E5-2650V2 (with 8 disks) uses around 300W plotting at nearly 100% processor utilisation.
Thanks again for this info.!
I got MadMax up and running today. I tested it with 2 NVME drives ( temp 1 and 2) first just to see the numbers and was getting about 29 minutes a plot. I then setup ram disk as temp 2 and am getting about 25 minutes a plot. I was hoping for around 20 minutes like you since we have basically the same setup, but I guess either your Ram or NVME drive is faster, otherwise I have no idea. But I guess I can’t complain too much about 25 minutes. Going to see if I can tweak anything to get it faster, maybe raid 0 2 of my 2TB NVME drives, can’t really think of anything else to do at the moment to speed it up.
What’s your configuration? Are you on windows or Linux? There’s many reports that the ramdisk implementations on Windows aren’t very good, and you need to run Linux to get big benefit from ramdisk.
My dual X5670 now does 12 plots a day consistently, going up to 24GB ECC reg. ram cost me ~18USD that’s the only thing I’ve spent on this. Plots directly to 7.2KRPM SAS disks (one per plot), not tried Mad Max (would it be better?). I could probably optimise it with better plot scheduling, but at the moment it’s not on the same network as everything else, and moving plots off it is a sneakernet job, so happy for it to do what it’s doing for now.
I have a dual X5670 also. I was able to do about 26 plots per day with 16 individual drives and 96 GB of ram. Assuming you have a smaller number of drives, what you have is not bad. Going to madmax probably would not be faster. Madmax works well when you have either enough ram for a ramdisk (128GB+) or if you have very fast nvme drives. This same X5670 can now do madmax plots in 45 minutes, but I achieved that by running a 12 disk raid0 using 10k SAS drives.
1 Like
I’m running latest Ubuntu desktop. One of two things I can think of is my Ram speed. I keep forgetting to check but I think the speed is 2133Mhz. I think with the 2400MHz it will shave some time off. I’m going to check tonight when I get a chance. I also tried a different NVME drive to see if it made a difference (was using a Samsung 970 Evo Plus and then tried an SK Hynix Gold P91) and the times are pretty much identical - 25 minutes give or take while running the Chia GUI simultaneously for farming. If I close the Chia GUI it saves about 20 to 30 seconds a plot. The other thing I can think of is I am running Ubuntu desktop and I heard running the server version may be faster. Does anyone think this will make a worthwhile difference to switch?