I have a HP Z210 Xeon E3-1230. It could do 120 minute a plot. I have been adding hard drives into it and made it a harvester. I noticed the plotting time slowed down. Now there are 8 hard drives connected by SATA and 4 hard drives by USB. I was making up some plot in it and it took over four hours for one plot. The plotting SSD is a lower end NVMe (crucial P2). Harvester function is running always. Does anybody experienced the similar situation?
Did you add the drives using the USB ports? The HP Z210 Xeon E3-1230 has only USB 2.0 which only transfers at .5GB/s. If you were using SATA drives and the problem started when you added USB drives/arrays then this is your problem.
If so, you can install a PCIe card with USB 3.0 5GB/s ports, often referred to and marked as “SS”. If you use any further hubs, they will also need to be powered, USB 3.0 5GB/s.
USB 2.0 transfers at .5GB/s. Regular USB 3.0 transfers at 1GB/s. USB 3.0 SS transfers at 5GB/s.
USB 2.0 is acceptable for farming but your response times will be within acceptable limits but imop high. Plotting transfers using 2.0 while farming take forever and have a huge impact on response time with many missed challenges.
Standard USB 3.0 farms without trouble as long as you are not laying plots.
USB 3.0 SS 5GB/s has my average response time under .15 seconds with longest response well under 2 seconds. I’m not plotting now, but when I was, my K33s transferred in 30 minutes and you could see the impact on response times, but they were not bad.
Thanks bro. I added a two port USB 3.0 PCI-E into the Z210. it is a quite old USB 3.0 card I bought it when USB 3.0 came out some years back. I tested speed it can do 100 MB/s copying plot file. The problem I have is not writing plot to destination. Max Mad plotting was 2 hours a piece. Now 4 hours. The only difference is I add several hard drives, some via on board SATA some by a PCIe SATA card (cheap one) and some by USB 3.0.
I’ve heard of farming response times slower when plotting, but not the other way around. I would disconnect the drives and see what the plotting time does just to confirm if that is the problem or not.
Hmmm … I am not at all up on madmax as I only pump out K33s. I have no idea why madmax’s plotting speed would be slowed by the addition of HDDS. Does madmax use any of newly added HDDs to do some of its work?
For your USB connected drives I would still suggest installing a more recent PCIe card that has 5.0GB/s USB 3.0 SS ports. If nothing else, it will at least improve your response times for the USB connected drives as long as they are in 5.0GB/s capable arrays.
I do not think it has anything to do with your hard drives
You probably need to TRIM your SSD
also your SSD probably just sucks, consider plotting with something like a used enterprise data center drive from eBay
2 Likes
Did you check your RAM usage? Seems that it’s topping and start using HDD as RAM for plotting.
That P2 grade Curcial brand NVMe is indeed slow maybe. Still slower. I have 12 hard drives connected to the computer. Anyway I will try to add hard drive by SATA. I have a question: Can two SATA controller cards work in one computer? Motherboard has its own SATA group. I added a cheap two port SATA card. I tried that 1-5 port multiplier. Widows crashed. Remove that 1-5 multiplier, Windows OK. Add 1-5 again, crashed. So can I just add TWO 4 port SATA card on two PCIe slots. Same two cards. It will add 8 SATA port internally. Thanks!
Yeah, i also couldn’t make those multipliers work on Windows (didn’t have crashes, just didn’t work), but they do work perfectly under Linux. I think that route may be dead for you.
Yes, you can put as many SATA card as you have PCI slots.
Another option would be to purchase SAS/SATA controler - about $50, plus a cable for $10-20. It is a good card, and can handle 8 SATA drives.
Just check your mb manual, and select a faster PCIe slots.
I just took a quick look at that box specs, and you may have there only one PCIe-Gen2x16 slot (actually, two, but looks like the second is only mechanically x16, but electrically x4).
Also, your SATA are only 2 with 6Gb/s, the other two are 3Gb/s. So, make sure that your SSD sits on that faster slot. That also mean that you can move your OS drive to that slower port, and get one more SSD (a bit faster ?) that you will use for tmp2 folder. MM likes to have two SSDs. Maybe that is the best upgrade that you can do.
Also, if you don’t mind, could you show us your MM line you use to run it?
Also, just noticed it. Do you have that NVMe already sitting on that faster PCIe slot?
Actually, I also missed that your PSU is only 240W, what may/will be a problem to also power your all drives. Your CPU takes about 80W during the boot time, and your 10 drives will draw about t00+W, add everything else that hangs off from that PSU, and i think it will not pull that load. So, unless you use an extra PSU to power those drives, you need to be on USB connections.
1 Like
Do you know how exactly TRIM works? Is it possible that it will cause 100% increase in plotting times?
I have rather limited understanding of that process. My take is that it is kind of a forced garbage collection. It means, if you do not TRIM, after one/two plots your SSD will be full of garbage (used clusters), and every single write from that point on will need to reclaim clusters that are needed for that write. As such, after a few runs, your SSD performance should stabilize, and not degrade anymore.
Therefore, for a normal (non-plotting) OS use, it takes long days to fully “garbage” such drive.
Assuming that this is the case, I run TRIM parallel to MM, and call it two/three times during the plot (on temp1, as temp2 is RAM for me). Although, I cannot say that I see much improvements - maybe a handful of percentage points difference. Also, when I started plotting, I didn’t trim at all, and I don’t think that my plotting times improved much since i started that parallel trimming.
If my assumptions hold water, then as you said, his NVMe is potentially going bad.
As such, maybe running Cristal Disk Info would shed some light on this issue?
2 Likes
I can only speak from my experience, as I also don’t know exactly what trim does.
My first plot is always fastest, 2nd is very slightly slower, every plot after matches the speed of the 2nd plot.
Only trimming gains me that tiny edge.
If I don’t trim for a week, windows won’t do it automatically as it doesn’t deem it degraded enough.
2 Likes
Yeah, that seems to be what i see as well. Assuming that you trim before the first plot, so it runs faster. Then one run gives about 400 GB for temp1 and 1.2 TB for temp2. Therefore you fully garbage 1 TB temp1 drive after a couple of runs. The second will be full after the first.
But, your observation rather confirms my statement, that the degradation only progresses for a short amount of time (during plotting, not normal operations), once saturated that drive slows down a bit, but not that much.
Therefore, most likely we eliminated TRIM as the reason for that slow down.
Maybe TRIM is beneficial when writing a bunch of small files, or just for synthetic comparisons.
I also setup TRIM to run daily (through drive properties), but I don’t think the OS is calling it at all, even though, when I check it, it says that the drive needs to be optimized. Kind of weird.
Here is the batch file that I use parallel to MM (to trim drive d: - you can specify “de”, and it will do both d: and e: at the same time; you can also drop that -Verbose, of course):
:TOP
powershell Start-Sleep -s 600
powershell Optimize-Volume -DriveLetter d -ReTrim -Verbose
goto TOP
1 Like
Thanks bro very much for writing such a long reply. Right now my MM line is -t = NVMe, no second temp, - D is whatever storage, -r = 8, -u = 256, with -w. NVMe is on a x8 (x4) slot.
CPU is Xeon E3-1230. No on board video. So I have use a video card. Eight 3.5 hard drive in there now, six one board SATA and 2 on a cheap PCIe SATA card. Currently the power supply holds. I pulled out an old medium tower case from basement and make it a 12 drive USB external, connected to the machine. Z210 has no USB3 so I added an old USB3.0 card.
I think the the Crucial P2 grade NVMe just sucks. Need a P5 grade.
Your CPU has 4 physical / 8 logical cores. Therefore, all the work (say 80%) is done by software threads sitting on those physical cores. By using -r 8, you are putting your harvester at disadvantage. I would use -r 7, or 6 instead. That shouldn’t change too much plotting times, but would give more breathing room to your harvester (not sure, whether you monitor your partial responses, but would say that they have to be rather high during plotting). You could try to play with CPU affinities, but as that is per thread, as such it changes with every new plot, so it is a hassle trying to do it with PowerShell.
Yeah, P2 may not be the best, but Gen2 PCIe2 and therefore PCIe lines you get may be the limiting factor. I would really run Crystal Info on that NVMe to see what results you get. I have not yet seen SSD going bad, so my understanding of that process is also limited. However, I would think that it doesn’t really slow down, but rather completely goes away at some point.
Actually, thinking about those PCIe lines, maybe the reason that your plotting speed dropped is that you have added all those extra cards, and as such your NVMe is getting less of those lines, and that would potentially cause such a big drop in those plotting speeds. You are just pushing that box really hard.
I have good experience with Samsung 970, WDC Black SN750. Although, WD started switching parts for inferior ones, so I would not go with WDC anymore. I also kind of cannot warm up to use Crucial. Also, recently Samsung 970 prices got back to a reasonable levels.
Your current box is good as a harvester, but the CPU and those PCIe2 slots are kind of limiting for a plotter. I built a box based on Dell T7610. You can get it possibly for less than $300 with about 16-32 GB RAM (and a 6 core CPU). You could get a 12 core CPU for less than $100. It has enough PCIe3 slots to run your current NVMe, and extra USB/SSD cards. If you have an extra ~$250, you could get 128GB ram, and you could use it for temp2. I am getting 40 minutes plots per CPU (it is a dual socket, but of course you can use one) with that box.
1 Like
My Z210 E3-1230 16g DDR3 was $130. MVMe extra. It can do 110 minutes a plot when it was running right. So that is about 13 plot a day. For that much money I paid, I have no complain on the result. That is hardware $10 = per plot output per day. How about $200 system doing 20 plots a day? Not possible, no? 
I have a newer Ryzen 7 5600G, two NMVe -t -2. It can do 55 minute a plot. Computer cost was upper $500 without NVMe. That is about 26 plot a day.
I want to try RAMDISK but considering the price / return ratio I don’t see a gain doing so. RAMDISK is certainly cool nevertheless.
1 Like
Maybe it is 
https://www.ebay.com/itm/284149183617?hash=item42289c7481:g:cDMAAOSwgtdhSNXT
That is 20 physical / 40 logical cores.
Drawback:
- it has only one PCIe3 slot, and only available when both CPUs are installed (so plotter only, although it has USB3)
- when you put some load on it, you cannot hear the nearby jet starting
- with high load, those passive heatsinks are kind of giving up
Pros:
- it can handle 2x128GB RAM, as such will let you run two MM in parallel, ending up in combined about 20 min plots (I think)
- it may take up to 512GB RAM, so may be able to run bladeBit - not sure about that.
Misc:
- you need to run Linux when using 2 CPUs
I guess, you could try to talk to that seller, and ask for just one CPU, and that would be really close to your $200 request.
Not something to purchase for home use, but is close to what you are asking for
So, as is for $250, that box may be capable of about 50-70 plots per day. Although, it may require two NVMes in one PCIe card (as it has just one slot).
1 Like
I have multiple HP Z400’s but added USB 3.x PCI cards to handle the external drives, I tried e-sata but that was a failure at my end … anyway they don’t cost a fortune, speedwise no issues at my end
actually multiple … above is just well not a good pic
edit: crap just noticed my cat jumped on the background chair… now sleeping there
1 Like
Before you buy that P5, take a look at this article. I would stay away from Crucial, and check whatever brand you would like to get for a similar language first.
1 Like
hi Jack, I read that PCIe2 x8 (4) that you have to assume the performance of x4 and that’s mean 2gb/s.