He has several videos about SAS, and yes, some are about SAS3, so I would just scan through some just to get an idea what worked for him (and should work for you). Based on what the videos that I watched (basically all) it should work well. (Basically, I was preparing my plotter to run off SAS drives, but XCH price went just too low (for me) to justify purchasing new drives. Although, I did run some tests on 4 SAS/SAS3 drives, and it was not that bad. From cost perspective, it blows everything else out of water.
I set a pretty high Ram allowance and it doesn’t seem to have been using anywhere near that. Though the logs reference the value. e.g. Bucket 126 uniform sort. Ram: 31.190GiB, u_sort min: 1.125GiB, qs min: 0.281GiB.
In reality, utilisation was 3.3GB out of a possible 50GB. At least for the first phase and last phase when I had a chance to review. I left it running overnight.
I also tried MaxMax but it ignored my temp dir (-t) and filled up my OS SSD. So I cleared that and used the above.
Yes, that is slow. Although maybe be due to chia default plotter being used (mostly is using just one physical core). I would forget about the default plotter, as it is rather worthless and focus only on MM. Start with one MM instance to get comfortable and move on to two and fine tuning when you will be ready.
I checked the specs of your setup. You can go up to 144 GB DDR3 RAM. I think that @RobbieL811 have just purchased rather inexpensive one recently (but a bit faster than yours - not sure it can be used to reach 1,600 MHz, though). I would upgrade your box to 128 GB of 1,600 MHz (if your 50 GB is slower, I would dump it). This way, your temp2 will be RAM, and your SAS temp1. Also, your box has 12 physical cores that run at 2.6 GHz, what is actually good. Although, those processors have rather small cache, what could be a limiting factor. I cannot tell how fast your box will be but would assume somewhere around 1 hour per plot or so.
Of course, all that depends on how many plots you want to generate. Still, the only upgrade you will need to make is to get that RAM (around $200 or so), and that is way less expensive than getting any other plotter.
Lastly, I didn’t plot for quite some time, so don’t really recall my MM settings (should be somewhere on this forum, I think), but you may want to search the forum to get the right thing (or someone will chime in).
Yes, the memory usage you mentioned is about right. Neither plotter is using much RAM. Where you need it is your temp2 folder (for MM).
I did recently purchase some new RAM. 32GB LRDIMMS @ 1866. Roughly $50/DIMM. Idk why, but for some reason they were a good bit cheaper than the 1600 ones. No idea why. To OP, or anyone else that might be interested, I will have about 18 16GB Micron RDIMMS to get rid of this week. They are pc3l-10600R @ about 1333 MHz. I currently set manual speed in my bios and run these @ 1600MHz without any issue. They will run at 1866 also, but I had trouble with reboots when running 2 MM instances @ 1866 MHz. They ran a single issue at 1866 without issue though. Idk. Let me know if you’re interested. I’ll let them go at what I think is pretty cheap. $20/DIMM shipped as long as you’re gonna get a few of them.
Sorry, I have never used MM that is built in to chia, I only used stand-alone version.
Here is an example (Win, but shows what is needed, and you can omit “-2 RAMDisk” for now). Just adjust the number of plots and cores to what you have.
As far as your line, I don’t understand “/media/computer/Chia/Chia\ Farm” You are using Linux, so no "" in the path and no spaces (unless you quote the whole path) there. I don’t know what “Farm” stands for.
In Linux, there are no spaces. You have to prelude a space with \ or you can notate it like so with quotes. “Chia Farm” or ‘chia farm’. the backslash just tells the CLI that there is a space, or special character.
Saying that, could you quote that temp path. MM is rather rock solid (at least the stand-alone version), but maybe it has a bug around parsing temp folder (kind of doubt that, though). One explanation why it ended up plotting in your OS drive is that for whatever reason couldn’t get temp folder right.
Or just for one run make your -d the same as -t (you will end up with just that one file in that folder).
Yeah. It is. Might be a qwirk with MM or something. Are you doing this in CLI? “Chia Farm” should be equal to Chia\ Farm. If you’re entering these into some sort of GUI or something might mess things up. Anyways, if Chia\ Farm works, then just use that. Same thing either way.
just to see whether that will bypass your OS drive.
Maybe in the built-in version there is a need to specify -2. Also, I doubt that there are too many people that run MM without providing both -t and -2, so if there is a bug there, it may be just not tested.
Actually, when you start MM, I think it dumps all the folder names that it will use, so no need to run through the whole process.
@Jacek I arrived on the exact same conclusion. It’s running now Dir in logging seem to be as I have defined. Will report back on speed in (hopefully) a few hours!
Would be really nice if you could post it on github Issues page, so Max could take a look at it (and fix, if there is something lingering there). Maybe he didn’t expect to not have -2 line, as that is where the MM power is.
Could you show your MM line? I mean, I would really start with defaults, just to get the base line.
That is about 1/5 or 1/6 what you had before. A good start. Did you monitor your resources to see what is choking?
Not sure, whether you checked Janis’ videos for OS tuning. Maybe now is a good time to follow his suggestions.
Few ways to move forward.
Get 128 GB RAM
1.1. Your box cannot accept 256 GB, as such you are kind of limited to one MM instance
Get one NVMes and use it for -t, and your RAID for -2
2.1. -t load/wear is about 1/4 of -2 folder
Potentially break down your RAID in two RAID0 sets, where -t will be just 2 drives, and -2 six drives
Not sure if or how this can be done, but with your existing setup, maybe you could use 40 GB RAM as a cache for just -2 directory (just make sure that -t and -2 are not the same directories, but can be on the same RAID). I have never looked into that, but maybe you can disable cache support for -t, that would give all cache for -2, but that option (if possible) may require two partitions (one for -t and one for -2)
Basically, MM doesn’t like to have -t and -2 pointing to the same place, as that media bandwidth will be the choking factor.
He should be fine to run 2 MM instances on 256GB. That’s what I have in my server now and it’s working a treat. My machine maxes out at 256GB with RDIMMs. I didn’t know this when I originally started buying RAM of course! I have to use LRDIMMs if I wanna get any above that as I’m sure it is with most servers. But that’s the setup I’ve got now and it runs good.
Dir in logging seem to be as I have defined. Will report back on speed in (hopefully) a few hours!