Was just trying to imagine what it will take to offload plots to long term storage when using a GPU to create sub 10 minute plots. Currently I’m getting 22-23min plots with my R620 and its taking 20-21 min to transfer the plot to my Netapp DS4246’s. Currently I have 10gb connection between the two, the limiting factor is write speed to a single drive at 84MB/s avg. So my question is what can be done so there won’t be a backlog at the plotter when the GPU plotter goes mainstream? My current setup is 2 R620 Plotters, 384gb memory each and they both direct connect to another R620 as a head unit to my 2 DS4246 Disk Shelves. My write speeds seem a bit slow, still need to chase that down. Anyways just curious on how you plan to setup your offloading from GPU rigs?
That is still TBs per day and the old plots will still work
I don’t have a solution to the bottleneck that you have. But honestly, the jump to the next plot version is not like starting from scratch. The plots you have will still be farming while you plot new ones to replace them. So there isn’t a huge rush like there was when starting from scratch.
I will be replotting when it comes time. But I will just use my current plotters and use my scripts to replace them one at a time.
That is currently my plan as well to replace each plot with my current plotters, I won’t be in any hurry. It just got me thinking about what might be the best way to handle that much output. I guess you could use a script to write one plot to one drive, and the next plot output to another drive.
You could just move to multiple disks in parallel.
I am using a little bash script that takes plots from a temp folder and moves them to the next target with free space and no running move operation.
The script can be adjusted to automatically delete old plots too.
I used that approach to re-plot from Chia to Chives and back.
Fill drive, move drive, will be how i get this done.
Something is wrong w/ your setup if your bandwidth is that slow to each drive. Unless your drive is horrible and broken it should do 160MB/s
You could directly connect them, brr, and then slot them after.
You could write a script that checks for free space before blasting a plot.
You could ask for a script to be written etc
will k33/k34 plots be available for plotting at less size as k32?
I agree something isn’t right with my current netapp setup, or maybe my test drive I’m working with. Because I tested writing directly to a SSD drive and was able to transfer a plot in 3 min. Before it was pretty much a don’t care on the transfer time because plots were taking close to an hour or more to create so there was plenty of time to copy. I’m currently fully plotted, just trying to make sure my setup is ready to replot when the time comes and finding the weakest link.
Normal HDD does average over 150MBs with peak 200+.
In 3-5 years when k33 may come…SSD suppose to replace HDD. Any SATA based SSD can do >300MBs.
You should worry why your storage system does only 84MBs…do you have PATA HDD or what?
morning,
can you post your R620 config?
Which CPU. How many. How much RAM and which one?
To your question:
I’m not quite sure… (check the docs)… but I think there was an option where you could save the finished plot and only then send it over the LAN.
Even if not… following solution.
Get an SSD with a high TBW. I’ve had good experiences with the Segate 520. They are inexpensive and have 1800 TBW.
Should be enough for around 18,000 plots.
Your target directory is then the SSD.
Then you write a script that checks every minute whether there is a plot file and then pushes it on from there.
Problem here: The SSD will gradually fill up because the plotter is faster than shifting.
1 TB SDD = approx. 9 plots.
You will only slightly shift your problem backwards… but not completely solve it.
My question was more on how other people plan on handling the fast rate of plotting when using GPU. They need to think about being able to off load that many plots and having enough bandwidth. Otherwise what is the point of plotting fast if its waiting to offload.
I currently have two Dell Poweredge R620 plotters, each has 2697 v2 processors, each has 384Gb ddr3 ram. I use Madmax and plot all in RAM. Each one running Linux has 23 minute plots on average. So in total I get a plot every 11-12 minutes. They are direct attached to a Third R620 with single processor and 96 Gb ram through 10gb sfp+ dac cables. This is my full node. This then attaches to my 2 netapp DS4246’s where I have 576Tb of long term storage. I have 7 free SFF slots in the Full Node R620 that I could add SSD’s as a buffer if I decide to play around with the GPU plotter. But right now I’m happy with my current plotting speed.
I only see one issue. If you get 12 minute plots let’s say, then 12min x 60 sec = 720 sec x .150 gb/sec = 108 gb. So every 11-12 minutes approximately you could store a plot @150 mb/sec to a drive
I realize you are not getting 150mb/sec transfer to your drives. The simple answer is you need to find out why and fix that. 150mb/sec is not fast, on a scale of writing to hard drives, but it not a terrible average to get. So that should be your 1st priority I would think, then you will not need to do any further upgrading or changes other than perhaps to use a small SSD as buffer between the drives and your plotter.
did you test it ?
is that faster then others?
Oh I think I found my issue for the slow transfer rate of 88Mb/sec, its a certain brand drive causing the problem, possibly SMR drives from something I shucked, but I haven’t looked them up yet. Luckily I only have two of those still in use, I’ve been slowly upgrading drives 8tb and smaller to at least 18tb when I can find deals.
@root87, No haven’t tested any GPU plotting yet. Still running a Nvidia 1060 in my windows system, but have been thinking about upgrading it lately to a 3060ti.
is that gpu plotter ? are you sure ?
False jeopardy. There is no need for fast plotting.
I hear you on that. I debated packing a small JBOD with Barracudas because the price was right when I got into this ($15/TB or so new https://www.newegg.com/seagate-barracuda-st8000dm004-8tb/p/N82E16822183793?Description=8tb%20baracuda&cm_re=8tb_baracuda-_-22-183-793-_-Product&quicklink=true). Now that power rates are going up around the world, there’s less of an incentive to hang on to the smaller SMR units with future rig changes/upgrades.