Here is the first experiment. I setup the new system last night (specs in first post) and ran a single plot with a 2nd temp drive configured. I then spent the next 5 hours watching it. I watched the folders and I/O on each folder. Here is what I found. (a lot of information, maybe too much, but I just wanted to document for my own sanity. Posted here for anyone that may be interested.)
From this point forward, I will refer to the 1st temporary folder as temp1 and the 2nd as temp2.
PHASE 1
When the job first starts, files start to appear in Temp1. At the same time, one file appears in Temp2. The file in Temp2 looks exactly like the final plot file in its temp state (extension .plot.2.tmp). It has 0KB size. Besides the creation of this file, there is no activity on temp2 during phase 1. During phase 1, the number of files in Temp1 climbed well over 200, I didn’t see the highest number. But then it started to decrease. It appears that the large number of files that are created are working files and then there are the plot table files. There are 7 files with the extension of .plot.table#.tmp and one file with plot.sort.tmp. The working files which have the extension .plot.p1.t#.sort_bucket_#.tmp seem to come and go as needed (there are 127 of these files per table) and the sort/table files grow during phase 1. The .plot.table1.tmp is the first to grow. Then table2 and so on all the way to table7. The largest I saw the folder get during this phase was 162GB. Here are the final table sizes.
table1 = 14.506 GB
table2 = 21.759 GB
table3 = 21.760 GB
table4 = 21.760 GB
table5 = 21.761 GB
table6 = 21.763 GB
table7 = 41.954 GB
PHASE 2
All the phase 1 working files disappeared and then files with the extension .plot.p2.t#.sort_bucket.tmp appeared (the first set were for table7). The file on temp2 still has a size of 0KB at this point. Then a short time later all the working files disappeared again. And then working sort files for table 5 & 6 showed up. It worked its way down through the tables. After the working files populated for each table, the original table file would drop to 0KB. It did this all the way down the list of tables. But tables 7 and 1 did not reduce in size. Disk activity on Temp1 (viewed with Task Manager) dramatically dropped when it entered phase 2. In phase 1 it was jumping around 20-30%, in phase 2 it is down around 2-6%. The largest I saw temp1 go was 236 GB. It made tables 2-6 0KB. Tables 7 and 1 stayed populated.
PHASE 3
New working files showed up on temp1. A second set of working files for table two. This time with .plot.p3.t2.sort_bucket_#.tmp as the extension. And for the first time, the file on Temp2 gained some size. 1KB. It begins to write from the .plot.p2 files into the .plot.p3 files. The .plot.p2 files disappear as they are written over. The sort file still has 0 size. As it finished with table 2, table1 finally went to 0KB. The size of the Temp2 file is climbing now. During this time, some new .plot.p3s.t2… files showed up (maybe s for sorting). As these files get processed the file on Temp2 grows. Currently Temp1 is at 211GB. The only table file left with any size is table7 and it is still at 41.9GB. The .plot.sort.tmp file still has no size. During this time, the read traffic on temp1 has spiked and the write traffic on temp2 is spiking in waves to about 30%. But not often. While all this is going on, table7 is still the same size as it was at the end of phase 1. As it finishes processing the .t#s files for each table, the size of the tmp file on temp2 grows. As it is processing .p3s.t6 files, the working files for table 7 showed up (both .p3 and .p3s). Basically it spends phase 3 sorting tables on temp1 and writing the output to the tmp file on temp2.
PHASE 4
There is still 71GB of data in temp1. The table 7 file is still there (41.9GB) and all the working files for table 7 (the .p3s.t7 files). The file on temp2 is now at 105.272 GB and temp2 now has decent I/O at this point. I/O on temp1 has dropped to almost nothing. As phase 4 progressed, the files in temp1 slowly started to disappear. As they did, the file in temp2 grew to 106.3GB.
PHASE 5
Then phase 5 started (the transfer to final drive).
So basically, during phase 4 and 5, temp1 I/O is eliminated and is offloaded to the temp2 and I’m sure there is some load removed in phase 3 because it is not writing that final tmp file to temp1.
Interesting observation, the file with the extension .plot.sort.tmp never did anything. I never saw that file have any size at all.
I never saw temp2 I/O anywhere near maxed out. The highest I saw was 30% and that was in short spikes. During the final transfer, it ran steady read at 21% (120MB/s Read) with the ethernet maxed out (1Gbps).
Now I am starting Swar’s with exactly the same setup as my first plotter but with the 2nd Temp Folder added. I will let it run for a couple days and get settled in. I will then report back to see if there is any performance improvement.
