Plotting efficiency, queues, Swap memory and performance questions



image
image

Hi Team, here is my rig:

i9-10900F
Asus ROG Strix Z490 e Gaming Mobo
Team T-Force Zeus DDR4-3200 64GB
2x WD Black SN850 2 TB each
Western Digital Gold 10 TB 3.5" 7200RPM Internal Hard Drive
RMx Series™ RM850x
Phanteks Enthoo Pro
CoolerMaster MasterLiquid ML240L RGB V2
Kingston 240GB A400 SSD 2.5’’ SATA 7MM 2.5-Inch SA400S37/240G for OS
ASUS GeForce GT 1030 LP 2GB (GT1030-2G-CSM)
Running on Ubuntu desktop

Questions:

Disclaimer: My plots are not timed well, I am still trying to find the most efficient gap window at the moment.

  1. Why is my Swap memory usage at 100% while there is plenty of unused actual RAM? Is there a fix? Should I be worried about this in the sense of damage to hardware or potential crashes?

  2. My Nvme SSDs are only at 50% utilization even though I am plotting 10 plots at a time. Is that normal? Does that mean I can plot more than 10 plots at the same time if I had higher thread count CPU?

Please let me know if you have any questions or need more info. Thank you!

1 Like

Bumping it up again to seek help. Thank you!

Bumping it up again to seek help. Thank you!

Swap on linux is controlled by vm.swappiness. Default value is 60, this means when 40% of real memory is used, some will get swapped out to configured swap files or swap partitions, the intent is to keep real memory available, and swap out whatever is less used, I don’t personally think swappiness of 60 makes sense with modern RAM sizes, because it is unlikely (although not impossible) that some imminent process is going to instantly need say 40GB of RAM, however this is just a default, easy to change and I recommend changing it.

You can check your current swappiness like this:
cat /proc/sys/vm/swappiness

You can set it temporarily like this (will be set up to the next reboot):
sudo sysctl vm.swappiness=5

Or permanently by adding this line to /etc/sysctl.conf
vm.swappiness=5

and then running this to set it to whatever is in sysctl.conf:
sudo sysctl -p

My preferred vm.swappiness value is 0, this special value means the kernel will only swap when necessary - it does not disable swap! I do not recommend turning swap off completely (although that’s also easy to do), because it is useful to have some headroom if you do have a runaway process and want to be able to interact with the system (even though it will be extremely sluggish if it actually needs to swap).

4 Likes

Thank you so much @aschen. I will try the different swappiness settings and see if it helps in terms of performance.

Hi, I did what you recommended here and set my swappiness=0.
I use BpyTOP for monitoring my system and I don’t see any changes in the Swap section. I have the Swap in 1.99 Gib with 88% used.
Is this OK? or I made some mistake in the configuration /etc/sysctl.conf?
If I check with cat /proc/sys/vm/swappiness returns 0.
Thanks!

Swappiness=0 isn’t turning off swap, it’s just indicating that you’d rather it not happen, the Kernel is still free to decide to do it, I’m not 100% sure why it sometimes seems to do it seemingly unnecessarily.

If you want to actually turn swap off, try first without rebooting:

sudo swapoff -a

This is safe to do on a running system as long as there is space in RAM to move everything currently in swap back.

To make this permanent you need to find and remove any swap entries from /etc/fstab (that is, lines with swap in the type column), and reboot.

Ah also, if you just change the swappiness on a running system, that doesn’t mean it will move everything currently in swap back to RAM - it just means you shouldn’t see nearly as many new allocations in swap. You’d need to compare 2 nearly identical tasks running from the same starting state with each setting to see the difference.

Personally, I get comfortable behaviour with swappiness=0, I still see some big processes occasionally have a bit of swap, but much less than with the default =60.

1 Like

Ok. I will let it lay with this setup (swappiness=0). Probably the process in the swap right now are not necessary to future plotting. So I can expect some better performance.
Am I right?