Solaris: How to limit ZFS ARC cache maximum size

Written by Claudio Kuenzler - 2 comments

Published on February 28th 2014 - Listed in Solaris Unix ZFS

---

One of the reasons why ZFS is a fast file system is the use of a file system caching called ARC (Adaptive Replacement Cache), which runs in memory. 

The current size of ARC can be seen with the following command:

kstat -p zfs:0:arcstats:size
zfs:0:arcstats:size     12310432872

At the time of this output, ARC was using 11.4GB.

Although ARC should be intelligent enough to not occupy RAM which is needed by zones or applications, I have seen cases where ARC uses 50% of the total available memory and the other 50% were fully occupied by zones and applications, causing applications (like MySQL) to swap. That's a performance nightmare.

But there's a way to limit the ARC size by setting the value of zfs_arc_max. At the end of /etc/system I added the following lines:

* Limit ZFS Arc
* http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache
set zfs:zfs_arc_max = 21474836480

This means that I have set a maximum limit for the zfs arc to 20GB (21474836480 Bytes).

After a reboot (and RAM-Upgrade at the same time), the memory graph clearly shows that the ARC size (represented by CACHES in the graph) has hit 20GB, but never went above it:

Add a comment

Show form to leave a comment

Comments (newest first)

ck from Switzerland wrote on May 27th, 2020:

Hi Jodi, your current action plan wants to set a minimum of 6GB and a maximum of 10GB for the ARC cache. Given the total memory capacity (256GB ?), I'd go for for a maximum of 50GB for the ARC cache. I havent worked with Solaris in the last couple of years but I kind of remember that the ARC cache should never be lower than 20% of the whole memory capacity - or you will get into file system performance problems. 50GB is around that 20% level. So you give Solaris the possibility to increase the ARC cache up to 50GB / 20% of the total system memory. Maybe this even needs to be set higher but you can start trying with that value. I would set these values in /etc/system:

set zfs:zfs_arc_max = 53687091200
set zfs:zfs_arc_min = 10737418240

---

Jodi from wrote on May 26th, 2020:

Hi Claudio

Thanks for sharing this article

i have memory performance issue in Ldom where ARC uses 60% of total memory and the other 10% were used by zone and oracle db,

with the following output grep by command ( zfs used 145Gb of ram and apps only 71G )
# echo "::memstat" | mdb -kw
Usage Type/Subtype Pages Bytes %Tot %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
ZFS 19061406 145.4g 55.0% --> zfs cache
User/Anon 9322025 71.1g 26.9% --> Application

also current memory setting as below :
- Ldom : ram=290g,swap=290g
- zone : ram=256g,swap=256g ( shared )

do the action plan, i adjusted the /etc/system file to add the followeing parameter and reboot os :
#vi /etc/system
set zfs:zfs_arc_max = 10737418240
set zfs:zfs_arc_min = 6442450944
:wq!
#reboot

kindly let me know if the above action plan is correct ?

Thank you and really appreciated for attantion and response

Best Regards
Jodi | unix support

---