Limit Disk I/O In Proxmox

ZFS by default will use quite a bit of ram for its ZFS ARC, if I’m correct up to 50%.

ZFS is great for certain use cases but I really wouldnt recommend it on a low memory server where you need to use as much of it for VM memory as possible.

You could try and tune your ZFS arc settings. But your also have a negative affect on disk performance but atleast wouldnt be running the server to the near point of being hit by OOM.

If not and your in a postion to backup and restore I’d suggest going to a standard mdadm Raid array.

2 Likes

Thanks for the reminder, using ZFS on a 2GB KVM is a terrible idea.

3 Likes

EXT4 or XFS, we all know :wink:

@Falzo do you use ZFS on your Proxmox installs?

I’ve always stayed as far away from ZFS on Proxmox. One thing you might want to do is remove deduplication unless you really need it.

no, not really. I use ZFS on one node, but only for a secondary set of harddrives using an ssd as caching device. I think that was more of an experiment :wink: anyway, so far no troubles with that.
I usually use just plain sw raid 1 or 10 with sparse disk images, not even thin lvm…

1 Like

Same here, last I remember ZFS was using 16 GB of RAM at boot just for being ZFS.

1 Like

@imok did you make any progress with fighting your troubles?

Not yet, I need to migrate 2 VMs before breaking the server.

This is for one

I’d suggest getting away from ZFS, as a trial.

If you need some temporary hosting space, let me know.

3 Likes

Moved 2/5 :dizzy_face:

LOL, you will get there. I’ve moved VMs many-a-time.

1 Like

Correct ZFS by default uses 50% of your system RAM.

Rule of thumb is you need 1GB RAM for every 1TB storage.

You can set limits on how much ZFS is allowed to use by visiting:
https://pve.proxmox.com/wiki/ZFS_on_Linux

Find on the page the section defined as: Limit ZFS Memory Usage

3 Likes

As for swap space, swapfiles running on ZFS doesn’t play too nicely. What you should do instead is when you install Proxmox is use the advanced setup, on that tab at the bottom (forgot what its called) but the bottom box allows you to set some free space at the end of your drive. Add however much space you want reserved for swap then continue installing the system as normal. After the machine is online, just create swap on that unused freespace like you would usually do (gdisk, mkswap etc). This will give you the best performance and stability.

You can read more on what I said here and search for SWAP on ZFS:
https://pve.proxmox.com/pve-docs/local-zfs-plain.html

4 Likes

Tune the ZFS ARC. I use 4-6GB on 32GB hosts.
zfsonlinux uses 50% of host RAM by default.

Next, MOVE/disable all swap on zfs zvols. (the default installer option I think)
ANY system(hypervisor) swap ‘should’ be on non ZFS volume for safety reasons. (explained later)
The newer proxmox installers provide a feature to ‘leave’ some unallocated space after the ZFS partition marker. Make a linux SWAP partion in that space and use that instead for the hypervisor linux system.

Having swap on ZFS during a critical OOM situation can cause catastrophic data loss.
I think this was the issue that alerted me to it https://github.com/openzfs/zfs/issues/7734
It’s been quite a while, there may have been movement on the bug.
But I doubt it due to the nature of ZFS design. Almost every operation in ZFS involves some memory allocation ops. So, an OOM is very bad thing for swap-out.

None of this applies to Guests. SWAP or no-SWAP is up to you there.

Better to just give enough MIN and big MAX ram and enable Balloon driver like Falzo mentioned.

Edit: Also https://github.com/openzfs/zfs/issues/342

2 Likes

Is it safe to put options zfs zfs_arc_max=8589934592 and run update-initramfs -u while the RAM usage is at 80%?

Depends what your RAM is for one, 80% of 1TB is a lot of ram left :slight_smile:

So remember thats 8GB and you only need that if you got 8TB of storage? do you?

Also keep in mind with the change you need to reboot as well.

2 Likes

I forgot to change the number from the docs. 1073741824 it is.

Currently using 23 of 32GB. That looks like a great amount of free RAM.

If you got around 1TB of storage then 1GB RAM is what you ideally need set as max which is what that number reflects. And tbh you prob find you are really using more then that right now for ZFS as it would be using 16GB (max) on that system.

2 Likes

The way I understand it is, 1GB would be the bare minimum for ARC_size per terabyte of addressable storage.

Additional ARC size functions as a hypervisor wide disk page cache. I get ssd-cached-like performance in my lxc containers (backed by zfs datasets by default in proxmox . KVMs are backed by ZVOLS; slightly different internally)

I say 4GB ARC_size and a minimal 256MB swap inside guests.

You want guests to take advantage of ZFS storage magic.

3 Likes