Limit Disk I/O In Proxmox

Found a citation in the new Proxmox 6 Admin guide as well.
Section 3.8.7
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs

3.8.7. SWAP on ZFS

Swap-space created on a zvol may generate some troubles, like blocking the server or generating a high IO load, often seen when starting a Backup to an external Storage.

We strongly recommend to use enough memory, so that you normally do not run into low memory situations. Should you need or want to add swap, it is preferred to create a partition on a physical disk and use it as swapdevice. You can leave some space free for this purpose in the advanced options of the installer.

… and thats reason why I said adjust the partition when your installing (a few posts up) and leave some space at the end of the drive for “real swap” if you need it rather then doing on ZFS.

1 Like

Hi!
I moved the VMs I could move to another server and there is a lot of free RAM. I still have disk IO problems here. Some examples:

  • YABS run on host or guests, takes down the whole node.
  • Inside a cPanel VM: I need to restore an account backup and the VM becomes unresponsive.

Anyone would like to take a look? PM an estimated price please, I don’t know where to look anymore. You must be a well-known member from HostBalls.

I think the problem is in writing stuff to disk. I was able to migrate data using rsync and installing OS from scratch for the old VMs.

For the VMs that are still running, I want to move them entirely using Proxmox backup, and for that I need to write to disk, which is not possible.

What storage type are you using on your server?

2 mirrored SSDs using ZFS.

What kind of tests should I run to discard hardware problems?

how’s your situation right now with swap/noswap on the host? see @vimalware comment above … if you added swap (which in general is a good idea) maybe this is causing the trouble now if it’s on zfs.

so maybe deactivate it again, just for the purpose of being able to get your backups done.

PS: once you moved out everything probably just set up the whole thing without zfs… just raid1 ext4 done.

Yep next time ext4 for sure.

Server has no swap and it does not have memory problems now. It’s past midnight right now so I powered off the cPanel VM and I’m taking a backup, just for testing but it’s very slow. Slow but host load average doesn’t go to 60 and not down.

Great! Backups will take about 3 hours of downtime but at least it works.

Now I have to find another machine for temporary use in case of disaster.

This server has around 250GB available to use, but VM disks are configured for 550GB in total (strange). Is it possible to shrink the disks? Is it safe?

If you’ve got backups you could boot from a gparted iso and try to shrink the partition size.

Edit, I have done it before without data loss but obviously ensure you have a copy just incase.

1 Like

you can’t shrink the images itself easily, I would rather consider it dangerous. as @Mr_Tom I have done it before, however even with shrinking the partition inside beforehand, I had cases where shrinking/cutting the image afterwards broke it.
so whatever you do a) have backups and b) only remove the old image after you made sure the new one works.

ideally you can do as you planned - move everything out to some other place temporarily and reinstall the box without zfs, then move your stuff back.

is this in US? I might be able to help you out with something… how many VMs are left (e.g. how many IPs would you need?)

Two VMs with 1 IPv4 for each one.

A. 240GB disk, using 10GB LOL.
B. 300GB disk, using 60GB.

I’d like to take backups, move them to a temporary server and start the VMs to check they work as expected (not sure if I need around 540GB or 70GB), then reinstall my Proxmox server and restore de backups.

If the problem is the hardware I won’t be able to put everything back, so I will probably need to stay in the temporary server some days until I get a new dedi.

if the images are sparse (which is default) then they won’t use/need the full space on the destination. depends a bit on the filesystem used in the guest and if you use discard etc.
you could manually run fstrim -a on each guest before taking the backups.

I have room on a dedi in charlotte (hi @qps ;-)) if you want to be my guest, doesn’t matter if it’s for a day or a month. disk size shouldn’t be a problem, RAM might be a bit tight but I guess, we can check what you need and take it from there :man_shrugging:t2: :wink:

2 Likes

@imok

You should be able to enable the discard flag, under the harddrive(s).

Then just enable the fstrim utility for your OS, and it will automatically shrink the VM to the current usage of the server.

What if this a physical RAM problem? How to test it?

Well… this is finally resolved but in an unexpected way.

I found a new provider and I’m happy with them. Almost same monthly price and better hardware (CPU upgraded from E3-1230 to E3-1270 v3). No more disk I/O problems.

The old provider asked me to pay double the price for a new machine instead of replacing the disks.

I guess we never checked if your SSDs were having issues?

No. I could not find the time to run the proper tests, I was thinking in paying an additional month to remove my doubt but I got angry when they refused to replace the disks unless I pay 2x.

They called an additional 105% “a little bit of an increase in price”.

That seems really rude of them. It might have been the SSDs were the issue and the drive was repairing itself. I’ve had something similar and the performance has massive and random hangs.

Was fun when I found out the issue and reported it to my then boss.

This has been going on for a while, but it’s similar to what happens to me when I try to restore the backup of a VM; it takes a long time and has high I/O. I use ZFS and Linstor with a 500 MB/s SSD.