RAM and SWAP were at full a minute ago. Where to start to diagnose the problem?
Exactly where you already are, htop
I couldn’t see anything relevant CPU and RAM seemed fine
Well… I’ve found a hacked account. An user with a Wordpress site and all of the themes injected with malicous code (no idea why he had a lot of themes there)
users being users…
“Free” so why not.
A lot of people just can’t resist the urge to install everything… and never sucumb to the need of actually choose what they need and uninstall what they don’t need. It’s a problem that won’t ever go away.
You should always be able to identify the most resource intensive websites from top/htop, since either the process will be running under that user or the $CWD of the process will show the username.
Install iotop and see what’s maxing out your I/O load.
Seems like I/O issues, run top and look if you see i/o wait (wa)
Kill the server, as in shutdown the webserver and php processes. That memory usage is causing massive IO thrashing to move things inbetween ram and swap. Once you have done that, and can see the memory usage return to normal then diagnose further.
If you are running this on a vm, try restarting the server. Sometimes the IO threads lock up.
Server returned to a normal (almost idle) state after some minutes. I cleaned the hacked account but I don’t know what the malicious code was doing.
Do you have a backup of those hacked files? Post them on github or somewhere and we can take a look.