Linux lets you set resource limits per process with the setrlimit call, or from a shell with ulimit. RLIMIT_AS caps the size of the process's whole address space, which includes its code, heap, stack, shared libraries and memory mappings. When a process tries to grow beyond the limit, allocation calls fail instead of succeeding.
Because it counts virtual address space and not memory actually in use, it is a blunt tool. Programs that reserve large ranges they never touch, such as some runtimes with a JIT, can hit the limit even though real memory use is small. Setting it too low breaks them.
For a service such as PHP-FPM it can still be a defense: a bug that makes a worker try to allocate huge amounts of memory fails sooner, and in some cases that changes whether a memory flaw can be exploited or only crashes the worker. It does not fix the bug.
Use it as a backstop alongside a cgroup memory limit and the application's own limits, and test under real load before applying it to production.