Reddit mentions of Operating Systems: Internals and Design Principles, Global Edition by Stallings, William (2014) Paperback

In short, the answer is virtual memory and the protected mode flat model that the past generations of CPU architecture and operating systems have been using.

As you may know, programs are never given full, direct access to the RAM banks; the operating system abstracts this layer away from them in the form of virtual memory. Virtual memory is basically a system whereby you can map physical memory addresses to non-physical ones the OS controls and can readily re-arrange. Thanks to virtual memory the OS can trick an application into thinking it has way more RAM than it actually has and this also enables swapping processes out to disk when the system is running out of memory because there are too many processes being run at the same time. As I pointed out before, since virtual memory is fully managed by the kernel, it can move out chunks of a program's address space to disk, a process known as "paging".

Now, back in the DOS era, virtual memory followed the real mode segmented model, which, in very simple terms meant that, even though processes could be shuffled back and forth between RAM and disk, there were no safeguards in place to prevent a process from messing up another process' memory space via a dodgy pointer pointing to a memory address beyond the scope of the faulty process.

One of the major goals of the successor to this virtual memory model, "protected mode flat model" was to allow the kernel to create a completely isolated address space for the processes it spawns and stopping a rogue program from altering other processes like before. Whenever such an attempt is made, a "segmentation fault" (SIGSEV) or "general protection fault" is raised by the kernel, which in the case of Linux, will prompt the kernel to swiftly kill the offending process.

In practical terms, this means your application won't be able to reach beyond the virtual memory address space it has been allocated (unless it's a kernel-space process, like a kernel subsystem or device driver) and can in no way communicate with other processes by reading or writing memory that belongs to them. In order to accomplish that, you'll need to make use of inter-process communication (IPC) techniques like Unix sockets, pipes, RPC, etc.

This is by no means an in-depth or 100% accurate explanation though. If you've got any follow-up questions I'm more than happy to answer them.

As for the literature, pretty much any textbook about operating system architecture will cover virtual memory to a great extent. I can recommend Operating Systems: Internals and Design Principles and Modern Operating Systems.

Here are a few more books that touch upon the topic of virtual memory:

Assembly Language Step-by-Step: Programming with Linux covers the topic of virtual memory and the different models that have evolved over time over its first few chapters.

The Linux Programming Interface: A Linux and UNIX System Programming Handbook covers this subject (and many, many more) from a Linux systems programmer perspective.

What Makes It Page?: The Windows 7 (x64) Virtual Memory Manager, in case you're interested in learning how Windows does it.

EDIT: added IPC info