shared memory ipc
POSIX Shared Memory and Memory-Mapped Files are the inter-process communication (IPC) mechanisms that allow multiple processes to access the same region of physical memory — providing the fastest possible data sharing between processes on the same machine (zero-copy, no kernel involvement after setup), essential for high-performance computing, database engines, ML inference serving, and any application where microsecond-level IPC latency matters.
Shared Memory vs. Other IPC
| IPC Method | Latency | Throughput | Complexity |
|---|---|---|---|
| Shared memory (mmap/shm) | ~100 ns | Memory bandwidth | Medium |
| Unix domain socket | ~1-5 µs | ~5 GB/s | Low |
| TCP/IP (localhost) | ~10-50 µs | ~2-5 GB/s | Low |
| Pipe/FIFO | ~1-5 µs | ~3-5 GB/s | Low |
| Message queue (POSIX) | ~5-10 µs | ~1-3 GB/s | Medium |
POSIX Shared Memory API
#include <sys/mman.h>
#include <fcntl.h>
// Process A: Create shared memory
int fd = shm_open("/my_shm", O_CREAT | O_RDWR, 0666);
ftruncate(fd, 4096); // Set size
void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0);
// Write data
memcpy(ptr, data, sizeof(data));
// Process B: Attach to same region
int fd = shm_open("/my_shm", O_RDONLY, 0666);
void *ptr = mmap(NULL, 4096, PROT_READ, MAP_SHARED, fd, 0);
// Read data — zero copy, same physical pagesMemory-Mapped Files
// Map a file into memory
int fd = open("large_dataset.bin", O_RDONLY);
void *data = mmap(NULL, file_size, PROT_READ,
MAP_PRIVATE, fd, 0);
// Access file as if it were memory
double value = ((double*)data)[1000000];
// OS handles page faults → loads from disk on demandKey Differences
| Feature | shm_open (POSIX SHM) | mmap (file-backed) |
|---|---|---|
| Backing | tmpfs (RAM only) | Filesystem (disk/SSD) |
| Persistence | Until shm_unlink or reboot | Persistent on disk |
| Size limit | Available RAM | Disk space |
| Use case | Fast IPC | Large dataset access, persistence |
| Survives reboot | No | Yes (file persists) |
Synchronization
- Shared memory has no built-in synchronization → multiple processes can corrupt data.
- Solutions:
- POSIX semaphores: sem_open/sem_wait/sem_post for mutual exclusion.
- Atomic operations: Lock-free algorithms using __atomic builtins.
- Futex: Fast userspace mutex (Linux) → no syscall in uncontended case.
- Reader-writer locks: pthread_rwlock in shared memory region.
ML / Data Pipeline Usage
- PyTorch DataLoader: Workers use shared memory to pass tensors to training process.
- Ray / Plasma: Object store backed by shared memory → zero-copy tensor sharing.
- Inference serving: Model weights in shared memory → multiple worker processes share one copy.
- Redis: Uses mmap for persistence (RDB/AOF), shared memory for module communication.
Huge Pages for Performance
- Default: 4 KB pages → many TLB misses for large shared regions.
- Huge pages (2 MB / 1 GB): Fewer TLB entries needed → 10-30% throughput improvement.
- Enable: mmap with MAP_HUGETLB or mount hugetlbfs.
- Critical for: Large ML models, HPC simulations, database buffer pools.
POSIX shared memory and memory-mapped files are the foundation of zero-copy IPC on modern systems — by allowing multiple processes to directly access the same physical memory pages without kernel-mediated data copies, they provide the highest possible throughput for local inter-process data sharing, making them indispensable for ML inference pipelines, database engines, and any high-performance system where data must flow between processes at memory bandwidth speeds.