↓ Ir para o conteúdo principal
Scholion
Scholion

← todas as notas

📎 Webclip

Building Distributed Systems in Elixir: Part 7 — Worker Pool

The article builds a worker pool from scratch with spawn/1, send/2, and receive/1. A coordinator process keeps the pool state, tracks idle workers, buffers waiting jobs, and stops the workers when all jobs are done. The design is meant to show bounded concurrency and backpressure before using higher-level abstractions.

Reading notes
#

  • A single registered process name points to one running process, so the article turns to a pool when many jobs must run without sending everything to one worker or spawning an unbounded number of processes.
  • The pool has three parts: a parent that starts it and waits, a coordinator that manages state, and worker processes that execute jobs and report back.
  • Each worker announces that it is available when it starts, then waits for work, processes a job, sends the result together with renewed availability, and stops when it receives :stop.
  • The coordinator keeps workers, available_workers, waiting_jobs, completed_jobs, and total_jobs in its recursive loop.
  • Job distribution is demand-driven: the coordinator assigns work only to workers that have explicitly sent :available.
  • The article says this model lets fast workers handle more jobs, slow workers handle fewer, and keeps concurrency within the worker count.
  • When completed_jobs == total_jobs, the coordinator sends :stop to every worker and notifies the parent that the pool has finished.
  • The article notes limits of the raw version: no failure handling or supervision, in-memory buffering only, and static sizing.
  • It points to OTP libraries such as NimblePool, Poolboy, and Broadway for production use.

Series
#

Building Distributed Systems in Elixir, by krishnadaspc. Research: Distributed Systems Patterns.