📎 Webclip
Building Resilient Payment Systems
Dr Milan Milanović summarizes Shopify Engineering’s ten main tips for building resilient payment systems, from tuning timeouts and adding circuit breakers to structured logging, reconciliation, and incident retrospectives. He also points readers to his newsletter for more details.
Reading notes#
- The thread presents Shopify Engineering’s list of ten useful tips and tricks for resilient payment systems.
- Lower timeouts are recommended, with an example that Ruby’s Net::HTTP 60s default is too slow when users are waiting in online applications.
- Circuit breakers such as Shopify’s Semian are suggested to detect when a service is down and fail fast instead of waiting for timeouts.
- Little’s Law is highlighted to relate queue size, arrival rate, and time, helping to design systems that handle load efficiently.
- Monitoring and alerting should cover the four golden signals from Google SRE: latency, traffic, errors, and saturation, to detect overload risk.
- Structured logging in a machine-readable format is recommended for better observability.
- Idempotency keys are advised so payments or refunds are executed exactly once while tracking attempts toward financial partners.
- Consistent reconciliation with financial partners’ records is emphasized, including recording mismatches and remediating them automatically when possible.
- Regular load testing is encouraged to probe system limits and safeguards, including simulating large flash sales; Shopify uses scriptable load balancers to throttle checkouts.
- Incident management at Shopify uses a Slack bot and defined roles for coordination, external communication, and restoring stability, triggered by alerts or manual reports.
- Incident retrospectives occur within a week to review what happened, adjust incorrect assumptions, and reduce the chance of recurrence.
- The author ends by inviting readers to subscribe to his “Techworld with Milan” newsletter for more information and provides related links and a card.
