Payment System Design

A system design case study for a payment platform covering payment orchestration, idempotency, ledgers, reconciliation, compliance, and fraud controls.

High-Level Architecture, Business Requirements & Core Concepts

Modern payment systems process millions of secure financial transactions every day. Whether you're paying with a credit card, Apple Pay, Google Pay, UPI, or a digital wallet, the payment platform coordinates multiple parties within seconds while ensuring security, consistency, and regulatory compliance.

In this series, we'll design a production-grade Payment System similar to Stripe, PayPal, Adyen, or Square.

Learning Objectives

By the end of this article, you'll understand:

Payment ecosystem
Payment lifecycle
Core payment concepts
Business requirements
Functional requirements
Non-functional requirements
Capacity planning
High-level architecture
Core microservices
Service responsibilities
End-to-end payment flow

What is a Payment System?

A payment system is a platform that securely transfers money between a payer and a payee.

It coordinates:

Customer
Merchant
Payment Gateway
Payment Processor
Acquiring Bank
Card Network
Issuing Bank

Its primary responsibilities are:

Process payments securely
Prevent fraud
Authorize transactions
Capture payments
Settle funds
Handle refunds
Support compliance

Types of Payment Systems

Card Payments

Examples

Visa
Mastercard
American Express
Discover

Digital Wallets

Examples

Apple Pay
Google Pay
Samsung Wallet
PayPal Wallet

Bank Transfers

Examples

ACH
Wire Transfer
SEPA
Faster Payments

Real-Time Payments

Examples

RTP
FedNow
UPI
PIX

Buy Now Pay Later

Examples

Klarna
Afterpay
Affirm

Cryptocurrency Payments

Examples

Bitcoin
Ethereum
Stablecoins

Payment Ecosystem

A payment transaction involves several participants.

flowchart LR

Customer

Merchant

Gateway

Processor

CardNetwork

IssuingBank

Customer --> Merchant

Merchant --> Gateway

Gateway --> Processor

Processor --> CardNetwork

CardNetwork --> IssuingBank

Payment Participants

Participant	Responsibility
Customer	Initiates payment
Merchant	Sells products or services
Payment Gateway	Securely accepts payment requests
Payment Processor	Routes payment transactions
Card Network	Connects acquiring and issuing banks
Issuing Bank	Approves or declines the payment
Acquiring Bank	Receives merchant payments
Settlement Bank	Transfers final funds

Business Requirements

The system should support:

Card payments
Wallet payments
Bank transfers
Partial payments
Multi-currency
Merchant onboarding
Customer management
Refunds
Chargebacks
Transaction history
Settlement
Reconciliation
Fraud detection
Notifications

Functional Requirements

The platform should allow users to:

Register merchants
Save payment methods
Create payment requests
Authorize payments
Capture payments
Cancel payments
Process refunds
Generate settlements
View transaction history
Download reports
Receive notifications

Non-Functional Requirements

The platform must provide:

High availability
Low latency
Strong consistency
Security
Scalability
Fault tolerance
Auditability
Compliance
Observability
Disaster recovery

Capacity Estimation

Example assumptions

Daily Transactions

120 Million

Peak TPS

12,000 Transactions/sec

Average Payment Record

3 KB

Daily Storage

120M × 3 KB

≈ 360 GB/day

Annual Storage

≈131 TB/year

Images and documents (receipts, invoices, disputes) should be stored separately in object storage.

Core Payment Concepts

Authorization

Authorization verifies:

Card validity
Available balance
Fraud checks
Spending limits

Money is reserved but not transferred.

Capture

Capture moves the authorized amount into the merchant's payment flow.

Example:

Hotel booking

Day 1

Authorize

After checkout

Capture

Settlement

Settlement transfers funds from the acquiring side to the merchant account.

Usually occurs:

Same day
Next business day
Based on merchant agreement

Refund

Refund transfers money back to the customer.

Types:

Full refund
Partial refund
Multiple partial refunds

Chargeback

Customers may dispute transactions.

Common reasons:

Fraud
Duplicate charge
Product not delivered
Incorrect amount

Chargebacks involve investigation between banks and merchants.

Payment Lifecycle

flowchart LR

PaymentRequest

PaymentRequest --> Authorization

Authorization --> Capture

Capture --> Settlement

Settlement --> Completed

Completed --> Refund

High-Level Architecture

flowchart TD

Customer

Customer --> API

API --> Authentication

API --> Merchant

API --> Payment

API --> Fraud

API --> Ledger

API --> Settlement

API --> Notification

Enterprise Architecture

flowchart TD

Customer

Merchant

Customer --> Gateway

Merchant --> Gateway

Gateway --> Authentication

Gateway --> PaymentService

Gateway --> MerchantService

Gateway --> WalletService

Gateway --> FraudService

Gateway --> LedgerService

Gateway --> SettlementService

Gateway --> NotificationService

Core Microservices

Service	Responsibility
API Gateway	Request routing
Authentication Service	Login, OAuth2, JWT
Customer Service	Customer profiles
Merchant Service	Merchant onboarding
Payment Service	Payment processing
Payment Method Service	Cards and wallets
Fraud Service	Fraud detection
Ledger Service	Financial accounting
Settlement Service	Merchant settlements
Refund Service	Refund processing
Chargeback Service	Dispute management
Notification Service	Email, SMS, Push
Reporting Service	Reports & analytics

Service Responsibilities

Customer Service

Responsible for:

Customer profile
Payment history
Saved payment methods

Merchant Service

Responsible for:

Merchant registration
Merchant verification
Business profile
Settlement configuration

Payment Service

Responsible for:

Payment authorization
Payment capture
Payment status
Payment cancellation

Ledger Service

Responsible for:

Financial transactions
Double-entry accounting
Account balances
Audit trail

Fraud Service

Responsible for:

Risk scoring
Velocity checks
Device fingerprinting
Rule engine
ML fraud detection

Settlement Service

Responsible for:

Merchant payouts
Settlement reports
Bank transfers
Reconciliation

Payment Flow

flowchart LR

Customer

Customer --> Merchant

Merchant --> Gateway

Gateway --> Payment

Payment --> Fraud

Fraud --> Processor

Processor --> IssuingBank

IssuingBank --> Approved

Approved --> Capture

Capture --> Settlement

Merchant Onboarding Flow

flowchart LR

Merchant

Merchant --> Registration

Registration --> Verification

Verification --> Approval

Approval --> SettlementAccount

SettlementAccount --> Active

Customer Payment Flow

flowchart LR

Customer

Customer --> Checkout

Checkout --> PaymentGateway

PaymentGateway --> Authorization

Authorization --> Success

Success --> Confirmation

Payment Status

Status	Description
Created	Payment initiated
Pending	Waiting for authorization
Authorized	Funds reserved
Captured	Funds captured
Settled	Funds transferred
Failed	Authorization failed
Cancelled	Payment cancelled
Refunded	Money returned
Chargeback	Customer dispute

Security Requirements

The platform must support:

OAuth2
JWT
TLS 1.3
PCI DSS compliance
Encryption at rest
Encryption in transit
Tokenization
Audit logging
Rate limiting
Multi-Factor Authentication for administrative access

Compliance Requirements

Common compliance standards include:

PCI DSS
SOC 2
ISO 27001
GDPR
CCPA

Depending on the region, additional financial regulations may apply.

High Availability Goals

Target:

99.99%

Recovery Objectives

Metric	Target
RPO	Near Zero
RTO	Less than 30 Minutes

Design Principles

Microservices architecture
Event-driven communication
API-first design
Database per service
Idempotent payment APIs
Strong consistency for financial operations
Immutable audit logs
Horizontal scalability
Security by design
Cloud-native deployment

Low-Level Architecture, Database Design, REST APIs & Event-Driven Architecture

In Part 1, we explored the payment ecosystem, business requirements, payment lifecycle, and high-level architecture.

In this part, we'll design the internal implementation of an enterprise payment platform similar to Stripe, PayPal, or Adyen.

Topics covered:

Low-Level Architecture
Database Design
Entity Relationship Diagram
Customer Management
Merchant Management
Payment Instrument Management
Payment Authorization
Payment Capture
Settlement
Refunds
Chargebacks
Double-Entry Ledger
REST APIs
Sequence Diagrams
Event-Driven Architecture
Kafka Topics

Low-Level Architecture

Each payment capability is implemented as an independent microservice.

flowchart LR

Gateway

Gateway --> Auth

Gateway --> Customer

Gateway --> Merchant

Gateway --> Payment

Gateway --> Instrument

Gateway --> Fraud

Gateway --> Ledger

Gateway --> Settlement

Gateway --> Refund

Gateway --> Chargeback

Gateway --> Notification

Why Database Per Service?

Every microservice owns its own database.

Benefits:

Independent deployment
Independent scaling
Loose coupling
Better fault isolation
Technology flexibility

Database Architecture

flowchart TD

CustomerService --> CustomerDB

MerchantService --> MerchantDB

PaymentService --> PaymentDB

InstrumentService --> InstrumentDB

LedgerService --> LedgerDB

SettlementService --> SettlementDB

RefundService --> RefundDB

ChargebackService --> ChargebackDB

Customer Database

Customer Table

Column	Type
customer_id	UUID
first_name	VARCHAR
last_name	VARCHAR
email	VARCHAR
phone	VARCHAR
status	VARCHAR
created_at	TIMESTAMP

Customer Address

Column	Type
address_id	UUID
customer_id	UUID
city	VARCHAR
state	VARCHAR
country	VARCHAR
postal_code	VARCHAR

Merchant Database

Merchant Table

Column	Type
merchant_id	UUID
merchant_name	VARCHAR
business_type	VARCHAR
email	VARCHAR
settlement_account	VARCHAR
status	VARCHAR

Merchant Configuration

Column	Type
config_id	UUID
merchant_id	UUID
settlement_frequency	VARCHAR
supported_currency	VARCHAR
webhook_url	VARCHAR

Payment Instrument Database

Supports:

Credit Cards
Debit Cards
Wallets
Bank Accounts
UPI
ACH

Payment Instrument

Column	Type
instrument_id	UUID
customer_id	UUID
token	VARCHAR
type	VARCHAR
last_four	VARCHAR
expiry_month	INTEGER
expiry_year	INTEGER
status	VARCHAR

Sensitive card information must never be stored directly. Store only tokenized references.

Payment Database

Payment Table

Column	Type
payment_id	UUID
customer_id	UUID
merchant_id	UUID
amount	DECIMAL
currency	VARCHAR
payment_status	VARCHAR
created_at	TIMESTAMP

Payment Status

Status
Created
Pending
Authorized
Captured
Settled
Failed
Cancelled
Refunded
Chargeback

Authorization Table

Column	Type
authorization_id	UUID
payment_id	UUID
authorization_code	VARCHAR
issuer_response	VARCHAR
authorized_amount	DECIMAL
status	VARCHAR

Capture Table

Column	Type
capture_id	UUID
payment_id	UUID
capture_amount	DECIMAL
captured_at	TIMESTAMP

Supports:

Full Capture
Partial Capture
Multiple Captures

Settlement Database

Settlement Table

Column	Type
settlement_id	UUID
merchant_id	UUID
payment_id	UUID
settlement_amount	DECIMAL
settlement_status	VARCHAR
settlement_date	TIMESTAMP

Refund Database

Refund Table

Column	Type
refund_id	UUID
payment_id	UUID
refund_amount	DECIMAL
refund_reason	VARCHAR
refund_status	VARCHAR

Chargeback Database

Chargeback Table

Column	Type
chargeback_id	UUID
payment_id	UUID
dispute_reason	VARCHAR
dispute_status	VARCHAR
created_at	TIMESTAMP

Ledger Database

Financial systems should never rely only on payment tables.

Maintain a dedicated immutable ledger.

Ledger Account

Column	Type
account_id	UUID
account_name	VARCHAR
account_type	VARCHAR

Ledger Entry

Column	Type
entry_id	UUID
payment_id	UUID
debit_account	UUID
credit_account	UUID
amount	DECIMAL
transaction_time	TIMESTAMP

Double-Entry Accounting

Every transaction has:

One Debit
One Credit

Example

Customer pays $250

Debit

Customer Account

$250

↓

Credit

Merchant Clearing Account

$250

Ledger remains balanced.

Entity Relationship Diagram

flowchart TD

Customer

Merchant

Payment

Authorization

Capture

Settlement

Refund

Chargeback

Ledger

Customer --> Payment

Merchant --> Payment

Payment --> Authorization

Payment --> Capture

Payment --> Settlement

Payment --> Refund

Payment --> Chargeback

Payment --> Ledger

Payment Authorization Flow

flowchart LR

Checkout

Checkout --> Gateway

Gateway --> Fraud

Fraud --> Processor

Processor --> Issuer

Issuer --> Approved

Payment Capture Flow

flowchart LR

Authorized

Authorized --> Capture

Capture --> Ledger

Ledger --> Settlement

Settlement Flow

flowchart LR

Captured

Captured --> Settlement

Settlement --> MerchantBank

Refund Flow

flowchart LR

RefundRequest

RefundRequest --> Validation

Validation --> Refund

Refund --> Ledger

Chargeback Flow

flowchart LR

Customer

Customer --> Bank

Bank --> Chargeback

Chargeback --> Merchant

REST API Design

Customer APIs

Create Customer

POST /customers

Get Customer

GET /customers/{id}

Update Customer

PUT /customers/{id}

Merchant APIs

Register Merchant

POST /merchants

Merchant Details

GET /merchants/{id}

Payment APIs

Create Payment

POST /payments

Example Request

{
  "merchantId":"MER123",
  "customerId":"CUS456",
  "amount":250.00,
  "currency":"USD"
}

Example Response

{
  "paymentId":"PAY10001",
  "status":"PENDING"
}

Authorize Payment

POST /payments/{id}/authorize

Capture Payment

POST /payments/{id}/capture

Cancel Payment

POST /payments/{id}/cancel

Payment Status

GET /payments/{id}

Settlement APIs

POST /settlements

GET /settlements/{id}

Refund APIs

POST /refunds

GET /refunds/{id}

Chargeback APIs

POST /chargebacks

GET /chargebacks/{id}

Sequence Diagram

sequenceDiagram

Customer->>Gateway: Create Payment

Gateway->>Fraud: Validate

Fraud-->>Gateway: OK

Gateway->>Processor: Authorize

Processor->>Issuer: Validate Card

Issuer-->>Processor: Approved

Processor-->>Gateway: Success

Gateway-->>Customer: Payment Authorized

Event-Driven Architecture

Business events are published to Kafka.

flowchart LR

Payment

Payment --> Kafka

Authorization --> Kafka

Settlement --> Kafka

Refund --> Kafka

Kafka --> Notification

Kafka --> Reporting

Kafka --> Analytics

Kafka Topics

Topic	Producer	Consumer
payment-created	Payment Service	Fraud
payment-authorized	Payment Service	Ledger
payment-captured	Payment Service	Settlement
settlement-created	Settlement Service	Merchant
refund-created	Refund Service	Ledger
refund-completed	Refund Service	Notification
chargeback-created	Chargeback Service	Risk
merchant-created	Merchant Service	Reporting
customer-created	Customer Service	Analytics

Sample Payment Event

{
  "event":"PAYMENT_AUTHORIZED",
  "paymentId":"PAY10001",
  "merchantId":"MER101",
  "customerId":"CUS500",
  "amount":250.00,
  "currency":"USD",
  "timestamp":"2026-08-01T15:30:45Z"
}

Service Communication

Use synchronous communication for:

Authentication
Payment authorization
Fraud validation
Payment status lookup
Merchant validation

Use asynchronous communication for:

Notifications
Reporting
Analytics
Settlement processing
Reconciliation
Audit logging

Data Consistency

Strong consistency is required for:

Payment authorization
Payment capture
Ledger entries
Settlement
Refunds

Eventual consistency is acceptable for:

Notifications
Analytics
Reporting
Dashboard metrics

Error Handling

Error	Action
Duplicate Payment	Reject using Idempotency Key
Card Declined	Return issuer response
Gateway Timeout	Retry safely
Fraud Detected	Reject transaction
Settlement Failure	Retry settlement
Refund Failure	Retry asynchronously

Best Practices

Maintain a separate immutable ledger.
Use UUIDs for distributed systems.
Never store raw card numbers.
Use tokenization for payment instruments.
Make payment APIs idempotent.
Publish domain events through Kafka.
Keep payment and ledger services independent.
Encrypt sensitive information.
Audit every financial transaction.
Use database-per-service architecture.

Authorization, Settlement, Ledger, CQRS, Security & Multi-Region Architecture

In Part 2, we designed the low-level architecture, databases, REST APIs, ledger schema, and event-driven communication.

In this part, we'll design the advanced architecture required by enterprise payment platforms such as Stripe, PayPal, Adyen, and Square.

Topics covered:

Payment Authorization Workflow
Payment Capture Workflow
Settlement Workflow
Refund Workflow
Chargeback Workflow
Double-Entry Ledger
CQRS
Saga Pattern
Idempotency Keys
Redis Caching
Fraud Detection
Risk Engine
PCI DSS
Encryption & Tokenization
Multi-Region Deployment

Enterprise Payment Architecture

flowchart LR

Customer

Customer --> Gateway

Gateway --> Payment

Gateway --> Fraud

Gateway --> Ledger

Gateway --> Settlement

Gateway --> Notification

Payment Authorization

Authorization verifies whether the payment can proceed.

Checks include:

Card validity
Available balance
Fraud rules
Merchant status
Card expiration
Velocity checks
Spending limits
Country restrictions

Funds are reserved but not transferred.

Authorization Workflow

flowchart LR

Checkout

Checkout --> Gateway

Gateway --> Fraud

Fraud --> Processor

Processor --> Issuer

Issuer --> Authorized

Authorization States

Status	Description
Pending	Awaiting authorization
Authorized	Funds reserved
Declined	Authorization failed
Expired	Authorization timeout
Reversed	Authorization released

Payment Capture

Capture transfers the authorized amount into the settlement pipeline.

Examples:

Hotel check-in
Car rental
Online marketplace
Food delivery

Capture Workflow

flowchart LR

Authorized

Authorized --> Capture

Capture --> Ledger

Ledger --> Settlement

Capture Types

Type	Description
Full Capture	Entire amount captured
Partial Capture	Portion captured
Multiple Capture	Multiple partial captures

Settlement

Settlement transfers money to the merchant.

Settlement usually happens:

Same day
Next business day
Weekly
Custom merchant schedule

Settlement Workflow

flowchart LR

Captured

Captured --> Settlement

Settlement --> MerchantAccount

MerchantAccount --> Completed

Settlement Process

Payment captured
Calculate processing fee
Calculate taxes
Calculate merchant payout
Create ledger entries
Transfer funds
Generate settlement report

Refund Workflow

Refunds reverse completed payments.

Types:

Full refund
Partial refund
Multiple refunds

Refund Flow

flowchart LR

RefundRequest

RefundRequest --> Validation

Validation --> Ledger

Ledger --> PaymentProcessor

PaymentProcessor --> Customer

Chargeback Workflow

Chargebacks occur when customers dispute payments.

Common reasons:

Fraud
Duplicate charge
Product not delivered
Product not as described

Chargeback Flow

flowchart LR

Customer

Customer --> Issuer

Issuer --> Chargeback

Chargeback --> Merchant

Merchant --> Evidence

Evidence --> Resolution

Double-Entry Ledger

Every financial transaction creates equal debit and credit entries.

Example:

Customer pays $100

Debit

Customer Cash

$100

↓

Credit

Merchant Clearing

$100

The ledger always remains balanced.

Ledger Workflow

flowchart LR

Payment

Payment --> Debit

Debit --> Credit

Credit --> Ledger

Ledger Rules

Never:

Update historical entries
Delete transactions
Modify financial history

Always:

Create adjustment entries
Preserve audit trail
Maintain immutable history

Why Separate Ledger?

Benefits:

Financial accuracy
Auditability
Regulatory compliance
Independent accounting
Easier reconciliation

Reconciliation

Daily reconciliation compares:

Gateway transactions
Processor records
Ledger entries
Settlement reports
Bank statements

Reconciliation Workflow

flowchart LR

Gateway

Gateway --> Ledger

Ledger --> Settlement

Settlement --> Bank

Bank --> Reconciliation

CQRS

Payment platforms receive many read requests.

Examples:

Writes

Create payment
Capture payment
Refund
Settlement

Reads

Transaction history
Merchant dashboard
Customer history
Reports

CQRS Architecture

flowchart LR

User

User --> CommandAPI

User --> QueryAPI

CommandAPI --> WriteDB

WriteDB --> Kafka

Kafka --> ReadDB

ReadDB --> QueryAPI

CQRS Benefits

Faster dashboards
Independent scaling
Better reporting
Reduced database contention
Optimized analytics

Saga Pattern

Payment workflows span multiple services.

Typical workflow:

Validate merchant
Fraud check
Authorize payment
Record ledger
Settlement
Notification

Each service owns its own transaction.

Saga Workflow

flowchart TD

Payment

Payment --> Fraud

Fraud --> Authorization

Authorization --> Ledger

Ledger --> Settlement

Settlement --> Notification

Compensation Example

Authorization succeeds

↓

Ledger fails

↓

Compensation:

flowchart LR

Authorized

Authorized --> LedgerFailure

LedgerFailure --> AuthorizationReversal

Idempotency Keys

Payment APIs must support idempotency.

Example:

Payment Request

↓

Network Timeout

↓

Retry

↓

Same Payment

No duplicate payment should be created.

Idempotency Example

Client sends:

Idempotency-Key

ABC123XYZ

If the request is retried with the same key:

No duplicate payment
Original response returned

Redis Caching

Redis stores frequently accessed information.

Examples:

Merchant profile
Customer profile
Exchange rates
Payment configuration
Fraud rules
Authentication sessions

Cache Architecture

flowchart LR

Application

Application --> Redis

Redis --> Database

Recommended Cache TTL

Data	TTL
Merchant Configuration	1 Hour
Exchange Rates	10 Minutes
Customer Profile	30 Minutes
Fraud Rules	5 Minutes
Supported Currencies	6 Hours
API Configuration	30 Minutes

Never Cache

Avoid caching:

Active payment authorization
Ledger balances
Settlement status
Refund state
Chargeback state

Fraud Detection

Fraud checks occur before authorization.

Checks include:

Velocity rules
Device fingerprint
IP reputation
Geographic mismatch
Spending behavior
Merchant risk
Blacklisted cards

Fraud Workflow

flowchart LR

Payment

Payment --> RiskEngine

RiskEngine --> Approved

RiskEngine --> ManualReview

RiskEngine --> Rejected

Risk Scoring

Example:

Score	Action
0–30	Approve
31–70	Manual Review
71–100	Reject

PCI DSS

Payment platforms must comply with PCI DSS.

Requirements include:

Encrypt cardholder data
Strong access control
Vulnerability scanning
Secure development
Continuous monitoring
Audit logging

Tokenization

Never store raw card numbers.

Instead:

4111111111111111

↓

TOKEN_ABC1001

Applications use the token instead of the PAN.

Benefits:

Better security
PCI scope reduction
Lower breach impact

Encryption

Protect:

Customer data
Merchant data
Payment tokens
Personal information
API credentials

Recommended:

TLS 1.3 for data in transit
AES-256 for encrypted storage

Secrets Management

Never hardcode:

Database passwords
API keys
Payment gateway credentials
JWT signing keys
Encryption keys

Use centralized secrets management.

Multi-Region Deployment

Large payment providers operate globally.

Objectives:

Low latency
Regulatory compliance
Disaster recovery
High availability

Multi-Region Architecture

flowchart LR

Users

Users --> RegionA

Users --> RegionB

RegionA --> PaymentDBA

RegionB --> PaymentDBB

PaymentDBA --> Replication

Replication --> PaymentDBB

High Availability

Target:

99.99%

Achieved using:

Multiple regions
Multiple availability zones
Load balancers
Auto scaling
Database replication
Health checks

Reliability Patterns

Implement:

Retry
Timeout
Circuit Breaker
Bulkhead
Dead Letter Queue
Rate Limiting
Outbox Pattern

These patterns improve resilience while preventing cascading failures.

Best Practices

Keep the ledger immutable.
Make every payment API idempotent.
Separate authorization from capture.
Tokenize sensitive payment information.
Perform fraud checks before authorization.
Encrypt all sensitive customer data.
Use CQRS for reporting workloads.
Coordinate distributed workflows using Saga.
Continuously reconcile financial records.
Design for multi-region failover.

Production Goals

A production payment platform should provide:

99.99% availability
Zero data loss
Zero duplicate payments
Low latency
High throughput
Automatic recovery
End-to-end observability
Disaster recovery
Regulatory compliance

Enterprise Production Architecture

flowchart TD

Clients

Clients --> DNS

DNS --> CDN

CDN --> WAF

WAF --> LoadBalancer

LoadBalancer --> APIGateway

APIGateway --> Kubernetes

Kubernetes --> Auth

Kubernetes --> Payment

Kubernetes --> Fraud

Kubernetes --> Ledger

Kubernetes --> Settlement

Kubernetes --> Refund

Kubernetes --> Notification

Infrastructure Stack

Layer	Technology
DNS	Route53 / Cloud DNS
CDN	CloudFront / Azure CDN
WAF	AWS WAF
Load Balancer	ALB / NGINX
API Gateway	Kong / Spring Cloud Gateway
Container Runtime	Docker
Container Orchestration	Kubernetes
Messaging	Kafka
Cache	Redis
Database	PostgreSQL / Oracle
Monitoring	Prometheus
Dashboards	Grafana
Logging	ELK / OpenSearch
Tracing	Jaeger / Zipkin

Docker

Every microservice is packaged as a Docker image.

Benefits:

Environment consistency
Faster deployments
Easy rollback
Dependency isolation
Immutable artifacts

Sample Dockerfile

FROM eclipse-temurin:21-jre

WORKDIR /app

COPY target/payment-service.jar app.jar

EXPOSE 8080

ENTRYPOINT ["java","-jar","app.jar"]

Kubernetes

Kubernetes manages:

Deployments
Scaling
Self-healing
Rolling updates
Service discovery
Health checks
Failover

Kubernetes Architecture

flowchart TD

Users

Users --> Ingress

Ingress --> Gateway

Gateway --> PaymentPods

Gateway --> LedgerPods

Gateway --> FraudPods

Gateway --> SettlementPods

Gateway --> RefundPods

Kubernetes Resources

Resource	Purpose
Pod	Application container
Deployment	Replica management
Service	Internal networking
Ingress	External traffic routing
ConfigMap	Configuration
Secret	Sensitive information
StatefulSet	Stateful workloads
Horizontal Pod Autoscaler	Auto scaling

Namespace Strategy

Recommended namespaces:

production

staging

uat

development

Benefits:

Resource isolation
Security boundaries
Easier deployments
Separate access controls

Service Discovery

Microservices communicate using logical service names.

Examples:

payment-service

ledger-service

fraud-service

settlement-service

Applications never rely on fixed IP addresses.

Internal Communication

flowchart LR

Gateway

Gateway --> Payment

Payment --> Fraud

Fraud --> Ledger

Ledger --> Settlement

Settlement --> Notification

API Gateway

Responsibilities:

Authentication
Authorization
SSL termination
Request routing
Rate limiting
Request validation
API aggregation
Request logging

Load Balancing

Traffic is distributed across healthy instances.

flowchart TD

Users

Users --> LoadBalancer

LoadBalancer --> Pod1

LoadBalancer --> Pod2

LoadBalancer --> Pod3

Benefits:

High availability
Better throughput
Lower latency
Fault tolerance

Horizontal Scaling

Increase pod count during high payment volume.

Example

5 Pods

↓

25 Pods

↓

100 Pods

Scaling metrics:

CPU usage
Memory usage
Transactions per second
Kafka consumer lag
Request latency

Vertical Scaling

Increase:

CPU
Memory
Storage

Typically applied to:

PostgreSQL
Oracle
Kafka Brokers
Redis
Elasticsearch

Service Scaling Strategy

Service	Scaling Requirement
API Gateway	Extremely High
Payment	Extremely High
Fraud	High
Ledger	High
Settlement	Medium
Refund	Medium
Notification	High
Reporting	High

CI/CD Pipeline

Every deployment follows an automated pipeline.

flowchart LR

Developer

Developer --> Git

Git --> Build

Build --> Test

Test --> SecurityScan

SecurityScan --> Docker

Docker --> Registry

Registry --> Kubernetes

Continuous Integration

Pipeline stages:

Checkout source code
Compile
Unit tests
Integration tests
Static code analysis
Dependency scanning
Build Docker image
Push to container registry

Continuous Delivery

Deployment flow

Development

↓

QA

↓

UAT

↓

Performance Testing

↓

Security Testing

↓

Production Approval

↓

Production Deployment

Deployment Strategies

Rolling Deployment

flowchart LR

Old

Old --> Mixed

Mixed --> New

Use for:

Reporting
Notification
Merchant services

Blue-Green Deployment

flowchart LR

Users

Users --> Blue

Blue --> Green

Recommended for:

Payment processing
Ledger
Settlement
Authorization

Canary Deployment

5%

↓

20%

↓

50%

↓

100%

Ideal for:

Fraud detection improvements
Recommendation engines
Reporting features

Monitoring Architecture

flowchart LR

Applications

Applications --> Metrics

Applications --> Logs

Applications --> Traces

Metrics --> Prometheus

Prometheus --> Grafana

Logs --> ELK

Traces --> Jaeger

Infrastructure Metrics

Monitor:

CPU
Memory
JVM Heap
Disk usage
Network latency
Kafka lag
Database connections
Redis hit ratio

Payment Metrics

Monitor:

Transactions per second
Authorization latency
Capture latency
Settlement latency
Refund latency
Chargeback rate
Payment success rate
Duplicate payment attempts

Business KPIs

Track:

Total payment volume
Merchant revenue
Refund percentage
Settlement completion time
Fraud detection rate
Merchant onboarding time
Failed payment percentage

Golden Signals

Signal	Description
Latency	API response time
Traffic	Requests per second
Errors	Failed requests
Saturation	Resource utilization

Logging Strategy

Every request should include:

Timestamp
Trace ID
Correlation ID
Payment ID
Merchant ID
Customer ID
Request ID
Response time

Sample Structured Log

{
  "traceId":"TR100001",
  "paymentId":"PAY50001",
  "merchantId":"MER200",
  "customerId":"CUS900",
  "status":"AUTHORIZED",
  "responseTime":42
}

Log Levels

Level	Usage
INFO	Business events
WARN	Recoverable issues
ERROR	Failures
DEBUG	Development only

Avoid DEBUG logging in production except for controlled troubleshooting.

Distributed Tracing

One payment request flows across multiple services.

flowchart LR

Gateway

Gateway --> Payment

Payment --> Fraud

Fraud --> Ledger

Ledger --> Settlement

Settlement --> Notification

Every service propagates the same Trace ID.

Health Checks

Expose application health endpoints.

GET /actuator/health

GET /actuator/liveness

GET /actuator/readiness

These endpoints enable Kubernetes to restart unhealthy pods automatically.

Alerting Strategy

Condition	Severity
Payment Failure > 2%	Critical
Authorization Latency > 500 ms	Critical
Settlement Delay	Warning
Kafka Consumer Lag	Warning
Ledger Write Failure	Critical
Database Replication Failure	Critical
Pod CrashLoop	Critical
CPU > 90%	Warning

Backup Strategy

Protect financial records with:

Hourly incremental backups
Daily full backups
Weekly archive
Cross-region replication
Immutable backup storage

Perform periodic restore testing.

Disaster Recovery

The platform should survive:

Region outage
Database failure
Kubernetes cluster failure
Kafka outage
Payment processor outage
Cloud availability zone failure

Disaster Recovery Architecture

flowchart TD

PrimaryRegion

PrimaryRegion --> DatabaseA

PrimaryRegion --> KafkaA

DatabaseA --> Replication

KafkaA --> Replication

Replication --> SecondaryRegion

SecondaryRegion --> DatabaseB

SecondaryRegion --> KafkaB

Recovery Objectives

Objective	Target
Recovery Point Objective (RPO)	Near Zero
Recovery Time Objective (RTO)	Less than 30 Minutes

Failure Scenarios

Payment Processor Failure

Recovery:

Retry request
Switch to secondary processor
Queue pending transactions
Notify merchants

Database Failure

Recovery:

Promote read replica
Redirect application traffic
Validate ledger consistency

Kafka Failure

Recovery:

Retry producers
Retry consumers
Dead Letter Queue
Cluster replication

Kubernetes Node Failure

Recovery:

Restart failed pods
Redistribute traffic
Automatic scheduling

Regional Outage

Recovery:

DNS failover
Activate secondary region
Promote standby databases
Resume payment processing

Performance Optimization

Improve performance using:

Redis caching
Database indexing
Connection pooling
Read replicas
Batch settlement
Asynchronous messaging
JVM tuning
HTTP compression

Cost Optimization

Reduce infrastructure costs through:

Horizontal Pod Autoscaler
Reserved instances
Spot instances for non-critical jobs
Storage lifecycle management
Kafka retention optimization
Log archival
Database partitioning

Security Operations

Production security includes:

Mutual TLS
Kubernetes Network Policies
Secret rotation
Runtime threat detection
RBAC
WAF
Image vulnerability scanning
Zero Trust networking

Production Readiness Checklist

Area	Ready
Docker	✓
Kubernetes	✓
Security Scan	✓
CI/CD	✓
Monitoring	✓
Logging	✓
Distributed Tracing	✓
Auto Scaling	✓
Backup Validation	✓
Disaster Recovery	✓
Alerting	✓
Rollback Strategy	✓

Production Operations

Operations teams should continuously monitor:

Authorization success rate
Payment success rate
Settlement completion time
Refund processing time
Fraud detection accuracy
Kafka consumer lag
Database replication health
Ledger consistency
Infrastructure cost
API latency

Best Practices

Deploy payment services independently.
Use Blue-Green deployments for critical payment components.
Continuously reconcile financial data.
Propagate Trace IDs across every service.
Monitor both business and infrastructure metrics.
Test disaster recovery regularly.
Secure all communication using TLS.
Rotate secrets automatically.
Automate rollback for failed deployments.
Validate production readiness before every release.

Real Production Challenges

Enterprise payment platforms process millions of secure transactions every day.

Typical production challenges include:

Payment gateway outages
Duplicate payment requests
Network failures
Settlement delays
Bank downtime
Fraud attacks
Chargeback spikes
Database failures
Regional outages
Regulatory compliance

Challenge 1 — Payment Gateway Failure

External processors occasionally become unavailable.

Examples:

Stripe API unavailable
Acquiring bank timeout
Card network latency
Processor maintenance

Gateway Failover

flowchart LR

Customer

Customer --> Gateway

Gateway --> ProcessorA

ProcessorA --> Success

ProcessorA --> ProcessorB

Recovery Strategy

Retry with exponential backoff
Circuit Breaker
Secondary payment processor
Queue pending requests
Notify merchant

Challenge 2 — Duplicate Payments

Customers often retry payments after:

Browser refresh
Mobile reconnect
Gateway timeout
Slow network

Without protection:

One Purchase

↓

Multiple Charges

Duplicate Payment Prevention

Use:

Idempotency Keys
Unique Payment ID
Distributed Lock
Request Hash Validation
Payment Status Lookup

Idempotency Flow

flowchart LR

Client

Client --> Payment

Payment --> Existing

Existing --> SameResponse

Challenge 3 — High Transaction Volume

Example:

Black Friday

150,000 TPS

Challenges:

API overload
Kafka lag
Database saturation
Queue growth

Solutions:

Horizontal scaling
Kafka partitioning
Redis caching
Auto Scaling
Load balancing

High-Volume Architecture

flowchart TD

Clients

Clients --> Gateway

Gateway --> PaymentPods

PaymentPods --> Kafka

Kafka --> Settlement

Kafka --> Ledger

Challenge 4 — Settlement Reconciliation

Every settlement must match:

Gateway records
Processor records
Ledger entries
Merchant reports
Bank statements

Even a $0.01 mismatch requires investigation.

Reconciliation Flow

flowchart LR

Gateway

Gateway --> Ledger

Ledger --> Settlement

Settlement --> Bank

Bank --> Reconciliation

Challenge 5 — Fraud Detection

Fraud patterns include:

Card testing
Stolen cards
Bot attacks
Account takeover
Velocity attacks
Geographic anomalies
Device spoofing

Fraud Detection Flow

flowchart LR

Payment

Payment --> RiskEngine

RiskEngine --> Approved

RiskEngine --> Review

RiskEngine --> Rejected

Challenge 6 — Chargeback Surge

Large merchants may receive thousands of disputes daily.

Typical workflow:

flowchart LR

Customer

Customer --> Issuer

Issuer --> Merchant

Merchant --> Evidence

Evidence --> Resolution

Best Practices:

Store audit logs
Keep immutable ledger
Save authorization records
Retain settlement reports

Challenge 7 — Regional Outage

A payment platform should continue processing during cloud region failures.

flowchart TD

Users

Users --> RegionA

Users --> RegionB

RegionA --> DatabaseA

RegionB --> DatabaseB

DatabaseA --> Replication

Replication --> DatabaseB

Recovery:

DNS failover
Active-Active deployment
Database replication
Automatic traffic routing

Challenge 8 — Ledger Consistency

Financial records must never become inconsistent.

Rules:

Never delete ledger entries
Never update historical entries
Create adjustment entries
Maintain audit history

Ledger Flow

flowchart LR

Payment

Payment --> Debit

Debit --> Credit

Credit --> Ledger

Challenge 9 — Kafka Consumer Lag

Symptoms:

Settlement delays
Notification delays
Reporting delays

Solutions:

Increase partitions
Scale consumers
Optimize processing
Monitor lag continuously

Challenge 10 — Secret Management

Never store:

Database passwords
API keys
Card encryption keys
JWT secrets
Processor credentials

Use centralized secret management with automatic rotation.

Scalability Strategy

Scale every service independently.

flowchart TD

Gateway

Gateway --> Payment

Gateway --> Fraud

Gateway --> Ledger

Gateway --> Settlement

Gateway --> Refund

Gateway --> Reporting

Gateway --> Notification

Scaling Strategy

Service	Scaling Need
API Gateway	Extremely High
Payment	Extremely High
Fraud	Very High
Ledger	High
Settlement	High
Refund	Medium
Reporting	Medium
Notification	High

Database Scaling

Use:

Read replicas
Connection pooling
Partitioning
Archiving
Query optimization

flowchart LR

Application

Application --> PrimaryDB

PrimaryDB --> Replica1

PrimaryDB --> Replica2

Kafka Scaling

Improve throughput using:

Topic partitioning
Consumer groups
Horizontal scaling

Example:

Payment Topic

↓

64 Partitions

↓

256 Consumers

Redis Scaling

flowchart LR

Application

Application --> RedisCluster

RedisCluster --> Node1

RedisCluster --> Node2

RedisCluster --> Node3

Ideal for:

Merchant configuration
Exchange rates
Session management
Fraud rules
Customer profile

Cost Optimization

Reduce operational costs through:

Horizontal Pod Autoscaler
Reserved instances
Storage lifecycle policies
Log archival
Kafka retention optimization
Right-sized Kubernetes nodes
Batch settlement processing

Reliability Patterns

Implement:

Retry
Timeout
Circuit Breaker
Bulkhead
Rate Limiting
Dead Letter Queue
Outbox Pattern

These patterns improve resilience while preventing cascading failures.

Architecture Trade-offs

Monolith vs Microservices

Monolith	Microservices
Faster to start	Better scalability
Shared database	Database per service
Easier debugging	Independent deployment
Lower operational overhead	Better fault isolation

SQL vs NoSQL

SQL	NoSQL
ACID	Flexible schema
Strong consistency	Massive scalability
Ledger	Analytics

REST vs Event-Driven

REST	Event-Driven
Immediate response	Asynchronous
Easier debugging	Loose coupling
Request/Response	Event streaming

Authorization vs Capture

Authorization	Capture
Reserve funds	Move funds
Reversible	Financial commitment
Hotel booking	Final payment

Business continuity and disaster recovery.

Common Production Issues

Issue	Solution
Duplicate Payments	Idempotency Keys
Gateway Timeout	Retry + Circuit Breaker
Fraud Spike	Risk Engine
Settlement Delay	Kafka Scaling
Database Contention	Read Replicas
Kafka Lag	Scale Consumers
High API Latency	Redis Cache
Ledger Mismatch	Daily Reconciliation
Pod CrashLoop	Kubernetes Self-Healing
Region Failure	Active-Active Deployment

Production Readiness Checklist

Area	Ready
Security Review	✓
PCI DSS Controls	✓
Performance Testing	✓
Load Testing	✓
Monitoring	✓
Logging	✓
Distributed Tracing	✓
Disaster Recovery	✓
Backup Validation	✓
Auto Scaling	✓
Capacity Planning	✓
Rollback Strategy	✓

Best Practices

Use immutable double-entry accounting.
Make every payment API idempotent.
Tokenize sensitive payment data.
Encrypt all customer information.
Separate authorization from capture.
Continuously reconcile settlement records.
Publish business events using Kafka.
Scale payment services independently.
Monitor technical and financial KPIs.
Test disaster recovery frequently.

Common Design Mistakes

Updating Ledger Records

Financial history should never be modified.

Storing Raw Card Numbers

Always tokenize payment instruments.

Missing Idempotency

Leads to duplicate charges.

Synchronous Settlement

Settlement should be asynchronous to improve resilience and throughput.

Shared Database

Creates tight coupling between services.

Weak Audit Logging

Makes investigations and compliance difficult.

Payment System Design Interview Questions

1. How would you design Stripe?

Use independently scalable microservices for payment processing, fraud detection, ledger management, settlements, refunds, reporting, and notifications connected through event-driven communication.

2. Why separate authorization from capture?

Authorization reserves funds, while capture transfers them. Separating them supports delayed capture scenarios such as hotels, rentals, and marketplaces.

3. What is an idempotency key?

A unique client-generated key that ensures repeated requests produce the same result instead of creating duplicate payments.

4. Why maintain a separate ledger?

A dedicated immutable ledger guarantees financial accuracy, reconciliation, auditing, and regulatory compliance.

5. What is double-entry accounting?

Every financial transaction records equal debit and credit entries so that the accounting ledger always remains balanced.

6. Why is reconciliation important?

It ensures gateway records, ledger entries, settlements, and bank statements all match.

7. How do you prevent duplicate payments?

Use idempotency keys, request hashing, unique payment identifiers, and distributed locking where appropriate.

8. How do you secure payment information?

Use TLS, tokenization, encryption, PCI DSS controls, RBAC, and centralized secret management.

9. What should be cached?

Merchant configuration, exchange rates, customer profiles, supported currencies, and fraud rules.

10. What should never be cached?

Authorization state, ledger balances, settlement status, refund state, and active payment transactions.

11. Why use Kafka?

To decouple payment processing from settlement, notifications, reporting, analytics, and reconciliation.

12. Why use Saga Pattern?

To coordinate payment authorization, ledger updates, settlements, refunds, and notifications without distributed database transactions.

13. How do you handle payment gateway failures?

Retry safely, apply circuit breakers, switch to secondary processors, and queue transactions if necessary.

14. What happens if ledger writing fails?

Compensate the workflow, reverse the authorization if appropriate, and alert operations.

15. Why tokenize cards?

To reduce PCI DSS scope and avoid storing raw card numbers.

16. How do you scale payment systems?

Scale stateless services horizontally, partition Kafka topics, cache reads with Redis, and optimize databases with replicas.

17. How do you detect fraud?

Use rule engines, risk scoring, behavioral analysis, velocity limits, geolocation checks, and device fingerprinting.

18. How do refunds work?

Validate eligibility, create reversing ledger entries, submit the refund to the processor, and notify the customer.

19. What causes chargebacks?

Fraud, duplicate charges, merchant disputes, incorrect amounts, and products not delivered.

20. How do you process settlements?

Aggregate captured payments, calculate fees, create ledger entries, transfer funds, and generate settlement reports.

21. What metrics should be monitored?

Authorization latency, payment success rate, settlement time, fraud rate, TPS, API latency, and reconciliation accuracy.

22. How do you support multiple currencies?

Maintain exchange rates, merchant settlement currencies, currency conversion, and localized reporting.

23. Why use CQRS?

To separate high-volume reporting and dashboard queries from transactional payment processing.

24. How do you achieve high availability?

Use multiple availability zones, multi-region deployment, database replication, and automated failover.

25. What happens during reconciliation?

Compare payment records across processors, ledgers, settlements, merchants, and banks to detect discrepancies.

26. Why use immutable audit logs?

They provide traceability for compliance, investigations, and financial audits.

27. Which deployment strategy is safest?

Blue-Green or Canary deployments combined with automated monitoring and rollback.

28. How do you protect secrets?

Use centralized secret management, automatic rotation, encryption, and strict access control.

29. What business KPIs should be monitored?

Payment volume, merchant revenue, authorization success rate, refund rate, chargeback ratio, settlement time, and fraud loss.

30. What is the most important design principle?

Maintain financial correctness through idempotent APIs, immutable ledgers, secure processing, and reliable reconciliation.

Payment Architecture Cheat Sheet

Area	Recommended Solution
Architecture	Microservices
API Style	REST + Event-Driven
Messaging	Kafka
Cache	Redis
Database	PostgreSQL / Oracle
Ledger	Double-Entry Accounting
Workflow	Saga Pattern
Read Optimization	CQRS
Authentication	OAuth2 + JWT
Card Security	Tokenization
Encryption	TLS 1.3 + AES-256
Compliance	PCI DSS
Deployment	Kubernetes
Monitoring	Prometheus + Grafana
Logging	ELK / OpenSearch
Tracing	Jaeger / Zipkin
Disaster Recovery	Active-Active Multi-Region

Complete Enterprise Payment Architecture

flowchart TD

Customer

Merchant

Customer --> Gateway

Merchant --> Gateway

Gateway --> Authentication

Gateway --> PaymentService

Gateway --> MerchantService

Gateway --> FraudService

Gateway --> LedgerService

Gateway --> SettlementService

Gateway --> RefundService

Gateway --> ChargebackService

Gateway --> NotificationService

PaymentService --> Kafka

FraudService --> Kafka

SettlementService --> Kafka

RefundService --> Kafka

Kafka --> ReportingService

Kafka --> AnalyticsService

Kafka --> ReconciliationService

PaymentService --> PaymentDB

MerchantService --> MerchantDB

LedgerService --> LedgerDB

SettlementService --> SettlementDB

RefundService --> RefundDB

Final Summary

Designing an enterprise payment platform requires much more than processing transactions. A successful solution must guarantee financial correctness, prevent duplicate charges, detect fraud, reconcile settlements accurately, comply with strict regulatory requirements, and remain highly available under massive transaction volumes.

Across this five-part series, we designed a production-ready payment platform from business requirements to cloud-native deployment. We applied microservices, event-driven architecture, CQRS, Saga Pattern, Kafka, Redis, immutable double-entry accounting, Kubernetes, observability, disaster recovery, and security best practices to build a resilient payment ecosystem capable of supporting millions of secure financial transactions every day.

Key Takeaways

✅ Keep financial records in an immutable double-entry ledger.
✅ Separate payment authorization from capture.
✅ Prevent duplicate payments with idempotency keys.
✅ Tokenize payment instruments instead of storing raw card data.
✅ Reconcile gateway, ledger, settlement, and bank records continuously.
✅ Publish domain events through Kafka for loose coupling.
✅ Scale stateless services horizontally while protecting stateful financial systems.
✅ Monitor business KPIs and technical metrics together.
✅ Design for PCI DSS compliance and secure-by-default architecture.
✅ Build for resilience using retries, circuit breakers, multi-region deployment, and comprehensive observability.