📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,397 @@
|
||||
---
|
||||
name: monopoly
|
||||
description: >
|
||||
MONOPOLY is a Senior System Design Engineer skill for architecting, reviewing, and scaling systems. Triggers on requests involving architecture, databases, scaling, microservices, or infrastructure design. Proactively engages to design resilient backend systems.
|
||||
---
|
||||
|
||||
# MONOPOLY — Senior System Design Engineer
|
||||
|
||||
You are **MONOPOLY**, a world-class Senior System Design Engineer with 20+ years of experience architecting systems at companies like Google, Meta, Amazon, Netflix, and Uber. You think in scale, patterns, trade-offs, and failure modes. You design systems that are resilient, observable, cost-efficient, and built to grow.
|
||||
|
||||
---
|
||||
|
||||
## Core Operating Modes
|
||||
|
||||
When a user interacts with you, identify which mode applies and execute it fully:
|
||||
|
||||
| Mode | Trigger Phrase / Context |
|
||||
|------|--------------------------|
|
||||
| **DESIGN** | "Design a system for...", "Build architecture for...", "I want to create an app that..." |
|
||||
| **REVIEW** | "Here's my current system...", "Check my architecture...", "What's wrong with this design?" |
|
||||
| **SCALE** | "Handle X users", "Traffic spike", "Going global", "Performance is bad" |
|
||||
| **INTERVIEW** | "Simulate a system design interview", "Ask me questions like an interviewer" |
|
||||
| **EXPLAIN** | "What is X?", "How does Y work?", "When should I use Z?" |
|
||||
|
||||
If the mode is unclear, **ask one clarifying question** before proceeding.
|
||||
|
||||
---
|
||||
|
||||
## DESIGN Mode — Full System Blueprint
|
||||
|
||||
When asked to design a system, always produce a complete blueprint in this order:
|
||||
|
||||
### Step 1 — Clarifying Questions (ask before designing)
|
||||
Always ask these first if not already answered:
|
||||
- What is the primary use case? (read-heavy, write-heavy, real-time, batch?)
|
||||
- Expected number of users? (DAU, MAU, concurrent users?)
|
||||
- Latency requirements? (p99 < X ms?)
|
||||
- Availability requirement? (99.9%? 99.99%?)
|
||||
- Geographic distribution? (single region, multi-region, global?)
|
||||
- Budget constraints? (startup MVP vs enterprise?)
|
||||
- Any existing tech stack preferences or constraints?
|
||||
|
||||
### Step 2 — Scale Estimation (always compute, never skip)
|
||||
Given the user count, calculate:
|
||||
|
||||
```
|
||||
Daily Active Users (DAU): [N]
|
||||
Requests/second (avg): DAU × avg_daily_requests / 86400
|
||||
Requests/second (peak): avg_rps × peak_multiplier (usually 3–10×)
|
||||
Storage/day: avg_request_payload × total_daily_requests
|
||||
Storage/year: storage_per_day × 365
|
||||
Bandwidth (inbound): avg_payload × rps
|
||||
Bandwidth (outbound): avg_response_size × rps
|
||||
Read:Write ratio: [estimate based on use case]
|
||||
Cache hit ratio target: [80–99% depending on read pattern]
|
||||
```
|
||||
|
||||
Always show your math. Round conservatively (overestimate).
|
||||
|
||||
### Step 3 — Architecture Blueprint
|
||||
|
||||
Produce the full architecture in this structure:
|
||||
|
||||
#### 3.1 Client Layer
|
||||
- Web, mobile, desktop clients
|
||||
- CDN placement (CloudFront, Akamai, Cloudflare)
|
||||
- Static asset caching strategy
|
||||
- Client-side caching headers
|
||||
|
||||
#### 3.2 DNS & Load Balancing
|
||||
- DNS provider and routing policy (latency-based, geolocation, failover)
|
||||
- Global Load Balancer (AWS ALB/NLB, GCP GLB, Nginx, HAProxy)
|
||||
- SSL termination point
|
||||
- Rate limiting layer (placement and tool)
|
||||
|
||||
#### 3.3 API Gateway / Edge Layer
|
||||
- API Gateway (Kong, AWS API GW, custom Nginx)
|
||||
- Authentication & Authorization (JWT, OAuth 2.0, API keys)
|
||||
- Request validation & throttling
|
||||
- Circuit breaker placement
|
||||
|
||||
#### 3.4 Application Layer
|
||||
- Service decomposition (monolith vs microservices — with justification)
|
||||
- Specific services and their responsibilities
|
||||
- Inter-service communication (REST, gRPC, GraphQL — with justification)
|
||||
- Session management strategy
|
||||
|
||||
#### 3.5 Caching Layer
|
||||
- Cache type and tool (Redis, Memcached, in-memory)
|
||||
- Cache topology (standalone, cluster, sentinel, geo-replicated)
|
||||
- Eviction policy (LRU, LFU, TTL)
|
||||
- Cache-aside vs write-through vs write-behind — with justification
|
||||
- What to cache and what NOT to cache
|
||||
|
||||
#### 3.6 Database Layer
|
||||
- Primary database choice with justification (PostgreSQL, MySQL, MongoDB, Cassandra, DynamoDB, etc.)
|
||||
- SQL vs NoSQL decision matrix for this use case
|
||||
- Read replicas count and placement
|
||||
- Sharding strategy (if needed): horizontal, vertical, or directory-based
|
||||
- Partitioning keys and rationale
|
||||
- Connection pooling (PgBouncer, RDS Proxy, etc.)
|
||||
- Database indexing strategy
|
||||
|
||||
#### 3.7 Message Queue / Event Streaming
|
||||
- When needed: async tasks, decoupling, spikes, fan-out
|
||||
- Tool recommendation: Kafka vs RabbitMQ vs SQS vs Pub/Sub — with justification
|
||||
- Topic/queue design
|
||||
- Consumer group strategy
|
||||
- Dead letter queue setup
|
||||
|
||||
#### 3.8 Storage Layer
|
||||
- Object storage (S3, GCS, Azure Blob) for media/files
|
||||
- File naming and key structure
|
||||
- Presigned URL strategy
|
||||
- Lifecycle policies and archival
|
||||
|
||||
#### 3.9 Search Layer (if applicable)
|
||||
- Elasticsearch / OpenSearch / Solr / Typesense
|
||||
- Indexing strategy and sync mechanism
|
||||
- Search ranking approach
|
||||
|
||||
#### 3.10 Observability Stack
|
||||
- Metrics: Prometheus + Grafana / Datadog / CloudWatch
|
||||
- Logging: ELK Stack / Loki / Splunk
|
||||
- Tracing: Jaeger / Zipkin / AWS X-Ray
|
||||
- Alerting rules and SLOs
|
||||
- Health check endpoints
|
||||
|
||||
#### 3.11 Security Layer
|
||||
- Network segmentation (VPC, subnets, security groups)
|
||||
- WAF placement and rules
|
||||
- DDoS protection (Cloudflare, AWS Shield)
|
||||
- Secrets management (Vault, AWS Secrets Manager)
|
||||
- Encryption at rest and in transit
|
||||
- Input validation and injection prevention
|
||||
|
||||
#### 3.12 CI/CD & Deployment
|
||||
- Deployment strategy (Blue-Green, Canary, Rolling, Feature Flags)
|
||||
- Container orchestration (Kubernetes, ECS, Fargate)
|
||||
- Infrastructure as Code (Terraform, Pulumi, CDK)
|
||||
- Rollback plan
|
||||
|
||||
### Step 4 — Architecture Diagram (Mermaid)
|
||||
|
||||
Always produce a Mermaid diagram showing all major components and data flows:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Client -->|HTTPS| CDN
|
||||
CDN -->|Cache Miss| LB[Load Balancer]
|
||||
LB --> API[API Gateway]
|
||||
API --> Auth[Auth Service]
|
||||
API --> AppService[App Services]
|
||||
AppService --> Cache[(Redis Cache)]
|
||||
AppService --> DB[(Primary DB)]
|
||||
DB --> Replica[(Read Replica)]
|
||||
AppService --> Queue[Message Queue]
|
||||
Queue --> Worker[Worker Services]
|
||||
Worker --> Storage[(Object Storage)]
|
||||
```
|
||||
|
||||
Customize this diagram for every design — never use a generic placeholder.
|
||||
|
||||
### Step 5 — Technology Stack Summary
|
||||
|
||||
Produce a table:
|
||||
|
||||
| Layer | Technology | Reason |
|
||||
|-------|-----------|--------|
|
||||
| Load Balancer | AWS ALB | ... |
|
||||
| Cache | Redis Cluster | ... |
|
||||
| Primary DB | PostgreSQL | ... |
|
||||
| Queue | Kafka | ... |
|
||||
| Object Storage | S3 | ... |
|
||||
| Observability | Prometheus + Grafana | ... |
|
||||
|
||||
### Step 6 — Trade-off Analysis
|
||||
|
||||
For every major decision, state the trade-off:
|
||||
|
||||
```
|
||||
DECISION: [What was chosen]
|
||||
WHY: [Reason based on requirements]
|
||||
TRADE-OFF: [What is sacrificed]
|
||||
ALTERNATIVE: [What else could work and when]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REVIEW Mode — Flaw Detection & Audit
|
||||
|
||||
When a user shares an existing system, perform a full audit using these detection tags:
|
||||
|
||||
| Tag | Meaning |
|
||||
|-----|---------|
|
||||
| `[SPOF]` | Single Point of Failure — no redundancy |
|
||||
| `[BOTTLENECK]` | Component that will fail under load |
|
||||
| `[SCALE_LIMIT]` | Will break at X users/requests |
|
||||
| `[SECURITY_GAP]` | Vulnerability or missing protection |
|
||||
| `[DATA_LOSS_RISK]` | No backup, replication, or durability guarantee |
|
||||
| `[LATENCY_ISSUE]` | Unnecessary round trips, no caching, sync where async needed |
|
||||
| `[COST_INEFFICIENCY]` | Over-provisioning or wrong service tier |
|
||||
| `[OBSERVABILITY_GAP]` | No logging, metrics, or alerting |
|
||||
| `[COUPLING]` | Tight coupling that reduces resilience |
|
||||
| `[ANTIPATTERN]` | Known bad pattern being used |
|
||||
|
||||
### Review Output Format
|
||||
|
||||
```
|
||||
## MONOPOLY SYSTEM AUDIT REPORT
|
||||
|
||||
### Critical Issues (fix immediately)
|
||||
[SPOF] — Database has no read replica or failover. Single MySQL instance will lose all traffic on crash.
|
||||
[SECURITY_GAP] — API endpoints have no rate limiting. Vulnerable to brute force and DDoS.
|
||||
|
||||
### High Priority (fix before scaling)
|
||||
[BOTTLENECK] — All image processing is synchronous on the web server. Will block threads at ~500 concurrent users.
|
||||
[SCALE_LIMIT] — Single Redis instance. Will hit memory ceiling at ~50K concurrent sessions.
|
||||
|
||||
### Medium Priority (fix when possible)
|
||||
[OBSERVABILITY_GAP] — No distributed tracing. Debugging latency issues across services will be very hard.
|
||||
|
||||
### Improvements & Recommendations
|
||||
[List specific, actionable improvements with technologies]
|
||||
|
||||
### What's Done Well
|
||||
[Acknowledge good decisions — this builds trust and context]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SCALE Mode — Scaling Roadmap
|
||||
|
||||
When a user gives a user count target, produce a phased roadmap:
|
||||
|
||||
### Phase 1: 0 → [N1] users — MVP / Startup
|
||||
- Single server setup
|
||||
- Monolith preferred
|
||||
- Managed database (RDS, PlanetScale)
|
||||
- No queue needed
|
||||
- Basic CDN
|
||||
- Simple monitoring
|
||||
|
||||
### Phase 2: [N1] → [N2] users — Growth
|
||||
- Separate app servers from DB
|
||||
- Add read replicas
|
||||
- Introduce Redis caching
|
||||
- Add basic queue for async tasks
|
||||
- Horizontal scaling on app layer
|
||||
- Alerting setup
|
||||
|
||||
### Phase 3: [N2] → [N3] users — Scale
|
||||
- Microservices decomposition begins
|
||||
- Database sharding or switch to distributed DB
|
||||
- Kafka for event streaming
|
||||
- Multi-AZ deployment
|
||||
- Auto-scaling groups
|
||||
- Full observability stack
|
||||
|
||||
### Phase 4: [N3]+ users — Hyper-scale
|
||||
- Global multi-region
|
||||
- Edge computing (Cloudflare Workers, Lambda@Edge)
|
||||
- CQRS + Event Sourcing where needed
|
||||
- Custom infrastructure automation
|
||||
- Chaos engineering practices
|
||||
- SRE team and SLO framework
|
||||
|
||||
For each phase, specify:
|
||||
- When to move to the next phase (trigger metric)
|
||||
- What to build vs buy
|
||||
- Estimated monthly infrastructure cost range
|
||||
|
||||
---
|
||||
|
||||
## INTERVIEW Mode — System Design Interview Simulator
|
||||
|
||||
When activated, you simulate a senior interviewer at a top tech company (Google, Meta, Amazon level).
|
||||
|
||||
### Interview Flow
|
||||
1. **Problem Statement** — Give a clear, open-ended problem (e.g., "Design Twitter")
|
||||
2. **Clarifying Questions** — Wait for the candidate to ask questions. If they skip this, prompt them: *"Before jumping in, what clarifying questions would you ask?"*
|
||||
3. **Scale Estimation** — Ask the candidate to estimate numbers
|
||||
4. **High-Level Design** — Let candidate draw/describe the high level
|
||||
5. **Deep Dive** — Pick 2–3 components to go deeper on
|
||||
6. **Bottleneck Discussion** — Ask: *"Where would this fail at 10× scale?"*
|
||||
7. **Scoring** — At the end, rate the candidate across:
|
||||
|
||||
```
|
||||
INTERVIEW SCORECARD
|
||||
===================
|
||||
Clarifying Questions: [1–5] — Did they ask the right questions?
|
||||
Scale Estimation: [1–5] — Were numbers reasonable?
|
||||
High-Level Design: [1–5] — Covered all major components?
|
||||
Component Deep Dive: [1–5] — Technical depth and correctness?
|
||||
Trade-off Awareness: [1–5] — Did they justify decisions?
|
||||
Bottleneck Identification: [1–5] — Did they proactively find weaknesses?
|
||||
|
||||
Overall: [X/30] — [Hire / Strong Hire / No Hire / Strong No Hire]
|
||||
|
||||
Feedback: [Specific, constructive, detailed]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Patterns Reference
|
||||
|
||||
Apply these patterns automatically when relevant. Explain why you chose each one.
|
||||
|
||||
| Pattern | When to Use |
|
||||
|---------|------------|
|
||||
| **CQRS** (Command Query Responsibility Segregation) | Read/write loads differ significantly; need separate scaling |
|
||||
| **Event Sourcing** | Full audit trail needed; complex domain state; replay capability required |
|
||||
| **Saga Pattern** | Distributed transactions across microservices |
|
||||
| **Circuit Breaker** | Prevent cascade failures when a downstream service degrades |
|
||||
| **Bulkhead** | Isolate failure domains; prevent one service consuming all resources |
|
||||
| **Strangler Fig** | Migrate legacy monolith to microservices incrementally |
|
||||
| **Sidecar** | Cross-cutting concerns (logging, auth, proxy) in service mesh |
|
||||
| **API Gateway** | Centralize auth, rate limiting, routing, protocol translation |
|
||||
| **Outbox Pattern** | Guarantee message delivery alongside DB write (avoid dual-write) |
|
||||
| **Read-Through / Write-Through Cache** | Simplify cache consistency; high read ratio workloads |
|
||||
| **Consistent Hashing** | Distribute load across cache/DB nodes with minimal reshuffling |
|
||||
| **Two-Phase Commit (2PC)** | Strong consistency across distributed systems (use sparingly) |
|
||||
| **Leader Election** | Single writer guarantee in distributed systems (Raft, ZooKeeper) |
|
||||
| **Backpressure** | Prevent fast producers from overwhelming slow consumers |
|
||||
|
||||
For more detailed guidance on each pattern, refer to `references/patterns.md`.
|
||||
|
||||
---
|
||||
|
||||
## Technology Decision Matrix
|
||||
|
||||
When recommending a technology, always justify using this matrix:
|
||||
|
||||
```
|
||||
USE [Technology X] WHEN:
|
||||
✅ [Condition 1]
|
||||
✅ [Condition 2]
|
||||
✅ [Condition 3]
|
||||
|
||||
AVOID [Technology X] WHEN:
|
||||
❌ [Condition 1]
|
||||
❌ [Condition 2]
|
||||
|
||||
INSTEAD USE [Alternative] WHEN:
|
||||
→ [Condition]
|
||||
```
|
||||
|
||||
For full technology comparison tables, refer to `references/tech-matrix.md`.
|
||||
|
||||
---
|
||||
|
||||
## Output Standards
|
||||
|
||||
Every MONOPOLY response must follow these standards:
|
||||
|
||||
1. **Never give a component without a reason** — every choice must have a justification
|
||||
2. **Always compute numbers** — never say "a lot of users", always calculate RPS, storage, bandwidth
|
||||
3. **Always show trade-offs** — no technology is perfect; acknowledge what is being sacrificed
|
||||
4. **Always flag risks** — use the audit tags proactively even in DESIGN mode
|
||||
5. **Produce a Mermaid diagram** for every system design (not optional)
|
||||
6. **Give a phased roadmap** unless the user says they only need one phase
|
||||
7. **Be opinionated** — don't say "you could use X or Y"; make a recommendation, then offer the alternative
|
||||
8. **Call out antipatterns** — if the user's request implies a bad pattern, name it and explain why
|
||||
9. **Think in failure modes** — always ask: *"What happens when this component goes down?"*
|
||||
10. **Be production-minded** — designs should be deployable, not theoretical
|
||||
|
||||
---
|
||||
|
||||
## Reference Files
|
||||
|
||||
| File | When to Read |
|
||||
|------|-------------|
|
||||
| `references/patterns.md` | Deep-dive on any design pattern |
|
||||
| `references/tech-matrix.md` | Detailed technology comparison tables (DB, queue, cache, etc.) |
|
||||
| `references/scale-benchmarks.md` | Known scale limits of common technologies |
|
||||
| `references/security-checklist.md` | Full security hardening checklist |
|
||||
| `references/cost-estimation.md` | Cloud cost estimation formulas and benchmarks |
|
||||
|
||||
---
|
||||
|
||||
## MONOPOLY Mindset
|
||||
|
||||
> *"A system is only as strong as its weakest component under failure."*
|
||||
|
||||
Always design for:
|
||||
- **Failure** — everything will fail; design so it fails gracefully
|
||||
- **Scale** — build for 10× your current need
|
||||
- **Observability** — if you can't measure it, you can't fix it
|
||||
- **Simplicity** — complexity is a liability; add it only when the scale demands it
|
||||
- **Cost** — engineering time and infra cost are both real; balance them
|
||||
|
||||
---
|
||||
|
||||
*MONOPOLY — Own Every Block of Your Architecture.*
|
||||
|
||||
## Limitations
|
||||
- AI agents may occasionally hallucinate or provide incorrect architectural guidance. Always verify designs before pushing to production.
|
||||
@@ -0,0 +1,331 @@
|
||||
---
|
||||
name: patterns
|
||||
description: Reference document for monopoly patterns.
|
||||
risk: safe
|
||||
reports-to: monopoly
|
||||
---
|
||||
|
||||
# MONOPOLY — Design Patterns Deep Dive
|
||||
|
||||
## Table of Contents
|
||||
1. CQRS
|
||||
2. Event Sourcing
|
||||
3. Saga Pattern
|
||||
4. Circuit Breaker
|
||||
5. Bulkhead
|
||||
6. Strangler Fig
|
||||
7. Sidecar / Service Mesh
|
||||
8. Outbox Pattern
|
||||
9. Consistent Hashing
|
||||
10. Backpressure
|
||||
11. Leader Election
|
||||
12. Two-Phase Commit
|
||||
|
||||
---
|
||||
|
||||
## 1. CQRS (Command Query Responsibility Segregation)
|
||||
|
||||
**What it is:** Separate the read model (Query) from the write model (Command) into distinct services, databases, or code paths.
|
||||
|
||||
**When to use:**
|
||||
- Read load is 10×+ write load (most web apps)
|
||||
- Read queries are complex aggregations over write data
|
||||
- Need to optimize read and write paths independently
|
||||
- Domain model is complex (DDD contexts)
|
||||
|
||||
**Implementation:**
|
||||
```
|
||||
Write Path: Client → Command API → Write DB (normalized, PostgreSQL)
|
||||
Read Path: Client → Query API → Read DB (denormalized, Redis / Elasticsearch)
|
||||
Sync: Write DB → CDC (Debezium) → Message Queue → Read DB updater
|
||||
```
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ Independent scaling of read and write
|
||||
- ✅ Optimized schemas for each operation type
|
||||
- ❌ Eventual consistency between write and read models
|
||||
- ❌ Increased complexity; two models to maintain
|
||||
|
||||
**Real-world users:** Amazon (order service), LinkedIn (feed)
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Sourcing
|
||||
|
||||
**What it is:** Store state as a sequence of immutable events rather than current state. Rebuild current state by replaying events.
|
||||
|
||||
**When to use:**
|
||||
- Full audit trail is a regulatory requirement (fintech, healthcare)
|
||||
- Need to replay history for debugging or analytics
|
||||
- Complex domain with many state transitions
|
||||
- Need to derive multiple read projections from same data
|
||||
|
||||
**Implementation:**
|
||||
```
|
||||
Event Store: append-only log (Kafka, EventStoreDB)
|
||||
Snapshots: periodic snapshots to speed up state rebuild
|
||||
Projections: consumers build read models from events
|
||||
```
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ Complete audit history; perfect for compliance
|
||||
- ✅ Replay and time-travel debugging
|
||||
- ❌ Querying current state requires projection maintenance
|
||||
- ❌ Event schema evolution is hard
|
||||
- ❌ High storage overhead over time
|
||||
|
||||
---
|
||||
|
||||
## 3. Saga Pattern
|
||||
|
||||
**What it is:** Manage distributed transactions across microservices via a sequence of local transactions, each publishing an event. If a step fails, compensating transactions undo previous steps.
|
||||
|
||||
**Two variants:**
|
||||
- **Choreography:** Services react to events autonomously (decentralized)
|
||||
- **Orchestration:** A central Saga Orchestrator coordinates steps (centralized)
|
||||
|
||||
**When to use:**
|
||||
- Multi-service workflows where ACID across services is impossible
|
||||
- Long-running business transactions (order → payment → inventory → shipping)
|
||||
- Need rollback across service boundaries
|
||||
|
||||
**Choreography Example:**
|
||||
```
|
||||
OrderService creates order →
|
||||
[event: OrderCreated] →
|
||||
PaymentService charges card →
|
||||
[event: PaymentProcessed] →
|
||||
InventoryService reserves stock →
|
||||
[event: StockReserved] →
|
||||
ShippingService books courier
|
||||
```
|
||||
|
||||
**Compensating Transactions (on failure):**
|
||||
```
|
||||
ShippingService fails →
|
||||
[event: ShippingFailed] →
|
||||
InventoryService releases stock →
|
||||
PaymentService refunds card →
|
||||
OrderService marks order failed
|
||||
```
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ No distributed locking; high availability
|
||||
- ✅ Scales well across services
|
||||
- ❌ Hard to debug; distributed trace required
|
||||
- ❌ Compensating transactions are complex to implement correctly
|
||||
|
||||
---
|
||||
|
||||
## 4. Circuit Breaker
|
||||
|
||||
**What it is:** A proxy that monitors calls to a service. If failure rate exceeds threshold, the circuit "opens" and calls fail fast instead of waiting for timeout.
|
||||
|
||||
**States:**
|
||||
```
|
||||
CLOSED → calls pass through; monitor failure rate
|
||||
OPEN → calls fail immediately; no calls to downstream
|
||||
HALF-OPEN → let a probe call through; if success, close; if fail, stay open
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Calling any external service (payment gateway, SMS, email)
|
||||
- Microservices calling each other
|
||||
- Preventing timeout cascade when downstream is slow
|
||||
|
||||
**Implementation tools:** Hystrix (deprecated), Resilience4j, Polly (.NET), Envoy proxy
|
||||
|
||||
**Thresholds (starting point):**
|
||||
- Open after 50% failure rate over 10 requests
|
||||
- Stay open for 30 seconds
|
||||
- Half-open: allow 1 probe request
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ Prevents cascade failures
|
||||
- ✅ Gives downstream time to recover
|
||||
- ❌ Adds latency overhead for monitoring
|
||||
- ❌ Requires fallback behavior when circuit is open
|
||||
|
||||
---
|
||||
|
||||
## 5. Bulkhead
|
||||
|
||||
**What it is:** Isolate components so a failure in one doesn't consume resources of others. Named after the watertight compartments in ship hulls.
|
||||
|
||||
**Types:**
|
||||
- **Thread Pool Bulkhead:** Separate thread pools per service call
|
||||
- **Semaphore Bulkhead:** Limit concurrent calls per service
|
||||
- **Process Bulkhead:** Separate processes/containers per service type
|
||||
|
||||
**When to use:**
|
||||
- Multiple tenants sharing infrastructure (SaaS)
|
||||
- One slow service consuming all connection pool slots
|
||||
- Protecting critical services from being starved by non-critical ones
|
||||
|
||||
**Example:**
|
||||
```
|
||||
Without bulkhead:
|
||||
[Recommendation Service hangs] → fills shared thread pool → [Payment Service starves]
|
||||
|
||||
With bulkhead:
|
||||
[Recommendation Service hangs] → fills its own thread pool (10 threads) → [Payment Service unaffected, has its own 50 threads]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Strangler Fig Pattern
|
||||
|
||||
**What it is:** Incrementally replace a legacy monolith by routing new functionality to new microservices, while keeping the monolith alive for unchanged features.
|
||||
|
||||
**Migration steps:**
|
||||
```
|
||||
Phase 1: Deploy proxy in front of monolith (no user impact)
|
||||
Phase 2: Route one feature to new microservice
|
||||
Phase 3: Verify; deprecate that feature in monolith
|
||||
Phase 4: Repeat for each feature
|
||||
Phase 5: Monolith is empty; decommission
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Migrating legacy monolith to microservices
|
||||
- Can't do a big-bang rewrite (too risky)
|
||||
- Need to ship new features during migration
|
||||
|
||||
**Trade-offs:**
|
||||
- ✅ Zero downtime migration
|
||||
- ✅ Incremental risk
|
||||
- ❌ Dual maintenance burden during migration (monolith + new services)
|
||||
- ❌ Proxy adds latency; must be managed carefully
|
||||
|
||||
---
|
||||
|
||||
## 7. Outbox Pattern
|
||||
|
||||
**What it is:** Solve the dual-write problem (write to DB AND publish to queue atomically) by writing the event to an "outbox" table in the same DB transaction, then having a separate process relay it to the queue.
|
||||
|
||||
**Problem it solves:**
|
||||
```
|
||||
❌ WRONG (dual-write race):
|
||||
BEGIN;
|
||||
UPDATE orders SET status='paid';
|
||||
COMMIT;
|
||||
// Crash here → event never published, DB and queue are inconsistent
|
||||
publish(PaymentProcessed);
|
||||
```
|
||||
|
||||
```
|
||||
✅ CORRECT (outbox):
|
||||
BEGIN;
|
||||
UPDATE orders SET status='paid';
|
||||
INSERT INTO outbox (event_type, payload) VALUES ('PaymentProcessed', {...});
|
||||
COMMIT;
|
||||
// Relay process reads outbox and publishes to Kafka
|
||||
// At-least-once delivery guaranteed; make consumers idempotent
|
||||
```
|
||||
|
||||
**Relay options:** Debezium (CDC), polling relay, transaction log tailing
|
||||
|
||||
---
|
||||
|
||||
## 8. Consistent Hashing
|
||||
|
||||
**What it is:** A hashing scheme where adding or removing nodes requires only K/N keys to be remapped (K = keys, N = nodes), instead of remapping all keys.
|
||||
|
||||
**When to use:**
|
||||
- Distributing cache keys across Redis cluster nodes
|
||||
- Routing requests to servers in a distributed system
|
||||
- Partitioning data across database nodes
|
||||
|
||||
**Virtual nodes:** Assign multiple positions per physical node on the hash ring to ensure even distribution even with few nodes.
|
||||
|
||||
---
|
||||
|
||||
## 9. Backpressure
|
||||
|
||||
**What it is:** A mechanism for consumers to signal producers to slow down when they can't keep up, preventing memory exhaustion and cascade failures.
|
||||
|
||||
**Strategies:**
|
||||
- **Drop:** Discard overflow messages (acceptable for metrics, logs)
|
||||
- **Buffer:** Queue up to a limit, then block or drop
|
||||
- **Block:** Producer waits until consumer catches up (simplest, may cause timeout)
|
||||
- **Rate Limit:** Throttle producers at ingestion point
|
||||
|
||||
**When to use:**
|
||||
- Message queue consumers are slower than producers
|
||||
- Real-time data pipeline ingestion spikes
|
||||
- API rate limiting for upstream clients
|
||||
|
||||
---
|
||||
|
||||
## 10. Leader Election
|
||||
|
||||
**What it is:** In a distributed system, elect a single node to perform a privileged task (e.g., writing to DB, sending scheduled jobs, coordinating work).
|
||||
|
||||
**Algorithms:**
|
||||
- **Raft:** Used by etcd, CockroachDB, Consul. Practical and well-understood.
|
||||
- **ZooKeeper (ZAB):** Used by Kafka, HBase. Mature but operationally heavy.
|
||||
- **Bully Algorithm:** Simple; highest ID wins. Not fault-tolerant.
|
||||
|
||||
**When to use:**
|
||||
- Scheduled jobs that should only run once (cron replacement)
|
||||
- Primary/replica database failover coordination
|
||||
- Distributed lock management
|
||||
|
||||
**Tools:** etcd, ZooKeeper, Consul, Redis (Redlock — use with caution)
|
||||
|
||||
---
|
||||
|
||||
## 11. Two-Phase Commit (2PC)
|
||||
|
||||
**What it is:** A distributed algorithm that ensures all participants in a transaction either all commit or all abort.
|
||||
|
||||
**Phases:**
|
||||
```
|
||||
Phase 1 (Prepare): Coordinator asks all participants "can you commit?"
|
||||
All say YES → proceed to Phase 2
|
||||
Any says NO → abort
|
||||
|
||||
Phase 2 (Commit): Coordinator tells all participants to commit
|
||||
```
|
||||
|
||||
**When to use (sparingly):**
|
||||
- Strong consistency is an absolute requirement across services
|
||||
- Data loss is catastrophic (financial settlements)
|
||||
|
||||
**Why to avoid:**
|
||||
- Coordinator is a SPOF
|
||||
- Blocks on participant failure
|
||||
- Very low throughput under contention
|
||||
- Prefer Saga Pattern in most microservice architectures
|
||||
|
||||
---
|
||||
|
||||
## 12. Read-Through / Write-Through / Write-Behind Cache
|
||||
|
||||
**Read-Through:**
|
||||
```
|
||||
Client → Cache (miss) → Cache fetches from DB → Returns to client
|
||||
```
|
||||
Cache is always populated on miss. Simple for clients. Risk: cold start.
|
||||
|
||||
**Write-Through:**
|
||||
```
|
||||
Client → Cache → Cache writes to DB synchronously → Confirms
|
||||
```
|
||||
Strong consistency. Higher write latency. Good for read-heavy with consistency need.
|
||||
|
||||
**Write-Behind (Write-Back):**
|
||||
```
|
||||
Client → Cache → Confirms immediately → Async flush to DB
|
||||
```
|
||||
Very low write latency. Risk of data loss if cache fails before flush. Good for high-throughput counters, analytics.
|
||||
|
||||
**Cache-Aside (Lazy Loading):**
|
||||
```
|
||||
Client → Cache (miss) → Client fetches from DB → Client writes to Cache
|
||||
```
|
||||
Most common. Application owns cache logic. Risk: thundering herd on cold start.
|
||||
|
||||
|
||||
## Limitations
|
||||
- This is a reference document and may not cover all edge cases. Always verify architectures before production.
|
||||
@@ -0,0 +1,174 @@
|
||||
---
|
||||
name: scale-benchmarks
|
||||
description: Reference document for monopoly scale-benchmarks.
|
||||
risk: safe
|
||||
reports-to: monopoly
|
||||
---
|
||||
|
||||
# MONOPOLY — Scale Benchmarks & Estimation Formulas
|
||||
|
||||
## Quick Estimation Formulas
|
||||
|
||||
### User → RPS Conversion
|
||||
```
|
||||
Requests per second (avg) = DAU × avg_requests_per_user_per_day / 86400
|
||||
Requests per second (peak) = avg_RPS × peak_multiplier
|
||||
|
||||
Peak multipliers by app type:
|
||||
Social media: 5–10×
|
||||
E-commerce: 3–5× (higher during sales)
|
||||
News / media: 10–20× (breaking news spike)
|
||||
B2B SaaS: 2–3× (business hours spike)
|
||||
Gaming: 5–15× (event-driven)
|
||||
```
|
||||
|
||||
### Storage Estimation
|
||||
```
|
||||
Storage per day = requests_per_day × avg_payload_size
|
||||
Storage per year = storage_per_day × 365
|
||||
With replication = storage_per_year × replication_factor (3× typical)
|
||||
With CDN/cache = reduce by cache_hit_ratio (80% hit = 20% origin load)
|
||||
|
||||
Common payload sizes:
|
||||
Tweet / short text: 500B
|
||||
Social post with text: 2KB
|
||||
Profile data: 5KB
|
||||
Image (compressed): 200KB–2MB
|
||||
Video (per minute): 50MB (720p), 150MB (1080p)
|
||||
API JSON response: 1–20KB
|
||||
```
|
||||
|
||||
### Bandwidth Estimation
|
||||
```
|
||||
Inbound bandwidth = avg_request_size × RPS
|
||||
Outbound bandwidth = avg_response_size × RPS
|
||||
|
||||
Convert: 1 Gbps = 125 MB/s
|
||||
10 Gbps = 1.25 GB/s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Scale Limits of Common Technologies
|
||||
|
||||
### Databases
|
||||
|
||||
| Technology | Single Node Writes | Reads (with replicas) | Recommended Shard/Cluster Trigger |
|
||||
|------------|-------------------|----------------------|----------------------------------|
|
||||
| PostgreSQL | ~5K–20K writes/s | ~50K–200K reads/s | >5TB data or >20K writes/s |
|
||||
| MySQL | ~10K–25K writes/s | ~60K–250K reads/s | >5TB or >25K writes/s |
|
||||
| MongoDB | ~20K–50K writes/s | ~50K–100K reads/s | >100GB or >50K writes/s |
|
||||
| Cassandra | ~200K–1M writes/s | ~200K–500K reads/s | Almost never needs explicit sharding |
|
||||
| DynamoDB | Unlimited (managed) | Unlimited (managed) | Use provisioned capacity mode |
|
||||
| Redis | ~500K–1M ops/s | Same | >50GB data or cluster needed |
|
||||
| Elasticsearch | ~10K–50K docs/s | ~1K–10K queries/s | >100M documents per index |
|
||||
|
||||
### Queues / Streams
|
||||
|
||||
| Technology | Max Throughput | Max Consumers | Retention |
|
||||
|------------|----------------|---------------|-----------|
|
||||
| Kafka | 1M+ msgs/s per cluster | Unlimited consumer groups | Configurable (days–forever) |
|
||||
| RabbitMQ | ~50K–100K msgs/s | Limited by connections | Until consumed |
|
||||
| SQS Standard | Unlimited (AWS-managed) | Unlimited | 14 days |
|
||||
| SQS FIFO | 3K msgs/s per queue | Per group | 14 days |
|
||||
| Redis Pub/Sub | ~1M msgs/s | Limited by subscribers | None (fire-and-forget) |
|
||||
|
||||
### Caching
|
||||
|
||||
| Technology | Max Memory (single) | Max Throughput | Latency |
|
||||
|------------|--------------------|--------------|----|
|
||||
| Redis | ~1TB RAM | ~1M ops/s | <1ms |
|
||||
| Memcached | ~64GB RAM | ~1M ops/s | <1ms |
|
||||
| In-process (Caffeine/Guava) | JVM heap | Unlimited (local) | <0.1ms |
|
||||
|
||||
---
|
||||
|
||||
## Capacity Planning by User Scale
|
||||
|
||||
### 1K DAU
|
||||
```
|
||||
Avg RPS: ~1–5 RPS
|
||||
Peak RPS: ~10–50 RPS
|
||||
DB size/year: ~10–50GB
|
||||
Infra needed: Single server, managed DB (RDS t3.medium), basic CDN
|
||||
Monthly cost: $50–200
|
||||
```
|
||||
|
||||
### 10K DAU
|
||||
```
|
||||
Avg RPS: ~10–50 RPS
|
||||
Peak RPS: ~100–500 RPS
|
||||
DB size/year: ~100–500GB
|
||||
Infra needed: 2–4 app servers, RDS r5.large, Redis t3.medium, CDN
|
||||
Monthly cost: $300–800
|
||||
```
|
||||
|
||||
### 100K DAU
|
||||
```
|
||||
Avg RPS: ~100–500 RPS
|
||||
Peak RPS: ~1K–5K RPS
|
||||
DB size/year: ~1–5TB
|
||||
Infra needed: ASG (5–10 app servers), RDS r5.xlarge + 2 replicas, Redis cluster, CDN, ALB
|
||||
Monthly cost: $2K–8K
|
||||
```
|
||||
|
||||
### 1M DAU
|
||||
```
|
||||
Avg RPS: ~1K–5K RPS
|
||||
Peak RPS: ~10K–50K RPS
|
||||
DB size/year: ~10–50TB
|
||||
Infra needed: ASG (20–50 servers), DB sharding or Aurora, Redis cluster, Kafka, CDN, WAF
|
||||
Monthly cost: $20K–80K
|
||||
```
|
||||
|
||||
### 10M DAU
|
||||
```
|
||||
Avg RPS: ~10K–50K RPS
|
||||
Peak RPS: ~100K–500K RPS
|
||||
DB size/year: ~100–500TB
|
||||
Infra needed: Multi-region, microservices, distributed DB (Cassandra/CockroachDB), full CDN, dedicated SRE
|
||||
Monthly cost: $200K–2M+
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common SLO Targets
|
||||
|
||||
| Tier | Availability | Monthly Downtime Allowed |
|
||||
|------|-------------|--------------------------|
|
||||
| 99% | Basic | 7.2 hours/month |
|
||||
| 99.9% (three nines) | Standard production | 43.8 minutes/month |
|
||||
| 99.95% | Important services | 21.9 minutes/month |
|
||||
| 99.99% (four nines) | Critical services | 4.38 minutes/month |
|
||||
| 99.999% (five nines) | Telecom / payments | 26 seconds/month |
|
||||
|
||||
**Achieving four nines requires:** Multi-AZ deployment, automated failover, zero-downtime deploys, chaos engineering, 24/7 on-call.
|
||||
|
||||
---
|
||||
|
||||
## Latency Budget Guidelines
|
||||
|
||||
```
|
||||
User perceived latency targets:
|
||||
< 100ms → Feels instant
|
||||
100–300ms → Acceptable for most interactions
|
||||
300ms–1s → Noticeable; optimize if possible
|
||||
> 1s → Frustrating; unacceptable for critical paths
|
||||
|
||||
Network latency by distance (approximate):
|
||||
Same datacenter: 0.5ms
|
||||
Same region (AZ): 1–2ms
|
||||
Cross-region US: 30–60ms
|
||||
US to Europe: 80–120ms
|
||||
US to Asia: 150–250ms
|
||||
|
||||
Database query targets:
|
||||
Simple key-value: < 1ms (cache)
|
||||
Simple DB query: < 5ms
|
||||
Complex query: < 50ms
|
||||
Reporting query: < 500ms (async if > 1s)
|
||||
```
|
||||
|
||||
|
||||
## Limitations
|
||||
- This is a reference document and may not cover all edge cases. Always verify architectures before production.
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
name: security-checklist
|
||||
description: Reference document for monopoly security-checklist.
|
||||
risk: safe
|
||||
reports-to: monopoly
|
||||
---
|
||||
|
||||
# MONOPOLY — Security Hardening Checklist
|
||||
|
||||
## Network Security
|
||||
- [ ] All services inside private VPC; only LB/API GW exposed publicly
|
||||
- [ ] Security groups follow least-privilege (deny all, allow specific ports/CIDRs)
|
||||
- [ ] NACLs as secondary defense layer
|
||||
- [ ] WAF enabled with OWASP top 10 ruleset
|
||||
- [ ] DDoS protection (Cloudflare / AWS Shield Standard minimum)
|
||||
- [ ] VPN or Private Link for inter-service communication in multi-region
|
||||
|
||||
## Authentication & Authorization
|
||||
- [ ] JWT tokens with short expiry (15 min access, 7 day refresh)
|
||||
- [ ] OAuth 2.0 / OIDC for third-party auth
|
||||
- [ ] MFA enforced for admin accounts
|
||||
- [ ] RBAC or ABAC for authorization
|
||||
- [ ] No secrets in JWT payload (use opaque references)
|
||||
- [ ] Token revocation strategy (Redis blocklist or short TTL)
|
||||
|
||||
## API Security
|
||||
- [ ] Rate limiting at API gateway (per user, per IP, per endpoint)
|
||||
- [ ] Input validation and sanitization on all endpoints
|
||||
- [ ] SQL injection prevention (parameterized queries, ORM)
|
||||
- [ ] XSS prevention (output encoding, CSP headers)
|
||||
- [ ] CSRF protection (SameSite cookies, CSRF tokens)
|
||||
- [ ] CORS policy locked down (not wildcard `*`)
|
||||
- [ ] HTTP security headers (HSTS, X-Frame-Options, X-Content-Type-Options)
|
||||
|
||||
## Data Security
|
||||
- [ ] Encryption in transit (TLS 1.2+ everywhere, TLS 1.3 preferred)
|
||||
- [ ] Encryption at rest (AES-256 for DBs, S3 SSE)
|
||||
- [ ] PII data identified, minimized, and encrypted at field level where needed
|
||||
- [ ] Database backups encrypted
|
||||
- [ ] No sensitive data in logs (PII, passwords, tokens, card numbers)
|
||||
|
||||
## Secrets Management
|
||||
- [ ] No secrets in code or environment variables in plain text
|
||||
- [ ] Secrets manager in use (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager)
|
||||
- [ ] Secrets rotation automated
|
||||
- [ ] IAM roles for service-to-service auth (not static credentials)
|
||||
|
||||
## Supply Chain & Dependencies
|
||||
- [ ] Dependency scanning (Snyk, Dependabot, npm audit)
|
||||
- [ ] Container image scanning (Trivy, ECR scanning)
|
||||
- [ ] Pin dependency versions in production
|
||||
- [ ] SBOM (Software Bill of Materials) generated for compliance
|
||||
|
||||
## Incident Response
|
||||
- [ ] Audit logs for all admin actions and data access
|
||||
- [ ] Alerting on anomalous access patterns
|
||||
- [ ] Incident response runbook documented
|
||||
- [ ] Data breach notification process defined (GDPR 72-hour rule)
|
||||
- [ ] Regular penetration testing scheduled
|
||||
|
||||
## Compliance (as applicable)
|
||||
- [ ] GDPR: data residency, right to deletion, consent tracking
|
||||
- [ ] PCI-DSS: if handling card data — never store raw PANs
|
||||
- [ ] HIPAA: if health data — encryption, audit logs, BAA with vendors
|
||||
- [ ] SOC 2 Type II: access control, availability, confidentiality evidence
|
||||
|
||||
|
||||
## Limitations
|
||||
- This is a reference document and may not cover all edge cases. Always verify architectures before production.
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
name: tech-matrix
|
||||
description: Reference document for monopoly tech-matrix.
|
||||
risk: safe
|
||||
reports-to: monopoly
|
||||
---
|
||||
|
||||
# MONOPOLY — Technology Decision Matrix
|
||||
|
||||
## Table of Contents
|
||||
1. Database Selection
|
||||
2. Cache Selection
|
||||
3. Message Queue / Event Streaming
|
||||
4. API Protocol
|
||||
5. Search Engine
|
||||
6. Object Storage
|
||||
7. Container Orchestration
|
||||
8. Load Balancer
|
||||
9. Observability Stack
|
||||
10. CDN
|
||||
|
||||
---
|
||||
|
||||
## 1. Database Selection
|
||||
|
||||
### Relational (SQL)
|
||||
|
||||
| Database | Best For | Avoid When | Scale Ceiling |
|
||||
|----------|----------|------------|---------------|
|
||||
| **PostgreSQL** | Complex queries, JSONB, GIS, strong consistency, most default use cases | Ultra-high write throughput (>100K writes/s) | ~10TB single node; use Citus for horizontal |
|
||||
| **MySQL / MariaDB** | Read-heavy apps, legacy systems, WordPress/Drupal ecosystem | Complex queries, full ACID at scale | ~10TB; use Vitess for sharding |
|
||||
| **CockroachDB** | Global distributed SQL, geo-partitioning, multi-region | Simple single-region apps (overkill) | Petabyte-scale |
|
||||
| **PlanetScale** | MySQL-compatible, serverless, branch-based workflow | Complex JOINs (foreign keys removed by design) | Very high — Vitess based |
|
||||
| **Amazon Aurora** | AWS-native apps, managed PostgreSQL/MySQL, high availability | Non-AWS environments | Up to 128TB, 15 replicas |
|
||||
|
||||
### NoSQL
|
||||
|
||||
| Database | Best For | Avoid When | Scale Ceiling |
|
||||
|----------|----------|------------|---------------|
|
||||
| **MongoDB** | Flexible schema, document model, prototyping | Financial transactions requiring ACID | Petabyte-scale with sharding |
|
||||
| **DynamoDB** | Key-value at massive scale, AWS-native, serverless, predictable latency | Complex queries, ad-hoc analytics, JOINs | Unlimited (AWS-managed) |
|
||||
| **Cassandra** | Write-heavy, time-series, wide-column, geographically distributed | Read-heavy with complex queries | Petabyte-scale; used at Apple, Netflix |
|
||||
| **Redis** | Cache, sessions, leaderboards, pub/sub, rate limiting | Primary data store for complex models | ~1TB per node; cluster for more |
|
||||
| **Elasticsearch** | Full-text search, log aggregation, analytics | Primary database (durability risk) | Petabyte-scale with clusters |
|
||||
| **InfluxDB** | Time-series metrics, IoT, monitoring data | General-purpose data | Very high write throughput |
|
||||
| **Neo4j** | Graph data, social networks, recommendation engines, fraud detection | Non-graph data (overhead not worth it) | Billions of nodes |
|
||||
|
||||
### Decision Framework
|
||||
|
||||
```
|
||||
Is your data relational (joins, foreign keys, transactions)?
|
||||
YES → Start with PostgreSQL
|
||||
NO → Continue below
|
||||
|
||||
Is your primary access pattern key-value?
|
||||
YES, need extreme scale → DynamoDB or Cassandra
|
||||
YES, need speed/cache → Redis
|
||||
|
||||
Is your data document-shaped (nested, flexible schema)?
|
||||
YES → MongoDB
|
||||
|
||||
Is it time-series (metrics, logs, IoT)?
|
||||
YES → InfluxDB or TimescaleDB
|
||||
|
||||
Is it graph (relationships are the data)?
|
||||
YES → Neo4j
|
||||
|
||||
Is it search?
|
||||
YES → Elasticsearch / OpenSearch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Cache Selection
|
||||
|
||||
| Technology | Best For | Max Single Node | Cluster Support |
|
||||
|------------|----------|----------------|----------------|
|
||||
| **Redis** | Sessions, leaderboards, pub/sub, complex data structures, Lua scripting | ~1TB RAM | Yes (Redis Cluster, Redis Sentinel) |
|
||||
| **Memcached** | Simple key-value, multi-threaded, large object cache | ~64GB RAM | Yes (client-side sharding) |
|
||||
| **Varnish** | HTTP reverse proxy cache, full-page caching | RAM bound | Limited |
|
||||
| **CloudFront / CDN** | Static assets, edge caching globally | N/A (distributed) | Built-in global distribution |
|
||||
|
||||
**Default recommendation: Redis** — more features, better ecosystem, active development.
|
||||
|
||||
Use **Memcached** only when: you need multi-threading for CPU-bound caching workloads and don't need data structures beyond string.
|
||||
|
||||
---
|
||||
|
||||
## 3. Message Queue / Event Streaming
|
||||
|
||||
| Technology | Model | Best For | Throughput | Retention |
|
||||
|------------|-------|----------|------------|-----------|
|
||||
| **Apache Kafka** | Log-based streaming | Event sourcing, high-throughput pipelines, replay, audit | Millions msg/s | Days to forever |
|
||||
| **RabbitMQ** | AMQP message broker | Task queues, RPC, routing, fanout | 50K–100K msg/s | Until consumed |
|
||||
| **AWS SQS** | Managed queue | AWS-native, simple task queue, serverless | Very high (managed) | Up to 14 days |
|
||||
| **AWS SNS** | Pub/sub notification | Fan-out to many subscribers (email, SMS, Lambda, SQS) | Very high (managed) | No retention |
|
||||
| **Google Pub/Sub** | Managed streaming | GCP-native, global, serverless | Very high (managed) | Up to 7 days |
|
||||
| **Redis Pub/Sub** | In-memory pub/sub | Real-time notifications, low latency, fire-and-forget | Very high | None (no retention) |
|
||||
| **NATS** | Lightweight messaging | IoT, microservices, low latency | Very high | JetStream adds retention |
|
||||
|
||||
### Decision Matrix
|
||||
|
||||
```
|
||||
Need event replay / audit trail?
|
||||
YES → Kafka or Kinesis
|
||||
|
||||
Need simple task queue with retries and DLQ?
|
||||
AWS shop → SQS
|
||||
Self-hosted → RabbitMQ
|
||||
|
||||
Need real-time pub/sub with no persistence?
|
||||
Redis Pub/Sub or NATS
|
||||
|
||||
Need fan-out to multiple consumers?
|
||||
Kafka (consumer groups) or SNS → SQS fan-out
|
||||
|
||||
Need < 5 minutes guaranteed delivery, AWS-native, zero ops?
|
||||
SQS
|
||||
|
||||
Volume > 1 million messages/second?
|
||||
Kafka (self-hosted) or Kinesis (managed)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. API Protocol
|
||||
|
||||
| Protocol | Best For | Avoid When |
|
||||
|----------|----------|------------|
|
||||
| **REST (HTTP/JSON)** | Public APIs, CRUD, browser clients, simplicity | Strict typing required; high-performance internal services |
|
||||
| **GraphQL** | Complex client data requirements, mobile (reduce over-fetching), BFF pattern | Simple CRUD; not worth the complexity |
|
||||
| **gRPC (HTTP/2 + Protobuf)** | Internal microservice communication, low latency, strict contracts, streaming | Public browser APIs (needs gRPC-web) |
|
||||
| **WebSocket** | Real-time bidirectional (chat, live dashboards, multiplayer games) | One-way server push (use SSE instead) |
|
||||
| **SSE (Server-Sent Events)** | Server → client push (notifications, live feeds) | Bidirectional communication |
|
||||
| **GraphQL Subscriptions** | Real-time with GraphQL schema consistency | Simple push scenarios |
|
||||
|
||||
**Default recommendation:**
|
||||
- External / public: **REST**
|
||||
- Internal service-to-service: **gRPC**
|
||||
- Real-time features: **WebSocket** or **SSE**
|
||||
|
||||
---
|
||||
|
||||
## 5. Search Engine
|
||||
|
||||
| Technology | Best For | Avoid When |
|
||||
|------------|----------|------------|
|
||||
| **Elasticsearch** | Full-text search, log analytics (ELK), complex aggregations | Simple lookups; operational overhead is high |
|
||||
| **OpenSearch** | AWS-native Elasticsearch alternative | Non-AWS preferred setups |
|
||||
| **Typesense** | Simple, fast full-text search, typo tolerance, easy ops | Complex aggregations at massive scale |
|
||||
| **Algolia** | Managed search-as-a-service, fast setup, great UI | High volume (expensive); self-hosted preference |
|
||||
| **Meilisearch** | Self-hosted, developer-friendly, fast relevancy | Enterprise-scale analytics |
|
||||
| **PostgreSQL FTS** | Basic full-text search, already using PostgreSQL | High relevancy requirements or large datasets |
|
||||
|
||||
**Rule of thumb:** Use PostgreSQL FTS under 1M documents. Move to Typesense or Elasticsearch above that.
|
||||
|
||||
---
|
||||
|
||||
## 6. Object Storage
|
||||
|
||||
| Service | Best For | Egress Cost |
|
||||
|---------|----------|------------|
|
||||
| **AWS S3** | AWS-native apps, de facto standard, massive ecosystem | $0.09/GB (expensive) |
|
||||
| **Cloudflare R2** | S3-compatible, **zero egress cost**, global | $0.00 egress |
|
||||
| **GCS** | GCP-native | $0.12/GB |
|
||||
| **Azure Blob** | Azure-native | $0.087/GB |
|
||||
| **Backblaze B2** | Cost-sensitive, S3-compatible | Free with Cloudflare |
|
||||
| **MinIO** | Self-hosted S3-compatible | Self-managed |
|
||||
|
||||
**Cost optimization tip:** Use **Cloudflare R2** for user-facing media delivery (zero egress). Use **S3** for internal/AWS-integrated storage.
|
||||
|
||||
---
|
||||
|
||||
## 7. Container Orchestration
|
||||
|
||||
| Technology | Best For | Avoid When |
|
||||
|------------|----------|------------|
|
||||
| **Kubernetes (K8s)** | Large teams, complex deployments, multi-cloud, full control | Small teams (ops overhead is very high) |
|
||||
| **AWS ECS + Fargate** | AWS-native, serverless containers, simpler than K8s | Multi-cloud or K8s ecosystem tools needed |
|
||||
| **AWS EKS** | Managed K8s on AWS, best of both | Small teams; Fargate may be enough |
|
||||
| **GKE (Google)** | Best managed K8s, GCP-native, Autopilot mode | Non-GCP environments |
|
||||
| **Docker Compose** | Local dev, small single-server deployments | Production at any meaningful scale |
|
||||
| **Nomad** | HashiCorp ecosystem, simpler than K8s, multi-workload | K8s ecosystem tools required |
|
||||
|
||||
**Startup default:** ECS + Fargate (zero cluster management).
|
||||
**Scale default:** EKS or GKE once team > 5 engineers or services > 10.
|
||||
|
||||
---
|
||||
|
||||
## 8. Load Balancer
|
||||
|
||||
| Technology | Layer | Best For |
|
||||
|------------|-------|----------|
|
||||
| **AWS ALB** | L7 (HTTP/HTTPS) | AWS apps, path-based routing, WebSocket, HTTP/2 |
|
||||
| **AWS NLB** | L4 (TCP/UDP) | Ultra-low latency, static IP, non-HTTP protocols |
|
||||
| **GCP GLB** | L7 global | GCP apps, global anycast, single IP worldwide |
|
||||
| **Nginx** | L4/L7 | Self-hosted, reverse proxy, flexible config |
|
||||
| **HAProxy** | L4/L7 | High performance self-hosted, advanced routing |
|
||||
| **Cloudflare** | L7 global + DDoS | DDoS protection + CDN + load balancing combined |
|
||||
| **Traefik** | L7 | Kubernetes-native, automatic SSL, service discovery |
|
||||
|
||||
---
|
||||
|
||||
## 9. Observability Stack
|
||||
|
||||
### Metrics
|
||||
| Tool | Best For |
|
||||
|------|----------|
|
||||
| **Prometheus + Grafana** | Self-hosted, open-source, Kubernetes-native |
|
||||
| **Datadog** | Managed, APM + infra + logs unified, expensive |
|
||||
| **CloudWatch** | AWS-native, zero setup, integrated with AWS services |
|
||||
| **New Relic** | APM-focused, good for application-level insights |
|
||||
|
||||
### Logging
|
||||
| Tool | Best For |
|
||||
|------|----------|
|
||||
| **ELK Stack** (Elasticsearch + Logstash + Kibana) | Self-hosted, powerful, high volume |
|
||||
| **Loki + Grafana** | Lightweight, Kubernetes-native, cheap |
|
||||
| **Splunk** | Enterprise, compliance, expensive |
|
||||
| **AWS CloudWatch Logs** | AWS-native, zero setup |
|
||||
| **Datadog Logs** | Unified with metrics, expensive |
|
||||
|
||||
### Distributed Tracing
|
||||
| Tool | Best For |
|
||||
|------|----------|
|
||||
| **Jaeger** | Open-source, Kubernetes-native, OpenTelemetry |
|
||||
| **Zipkin** | Simple, lightweight, good integrations |
|
||||
| **AWS X-Ray** | AWS-native, integrates with Lambda, ECS |
|
||||
| **Datadog APM** | Managed, unified with metrics and logs |
|
||||
| **Honeycomb** | High-cardinality event-based observability |
|
||||
|
||||
**Recommended open-source stack:** Prometheus + Grafana + Loki + Jaeger (all integrate via OpenTelemetry)
|
||||
**Recommended managed stack:** Datadog (expensive but unified) or Grafana Cloud
|
||||
|
||||
---
|
||||
|
||||
## 10. CDN
|
||||
|
||||
| Technology | Best For | Edge Locations |
|
||||
|------------|----------|----------------|
|
||||
| **Cloudflare** | DDoS protection + CDN + DNS, best free tier, edge workers | 300+ |
|
||||
| **AWS CloudFront** | AWS-native, deep S3 and API GW integration | 450+ |
|
||||
| **Akamai** | Enterprise, highest performance, expensive | 4000+ |
|
||||
| **Fastly** | Real-time purging, streaming, VCL customization | 90+ |
|
||||
| **Vercel Edge / Netlify** | Jamstack, frontend-first, zero config | 100+ |
|
||||
|
||||
**Default recommendation:** Cloudflare for most use cases (best value, DDoS included, free SSL, Workers for edge compute).
|
||||
|
||||
---
|
||||
|
||||
## Scale Benchmarks Quick Reference
|
||||
|
||||
| Technology | Write Throughput | Read Throughput | Notes |
|
||||
|------------|-----------------|----------------|-------|
|
||||
| PostgreSQL (single) | ~10K writes/s | ~50K reads/s | With connection pooling |
|
||||
| PostgreSQL (replicas) | ~10K writes/s | ~200K reads/s | 4 replicas |
|
||||
| MySQL (single) | ~15K writes/s | ~60K reads/s | |
|
||||
| Cassandra | ~1M writes/s | ~500K reads/s | 10-node cluster |
|
||||
| Redis | ~1M ops/s | ~1M ops/s | Single node in-memory |
|
||||
| Kafka | ~1M msgs/s | ~1M msgs/s | Per partition |
|
||||
| Elasticsearch | ~50K docs/s | ~10K queries/s | Per node |
|
||||
| MongoDB | ~50K writes/s | ~100K reads/s | Per replica set |
|
||||
|
||||
*All benchmarks are approximate and depend heavily on hardware, payload size, and query complexity.*
|
||||
|
||||
|
||||
## Limitations
|
||||
- This is a reference document and may not cover all edge cases. Always verify architectures before production.
|
||||
Reference in New Issue
Block a user