High-Performance URL Shortener
Backend Engineering · Go · PostgreSQL · Redis · Docker
A production-style URL shortening service with clean API boundaries, persistent storage, Redis-backed caching and rate limiting, asynchronous click analytics and automated testing — designed and verified stage by stage.
Measured in the repository's own documentation (docs/PERFORMANCE.md, docs/SECURITY.md).
Overview
The URL shortener accepts a long URL and returns a short code that redirects to the original destination. Around that core flow it adds production concerns: persistent storage, caching for hot reads, per-client rate limiting, and an asynchronous pipeline for click analytics so metrics never slow down the redirect path.
Problem
A naive shortener is easy — but a useful one needs to answer: how do you serve millions of redirects with low latency, avoid duplicate short codes, keep analytics from slowing the hot path, and stay up when a cache layer fails? This project treats those as engineering problems rather than curiosities, and verifies each answer with tests and load tests.
Architecture
A read hits the Redis cache first; on a miss it falls back to PostgreSQL and warms the cache. Every redirect pushes a non-blocking analytics event into a Redis Stream, consumed asynchronously by a worker.
Technology Stack
| Layer | Technology |
|---|---|
| Language | Go |
| Database | PostgreSQL (system of record, migrations) |
| Cache & queue | Redis (cache-aside, rate limiting, Streams) |
| Proxy | Nginx (reverse proxy to 3 API replicas) |
| Deployment | Docker, Docker Compose |
| Observability | Prometheus metrics, Grafana dashboard, structured logs |
| Testing & load | Go unit/integration tests, k6 |
| CI | GitHub Actions, govulncheck, Trivy |
Key Engineering Decisions
- Layered architecture — handlers, service and repository layers keep HTTP, business logic and data access separate and independently testable.
- Cache-aside with fail-open reads — a Redis outage degrades latency but never breaks redirects; verified explicitly.
- Base62 short codes — compact, URL-safe identifiers with custom alias and expiry support.
- Idempotent URL creation — retries don't produce duplicate short codes.
- Analytics off the hot path — redirects enqueue events to Redis Streams; a consumer-group worker aggregates into PostgreSQL.
- Stateless replicas — identical API instances behind Nginx make scaling and failover straightforward.
API Design
Database Design
PostgreSQL is the source of truth. The two main data stores:
- urls — long URL, unique short code, optional custom alias, expiry, created metadata
- analytics_daily — aggregate click counts per URL / day / referrer, with idempotency dedupe
Schema is managed with migrations run as part of setup and CI. Raw click events flow through Redis Streams before being aggregated, keeping high-frequency writes away from the redirect path's database queries.
Authentication & Security
- Input validation on creation (URL, alias, expiry).
- Per-IP rate limiting (60 req/min) on URL creation via Redis.
- Dependency scanning —
govulncheckin CI. - Container scanning — Trivy image scan failing CI on HIGH/CRITICAL; 0 findings in the documented run.
No user authentication layer — this project focuses on the redirect service itself, with rate limiting as the abuse control.
Testing
21 unit and integration tests cover the codec (Base62), service logic, repository, cache behaviour, analytics consumer and database migrations. Integration tests run against real PostgreSQL and Redis containers in CI.
Beyond unit tests, the repo documents load and failure tests:
- k6 load test — 2,500 req/s sustained for 60s with 0 failed requests on 3 replicas
- Server-side latency p50 / p95 / p99 of 2.50 / 4.75 / 4.95 ms
- Redis stopped — fail-open keeps requests served at ~434 ms/request
- Replica failure under load — 10/10 requests still redirected correctly
CI/CD
A GitHub Actions workflow runs on push to main and on pull requests:
CI builds API and worker binaries and verifies the containers are clean to ship — there is no live deployment for this project.
Challenges & Solutions
- Keeping analytics off the redirect path — solved with Redis Streams buffering and a consumer-group worker that aggregates asynchronously.
- Cache consistency — cache-aside with a 60-second TTL, invalidation on delete, and fail-open reads so a cold or failing cache never breaks redirects.
- Duplicate creation on retries — idempotency at the service layer prevents identical short codes being created more than once.
- Scaling out — stateless API replicas behind Nginx let load tests verify horizontal scaling and failover behaviour rather than assuming it.
What I Learned
Building under testable disciplines: every feature was implemented with infrastructure that proved it — load tests for latency claims, failure tests for resilience claims, and CI gates for quality. I learned how much design discipline it takes to keep a read-heavy hot path fast, and how to verify "it works" with evidence rather than assumption.