Step 1: Requirements and estimates
Functional requirements:
- Upload photos, videos and documents, from a few KB up to a few GB.
- Show progress, and resume after a lost connection or after the app was closed.
- Process files after upload: thumbnails for images, previews for videos, virus scanning for documents.
- Files are private: only people allowed to see the ticket or chat can download them.
Non-functional requirements:
- API servers never handle the file bytes.
- Nothing is used (shown to others, attached to a message) until it’s been checked.
- An upload interrupted at 92% continues from about 92%, not from zero.
Clarify: file size distribution (it decides single-request vs multipart), allowed types, who can read files (private with signed downloads vs public via CDN), retention and deletion (GDPR, user deletion), processing needs (transcoding is a large subsystem of its own), deduplication, and quotas per user. The design follows from three principles: direct-to-storage data path with the API only issuing permissions and recording metadata; untrusted until verified (the client’s declared type, size and checksum are claims); and resumability at the chunk level, with the client owning retry and the server owning state.
Daily active users: 20 million
Uploads: ~0.5 per user per day → 10 million / day ≈ 120 / s (peak ~500 / s)
Average size: ~3 MB (photos), long tail of videos up to 2 GB
Ingest: 10 million × 3 MB ≈ 30 TB / day ≈ 350 MB/s average
Storage after a year: ~11 PB before lifecycle rules (thumbnails and previews add ~10%)
If bytes went through the API: 350 MB/s of inbound traffic, plus the same again outbound to storage,
and thousands of slow requests holding connections for minutes- Bandwidth, not requests, is the problem: 120 uploads per second is little; 350 MB per second through your API servers is a lot.
- Mobile uploads are slow: a 40 MB video on a weak connection can take several minutes, so anything holding a server connection that long scales badly.
- Sizes from KB to GB; resumable across network loss and app restarts; private by default.
- ~10M uploads/day, ~30 TB/day ingest: bandwidth-dominated, slow long-lived client connections.
- Principles: direct-to-storage data path, client claims untrusted until verified, chunk-level resumability.
- Probe retention/deletion, quotas, dedup, and whether transcoding is in scope.
Check this part
Why shouldn't file bytes pass through the API servers?
Which client-provided values can the server trust without checking?