Resilient File Upload Architecture: S3/GCS Presigned URLs, Resumable Chunking & Progress
Stop routing multi-gigabyte uploads through your API application servers. Build direct-to-cloud presigned upload pipelines with pause, resume, checksum verification, and SHA-256 integrity checks.
The Architecture Anti-Pattern: Routing Uploads Through Node.js
In early development, teams frequently implement file uploads by posting `multipart/form-data` directly to an Express or Next.js server endpoint (`/api/upload`). The server accepts the incoming binary stream, writes it to a temporary directory, and then uploads it to cloud storage like AWS S3 or Cloudflare R2.
At scale, this architecture fails catastrophically. Uploading a 500MB video ties up an application server worker for minutes. Node.js memory spikes, server bandwidth saturates, and serverless runtimes (like AWS Lambda or Vercel) time out after 15 to 30 seconds with 413 Payload Too Large errors.
The enterprise standard is direct-to-object-storage uploads using presigned URLs and S3 Multipart Upload APIs. The application server only issues lightweight, cryptographically signed permission tokens; the client streams raw bytes directly to storage bucket endpoints.
This completely decouples application servers from bandwidth-heavy binary traffic.
Step-by-Step Presigned Upload Flow
The direct upload protocol operates in three distinct phases:
1. Request Permission: The client sends file metadata (filename, byte size, MIME type, SHA-256 hash) to `/api/uploads/presign`.
2. Issue URL: The server validates user authorization and storage quotas, generates a presigned `PUT` URL with a short 10-minute expiry, and returns the URL to the client.
3. Direct Streaming: The browser uploads the file directly to the object storage endpoint via `fetch()` or `XMLHttpRequest`, tracking upload progress in real-time.
export async function uploadFileDirectly(
file: File,
onProgress?: (percentage: number) => void
): Promise<{ fileUrl: string }> {
// 1. Request presigned upload URL from our API
const presignRes = await fetch("/api/uploads/presign", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
filename: file.name,
fileSize: file.size,
contentType: file.type,
}),
});
if (!presignRes.ok) throw new Error("Failed to authorize upload");
const { uploadUrl, publicUrl } = await presignRes.json();
// 2. Stream bytes directly to S3 / Cloudflare R2 with progress tracking
return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("PUT", uploadUrl, true);
xhr.setRequestHeader("Content-Type", file.type);
xhr.upload.onprogress = (event) => {
if (event.lengthComputable && onProgress) {
const percent = Math.round((event.loaded / event.total) * 100);
onProgress(percent);
}
};
xhr.onload = () => {
if (xhr.status >= 200 && xhr.status < 300) {
resolve({ fileUrl: publicUrl });
} else {
reject(new Error(`Upload failed with status: ${xhr.status}`));
}
};
xhr.onerror = () => reject(new Error("Network error during file upload"));
xhr.send(file);
});
}Resumable Uploads for Large Files (>100MB) with S3 Multipart
For files exceeding 100MB, a single network hiccup forces the user to restart the upload from 0%. Implement S3 Multipart Upload or the open-source TUS protocol.
Slice the file into 5MB chunks (`file.slice(start, end)`), upload chunks in parallel batches, and assemble the parts on the storage provider once all chunks successfully resolve.
If the user loses internet connection, the upload pauses and resumes from the last confirmed byte offset upon reconnecting.
const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB minimum S3 chunk size
export async function uploadMultipartFile(file: File, uploadId: string, key: string) {
const totalChunks = Math.ceil(file.size / CHUNK_SIZE);
const parts: Array<{ ETag: string; PartNumber: number }> = [];
for (let partNumber = 1; partNumber <= totalChunks; partNumber++) {
const start = (partNumber - 1) * CHUNK_SIZE;
const end = Math.min(start + CHUNK_SIZE, file.size);
const chunkBlob = file.slice(start, end);
// Fetch presigned URL for this specific chunk
const { presignedUrl } = await getPartPresignedUrl({ key, uploadId, partNumber });
const res = await fetch(presignedUrl, { method: "PUT", body: chunkBlob });
const etag = res.headers.get("ETag")!;
parts.push({ ETag: etag.replace(/"/g, ""), PartNumber: partNumber });
}
// Finalize multipart assembly
await completeMultipartUpload({ key, uploadId, parts });
}Client-Side Checksum Verification with Web Crypto
To guarantee data integrity across lossy mobile networks, calculate the file's SHA-256 hash client-side before upload begins. Pass the base64 checksum in the `x-amz-checksum-sha256` header.
The object storage service verifies the incoming byte stream against the signature and rejects any corrupted transfers immediately.
export async function calculateFileSha256(file: File): Promise<string> {
const buffer = await file.arrayBuffer();
const digestBuffer = await crypto.subtle.digest("SHA-256", buffer);
const hashArray = Array.from(new Uint8Array(digestBuffer));
return btoa(String.fromCharCode(...hashArray));
}Security Scans and Asynchronous Webhooks
Direct uploads introduce risk if users upload infected files or spoof file extensions. Mitigate this by creating an asynchronous post-upload pipeline.
Configure S3 Event Notifications to trigger a serverless function upon `s3:ObjectCreated:*`. The function runs ClamAV virus scans and verifies MIME magic bytes before updating the database to mark the file as 'Active'.
Isolate unverified files in a dedicated 'Quarantine' bucket with zero public read access until malware scanning pipelines emit an affirmative clean signal.
If a payload is flagged as malicious, immediately trigger an automated security alert webhook and permanently delete the quarantined object from cloud storage.
Graceful Network Retries with Exponential Jitter in Mobile Web
Mobile uploads frequently traverse cellular handoffs (switching from LTE to Wi-Fi). A transient network blip will abort an in-flight byte stream with a fetch TypeError.
Implement defensive retries with randomized exponential backoff ('full jitter'). If a chunk fails, wait `min(base * 2^attempt + random(0, 500), max)` before retrying up to 5 times before failing the upload task.
export async function retryWithBackoff<T>(
operation: () => Promise<T>,
maxRetries = 4,
baseDelayMs = 1000
): Promise<T> {
let attempt = 0;
while (true) {
try {
return await operation();
} catch (err) {
if (attempt >= maxRetries) throw err;
const jitter = Math.random() * 500;
const delay = Math.min(baseDelayMs * Math.pow(2, attempt) + jitter, 10000);
attempt++;
console.warn(`Upload chunk failed. Retrying attempt ${attempt} in ${Math.round(delay)}ms...`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}How do we prevent malicious users from uploading executable malware or spoofed file types?
Enforce strict Content-Type and file size headers inside the presigned URL signature. Configure asynchronous bucket webhooks that scan files with ClamAV before marking them as accessible.
Do presigned URLs require enabling CORS on the S3 bucket?
Yes. The S3/R2 bucket must have a CORS policy allowing PUT methods and headers (`Content-Type`) from your application domain.
How do we generate thumbnails for uploaded video or PDF files?
Trigger an asynchronous cloud worker (AWS Lambda or Cloudflare Worker) that executes FFmpeg to extract video keyframes or pdf-lib to render first-page PNG previews.