I’ll be upfront about something most tutorials on this topic gloss over: Cassandra isn’t natively an object store, and there’s no mainstream open-source project that uses Cassandra as its primary backend for full S3 semantics the way MinIO uses erasure-coded drives. What people actually mean — and what I’ve built in practice — is a metadata + chunked-blob storage layer on top of Cassandra, fronted by an S3-compatible API gateway. That’s a legitimate and genuinely useful pattern (it’s close to what early versions of things like OpenStack Swift-on-Cassandra experiments and several homegrown CDN backends have done), so that’s exactly what I’ll walk through: a real, working architecture, not a fictional “Cassandra has an S3 mode” shortcut.
The Architecture
+-------------------+
S3 clients ---> | Gateway (API) |
(aws-cli, SDKs) | S3-compatible |
+-------------------+
| |
+-------------+ +----------------+
| Cassandra | | Object chunks |
| (metadata + | | (stored as blobs|
| small obj) | | in Cassandra |
+-------------+ | blob tables) |
+----------------+
All running as StatefulSets/Deployments in Kubernetes
We’ll deploy:
- A Cassandra StatefulSet (3 nodes) for metadata and object-chunk storage.
- A lightweight S3 gateway service that translates S3 REST calls into Cassandra reads/writes (implemented conceptually — I’ll show the CQL schema and a minimal Node.js gateway you can extend).
- Kubernetes networking (headless Service for Cassandra gossip, ClusterIP for the gateway, Ingress for external S3 access).
If you want a production-grade, fully S3-compliant system without building the gateway yourself, the honest recommendation is MinIO or Ceph RGW — I’ll note that in best practices below. But if your goal is specifically “S3-compatible API in front of Cassandra-backed storage” (common in multi-region, tunable-consistency use cases), here’s how it’s built.
Step 1: Namespace and Storage Class
kubectl create namespace object-store
Confirm you have a StorageClass that supports dynamic provisioning (cloud environments usually do by default):
kubectl get storageclass
Expected output (varies by provider):
NAME PROVISIONER RECLAIMPOLICY
standard (default) kubernetes.io/gce-pd Delete
Step 2: Headless Service for Cassandra Gossip
Cassandra nodes discover each other via gossip, which requires stable network identities — a headless Service (clusterIP: None) gives each pod a predictable DNS name.
# cassandra-headless-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: cassandra
namespace: object-store
labels:
app: cassandra
spec:
clusterIP: None
ports:
- port: 7000
name: intra-node
- port: 7001
name: tls-intra-node
- port: 7199
name: jmx
- port: 9042
name: cql
selector:
app: cassandra
kubectl apply -f cassandra-headless-svc.yaml
Step 3: Cassandra StatefulSet
# cassandra-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
namespace: object-store
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
image: cassandra:5.0
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
env:
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.object-store.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "ObjectStoreCluster"
- name: CASSANDRA_DC
value: "dc1"
- name: CASSANDRA_RACK
value: "rack1"
- name: CASSANDRA_ENDPOINT_SNITCH
value: "GossipingPropertyFileSnitch"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command: ["/bin/bash", "-c", "nodetool status | grep -E \"^UN\\s+${POD_IP}\""]
initialDelaySeconds: 60
periodSeconds: 15
volumeMounts:
- name: cassandra-data
mountPath: /var/lib/cassandra
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
kubectl apply -f cassandra-statefulset.yaml
kubectl get pods -n object-store -w
Expected output as nodes come up one by one (StatefulSets deploy sequentially):
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 2m
cassandra-1 1/1 Running 0 90s
cassandra-2 1/1 Running 0 40s
Verify cluster formation:
kubectl exec -it cassandra-0 -n object-store -- nodetool status
Expected output:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.244.1.5 115 KiB 16 33.4% ... rack1
UN 10.244.2.7 102 KiB 16 33.3% ... rack1
UN 10.244.3.9 98 KiB 16 33.3% ... rack1
UN means Up/Normal — all three nodes are healthy and joined.
Step 4: Schema for Object Storage
Connect via cqlsh:
kubectl exec -it cassandra-0 -n object-store -- cqlsh
CREATE KEYSPACE object_store
WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3};
USE object_store;
CREATE TABLE buckets (
bucket_name text PRIMARY KEY,
created_at timestamp,
owner text
);
CREATE TABLE objects (
bucket_name text,
object_key text,
version_id timeuuid,
size bigint,
etag text,
content_type text,
created_at timestamp,
chunk_count int,
PRIMARY KEY ((bucket_name), object_key, version_id)
) WITH CLUSTERING ORDER BY (object_key ASC, version_id DESC);
CREATE TABLE object_chunks (
bucket_name text,
object_key text,
version_id timeuuid,
chunk_index int,
data blob,
PRIMARY KEY ((bucket_name, object_key, version_id), chunk_index)
);
Design notes: objects are split into chunks (e.g., 1MB each) and stored as blob rows rather than one giant row, because Cassandra performs poorly with very large single cells/partitions. The objects table holds metadata; object_chunks holds the payload, partitioned so retrieval streams chunk-by-chunk.
Step 5: Minimal S3-Compatible Gateway (Deployment)
A trimmed example gateway (Node.js + Express + cassandra-driver) exposing PUT/GET for objects, mapped to basic S3 REST semantics:
// gateway/index.js
const express = require("express");
const cassandra = require("cassandra-driver");
const { v1: uuidv1 } = require("uuid");
const client = new cassandra.Client({
contactPoints: ["cassandra-0.cassandra.object-store.svc.cluster.local"],
localDataCenter: "dc1",
keyspace: "object_store",
});
const app = express();
const CHUNK_SIZE = 1024 * 1024;
app.put("/:bucket/:key", express.raw({ type: "*/*", limit: "5gb" }), async (req, res) => {
const { bucket, key } = req.params;
const versionId = uuidv1();
const data = req.body;
const chunks = Math.ceil(data.length / CHUNK_SIZE);
for (let i = 0; i < chunks; i++) {
const chunk = data.slice(i * CHUNK_SIZE, (i + 1) * CHUNK_SIZE);
await client.execute(
"INSERT INTO object_chunks (bucket_name, object_key, version_id, chunk_index, data) VALUES (?, ?, ?, ?, ?)",
[bucket, key, versionId, i, chunk],
{ prepare: true }
);
}
await client.execute(
"INSERT INTO objects (bucket_name, object_key, version_id, size, chunk_count, created_at) VALUES (?, ?, ?, ?, ?, toTimestamp(now()))",
[bucket, key, versionId, data.length, chunks],
{ prepare: true }
);
res.set("ETag", versionId.toString()).status(200).end();
});
app.get("/:bucket/:key", async (req, res) => {
const { bucket, key } = req.params;
const meta = await client.execute(
"SELECT * FROM objects WHERE bucket_name = ? AND object_key = ? LIMIT 1",
[bucket, key],
{ prepare: true }
);
if (meta.rowLength === 0) return res.status(404).end();
const row = meta.first();
for (let i = 0; i < row.chunk_count; i++) {
const chunkRes = await client.execute(
"SELECT data FROM object_chunks WHERE bucket_name = ? AND object_key = ? AND version_id = ? AND chunk_index = ?",
[bucket, key, row.version_id, i],
{ prepare: true }
);
res.write(chunkRes.first().data);
}
res.end();
});
app.listen(8080, () => console.log("S3 gateway listening on 8080"));
Dockerfile:
FROM node:20-slim
WORKDIR /app
COPY package.json .
RUN npm install --production
COPY index.js .
EXPOSE 8080
CMD ["node", "index.js"]
Build and push:
docker build -t your-registry/s3-cassandra-gateway:1.0 ./gateway
docker push your-registry/s3-cassandra-gateway:1.0
Deployment manifest:
# gateway-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: s3-gateway
namespace: object-store
spec:
replicas: 2
selector:
matchLabels:
app: s3-gateway
template:
metadata:
labels:
app: s3-gateway
spec:
containers:
- name: s3-gateway
image: your-registry/s3-cassandra-gateway:1.0
ports:
- containerPort: 8080
resources:
requests:
cpu: "250m"
memory: 256Mi
limits:
cpu: "500m"
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: s3-gateway
namespace: object-store
spec:
selector:
app: s3-gateway
ports:
- port: 80
targetPort: 8080
type: ClusterIP
kubectl apply -f gateway-deployment.yaml
Expose externally with an Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: s3-gateway-ingress
namespace: object-store
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "5g"
spec:
ingressClassName: nginx
rules:
- host: s3.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: s3-gateway
port:
number: 80
Step 6: Test With a Standard S3 Client
kubectl port-forward svc/s3-gateway -n object-store 8080:80
curl -X PUT --data-binary @localfile.txt http://localhost:8080/my-bucket/localfile.txt
curl http://localhost:8080/my-bucket/localfile.txt -o downloaded.txt
diff localfile.txt downloaded.txt
Expected: diff returns nothing, confirming round-trip integrity.
Internals: Why This Works (and Where It Strains)
Cassandra’s NetworkTopologyStrategy with RF=3 gives you the same durability model S3 relies on internally — every chunk is written to 3 nodes across the ring before being acknowledged (tunable via consistency level, e.g. LOCAL_QUORUM). Partitioning chunks by (bucket, key, version_id) spreads large objects across the cluster instead of hammering one node with a giant blob.
Where this genuinely strains compared to purpose-built object stores like MinIO or Ceph: Cassandra wasn’t designed for large binary blobs, and very large objects (multi-GB) will pressure compaction and heap usage. In practice, teams doing this successfully cap object size, tune compaction throughput, and often route only metadata and small objects through Cassandra while large blobs go to a separate blob store — a hybrid model.
Security
- Enable Cassandra internode and client-to-node TLS via
cassandra.yaml(server_encryption_options,client_encryption_options) — don’t run gossip traffic unencrypted across namespaces. - Use Kubernetes
NetworkPolicyto restrict which pods can reach Cassandra’s CQL port (9042):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: cassandra-allow-gateway-only
namespace: object-store
spec:
podSelector:
matchLabels:
app: cassandra
ingress:
- from:
- podSelector:
matchLabels:
app: s3-gateway
ports:
- port: 9042
- Store the gateway’s Cassandra credentials in a Kubernetes
Secret, not plaintext env vars in the manifest. - Add authentication (SigV4 or a simpler bearer token scheme) to the gateway before exposing it publicly — the example above has none, intentionally, to keep the code focused on the storage mechanics.
Monitoring and Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
nodetool status shows DN (Down/Normal) | Pod crashed or network partition | kubectl describe pod cassandra-N, check node resource pressure |
| Gateway 504s on large uploads | Ingress body-size limit too low | Raise proxy-body-size annotation |
| High latency on reads | Chunk reads happening serially | Parallelize chunk fetches in the gateway with Promise.all |
| Compaction pressure / high disk I/O | Storing very large blobs directly | Cap object size or route large files to dedicated blob storage |
Deploy cassandra-exporter or JMX-to-Prometheus exporters as a sidecar for cluster-level metrics (pending compactions, read/write latency, heap usage), and scrape gateway request metrics via a /metrics endpoint for Prometheus + Grafana dashboards.
Summary
Cassandra doesn’t give you S3 out of the box, but its distributed, replicated, partition-tolerant design makes it a solid metadata and chunk-storage backend when you build an S3-compatible API layer on top — especially in Kubernetes, where StatefulSets and headless Services handle the hard parts of stable node identity and gossip discovery for you. For most production object-storage needs, evaluate MinIO or Ceph RGW first; reach for this Cassandra-backed pattern when you specifically need Cassandra’s tunable consistency and multi-datacenter replication model married to an S3-shaped API.
References
- Apache Cassandra Documentation
- Kubernetes StatefulSets
- Kubernetes Headless Services
- Kubernetes NetworkPolicy
- Amazon S3 API Reference
- MinIO Documentation (for a purpose-built alternative)
