Start a shell in an Ubuntu pod (kubectl run -i --rm --tty shell --image=ubuntu:24.04 -- bash), install nslookup or dig, and attempt a DNS resolution of:
mongo
mongo-1.mongo
What do you observe?
The pods run mongod --replSet rs0, but the replica set is not configured yet. Initialize it from mongo-0, then add the two other members:
kubectl exec mongo-0 -c mongodb -- mongosh --quiet --eval '
rs.initiate({_id: "rs0", members: [{_id: 0, host: "mongo-0.mongo:27017"}]});
// Wait until mongo-0 is the primary: rs.add() only works on the primary
while (!db.hello().isWritablePrimary) { sleep(1000); }
rs.add("mongo-1.mongo:27017");
rs.add("mongo-2.mongo:27017");'
The commands are run by mongosh --eval, not typed in an interactive mongosh shell: pasted after kubectl exec -it mongo-0 -- mongosh, they would be run by bash once mongosh exits.
MongoServerError: no replset config has been received: rs.initiate() was not run on this replica set, run the initialization above.
A member stays in STARTUP or is (not reachable/healthy): mongo-0 cannot resolve or reach it. Check the headless Service (kubectl get svc mongo, CLUSTER-IP must be None), the DNS name from another pod (mongo-1.mongo), and the logs of the member: kubectl logs mongo-1.
rs.add() fails with NotWritablePrimary: it was run before mongo-0 became primary, or on another member. Run it again on the primary.
Try scaling the StatefulSet using kubectl scale. What happens?
Automated Installation
What are the areas for improvement for this StatefulSet?
The configuration script in the configmap is not highly robust and has some side effects.
If one of the mongo pods is not present in the mongo replicaset, a simple solution is to delete the affected pod so that it is automatically recreated and the configuration script runs again.
Data Loading
curl -O https://raw.githubusercontent.com/mongodb/docs-assets/primer-dataset/primer-dataset.json
# Only the primary accepts writes: connect to the replica set, not to a podRS="mongodb://mongo-0.mongo:27017/test?replicaSet=rs0"kubectl exec -i mongo-0 -c mongodb -- mongoimport --uri "$RS" --collection restaurants --drop < primer-dataset.json
Display the update strategy of the StatefulSet. Then update onlymongo-2 to docker.io/library/mongo:7.0.45, and check the MongoDB version of each member.
Solution
kubectl get statefulset mongo -o jsonpath='{.spec.updateStrategy}{"\n"}'# {"rollingUpdate":{"maxUnavailable":1,"partition":0},"type":"RollingUpdate"}# Only the pods with an ordinal >= partition are updatedkubectl patch statefulset mongo -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'kubectl set image statefulset/mongo mongodb=docker.io/library/mongo:7.0.45 init-mongo=docker.io/library/mongo:7.0.45
kubectl rollout status statefulset/mongo
# partitioned roll out complete: 1 new pods have been updated...kubectl get pods -l app=mongo -o custom-columns='NAME:.metadata.name,IMAGE:.spec.containers[0].image,REV:.metadata.labels.controller-revision-hash'# NAME IMAGE REV# mongo-0 docker.io/library/mongo:7.0.43 mongo-844478b6d8# mongo-1 docker.io/library/mongo:7.0.43 mongo-844478b6d8# mongo-2 docker.io/library/mongo:7.0.45 mongo-74bcf55988for i in 01 2; do echo "mongo-$i$(kubectl exec mongo-$i -c mongodb -- mongosh --quiet --eval 'db.version()')"; done
The controller-revision-hash label points to a ControllerRevision: the StatefulSet keeps its pod templates in these objects (kubectl get controllerrevisions), like a Deployment keeps its ReplicaSets.
partition is the canary of a StatefulSet: the pods with an ordinal below the partition keep the old revision, even if they are deleted and recreated.
Finish the update
Update the other members. In which order are the pods updated? Which member is the primary at the end? Are the data and the PVCs still the same?
The pods are updated one at a time, in reverse ordinal order (mongo-1, then mongo-0), and each pod must be Ready before the next one is deleted.
When the pod of the primary is deleted, the replica set elects a new primary among the secondaries: the primary is not mongo-0 any more. This is why the clients must connect to the replica set (?replicaSet=rs0) and not to a pod.
Each new pod gets the PVC of its ordinal back (database-mongo-<i>, same volume): the data is not copied, it stays in the volume.
Rollback
Roll back the StatefulSet to the previous version, and check the versions and the data.
Solution
kubectl rollout history statefulset/mongo
kubectl rollout undo statefulset/mongo
kubectl rollout status statefulset/mongo
for i in 01 2; do echo "mongo-$i$(kubectl exec mongo-$i -c mongodb -- mongosh --quiet --eval 'db.version()')"; done# mongo-0 7.0.43# mongo-1 7.0.43# mongo-2 7.0.43
The rollback is a rolling update to the previous ControllerRevision. Kubernetes only rolls back the pod template, never the data: a patch downgrade (7.0.45 to 7.0.43) is safe, but a major downgrade (8.0 to 7.0) needs to lower the featureCompatibilityVersion of MongoDB before. This is the job of a database operator.
Going further
updateStrategy.type: OnDelete: the StatefulSet only updates a pod when you delete it, which gives full control on the order (for example: the secondaries first, then rs.stepDown() on the primary).
A database operator (CloudNativePG for PostgreSQL, MongoDB Community Operator) knows the topology: it updates the replicas first and switches the primary over in a controlled way.