I'm facing an issue with my MongoDB setup on Kubernetes in a replica set configuration. I have 2 pods and an arbiter, and my Java SpringBoot application is interacting with MongoDB using Reactive Mongo Repository. The connection URL I'm using is:
mongodb://{{mongoURL}}/?directConnection=true&serverSelectionTimeoutMS=2000&readPreference=primary
However, I keep getting a MongoNotPrimaryException with the error message: "Command failed with error 10107 (NotWritablePrimary): 'not primary'."
I'd like to understand and resolve this problem. Any guidance or suggestions would be greatly appreciated. Thank you!
What I've Tried: I've already attempted to diagnose and address this issue. Here's what I've tried and the results:
Checked MongoDB Configuration: I've verified the MongoDB configuration for the replica set and the arbiter. Everything seems to be correctly set up in the Kubernetes cluster.
Application Code: I've examined my Java SpringBoot application's code that interacts with MongoDB using Reactive Mongo Repository.
Actual Results: Despite these checks, I continue to encounter the MongoNotPrimaryException with the "not primary" error message when my application tries to write to MongoDB. It's unexpected as I'm pointing to the primary in the connection string.
Any insights or suggestions on how to resolve this issue would be greatly appreciated. Thank you!
These are logs in the mongo pod
{"t":{"$date":"2023-11-09T09:05:48.481+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:54236","uuid":{"uuid":{"$uuid":"87592666-c1a1-4d21-9f2d-9e992d8ad268"}},"connectionId":40918,"connectionCount":32}}
{"t":{"$date":"2023-11-09T09:05:48.481+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:54244","uuid":{"uuid":{"$uuid":"795de4a0-3d8b-4688-a046-e6e64330752f"}},"connectionId":40919,"connectionCount":33}}
{"t":{"$date":"2023-11-09T09:05:48.482+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn40918","msg":"client metadata","attr":{"remote":"127.0.0.1:54236","client":"conn40918","doc":{"application":{"name":"mongosh 1.10.6"},"driver":{"name":"nodejs|mongosh","version":"5.7.0|1.10.6"},"platform":"Node.js v16.20.2, LE","os":{"name":"linux","architecture":"x64","version":"5.15.0-1041-azure","type":"Linux"}}}}
{"t":{"$date":"2023-11-09T09:05:48.482+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn40919","msg":"client metadata","attr":{"remote":"127.0.0.1:54244","client":"conn40919","doc":{"application":{"name":"mongosh 1.10.6"},"driver":{"name":"nodejs|mongosh","version":"5.7.0|1.10.6"},"platform":"Node.js v16.20.2, LE","os":{"name":"linux","architecture":"x64","version":"5.15.0-1041-azure","type":"Linux"}}}}
{"t":{"$date":"2023-11-09T09:05:48.485+00:00"},"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn40918","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":3}}
{"t":{"$date":"2023-11-09T09:05:48.485+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:54252","uuid":{"uuid":{"$uuid":"f1419771-32a0-4760-9e68-6736d27f7a71"}},"connectionId":40920,"connectionCount":34}}
{"t":{"$date":"2023-11-09T09:05:48.489+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn40920","msg":"client metadata","attr":{"remote":"127.0.0.1:54252","client":"conn40920","doc":{"application":{"name":"mongosh 1.10.6"},"driver":{"name":"nodejs|mongosh","version":"5.7.0|1.10.6"},"platform":"Node.js v16.20.2, LE","os":{"name":"linux","architecture":"x64","version":"5.15.0-1041-azure","type":"Linux"}}}}
{"t":{"$date":"2023-11-09T09:05:48.499+00:00"},"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn40920","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":10}}
{"t":{"$date":"2023-11-09T09:05:49.503+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn40917","msg":"Interrupted operation as its client disconnected","attr":{"opId":1375581}}
{"t":{"$date":"2023-11-09T09:05:49.503+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn40920","msg":"Connection ended","attr":{"remote":"127.0.0.1:54252","uuid":{"uuid":{"$uuid":"f1419771-32a0-4760-9e68-6736d27f7a71"}},"connectionId":40920,"connectionCount":33}}
{"t":{"$date":"2023-11-09T09:05:49.503+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn40918","msg":"Connection ended","attr":{"remote":"127.0.0.1:54236","uuid":{"uuid":{"$uuid":"87592666-c1a1-4d21-9f2d-9e992d8ad268"}},"connectionId":40918,"connectionCount":32}}
{"t":{"$date":"2023-11-09T09:05:49.503+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn40919","msg":"Connection ended","attr":{"remote":"127.0.0.1:54244","uuid":{"uuid":{"$uuid":"795de4a0-3d8b-4688-a046-e6e64330752f"}},"connectionId":40919,"connectionCount":31}}
{"t":{"$date":"2023-11-09T09:05:49.503+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn40917","msg":"Connection ended","attr":{"remote":"127.0.0.1:54232","uuid":{"uuid":{"$uuid":"a571fb30-30ec-4a87-83e8-0b0ed0817385"}},"connectionId":40917,"connectionCount":30}}