SolrCloud 8.1.1 CDCR Replication not working: failed due to (409) version conflict actual=-1

Question

Trying to setup Uni-directional CDCR with Solr 8.1.1,

after syncing data directories from source to target replicas (1 to 1) in source logs I see:

2020-03-09 11:04:28.873 INFO  (cdcr-replicator-17-thread-4) [   ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target search1
2020-03-09 11:04:30.872 INFO  (cdcr-replicator-17-thread-5) [   ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target search1
2020-03-09 11:04:32.873 INFO  (cdcr-replicator-17-thread-6) [   ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target search1
2020-03-09 11:04:34.873 INFO  (cdcr-replicator-17-thread-7) [   ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target search1
2020-03-09 11:04:36.873 INFO  (cdcr-replicator-17-thread-8) [   ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target search1

I started the CDCR on source collection, and when I index new data into source collection, getting following error:

2020-03-09 11:14:22.933 ERROR (cdcr-replicator-17-thread-5) [   ] o.a.s.c.s.i.BaseCloudSolrClient Request to collection [search1] failed due to (409) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.11.33.159:8983/solr/search1_shard1_replica_n1: version conflict for THREE_INDEXING_01 expected=1660684743870513152 actual=-1, retry=0 commError=false errorCode=409
2020-03-09 11:14:22.933 INFO  (cdcr-replicator-17-thread-5) [   ] o.a.s.c.s.i.BaseCloudSolrClient request was not communication error it seems
2020-03-09 11:14:22.933 WARN  (cdcr-replicator-17-thread-5) [   ] o.a.s.h.CdcrReplicator Failed to forward update request to target: search1 => org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://10.11.33.159:8983/solr/search1_shard1_replica_n1: version conflict for THREE_INDEXING_01 expected=1660684743870513152 actual=-1
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://10.11.33.159:8983/solr/search1_shard1_replica_n1: version conflict for THREE_INDEXING_01 expected=1660684743870513152 actual=-1
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125) ~[?:?]
        at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46) ~[?:?]
        at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:489) ~[?:?]
        at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:977) ~[?:?]
        at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:837) ~[?:?]
        at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:769) ~[?:?]
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) ~[?:?]
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) ~[?:?]
        at org.apache.solr.handler.CdcrReplicator.sendRequest(CdcrReplicator.java:140) ~[?:?]
        at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:120) ~[?:?]
        at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81) ~[?:?]
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.11.33.159:8983/solr/search1_shard1_replica_n1: version conflict for THREE_INDEXING_01 expected=1660684743870513152 actual=-1
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:649) ~[?:?]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[?:?]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[?:?]
        at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) ~[?:?]
        at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) ~[?:?]
        at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:465) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
        ... 4 more

Answer 1

If there is a version conflict (HTTP error code 409), then you get the HTTP error code 409.

By default, Solr’s Schema includes a version field, and this field is automatically added to each new document. This feature works by requiring a version field on all documents in the index, and comparing that to a version specified as part of the update command.

$ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/techproducts/update?_version_=1632740120218042368&versions=true&commit=true&omitHeader=true' --data-binary '
[{ "id" : "aaa",
   "foo_s" : "update attempt with correct existing version" }]'

an update with a value for version that matches the value in the index, and it succeeds. Because we included versions=true to the update request, the response includes a different value for the version field.

If an update with a value for version embedded in the document itself. The request fails because you have specified the wrong version. Below would be the error for it.

{
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"version conflict for aaa expected=100 actual=1632740462042284032",
    "code":409
    }
}

The -1 here is meant that Solr is not able to find a document with that version. I would suggest you to try sending one of the document to solr yourself by hand on the Solr admin UI. Select your core/collection name, then click the Documents link(on the solr admin page) and you'll be at the page where you could send the document for update to solr.

Please refer the solr documentation for more details. Solr Documentation