FW: CDCR performance issues

-----Original Message-----
From: Tom Peters [mailto:tpeters@synacor.com]
Sent: 08 March 2018 04:49
To: solr-user@lucene.apache.org
Subject: CDCR performance issues

I'm having issues with the target collection staying up-to-date with
indexing from the source collection using CDCR.

This is what I'm getting back in terms of OPS:

curl -s 'solr2-a:8080/solr/mycollection/cdcr?action=OPS' | jq .
{
"responseHeader": {
"status": 0,
"QTime": 0
},
"operationsPerSecond": [
"zook01,zook02,zook03/solr",
[
"mycollection",
[
"all",
49.10140553500938,
"adds",
10.27612635309587,
"deletes",
38.82527896994054
]
]
]
}

The source and target collections are in separate data centers.

Doing a network test between the leader node in the source data center and
the ZooKeeper nodes in the target data center show decent enough network
performance: ~181 Mbit/s

I've tried playing around with the "batchSize" value (128, 512, 728, 1000,
2000, 2500) and they've haven't made much of a difference.

Any suggestions on potential settings to tune to improve the performance?

Thanks

--

Here's some relevant log lines from the source data center's leader:

2018-03-07 23:16:11.984 INFO
(cdcr-replicator-207-thread-3-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
2018-03-07 23:16:23.062 INFO
(cdcr-replicator-207-thread-4-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 510 updates to target mycollection
2018-03-07 23:16:32.063 INFO
(cdcr-replicator-207-thread-5-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
2018-03-07 23:16:36.209 INFO
(cdcr-replicator-207-thread-1-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection
2018-03-07 23:16:42.091 INFO
(cdcr-replicator-207-thread-2-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection
2018-03-07 23:16:46.790 INFO
(cdcr-replicator-207-thread-3-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 511 updates to target mycollection
2018-03-07 23:16:50.004 INFO
(cdcr-replicator-207-thread-4-processing-n:solr2-a:8080_solr
x:mycollection_shard1_replica_n6 s:shard1 c:mycollection r:core_node9)
[c:mycollection s:shard1 r:core_node9 x:mycollection_shard1_replica_n6]
o.a.s.h.CdcrReplicator Forwarded 512 updates to target mycollection


And what the log looks like in the target:

2018-03-07 23:18:46.475 INFO (qtp1595212853-26) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067896487950&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.500 INFO (qtp1595212853-25) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067896487951&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.525 INFO (qtp1595212853-24) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536512&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.550 INFO (qtp1595212853-3793) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536513&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.575 INFO (qtp1595212853-30) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536514&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.600 INFO (qtp1595212853-26) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536515&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.625 INFO (qtp1595212853-25) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536516&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.651 INFO (qtp1595212853-24) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536517&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.676 INFO (qtp1595212853-3793) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536518&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0
2018-03-07 23:18:46.701 INFO (qtp1595212853-30) [c:mycollection
s:shard1 r:core_node2 x:mycollection_shard1_replica_n1] o.a.s.c.S.Request
[mycollection_shard1_replica_n1] webapp=/solr path=/update
params={_stateVer_=mycollection:30&_version_=-1594317067897536519&cdcr.updat
e=&wt=javabin&version=2} status=0 QTime=0



This message and any attachment may contain information that is confidential
and/or proprietary. Any use, disclosure, copying, storing, or distribution
of this e-mail or any attached file by anyone other than the intended
recipient is strictly prohibited. If you have received this message in
error, please notify the sender by reply email and delete the message and
any attachments. Thank you.