FW: Copying a SolrCloud collection to other hosts

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: 15 March 2018 20:39
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Copying a SolrCloud collection to other hosts

yeah, it's on a core-by-core basis. Which also makes getting it propagated
to all replicas something you have to be sure happens...

Glad it's working for you!
Erick

On Thu, Mar 15, 2018 at 1:54 AM, Patrick Schemitz <ps@solute.de> wrote:
> Hi Erick,
>
> thanks a lot, that solved our problem nicely.
>
> (It took us a try or two to notice that this will not copy the entire
> collection but only the shard on the source instance, and we need to
> do this for all instances explicitly. But hey, we had to do the same
> for the old approch of scp'ing the data directories.)
>
> Ciao, Patrick
>
> On Tue, Mar 06, 2018 at 07:18:15AM -0800, Erick Erickson wrote:
>> this is part of the "different replica types" capability, there are
>> NRT (the only type available prior to 7x), PULL and TLOG which would
>> have different names. I don't know of any way to switch it off.
>>
>> As far as moving the data, here's a little known trick: Use the
>> replication API to issue a fetchindexk, see:
>> https://lucene.apache.org/solr/guide/6_6/index-replication.html As
>> long as the target cluster can "see" the source cluster via http,
>> this should work. This is entirely outside SolrCloud and ZooKeeper is
>> not involved. This would even work with, say, one side being
>> stand-alone and the other being SolrCloud (not that you want to do
>> that, just illustrating it's not part of SolrCloud)...
>>
>> So you'd specify something like:
>> http://target_node:port/solr/core_name/replication?command=fetchindex
>> &masterUrl=http://source_node:port/solr/core_name
>>
>> "core_name" in these cases is what appears in the "cores" dropdown on
>> the admin UI page. You do not have to shut Solr down at all on either
>> end to use this, although last I knew the target node would not serve
>> queries while this was happening.
>>
>> An alternative is to not hard-code the names in your copy script,
>> rather look at the information in ZooKeeper for your source and
>> target information, you could do this by using the CLUSTERSTATUS
>> collections API call.
>>
>> Best,
>> Erick
>>
>> On Tue, Mar 6, 2018 at 6:47 AM, Patrick Schemitz <ps@solute.de> wrote:
>> > Hi List,
>> >
>> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8
>> > shards on 2 servers, with 4 instances per server, no replicas, i.e.
>> > 1 shard per instance).
>> >
>> > Building the index afresh takes 15+ hours, so when I have to deploy
>> > a new index, I build it once, on one cluster, and then copy (scp)
>> > over the data/<main_index>/index directories (shutting down the Solr
instances first).
>> >
>> > I could get Solr 6.5.1 to number the shard/replica directories
>> > nicely via the createNodeSet and createNodeSet.shuffle options:
>> >
>> > Solr 6.5.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica1
>> > instance01/data/main_index_shard2_replica1
>> > instance02/data/main_index_shard3_replica1
>> > instance03/data/main_index_shard4_replica1
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica1
>> > instance01/data/main_index_shard6_replica1
>> > instance02/data/main_index_shard7_replica1
>> > instance03/data/main_index_shard8_replica1
>> >
>> > However, while attempting to upgrade to 7.2.1, this numbering has
changed:
>> >
>> > Solr 7.2.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica_n1
>> > instance01/data/main_index_shard2_replica_n2
>> > instance02/data/main_index_shard3_replica_n4
>> > instance03/data/main_index_shard4_replica_n6
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica_n8
>> > instance01/data/main_index_shard6_replica_n10
>> > instance02/data/main_index_shard7_replica_n12
>> > instance03/data/main_index_shard8_replica_n14
>> >
>> > This new numbering breaks my copy script, and furthermode, I'm
>> > worried as to what happens when the numbering is different among target
clusters.
>> >
>> > How can I switch this back to the old numbering scheme?
>> >
>> > Side note: is there a recommended way of doing this? Is the
>> > backup/restore mechanism suitable for this? The ref guide is kind
>> > of terse here.
>> >
>> > Thanks in advance,
>> >
>> > Ciao, Patrick