FW: FW: Question about Overseer calling SPLITSHARD collection API command during autoscaling

-----Original Message-----
From: Cassandra Targett [mailto:casstargett@gmail.com]
Sent: 15 March 2018 22:49
To: solr-user@lucene.apache.org
Subject: Re: FW: Question about Overseer calling SPLITSHARD collection API
command during autoscaling

Hi Matthew -

It's cool to hear you're using the new autoscaling features.

To answer your first question, SPLITSHARD as an action for autoscaling is
not yet supported. As for when it might be, it's the next big gap to fill in
the autoscaling functionality, but there is some work to do first to make
splitting shards faster and safer overall. So, I hope we'll see it in 7.4,
but there's a chance it won't be ready until the release after (7.5, I'd
assume).

AFAICT, there isn't a JIRA issue specifically for the SPLITSHARD support
yet, but there will be one relatively soon. There's an umbrella issue for a
many of the open tasks if you're interested in that:
https://issues.apache.org/jira/browse/SOLR-9735 (although, it's not an
exhaustive roadmap, I don't think).

I think for the time being if you want/need to split a shard, you'd still
need to do it manually.

Hope this helps -
Cassandra

On Thu, Mar 15, 2018 at 11:41 AM, Matthew Faw <Matthew.Faw@verato.com>
wrote:

> I sent this a few mins ago, but wasn't yet subscribed. Forwarding the
> message along to make sure it's received!
>
> From: Matthew Faw <Matthew.Faw@verato.com>
> Date: Thursday, March 15, 2018 at 12:28 PM
> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
> Cc: Matthew Faw <Matthew.Faw@verato.com>, Alex Meijer <
> Alex.Meijer@verato.com>
> Subject: Question about Overseer calling SPLITSHARD collection API
> command during autoscaling
>
> Hi,
>
> So I've been trying out the new autoscaling features in solr 7.2.1. I
> run the following commands when creating my solr cluster:
>
>
> Set up overseer role:
> curl -s "solr-service-core:8983/solr/admin/collections?action=
> ADDROLE&role=overseer&node=$thenode"
>
> Create cluster prefs:
> clusterprefs=$(cat <<-EOF
> {
> "set-cluster-preferences" : [
> {"minimize":"sysLoadAvg"},
> {"minimize":"cores"}
> ]
> }
> EOF
> )
> echo "The cluster prefs request body is: $clusterprefs"
> curl -H "Content-Type: application/json" -X POST -d
> "$clusterprefs" solr-service-core:8983/api/cluster/autoscaling
>
> Cluster policy:
> clusterpolicy=$(cat <<-EOF
> {
> "set-cluster-policy": [
> {"replica": 0, "nodeRole": "overseer"},
> {"replica": "<2", "shard": "#EACH", "node": "#ANY"},
> {"cores": ">0", "node": "#ANY"},
> {"cores": "<5", "node": "#ANY"},
> {"replica": 0, "sysLoadAvg": ">80"}
> ]
> }
> EOF
> )
> echo "The cluster policy is $clusterpolicy"
> curl -H "Content-Type: application/json" -X POST -d
> "$clusterpolicy" solr-service-core:8983/api/cluster/autoscaling
>
> nodeaddtrigger=$(cat <<-EOF
> {
> "set-trigger": {
> "name" : "node_added_trigger",
> "event" : "nodeAdded",
> "waitFor" : "1s"
> }
> }
> EOF
> )
> echo "The node added trigger request: $nodeaddtrigger"
> curl -H "Content-Type: application/json" -X POST -d
> "$nodeaddtrigger" solr-service-core:8983/api/cluster/autoscaling
>
>
> I then create a collection with 2 shards and 3 replicas, under a set
> of nodes in an autoscaling group (initially 4, scales up to 10):
> curl -s
> "solr-service-core:8983/solr/admin/collections?action=
> CREATE&name=${COLLECTION_NAME}&numShards=${NUM_SHARDS}&
> replicationFactor=${NUM_REPLICAS}&autoAddReplicas=${
> AUTO_ADD_REPLICAS}&collection.configName=${COLLECTION_NAME}&
> waitForFinalState=true"
>
>
> I've observed several autoscaling actions being performed -
> automatically re-adding replicas, and moving shards to nodes based on
> my cluster policy/prefs. However, I have not observed a SPLITSHARD
> operation. My question is:
> 1) should I expect the Overseer to be able to call the SPLITSHARD
> command, or is this feature not yet implemented?
> 2) If it is possible, do you have any recommendations as to how I
> might force this type of behavior to happen?
> 3) If it's not implemented yet, when could I expect the feature to be
> available?
>
> If you need any more details, please let me know! Really excited about
> these new features.
>
> Thanks,
> Matthew
>
> The content of this email is intended solely for the individual or
> entity named above and access by anyone else is unauthorized. If you
> are not the intended recipient, any disclosure, copying, distribution,
> or use of the contents of this information is prohibited and may be
> unlawful. If you have received this electronic transmission in error,
> please reply immediately to the sender that you have received the message
in error, and delete it.
> Thank you.
>