FW: CLUSTERSTATUS API and Error loading specified collection / config in Solr 5.3.2.

-----Original Message-----
From: Atita Arora [mailto:atitaarora@gmail.com]
Sent: 13 March 2018 20:49
To: solr-user@lucene.apache.org
Subject: Re: CLUSTERSTATUS API and Error loading specified collection /
config in Solr 5.3.2.

Hi Hendrik and Shalin,

Really appreciate your valuable inputs on this.

I looked up to the two issues that were being referred to (SOLR-8804 and
SOLR-10720) and thats exactly what I'm running into.
Glad they have been fixed in later versions.

Thanks much ,
Atita

On Tue, Mar 13, 2018 at 10:38 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:

> I think you are running into race conditions in the API which have
> been fixed. See SOLR-8804 and SOLR-10720. The first is available in
> 5.5.1 but the latter fix will be released in the upcoming 7.3 release.
> The best workaround for your version is to just retry a few times
> until the API succeeds.
>
> On Sun, Mar 11, 2018 at 11:57 PM, Atita Arora <atitaarora@gmail.com>
> wrote:
>
> > Hi ,
> >
> > I am working on an application which involves working on a highly
> > distributed Solr cloud environment. The application supports
> multi-tenancy
> > and we have around 250-300 collections on Solr where each client has
> their
> > own collection with a new shard being created as
> clientid-<timestamp_long>
> > where the timestamp is whenever the new data comes in for the client
> > (typically every 4-8 hrs) , the reason for this convention is to
> > make
> sure
> > when the Indexes are being built (on demand) the timestamp matches
> closely
> > to the time when the last indexing was run (the earlier shard is
> > de-provisioned as soon as the new one is created). Whenever the
> > indexing
> is
> > triggered it first makes a DB entry and then creates a catalog with
> > timestamp in solr.
> > The Solr cloud has 10 Nodes distributed geographically among 10
> > datacenters.
> > The replication factor is 2. The Solr version is 5.3.2.
> > Coming to my problem - I had to write a utility to ensure that the
> > DB insert timestamp matches closely to the Solr index timestamp
> > wherein I
> can
> > ensure that if the difference between DB timestamp and Solr Index
> tinestamp
> > is <= 2 hrs , we have fresh index. The new index contains revised
> > prices
> of
> > products or offers etc which are critical to be updated as in when
> > they come. Hence this utility is to track that the required updates
> > have been successfully made.
> > I used *CLUSTERSTATUS* api for this task. It is serving the purpose
> > well
> so
> > far , but pretty recently our solr cloud started complaining of
> > strange things because of which the *CLUSTERSTATUS* api keeps returning
as error.
> >
> > The error claims to be of missing config & sometime missing
> > collections like.
> >
> > org.apache.solr.common.SolrException: Could not find collection :
> > > 1785-1520548816454
> >
> > org.apache.solr.common.SolrException: Could not find collection :
> > 1785-1520548816454
> > at
> > org.apache.solr.common.cloud.ClusterState.getCollection(
> > ClusterState.java:165)
> > at
> > org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(
> > ClusterStatus.java:110)
> > at
> > org.apache.solr.handler.admin.CollectionsHandler$
> > CollectionOperation$19.call(CollectionsHandler.java:614)
> > at
> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(
> > CollectionsHandler.java:166)
> > at
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(
> > RequestHandlerBase.java:143)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> > HttpSolrCall.java:678)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> > SolrDispatchFilter.java:215)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> > SolrDispatchFilter.java:179)
> > at
> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> > doFilter(ServletHandler.java:1652)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:585)
> >
> > The other times it would complain of missing the config for same or
> > different client id- timestamp like :
> >
> > 1532-1518669619526_shard1_replica3:
> > org.apache.solr.common.cloud.ZooKeeperException:org.apache.
> > solr.common.cloud.ZooKeeperException:
> > Specified config does not exist in ZooKeeper:1532-1518669619526I
> >
> > I would really appreciate if :
> >
> >
> > 1. Someone can possibly guide me as to whats going on Solr Cloud
> > 2. If CLUSTERSTATUS is the right pick to build such utility. Do
> > we have any other option?
> >
> >
> > Thanks for any pointers and suggestions.
> >
> > Appreciate your attention looking this through.
> >
> > Atita
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

FW: Parameter expansion and functions

-----Original Message-----
From: Dariusz Wojtas [mailto:dwojtas@gmail.com]
Sent: 13 March 2018 19:53
To: solr-user@lucene.apache.org
Subject: Parameter expansion and functions

Hi,

I am using SOLR 7.1
I've read some time ago the article
http://yonik.com/solr-query-parameter-substitution/
It's very helpful, knowing the trick with dummy parameter as a workaround in
some cases is very helpful.
${dummyParam:${localVariable}}

It is possible to use constructs like this in solrsonfig.xml <lst
name="defaults">
<str name="varA">10</str>
<str name="varB">${dummyParam:${varA}} and some other content</str>
</lst> It resolves as expected. This way I may calculate some parameters on
the fly.

But is there any chance to make this feature more complex?
Can I do something conditionally inside?
Can I use logical operators, or some other functions to construct local
parameters?
I know, one may say it may break security, etc.
Let it be turned off by default, let it require allowing local parameters by
name, to be allowed this. but can it be done?

My current need is to define the buusted query on the fly, but this feature
could help me in the past also. Example:

I have static 'q' param:
<str name="q">{!edismax
qf='idx_IDs^5 idx_names^3 idx_address^1'
v=$searchedTerms}</str>

Sometimes it is longer.
And I'd like to decide on the fly which fields should be included in the
query, same as their weights.
It works if I want to assign parametrized weight:
<str name="q">{!edismax
qf='idx_IDs^5 idx_names^3
idx_address^${dummyParam:${myWeightParam}}'
v=$searchedTerms}</str>

But can I make it more dynamic? Conditionals? I ytried with the {!switch
case.... default... v... } but it looks like this function expansion is
applied at some later stage, and it fails when I try to put it in "q"
definition.


Best regards,
Dariusz Wojtas

FW: How to store files larger than zNode limit

-----Original Message-----
From: Atita Arora [mailto:atitaarora@gmail.com]
Sent: 14 March 2018 03:08
To: solr-user@lucene.apache.org
Subject: How to store files larger than zNode limit

Hi ,

I have a use case supporting multiple clients and multiple languages in a
single application.
So , In order to improve the language support, we want to leverage the Solr
dictionary (userdict.txt) files as large as 10MB.
I understand that ZooKeeper's default zNode file size limit is 1MB.
I'm not sure sure if someone tried increasing it before and how does that
fares in terms of performance.
Looking at - https://zookeeper.apache.org/doc/r3.2.2/zookeeperAdmin.html
It states -
Unsafe Options

The following options can be useful, but be careful when you use them. The
risk of each is explained along with the explanation of what the variable
does.
jute.maxbuffer:

(Java system property:* jute.maxbuffer*)

This option can only be set as a Java system property. There is no zookeeper
prefix on it. It specifies the maximum size of the data that can be stored
in a znode. The default is 0xfffff, or just under 1M. If this option is
changed, the system property must be set on all servers and clients
otherwise problems will arise. This is really a sanity check.
ZooKeeper is designed to store data on the order of kilobytes in size.
I would appreciate if someone has any suggestions on what are the best
practices for handling large config/dictionary files in ZK?

Thanks ,
Atita

FW: Continuing Saga of Authorization on 6.6.0

-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: 14 March 2018 01:33
To: solr-user@lucene.apache.org
Subject: Re: Continuing Saga of Authorization on 6.6.0

On 3/13/2018 12:00 PM, Terry Steichen wrote:
> So, Shawn, every time zookeeper gets shut down (intentionally or
> otherwise), I have to recreate the credentials and permissions via a
> set of API calls?  Is there some way to have it save and read that
> stuff from disk?

If your ZK ensemble doesn't keep that info around through restarts, then it
is badly configured.  The ZK database should be persistent.  They do have a
concept of an ephemeral node, and I think Solr does use that feature, but it
should NOT be in use for the majority of what Solr puts in ZK.

Thanks,
Shawn

FW: Continuing Saga of Authorization on 6.6.0

-----Original Message-----
From: Terry Steichen [mailto:terry@net-frame.com]
Sent: 13 March 2018 22:17
To: solr-user@lucene.apache.org
Subject: Continuing Saga of Authorization on 6.6.0

I switched solr from standalone to cloud and created the two collections
(emails1 and emails2). 

I was able to create a basic set of credentials via the curl-based API's.  I
could create users, and toggle the blockUnknown property status. However,
the system refused to allow me to delete a user, or to set a permission. 

Here are the curl commands (with *terry:admin* as admin credentials) and
results:

*succeeded in setting blockUnknown property (verified by
admin/authentication dump):*

curl --user terry:admin http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{
  "set-property": {"blockUnknown" : true}}'

*succeeded in adding a user (verified by admin/authentication dump):*

curl --user terry:admin http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{
>   "set-user": {"lanny" : "hawaii"}}'

*succeeded in changing lanny's password (verified by admin/authentication
dump):*

curl --user terry:admin http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{
 "set-user": {"lanny" : "hawaii_five_o"}}'

*failed to delete a user:*

 curl --user terry:admin http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{
 "delete-user": {"lanny"}}'
{
  "responseHeader":{
    "status":500,
    "QTime":1},

  "error":{ "msg":"Expected key,value separator ':': char=},position=26
BEFORE='{ \"delete-user\": {\"lanny\"}' AFTER='}'", [terry here: plus a very
long stack trace}

*failed to set a permission: *

curl --user terry:admin http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{"set-permission" :
{"name":"collection-admin-edit", "role":"admin"}}'
{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "errorMessages":[{
      "set-permission":{
        "name":"collection-admin-edit",
        "role":"admin"},
      "errorMessages":["Unknown operation 'set-permission' "]}]}


This really makes no sense at all (or, I'm really losing it - always a
distinct possibility).  It's almost as if half of the documented parameters
must have been changed, though I can't find any references to any such
changes. 

I confess I'm about to just give up and find some other route to go. 

Terry


On 03/12/2018 11:15 PM, Shawn Heisey wrote:
> On 3/12/2018 8:39 PM, Terry Steichen wrote:
>> I'm increasingly of the view that Solr's authentication/authorization
>> mechanism doesn't work correctly in a _standalone_ mode.  It was
>> present in the cloud mode for quite a few versions back, but as of
>> 6.0.0 (or so) it was supposed to be available in standalone mode too. 
>> It seems to partly work (when using the built-in permissions), but
>> does not seem to work with customized, core-specific permissions.
>
> I suspected based on your last message that the authorization feature
> might only work correctly in SolrCloud.  The entire authentication
> feature was designed for SolrCloud.  Version 6.5 brought the
> security.json file to standalone mode.  This was LONG after the
> feature was introduced in 5.2 and had a LOT of bugs fixed in the three
> 5.3.x releases.
>
> I just found the section in the documentation confirming what I
> suspected.
>
> https://lucene.apache.org/solr/guide/7_2/authentication-and-authorizat
> ion-plugins.html#authorization
>
>
> There is a note here that says "The authorization plugin is only
> supported in SolrCloud mode. Also, reloading the plugin isn't yet
> supported and requires a restart of the Solr installation (meaning,
> the JVM should be restarted, not simply a core reload)."  The 6.6
> documentation contains the same note that you can see here in the
> latest docs.
>
> I have no idea how hard it would be to extend the authorization plugin
> to support standalone cores as well as collections.  I imagine that if
> it were easy, it would have been done already.
>
> Thanks,
> Shawn
>
>

FW: Boosting with 0 factor

-----Original Message-----
From: Dariusz Wojtas [mailto:dwojtas@gmail.com]
Sent: 13 March 2018 21:59
To: solr-user@lucene.apache.org
Subject: Boosting with 0 factor

Hi,

I have a question about boosting queries with ^0

I am using LTR. In the 1st step I want to narrow the query, but limit
'noise' results as much as possible. The 1st step is defined as follows:
<str name="q">{!edismax qf='keyword_id^10 keyword_nonid^2
keyword_lastNames^2 keyword_names^1' v=$searchedTerms}</str>

And I have lots of such boosted fields, but want to narrow them dynamically.
I may pass boost factors as local params, with ${boostVal}. This part works.
If I set boost to 0, as in the example below for field 'keyword_id', will it
execute query on that field or skip it since boost of factor 0 will boost
the result?
In short: will it bring performance savings, or not?

<str name="q">{!edismax qf='keyword_id^0 keyword_nonid^2
keyword_lastNames^2 keyword_names^1' v=$searchedTerms}</str>


Best regards,
Dariusz Wojtas

FW: Navigation/Paging

-----Original Message-----
From: Sebastian Riemer [mailto:s.riemer@littera.eu]
Sent: 13 March 2018 21:56
To: solr-user@lucene.apache.org
Subject: Navigation/Paging

Hi,

In our web app, when displaying result lists from solr, we've successfully
introduced paging via the params 'start' and 'rows' and it's working quite
well.

Our navigation in list screens look like this:


<< First < Prev 1 - 15 of 62181 Next
><http://test.litterare.local:3100/littera/libraries/2/cat/man?locale=en>
Last
>><http://test.litterare.local:3100/littera/libraries/2/cat/man?locale=en>

One can navigate to the first page, previous page, next page and last page.
All is done via adapting the param "start" accordingly by simply adding the
page size.

However, now we want to introduce a similar navigation in our detail views,
where only ever one document is displayed. Again, the navigation bar looks
like this:

<< First < Prev 1 - 15 of 62181 Next
><http://test.litterare.local:3100/littera/libraries/2/cat/man?locale=en>
Last
>><http://test.litterare.local:3100/littera/libraries/2/cat/man?locale=en>

But now, Prev / Next shall open up the previous / next _document_ instead of
the next page. The same goes for First and Last, it shall open the first /
last _document_ not the page.

Our first approach to this was to simply add the param "fl=id" so we only
get the IDs of documents and set page size to ALL (i.e. no restriction on
param "rows"). That way, it was easy to extract the current document id from
the result list, and check which id was preceding and succeeding the current
id, as well as getting the very first id and the very last id, in order to
render the navigation bar.

This lead to solr being heavily under load since it must load 62181
documents (in this example) in order to return the ids. I somehow thought
this would be easy for solr to do, but it isn't.

Our second approach was, to simply keep the same value for params "start"
and "rows" since the user is always selecting a document from the list -
thus the selected document already is within the page. However, the edge
cases are, the selected document is the very first on the page or the very
last one, thus the previous or next document id is not within the page
result from solr -> I guess this we could handle by simply checking and
sending a second query where the param "start" would be adjusted
accordingly.

However I would not know how to retrieve the id of the very first document
and the very last document (except for executing separate queries with I
guess start=0, rows=1 and start=62181 and rows=1)

TL,DR:
For any query and a documentId (of which it is known it is within the query
result), what is a simple and efficient enough way, to get the following
navigational information:

- Previous document Id

- Next document id

- First document id

- Last document id

Can this sort of requirement be handled within one solr query? Should I user
cursorMark in this scenario?

Best regards,

Sebastian