FW: Schema update/Reload during Live traffic

-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: 24 March 2018 10:48
To: solr-user@lucene.apache.org
Subject: Re: Schema update/Reload during Live traffic

On 3/23/2018 10:32 PM, Susheel Kumar wrote:
> I did schema update to Solr cloud of Source CDCR cluster and same on
> target. After Collection Reload, noticed "error opening searcher" /
> IndexWriter closed etc. on leader node while all replica's went into
> recovery mode.
>
> Later after restarting Solr on Leader noticed below too many file open
> errors. The ulimit shows unlimited.
>
> What could have gone wrong above/and how can we avoid. This is with
> 6.6.2 and there were ingestions/delete and searches happening the time
> I reload the collection.
>
> Caused by: org.apache.solr.common.SolrException:
> java.io.FileNotFoundException:
> /app/solr/data/COLL_shard1_replica4/data/tlog/tlog.0000000000000002214
> .1591528549576081408
> (Too many open files)

A 'ulimit' command that you typed may have shown unlimited, but the Solr
process apparently DOES have a limit.

If you are trying to change limits with the ulimit command for Solr, I have
frequently seen this NOT work.  On Linux, /etc/security/limits.conf appears
to be the only way to successfully increase the open file limit.  The
default limit on Linux is normally 1024, which is pretty low.

I would recommend not doing reloads when there is a lot of indexing
activity.  When indexing is heavy, it can lead to recovery operations, if
the number of documents indexed during the reload exceeds the limit for
syncing from transaction logs.  This limit is normally 100 docs.

Thanks,
Shawn