FW: Schema update/Reload during Live traffic

-----Original Message-----
From: Susheel Kumar [mailto:susheel2777@gmail.com]
Sent: 24 March 2018 10:02
To: solr-user@lucene.apache.org
Subject: Schema update/Reload during Live traffic

Hello,

I did schema update to Solr cloud of Source CDCR cluster and same on target.
After Collection Reload, noticed "error opening searcher" / IndexWriter
closed etc. on leader node while all replica's went into recovery mode.

Later after restarting Solr on Leader noticed below too many file open
errors. The ulimit shows unlimited.

What could have gone wrong above/and how can we avoid. This is with 6.6.2
and there were ingestions/delete and searches happening the time I reload
the collection.

Caused by: org.apache.solr.common.SolrException:
java.io.FileNotFoundException:
/app/solr/data/COLL_shard1_replica4/data/tlog/tlog.0000000000000002214.15915
28549576081408
(Too many open files)

at
org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransaction
Log.java:250)

at
org.apache.solr.update.CdcrTransactionLog.incref(CdcrTransactionLog.java:179
)

at
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1342)

at org.apache.solr.update.UpdateLog.init(UpdateLog.java:393)

at org.apache.solr.update.CdcrUpdateLog.init(CdcrUpdateLog.java:77)

at
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:153)

at
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:110)

at
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java
:108)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:62)

at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:760)

at
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:822)

at
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)

at org.apache.solr.core.SolrCore.<init>(SolrCore.java:947)