FW: edit gc parameters in solr.in.sh or solr?

-----Original Message-----
From: Shawn Heisey [mailto:elyograg@elyograg.org]
Sent: 28 March 2018 13:44
To: solr-user@lucene.apache.org
Subject: Re: edit gc parameters in solr.in.sh or solr?

On 3/28/2018 1:06 AM, Bernd Fehling wrote:
> Humongous Allocations are not genrally bad. Sure, the G1 part for
> humongous allocations is not that performant and takes time. But just
> try to limit humongous allocations and not to avoid it under all
circumstances.

I was told by Oracle engineers that humongous allocations can ONLY be
collected by a full GC. A full GC on my servers, with an 8GB heap, can take
10-15 seconds.  This isn't a guess -- I've examined the GC logs. And that's
a full stop-the-world pause for the entire collection.  If I increase the
region size, then filterCache entries are not tagged as humongous, and full
GCs almost never happen.

Every single filterCache entry on my large shards was more than 2MB at the
time I was experimenting.  With an 8GB heap, the calculated G1 region size
is 4MB.  Anything that's half the region size or larger is considered a
humongous allocation, so objects over 2MB get that tag.

Each server has three of those large shards, so when a new query goes into
the filterCache, three large filterCache entries are created in one JVM. 
These days, each of those shards has 31 million docs, which means that a
filterCache entry has almost doubled in size since then.  It's almost 4 MB.

If one of those cores gets updated, multiple large objects become garbage,
and because of cache autowarming (with autowarmCount set to 4), a few more
get created.  Each of our queries typically has several filter queries in
it.  Some of them don't really change, but a couple of them are highly
variable from user to user.  Solr burns through the heap pretty fast with
those large filterCache entries.  When it fills up with garbage objects
tagged humongous, the only way to clear it out is a full GC, because the
concurrent low-pause collectors are unable to collect humongous objects.

Because the region size tops out at 32MB, that means that any object over
16MB is always humongous.  So for indexes with a very large number of
documents per core (more than about 134 million), G1 is not a good choice. 
Unless the filterCache is completely disabled, which isn't normally
recommended, G1 can't be tuned to avoid full GC with an index that size.

Thanks,
Shawn