[ome-devel] OMERO QA submisssion #9383

[Josh Moore]

Moving to ome-devel.

See:  https://www.openmicroscopy.org/qa2/qa/feedback/9383

Caused by: Ice.UnknownUserException
  unknown = "omero::OverUsageException
  serverStackTrace = "ome.conditions.OverUsageException: servantsPerSession reached for 59d08ac3-7ffb-46be-ba8d-ede69be27237: 10000
    at omero.util.ServantHolder.put(ServantHolder.java:161)
    at omero.cmd.SessionI.registerServant(SessionI.java:589)
    at omero.cmd.SessionI.submit_async(SessionI.java:203)


On Jul 2, 2014, at 11:07 AM, Kenneth Gillen wrote:
> On 01/07/2014 18:16, "S. Simard" <sebastien.simard at pasteur.fr> wrote:
> 
>> Hi Kenny,

Hi Sebastien,


>> Thanks for following this up - fyi, here is the current state of things.
>> 
>> My user attempts to upload large-ish datasets (around 1600 TIF files,
>> about 1.7 M each) using OMERO.insight.
>> This happens sequentially, so you typically have two or three such bulk
>> imports running within Insight/Importer tabs.

>> The problems that arise are then:
>> 1 - partial failure to import due to the "servantsPerSession" error you
>> received
>> 2 - in order to cleanup the inconsistent dataset, my user attempts to
>> delete it, and hits another set of errors
>> 
>> 
>> The import issue looks partly similar to [trac 12012] - lsof gives
>> around 3500 open files after the OverUsageException is reported, but
>> they only seem to get cleared on a server restart.

https://trac.openmicroscopy.org.uk/ome/ticket/12012 points to
https://github.com/openmicroscopy/openmicroscopy/pull/2207
which is currently only on develop.


>> Trying to use the CLI as an alternative to perform parallel imports hits
>> [trac 11096].


https://trac.openmicroscopy.org.uk/ome/ticket/11096 occurs with imports
from one terminal, or multiple?


>> I am currently running some more tests with Insight and the following
>> server-side changes:
>> - increase the "nofile" limit for the "omero" user from 4096 to 8192
>> (soft) and 12288 (hard)
>> - omero config set omero.throttling.servants_per_session 20000
>> If you are aware of caveats with this, or sensible alternatives, I would
>> be interested to hear them.

The only caveat is that until 12012 is fixed, you're consuming ever more
server resources by raising this limit. That may be worth it to you, but
you're essentially only delaying the problem.


>> WRT the delete issue, it seems reproducible locally with some dummy data:
>> - mkdir test-dataset && cd test-dataset
>> - for i in `seq 0 1600`; do touch "$i.fake"; done
>> - import the test-dataset folder into OMERO
>> - then using Insight or web, try and delete the data:
>>  * first select a bunch of images and delete them, then delete the
>> dataset: works fine
>>  * select and delete the dataset directly:
>>    i) just a note: fails with a Blitz OOM on a stock 5.0.2 server zip
>>    ii) with 2048M heap allocated to Blitz: the process eventually
>> fails with a "org.postgresql.util.PSQLException: ERROR: out of shared
>> memory  Hint: You might need to increase max_locks_per_transaction." on
>> postgres 8.4 (I haven't tried upping this setting, but I'm not sure it
>> would be the right thing to do either)

Increasing the memory available to OMERO will help, but once you run out of
memory in postgresql, you'll have to bump the value as suggested. Yanling
recently had to do the same, and said that 2048 was working for the moment.

>> Regards,
>> Sebastien
>> 
>> [trac 12012] http://trac.openmicroscopy.org.uk/ome/ticket/12012
>> [trac 11096] http://trac.openmicroscopy.org.uk/ome/ticket/11096

Cheers,
~Josh.