[ome-users] BlitzGateway: keep connection alive to write large ROIs

Wed May 10 15:33:49 BST 2017

Hi Paul,

On Wed, May 10, 2017 at 3:46 PM, Paul Korir <pkorir at ebi.ac.uk> wrote:
> Hi Josh,
>
> Thanks for your reply.
>
> I've set omero.dump=1 but I don't see any output on console nor do I get a
> dump file. It seems there's something amiss with my setup.

If omero.dump is set, then you should get all key-value pairs to standard out:

https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.8/components/tools/OmeroPy/src/omero/clients.py#L254

> The 84MB is a set of ROIs resulting from slicing a segmentation (of a
> ribosome - many polygons result). I expect to get much bigger files with
> time. I'd be happy to learn another way to do this. I'm not familiar with
> working with the object graph (please shed some light).

Ultimately you'll need to introduce batching, e.g. uploading every 100
or 1000 ROIs that you parse out of an image. If you can show us a
snippet, we may be able to make a suggestion.

> The code is not public but it doesn't do anything more than described in the
> OMERO Python examples.
>
> Thanks.
> Paul

All the best,
~Josh.

> On 10/05/2017 10:49, Josh Moore wrote:
>>
>> Hi Paul,
>>
>>
>> On Tue, May 9, 2017 at 4:39 PM, Paul Korir <pkorir at ebi.ac.uk> wrote:
>>>
>>> Hi Will,
>>>
>>> It turns out the problem had to do with a mismatch in the client and
>>> server
>>> versions. I was using a 5.2.5 client with a 5.1.4 server. The actual
>>> problem
>>> was a memory exception (when I match client and server version) which is
>>> proving harder to resolve than I thought. I've asked this question before
>>> on
>>> this mailing list and I think I resolved that case.
>>>
>>> But right now setting Ice.MessageSizeMax does note seem to have any
>>> effect.
>>>
>>> I've used:
>>>
>>> bin/omero config edit (then added Ice.MessageSizeMax=131072; I've also
>>> done
>>> this using bin/omero config set Ice.MessageSizeMax 131072)
>>>
>>> and edited both etc/ice.config and etc/grid/templates.xml files for both
>>> client and server with no effect. All I keep getting is a data dump and:
>>
>> Can you also add "omero.dump=1" to your etc/ice.config setting and
>> make sure you are seeing the expected MessageSizeMax?
>>
>>
>>> Traceback (most recent call last):
>>>    File
>>>
>>> "/Users/pkorir/omero/OMERO.py-5.1.4-ice35-b55/lib/python/omero/gateway/__init__.py",
>>> line 4160, in __call__
>>>      return self.f(*args, **kwargs)
>>>    File
>>>
>>> "/Users/pkorir/omero/OMERO.py-5.1.4-ice35-b55/lib/python/omero_api_IUpdate_ice.py",
>>> line 107, in saveObject
>>>      return _M_omero.api.IUpdate._op_saveObject.invoke(self, ((obj, ),
>>> _ctx))
>>> MemoryLimitException: exception ::Ice::MemoryLimitException
>>> {
>>>      reason = requested 83930389 bytes, maximum allowed is 67108864 bytes
>>> (see Ice.MessageSizeMax)
>>> }
>>
>> What are you trying to save that's 84 MB? Is this code available
>> somewhere? Would it be possible to save parts of the object graph
>> instead?
>> ~Josh
>>
>>
>>
>>> Any help will be gratefully received.
>>>
>>> Paul
>>>
>>>
>>>
>>> On 01/05/2017 13:15, William Moore (Staff) wrote:
>>>
>>> Hi Paul,
>>>
>>> I used the tiny script below to test what you’re seeing….
>>>
>>> First I set the session timeout to 10 seconds:
>>>
>>> $ bin/omero config set omero.sessions.timeout 10000
>>>
>>> Then I ran the script to see if the  conn.c.enableKeepAlive(5)  would
>>> allow
>>> me to
>>> perform other calls that lasted longer than 10 seconds, such as
>>> time.sleep(15).
>>>
>>> This seems to work.
>>> I think you may have your keepAlive interval longer than the session
>>> timeout?
>>> This is confusing since the session timeout is in milliseconds but the
>>> keepAlive interval is in seconds, so it’s easy to accidentally set the
>>> keepAlive
>>> to be too long (longer than the session timeout).
>>>
>>> In the example below, if I use conn.c.enableKeepAlive(50)
>>> then I see the same as you "warning: Proxy keep alive failed.”
>>> because this is longer than 10 seconds (omero.sessions.timeout 10000)
>>>
>>> Hopefully that was the problem?
>>> If not, you’ll need to show us more of your code, or a sample script that
>>> replicates the problem, and say what your session timeout is.
>>>
>>>   Regards,
>>>
>>>     Will.
>>>
>>>
>>>
>>> from omero.gateway import BlitzGateway
>>>
>>> conn = BlitzGateway(“username", “password", host="localhost", port=4064)
>>> conn.connect()
>>>
>>> # Ping interval in SECONDS - KeepAlive Interval must be shorter than
>>> # omero config set omero.sessions.timeout 10000
>>> # which is in MILLISECONDS
>>> conn.c.enableKeepAlive(5)
>>>
>>> print "start..."
>>> time.sleep(15)
>>>
>>> print conn.c.sf.getAdminService().getEventContext().userName
>>>
>>> conn.close()
>>>
>>>
>>>
>>> On 27 Apr 2017, at 14:04, Paul Korir <pkorir at ebi.ac.uk> wrote:
>>>
>>> By the way, the keepalive gave the following:
>>>
>>> -! 04/27/17 12:07:15.006 warning: Proxy keep alive failed.
>>>
>>>
>>> On 26/04/2017 11:16, William Moore (Staff) wrote:
>>>
>>> Hi Paul,
>>>
>>>   You can enable the keepAlive on the client object with
>>> conn.c.enableKeepAlive(300)
>>> See 5.2 docs:
>>>
>>> https://downloads.openmicroscopy.org/omero/5.2.0/api/omero/client.html#enableKeepAlive-int-
>>>
>>> Also you can set this in the etc/omero.properties file.
>>> See discussion at
>>> https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=495
>>>
>>> Hope that helps,
>>>
>>>    Will.
>>>
>>>
>>> --
>>> Paul K. Korir, PhD
>>> Scientific Programmer
>>> EMBL-EBI
>>> 01223494422
>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>
>>>
>>> --
>>> Paul K. Korir, PhD
>>> Scientific Programmer
>>> EMBL-EBI
>>> 01223494422
>>>
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
> --
> Paul K. Korir, PhD
> Scientific Programmer
> EMBL-EBI
> 01223494422
>