[ome-users] BlitzGateway: keep connection alive to write large ROIs

Wed May 10 16:27:15 BST 2017

Thanks, Josh.

I'll first try and figure out why I don't get a dump of configs. I'm 
sure that will solve the immediate problem. Afterwards, I'll turn my 
attention to batching then.

Paul

On 10/05/2017 15:33, Josh Moore wrote:
> Hi Paul,
>
> On Wed, May 10, 2017 at 3:46 PM, Paul Korir <pkorir at ebi.ac.uk> wrote:
>> Hi Josh,
>>
>> Thanks for your reply.
>>
>> I've set omero.dump=1 but I don't see any output on console nor do I get a
>> dump file. It seems there's something amiss with my setup.
> If omero.dump is set, then you should get all key-value pairs to standard out:
>
> https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.8/components/tools/OmeroPy/src/omero/clients.py#L254
>
>
>> The 84MB is a set of ROIs resulting from slicing a segmentation (of a
>> ribosome - many polygons result). I expect to get much bigger files with
>> time. I'd be happy to learn another way to do this. I'm not familiar with
>> working with the object graph (please shed some light).
> Ultimately you'll need to introduce batching, e.g. uploading every 100
> or 1000 ROIs that you parse out of an image. If you can show us a
> snippet, we may be able to make a suggestion.
>
>
>> The code is not public but it doesn't do anything more than described in the
>> OMERO Python examples.
>>
>> Thanks.
>> Paul
> All the best,
> ~Josh.
>
>
>
>
>> On 10/05/2017 10:49, Josh Moore wrote:
>>> Hi Paul,
>>>
>>>
>>> On Tue, May 9, 2017 at 4:39 PM, Paul Korir <pkorir at ebi.ac.uk> wrote:
>>>> Hi Will,
>>>>
>>>> It turns out the problem had to do with a mismatch in the client and
>>>> server
>>>> versions. I was using a 5.2.5 client with a 5.1.4 server. The actual
>>>> problem
>>>> was a memory exception (when I match client and server version) which is
>>>> proving harder to resolve than I thought. I've asked this question before
>>>> on
>>>> this mailing list and I think I resolved that case.
>>>>
>>>> But right now setting Ice.MessageSizeMax does note seem to have any
>>>> effect.
>>>>
>>>> I've used:
>>>>
>>>> bin/omero config edit (then added Ice.MessageSizeMax=131072; I've also
>>>> done
>>>> this using bin/omero config set Ice.MessageSizeMax 131072)
>>>>
>>>> and edited both etc/ice.config and etc/grid/templates.xml files for both
>>>> client and server with no effect. All I keep getting is a data dump and:
>>> Can you also add "omero.dump=1" to your etc/ice.config setting and
>>> make sure you are seeing the expected MessageSizeMax?
>>>
>>>
>>>> Traceback (most recent call last):
>>>>     File
>>>>
>>>> "/Users/pkorir/omero/OMERO.py-5.1.4-ice35-b55/lib/python/omero/gateway/__init__.py",
>>>> line 4160, in __call__
>>>>       return self.f(*args, **kwargs)
>>>>     File
>>>>
>>>> "/Users/pkorir/omero/OMERO.py-5.1.4-ice35-b55/lib/python/omero_api_IUpdate_ice.py",
>>>> line 107, in saveObject
>>>>       return _M_omero.api.IUpdate._op_saveObject.invoke(self, ((obj, ),
>>>> _ctx))
>>>> MemoryLimitException: exception ::Ice::MemoryLimitException
>>>> {
>>>>       reason = requested 83930389 bytes, maximum allowed is 67108864 bytes
>>>> (see Ice.MessageSizeMax)
>>>> }
>>> What are you trying to save that's 84 MB? Is this code available
>>> somewhere? Would it be possible to save parts of the object graph
>>> instead?
>>> ~Josh
>>>
>>>
>>>
>>>> Any help will be gratefully received.
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>> On 01/05/2017 13:15, William Moore (Staff) wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> I used the tiny script below to test what you’re seeing….
>>>>
>>>> First I set the session timeout to 10 seconds:
>>>>
>>>> $ bin/omero config set omero.sessions.timeout 10000
>>>>
>>>> Then I ran the script to see if the  conn.c.enableKeepAlive(5)  would
>>>> allow
>>>> me to
>>>> perform other calls that lasted longer than 10 seconds, such as
>>>> time.sleep(15).
>>>>
>>>> This seems to work.
>>>> I think you may have your keepAlive interval longer than the session
>>>> timeout?
>>>> This is confusing since the session timeout is in milliseconds but the
>>>> keepAlive interval is in seconds, so it’s easy to accidentally set the
>>>> keepAlive
>>>> to be too long (longer than the session timeout).
>>>>
>>>> In the example below, if I use conn.c.enableKeepAlive(50)
>>>> then I see the same as you "warning: Proxy keep alive failed.”
>>>> because this is longer than 10 seconds (omero.sessions.timeout 10000)
>>>>
>>>> Hopefully that was the problem?
>>>> If not, you’ll need to show us more of your code, or a sample script that
>>>> replicates the problem, and say what your session timeout is.
>>>>
>>>>    Regards,
>>>>
>>>>      Will.
>>>>
>>>>
>>>>
>>>> from omero.gateway import BlitzGateway
>>>>
>>>> conn = BlitzGateway(“username", “password", host="localhost", port=4064)
>>>> conn.connect()
>>>>
>>>> # Ping interval in SECONDS - KeepAlive Interval must be shorter than
>>>> # omero config set omero.sessions.timeout 10000
>>>> # which is in MILLISECONDS
>>>> conn.c.enableKeepAlive(5)
>>>>
>>>> print "start..."
>>>> time.sleep(15)
>>>>
>>>> print conn.c.sf.getAdminService().getEventContext().userName
>>>>
>>>> conn.close()
>>>>
>>>>
>>>>
>>>> On 27 Apr 2017, at 14:04, Paul Korir <pkorir at ebi.ac.uk> wrote:
>>>>
>>>> By the way, the keepalive gave the following:
>>>>
>>>> -! 04/27/17 12:07:15.006 warning: Proxy keep alive failed.
>>>>
>>>>
>>>> On 26/04/2017 11:16, William Moore (Staff) wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>>    You can enable the keepAlive on the client object with
>>>> conn.c.enableKeepAlive(300)
>>>> See 5.2 docs:
>>>>
>>>> https://downloads.openmicroscopy.org/omero/5.2.0/api/omero/client.html#enableKeepAlive-int-
>>>>
>>>> Also you can set this in the etc/omero.properties file.
>>>> See discussion at
>>>> https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=495
>>>>
>>>> Hope that helps,
>>>>
>>>>     Will.
>>>>
>>>>
>>>> --
>>>> Paul K. Korir, PhD
>>>> Scientific Programmer
>>>> EMBL-EBI
>>>> 01223494422
>>>>
>>>>
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>
>>>>
>>>> --
>>>> Paul K. Korir, PhD
>>>> Scientific Programmer
>>>> EMBL-EBI
>>>> 01223494422
>>>>
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> --
>> Paul K. Korir, PhD
>> Scientific Programmer
>> EMBL-EBI
>> 01223494422
>>

-- 
Paul K. Korir, PhD
Scientific Programmer
EMBL-EBI
01223494422