[ome-devel] Plan for transferring data from one OMERO to another

Wed Nov 1 22:00:00 GMT 2017

Hi Will, Josh,

That does work! Thank you very much for your help.

In the gist code, this line:
conn = BlitzGateway(client_obj=client)
should instead be:
conn = BlitzGateway(client_obj=c)
to access remote instead of local.

Cheers,
Eilidh

> On 1 Nov 2017, at 11:54, William Moore (Staff) <W.Moore at dundee.ac.uk <mailto:W.Moore at dundee.ac.uk>> wrote:
> 
> 
> Hi Eilidh,
> 
>  I updated my gist at https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c <https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c>
> with Josh’s fix.
> This also now uses the BlitzGateway to list projects on the remote server.
> 
> c = omero.client(host=‘demo.openmicroscopy.org <http://demo.openmicroscopy.org/>', port=4064, args=["--Ice.Config=/dev/null"])
> c.createSession(‘username', ‘password')
> print c
> conn = BlitzGateway(client_obj=client)
> …
> 
> Hope that helps,
> 
>   Will.
> 
> 
>> On 1 Nov 2017, at 10:53, Josh Moore <josh at glencoesoftware.com <mailto:josh at glencoesoftware.com>> wrote:
>> 
>> Hi Eilidh,
>> 
>> On Wed, Nov 1, 2017 at 12:36 AM, William Moore (Staff)
>> <W.Moore at dundee.ac.uk <mailto:W.Moore at dundee.ac.uk>> wrote:
>>> Hi Eilidh,
>>> 
>>> I created a test python script to illustrate the issue:
>>> https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c <https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c>
>>> 
>>> Hopefully that will help debugging…
>>> 
>>> We’ll let you know if we make progress….
>> 
>> using Will's gist as a basis, the problem is that scripts are run with
>> a default configuration, pointing to the launching server. In the
>> script itself, line 28 [1] creates the remote connection:
>> 
>>    c = omero.client(host='%s', port=4064)
>> 
>> To see the default configuration that is being passed, use:
>> 
>>    c = omero.client(host='%s', port=4064, args=["--omero.dump=1"])
>> 
>> To _disable_ the default configuration, use:
>> 
>>    c = omero.client(host='%s', port=4064, args=["--Ice.Config=/dev/null"])
>> 
>> All the best,
>> ~Josh
>> 
>> 
>> 
>> [1] https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28 <https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28>
>> 
>> 
>> 
>>> Cheers,
>>> 
>>>  Will.
>>> 
>>> 
>>> 
>>> On 31 Oct 2017, at 22:18, William Moore (Staff) <W.Moore at dundee.ac.uk <mailto:W.Moore at dundee.ac.uk>>
>>> wrote:
>>> 
>>> Hi Eilidh,
>>> 
>>> Yes, I’m afraid the output from conn.connect() is not very helpful.
>>> 
>>> You can try the underlying omero.client connection like this:
>>> 
>>>>>> c = omero.client(host=‘demo.openmicroscopy.org <http://demo.openmicroscopy.org/>', port=4064)
>>>>>> c.createSession(‘username', ‘password’)
>>> 
>>> which should give you an exception if this fails.
>>> 
>>> e.g.
>>> 
>>> $ omero script launch 87923
>>> Using session 009e7c47-5b59-4156-a664-21bd8d84ea68 (root at localhost:4064).
>>> Idle timeout: 10 min. Current group: system
>>> Enter value for "IDs": 1
>>> Job 89261 ready
>>> Waiting....
>>> Callback received: FINISHED
>>> 
>>> *** start stderr (id=89762)***
>>> * Traceback (most recent call last):
>>> *   File "./script", line 394, in <module>
>>> *     run_script()
>>> *   File "./script", line 369, in run_script
>>> *     ok = c.createSession('user', ‘password')
>>> *   File
>>> "/Users/wmoore/Desktop/OMERO/openmicroscopy/dist/lib/python/omero/clients.py",
>>> line 569, in createSession
>>> *     prx = rtr.createSession(username, password, ctx)
>>> *   File "/usr/local/lib/python2.7/site-packages/Glacier2_Router_ice.py",
>>> line 258, in createSession
>>> *     return _M_Glacier2.Router._op_createSession.invoke(self, ((userId,
>>> password), _ctx))
>>> * Glacier2.PermissionDeniedException: exception
>>> ::Glacier2::PermissionDeniedException
>>> * {
>>> *     reason = Password check failed for 'user': []
>>> * }
>>> * !! 10/31/17 21:30:57.788 error: communicator not destroyed during global
>>> destruction.
>>> *** end stderr ***
>>> 
>>> 
>>> *** out parameters ***
>>> ***  done ***
>>> 
>>> In fact, I’ve found the same issue as you - Connection to a different server
>>> doesn’t work from within an OMERO.script
>>> but does work locally.
>>> Not sure what is going on here but I’ll ask others on the OME team.
>>> 
>>> You shouldn’t be seeing a "ValidationException Can’t find params…” when
>>> you run the script.
>>> 
>>> Do you see any errors when uploading the script with $ omero script upload
>>> ?
>>> Or are you simply putting the script into the lib/scripts directory?
>>> 
>>> Cheers,
>>> 
>>>   Will.
>>> 
>>> 
>>> 
>>> On 31 Oct 2017, at 16:15, Eilidh Troup <e.troup at epcc.ed.ac.uk <mailto:e.troup at epcc.ed.ac.uk>> wrote:
>>> 
>>> Hi,
>>> 
>>> Thank you very much for your thoughtful response to my question on how to
>>> transfer data from one OMERO to another.
>>> 
>>> I was able to create a script to take an Image ID as input, along with
>>> username & password of central server, and could list the files belonging to
>>> the image too.
>>> 
>>> I got stuck when I tried to create a BlitzGateway connection to central
>>> server. I’m using the demo OMERO server as my remote OMERO.
>>> 
>>> This is a short script that reproduces the problem. I can connect to a local
>>> or remote OMERO.server using the following script when I run it as a python
>>> script directly from the command line, but when I run it as an omero script,
>>> it will connect to the local server (itself), but not to the remote server:
>>> remote_conn.connect() returns False. Is there a way to get more information
>>> about why it won’t connect? I’m at a loss as to how to debug this.
>>> 
>>> --------------------------------------------------
>>> Connect_to_remote_omero.py
>>> 
>>> from omero.gateway import BlitzGateway
>>> 
>>> # remote
>>> username = "etroup"
>>> password = “redacted"
>>> host = 'demo.openmicroscopy.org <http://demo.openmicroscopy.org/>'
>>> 
>>> # local
>>> # username = "root"
>>> # password = “redacted"
>>> # host = 'localhost'
>>> port = 4064
>>> 
>>> # Connect to remote server.
>>> remote_conn = BlitzGateway(username, password, host=host, port=port)
>>> connected = remote_conn.connect() # returns true if connected
>>> 
>>> if (connected):
>>>   print "connected"
>>>   for p in remote_conn.listProjects():
>>>       print p.getName()
>>> else:
>>>   print "not connected"
>>> 
>>> remote_conn.close()
>>> —————————————————————————
>>> 
>>> Running from python (works for local or remote server):
>>> 
>>> python Connect_to_remote_omero.py
>>> 
>>> connected
>>> Project1
>>> 
>>> Running as OMERO script, fails to connect to remote server:
>>> 
>>> bin/omero script list
>>> 
>>> 623 | /rdm/Connect_to_remote_omero.py
>>> 524 | /rdm/Export_to_other_omero.py
>>> 547 | /rdm/ping.py
>>> (20 rows)
>>> 
>>> bin/omero script launch 623
>>> 
>>> Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
>>> Idle timeout: 10 min. Current group: system
>>> ValidationException: Can't find params for 623!
>>> stdout is in file 624:
>>> ---------------------------------
>>> not connected
>>> 
>>> ---------------------------------
>>> No stderr.
>>> 
>>> Running as OMERO script, does connect to local server:
>>> bin/omero script launch 625
>>> Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
>>> Idle timeout: 10 min. Current group: system
>>> ValidationException: Can't find params for 625!
>>> stdout is in file 626:
>>> ---------------------------------
>>> connected
>>> project2
>>> project1
>>> 
>>> ————————————————
>>> 
>>> Thanks,
>>> Eilidh
>>> 
>>> 
>>> 
>>> 
>>> On 17 Oct 2017, at 12:00, ome-devel-request at lists.openmicroscopy.org.uk <mailto:ome-devel-request at lists.openmicroscopy.org.uk>
>>> wrote:
>>> 
>>> Send ome-devel mailing list submissions to
>>> ome-devel at lists.openmicroscopy.org.uk <mailto:ome-devel at lists.openmicroscopy.org.uk>
>>> 
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
>>> or, via email, send a message with subject or body 'help' to
>>> ome-devel-request at lists.openmicroscopy.org.uk <mailto:ome-devel-request at lists.openmicroscopy.org.uk>
>>> 
>>> You can reach the person managing the list at
>>> ome-devel-owner at lists.openmicroscopy.org.uk <mailto:ome-devel-owner at lists.openmicroscopy.org.uk>
>>> 
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of ome-devel digest..."
>>> 
>>> 
>>> Today's Topics:
>>> 
>>> 1. Re: Plan for transferring data from one OMERO to another
>>>    (William Moore (Staff))
>>> 
>>> 
>>> ----------------------------------------------------------------------
>>> 
>>> Message: 1
>>> Date: Tue, 17 Oct 2017 10:18:18 +0000
>>> From: "William Moore (Staff)" <W.Moore at dundee.ac.uk <mailto:W.Moore at dundee.ac.uk>>
>>> To: OME External Developer List
>>> <ome-devel at lists.openmicroscopy.org.uk <mailto:ome-devel at lists.openmicroscopy.org.uk>>
>>> Subject: Re: [ome-devel] Plan for transferring data from one OMERO to
>>> another
>>> Message-ID: <818C04E1-B318-472F-AEF4-16F998AEB015 at dundee.ac.uk <mailto:818C04E1-B318-472F-AEF4-16F998AEB015 at dundee.ac.uk>>
>>> Content-Type: text/plain; charset="utf-8"
>>> 
>>> Hi,
>>> 
>>> This could be quite a bit of work, so you will need to start small and
>>> build-up functionality as you go. Some investigation will be required - Some
>>> things will depend on the type of images you?ll be transferring. You?ll also
>>> need to have both OMERO servers using the same major version. E.g. 5.3.x.
>>> 
>>> You?ll really need to re-import the images to the central server instead of
>>> manually creating new DB entries for all image metadata etc. This can be
>>> acheived from the command line, with your OMERO.script invoking ?bin/omero
>>> import ?? to the central server.
>>> 
>>> Are these images imported in OMERO 5.0 or later (original files will be in
>>> managed repo)?
>>> Your script, running on the source OMERO server *may* be able to access the
>>> managed repository directly, and call e.g. ?bin/omero import
>>> /OMERO/ManagedRepository/will_3/2017-10/05/15-31-34.090/control.lsm?
>>> This is certainly the easiest solution.
>>> 
>>> Alternatively, if the managed repo is not available then you can download
>>> the original images to the folder where the OMERO.script is running and
>>> import the images from there.
>>> In the simplest case, a single OMERO Image has a single image file. However,
>>> some Images are stored as multiple original files in the managed repo, which
>>> means they will all need to be downloaded to the script before they can be
>>> imported if you go for that option.
>>> 
>>> In some cases, a single original file (or set of original files) will
>>> generate multiple OMERO Images when imported. So you may find that multiple
>>> images in your source server come from the same original file(s). In that
>>> case, you will have to be careful when you iterate through these images that
>>> you don?t import the same original file multiple times.
>>> 
>>> It?s not really possible for OMERO.scripts to be ?interactive?. The easiest
>>> option for choosing a target is to have your users log in to the webclient
>>> (or Insight) in the central server and pick a Project or Dataset ID to enter
>>> into the script dialog in the other server.
>>> 
>>> Traversing the graph of linked objects Dataset -> Image and handling of file
>>> Annotations / duplicate tags etc should be pretty straight-forward once you
>>> have the re-import of images working.
>>> 
>>> Please put your script in github and commit all changes as you go. This will
>>> allow us to see what you have working and to help or contribute much easier.
>>> One option if you don?t already have a repo is to fork
>>> https://github.com/ome/omero-user-scripts <https://github.com/ome/omero-user-scripts> which allows us and the community
>>> to find your repo from there.
>>> 
>>> Be careful when handling usernames and passwords in your OMERO.script. Make
>>> sure you?re not logging these etc.
>>> 
>>> Initial goals:
>>> - Create an script that can take an Image ID as input, along with username &
>>> password of central server. Can use
>>> https://github.com/openmicroscopy/openmicroscopy/blob/develop/examples/Training/python/Scripting_Service_Example.py <https://github.com/openmicroscopy/openmicroscopy/blob/develop/examples/Training/python/Scripting_Service_Example.py>
>>> - List original files from source Image
>>> for f in image.getImportedImageFiles():
>>>  print f.path, f.name
>>> - Create a BlitzGateway connection to central server
>>> - You can get a ?client? object using  client = conn.c
>>> - Command line import using code similar to that in our python test
>>> framework (using ?client? from above as in the example)
>>> :https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/src/omero/testlib/__init__.py#L277 <https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/src/omero/testlib/__init__.py#L277>
>>> 
>>> 
>>> Let us know how this goes,
>>> 
>>> Regards,
>>> 
>>> Will.
>>> 
>>> 
>>> On 12 Oct 2017, at 12:46, Eilidh Troup
>>> <e.troup at epcc.ed.ac.uk <mailto:e.troup at epcc.ed.ac.uk><mailto:e.troup at epcc.ed.ac.uk <mailto:e.troup at epcc.ed.ac.uk>>> wrote:
>>> 
>>> Hi,
>>> 
>>> I?m a software engineer working on data management in biology. Our use case
>>> is that lab groups have their own instances of OMERO, and there is also a
>>> centrally-hosted university OMERO server which they would like to transfer
>>> data to. I?m planning to write a script to allow users to transfer data from
>>> one OMERO to another. I wanted to run the plans by this forum to see what
>>> you thought of the idea, and whether my planned implementation seems
>>> sensible.
>>> 
>>> I plan to write a script (OMERO.scripts) to run on the local instances of
>>> OMERO and be used through OMERO.server or OMERO.web. This will let the user
>>> choose the image, dataset or project that they want to move, and enter their
>>> credentials for the central OMERO server. Then the script will use the Blitz
>>> Gateway to read data from the local server, and then to write to the remote
>>> server.
>>> 
>>> issues:
>>> 
>>> - How will users choose where to put data? It will have to be either in a
>>> default project on the central server, or the script would have to be
>>> interactive, where the user enters their credentials, and then the central
>>> server is contacted so that a list of projects and datasets belonging to
>>> that user can be presented.
>>> 
>>> - There will be a graph of objects linked to any given one, for example, a
>>> dataset with images, tags and attachments. Need to traverse this to transfer
>>> all associated data.
>>> 
>>> - Some IDs will need to be mapped from one OMERO to the other, for example
>>> tags - shouldn?t create a new one, but link to existing tag with same name
>>> on the other OMERO.
>>> 
>>> I?m planning to write the code in Python and use OMERO.scripts and the Blitz
>>> Gateway. Please let me know what you think.
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk <mailto:ome-devel at lists.openmicroscopy.org.uk>
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
> 
> 
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk <mailto:ome-devel at lists.openmicroscopy.org.uk>
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20171101/6d5c6e9d/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20171101/6d5c6e9d/attachment.ksh>