[ome-devel] Plan for transferring data from one OMERO to another

[Josh Moore]

Hi Eilidh,

On Wed, Nov 1, 2017 at 12:36 AM, William Moore (Staff)
<W.Moore at dundee.ac.uk> wrote:
> Hi Eilidh,
>
>  I created a test python script to illustrate the issue:
> https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c
>
>  Hopefully that will help debugging…
>
> We’ll let you know if we make progress….

using Will's gist as a basis, the problem is that scripts are run with
a default configuration, pointing to the launching server. In the
script itself, line 28 [1] creates the remote connection:

    c = omero.client(host='%s', port=4064)

To see the default configuration that is being passed, use:

    c = omero.client(host='%s', port=4064, args=["--omero.dump=1"])

To _disable_ the default configuration, use:

    c = omero.client(host='%s', port=4064, args=["--Ice.Config=/dev/null"])

All the best,
~Josh



[1] https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28



>  Cheers,
>
>   Will.
>
>
>
> On 31 Oct 2017, at 22:18, William Moore (Staff) <W.Moore at dundee.ac.uk>
> wrote:
>
> Hi Eilidh,
>
>  Yes, I’m afraid the output from conn.connect() is not very helpful.
>
> You can try the underlying omero.client connection like this:
>
>>>> c = omero.client(host=‘demo.openmicroscopy.org', port=4064)
>>>> c.createSession(‘username', ‘password’)
>
> which should give you an exception if this fails.
>
> e.g.
>
> $ omero script launch 87923
> Using session 009e7c47-5b59-4156-a664-21bd8d84ea68 (root at localhost:4064).
> Idle timeout: 10 min. Current group: system
> Enter value for "IDs": 1
> Job 89261 ready
> Waiting....
> Callback received: FINISHED
>
> *** start stderr (id=89762)***
> * Traceback (most recent call last):
> *   File "./script", line 394, in <module>
> *     run_script()
> *   File "./script", line 369, in run_script
> *     ok = c.createSession('user', ‘password')
> *   File
> "/Users/wmoore/Desktop/OMERO/openmicroscopy/dist/lib/python/omero/clients.py",
> line 569, in createSession
> *     prx = rtr.createSession(username, password, ctx)
> *   File "/usr/local/lib/python2.7/site-packages/Glacier2_Router_ice.py",
> line 258, in createSession
> *     return _M_Glacier2.Router._op_createSession.invoke(self, ((userId,
> password), _ctx))
> * Glacier2.PermissionDeniedException: exception
> ::Glacier2::PermissionDeniedException
> * {
> *     reason = Password check failed for 'user': []
> * }
> * !! 10/31/17 21:30:57.788 error: communicator not destroyed during global
> destruction.
> *** end stderr ***
>
>
> *** out parameters ***
> ***  done ***
>
> In fact, I’ve found the same issue as you - Connection to a different server
> doesn’t work from within an OMERO.script
> but does work locally.
> Not sure what is going on here but I’ll ask others on the OME team.
>
> You shouldn’t be seeing a "ValidationException Can’t find params…” when
> you run the script.
>
> Do you see any errors when uploading the script with $ omero script upload
> ?
> Or are you simply putting the script into the lib/scripts directory?
>
>  Cheers,
>
>    Will.
>
>
>
> On 31 Oct 2017, at 16:15, Eilidh Troup <e.troup at epcc.ed.ac.uk> wrote:
>
> Hi,
>
> Thank you very much for your thoughtful response to my question on how to
> transfer data from one OMERO to another.
>
> I was able to create a script to take an Image ID as input, along with
> username & password of central server, and could list the files belonging to
> the image too.
>
> I got stuck when I tried to create a BlitzGateway connection to central
> server. I’m using the demo OMERO server as my remote OMERO.
>
> This is a short script that reproduces the problem. I can connect to a local
> or remote OMERO.server using the following script when I run it as a python
> script directly from the command line, but when I run it as an omero script,
> it will connect to the local server (itself), but not to the remote server:
> remote_conn.connect() returns False. Is there a way to get more information
> about why it won’t connect? I’m at a loss as to how to debug this.
>
> --------------------------------------------------
> Connect_to_remote_omero.py
>
> from omero.gateway import BlitzGateway
>
> # remote
> username = "etroup"
> password = “redacted"
> host = 'demo.openmicroscopy.org'
>
> # local
> # username = "root"
> # password = “redacted"
> # host = 'localhost'
> port = 4064
>
> # Connect to remote server.
> remote_conn = BlitzGateway(username, password, host=host, port=port)
> connected = remote_conn.connect() # returns true if connected
>
> if (connected):
>    print "connected"
>    for p in remote_conn.listProjects():
>        print p.getName()
> else:
>    print "not connected"
>
> remote_conn.close()
> —————————————————————————
>
> Running from python (works for local or remote server):
>
> python Connect_to_remote_omero.py
>
> connected
> Project1
>
> Running as OMERO script, fails to connect to remote server:
>
> bin/omero script list
>
> 623 | /rdm/Connect_to_remote_omero.py
> 524 | /rdm/Export_to_other_omero.py
> 547 | /rdm/ping.py
> (20 rows)
>
> bin/omero script launch 623
>
> Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
> Idle timeout: 10 min. Current group: system
> ValidationException: Can't find params for 623!
> stdout is in file 624:
> ---------------------------------
> not connected
>
> ---------------------------------
> No stderr.
>
> Running as OMERO script, does connect to local server:
> bin/omero script launch 625
> Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
> Idle timeout: 10 min. Current group: system
> ValidationException: Can't find params for 625!
> stdout is in file 626:
> ---------------------------------
> connected
> project2
> project1
>
> ————————————————
>
> Thanks,
> Eilidh
>
>
>
>
> On 17 Oct 2017, at 12:00, ome-devel-request at lists.openmicroscopy.org.uk
> wrote:
>
> Send ome-devel mailing list submissions to
> ome-devel at lists.openmicroscopy.org.uk
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> or, via email, send a message with subject or body 'help' to
> ome-devel-request at lists.openmicroscopy.org.uk
>
> You can reach the person managing the list at
> ome-devel-owner at lists.openmicroscopy.org.uk
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ome-devel digest..."
>
>
> Today's Topics:
>
>  1. Re: Plan for transferring data from one OMERO to another
>     (William Moore (Staff))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 17 Oct 2017 10:18:18 +0000
> From: "William Moore (Staff)" <W.Moore at dundee.ac.uk>
> To: OME External Developer List
> <ome-devel at lists.openmicroscopy.org.uk>
> Subject: Re: [ome-devel] Plan for transferring data from one OMERO to
> another
> Message-ID: <818C04E1-B318-472F-AEF4-16F998AEB015 at dundee.ac.uk>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> This could be quite a bit of work, so you will need to start small and
> build-up functionality as you go. Some investigation will be required - Some
> things will depend on the type of images you?ll be transferring. You?ll also
> need to have both OMERO servers using the same major version. E.g. 5.3.x.
>
> You?ll really need to re-import the images to the central server instead of
> manually creating new DB entries for all image metadata etc. This can be
> acheived from the command line, with your OMERO.script invoking ?bin/omero
> import ?? to the central server.
>
> Are these images imported in OMERO 5.0 or later (original files will be in
> managed repo)?
> Your script, running on the source OMERO server *may* be able to access the
> managed repository directly, and call e.g. ?bin/omero import
> /OMERO/ManagedRepository/will_3/2017-10/05/15-31-34.090/control.lsm?
> This is certainly the easiest solution.
>
> Alternatively, if the managed repo is not available then you can download
> the original images to the folder where the OMERO.script is running and
> import the images from there.
> In the simplest case, a single OMERO Image has a single image file. However,
> some Images are stored as multiple original files in the managed repo, which
> means they will all need to be downloaded to the script before they can be
> imported if you go for that option.
>
> In some cases, a single original file (or set of original files) will
> generate multiple OMERO Images when imported. So you may find that multiple
> images in your source server come from the same original file(s). In that
> case, you will have to be careful when you iterate through these images that
> you don?t import the same original file multiple times.
>
> It?s not really possible for OMERO.scripts to be ?interactive?. The easiest
> option for choosing a target is to have your users log in to the webclient
> (or Insight) in the central server and pick a Project or Dataset ID to enter
> into the script dialog in the other server.
>
> Traversing the graph of linked objects Dataset -> Image and handling of file
> Annotations / duplicate tags etc should be pretty straight-forward once you
> have the re-import of images working.
>
> Please put your script in github and commit all changes as you go. This will
> allow us to see what you have working and to help or contribute much easier.
> One option if you don?t already have a repo is to fork
> https://github.com/ome/omero-user-scripts which allows us and the community
> to find your repo from there.
>
> Be careful when handling usernames and passwords in your OMERO.script. Make
> sure you?re not logging these etc.
>
> Initial goals:
> - Create an script that can take an Image ID as input, along with username &
> password of central server. Can use
> https://github.com/openmicroscopy/openmicroscopy/blob/develop/examples/Training/python/Scripting_Service_Example.py
> - List original files from source Image
> for f in image.getImportedImageFiles():
>   print f.path, f.name
> - Create a BlitzGateway connection to central server
> - You can get a ?client? object using  client = conn.c
> - Command line import using code similar to that in our python test
> framework (using ?client? from above as in the example)
> :https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/src/omero/testlib/__init__.py#L277
>
>
> Let us know how this goes,
>
> Regards,
>
> Will.
>
>
> On 12 Oct 2017, at 12:46, Eilidh Troup
> <e.troup at epcc.ed.ac.uk<mailto:e.troup at epcc.ed.ac.uk>> wrote:
>
> Hi,
>
> I?m a software engineer working on data management in biology. Our use case
> is that lab groups have their own instances of OMERO, and there is also a
> centrally-hosted university OMERO server which they would like to transfer
> data to. I?m planning to write a script to allow users to transfer data from
> one OMERO to another. I wanted to run the plans by this forum to see what
> you thought of the idea, and whether my planned implementation seems
> sensible.
>
> I plan to write a script (OMERO.scripts) to run on the local instances of
> OMERO and be used through OMERO.server or OMERO.web. This will let the user
> choose the image, dataset or project that they want to move, and enter their
> credentials for the central OMERO server. Then the script will use the Blitz
> Gateway to read data from the local server, and then to write to the remote
> server.
>
> issues:
>
> - How will users choose where to put data? It will have to be either in a
> default project on the central server, or the script would have to be
> interactive, where the user enters their credentials, and then the central
> server is contacted so that a list of projects and datasets belonging to
> that user can be presented.
>
> - There will be a graph of objects linked to any given one, for example, a
> dataset with images, tags and attachments. Need to traverse this to transfer
> all associated data.
>
> - Some IDs will need to be mapped from one OMERO to the other, for example
> tags - shouldn?t create a new one, but link to existing tag with same name
> on the other OMERO.
>
> I?m planning to write the code in Python and use OMERO.scripts and the Blitz
> Gateway. Please let me know what you think.