[ome-devel] Plan for transferring data from one OMERO to another
William Moore (Staff)
W.Moore at dundee.ac.uk
Wed Nov 1 22:58:44 GMT 2017
Hi Eilidh,
Glad that worked and thanks for spotting the mistake in the gist - fixed now.
Cheers,
Will.
On 1 Nov 2017, at 22:00, Eilidh Troup <e.troup at epcc.ed.ac.uk<mailto:e.troup at epcc.ed.ac.uk>> wrote:
Hi Will, Josh,
That does work! Thank you very much for your help.
In the gist code, this line:
conn = BlitzGateway(client_obj=client)
should instead be:
conn = BlitzGateway(client_obj=c)
to access remote instead of local.
Cheers,
Eilidh
On 1 Nov 2017, at 11:54, William Moore (Staff) <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>> wrote:
Hi Eilidh,
I updated my gist at https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c
with Josh’s fix.
This also now uses the BlitzGateway to list projects on the remote server.
c = omero.client(host=‘demo.openmicroscopy.org<http://demo.openmicroscopy.org/>', port=4064, args=["--Ice.Config=/dev/null"])
c.createSession(‘username', ‘password')
print c
conn = BlitzGateway(client_obj=client)
…
Hope that helps,
Will.
On 1 Nov 2017, at 10:53, Josh Moore <josh at glencoesoftware.com<mailto:josh at glencoesoftware.com>> wrote:
Hi Eilidh,
On Wed, Nov 1, 2017 at 12:36 AM, William Moore (Staff)
<W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>> wrote:
Hi Eilidh,
I created a test python script to illustrate the issue:
https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c
Hopefully that will help debugging…
We’ll let you know if we make progress….
using Will's gist as a basis, the problem is that scripts are run with
a default configuration, pointing to the launching server. In the
script itself, line 28 [1] creates the remote connection:
c = omero.client(host='%s', port=4064)
To see the default configuration that is being passed, use:
c = omero.client(host='%s', port=4064, args=["--omero.dump=1"])
To _disable_ the default configuration, use:
c = omero.client(host='%s', port=4064, args=["--Ice.Config=/dev/null"])
All the best,
~Josh
[1] https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28
Cheers,
Will.
On 31 Oct 2017, at 22:18, William Moore (Staff) <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>>
wrote:
Hi Eilidh,
Yes, I’m afraid the output from conn.connect() is not very helpful.
You can try the underlying omero.client connection like this:
c = omero.client(host=‘demo.openmicroscopy.org<http://demo.openmicroscopy.org/>', port=4064)
c.createSession(‘username', ‘password’)
which should give you an exception if this fails.
e.g.
$ omero script launch 87923
Using session 009e7c47-5b59-4156-a664-21bd8d84ea68 (root at localhost:4064).
Idle timeout: 10 min. Current group: system
Enter value for "IDs": 1
Job 89261 ready
Waiting....
Callback received: FINISHED
*** start stderr (id=89762)***
* Traceback (most recent call last):
* File "./script", line 394, in <module>
* run_script()
* File "./script", line 369, in run_script
* ok = c.createSession('user', ‘password')
* File
"/Users/wmoore/Desktop/OMERO/openmicroscopy/dist/lib/python/omero/clients.py",
line 569, in createSession
* prx = rtr.createSession(username, password, ctx)
* File "/usr/local/lib/python2.7/site-packages/Glacier2_Router_ice.py",
line 258, in createSession
* return _M_Glacier2.Router._op_createSession.invoke(self, ((userId,
password), _ctx))
* Glacier2.PermissionDeniedException: exception
::Glacier2::PermissionDeniedException
* {
* reason = Password check failed for 'user': []
* }
* !! 10/31/17 21:30:57.788 error: communicator not destroyed during global
destruction.
*** end stderr ***
*** out parameters ***
*** done ***
In fact, I’ve found the same issue as you - Connection to a different server
doesn’t work from within an OMERO.script
but does work locally.
Not sure what is going on here but I’ll ask others on the OME team.
You shouldn’t be seeing a "ValidationException Can’t find params…” when
you run the script.
Do you see any errors when uploading the script with $ omero script upload
?
Or are you simply putting the script into the lib/scripts directory?
Cheers,
Will.
On 31 Oct 2017, at 16:15, Eilidh Troup <e.troup at epcc.ed.ac.uk<mailto:e.troup at epcc.ed.ac.uk>> wrote:
Hi,
Thank you very much for your thoughtful response to my question on how to
transfer data from one OMERO to another.
I was able to create a script to take an Image ID as input, along with
username & password of central server, and could list the files belonging to
the image too.
I got stuck when I tried to create a BlitzGateway connection to central
server. I’m using the demo OMERO server as my remote OMERO.
This is a short script that reproduces the problem. I can connect to a local
or remote OMERO.server using the following script when I run it as a python
script directly from the command line, but when I run it as an omero script,
it will connect to the local server (itself), but not to the remote server:
remote_conn.connect() returns False. Is there a way to get more information
about why it won’t connect? I’m at a loss as to how to debug this.
--------------------------------------------------
Connect_to_remote_omero.py
from omero.gateway import BlitzGateway
# remote
username = "etroup"
password = “redacted"
host = 'demo.openmicroscopy.org<http://demo.openmicroscopy.org/>'
# local
# username = "root"
# password = “redacted"
# host = 'localhost'
port = 4064
# Connect to remote server.
remote_conn = BlitzGateway(username, password, host=host, port=port)
connected = remote_conn.connect() # returns true if connected
if (connected):
print "connected"
for p in remote_conn.listProjects():
print p.getName()
else:
print "not connected"
remote_conn.close()
—————————————————————————
Running from python (works for local or remote server):
python Connect_to_remote_omero.py
connected
Project1
Running as OMERO script, fails to connect to remote server:
bin/omero script list
623 | /rdm/Connect_to_remote_omero.py
524 | /rdm/Export_to_other_omero.py
547 | /rdm/ping.py
(20 rows)
bin/omero script launch 623
Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
Idle timeout: 10 min. Current group: system
ValidationException: Can't find params for 623!
stdout is in file 624:
---------------------------------
not connected
---------------------------------
No stderr.
Running as OMERO script, does connect to local server:
bin/omero script launch 625
Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root at localhost:4064).
Idle timeout: 10 min. Current group: system
ValidationException: Can't find params for 625!
stdout is in file 626:
---------------------------------
connected
project2
project1
————————————————
Thanks,
Eilidh
On 17 Oct 2017, at 12:00, ome-devel-request at lists.openmicroscopy.org.uk<mailto:ome-devel-request at lists.openmicroscopy.org.uk>
wrote:
Send ome-devel mailing list submissions to
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
or, via email, send a message with subject or body 'help' to
ome-devel-request at lists.openmicroscopy.org.uk<mailto:ome-devel-request at lists.openmicroscopy.org.uk>
You can reach the person managing the list at
ome-devel-owner at lists.openmicroscopy.org.uk<mailto:ome-devel-owner at lists.openmicroscopy.org.uk>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of ome-devel digest..."
Today's Topics:
1. Re: Plan for transferring data from one OMERO to another
(William Moore (Staff))
----------------------------------------------------------------------
Message: 1
Date: Tue, 17 Oct 2017 10:18:18 +0000
From: "William Moore (Staff)" <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>>
To: OME External Developer List
<ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-devel] Plan for transferring data from one OMERO to
another
Message-ID: <818C04E1-B318-472F-AEF4-16F998AEB015 at dundee.ac.uk<mailto:818C04E1-B318-472F-AEF4-16F998AEB015 at dundee.ac.uk>>
Content-Type: text/plain; charset="utf-8"
Hi,
This could be quite a bit of work, so you will need to start small and
build-up functionality as you go. Some investigation will be required - Some
things will depend on the type of images you?ll be transferring. You?ll also
need to have both OMERO servers using the same major version. E.g. 5.3.x.
You?ll really need to re-import the images to the central server instead of
manually creating new DB entries for all image metadata etc. This can be
acheived from the command line, with your OMERO.script invoking ?bin/omero
import ?? to the central server.
Are these images imported in OMERO 5.0 or later (original files will be in
managed repo)?
Your script, running on the source OMERO server *may* be able to access the
managed repository directly, and call e.g. ?bin/omero import
/OMERO/ManagedRepository/will_3/2017-10/05/15-31-34.090/control.lsm?
This is certainly the easiest solution.
Alternatively, if the managed repo is not available then you can download
the original images to the folder where the OMERO.script is running and
import the images from there.
In the simplest case, a single OMERO Image has a single image file. However,
some Images are stored as multiple original files in the managed repo, which
means they will all need to be downloaded to the script before they can be
imported if you go for that option.
In some cases, a single original file (or set of original files) will
generate multiple OMERO Images when imported. So you may find that multiple
images in your source server come from the same original file(s). In that
case, you will have to be careful when you iterate through these images that
you don?t import the same original file multiple times.
It?s not really possible for OMERO.scripts to be ?interactive?. The easiest
option for choosing a target is to have your users log in to the webclient
(or Insight) in the central server and pick a Project or Dataset ID to enter
into the script dialog in the other server.
Traversing the graph of linked objects Dataset -> Image and handling of file
Annotations / duplicate tags etc should be pretty straight-forward once you
have the re-import of images working.
Please put your script in github and commit all changes as you go. This will
allow us to see what you have working and to help or contribute much easier.
One option if you don?t already have a repo is to fork
https://github.com/ome/omero-user-scripts which allows us and the community
to find your repo from there.
Be careful when handling usernames and passwords in your OMERO.script. Make
sure you?re not logging these etc.
Initial goals:
- Create an script that can take an Image ID as input, along with username &
password of central server. Can use
https://github.com/openmicroscopy/openmicroscopy/blob/develop/examples/Training/python/Scripting_Service_Example.py
- List original files from source Image
for f in image.getImportedImageFiles():
print f.path, f.name
- Create a BlitzGateway connection to central server
- You can get a ?client? object using client = conn.c
- Command line import using code similar to that in our python test
framework (using ?client? from above as in the example)
:https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/src/omero/testlib/__init__.py#L277
Let us know how this goes,
Regards,
Will.
On 12 Oct 2017, at 12:46, Eilidh Troup
<e.troup at epcc.ed.ac.uk<mailto:e.troup at epcc.ed.ac.uk><mailto:e.troup at epcc.ed.ac.uk>> wrote:
Hi,
I?m a software engineer working on data management in biology. Our use case
is that lab groups have their own instances of OMERO, and there is also a
centrally-hosted university OMERO server which they would like to transfer
data to. I?m planning to write a script to allow users to transfer data from
one OMERO to another. I wanted to run the plans by this forum to see what
you thought of the idea, and whether my planned implementation seems
sensible.
I plan to write a script (OMERO.scripts) to run on the local instances of
OMERO and be used through OMERO.server or OMERO.web. This will let the user
choose the image, dataset or project that they want to move, and enter their
credentials for the central OMERO server. Then the script will use the Blitz
Gateway to read data from the local server, and then to write to the remote
server.
issues:
- How will users choose where to put data? It will have to be either in a
default project on the central server, or the script would have to be
interactive, where the user enters their credentials, and then the central
server is contacted so that a list of projects and datasets belonging to
that user can be presented.
- There will be a graph of objects linked to any given one, for example, a
dataset with images, tags and attachments. Need to traverse this to transfer
all associated data.
- Some IDs will need to be mapped from one OMERO to the other, for example
tags - shouldn?t create a new one, but link to existing tag with same name
on the other OMERO.
I?m planning to write the code in Python and use OMERO.scripts and the Blitz
Gateway. Please let me know what you think.
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20171101/914f3971/attachment.html>
More information about the ome-devel
mailing list