<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">Hi Eilidh,</div>
<div class=""><br class="">
</div>
<div class=""> I updated my gist at <a href="https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c" class="">https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c</a></div>
<div class="">with Josh’s fix.</div>
<div class="">This also now uses the BlitzGateway to list projects on the remote server.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">c = omero.client(host=‘<a href="http://demo.openmicroscopy.org" class="">demo.openmicroscopy.org</a>', port=4064, args=["--Ice.Config=/dev/null"])</div>
<div class="">c.createSession(‘username', ‘password')</div>
<div class="">print c</div>
<div class="">conn = BlitzGateway(client_obj=client)</div>
</div>
<div class="">…</div>
<div class=""><br class="">
</div>
<div class="">Hope that helps,</div>
<div class=""><br class="">
</div>
<div class=""> Will.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 1 Nov 2017, at 10:53, Josh Moore <<a href="mailto:josh@glencoesoftware.com" class="">josh@glencoesoftware.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Hi Eilidh,<br class="">
<br class="">
On Wed, Nov 1, 2017 at 12:36 AM, William Moore (Staff)<br class="">
<<a href="mailto:W.Moore@dundee.ac.uk" class="">W.Moore@dundee.ac.uk</a>> wrote:<br class="">
<blockquote type="cite" class="">Hi Eilidh,<br class="">
<br class="">
I created a test python script to illustrate the issue:<br class="">
<a href="https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c" class="">https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c</a><br class="">
<br class="">
Hopefully that will help debugging…<br class="">
<br class="">
We’ll let you know if we make progress….<br class="">
</blockquote>
<br class="">
using Will's gist as a basis, the problem is that scripts are run with<br class="">
a default configuration, pointing to the launching server. In the<br class="">
script itself, line 28 [1] creates the remote connection:<br class="">
<br class="">
c = omero.client(host='%s', port=4064)<br class="">
<br class="">
To see the default configuration that is being passed, use:<br class="">
<br class="">
c = omero.client(host='%s', port=4064, args=["--omero.dump=1"])<br class="">
<br class="">
To _disable_ the default configuration, use:<br class="">
<br class="">
c = omero.client(host='%s', port=4064, args=["--Ice.Config=/dev/null"])<br class="">
<br class="">
All the best,<br class="">
~Josh<br class="">
<br class="">
<br class="">
<br class="">
[1] <a href="https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28" class="">
https://gist.github.com/will-moore/90b26a5d39f8084a22dcc8c4af44fa4c#file-connect_to_remote_omero-py-L28</a><br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Cheers,<br class="">
<br class="">
Will.<br class="">
<br class="">
<br class="">
<br class="">
On 31 Oct 2017, at 22:18, William Moore (Staff) <<a href="mailto:W.Moore@dundee.ac.uk" class="">W.Moore@dundee.ac.uk</a>><br class="">
wrote:<br class="">
<br class="">
Hi Eilidh,<br class="">
<br class="">
Yes, I’m afraid the output from conn.connect() is not very helpful.<br class="">
<br class="">
You can try the underlying omero.client connection like this:<br class="">
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">c = omero.client(host=‘<a href="http://demo.openmicroscopy.org" class="">demo.openmicroscopy.org</a>', port=4064)<br class="">
c.createSession(‘username', ‘password’)<br class="">
</blockquote>
</blockquote>
</blockquote>
<br class="">
which should give you an exception if this fails.<br class="">
<br class="">
e.g.<br class="">
<br class="">
$ omero script launch 87923<br class="">
Using session 009e7c47-5b59-4156-a664-21bd8d84ea68 (root@localhost:4064).<br class="">
Idle timeout: 10 min. Current group: system<br class="">
Enter value for "IDs": 1<br class="">
Job 89261 ready<br class="">
Waiting....<br class="">
Callback received: FINISHED<br class="">
<br class="">
*** start stderr (id=89762)***<br class="">
* Traceback (most recent call last):<br class="">
* File "./script", line 394, in <module><br class="">
* run_script()<br class="">
* File "./script", line 369, in run_script<br class="">
* ok = c.createSession('user', ‘password')<br class="">
* File<br class="">
"/Users/wmoore/Desktop/OMERO/openmicroscopy/dist/lib/python/omero/clients.py",<br class="">
line 569, in createSession<br class="">
* prx = rtr.createSession(username, password, ctx)<br class="">
* File "/usr/local/lib/python2.7/site-packages/Glacier2_Router_ice.py",<br class="">
line 258, in createSession<br class="">
* return _M_Glacier2.Router._op_createSession.invoke(self, ((userId,<br class="">
password), _ctx))<br class="">
* Glacier2.PermissionDeniedException: exception<br class="">
::Glacier2::PermissionDeniedException<br class="">
* {<br class="">
* reason = Password check failed for 'user': []<br class="">
* }<br class="">
* !! 10/31/17 21:30:57.788 error: communicator not destroyed during global<br class="">
destruction.<br class="">
*** end stderr ***<br class="">
<br class="">
<br class="">
*** out parameters ***<br class="">
*** done ***<br class="">
<br class="">
In fact, I’ve found the same issue as you - Connection to a different server<br class="">
doesn’t work from within an OMERO.script<br class="">
but does work locally.<br class="">
Not sure what is going on here but I’ll ask others on the OME team.<br class="">
<br class="">
You shouldn’t be seeing a "ValidationException Can’t find params…” when<br class="">
you run the script.<br class="">
<br class="">
Do you see any errors when uploading the script with $ omero script upload<br class="">
?<br class="">
Or are you simply putting the script into the lib/scripts directory?<br class="">
<br class="">
Cheers,<br class="">
<br class="">
Will.<br class="">
<br class="">
<br class="">
<br class="">
On 31 Oct 2017, at 16:15, Eilidh Troup <<a href="mailto:e.troup@epcc.ed.ac.uk" class="">e.troup@epcc.ed.ac.uk</a>> wrote:<br class="">
<br class="">
Hi,<br class="">
<br class="">
Thank you very much for your thoughtful response to my question on how to<br class="">
transfer data from one OMERO to another.<br class="">
<br class="">
I was able to create a script to take an Image ID as input, along with<br class="">
username & password of central server, and could list the files belonging to<br class="">
the image too.<br class="">
<br class="">
I got stuck when I tried to create a BlitzGateway connection to central<br class="">
server. I’m using the demo OMERO server as my remote OMERO.<br class="">
<br class="">
This is a short script that reproduces the problem. I can connect to a local<br class="">
or remote OMERO.server using the following script when I run it as a python<br class="">
script directly from the command line, but when I run it as an omero script,<br class="">
it will connect to the local server (itself), but not to the remote server:<br class="">
remote_conn.connect() returns False. Is there a way to get more information<br class="">
about why it won’t connect? I’m at a loss as to how to debug this.<br class="">
<br class="">
--------------------------------------------------<br class="">
Connect_to_remote_omero.py<br class="">
<br class="">
from omero.gateway import BlitzGateway<br class="">
<br class="">
# remote<br class="">
username = "etroup"<br class="">
password = “redacted"<br class="">
host = '<a href="http://demo.openmicroscopy.org" class="">demo.openmicroscopy.org</a>'<br class="">
<br class="">
# local<br class="">
# username = "root"<br class="">
# password = “redacted"<br class="">
# host = 'localhost'<br class="">
port = 4064<br class="">
<br class="">
# Connect to remote server.<br class="">
remote_conn = BlitzGateway(username, password, host=host, port=port)<br class="">
connected = remote_conn.connect() # returns true if connected<br class="">
<br class="">
if (connected):<br class="">
print "connected"<br class="">
for p in remote_conn.listProjects():<br class="">
print p.getName()<br class="">
else:<br class="">
print "not connected"<br class="">
<br class="">
remote_conn.close()<br class="">
—————————————————————————<br class="">
<br class="">
Running from python (works for local or remote server):<br class="">
<br class="">
python Connect_to_remote_omero.py<br class="">
<br class="">
connected<br class="">
Project1<br class="">
<br class="">
Running as OMERO script, fails to connect to remote server:<br class="">
<br class="">
bin/omero script list<br class="">
<br class="">
623 | /rdm/Connect_to_remote_omero.py<br class="">
524 | /rdm/Export_to_other_omero.py<br class="">
547 | /rdm/ping.py<br class="">
(20 rows)<br class="">
<br class="">
bin/omero script launch 623<br class="">
<br class="">
Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root@localhost:4064).<br class="">
Idle timeout: 10 min. Current group: system<br class="">
ValidationException: Can't find params for 623!<br class="">
stdout is in file 624:<br class="">
---------------------------------<br class="">
not connected<br class="">
<br class="">
---------------------------------<br class="">
No stderr.<br class="">
<br class="">
Running as OMERO script, does connect to local server:<br class="">
bin/omero script launch 625<br class="">
Using session 5f02548b-6655-490b-981a-85edec1e70c4 (root@localhost:4064).<br class="">
Idle timeout: 10 min. Current group: system<br class="">
ValidationException: Can't find params for 625!<br class="">
stdout is in file 626:<br class="">
---------------------------------<br class="">
connected<br class="">
project2<br class="">
project1<br class="">
<br class="">
————————————————<br class="">
<br class="">
Thanks,<br class="">
Eilidh<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On 17 Oct 2017, at 12:00, <a href="mailto:ome-devel-request@lists.openmicroscopy.org.uk" class="">
ome-devel-request@lists.openmicroscopy.org.uk</a><br class="">
wrote:<br class="">
<br class="">
Send ome-devel mailing list submissions to<br class="">
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" class="">ome-devel@lists.openmicroscopy.org.uk</a><br class="">
<br class="">
To subscribe or unsubscribe via the World Wide Web, visit<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel<br class="">
or, via email, send a message with subject or body 'help' to<br class="">
ome-devel-request@lists.openmicroscopy.org.uk<br class="">
<br class="">
You can reach the person managing the list at<br class="">
ome-devel-owner@lists.openmicroscopy.org.uk<br class="">
<br class="">
When replying, please edit your Subject line so it is more specific<br class="">
than "Re: Contents of ome-devel digest..."<br class="">
<br class="">
<br class="">
Today's Topics:<br class="">
<br class="">
1. Re: Plan for transferring data from one OMERO to another<br class="">
(William Moore (Staff))<br class="">
<br class="">
<br class="">
----------------------------------------------------------------------<br class="">
<br class="">
Message: 1<br class="">
Date: Tue, 17 Oct 2017 10:18:18 +0000<br class="">
From: "William Moore (Staff)" <W.Moore@dundee.ac.uk><br class="">
To: OME External Developer List<br class="">
<ome-devel@lists.openmicroscopy.org.uk><br class="">
Subject: Re: [ome-devel] Plan for transferring data from one OMERO to<br class="">
another<br class="">
Message-ID: <818C04E1-B318-472F-AEF4-16F998AEB015@dundee.ac.uk><br class="">
Content-Type: text/plain; charset="utf-8"<br class="">
<br class="">
Hi,<br class="">
<br class="">
This could be quite a bit of work, so you will need to start small and<br class="">
build-up functionality as you go. Some investigation will be required - Some<br class="">
things will depend on the type of images you?ll be transferring. You?ll also<br class="">
need to have both OMERO servers using the same major version. E.g. 5.3.x.<br class="">
<br class="">
You?ll really need to re-import the images to the central server instead of<br class="">
manually creating new DB entries for all image metadata etc. This can be<br class="">
acheived from the command line, with your OMERO.script invoking ?bin/omero<br class="">
import ?? to the central server.<br class="">
<br class="">
Are these images imported in OMERO 5.0 or later (original files will be in<br class="">
managed repo)?<br class="">
Your script, running on the source OMERO server *may* be able to access the<br class="">
managed repository directly, and call e.g. ?bin/omero import<br class="">
/OMERO/ManagedRepository/will_3/2017-10/05/15-31-34.090/control.lsm?<br class="">
This is certainly the easiest solution.<br class="">
<br class="">
Alternatively, if the managed repo is not available then you can download<br class="">
the original images to the folder where the OMERO.script is running and<br class="">
import the images from there.<br class="">
In the simplest case, a single OMERO Image has a single image file. However,<br class="">
some Images are stored as multiple original files in the managed repo, which<br class="">
means they will all need to be downloaded to the script before they can be<br class="">
imported if you go for that option.<br class="">
<br class="">
In some cases, a single original file (or set of original files) will<br class="">
generate multiple OMERO Images when imported. So you may find that multiple<br class="">
images in your source server come from the same original file(s). In that<br class="">
case, you will have to be careful when you iterate through these images that<br class="">
you don?t import the same original file multiple times.<br class="">
<br class="">
It?s not really possible for OMERO.scripts to be ?interactive?. The easiest<br class="">
option for choosing a target is to have your users log in to the webclient<br class="">
(or Insight) in the central server and pick a Project or Dataset ID to enter<br class="">
into the script dialog in the other server.<br class="">
<br class="">
Traversing the graph of linked objects Dataset -> Image and handling of file<br class="">
Annotations / duplicate tags etc should be pretty straight-forward once you<br class="">
have the re-import of images working.<br class="">
<br class="">
Please put your script in github and commit all changes as you go. This will<br class="">
allow us to see what you have working and to help or contribute much easier.<br class="">
One option if you don?t already have a repo is to fork<br class="">
https://github.com/ome/omero-user-scripts which allows us and the community<br class="">
to find your repo from there.<br class="">
<br class="">
Be careful when handling usernames and passwords in your OMERO.script. Make<br class="">
sure you?re not logging these etc.<br class="">
<br class="">
Initial goals:<br class="">
- Create an script that can take an Image ID as input, along with username &<br class="">
password of central server. Can use<br class="">
https://github.com/openmicroscopy/openmicroscopy/blob/develop/examples/Training/python/Scripting_Service_Example.py<br class="">
- List original files from source Image<br class="">
for f in image.getImportedImageFiles():<br class="">
print f.path, f.name<br class="">
- Create a BlitzGateway connection to central server<br class="">
- You can get a ?client? object using client = conn.c<br class="">
- Command line import using code similar to that in our python test<br class="">
framework (using ?client? from above as in the example)<br class="">
:https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/src/omero/testlib/__init__.py#L277<br class="">
<br class="">
<br class="">
Let us know how this goes,<br class="">
<br class="">
Regards,<br class="">
<br class="">
Will.<br class="">
<br class="">
<br class="">
On 12 Oct 2017, at 12:46, Eilidh Troup<br class="">
<e.troup@epcc.ed.ac.uk<mailto:e.troup@epcc.ed.ac.uk>> wrote:<br class="">
<br class="">
Hi,<br class="">
<br class="">
I?m a software engineer working on data management in biology. Our use case<br class="">
is that lab groups have their own instances of OMERO, and there is also a<br class="">
centrally-hosted university OMERO server which they would like to transfer<br class="">
data to. I?m planning to write a script to allow users to transfer data from<br class="">
one OMERO to another. I wanted to run the plans by this forum to see what<br class="">
you thought of the idea, and whether my planned implementation seems<br class="">
sensible.<br class="">
<br class="">
I plan to write a script (OMERO.scripts) to run on the local instances of<br class="">
OMERO and be used through OMERO.server or OMERO.web. This will let the user<br class="">
choose the image, dataset or project that they want to move, and enter their<br class="">
credentials for the central OMERO server. Then the script will use the Blitz<br class="">
Gateway to read data from the local server, and then to write to the remote<br class="">
server.<br class="">
<br class="">
issues:<br class="">
<br class="">
- How will users choose where to put data? It will have to be either in a<br class="">
default project on the central server, or the script would have to be<br class="">
interactive, where the user enters their credentials, and then the central<br class="">
server is contacted so that a list of projects and datasets belonging to<br class="">
that user can be presented.<br class="">
<br class="">
- There will be a graph of objects linked to any given one, for example, a<br class="">
dataset with images, tags and attachments. Need to traverse this to transfer<br class="">
all associated data.<br class="">
<br class="">
- Some IDs will need to be mapped from one OMERO to the other, for example<br class="">
tags - shouldn?t create a new one, but link to existing tag with same name<br class="">
on the other OMERO.<br class="">
<br class="">
I?m planning to write the code in Python and use OMERO.scripts and the Blitz<br class="">
Gateway. Please let me know what you think.<br class="">
</blockquote>
_______________________________________________<br class="">
ome-devel mailing list<br class="">
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" class="">ome-devel@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>