[ome-devel] Plan for transferring data from one OMERO to another

[Andrea Falconi]

Oh, perhaps I should've mentioned that Smuggler doesn't store usernames and
passwords anywhere, they're kept in memory just for the time it takes to
create an OMERO session and then discarded. also, you can configure
Smuggler to use HTTPS to protect your passwords from prying eyes ;)


On 15 December 2017 at 20:13, Andrea Falconi <andrea.falconi at gmail.com>
wrote:

> Hi guys,
>
> If running an additional service is an option, then you could try using
> Smuggler:
>
>    - http://c0c0n3.github.io/ome-smuggler/
>
> It comes with a JSON API that lets you create an OMERO session:
>
> * http://c0c0n3.github.io/ome-smuggler/javadoc/server/ome/
> smuggler/web/omero/CreateSessionRequest.html
>
> You can use the same Smuggler instance to create sessions with different
> OMERO servers and optionally specify a session keep-alive duration. If you
> specify a keep-alive of D minutes/hours/etc. Smuggler will take care of
> pinging OMERO to keep that session alive for D minutes/hrs/etc. This may
> come in handy if you want to run a script multiple times with the same
> session key without having to worry about OMERO expiring the session
> between runs.
>
> You can find an example of creating a session with curl in this directory:
>
> * https://github.com/c0c0n3/omero-ms-queue/tree/master/
> components/server/src/test/scripts/http-omero
>
> I'll gladly provide more details if you think this is a good fit for your
> scenario :)
>
> Cheers
>
> /a
>
>
>
>
> On 15 December 2017 at 17:45, William Moore (Staff) <W.Moore at dundee.ac.uk>
> wrote:
>
>> Hi Eilidh,
>>
>>   We are still discussing this, but one option we are exploring is to use
>> an existing login via the JSON api in webclient that can give you a
>> session ID.
>>
>> If you could go to your remote server’s /webclient/login/ page and open
>> the browser dev-tools (right-click on the page and -> Inspect Element)
>> Then open the Console.
>> Then paste in the following command, editing the username and password to
>> yours… (this is easier in Chrome than Firefox, since Firefox you have to
>> type ‘allow pasting’ for security).
>>
>> $.post('/api/v0/login/', {'username': ‘your_username',
>> 'password’:'secret', 'server': 1}, data => console.log(data.eventContext.
>> sessionUuid));
>>
>> This should give you a session ID.
>> Then you can paste this into your script UI (in Insight) and use the code
>> sample in my previous e-mail to join a session from the script.
>>
>> NB: it’s important to close this session as soon as you’ve finished with
>> it, in case someone got hold of the session ID.
>>
>> Give this a try and see if it works for you.
>>
>> There’s maybe other tests, checks and gotcha’s we need to consider, so we
>> need more discussion before
>> this workflow gets the green light for production use.
>>
>> But it would be good to know if this looks viable from your point of view?
>>
>>  Cheers,
>>
>>    Will.
>>
>>
>>
>> On 15 Dec 2017, at 15:21, William Moore (Staff) <wmoore at dundee.ac.uk>
>> wrote:
>>
>> Hi Eilidh,
>>
>>  Unfortunately, following more discussion in the wider team, we can’t
>> recommend the entry of passwords into script parameters.
>> You can see some of the ongoing discussion at https://github.com/openmicr
>> oscopy/openmicroscopy/pull/5598 .
>>
>> Fundamentally, the script service wasn’t designed to handle that level of
>> security.
>> Everywhere we handle passwords in OMERO, we need to use a secure
>> (encrypted) connection and ensure nothing is
>> logged, printed or stored anywhere.
>>
>> We know that some DEBUG logging levels will log script parameters in
>> OMERO.web (see above) and that OMERO.insight’s
>> connection to the server is not encrypted (except during login) and we
>> just can’t guarantee that there aren’t other security loop-holes.
>>
>> The support of a “Password” field in the script UI would give users the
>> false impression that
>> their passwords are guaranteed to be safe, which we’d rather not do.
>>
>> One suggestion was to use some other key to look-up passwords from some
>> protected resource that was only available
>> to your script to read. Although I’m not sure whether this is viable or
>> will scale for your use case?
>> This is included in the ongoing discussion on the PR above.
>>
>> The safest way to do what you want is to use a Session key to “join” a
>> current session on the server, although the webclient and Insight
>> don’t currently let users get hold of their session key.
>>
>> If your users could use the command line to login to the remote server
>> then you’d do something like this….
>>
>> $ bin/omero login
>> Previous session expired for root on eel.openmicroscopy.org:4064
>> Server: [eel.openmicroscopy.org:4064]
>> Username: [root]user-3
>> Password:
>>
>> $ bin/omero sessions key
>> b2c87a07-6728-4a0b-a9a0-1444cdd35ad7
>>
>> They would then enter this key as a script parameter and you’d get a
>> connection like this:
>>
>> >>> from omero.gateway import BlitzGateway
>> >>> conn=BlitzGateway(host='eel.openmicroscopy.org', port=4064)
>> >>> conn.connect('b2c87a07-6728-4a0b-a9a0-1444cdd35ad7’)
>> True
>>
>>
>> We’ll look at supporting session key in webclient but this won’t be in
>> the next release I’m afraid.
>> So hopefully there’s some workaround you can use in the meantime.
>> Please let us know if there’s any other way we can help with this...
>>
>> Regards,
>>
>>
>>   Will.
>>
>>
>>
>> On 11 Dec 2017, at 14:33, Eilidh Troup <e.troup at epcc.ed.ac.uk> wrote:
>>
>> Thanks very much Will.
>>
>> On 11 Dec 2017, at 13:37, William Moore (Staff) <W.Moore at dundee.ac.uk>
>> wrote:
>>
>> Hi Eilidh,
>>
>> I’ve opened at PR for this in web: https://github.com/openmi
>> croscopy/openmicroscopy/pull/5598 and Jean-Marie tells me
>> that it should also be doable in Insight, so I’ve created a card at
>> https://trello.com/c/C3EqebXJ/61-script-parameter-password
>> These are likely to be included in the next release of OMERO.
>>
>> Apart from the visibility of the password in the script UI, it’s
>> important to be aware of other places that the password
>> could be exposed to others:
>>
>>  - The connection between Insight and OMERO is not encrypted by default
>> (except for the login). We’ll look at using encrypted connection when
>> submitting script parameters containing passwords, but in the meantime
>> there is the potential for this data to be read.
>>  - Make sure you’re not logging or printing the script params (as we do
>> in many of our example scripts). Currently, Blitz.log and Processor.log
>> don’t log script params but make sure you don’t log them yourself.
>>  - If you’re using the webclient, you should use https to encrypt the
>> connection between the browser and OMERO.web.
>>  - Currently in webcilent, errors in the handling of script submission
>> are not handled well (display error on page instead of displaying feedback
>> error submission page). This shouldn’t lead to any exposure of passwords,
>> but be careful not to submit feedback etc that contains the password and if
>> you think you might have done, then be sure to change your password.
>>
>>
>> Regards,
>>
>>   Will.
>>
>>
>> On 8 Dec 2017, at 11:55, Eilidh Troup <e.troup at epcc.ed.ac.uk> wrote:
>>
>> Hi Josh,
>>
>> Thanks very much. This is a screenshot from running the script in
>> OMERO.insight. I’d like the text entered in the password to be hidden, like
>> this *****.
>>
>> Eilidh
>>
>> <PastedGraphic-2.png>
>>
>> On 7 Dec 2017, at 14:39, Josh Moore <josh at glencoesoftware.com> wrote:
>>
>> Hi Eilidh,
>>
>> On Tue, Dec 5, 2017 at 10:51 PM, Eilidh Troup <e.troup at epcc.ed.ac.uk>
>> wrote:
>>
>> Hi Josh,
>>
>> Thanks for your help. I can now transfer an image from one OMERO to
>> another
>> : )
>>
>>
>> Congratulations.
>>
>>
>> I’ve put the latest version of the script at
>> https://github.com/SynthSys/omero-user-scripts/blob/master/
>> Export_to_other_omero.py
>>
>> I’ve got more questions though:
>> How can I find the managed repository directory? (Something like
>> omero.managed.dir, Default: ${omero.data.dir}/ManagedRepository ?)
>>
>>
>> In [2]: client.sf.getConfigService().getConfigValue("omero.managed.dir")
>>
>>
>> How can I hide the password text entry?
>>
>>
>> Where are you seeing password text entry?
>>
>>
>> How can I get the newly created image id? It is printed to stdout as
>> "Image:920" for example, but I don't know how to get hold of that number
>> in
>> my script.
>>
>>
>> You can use something like:
>>
>>   import sys
>>   old_stdout = sys.stdout
>>   sys.stdout = open(temporary_file_name, "w")
>>   try:
>>       # do import
>>   finally:
>>       sys.stdout = old_stdout
>>
>>
>> Also, the docker script you sent didn’t quite work (don’t worry about it
>> though, I got what I needed from your example) - I got this error when I
>> ran
>> the python script:
>>
>> bash-4.2$ python /gist/export_to_remote_omero.py
>> Traceback (most recent call last):
>>
>> ...
>>
>> ::Glacier2::PermissionDeniedException
>> {
>>    reason = internal server error
>> }
>>
>>
>> Alright. If you'd like to investigate further, we'll need the server logs.
>>
>>
>> Thanks,
>> Eilidh
>>
>>
>> All the best,
>> ~Josh
>>
>>
>> On 10 Nov 2017, at 09:22, Josh Moore <josh at glencoesoftware.com> wrote:
>>
>> Hi Elidh,
>>
>> On Thu, Nov 9, 2017 at 5:07 PM, Eilidh Troup <e.troup at epcc.ed.ac.uk>
>> wrote:
>>
>> Hi,
>>
>> I’ve run into a similar problem as last time.
>>
>>
>> Turns out, there's a similar solution.
>>
>> I’ve written a script that
>> will upload an image to a remote OMERO server, but only from the python
>> command line, and not from within a script running in a local OMERO. As
>> before, the script will work from the command line or from with OMERO when
>> transferring the image to the local OMERO server. When I try to import an
>> image on the remote server, I get the following error:
>>
>> ...
>>
>>
>> The code, and full error output are below.
>> I’ve also put it into a different script on gist that demonstrates the
>> error
>> by running it from within OMERO as a script.
>> https://gist.github.com/eilidh-t/e763dab7d4f3515cc85b0b6ddfb8d9f4
>>
>>
>> I've update your script:
>> https://gist.github.com/joshmoore/0c2e5319d8d5ae7f79ee82e3e68b721b
>>
>> The most important change is:
>>
>>   del os.environ["ICE_CONFIG"] before importing
>>
>> which does the equivalent of `Ice.Config=/dev/null` but for the import
>> process. Likely, that should be done closer to the top.
>>
>> Other changes include:
>>
>> * use ome.cli.CLI rather than calling subprocess (simplicity)
>> * only register the script if it doesn't exist (less test mess)
>> * pass a key to not store the root's password (security)
>> * adding a warning about the security whole that this script could
>> cause (security)
>>
>> My general workflow was:
>>
>> * docker-compose up -d
>> * docker-compose exec omero1 bash
>> * cd /opt/omero/server/OMERO.server
>> * export PYTHONPATH=lib/python
>> * python /gist/export_to_remote_omero.py
>>
>> If I needed to change the script, I deleted the uploaded file from
>> lib/script/export_to_remote_omero.py and re-ran.
>>
>> Thanks for your help with this,
>> Eilidh
>>
>>
>>
>> Let us know if that works for you.
>> ~J
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20171215/220f0b6d/attachment.html>