<div dir="ltr"><div><div>Excellent!</div><div><br></div><div>So if I want to use this with DropBox I simply put --transfer=upload_rm to omero.fs.importArgs in <span style="color:rgb(0,0,0);font-family:'Lucida Grande','Lucida Sans Unicode',Geneva,Verdana,sans-serif;font-size:14px;letter-spacing:-0.159999996423721px;line-height:24px"> </span><tt class="" style="font-family:Consolas,'Deja Vu Sans Mono','Bitstream Vera Sans Mono',monospace;font-size:14px;letter-spacing:0.01em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(221,221,221);color:rgb(51,51,51);line-height:24px;background-image:none;background-repeat:initial"><span class="">etc/grid/templates.xml</span></tt>, right?</div><div><br></div><div>Thanks,</div><div>Yanling</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 1, 2014 at 1:30 PM, Colin Blackburn <span dir="ltr"><<a href="mailto:C.Blackburn@dundee.ac.uk" target="_blank">C.Blackburn@dundee.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Yanling,<br>
<br>
Although we plan to release this in 5.1 it is already available in 5.0.5 as an experimental feature. However, it is not exposed in the help or documentation as it is only intended as an experimental feature for testing purposes. To import a file by first uploading and then deleting the original source files you would need to use this transfer option:<br>
<br>
-—transfer=upload_rm<br>
<br>
Please note that you should use this for test purposes only! Please do not use it in a production system and please ensure that any data you use for testing is backed-up elsewhere.<br>
<br>
Any feedback would be most welcome.<br>
<div class="HOEnZb"><div class="h5"><br>
Cheers,<br>
<br>
Colin<br>
<br>
On 1 Oct 2014, at 17:35, Yanling Liu <<a href="mailto:vrnova@gmail.com">vrnova@gmail.com</a>> wrote:<br>
<br>
> Colin,<br>
><br>
> I'd be more than glad to test this feature.<br>
><br>
> Best,<br>
> Yanling<br>
><br>
> On Wed, Oct 1, 2014 at 12:30 PM, Colin Blackburn <<a href="mailto:C.Blackburn@dundee.ac.uk">C.Blackburn@dundee.ac.uk</a>> wrote:<br>
> Hi Yanling,<br>
><br>
> Again, we are planning to release a transfer mode that uploads, imports and then deletes the original files. It would do this only on a successful upload (all files checked against their sha1s) and a successful import (completes without error). This transfer mode could then be used for DropBox to effectively clean the DropBox directories of files once imported.<br>
><br>
> This feature is intended to be available in the 5.1 release. If you would like to test the this feature prior to that release please do let us know.<br>
><br>
> Cheers,<br>
><br>
> Colin<br>
><br>
> On 1 Oct 2014, at 16:54, Yanling Liu <<a href="mailto:vrnova@gmail.com">vrnova@gmail.com</a>> wrote:<br>
><br>
> > This is fabulous!<br>
> ><br>
> > Then the only thing I need to work on is to setup a cron job to delete old files from the DropBox folder.<br>
> ><br>
> > Can I dream about DropBox facility to be able to auto-delete imported files? I know there's in-place auto-delete but our data is 20TB+ on NFS storage and OME doesn't support NFS.<br>
> ><br>
> > Cheers,<br>
> ><br>
> > Yanling<br>
> ><br>
> > On Wed, Oct 1, 2014 at 11:40 AM, Colin Blackburn <<a href="mailto:C.Blackburn@dundee.ac.uk">C.Blackburn@dundee.ac.uk</a>> wrote:<br>
> > Hi Yanling,<br>
> ><br>
> > We are planning to implement a DropBox facility for selecting the destination container for imports in our next major release, 5.1.<br>
> ><br>
> > Cheers,<br>
> ><br>
> > Colin<br>
> ><br>
> ><br>
> > On 30 Sep 2014, at 19:33, Yanling Liu <<a href="mailto:vrnova@gmail.com">vrnova@gmail.com</a>> wrote:<br>
> ><br>
> > > Tested Dropbox and it works very well.<br>
> > ><br>
> > > Just wondering when will the extended path /OMERO/DropBox/user/group/project/dataset being implemented? This is very very useful to make OME "useful" for imaging facilities where imaging facilities upload images for users instead of users themselves, which is exactly the case we have.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Yanling<br>
> > ><br>
> > > On Fri, Sep 26, 2014 at 9:13 AM, Aleksandra Tarkowska <<a href="mailto:A.Tarkowska@dundee.ac.uk">A.Tarkowska@dundee.ac.uk</a>> wrote:<br>
> > > Hi Yanling<br>
> > ><br>
> > > > Dropbox documentation says NAS is not supported, is a NFS mount considered a network attached share (NAS) to OMERO?<br>
> > ><br>
> > > Dropbox can monitor only local hard drive. But you can set up samba dir locally to drop files and set configuration parameters like: "--transfer=ln_rm" or "--transfer=upload_rm" to delete when they are imported. If you prefer to use NFS you can use In-place import <a href="https://www.openmicroscopy.org/site/support/omero5/sysadmins/in-place-import.html" target="_blank">https://www.openmicroscopy.org/site/support/omero5/sysadmins/in-place-import.html</a>. That should simplify your workflow.<br>
> > ><br>
> > > > Dropbox import new images under "Orphaned Images", then user has to login to organise new images into projects and datasets, is that correct?<br>
> > ><br>
> > > That is correct. Extended path /OMERO/DropBox/user/group/project/dataset/ is on our list to do <a href="http://trac.openmicroscopy.org.uk/ome/ticket/4032" target="_blank">http://trac.openmicroscopy.org.uk/ome/ticket/4032</a> as well as templates <a href="http://trac.openmicroscopy.org.uk/ome/ticket/12039" target="_blank">http://trac.openmicroscopy.org.uk/ome/ticket/12039</a><br>
> > ><br>
> > > You can customise name of "Orphaned Images" if is not suitable.<br>
> > ><br>
> > > Kind regards<br>
> > > Ola<br>
> > ><br>
> > > On Fri, Sep 26, 2014 at 8:38 AM, Aleksandra Tarkowska <<a href="mailto:A.Tarkowska@dundee.ac.uk">A.Tarkowska@dundee.ac.uk</a>> wrote:<br>
> > > Hi Yanling<br>
> > ><br>
> > > As Carne mentioned you can invoke import process via python script.<br>
> > > Similar test is available on<br>
> > > <a href="https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/to" target="_blank">https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/to</a><br>
> > > ols/OmeroPy/test/integration/library.py#L187<br>
> > ><br>
> > > args.extend(["-s", server, "-k", key, "-p", port, "import", "--"])<br>
> > ><br>
> > > Allows you to pass all parameters you normally pass via bin/omero import Š<br>
> > > For more details refer to --help.<br>
> > ><br>
> > > Can I ask why you just simply cannot use dropbox to do that work<br>
> > > automatically for you?<br>
> > > Did you see<br>
> > > <a href="http://www.openmicroscopy.org/site/support/omero5/sysadmins/dropbox.html" target="_blank">http://www.openmicroscopy.org/site/support/omero5/sysadmins/dropbox.html</a><br>
> > ><br>
> > > Kind regards<br>
> > > Ola<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On 26/09/2014 12:48, "Carnë Draug" <<a href="mailto:carandraug%2Bdev@gmail.com">carandraug+dev@gmail.com</a>> wrote:<br>
> > ><br>
> > > >On Thu, 25 Sep 2014 13:31:53 -0400, Yanling Liu <<a href="mailto:vrnova@gmail.com">vrnova@gmail.com</a>> wrote:<br>
> > > >> Hi,<br>
> > > >><br>
> > > >> I was able to use the CLI import (bin/omero import) to import<br>
> > > >> images into OMERO. My question is that if OMERO python binding has<br>
> > > >> similar capability? Can I write a little python script and call OMERO<br>
> > > >> import function to upload some images to OMERO?<br>
> > > >><br>
> > > >> What I mean is if there's python binding to the import function,<br>
> > > >> rather than doing system call in python script to call bin/omero.<br>
> > > ><br>
> > > >You can do something like the following:<br>
> > > ><br>
> > > > import omero<br>
> > > > import omero.cli<br>
> > > ><br>
> > > > client = omero.client (host = server_ip_or_hostname)<br>
> > > > client.createSession (username = "username", password = "password")<br>
> > > ><br>
> > > > cli = omero.cli.CLI()<br>
> > > > cli.loadplugins()<br>
> > > > cli._client = client.createClient(secure = True)<br>
> > > > cli.invoke(["import", path_to_image_file])<br>
> > > ><br>
> > > >Note that you need to create a copy of the client when setting `_client`<br>
> > > >because that session will be closed after the image import.<br>
> > > ><br>
> > > >To find out the ID of the image you imported, you will need to get<br>
> > > >this call STDOUT. I have routinely [1] been doing the following (note<br>
> > > >that this imports a single file but should be easy to adjust for<br>
> > > >multiple files):<br>
> > > ><br>
> > > ><br>
> > > > cli = omero.cli.CLI()<br>
> > > > cli.loadplugins()<br>
> > > > cli._client = self.client.createClient(secure = True)<br>
> > > > cmd = [<br>
> > > > "import",<br>
> > > > "--debug", "ERROR",<br>
> > > > ]<br>
> > > > if self.datasetID:<br>
> > > > cmd.extend(["-d", str(self.datasetID)])<br>
> > > > if self.child_name:<br>
> > > > cmd.extend(["-n", self.child_name])<br>
> > > ><br>
> > > > cid = None<br>
> > > > with tempfile.NamedTemporaryFile(suffix=".stdout") as stdout:<br>
> > > > cmd.extend([<br>
> > > > "---errs", os.devnull,<br>
> > > > "---file", <a href="http://stdout.name" target="_blank">stdout.name</a>,<br>
> > > > ])<br>
> > > > cmd.append(<a href="http://self.fout.name" target="_blank">self.fout.name</a>)<br>
> > > ><br>
> > > > STDERR = sys.stderr<br>
> > > > try:<br>
> > > > with open(os.devnull, 'w') as DEVNULL:<br>
> > > > sys.stderr = DEVNULL<br>
> > > > cli.invoke(cmd)<br>
> > > > finally:<br>
> > > > sys.stderr = STDERR<br>
> > > > ret_code = cli.rv<br>
> > > > if ret_code == 0:<br>
> > > > ## we only need to read one line or something is very wrong<br>
> > > > cid = int(stdout.readline())<br>
> > > > if not cid:<br>
> > > > raise Exception("unable to get exported image ID")<br>
> > > > else:<br>
> > > > raise Exception("failed to import processed image")<br>
> > > ><br>
> > > > self.child = self.conn.getObject("Image", cid)<br>
> > > ><br>
> > > ><br>
> > > >[1]<br>
> > > ><a href="https://github.com/MicronOxford/omero_scripts_processing/blob/40e29af8/ome" target="_blank">https://github.com/MicronOxford/omero_scripts_processing/blob/40e29af8/ome</a><br>
> > > >ro_scripts_processing.py#L293<br>
> > > >_______________________________________________<br>
> > > >ome-devel mailing list<br>
> > > ><a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
> > > ><a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
> > ><br>
> > ><br>
> > > The University of Dundee is a registered Scottish Charity, No: SC015096<br>
> > ><br>
> > > _______________________________________________ ome-devel mailing list <a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a> <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
> > > The University of Dundee is a registered Scottish Charity, No: SC015096<br>
> > ><br>
> > > _______________________________________________<br>
> > > ome-devel mailing list<br>
> > > <a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
> > > <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
> ><br>
> ><br>
> > The University of Dundee is a registered Scottish Charity, No: SC015096<br>
> ><br>
><br>
><br>
> The University of Dundee is a registered Scottish Charity, No: SC015096<br>
><br>
<br>
<br>
The University of Dundee is a registered Scottish Charity, No: SC015096<br>
</div></div></blockquote></div><br></div>