[ome-users] OMERO with iRODS?
Colin Blackburn
c.blackburn at dundee.ac.uk
Fri Mar 25 10:24:17 GMT 2011
Hi Harri,
The issue of faster than acquisition-rate speeds is something we are
looking at. There are limitations imposed by the different monitoring
systems on the different OSes. On Linux some file creation events are
missed under some circumstances:
https://github.com/seb-m/pyinotify/issues#issue/2
This particularly affects copying of entire projects. We mitigate this
but we are currently looking into concurrency issues when trying to
import files appearing rapidly:
https://trac.openmicroscopy.org/ome/ticket/4749
We would, of course, welcome any feedback at all on performance issues
or problems when undertaking these sorts of imports.
Cheers,
Colin
On 21 Mar 2011, at 14:03, Harri Jäälinoja wrote:
>
> Hi Colin,
>
> thanks for your reply! One thing that comes to mind: at the moment
> DropBox works fine, when new images arrive at acquisition speed.
> However, in this proposed setup someone might easily get the idea to
> import an entire project to OMERO via DropBox, so it would be good
> to be prepared for this somehow.
>
> Cheers,
> Harri
>
> Colin Blackburn wrote:
>> Hi Harri,
>> This looks great, thanks for letting us know how thing are going.
>> We'd certainly be interested with any feedback on performance
>> issues when you develop things further.
>> Cheers,
>> Colin
>> On 15 Mar 2011, at 12:17, Harri Jäälinoja wrote:
>>>
>>> Hello all,
>>>
>>> I've made a simple iRODS/OMERO integration prototype. The idea is
>>> simply to make the iRODS storage area readable by DropBox.
>>>
>>> You can find documentation for the prototype here:
>>> http://wiki.helsinki.fi/display/LMUISA
>>>
>>> The OMERO specific part is here:
>>> http://wiki.helsinki.fi/display/LMUISA/iRODS+and+OMERO+integration
>>>
>>> Here I've listed some points why this approach might be interesting:
>>> http://wiki.helsinki.fi/pages/viewpage.action?pageId=67212966
>>>
>>> The prototype is basically functional, but slow, unreliable and
>>> not connected to any serious disk space. I will build the next
>>> version on my new desktop computer when it arrives. Then I will
>>> hopefully be able to have a better idea about performance.
>>>
>>> Cheers,
>>> Harri
>>>
>>>
>>> Curtis Rueden wrote:
>>>> Hi Harri,
>>>> Microservices are written in C. So we couldn't access the
>>>> Bio-Formats C++ bindings from there. Maybe just use a system()
>>>> call
>>>> to run B-F?
>>>> Sure, this is how we integrated Bio-Formats with the original C-
>>>> and Perl-driven OME server. The "OMEIS" process (written in C)
>>>> just invoked a particular Bio-Formats Java class responsible for
>>>> parsing the file in question. This Bio-Formats class then wrote
>>>> the pixel out to a particular binary file designated by OMEIS,
>>>> and dumped the OME-XML metadata to stdout, which the OME server's
>>>> perl layer then consumed. Maybe sounds complicated, but it worked
>>>> like a charm!
>>>> The main thing is to create a Java helper class that behaves the
>>>> way you want. You then make a single system call from C to that
>>>> class.
>>>> Making the conversion to OME-TIFF in iRODS requires some
>>>> coding, the
>>>> operation should be wrapped into a "microservice":
>>>> https://www.irods.org/index.php/Micro-Services
>>>> Since all you need to do is convert to OME-TIFF, system calls
>>>> will be just fine. The main advantage of the C++ bindings is the
>>>> ability to perform very granular operations (at the same level as
>>>> the Java API), which is nice for some applications.
>>>> You might even be able to get away with calling the Bio-Formats
>>>> "bfconvert" tool (loci.formats.tools.ImageConverter) without need
>>>> for your own Java helper class.
>>>> -Curtis
>>>> 2010/10/22 Harri Jäälinoja <harri.jaalinoja at helsinki.fi <mailto:harri.jaalinoja at helsinki.fi
>>>> >>
>>>> Hi Josh and others,
>>>> Does iRODS have facilities for monitoring file access times
>>>> and
>>>> taking actions based on them?
>>>> yes, seems so. The attributes that are stored in the iRODS
>>>> database
>>>> include timestamps for modification and creation, and I assume
>>>> it is
>>>> possible to use those for querying:
>>>> https://www.irods.org/index.php/icatAttributes
>>>> It is possible to create delayed or repeated rules, there is a
>>>> server process that checks every two minutes if there is
>>>> something
>>>> to do. So one could run a job every day to move old files to
>>>> another
>>>> location for example.
>>>> From the OMERO point-of-view, this sounds very
>>>> straight-forward. The only issue I can think of off-hand is
>>>> that
>>>> OMERO.fs would have to handle the read-only status. I've
>>>> added a
>>>> story for this: http://trac.openmicroscopy.org.uk/omero/ticket/3162
>>>> Thanks. This is my setup so far:
>>>> 1. host1 (university general purpose server)
>>>> - iRODS main server, with iCAT database
>>>> 2. host2 (Centos virtual server running in VMWare)
>>>> - OMERO
>>>> -- still need to enable OMERO.fs
>>>> - iRODS server 2, no database.
>>>> -- cannot connect to host1 iRODS, have to figure out why
>>>> Once I get host2 set up, I can start learning iRODS controls with
>>>> some simple tests.
>>>> This is certainly a much larger undertaking, and could
>>>> possibly
>>>> involve making Bio-Formats iRODS-aware (just glancing at
>>>> Jargon
>>>> -- it does provide a RandomAccessFile interface, so the
>>>> necessary modifications may be limited). Perhaps there's a
>>>> way
>>>> using callbacks to do the reverse of dropbox? i.e. OMERO
>>>> writes
>>>> out to a location, and then iRODS takes control of that
>>>> location
>>>> on receiving notification?
>>>> I think it might be possible to accomplish the reverse drop-box
>>>> with
>>>> either FUSE (Linux) or WebDAV (Windows, Linux, Mac):
>>>> https://www.irods.org/index.php/iRODS_FUSE
>>>> https://projects.arcs.org.au/trac/davis
>>>> It seems those will provide an area iRODS is aware of, or a
>>>> drop-box
>>>> into iRODS. I am not sure about this, though. For example, I was
>>>> explained that WebDAV doesn't really make a folder where you can
>>>> write, but rather just a GUI like WinSCP. I don't know about
>>>> FUSE yet.
>>>> That's of course difficult for everyone involved to say. When
>>>> you've got both systems running, perhaps we could start
>>>> with a
>>>> very simple test, like opening a file stored in iRODS via
>>>> Bio-Formats and converting it to OME-TIFF in iRODS. That
>>>> would
>>>> let us know if there are any barriers to moving forward.
>>>> Making the conversion to OME-TIFF in iRODS requires some
>>>> coding, the
>>>> operation should be wrapped into a "microservice":
>>>> https://www.irods.org/index.php/Micro-Services
>>>> Microservices are written in C. So we couldn't access the
>>>> Bio-Formats C++ bindings from there. Maybe just use a system()
>>>> call
>>>> to run B-F?
>>>> I will let you know when I make progress.
>>>> Cheers,
>>>> Harri
>>>> Hoping to hear your comments!
>>>> Best regards,
>>>> Harri
>>>> Cheers,
>>>> ~Josh.
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> <mailto:ome-users at lists.openmicroscopy.org.uk>
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> Colin Blackburn
>> Wellcome Trust Centre for Gene Regulation & Expression
>> College of Life Sciences
>> MSI/WTB/JBC Complex
>> University of Dundee
>> Dow Street
>> Dundee DD1 5EH
>> http://openmicroscopy.org
>
Colin Blackburn
Wellcome Trust Centre for Gene Regulation & Expression
College of Life Sciences
MSI/WTB/JBC Complex
University of Dundee
Dow Street
Dundee DD1 5EH
http://openmicroscopy.org
More information about the ome-users
mailing list