[ome-devel] [ome-users] question re bioformats.jar

[Ilya Goldberg]

Its pretty straight forward if you have the meta data in OME-XML  
already.  The easiest way to do it is write a couple of Java CGIs or  
command-line programs, where they deal with files on the local  
filesystem (i.e. OMEIS's).  Then we can institute a couple of new  
methods in OMEIS nearly identical to those that are used for dealing  
with OME-XML files (IsOMExml and ImportOMEfile).  The corresponding  
BioFormats Perl importer would be trivial to implement, because it  
would be essentially identical to the OME-XML importer and not as  
complicated as the OME-TIFF importer.

The command-line Java program would take a list of paths to local  
image files, and dump XML and raw pixels also to local files and to  
stdout.
The programs don't need to know they're CGIs - those details can be  
easily taken care of by OMEIS - but there are a couple of tricks to  
them.  The path to the file (for opening and reading it) and the  
original file name (for parsing out patterns for fusing multiple  
files together) need to be taken as separate parameters (or, the name  
can be gotten from the path with a local OMEIS command-line call).   
The second trick is that ideally they should be able to group files  
into OME images based on information already in the list of files and  
the files themselves rather than additional user parameters.
A third trick is that instead of combining the pixels and the XML  
into a tiff file, they would dump the XML on stdout and the raw  
pixels into separate (local) files.
Since the easiest interface to implement is just command-line, they  
can be written entirely in Java, and don't need to have any  
additional interaction with any OME servers since they would be  
driven by OMEIS and by the OME back-end and use only local files.

To outline the process, it would go something like this:
User uploads a group of files to OMEIS (using the standard web-based  
mechanism, or any other remote file importing mechanism).
ImportEngine eventually calls BioFormats to arrange the list of files  
into groups.  The result can be one OME Image per group, or the whole  
set of recognizable files in one multi-image OME-XML "chunk".  The  
output of this call would be the input list arranged into one or more  
groups.
ImportEngine then calls BioFormats to import each group.  The result  
of this call is the OME-XML, which is shipped back to the OME back- 
end.  Since this will be executed as a CGI, the Java program would  
just dump the XML to stdout so that it gets received by the remote  
OME back-end, and sucked up into its standard OME-XML importer (this  
is how ImportOMEfile works).  In addition to dumping the XML to  
stdout, the Java program needs to dump the raw pixel data to one or  
more local files.

To glue it together without getting the Java program involved in any  
OME calls, it would need to report the names of the raw pixel files  
it created.  The OMEIS interface to the Java program can read those,  
and move them around to make them proper OMEIS Pixels.  The Perl  
importer would need to square the Pixels IDs with what came in the  
OME-XML.  In this case, the Java program is just a command-line  
utility to get a raw pixel dump and the XML as separate files, which  
would obviously be very useful to people besides acting as the OMEIS  
importer.  It would require more interface code in C and Perl  
though.  All that's needed to eliminate that is to ask OMEIS for the  
local paths to the pixels files instead of making them up.  This  
would let it put proper PixelsIDs directly into the XML.  Getting the  
Pixels ID can be done by (you guessed it) executing OMEIS locally on  
the command line with three calls:  1. NewPixels, getting a temporary  
PixelsID, 2. GetLocalPath to get the local path to the raw pixels  
file corresponding to the PixelsID, 3.  Dumping the raw pixels into  
the provided file path, and 3.  FinishPixels, getting the permanent  
PixelsID which goes into the OME-XML "Pixels" attribute.  After  
writing the pixels to local files, it would dump the XML on stdout to  
be picked up by the back-end importer.  Doing this with command-line  
calls to OMEIS is easy and doesn't cost much of anything since we're  
only communicating IDs and file paths.

On the C side of OMEIS, we would have to fire off these command line  
programs (which I can easily do "in an afternoon"), and on the perl  
side, we would need to essentially duplicate the OME-XML importer  
under a different class name (so it would call the new OMEIS methods  
instead of ImportOMEfile).  This I could probably also do "in an  
afternoon".

This way, you can put any supported file on OMEIS in the regular way,  
and import it using BioFormats as if it was any other file type  
directly supported by a back-end importer.  OMEIS would keep a copy  
of the original file and a cache of the extracted pixels. There's no  
additional disk writes or wire transfers, so the process would be  
efficient, and the whole thing would be centrally managed by OMEIS.   
If there's a way to get an OME-TIFF from a command-line converter  
that accepts a list of file paths, then the code to do all this is  
essentially already there (and if not, such a converter would  
probably be very useful and almost all of its code would be recycled  
to make the CGIs).  All that's needed is a little spackle in Java,  
and no C or Perl.

-Ilya



On Oct 16, 2006, at 1:02 PM, Curtis Rueden wrote:

> Hi Keith,
>
> Sorry for the delay in my response; I was out sick on Friday. I can't
> speak to your question #2, but have some thoughts on #1 and #3.
>
>> 1) we are not able to import any type of images other than TIF, (for
>> example leica format)
>>
>> As OME is installed in /OME and is owned by user OME, we have  
>> tried putting
>> the LOCI bioformats.jar in the OME directory, or in Java/lib and  
>> neither
>> seem to change things.  I don't know how to change the classpath  
>> of the OME
>> user, as it doesn't have a home directory, thus a .cshrc etc... to  
>> add the
>> classpath to its environment...
>>
>> Any hints?
>
> Unfortunately, Bio-Formats is not so integrated with the OME server
> yet. Ultimately, we may include the Bio-Formats JARs with OME, or
> allow them to be dropped into the distribution as you tried, and have
> the perl code automatically attempt to use them. But such
> functionality is not yet in place.
>
> The OME server is perl-based, with several file format importers on
> the backend, written in Perl (modules in
> OME/src/perl2/OME/ImportEngine), including: Bio-Rad PIC, DICOM,
> Deltavision DV, Molecular Dynamics GEL, Zeiss LSM, Metamorph HTD,
> OME-TIFF, Metamorph STK, TIFF, and OME-XML.
>
> Bio-Formats is Java-based, with many file format readers implemented
> (see the Bio-Formats web site:
> http://www.loci.wisc.edu/ome/formats.html). But we have not fully
> integrated Bio-Formats with the OME server.
>
> Currently, the easiest way to get files supported by Bio-Formats into
> OME is to use the OME Plugin for ImageJ
> (http://www.loci.wisc.edu/ome/imagej.html). But even though
> Bio-Formats is capable of producing OME-XML metadata for each file,
> the OME Plugin only supports upload of pixel data to the server. We
> are actively working to improve this support, but there is still much
> to be done.
>
> There is also a command line tool called omeul packaged with VisBio,
> which we are in the middle of refactoring into Bio-Formats directly.
> If you need to batch upload many files to OME, you could try
> downloading VisBio and using the packaged version of omeul; it should
> be able to upload pixel data in any supported file format.
>
>> 3) in regards to accessing the images from a browser on a second  
>> machine,
>> the images are refered to by a url pointing to the image file  
>> which works
>> on the host machine, but not on the second machine.  How do you  
>> set OME to
>> use a relative not absolute link to the images?
>
> I am not certain, but this sounds like an issue with the path to your
> OMEIS (since OMEIS is what provides the thumbnails). The OME database
> needs the full hostname to OMEIS, even if it is on the same machine
> (which it is by default). Did you specify a complete URL to OMEIS when
> it prompted in the installer? You can check as follows (assuming you
> are logged in with the account used to install OME):
>
> % psql ome
> Welcome to psql 8.0.7, the PostgreSQL interactive terminal.
>
> Type:  \copyright for distribution terms
>        \h for help with SQL commands
>        \? for help with psql commands
>        \g or terminate with semicolon to execute query
>        \q to quit
>
> ome=# select * from repositories;
>  attribute_id | module_execution_id |                 image_server_url
>                 | path | is_local
> --------------+--------------------- 
> +--------------------------------------------------+------+----------
>             4 |                   1 |
> http://skyking.microscopy.wisc.edu/cgi-bin/omeis |      | f
> (1 row)
>
> And if necessary, you can update the URL like so:
>
> ome=# update repositories set
> image_server_url='http://skyking.microscopy.wisc.edu/cgi-bin/omeis'
> where module_execution_id=1;
> UPDATE 1
> ome=# \q
> %
>
> -Curtis
>
> On 10/13/06, Keith.Neil at sanofi-aventis.com
> <Keith.Neil at sanofi-aventis.com> wrote:
>> I have 3 questions regarding OME on OSX 10.4 (intel)
>>
>> 1) we are not able to import any type of images other than TIF, (for
>> example leica format)
>>
>> As OME is installed in /OME and is owned by user OME, we have  
>> tried putting
>> the LOCI bioformats.jar in the OME directory, or in Java/lib and  
>> neither
>> seem to change things.  I don't know how to change the classpath  
>> of the OME
>> user, as it doesn't have a home directory, thus a .cshrc etc... to  
>> add the
>> classpath to its environment...
>>
>> Any hints?
>>
>> 2) we would like to have a Samba mount to a network disk in the  
>> OME_images
>> directory we created in the default user home in OSX.  We can do  
>> this on
>> osx by creating an alias to the samba mount.  Problem is this is not
>> recognised by the Apache browser.  I imagine we can create a  
>> symlink to the
>> network mount, but I haven't done this osx before...
>>
>> 3) in regards to accessing the images from a browser on a second  
>> machine,
>> the images are refered to by a url pointing to the image file  
>> which works
>> on the host machine, but not on the second machine.  How do you  
>> set OME to
>> use a relative not absolute link to the images?
>>
>> Thanks for any help.
>>
>> Keith
>>
>>
>> ------------------------------
>> Le présent message ainsi que ses éventuelles pièces jointes est
>> exclusivement destiné au(x) destinataire(s), personnes physiques ou
>> morales, qu'il désigne.
>> Il constitue de ce fait une correspondance à caractère privé et peut
>> contenir des informations confidentielles.
>> Si ce message vous est parvenu par erreur, nous vous remercions  
>> d'en aviser
>> immédiatement l'expéditeur par retour de ce courrier électronique  
>> puis de
>> le détruire, ainsi que ses éventuellement pièces jointes, sans en  
>> conserver
>> de copie.
>>
>> This message, including any attachment, is intended for the use of  
>> the
>> individual or entity to which it is addressed.
>> It is therefore to be considered as a private correspondence which  
>> may
>> contain confidential information.
>> If you are not the intended recipient, please advise the sender  
>> immediately
>> by reply e.mail and delete this message and any attachment thereto  
>> without
>> retaining a copy.
>> ------------------------------
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>