[ome-devel] Importing TIFFs: Bugs 168 & 169

[Ilya Goldberg]

On Jul 16, 2004, at 7:11 AM, Joshua Moore wrote:

> All,
>
> To keep up with the work on the OME::ImportEngine::TIFFreader I've 
> added two bugs:
> Bug 168 for the Z&T and Bug 169 for the file name patterns. I know 
> that these are things on the general todo list, but at our site TIFF 
> import is top priority.

If your TIFF files have a specific (and consistent) naming scheme that 
indicates how they get stitched into a 5D format, the most expedient 
option is to deal with your specific case.  The current TIFF importer 
already looks for a pattern in TIFF names, but its a different pattern 
than yours, apparently.  The specific one it looks for is generated by 
MetaMorph.  So, the quickest way to get what you want is to change 
OME/src/perl2/OME/ImportEngine/TIFFreader.pm.  Better would be to copy 
that file to a new class/package, change what you need, and put this 
new class in front of OME::ImportEngine::TIFFreader in the import list 
(in OME::Install::CoreDatabaseTablesTask::IMPORT_FORMATS).  That way 
your importer will intercept any tiff files before they get to the 
generic one.
The only unfortunate thing about this is that TIFFreader.pm could stand 
a wee bit of cleanup.

>
> Ilya, you mentioned in your email to Zachary the VisBio software. Is 
> this a planned UI for getting TIFFs into OME? How exactly would it 
> work?
VisBio already deals well with image planes sprayed over a collection 
of files.  It asks the user to specify how to arrange them, and has a 
number of very nice heuristics to make intelligent guesses.  The new 
version already knows how to talk to the OME back-end directly, so once 
it sorts out the 5D structure of your files, it can send the 5D object 
to OME.  Curtis Rueden is the man in charge.  The software is in alpha 
mode, but this is known to work.

>
> Also, I've discussed briefly with Chris Allan the possibility of an 
> incremental import. I'll explain the use case here and see what 
> everyone thinks:
>
>    The folks at EMBL want to upload the first few steps in a time 
> series, use the browser to examine those images, and then decide 
> whether or not to cancel the times series. This, as far as I know, 
> requires postponing the fixation of the pixels. Is this doable and 
> what would it look like in the UI?
There are several very good reasons why you can't do this through OME 
(see below).  This is where OMEIS comes to the rescue.  Essentially 
during acquisition, you can stream your pixels to OMEIS and ask it to 
draw them for you (look at the Composite call in the OMEIS docs: 
http://docs.openmicroscopy.org.uk/api/omeis/composite.html#Composite).  
Ideally you would stream the pixels right out of hardware (see the Set* 
methods for OMEIS), or much less ideally out of intermediate files.  
OMEIS right now requires a finalize call before any Composite calls, 
but that's probably a fair change to make and wouldn't be very 
difficult.  Note that the Composite call is simply a URL.  Just type it 
into your browser, and presto - you get an image.  Once you're 
satisfied that you want to keep the image, you just import into OME - 
except you don't need to download or upload anything.  You just tell 
OME where the image is in OMEIS.  This is how all our importers work.  
For a specific case of a remote importer that does such things in Java, 
look at the VisBio code.

To all the hardware manufacturers out there:  What do y'all think about 
putting OMEIS hooks into acquisition hardware?

And now, for some philosophy. One of the fundamental assumptions in OME 
is that readable objects are "finalized".  If you can read it today, we 
guarantee that if you read it tomorrow you will get the same exact 
thing.  We couldn't very well call this an image repository without 
making this guarantee, could we?  There is one exception to this in 
that attributes with null MEXes are considered mutable.  Since they 
have null MEXes though, they're not connected to a data history and so 
they would not be reported in questions like "What are all the known 
attributes of object X".  You have to ask for them specifically.  Many 
of us consider this an ugly hack and clamor for this "feature"'s 
deprecation.
The practical reason why this won't work through OME is that imports 
are transactionally isolated (as is every other module execution).  
Other than the process that connects to the DB to do the actual import, 
the import is completely invisible to any other process until the 
original one commits its transaction.  This happens at the DB level and 
literally there is no hint of the image in the database until the 
import is complete.  I don't want to go into why transactions are 
important other than to say that 30 years of database design concurs.
So the only way to do it through OME is to make a mutable Pixels object 
that you can add pixels to at a later time.  That's possible, but its 
kind of a hack.  Its also likely to have consequences throughout the 
code base, and could even be a major hack.  If the information 
management purists win out, this may not even be possible in the 
future.

Another option:  An OME Image can have several sets of pixels (Pixels 
are just attributes - remember?).  In principle, you can add pixels to 
an Image a plane at a time at your leisure.  At some later point, you 
can have a module execute that unites all those planes into a single 5D 
set of pixels.  The Pixels associated with an image don't need to have 
the same dimensionality (by design, but we haven't tested this).  Other 
than the plane-merge module itself, this is all within the 
functionality of the software and consistent with the existing data 
model.  The drawback is that Pixels objects have 5D granularity in all 
cases, so there is no practical way to recover the extra storage space. 
  We don't even have a deletion mechanism right now, but obviously this 
is on our plate.  This plane-merge module might even be generally 
useful (i.e. your case above).

-Ilya



>
> One last question more to the java crew, what are the plans for 
> importing from the java client?
>
> Best wishes & good importing,
>
> Josh Moore
> DKFZ-Heidelberg
> Germany
>
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>