[ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Tue Oct 28 17:27:30 GMT 2014

Thanks, Sebastien.

I found that this made some improvement:  The time is now 4.66 sec (vs. previous 70 sec and alternate 1.26 sec), and the filesize is slightly improved (4.5 MB, vs previous 5.3 MB and alternate 1.65 MB).

I've enclosed the modified code, which includes my interpretation of your suggested changes (I created the new IFD only once, and then set owriter=writer.getWriter(fname)).  Please let me know if this is not quite what you meant.

Best,
Gary

From: Sebastien Besson [mailto:s.besson at dundee.ac.uk]
Sent: Tuesday, October 28, 2014 12:25 PM
To: Yellen, Gary
Cc: OME-devel OME
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Gary,

thanks for sending the code and some test data. After some debugging and with Melissa's input,
the major difference between the whole Bio-Formats version and thelibTiff version is the following
option:

tagstruct.RowsPerStrip = szY;

If writing using Bio-Formats, a workaround is to construct a loci.formats.tiff.IFD object with the
RowsPerStrip equal to the desired value, i.e.

ifd = loci.formats.tiff.IFD();
ifd.putIFDValue(ifd.ROWS_PER_STRIP, szY);

This  object can then be passed to TiffWriter.saveBytes():

1- either if you initialize an ImageWriter() using

  writer.getWriter(fname).saveBytes(index, getBytes(plane(:)), ifd);

2- or if you directly initialize an OMETiffWriter() using

  writer.saveBytes(index, getBytes(plane(:)), ifd);

Locally adding this option brought your Bio-Formats code to similar levels of performance
in terms of file size and writing time.

Could you try this and let us know?
Sebastien

On 28 Oct 2014, at 12:33, Yellen, Gary <gary_yellen at hms.harvard.edu<mailto:gary_yellen at hms.harvard.edu>> wrote:

Sorry, those are MB, not GB!

From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk] On Behalf Of Yellen, Gary
Sent: Tuesday, October 28, 2014 8:27 AM
To: Besson, Sebastien (forwarding)
Cc: OME-devel OME
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Sebastien -
I've created some test data and code and uploaded the zip as gyBFWriting.zip.
If you unzip and set the MATLAB current directory to this folder, you should be able to run gyTestScript.  It will delete the product files (.ome.tif's) and rewrite them using the two different codes.  On my machine, the timing is about 1.3 sec for the hybrid code, and 70 sec for the all Bio-Formats version.  You can also see in the .zip that the hybrid-written file is 1.6 GB and the all-Bio-formats-written file is 5.3 GB.

Hopefully I've removed all external refs and it will run smoothly.

Also, you can set verbose=1; in each routine to see the progress of the writing.

Please let me know how it goes.

Best,
Gary

From: Sebastien Besson [mailto:s.besson at dundee.ac.uk]
Sent: Tuesday, October 28, 2014 6:42 AM
To: Yellen, Gary
Cc: Patterson Andrew; OME-devel OME
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Gary,

although setting compression on certainly has an impact on writing speed, both the writing duration
and the file size excess look unexpected. Would you have by any chance some test data and the code
that uses Bio-Formats methods alongside with your hybrid solution to help us reproduce your issue?

You should be able to upload everything as a zip file at http://qa.openmicroscopy.org.uk/qa/upload/.

Best regards,
Sebastien

On 27 Oct 2014, at 15:59, Yellen, Gary <gary_yellen at hms.harvard.edu<mailto:gary_yellen at hms.harvard.edu>> wrote:

Hi Andrew - I do realize that this is a ridiculously long Experiment Description string.  It contains unparsed metadata strings from MATLAB - I'll take care of the parsing if and when I can establish the practicality of my storing acquired data in the OME-TIFF format.

I mentioned it in the context of the excessive file size being produced by the LOCI calls, simply because eliminating it was an easy way to rule out the wholesale duplication of all the XML metadata.
Thanks - Gary

On Oct 27, 2014, at 11:54 AM, Andrew Patterson <ajpatterson at lifesci.dundee.ac.uk<mailto:ajpatterson at lifesci.dundee.ac.uk>> wrote:

Hello Gary,

I am coming to this discussion late and you may only be talking about dummy test data, but I was wonder why your Experiment Description would be that large?

This is normally used for a few lines of text that can be displayed to the user, perhaps in a tool-tip, to help then tell similarly named experiments apart.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#Experiment_Description

If you have a quantity of data you wish to store the better location might be an annotation on a Dataset, Project, Experimenter, or Image.
CommentAnnotation can store a long string.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#CommentAnnotation
Or XMLAnnotation more structured data.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#XMLAnnotation

Hope this helps,

Andrew

On 27 Oct 2014, at 15:20, "Yellen, Gary" <gary_yellen at hms.harvard.edu<mailto:gary_yellen at hms.harvard.edu>> wrote:

Ian - I did give this a try and it did not fix either the file size or the slow speed problem.
Also, I reduced the size of the metadata (at least, of the ExperimentDescription string) by 5 KB, and the file size was reduced only by this amount.  So at least this particular metadata item is not being duplicated.
Best,
Gary

From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk] On Behalf Of Munro, Ian
Sent: Monday, October 27, 2014 9:02 AM
Cc: ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Again Gary

Good to know that you've found a solution. However,  I find it puzzling that the files are different sizes.
We had a problem where bio-formats was writing multiple copies of the xml when called from Matlab.
The work-around was to add the following line:

java.lang.System.setProperty('javax.xml.transform.TransformerFactory', 'com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl');

to the Matlab code as in the example that I sent.
Perhaps this might help.

Regards

Ian
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

Dr Sébastien Besson
Open Microscopy Environment / Harvard Medical School
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096

Dr Sébastien Besson
Open Microscopy Environment / Harvard Medical School
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20141028/4d1175bd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gyTestAllBF.m
Type: application/octet-stream
Size: 7873 bytes
Desc: gyTestAllBF.m
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20141028/4d1175bd/attachment-0001.obj>