<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Gary,
<div><br>
</div>
<div>thanks for sending the code and some test data. After some debugging and with Melissa's input,</div>
<div>the major difference between the whole Bio-Formats version and thelibTiff version is the following</div>
<div>option:</div>
<div><br>
</div>
<div>tagstruct.RowsPerStrip = szY;</div>
<div><br>
</div>
<div>If writing using Bio-Formats, a workaround is to construct a loci.formats.tiff.IFD object with the</div>
<div>RowsPerStrip equal to the desired value, i.e.</div>
<div><br>
</div>
<div>
<div>ifd = loci.formats.tiff.IFD();</div>
<div>ifd.putIFDValue(ifd.ROWS_PER_STRIP, szY);</div>
</div>
<div><br>
</div>
<div>This object can then be passed to TiffWriter.saveBytes():</div>
<div><br>
</div>
<div>1- either if you initialize an ImageWriter() using</div>
<div><br>
</div>
<div> writer.getWriter(fname).saveBytes(index, getBytes(plane(:)), ifd);</div>
<div><br>
</div>
<div>2- or if you directly initialize an OMETiffWriter() using</div>
<div><br>
</div>
<div> writer.saveBytes(index, getBytes(plane(:)), ifd);</div>
<div><br>
</div>
<div>Locally adding this option brought your Bio-Formats code to similar levels of performance</div>
<div>in terms of file size and writing time. </div>
<div><br>
</div>
<div>Could you try this and let us know?</div>
<div>Sebastien</div>
<div><br>
</div>
<div>
<div>
<div>On 28 Oct 2014, at 12:33, Yellen, Gary <<a href="mailto:gary_yellen@hms.harvard.edu">gary_yellen@hms.harvard.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Sorry, those are MB, not GB!<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;"><span class="Apple-converted-space"> </span>ome-devel [<a href="mailto:ome-devel-bounces@lists.openmicroscopy.org.uk">mailto:ome-devel-bounces@lists.openmicroscopy.org.uk</a>]<span class="Apple-converted-space"> </span><b>On
Behalf Of<span class="Apple-converted-space"> </span></b>Yellen, Gary<br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Tuesday, October 28, 2014 8:27 AM<br>
<b>To:</b><span class="Apple-converted-space"> </span>Besson, Sebastien (forwarding)<br>
<b>Cc:</b><span class="Apple-converted-space"> </span>OME-devel OME<br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation<o:p></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Hi Sebastien –<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">I’ve created some test data and code and uploaded the zip as gyBFWriting.zip.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">If you unzip and set the MATLAB current directory to this folder, you should be able to run gyTestScript. It will delete the product files (.ome.tif’s) and rewrite them
using the two different codes. On my machine, the timing is about 1.3 sec for the hybrid code, and 70 sec for the all Bio-Formats version. You can also see in the .zip that the hybrid-written file is 1.6 GB and the all-Bio-formats-written file is 5.3 GB.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Hopefully I’ve removed all external refs and it will run smoothly.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Also, you can set verbose=1; in each routine to see the progress of the writing.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Please let me know how it goes.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Best,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Gary<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"> </span></div>
<div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;"><span class="Apple-converted-space"> </span>Sebastien Besson [<a href="mailto:s.besson@dundee.ac.uk" style="color: purple; text-decoration: underline;">mailto:s.besson@dundee.ac.uk</a>]<span class="Apple-converted-space"> </span><br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Tuesday, October 28, 2014 6:42 AM<br>
<b>To:</b><span class="Apple-converted-space"> </span>Yellen, Gary<br>
<b>Cc:</b><span class="Apple-converted-space"> </span>Patterson Andrew; OME-devel OME<br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation<o:p></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Hi Gary,<br>
<br>
although setting compression on certainly has an impact on writing speed, both the writing duration<br>
and the file size excess look unexpected. Would you have by any chance some test data and the code<br>
that uses Bio-Formats methods alongside with your hybrid solution to help us reproduce your issue?<br>
<br>
You should be able to upload everything as a zip file at <a href="http://qa.openmicroscopy.org.uk/qa/upload/" style="color: purple; text-decoration: underline;">http://qa.openmicroscopy.org.uk/qa/upload/</a>.<br>
<br>
Best regards,<br>
Sebastien<o:p></o:p></div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
On 27 Oct 2014, at 15:59, Yellen, Gary <<a href="mailto:gary_yellen@hms.harvard.edu" style="color: purple; text-decoration: underline;">gary_yellen@hms.harvard.edu</a>> wrote:<o:p></o:p></div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></p>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Hi Andrew - I do realize that this is a ridiculously long Experiment Description string. It contains unparsed metadata strings from MATLAB - I'll take care of the parsing if and when I can establish the practicality of my storing acquired data in the OME-TIFF
format.<br>
<br>
I mentioned it in the context of the excessive file size being produced by the LOCI calls, simply because eliminating it was an easy way to rule out the wholesale duplication of all the XML metadata.<span class="Apple-converted-space"> </span><br>
Thanks - Gary<span class="Apple-converted-space"> </span><br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
On Oct 27, 2014, at 11:54 AM, Andrew Patterson <<a href="mailto:ajpatterson@lifesci.dundee.ac.uk" style="color: purple; text-decoration: underline;">ajpatterson@lifesci.dundee.ac.uk</a>> wrote:<br>
<br>
Hello Gary,<br>
<br>
I am coming to this discussion late and you may only be talking about dummy test data, but I was wonder why your Experiment Description would be that large?<br>
<br>
This is normally used for a few lines of text that can be displayed to the user, perhaps in a tool-tip, to help then tell similarly named experiments apart.<br>
<a href="http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#Experiment_Description" style="color: purple; text-decoration: underline;">http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#Experiment_Description</a><br>
<br>
If you have a quantity of data you wish to store the better location might be an annotation on a Dataset, Project, Experimenter, or Image.<br>
CommentAnnotation can store a long string.<br>
<a href="http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#CommentAnnotation" style="color: purple; text-decoration: underline;">http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#CommentAnnotation</a><br>
Or XMLAnnotation more structured data.<br>
<a href="http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#XMLAnnotation" style="color: purple; text-decoration: underline;">http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#XMLAnnotation</a><br>
<br>
Hope this helps,<br>
<br>
Andrew<br>
<br>
<br>
<o:p></o:p></p>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
On 27 Oct 2014, at 15:20, "Yellen, Gary" <<a href="mailto:gary_yellen@hms.harvard.edu" style="color: purple; text-decoration: underline;">gary_yellen@hms.harvard.edu</a>> wrote:<br>
<br>
Ian – I did give this a try and it did not fix either the file size or the slow speed problem.<span class="Apple-converted-space"> </span><br>
Also, I reduced the size of the metadata (at least, of the ExperimentDescription string) by 5 KB, and the file size was reduced only by this amount. So at least this particular metadata item is not being duplicated.<br>
Best,<br>
Gary<br>
<br>
From: ome-devel [<a href="mailto:ome-devel-bounces@lists.openmicroscopy.org.uk" style="color: purple; text-decoration: underline;">mailto:ome-devel-bounces@lists.openmicroscopy.org.uk</a>] On Behalf Of Munro, Ian<br>
Sent: Monday, October 27, 2014 9:02 AM<br>
Cc:<span class="Apple-converted-space"> </span><a href="mailto:ome-devel@lists.openmicroscopy.org.uk" style="color: purple; text-decoration: underline;">ome-devel@lists.openmicroscopy.org.uk</a><br>
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation<br>
<br>
Hi Again Gary<span class="Apple-converted-space"> </span><br>
<br>
Good to know that you’ve found a solution. However, I find it puzzling that the files are different sizes.<br>
We had a problem where bio-formats was writing multiple copies of the xml when called from Matlab.<br>
The work-around was to add the following line:<br>
<br>
java.lang.System.setProperty('javax.xml.transform.TransformerFactory', 'com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl’);<br>
<br>
to the Matlab code as in the example that I sent.<br>
Perhaps this might help.<br>
<br>
Regards<br>
<br>
Ian<span class="Apple-converted-space"> </span><br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" style="color: purple; text-decoration: underline;">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" style="color: purple; text-decoration: underline;">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><o:p></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" style="color: purple; text-decoration: underline;">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" style="color: purple; text-decoration: underline;">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><o:p></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></p>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-family: Helvetica, sans-serif;">Dr Sébastien Besson<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-family: Helvetica, sans-serif;">Open Microscopy Environment / Harvard Medical School<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-family: Helvetica, sans-serif;">Wellcome Trust Centre for Gene Regulation and Expression,<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-family: Helvetica, sans-serif;">College of Life Sciences, University of Dundee, Dow Street,<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-family: Helvetica, sans-serif;">Dundee DD1 5EH Scotland UK Tel: (01382) 386364<o:p></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<br>
<br>
<span style="font-size: 10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span></div>
</div>
</div>
</blockquote>
</div>
<br>
<div><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>Dr Sébastien Besson</div>
<div>Open Microscopy Environment / Harvard Medical School</div>
<div>Wellcome Trust Centre for Gene Regulation and Expression,</div>
<div>College of Life Sciences, University of Dundee, Dow Street,</div>
<div>Dundee DD1 5EH Scotland UK Tel: (01382) 386364</div>
</div>
</span></div>
</span></span></div>
<br>
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>