[ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

[Yellen, Gary]

Hi Sebastien -
I've created some test data and code and uploaded the zip as gyBFWriting.zip.
If you unzip and set the MATLAB current directory to this folder, you should be able to run gyTestScript.  It will delete the product files (.ome.tif's) and rewrite them using the two different codes.  On my machine, the timing is about 1.3 sec for the hybrid code, and 70 sec for the all Bio-Formats version.  You can also see in the .zip that the hybrid-written file is 1.6 GB and the all-Bio-formats-written file is 5.3 GB.

Hopefully I've removed all external refs and it will run smoothly.

Also, you can set verbose=1; in each routine to see the progress of the writing.

Please let me know how it goes.

Best,
Gary



From: Sebastien Besson [mailto:s.besson at dundee.ac.uk]
Sent: Tuesday, October 28, 2014 6:42 AM
To: Yellen, Gary
Cc: Patterson Andrew; OME-devel OME
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Gary,

although setting compression on certainly has an impact on writing speed, both the writing duration
and the file size excess look unexpected. Would you have by any chance some test data and the code
that uses Bio-Formats methods alongside with your hybrid solution to help us reproduce your issue?

You should be able to upload everything as a zip file at http://qa.openmicroscopy.org.uk/qa/upload/.

Best regards,
Sebastien

On 27 Oct 2014, at 15:59, Yellen, Gary <gary_yellen at hms.harvard.edu<mailto:gary_yellen at hms.harvard.edu>> wrote:


Hi Andrew - I do realize that this is a ridiculously long Experiment Description string.  It contains unparsed metadata strings from MATLAB - I'll take care of the parsing if and when I can establish the practicality of my storing acquired data in the OME-TIFF format.

I mentioned it in the context of the excessive file size being produced by the LOCI calls, simply because eliminating it was an easy way to rule out the wholesale duplication of all the XML metadata.
Thanks - Gary



On Oct 27, 2014, at 11:54 AM, Andrew Patterson <ajpatterson at lifesci.dundee.ac.uk<mailto:ajpatterson at lifesci.dundee.ac.uk>> wrote:

Hello Gary,

I am coming to this discussion late and you may only be talking about dummy test data, but I was wonder why your Experiment Description would be that large?

This is normally used for a few lines of text that can be displayed to the user, perhaps in a tool-tip, to help then tell similarly named experiments apart.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#Experiment_Description

If you have a quantity of data you wish to store the better location might be an annotation on a Dataset, Project, Experimenter, or Image.
CommentAnnotation can store a long string.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#CommentAnnotation
Or XMLAnnotation more structured data.
http://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/SA_xsd.html#XMLAnnotation

Hope this helps,

Andrew



On 27 Oct 2014, at 15:20, "Yellen, Gary" <gary_yellen at hms.harvard.edu<mailto:gary_yellen at hms.harvard.edu>> wrote:

Ian - I did give this a try and it did not fix either the file size or the slow speed problem.
Also, I reduced the size of the metadata (at least, of the ExperimentDescription string) by 5 KB, and the file size was reduced only by this amount.  So at least this particular metadata item is not being duplicated.
Best,
Gary

From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk] On Behalf Of Munro, Ian
Sent: Monday, October 27, 2014 9:02 AM
Cc: ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
Subject: Re: [ome-devel] OME-Tiff writing using LOCI calls from MATLAB, and ModuloAlongT annotation

Hi Again Gary

Good to know that you've found a solution. However,  I find it puzzling that the files are different sizes.
We had a problem where bio-formats was writing multiple copies of the xml when called from Matlab.
The work-around was to add the following line:

java.lang.System.setProperty('javax.xml.transform.TransformerFactory', 'com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl');

to the Matlab code as in the example that I sent.
Perhaps this might help.

Regards

Ian
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel


Dr Sébastien Besson
Open Microscopy Environment / Harvard Medical School
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: (01382) 386364


The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20141028/0ed8ed60/attachment.html>