[ome-users] 3DHistech .mrxs files - update?

[Jason Swedlow (Staff)]

Dear Sho,

Yes, several people have contacted us about 3D HISTECH .mrxs. At the moment, the complexity of the format, and as noted below, the work we need to do to deliver a complete solution for the format means that we have to lower its priority. I'm afraid I cannot give you an ETA. My understanding is that you are already CC'd on the Trac ticket that records any milestone progress on the implementation:

        http://trac.openmicroscopy.org/ome/ticket/4121

I can assure you that this is the best way to be notified if the priority changes or there is development of note.

A few more details:  we haven’t delivered a full reader for .mrxs, and there hasn’t been substantial public development, but we have spent a great deal of time examining the format and potential solutions, and building test readers.  So we know a lot about this format. We can say with some confidence that the 3D HISTECH .mrxs file format is the most complex whole slide imaging file format we have ever encountered. Thanks to the example data the community has generously provided, we have been able to analyse the on disk layout as well as the compression types and map out the details of what an implementation would entail.

One thing to understand about our work, strategy and commitment for supporting all file formats, especially formats used in production-scale facilities that use tech like whole slide imaging, is that we insist on delivering as close to complete support as possible. This is important given the size of community we support, the breadth of applications that use our software, and the need for utility and reliability in the software we deliver to the community. With .mrxs, it is very hard and expensive to meet this goal. To be specific:

  *   The design decisions of 3D HISTECH with respect to image pyramid layout are at odds with what we can reasonably handle within the infrastructure currently in place. Our analysis suggests we will have to re-calculate several of the resolution levels, because of choices 3D HISTECH has made in their tiling strategy.  This will create a substantial performance penalty for anyone using Bio-Formats to read this format.
  *   The strategy for storing binary data on disk in 3D HISTECH .mrxs brightfield differs substantially from fluorescence images stored in .mrxs. They are essentially two different file formats, thus doubling the work required.
  *   Based on recent data submissions and information from the community, 3DHISTECH scanners default to JPEG-XR compression when acquiring fluorescence data. Another doubling of work and complexity, as we support both compressed and uncompressed data, in brightfield and fluorescence.

As a team we are not comfortable with releasing a reader implementation that works on a limited set of 3D HISTECH .mrxs variants, or requires time consuming and computationally expensive reprocessing and pyramid creation, just because of the implementation choices made in the .mrxs format.  Also, while the software community at large has made significant strides in expanding JPEG-XR support beyond its original exclusive, patent-encumbered Microsoft Windows implementation, a pure Java decoder remains elusive. Work is underway to explore pure Java decoder alternatives as support for this compression type extends beyond 3D HISTECH .mrxs but it is very far from a simple addition.

Finally, we’ll finish with a philosophical point.  The OME Consortium and the wider community has worked steadily since 2002, funded mostly by grants from non-profit charities and public funders, to build tools for the wider scientific community. Building readers for proprietary formats has never been funded, and we don’t think it would ever be funded by any grant funding panel. New readers are created either by diverting our precious resources from other projects, by contributions by the community (most recently by the companies themselves) or by work commissioned by customers of Glencoe Software.  We certainly listen to the community and adjust our priorities based on requests, but we can’t do everything with limited resources.  Perhaps we could crowdsource the funding for .mrxs  but that misses the point— .mrxs is a complex, expensive, difficult, proprietary, closed format, designed to lock its users into a single, proprietary software application. The community’s resources are finite and should be used for other things than reverse engineering this type of format— work that if subjected to peer review would be declined as a waste of community resources.  Several of those “other things” were discussed at our recent Annual Users Meeting (Presentations<https://www.openmicroscopy.org/site/community/minutes/meetings/10th-annual-users-meeting-june-2015> and Trello<https://trello.com/b/CTG2teAV/paris-2015>) and represent key technologies that the community needs to achieve its scientific goals.

Over the last few years, we have seen efforts by several commercial imaging companies to support open formats, provide open APIs, and to make it easier for researchers and clinicians to work with the data acquired by their instruments.  We have also received specifications and input from several imaging companies, which we have used to improve our own work and output.  We applaud this trend— ultimately it means scientists, clinicians, engineers and developers spend less time dealing with data formats and more time doing science, developing new technologies and treating patients.

In the case of 3DHISTECH .mrxs, the community has the power to change the situation. You can condition your purchase, continued payment of support and maintenance fees, etc on

— the delivery of a rational, well-designed, efficient, open format
— use of open compression schemes
— support for the community’s efforts to deliver open readers for these files

You can of course also commit your own development resources to help solve this problem.

I’m sorry we can’t give a more positive response to your email,  several others on this list, and many other private emails we’ve received.  As described above, we’ve explored this format extensively, and have already spent a few developer months on it.  We’d strongly encourage a discussion between the user community and the creators of this format.

Cheers,

Jason

--------------------
Centre for Gene Regulation & Expression | Open Microscopy Environment | University of Dundee

Phone:  +44 (0) 1382 385819
email: j.swedlow at dundee.ac.uk<mailto:j.swedlow at dundee.ac.uk>

Web: http://www.lifesci.dundee.ac.uk/people/jason-swedlow
Open Microscopy Environment: http://openmicroscopy.org<http://openmicroscopy.org/>


From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> on behalf of "fujisaws at mskcc.org<mailto:fujisaws at mskcc.org>" <fujisaws at mskcc.org<mailto:fujisaws at mskcc.org>>
Reply-To: OME Users <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Date: Thursday, 10 September 2015 14:23
To: OME Users <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: [ome-users] 3DHistech .mrxs files - update?

Hello -

I assume a lot of people have contacted you about this, but we are biting our fingers waiting for OME to support 3DHistech .mrxs files. Any update or ETA?


Sho Fujisawa
Sr. Digital Microscopist
Molecular Cytology Core Facility
Memorial Sloan-Kettering Cancer Center
417 E. 68th St.
Zuckerman Room 1834
New York NY 10065
(212) 639-6667
(646) 888-2186


=====================================================================

Please note that this e-mail and any files transmitted from
Memorial Sloan Kettering Cancer Center may be privileged, confidential,
and protected from disclosure under applicable law. If the reader of
this message is not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient,
you are hereby notified that any reading, dissemination, distribution,
copying, or other use of this communication or any of its attachments
is strictly prohibited. If you have received this communication in
error, please notify the sender immediately by replying to this message
and deleting this message, any attachments, and all copies and backups
from your computer.

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20150914/5c48dbe3/attachment.html>