<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Pasi<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 17 Apr 2019, at 14:02, Pasi Kankaanpää <<a href="mailto:pkankaan@abo.fi" class="">pkankaan@abo.fi</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi everyone,<br class="">
<br class="">
We at Turku BioImaging and the Finnish Euro-BioImaging Node are currently making plans for an OMERO server setup, and some questions have come up in this context. The OMERO team recommended posting these questions here, in case they would benefit also others.
Any experiences, thought and comments would be greatly appreciated.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-family: Arial; font-size: 9pt; white-space: pre-wrap;" class="">Thanks for raising these questions publicly. Most of these answers are informed by our own use case
with IDR. Other community members with large-scale deployments might want to share their experience as well.</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-family: Arial; font-size: 9pt; white-space: pre-wrap;" class=""><br class="">
</span></div>
<blockquote type="cite" class="">
<div class="">
<div class="">1) How well would you estimate OMERO handles large files, such as data from a light sheet microscope, in these two scenarios a) a large amount of data consisting of numerous small files and b) a large amount of data consisting of just one or few
very large files.<br class="">
</div>
</div>
</blockquote>
<br class="">
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">IDR
has multiple representative examples of both situations:</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class=""><br class="">
</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">1-
the case of numerous small files is frequent in the High-Content Screening domain where individual plates can typically span over 10-100K files [1]</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">2-
the case of 100GB-1TB sized individual files is becoming increasingly common e.g. from the light-sheet technology experiments [2]</span></div>
<br class="">
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Re
“How well…?”, we won’t pretend that routinely loading multi-TB datasets with many 1000s of files or very large TB-scale files does not require some care and consideration. For example, copying the data often is no longer an option and we use “in-place” import
for these dataset or we will delay thumbnail calculation until the server is less busy-- these fall under what we refer to as “advanced import mechanisms” [3]. Hopefully, the two previous links demonstrate that both scenarios can be handled with appropriate
use of OMERO’s capabilities.</span></div>
<div><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[1]
</span><a href="https://idr.openmicroscopy.org/webclient/?show=image-1898184" style="text-decoration: none;" class=""><span style="font-size: 9pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; -webkit-text-decoration-skip: none; vertical-align: baseline; white-space: pre-wrap;" class="">https://idr.openmicroscopy.org/webclient/?show=image-1898184</span></a><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">
</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[2]
</span><a href="https://idr.openmicroscopy.org/webclient/?show=image-4007801" style="text-decoration: none;" class=""><span style="font-size: 9pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; -webkit-text-decoration-skip: none; vertical-align: baseline; white-space: pre-wrap;" class="">https://idr.openmicroscopy.org/webclient/?show=image-4007801</span></a></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[3]
</span><a href="https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/import-scenarios.html" style="text-decoration:none;" class=""><span style="font-size: 9pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; -webkit-text-decoration-skip: none; vertical-align: baseline; white-space: pre-wrap;" class="">https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/import-scenarios.html</span></a><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">
</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class=""><br class="">
</span></div>
<blockquote type="cite" class="">
<div class="">
<div class="">2) If we think of a 5-10 year spectrum and a total cumulative data amount during this time of 10 petabytes, will OMERO handle this without problems?<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Large
deployments like IDR are routinely serving 100TB-1PB datasets these days. Considering the data growth, I would say the data volume will naturally fall into your range over the next decade. Probably even more than disk size, the choice of hardware and the access
speed to the storage/database are important architectural decisions.</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class=""><br class="">
</span></div>
<blockquote type="cite" class="">
<div class="">
<div class="">3) If we think of a scenario where of the total storage capacity 10-20% would be faster access working storage and the rest more permanent archiving storage (that can have slower access times), how would you envision OMERO working with this scenario,
or what would you see as pros and cons of these alternatives:<br class="">
<br class="">
a) OMERO would run only in working storage and not archiving (how would the transfer between them take place?)<br class="">
<br class="">
b) OMERO would run only in archiving and not working storage (how would the transfer between them take place?)<br class="">
<br class="">
c) OMERO would run both in working storage and archiving (how would the transfer between them take place?)<br class="">
<br class="">
d) 100% of the capacity would be with high access speed, operated by OMERO, so no separate working and archiving storages<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">The
OMERO server configuration includes some mechanisms for extending the repository and accommodate for different underlying storage volumes [4]. Some people had also looked at integrating production archiving solutions like Arkivum with OMERO - see for instance
[5].</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">Vanilla
OMERO does not include support for hierarchical storage-- we have been asked this many times. You might consider having OMERO trigger data migration between different tiers or use some other applications. As each archiving implementation ends up having custom
features, we’d encourage you to look at various integration points and consider what works best for your needs.</span></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><br class="">
</div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class="">
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[4]
</span><a href="https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/repository-move.html#extending-the-managed-repository" style="text-decoration: none;" class=""><span style="font-size: 9pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; -webkit-text-decoration-skip: none; vertical-align: baseline; white-space: pre-wrap;" class="">https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/repository-move.html#extending-the-managed-repository</span></a></div>
<div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">[5]
</span><a href="https://downloads.openmicroscopy.org/presentations/2017/Users-Meeting/Lightning-Talks/Alex%20Herbert%20-%20Archiving%20images%20from%20OMERO%20to%20Arkivum%202017.pdf" style="text-decoration: none;" class=""><span style="font-size: 9pt; font-family: Arial; color: rgb(17, 85, 204); font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: underline; -webkit-text-decoration-skip: none; vertical-align: baseline; white-space: pre-wrap;" class="">https://downloads.openmicroscopy.org/presentations/2017/Users-Meeting/Lightning-Talks/Alex%20Herbert%20-%20Archiving%20images%20from%20OMERO%20to%20Arkivum%202017.pdf</span></a><span style="font-size: 9pt; font-family: Arial; font-variant-ligatures: normal; font-variant-east-asian: normal; font-variant-position: normal; vertical-align: baseline; white-space: pre-wrap;" class="">
</span></div>
<div class=""><br class="">
</div>
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">Thanks a lot in advance for any comments,<br class="">
<br class="">
Pasi<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Best,</div>
</div>
<div>Sebastien & the OME/IDR team</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
--<br class="">
Pasi Kankaanpää, PhD<br class="">
Administrative Director, Turku BioImaging<br class="">
Project Manager, Euro-BioImaging<br class="">
Åbo Akademi University and University of Turku<br class="">
Turku, Finland<br class="">
<br class="">
<a href="mailto:pkankaan@abo.fi" class="">pkankaan@abo.fi</a><br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
https://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>