[ome-users] Partial Replication of OMERO data for TEST system: how to obtain user specific Pixels data as well?

[Josh Moore]

On Wed, Jul 13, 2016 at 10:03 AM, Rainer Poehlmann
<rainer.poehlmann at unibas.ch> wrote:
> Dear all,

Hi Rainer,


> in order to have an OMERO TEST server with a reasonable subset of real data,
> I have started to replicate data from our productive OMERO storage volume to
> a new test volume connected to the OMERO TEST environment. Since we have
> more than 40TB of productive data, I am obviously interested to only
> replicate parts of the data, e.g. by replicating data from certain users
> only.
>
> The easy part:
> ==============
> replicate specific user data from the ManagedRepository, e.g.
> "{omero.data.dir}/OMERO/ManagedRepository/userA_n" PROD to TEST
>
> The problem:
> ============
> certain replicated data from "userA_n" cannot be displayed in the "detailed
> view". Obvious error:
> A "Pyramid" of zoom levels is currently being calculated for this image.
> Please try viewing again later"
>
> => Pyramid data for this particular user/image as present somewhere under
> "{omero.data.dir}/OMERO/Pixels" were not replicated and are therefore
> missing!
>
> First of all a more general question to better understand this topic: what
> are the "rules" according to which pyramid data for a particular image
> (format) will be generated or not? We discussed this internally but did not
> really came to a satisfying answer.

Pyramids are generated for files which are A) larger than 3K x 3K
_and_ B) in a format which itself does not include pyramids (.svs,
.ims, and a few others *do* provide pyramids).

The 3K value can be increased via
https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/config.html#omero-pixeldata-max-plane-height
though going much higher will lead to memory issues.



> And, more specific to the PROD to TEST data replication problem:
> in order to have a fully working TEST environment, I do not want to
> replicate all of our 3TB of "Pixels" data from PROD => TEST. Since the
> "Pixels" data are organized in a more "arbitrary" directory structure
> "(Dir-001 to Dir-nnn), is there an (easy?) way to identify *all* "Pixels"
> data belonging to one user to just replicate those and not the whole
> "Pixels" directory?

The pyramid files can be found with `find /OMERO/Pixels -name "*_pyramid"`

In order to limit by user, you'd have to cross-check the pixel IDs in
the DB. Happy to help with this if you'd like.  The longer-term
solution is certainly to have OMERO move all the pyramid files to the
managed repository. I've filed your email under
https://trello.com/c/Zkau64CA/108-store-pyramids-under-managedrepository


> Any hints would be appreciated.
>
> Thanks a lot for your help, best regards,
> -Rainer


Cheers,
~Josh.