[ome-users] tech queries

[Chris Allan]

On Fri, Jun 10, 2005 at 11:21:41AM +0100, Alastair Kerr wrote:
> Dear all, 

Hi Alastair.

> we are getting new hardware which will run OME and connect to a remote
> SAN for storage. We have some restrictions on what to get but we do have
> some leeway.  I have here some queries from the tech guys about OME
> functionality. I can partially answer some of them but I would prefer to
> make sure.  I've commented my (lack of)  understanding of the issue to
> each query. Help much appreciated. 
> 
> Background:
> ----------
> [SRIF= network with a SAN array used as storage]
> 
> The basic setup is we need to buy a box to sit on a local Gb fibre on 
> the SRIF network with, at present, 3 NFS mounted volumes (more will 
> follow as we request more space). Each volume is 3Tb.
> 
> OME will be run on this system, along with general services to allow 
> staff access to the space on the SAN system on the SRIF network.
> 
> As nothing is stored on the server the disk capacity need only be
> minimal.
> 
> 
> Questions about the system are:
> -------------------------------
> 
> "Can OME utilise 64 bit dual processor systems - is this worth it"
> [Alastair: If perl does, OME should - it this correct? I imagine there
> would be not much speed boost but I can't see why it would be
> detrimental]

Yes, and in fact in order to work with images larger than ~800MB at
present you need a 64-bit machine. This 64-bit requirement for such
images will disappear in 2.4.2.

> 
> "How can OME handle the 3Tb volumes, presumably the master db is held on
> one, while the data files can be scattered across several"
> [Alastair: 
> My understanding of how OME works is that 
> 1. Files from the microscopes - as long as under the 'data' directory
> the web interface can view/import them. Once imported into OMEIS then
> they can be deleted. No problems to symlink the dir structure to other
> partitions(?)

OME copies data into the OMEIS repository, yes. There are no problems in
symlinking the entire OMEIS repository directory structure or part of
it.

> 2. OMEIS + OME Can be in separate partitions. Do not know enough to
> figure out how to spread between partitions or to add a partition later
> ]
> 

/OME/OMEIS can be symlinked anywhere you like. Repository files can be
moved anywhere you like as long as you symlink them as well. Example:

mv /OME/OMEIS /really_big_fs
ln -s /really_big_fs/OMEIS /OME/OMEIS

--and--

mv /OME/OMEIS/Dir-001 /really_big_fs2
ln -s /really_big_fs2/Dir-001 /OME/OMEIS/Dir-001

Obvously this does require maintenance and a brief understanding of how
much data is in which folders.

> 
> 
> "We have a cosign/kerberos based authentication system which can be used
> for general ssh logins and web authentication - can OME utilise this, 
> particularly for the java side (possibly with tomcat?)"
> [Alastair: I think this could be an issue  but I do not fully understand
> the problem]

OME, at present, has no support for extended authentication systems. I
also don't see us intergrating with or implementing Cosign single sign-on
authentication in our software in the near future.

> 
> "As the server can analyse data what RAM or swap space requirements does
> it have, especially if it works on movies"
> [Alastair: I assume that the analysis using shoola/web is performed
> server side and not client side - therefore I think: as much as
> possible...]

More is done on the client side with the Shoola Java client but *all* is
done on the server side with the web interface. So yes, "as much as
possible" certainly fits the bill quite accurately. 2GB would certainly
be adequate but it really does depend on your usage 4 or 8GB would not
be unreasonable.

> 
> "is all OME functionality usable in the situation where all storage for 
> the program is available as a single NFS mounted volume, and where only 
> a single uid/gid  (eg OMEuser) is used on that storage"
> [Alastair: not sure]

As long as you maintain the configured UID and GID, yes. OME makes no
use of filesystem ACLs.

A note, your performance milage with mmap(2) on an NFS mounted volume
may vary. OMEIS uses mmap(2) for almost all its I/O routines and you
will have to evaluate the performance penalty of having the OMEIS
repository on NFS. I would not suggest you architect your system this
way if you want to achieve reliability and performance. Without a more
accurate picture of your network topology and the equipment involved I
can't really comment more on this.

> 
> 
> thanks in advance for your input

You're welcome.

> - Alastair 

Ciao.

-Chris