[ome-users] LDAP and posix groups

Thu Jul 29 14:50:22 BST 2010

Hi Mark,

Hrm... that's definitely not good. Can you send me the output?

-Chris

On 27 Jul 2010, at 15:39, Woodbridge, Mark R wrote:

> Thanks Josh! My fault for not looking carefully enough at the documentation.
> 
> Though I am a bit concerned that on a dry-run it reports that it will remove 19448 files and keep only 4...
> 
> Mark.
> ________________________________________
> From: Josh Moore [josh at glencoesoftware.com]
> Sent: 27 July 2010 15:27
> To: Woodbridge, Mark R
> Cc: Alessandro Dellavedova; ome-users at lists.openmicroscopy.org.uk
> Subject: Re: [ome-users] LDAP and posix groups
> 
> Mark et al,
> 
> You may want to take a look at:
> 
> https://www.openmicroscopy.org/site/support/omero4/server/backup-and-restore#section-1
> 
> for information on cleansing the deleted binary files.
> ~J.
> 
> 
> On Jul 27, 2010, at 10:56 AM, Woodbridge, Mark R wrote:
> 
>> Hi,
>> 
>> We're using ZFS mounted from CentOS (Linux) using NFS4 which hasn't given us any problems (yet!).
>> 
>> Regarding data de-duplication: this is an advantage of using ZFS, along with other nice features such as filesystem snapshots (good for doing upgrades) and on-the-fly compression. So I guess this can deal with the case where a researcher uploads an image more than once. But it doesn't deal with the two data duplication scenarios discussed previously. One of these cases is where a researcher (or research group) want to keep their data locally but also want to use the features of OMERO, so they upload their data but also keep it locally. This is something that OMERO.fs can (or will?) help with. The other type of duplication is when you choose the 'archival' option in the importer, in which case your image is stored twice on the server (once as the original and once as a 'flattened' bitmap). I think this is a good thing, but something you need to be aware of when budgeting for disk space if you plan to encourage users to use this feature.
>> 
>> One other thing regarding disk space that I previously mentioned on this list is that when you delete an image from insight it seems to remain on disk (on the server) even though references to it are correctly removed from the database. Looking at our filesystem, these orphaned images seem to be currently taking about 10% of our space (if i am understanding the situation correctly).
>> 
>> Mark.
>> ________________________________________
>> From: ome-users-bounces at lists.openmicroscopy.org.uk [ome-users-bounces at lists.openmicroscopy.org.uk] On Behalf Of Alessandro Dellavedova [alessandro.dellavedova at ifom-ieo-campus.it]
>> Sent: 26 July 2010 23:44
>> To: Josh Moore
>> Cc: ome-users at lists.openmicroscopy.org.uk
>> Subject: Re: [ome-users] LDAP and posix groups
>> 
>> Hi Josh,
>> 
>> on behalf of Futhwo I'd like to say "Thank you" for the quick OMERO dollar workaround, tomorrow we'll test it and we'll post a feedback on the mailing list.
>> 
>> Futhwo (our senior sysadmin, cloaked) is extensively testing OMERO under Solaris and we'll be very happy to share the setup instructions and all the technical informations that are needed in order to run OMERO under this platform (we know that this is officially not supported, yet).
>> 
>> Basically we did choose Solaris due to the fact that we are deploying a multi-terabyte installation of OMERO 4.2.0 (88TB on a Sun..err..Oracle X4540 server) and we needed a reliable filesystem like ZFS, in order to protect ourselves from silent data corruption and other problems that arise when dealing with this tremendous amount of data (please see References).
>> 
>> Moreover, as soon as the ZFS version under Solaris will match the ZFS versions currently implemented under OpenSolaris and Nexenta Core, we'll get data-deduplication basically for free, and that will enable us to save some space on local storage (an early test showed a data-deduplication ratio of 1.55 for timelapse experiments, with a minor performance hit).
>> 
>> The need for local storage data deduplication was outlined by Martin Spitaler and Mark Woodbridge during their talk at the OME Meeting in Paris (slide 14), and data deduplication is a desiderata for OMERO.FS (http://www.openmicroscopy.org/site/support/omero4/server/fs). By using ZFS this will come basically for free.
>> 
>> HTH,
>> 
>> Alessandro
>> 
>> DISCLAIMER: We are not sponsored in any way by Sun/Oracle, we are just sharing our experience and we hope that this can be useful to someone else. If you are interested in the ZFS filesystem and you don't like Solaris/OpenSolaris/Nexenta Core you can also try FreeBSD 8.1 that implements ZFS version 15 (http://wiki.freebsd.org/ZFS).
>> 
>> REFERENCES:
>> End-to-end Data Integrity for File Systems: A ZFS Case Study - http://www.cs.wisc.edu/wind/Publications/zfs-corruption-fast10.html
>> 
>> OMERO Implementation at Imperial College London - http://openmicroscopy.org/site/community/minutes/meetings/june-2010-paris-users-meeting/presentations/3%20%20Martin%20Spitaler/spitaler%20100615%20OME%20metting.pdf/at_download/file
>> 
>> ZFS Deduplication - http://blogs.sun.com/bonwick/entry/zfs_dedup
>> 
>> Alessandro Dellavedova
>> 
>> Responsabile Sistemi Informativi
>> 
>> COGENTECH - Consortium for Genomic Technologies
>> 
>> Via Adamello, 16 - 20139 Milan, Italy
>> T +39 02 57489.857
>> F +39 02 9437.5990
>> E alessandro.dellavedova at ifom-ieo-campus.it
>> W www.ifom-ieo-campus.it        www.ieo.it
>> 
>> “There are risks and costs to a program of action. But, they are far less than the long-range risks and costs of comfortable inaction.” – John F. Kennedy (1917-1963)–
>> 
>> On Jul 26, 2010, at 8:03 PM, Josh Moore wrote:
>> 
>>> Hi Futhwo,
>>> 
>>> Unforunately, you've hit upon a rather interesting bug in 4.2.0.
>>> 
>>> I've created a ticket to track the issue:
>>> 
>>> http://trac.openmicroscopy.org.uk/omero/ticket/2613
>>> 
>>> To workaround the issue you'll need to set an extra property:
>>> 
>>> ./omero config set omero.dollar '$'
>>> 
>>> And then:
>>> 
>>> ./omero config set omero.ldap.new_user_group ':query:(memberUid=$${omero.dollar}{uid})'
>>> 
>>> As odd as it may seem, an OMERO dollar should get you what you want. </end-bad-joke>
>>> ~Josh.
>>> 
>>> 
>>> On Jul 23, 2010, at 4:00 PM, Futhwo wrote:
>>> 
>>>> Hi
>>>> 
>>>> I am trying to set up OMERO to insert new users in the same groups he has on
>>>> the ldap directory (we use RFC 2307 standard).
>>>> 
>>>> In this standard group membership is defined by the "memberUid" multi value
>>>> in the group entry, wich value is the uid of the user belonging to the group
>>>> defined in the entry.
>>>> 
>>>> So to set up this for omero i used, as pointed in the examples:
>>>> 
>>>> ./omero config set omero.ldap.new_user_group ':query:(memberUid=${uid})'
>>>> 
>>>> To double-ckeck it:
>>>> 
>>>> ./omero config get
>>>> omero.config.updated=4.2.0
>>>> omero.ldap.base=dc=MYDOMAIN,dc=it
>>>> omero.ldap.config=true
>>>> omero.ldap.group_filter=(objectClass=posixGroup)
>>>> omero.ldap.group_mapping=name=cn
>>>> omero.ldap.new_user_group=:query:(memberUid=${uid})
>>>> omero.ldap.password=
>>>> omero.ldap.urls=ldap://MYLDAPSERVER:389
>>>> omero.ldap.user_filter=(objectClass=posixAccount)
>>>> omero.ldap.user_mapping=omeName=uid,firstName=givenName,lastName=sn,email=mail
>>>> omero.ldap.username=
>>>> 
>>>> (I substituted MYDOMAIN and MYLDAPSERVER of course).
>>>> 
>>>> This do not work, group membership still use the previous value for
>>>> omero.ldap_new_user_group, even if "omero config get" reports the new value.
>>>> 
>>>> If i restart the server i see in master.err:
>>>> 
>>>> 07/23/10 15:46:07.852 icegridnode: warning: failed to deploy application
>>>> `/opt/omero_dist/etc/grid/default.xml':
>>>> IceGrid::DeploymentException: application `OMERO':
>>>> invalid value for attribute `property set `__ACTIVE__' property value':
>>>> invalid variable `:query:(memberUid=${uid})':
>>>> undefined variable `uid'
>>>> 
>>>> I tried using ${cn} and ${omeName} with the same result.
>>>> 
>>>> If i try something like:
>>>> 
>>>> ./omero config set omero.ldap.new_user_group ':query:(memberUid=$uid)'
>>>> 
>>>> the server stop complaining at start, but the query issued to ldap will be
>>>> (taken from the openldap server debug):
>>>> 
>>>> filter="(&(objectClass=posixGroup)(memberUid=$uid))"
>>>> 
>>>> without the substitution of the $uid string with logging user id, so users
>>>> cannot login.
>>>> 
>>>> Thanks in advance to anyone who may help
>>>> 
>>>> Cheers
>>>> Futhwo
>>> 
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> 
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> 
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users