[ome-devel] Failed root logins at midnight

Tue Nov 1 17:33:23 GMT 2016

Hi Jush, Rainer and all,

Thank you very much for helping us to find the bug, which also resolved
my issue. Since Rainer had adjusted the LDAP settings I can finally work
with my OMERO scripts again.

Cheers,

Frederik

On 01.11.2016 17:57, Rainer Poehlmann wrote:
> Hi Josh,
>
> thanks for your reply and no worries about the delay. I was also
> "distracted" by some other issues ...
>
> 1.) failed root logins
> ++++++++++++++++++++++
> Finally, we could identify a script that was wrongly configured. I
> have to apologize, I did not know anything about it and it took us
> some internal email conversation ping-pong to finally become aware of
> the culprit ;-)
> Since last Friday we now no longer receive any login complaints. Sorry
> again, we shot ourselves in the foot :-(
>
> 2.) LDAP exception
> ++++++++++++++++++
> That's very interesting to have this potentially correlated to
> Frederik's "SHA1 does not match after script upload" post at
> "http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2016-November/003794.html"?!?
>
> Yes, indeed, we had some infrastructure changes! We could meanwhile
> figure out that the LDAP from our Active Directory might have caused
> substantial trouble during the past week!
>
> The AD cluster to which our OMERO LDAP URL points to was upgraded to a
> newer version. Unfortunately this was done in such a way that our
> configured virtual AD cluster connection address temporarily resolves
> to both new (=active) and old (=deactivated) AD domain controllers! As
> a result, certain request just by chance might end up with a meanwhile
> de-activated one and will cause authentication errors. Unfortunately,
> it seems that we have to wait for another week until the last "old"
> one can be de-activated. And only then all de-activated ones will be
> also finally removed from the virtual cluster connection address. :-(
>
> I therefore now adjusted our OMERO LDAP config to specifically point
> towards a single domain controlled instead of using the previous
> virtual one.
>
> I hope that this change might also help to solve Frederik's SHA1 issue.
>
> And again: apologies from our side for all the hassle!
>
> Cheers,
> -Rainer
>
> On 11/01/2016 04:19 PM, Josh Moore wrote:
>> Hi Rainer,
>>
>> sorry for the delay in getting back to you. I don't think we've seen
>> this type of behavior before nor have I yet found a reason for the
>> failures. Based on the logs, I would have hoped that your latest
>> restart:
>>
>> $ grep Ready Blitz-0.log.1
>> 2016-10-24 17:36:17,596 INFO  [
>> ome.services.util.ServerVersionCheck] (      main) OMERO Version:
>> 5.2.5-ice35-b28 Ready.
>>
>> would have corrected the issue. Unfortunately, that's not the case.
>> Interestingly, though, starting with Blitz-0.log.1, a new exception
>> has appeared:
>>
>>
>> 2016-10-27 08:26:53,504 ERROR [
>> o.s.blitz.fire.PermissionsVerifierI] (l.Server-6) Exception thrown
>> while checking password for:witzg
>> ome.conditions.InternalException:  Wrapped Exception:
>> (org.springframework.ldap.CommunicationException):
>> unibasel.ads.unibas.ch:3268; nested exception is
>> javax.naming.CommunicationException: unibasel.ads.unibas.ch:3268 [Root
>> exception is java.net.ConnectException: Connection refused]
>>         at
>> org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:98)
>> ~[spring-ldap-core.jar:1.3.0.RELEASE]
>>
>>
>> There's not necessarily a connection between the failures before the
>> restart and the new LDAP issue, but I do wonder if you know of any
>> infrastructural changes which may have happened in the last few days:
>>
>>  * change of passwords for the root user
>>  * change in your institution's LDAP server
>>  * networking changes or the like
>>
>> Thanks for helping us to track this down.
>> ~Josh.
>>
>>
>> On Fri, Oct 28, 2016 at 1:50 PM, Rainer Poehlmann
>> <rainer.poehlmann at unibas.ch> wrote:
>>> Hi Will,
>>>
>>>> I’m afraid the majority of the team are at a conference or on leave
>>>> today,
>>>> but we’ll try to investigate the issue and your logs next week.
>>>
>>>
>>> no worries.
>>>
>>> This issue does not impact any operations of OMERO. It's really more to
>>> understand what's going on behind the scenes with those failed logins.
>>>
>>> Thanks for letting me know.
>>>
>>> Regards,
>>> -Rainer
>>>
>>>
>>>
>>>>> On 28 Oct 2016, at 10:15, Rainer Poehlmann
>>>>> <rainer.poehlmann at unibas.ch>
>>>>> wrote:
>>>>>
>>>>> Dear Mark,
>>>>>
>>>>> just to let you know that upon changing the omero.scripts.cache.cron
>>>>> entry to 2:00 in the momring I now received those 2 "failed root
>>>>> login"
>>>>> emails at exactly this time ;-)
>>>>>
>>>>> Cheers,
>>>>> -Rainer
>>>>>
>>>>> On 10/27/2016 04:13 PM, Rainer Poehlmann wrote:
>>>>>>
>>>>>> Dear Mark,
>>>>>>
>>>>>> I uploaded the whole OMERO log directory as "tar.gz" file.
>>>>>>
>>>>>> I also issued an
>>>>>>
>>>>>> bin/omero config set omero.scripts.cache.cron "0 0 2 * * ?"
>>>>>>
>>>>>> to shift the script reloading to 2:00 in the morning. Let's see
>>>>>> if this
>>>>>> will have effects on the "failed root logins" as
>>>>>> well ;-)
>>>>>>
>>>>>> Anyhow, thanks a lot for your support!
>>>>>>
>>>>>> Cheers,
>>>>>> -Rainer
>>>>>>
>>>>>>
>>>>>> On 10/27/2016 11:06 AM, Mark Carroll wrote:
>>>>>>>
>>>>>>> Dear Rainer,
>>>>>>>
>>>>>>> I am afraid that we too are puzzled: we are not seeing similar
>>>>>>> on any
>>>>>>> of
>>>>>>> the production systems that we have checked so far. Could you
>>>>>>> please
>>>>>>> zip
>>>>>>> up your OMERO server's var/log/ folder and upload it to
>>>>>>> http://qa.openmicroscopy.org.uk/qa/upload/ ?  Also, it might be
>>>>>>> interesting to adjust the value of omero.scripts.cache.cron in your
>>>>>>> server's configuration properties from the default of "0 0 0 * *
>>>>>>> ?" to
>>>>>>> see if the timing of these failed root logins moves accordingly.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> The University of Dundee is a registered Scottish Charity, No:
>>>>>>> SC015096
>>>>>>> _______________________________________________
>>>>>>> ome-devel mailing list
>>>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>>>>
>>>>> _______________________________________________
>>>>> ome-devel mailing list
>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>
>>>>
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No:
>>>> SC015096
>>>> _______________________________________________
>>>> ome-devel mailing list
>>>> ome-devel at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>
>>> _______________________________________________
>>> ome-devel mailing list
>>> ome-devel at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>

-- 
Dr. Frederik Grüll | Imaging Expert | G1055, Biozentrum, University of
Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41 (61) 207 2250 |
frederik.gruell at unibas.ch | www.biozentrum.unibas.ch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20161101/b5193195/attachment.asc>