[ome-devel] Failed root logins at midnight

Tue Nov 1 16:57:03 GMT 2016

Hi Josh,

thanks for your reply and no worries about the delay. I was also "distracted" by some other issues ...

1.) failed root logins
++++++++++++++++++++++
Finally, we could identify a script that was wrongly configured. I have to apologize, I did not know anything about it 
and it took us some internal email conversation ping-pong to finally become aware of the culprit ;-)
Since last Friday we now no longer receive any login complaints. Sorry again, we shot ourselves in the foot :-(

2.) LDAP exception
++++++++++++++++++
That's very interesting to have this potentially correlated to Frederik's "SHA1 does not match after script upload" post 
at "http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2016-November/003794.html"?!?

Yes, indeed, we had some infrastructure changes! We could meanwhile figure out that the LDAP from our Active Directory 
might have caused substantial trouble during the past week!

The AD cluster to which our OMERO LDAP URL points to was upgraded to a newer version. Unfortunately this was done in 
such a way that our configured virtual AD cluster connection address temporarily resolves to both new (=active) and old 
(=deactivated) AD domain controllers! As a result, certain request just by chance might end up with a meanwhile 
de-activated one and will cause authentication errors. Unfortunately, it seems that we have to wait for another week 
until the last "old" one can be de-activated. And only then all de-activated ones will be also finally removed from the 
virtual cluster connection address. :-(

I therefore now adjusted our OMERO LDAP config to specifically point towards a single domain controlled instead of using 
the previous virtual one.

I hope that this change might also help to solve Frederik's SHA1 issue.

And again: apologies from our side for all the hassle!

Cheers,
-Rainer

On 11/01/2016 04:19 PM, Josh Moore wrote:
> Hi Rainer,
>
> sorry for the delay in getting back to you. I don't think we've seen
> this type of behavior before nor have I yet found a reason for the
> failures. Based on the logs, I would have hoped that your latest
> restart:
>
> $ grep Ready Blitz-0.log.1
> 2016-10-24 17:36:17,596 INFO  [
> ome.services.util.ServerVersionCheck] (      main) OMERO Version:
> 5.2.5-ice35-b28 Ready.
>
> would have corrected the issue. Unfortunately, that's not the case.
> Interestingly, though, starting with Blitz-0.log.1, a new exception
> has appeared:
>
>
> 2016-10-27 08:26:53,504 ERROR [
> o.s.blitz.fire.PermissionsVerifierI] (l.Server-6) Exception thrown
> while checking password for:witzg
> ome.conditions.InternalException:  Wrapped Exception:
> (org.springframework.ldap.CommunicationException):
> unibasel.ads.unibas.ch:3268; nested exception is
> javax.naming.CommunicationException: unibasel.ads.unibas.ch:3268 [Root
> exception is java.net.ConnectException: Connection refused]
>         at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:98)
> ~[spring-ldap-core.jar:1.3.0.RELEASE]
>
>
> There's not necessarily a connection between the failures before the
> restart and the new LDAP issue, but I do wonder if you know of any
> infrastructural changes which may have happened in the last few days:
>
>  * change of passwords for the root user
>  * change in your institution's LDAP server
>  * networking changes or the like
>
> Thanks for helping us to track this down.
> ~Josh.
>
>
> On Fri, Oct 28, 2016 at 1:50 PM, Rainer Poehlmann
> <rainer.poehlmann at unibas.ch> wrote:
>> Hi Will,
>>
>>> I’m afraid the majority of the team are at a conference or on leave today,
>>> but we’ll try to investigate the issue and your logs next week.
>>
>>
>> no worries.
>>
>> This issue does not impact any operations of OMERO. It's really more to
>> understand what's going on behind the scenes with those failed logins.
>>
>> Thanks for letting me know.
>>
>> Regards,
>> -Rainer
>>
>>
>>
>>>> On 28 Oct 2016, at 10:15, Rainer Poehlmann <rainer.poehlmann at unibas.ch>
>>>> wrote:
>>>>
>>>> Dear Mark,
>>>>
>>>> just to let you know that upon changing the omero.scripts.cache.cron
>>>> entry to 2:00 in the momring I now received those 2 "failed root login"
>>>> emails at exactly this time ;-)
>>>>
>>>> Cheers,
>>>> -Rainer
>>>>
>>>> On 10/27/2016 04:13 PM, Rainer Poehlmann wrote:
>>>>>
>>>>> Dear Mark,
>>>>>
>>>>> I uploaded the whole OMERO log directory as "tar.gz" file.
>>>>>
>>>>> I also issued an
>>>>>
>>>>> bin/omero config set omero.scripts.cache.cron "0 0 2 * * ?"
>>>>>
>>>>> to shift the script reloading to 2:00 in the morning. Let's see if this
>>>>> will have effects on the "failed root logins" as
>>>>> well ;-)
>>>>>
>>>>> Anyhow, thanks a lot for your support!
>>>>>
>>>>> Cheers,
>>>>> -Rainer
>>>>>
>>>>>
>>>>> On 10/27/2016 11:06 AM, Mark Carroll wrote:
>>>>>>
>>>>>> Dear Rainer,
>>>>>>
>>>>>> I am afraid that we too are puzzled: we are not seeing similar on any
>>>>>> of
>>>>>> the production systems that we have checked so far. Could you please
>>>>>> zip
>>>>>> up your OMERO server's var/log/ folder and upload it to
>>>>>> http://qa.openmicroscopy.org.uk/qa/upload/ ?  Also, it might be
>>>>>> interesting to adjust the value of omero.scripts.cache.cron in your
>>>>>> server's configuration properties from the default of "0 0 0 * * ?" to
>>>>>> see if the timing of these failed root logins moves accordingly.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>>> _______________________________________________
>>>>>> ome-devel mailing list
>>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>>>
>>>> _______________________________________________
>>>> ome-devel mailing list
>>>> ome-devel at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>> _______________________________________________
>>> ome-devel mailing list
>>> ome-devel at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>