How to switch to LDAPS as authentication backend in Elasticsearch and solve certificate errors

Written by Claudio Kuenzler - 0 comments

Published on December 14th 2023 - Listed in Elasticsearch ELK Windows

One of the ELK stacks I manage is using LDAP as authentication backend. This is configured in /etc/elasticsearch.yml under the xpack setting (xpack -> security -> authc -> realms to be more precise).

Change from LDAP to LDAPS

Until recently we used LDAP to connect to the LDAP server(s) in the internal network, but we wanted to increase security (even in internal networks) and use LDAPS from now on.

Let's adjust this in elasticsearch.yml, changing ldap protocol to ldaps and LDAP port from 389 to LDAPS port 636:

xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
        ldap:
          ldap1:
            order: 1
            url: "ldaps://ldap.example.com:636"
            bind_dn: "cn=sa_elk, ou=users, o=services, dc=example, dc=com"
            bind_password: topGunSecret
            user_search:
              base_dn: "OU=Users,DC=example,DC=com"
              attribute: sAMAccountName
            group_search:
              base_dn: "OU=Groups,DC=example,DC=com"

After this quick change, restart Elasticsearch and that's it. Right? Right?!

Failed to obtain LDAP connection from pool

After restarting Elasticsearch and following the logs, it became quickly obvious that the supposedly quick config change did not work - but instead resulted in the following error:

[WARN ][o.e.x.s.a.l.s.LdapUtils  ] [es01] Failed to obtain LDAP connection from pool - LDAPException(resultCode=81 (server down), diagnosticMessage='An error occurred while attempting to send the LDAP message to server ldap.example.com:636:  SocketException(Connection or outbound has closed), ldapSDKVersion=4.0.8, revision=28812')
[WARN ][o.e.x.s.a.RealmsAuthenticator] [es01] Authentication to realm ldap1 failed - authenticate failed (Caused by LDAPException(resultCode=81 (server down), diagnosticMessage='An error occurred while attempting to send the LDAP message to server ldap.example.com:636:  SocketException(Connection or outbound has closed), ldapSDKVersion=4.0.8, revision=28812'))

Reading through the Elasticsearch LDAP user authentication documentation, once again, turned out be a good idea. The following (marked bold) part is quickly overlooked:

To protect the user credentials that are sent for authentication in an LDAP realm, it’s highly recommended to encrypt communications between Elasticsearch and your LDAP server. Connecting via SSL/TLS ensures that the identity of the LDAP server is authenticated before Elasticsearch transmits the user credentials and the contents of the connection are encrypted. Clients and nodes that connect via TLS to the LDAP server need to have the LDAP server’s certificate or the server’s root CA certificate installed in their keystore or truststore.

This means: Elasticsearch needs to have the LDAP's server or Root CA certificate locally installed!

Installing the LDAP server certificate

The LDAP server's certificate can be obtained on the Elasticsearch server itself - just the openssl command is necessary for this. Besides additional output and information about the certificate, the certificate itself is shown in PEM format:

root@es01:~# openssl s_client -connect ldap.example.com:636
CONNECTED(00000003)
depth=0
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:
   i:C = CH, O = Example Corp, CN = ACME Enterprise CA 1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGcTCCBFmgAwIBAgITHQAAQid5DfTJtp4qUwAAAABCJzANBgkqhkiG9w0BAQsF
[...]
siDlAPcj/29EctRH/DpAlqHih/9X
-----END CERTIFICATE-----
subject=

issuer=C = CH, O = Example Corp, CN = ACME Enterprise CA 1

---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
[...]

From this output we can see a couple of things:

• The certificate seems to be self-signed or at least not issued by a known Root CA as openssl was not able to verify the server certificate against any locally installed Root CA (usually found in /etc/ssl/certs).
• The certificate chain only shows one certificate. This means the LDAP server has only the (local) server certificate configured, without the issuing certificate
• The LDAP's full server certificate is shown in the output and can be copied from the lines BEGIN CERTIFICATE until (and including) END CERTIFICATE.
• The issuer certificate (Root CA) seems to be called "ACME Enterprise CA 1", which we have no knowledge of or have access to.
  

Now that we obtained the LDAP server certificate, we can put the certificate in PEM format (including the lines BEGIN CERTIFICATE and END CERTIFICATE) into a file inside /etc/elasticsearch/. In this case we just created /etc/elasticsearch/ldap_cert.pem.

The certificate file name needs to be added in Elasticsearch's config. Within the ldap1 realm, a new sub-option "ssl" is added under which "certificate_authorities" points to the just created pem file:

xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
        ldap:
          ldap1:
            order: 1
            url: "ldaps://ldap.example.com:636"
            ssl:
              certificate_authorities: [ "ldap_cert.pem" ]
            bind_dn: "cn=sa_elk, ou=users, o=services, dc=example, dc=com"
            bind_password: topGunSecret
            user_search:
              base_dn: "OU=Users,DC=example,DC=com"
              attribute: sAMAccountName
            group_search:
              base_dn: "OU=Groups,DC=example,DC=com"

Another restart of Elasticsearch, and this should finally work.

root@es01:~# systemctl restart elasticsearch

At least that's what I hoped for.

SSLPeerUnverifiedException(peer not authenticated)

Shortly after Elasticsearch was restarted, the following errors started to show up in the logs:

[WARN ][o.e.x.s.a.l.s.LdapUtils  ] [es01] Failed to obtain LDAP connection from pool - LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to connect to server ldap.example.com:636:  IOException(LDAPException(resultCode=91 (connect error), errorMessage='Unable to verify an attempt to to establish a secure connection to 'ldap.example.com:636' because an unexpected error was encountered during validation processing:  SSLPeerUnverifiedException(peer not authenticated), ldapSDKVersion=4.0.8, revision=28812'))')
[WARN ][o.e.x.s.a.RealmsAuthenticator] [es01] Authentication to realm ldap1 failed - authenticate failed (Caused by LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to connect to server ldap.example.com:636:  IOException(LDAPException(resultCode=91 (connect error), errorMessage='Unable to verify an attempt to to establish a secure connection to 'ldap.example.com:636' because an unexpected error was encountered during validation processing:  SSLPeerUnverifiedException(peer not authenticated), ldapSDKVersion=4.0.8, revision=28812'))'))

Ohh what now?!

After another investigation it turned out that the LDAP URL (ldap.example.com) is not a LDAP server but rather a layer4 tcp proxy in front of multiple LDAP servers. This ensures a highly available LDAP backend for Elasticsearch.

By looking at the LDAP server certificates with openssl once more, we could verify each LDAP server presents its own server certificate. This can be seen by looking for the Serial Number or the SAN (Subject Alternative Name) fields:

root@es01:~# openssl s_client -connect ldap.example.com:636 | openssl x509 -text -in -  | egrep "(Serial|X509v3 Subject Alternative Name)" -A 1
[...]
        Serial Number:
            1d:00:00:42:13:24:78:10:0b:93:f6:83:ab:00:00:00:00:42:13
--
  X509v3 Subject Alternative Name: critical
DNS:dc01.example.com, DNS:example.com, DNS:EXAMPLE

root@es01:~# openssl s_client -connect ldap.example.com:636 | openssl x509 -text -in -  | egrep "(Serial|X509v3 Subject Alternative Name)" -A 1
[...]
        Serial Number:
            1d:00:00:42:27:79:0d:f4:c9:b6:9e:2a:53:00:00:00:00:42:27
--
            X509v3 Subject Alternative Name: critical
DNS:dc03.example.com, DNS:example.com, DNS:EXAMPLE

Depending on the balancing on the layer4 proxy, we could (by chance) connect to the LDAP server we know and we installed the certificate in /etc/elasticsearch/ldap_cert.pem. But obviously taking chances is not a technical solution.

At this moment in time there were three possibilities:

• Remove the Layer4 proxy in the LDAP URL and replace it with the IP or DNS name of a LDAP server - but lose high availability
• Retrieve all server certificates from all the LDAP servers, save them as pem files and add them in /etc/elasticsearch.yml inside the certificate_authorities array -> but if new LDAP servers are added behind the layer 4 proxy or if new LDAP server certificates are installed, Elasticsearch will run into errors again
• Obtain the Root CA, which has issued all LDAP server certificates and install it in Elasticsearch

From the openssl output we could verify that all LDAP server certificates were indeed issued by the same Root CA: ACME Enterprise CA 1.

Using the Root CA certificate

As the LDAP server's did not include the Root CA in the certificate chain, it was time to involve the Windows administrators (as the LDAP servers in question are actually Windows Active Directory Domain Controllers). Once the Root CA certificate was obtained, it was saved as pem file in /etc/elasticsearch/acme_enterprise_ca1.pem and configured in Elasticsearch:

xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
        ldap:
          ldap1:
            order: 1
            url: "ldaps://ldap.example.com:636"
            ssl:
              certificate_authorities: [ "acme_enterprise_ca1.pem" ]
            bind_dn: "cn=sa_elk, ou=users, o=services, dc=example, dc=com"
            bind_password: topGunSecret
            user_search:
              base_dn: "OU=Users,DC=example,DC=com"
              attribute: sAMAccountName
            group_search:
              base_dn: "OU=Groups,DC=example,DC=com"

Another restart and this should now work, it has to!

root@es01:~# systemctl restart elasticsearch

LDAPException: Hostname verification failed

But after another Elasticsearch restart, another kind of error popped up as soon as someone wanted to authenticate against Elasticsearch's LDAP backend (using Kibana):

[WARN ][o.e.x.s.a.l.s.LdapUtils  ] [inf-elk01-t] Failed to obtain LDAP connection from pool - LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to connect to server ldap.example.com:636:  IOException(LDAPException(resultCode=91 (connect error), errorMessage='Hostname verification failed because the expected hostname 'ldap.example.com' was not found in peer certificate 'subject='' dNSName='dc04.example.com' dNSName='example.com' dNSName='EXAMPLE''.', ldapSDKVersion=4.0.8, revision=28812))')
[WARN ][o.e.x.s.a.RealmsAuthenticator] [inf-elk01-t] Authentication to realm ldap1 failed - authenticate failed (Caused by LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to connect to server ldap.example.com:636:  IOException(LDAPException(resultCode=91 (connect error), errorMessage='Hostname verification failed because the expected hostname 'ldap.example.com' was not found in peer certificate 'subject='' dNSName='dc04.example.com' dNSName='example.com' dNSName='EXAMPLE''.', ldapSDKVersion=4.0.8, revision=28812))'))

ARGH! Whyyy!

Elasticsearch throws this error because the configured LDAP URL (ldap.example.com) didn't match the certificate names, which show the LDAP servername. Unfortunately the server certificates don't contain ldap.example.com as an alternative name (SAN) either.

Luckily there's a (configuration) workaround for this and this is even mentioned in the Elasticsearch LDAP documentation at the very bottom:

By default, when you configure Elasticsearch to connect to an LDAP server using SSL/TLS, it attempts to verify the hostname or IP address specified with the url attribute in the realm configuration with the values in the certificate. If the values in the certificate and realm configuration do not match, Elasticsearch does not allow a connection to the LDAP server. This is done to protect against man-in-the-middle attacks. If necessary, you can disable this behavior by setting the ssl.verification_mode property to certificate.

This situation applies to this Elasticsearch setup and the "verification_mode" options was added under the "ssl" option (on the same level as "certificate_authorities"):

xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
        ldap:
          ldap1:
            order: 1
            url: "ldaps://ldap.example.com:636"
            ssl:
              certificate_authorities: [ "acme_enterprise_ca1.pem" ]
              verification_mode: certificate
            bind_dn: "cn=sa_elk, ou=users, o=services, dc=example, dc=com"
            bind_password: topGunSecret
            user_search:
              base_dn: "OU=Users,DC=example,DC=com"
              attribute: sAMAccountName
            group_search:
              base_dn: "OU=Groups,DC=example,DC=com"

Finally, LDAPS authentication is working

Another restart of Elasticsearch:

root@es01:~# systemctl restart elasticsearch

And finally the authentication worked again! Users were again able to login in Kibana, which then uses the Elasticsearch user authentication backend - in this case the ldap1 realm.

Add a comment

Comments (newest first)

No comments yet.