Elasticsearch: unknown setting [xpack notification slack account my-watcher url]

Written by Claudio Kuenzler - 0 comments

Published on October 12th 2021 - Listed in Elasticsearch ELK

---

After upgrading Elasticsearch from 6.8.x to 7.15.x, xpack.notification settings in elasticsearch.yml resulted in an error during start of Elasticsearch:

java.lang.IllegalArgumentException: unknown setting [xpack.notification.slack.account.my-watcher.url] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Slack notifications with X-Pack

X-Pack allows to configure notifications sent by Elasticsearch directly to Slack. In the past this could be configured by defining an xpack.notification.slack snippet in /etc/elasticsearch/elasticsearch.yml:

# This worked fine with Elasticsearch 6.x
xpack.notification.slack:
  account:
    my-watcher:
      url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      message_defaults:
        from: watcher

But this very same config now prevents Elasticsearch from starting.

Breaking change!

Once more, I ran into a breaking change with the newest Elasticsearch 7.15 release, after seeing a couple of them in the past few days during the TEST upgrade:

• Elasticsearch error: The index was created with version [x.x.x] but the minimum compatible version is [x.x.x]
• Elasticsearch X-Pack error after upgrade: Incorrect realm settings found
• Elasticsearch failed on parsing mappings on index creation ([_default_] mappings are not allowed)
  

But in this situation, concerning the xpack notification, the breaking changes documentation is actually more confusing than straightforward:

By reading this, it actually sounds pretty easy; just replace <url> with <secure_url>, right? Ha! No!

The catch is that secure_url is not allowed to be defined in elasticsearch.yml!

# Do not use this, secure_url is not allowed in elasticsearch.yml!
xpack.notification.slack:
  account:
    my-watcher:
      secure_url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      message_defaults:
        from: watcher

If you do, the following error will show up during Elasticsearch start (and prevents ES from starting):

[2021-10-12T15:30:55,760][ERROR][o.e.b.Bootstrap          ] [elastic01] Exception
java.lang.IllegalArgumentException: Setting [xpack.notification.slack.account.my-watcher.secure_url] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml

In order to continue, disable the xpack.notification.slack settings from elasticsearch.yml and restart Elasticsearch. 

Save secure_url in Elasticsearch's keystore

To prevent having some clear text secure information in the config file, certain settings (such as passwords) have been moved into Elasticsearch itself. Whether this is more secure or not is questionable, but that's the case. The "keystore" can be executed on the command line, the binary should be located under /usr/share/elasticsearch/bin/elasticsearch-keystore (at least in deb packages).

The elasticsearch-keystore command is executed with an additional input - setting the "key name", such as it would be set in elasticsearch.yml:

root@elastic01:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.notification.slack.account.my-watcher.secure_url
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for xpack.notification.slack.account.my-watcher.secure_url: [paste url here]

The command asks for an additional input as a prompt. Copy the previous "url" and paste it here. Note that the input here is not shown (not even with asterisks), as it is considered a secure input (as a password).

Now you can enable the xpack.notification.slack configuration in /etc/elasticsearch/elasticsearch.yml again and do another Elasticsearch restart:

xpack.notification.slack:
  account:
    my-watcher:
      message_defaults:
        from: watcher

root@elastic01:~# systemctl restart elasticsearch.service

This time, Elasticsearch should start up correctly and read the relevant "secure_url" settings from the keystore.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.