Solvedansible elasticsearch Permissions on elasticsearch.keystore prevent Elasticsearch from starting

From 6.2, and perhaps earlier, it appears the elasticsearch.keystore file is created even if X-Pack security isn't enabled. This then prevents Elasticsearch from starting up

Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/0/elasticsearch.keystore

from tasks/xpack/elasticsearch-xpack.yml

#Make sure elasticsearch.keystore has correct Permissions
- name: Set elasticsearch.keystore Permissions
  become: yes
  file: state=file path={{ conf_dir }}/elasticsearch.keystore owner={{ es_user }} group={{ es_group }}
  when: es_enable_xpack and "security" in es_xpack_features and (es_version | version_compare('6.0.0', '>'))

I'd suggest removing "security" in es_xpack_features.
I can create a PR to relax the conditional, and without testing back on 6.x versions ignore errors. Though based on this and other issues, it seems we need some more automated tests around X-Pack features in enabled/disabled states. I will try to look into this when I get some time.

18 Answers

✔️Accepted Answer

This entire problem is being caused by an incorrect mixing of static, read-only configuration (elasticsearch.yml, jvm.options, with writable, run-time files (elasticsearch.keystore).

The /etc directory, and its subdirectories, should not need to be writable by non-privileged users. That's what /var and /tmp are for.

My contention is that elasticsearch.keystore is being improperly stored in /etc/elasticsearch/, when it should really be in something like /var/lib/elasticsearch/, which would be owned by the elasticsearch user. Unfortunately, the location of elasticsearch.keystore does not seem to be configurable.

Other Answers:

Confirming that the issue still exists in 6.3.2, when I tried to start ES after upgrading from 5.6.
In my case, there was no write permission to /etc/elasticsearch, had to explicitly set both guid and w,

chmod g+ws /etc/elasticsearch/

I agree with @kzalewski . I am running elasticsearch in kubernetes, providing its configuration via a ConfigMap. ConfigMaps are read-only, meaning that if I mount it as a directory, and point elasticsearch to that directory as the config location, elasticsearch fails to work. This is a very simple deployment method, and I would really expect it to work.

At the very least, there should be a method to override the keystore location. But, it should probably be completely decoupled from static config by default.

Related Issues:

ansible elasticsearch Permissions on elasticsearch.keystore prevent Elasticsearch from starting
This entire problem is being caused by an incorrect mixing of static read-only configuration (elasti...
ansible module_stdout: "/bin/sh: 1: /usr/bin/python: not found\r\n",
Just use ansible_python_interpreter=/usr/bin/python3 in ur inventory file ansible -m ping -u ubuntu ...
ansible error in cryptography setup command: Invalid environment marker: python_version < '3'
I had the same problem in Debian Jessie This is what I did to get it working for me: After this I wa...
ansible OSX crash complaining of operation in progress in another thread when fork() was called
This is apparently due to some new security changes made in High Sierra that are breaking lots of Py...
drupal vm Composer install fails without proper swap
or you can create a swap file sudo fallocate -l 2G /swapfile sudo chmod 600 /swapfile sudo mkswap /s...
kubespray After the certificate expires how use kubespray to renew certificate
@kerOssinas you are right the upgrade-cluster.yml of Kubespray will also rotate the certificates ...
ansible Failed to connect to the host via ssh: Permission denied (publickey,password)
Good It's a bit hard to debug when you specify all in your command I have this error I use Debian St...
ansible Describe how to use "postgresql_user" properly with ansible >=
I managed to get this temporarily working with pipelining per task and becoming postgres user: Hopef...
ansible ansible unable to find boto: boto required for this module
@stevenscg still working me with this in my inventory file: Let me know if that does anything for yo...
ansible why is ansible's default output not more human readable... stilll?
Ansible 2.4+ has built-in support for human-readable results: Temporarily by setting ANSIBLE_STDOUT_...
ansible Reboot and Wait for
An update of the docs and/or the support article to use the preferred full YAML format for tasks wou...
ansible ERROR! Timeout (12s) waiting for privilege escalation prompt:
Just as a note I switched the connection over to paramiko and the issue went away and the playbook r...
ansible Failed to import docker-py for docker_container module
docker-py is just the name of the project It installs a python package named docker ...
ansible json_query filter fails when using the functions "contains", "starts_with", others
The problem is related to the fact that Ansible uses own types for strings: AnsibleUnicode and Ansib...
kubespray Current install documentation is incorrect and does not work due to inventory script changes
@elfiii good luck. The install/usage documentation here:
ansible feature: controlling ignore-errors output
From a UX perspective it seems reasonable to give visual distinction between explicitly ignored erro...
ansible Support specifying collections in git repositories in requirements.yml
This has become much more frustrating lately SUMMARY When I develop collections I like to store them...
ansible SSH works, but ansible throws unreachable error
This happende all of a sudden when I upgraded Ansible ISSUE TYPE Bug Report ANSIBLE VERSION CONFIGUR...
ansible Ansible evaluates with_items for tasks in blocks skipped by the block when condition
For anyone who finds this in future the way to have this work without the warning is to use with_ite...
ansible "template error while templating string: Missing end of comment tag" error
EDIT: When unsafe characters are defined in vars follow @inossidabile's recommendation to use !unsaf...
drupal vm Failing to install Drupal on macOS High Sierra - NFS filesystem issues
@ajhoddinott OMG That works thank you! For explicit instructions on Mac OS High Sierra open the app ...
kubespray etcd cluster is unavailable or misconfigured: connection refused
Run on master nodes: Run no all nodes: btw SELinux is working fine i did not had to do any adjustmen...
ansible ansible-galaxy should download dependencies in meta/main.yml
I heavily work with dependencies and meta/main.yml and it would be great to spare the necessity to m...
ansible Add an option lock_wait to the apt module
This should integrate with systemd ISSUE TYPE Feature Idea This is a copy of the issue on the old re...
ansible Windows 10/WSL: Ansible cannot read ansible.cfg from NTFS mounts
I think I found a solution for 2.6.1 and so on.. SUMMARY Ansible 2.6.1 added #42070 which makes Ansi...
kubespray Unable to add new master/etcd node to cluster
You should be able to In the past we managed to replace all nodes in the cluster: master etcd and wo...
ansible lint Re-evaluate E0010 - Package installs should not use latest
The official Ansible yum module docs prominently recommend using state=latest with name=* to update ...
ansible Anisble does not allow handling of "host unreachable" errors
Does anyone else agree we need to revisit how we are handling unreachable errors? We have a use case...
ansible delegate_to not propagated to include_role
I would say this is a huge issue If Ansible would have raised an error for combination of delegate_t...
ansible shared connection closed
openshift ansible Could not resolve host:; Unknown error
Any updates on this? I have the exact same issue on an oc cluster up on CentOS 7 ...
algo Windows WSL: "Ansible is being run in a world writable directory"
Try to correct the problem by running the following while in the algo directory: ...
ansible Handle omit value in task attributes (like environment or become_user)
I too am interested in something similar to this In my use case we use the same playbook for multipl...
ansible FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}
This happened to me after my internet connect dropped while running a playbook I fixed it by running...
awx Error with pip-docker (3.0.1): "Error connecting: load_config() got an unexpected keyword argument 'config_dict'"
For me even installing docker 3.1.3 resulted in errors What I had to do was: And the error stopped. ...
openshift ansible No package matching 'origin-docker-excluder-3.10**' found available, installed or updated
Currently with that running openshift-ansible from master for all-on-one setup works internal reprod...
ansible apt_key module ignores the proxy environment
I'm using this as a workaround: ISSUE TYPE Bug Report COMPONENT NAME apt_key ANSIBLE VERSION SUMMARY...
awx Playbook fails when ssh host key changes
Giant pain in Ansible Tower Anytime an IP is recycled we've got manually clear it from the known_hos...
molecule master: pip install -e . fails with TypeError: expected str, bytes or os.PathLike object, not NoneType
upgrade your pip via curl | python - I had same problem and I w...
ansible Failure in apt. "Please install python-apt", but it is installed
I ran into this issue using the local connection mode -c local using ansible from a virtualenv ...
ceph ansible ceph luminous with bluestore add osd failed
The problem here is that your old OSD auth details are with mon If you remove the OSD entry and its ...
drupal vm 503 Service Unavailable
@TheNugg Even I was facing the same issue I was not able access Drupal website it was throwing 503 e...
algo DigitalOcean: Error creating tags
Here's a patch file you can use to get going again Save the following patch to your algo directory i...
chrome aws lambda [BUG] Failed to launch chrome
I got this to work by only using await chromium.executablePath in prod as I know this issue has been...
molecule Make Ansible default verifier
I see some value in having a tool that is NOT Ansible to verify something that was modified by Ansib...
ansible [mac os x] ansible-galaxy: "unexpected Exception: name must be a byte string" when installing from requirements file
Upgrading urllib3 solved this problem for me: sudo pip install --upgrade urllib3 ...
ansible Support apt-mark hold
Full working example for reference from Ubuntu 16.04 and docker: From @scottnonnenberg on September ...
ansible Single Vault Encrypted value not decrypted in jinja2 pipeline
It still not work for password_hash It need to add string before using password_hash ...
ansible file touch always 'changed' - [was: need a separate touch module]
FYI: In Ansible 2.7 was added access_time and modification_time so you can use that to avoid change ...
ansible podman support (podman_container)
I am working on the following modules for inclusion in TripleO: podman_image podman_container I also...