Solvedansible json_query filter fails when using the functions "contains", "starts_with", others

ISSUE TYPE
  • Bug Report
COMPONENT NAME

json_query

ANSIBLE VERSION
ansible 2.3.1.0                                                                
  config file = /etc/ansible/ansible.cfg                                       
  configured module search path = [u'/usr/share/my_modules/']                  
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
CONFIGURATION

Standard

OS / ENVIRONMENT

N/A

SUMMARY

When using the json_query filter with the query [?interface.name=='irb.1111'].{interface: interface.name, address: address} everything functions fine and I get the correct results back. When that filter is changed to [?contains(interface.name,'irb')].{interface: interface.name, address: address} traceback errors with jmespath/functions.py start occurring that complain about invalid type for value.

A previous bug (#20379) along the same lines was opened back in January against Ansible v2.2.1, but hasn't been resolved yet.

STEPS TO REPRODUCE
- name: json_query testing
  hosts: localhost
  connection: local
  gather_facts: no

  vars:
    json_file: "{{lookup('file','results.json') | from_json }}"
    jq_old: "[?interface.name=='irb.1111'].{interface: interface.name, address: address}"
    jq_new: "[?contains(interface.name,'irb')].{interface: interface.name, address: address}"

  tasks:

    - debug:
        var: item
      with_items: "{{ json_file.results | json_query(jq_old) }}"

Provided results.json

That works as expected with the following results:

ok: [localhost] => (item={u'interface': u'irb.1111', u'address': u'10.0.1.1/29'}) => {
    "item": {
        "address": "10.0.1.1/29",
        "interface": "irb.1111"
    }
}

However, when you change the line with_items in the PB above to use jq_new instead of jq_old, this traceback occurs:

The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 89, in run
    items = self._get_loop_items()
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 202, in _get_loop_items
    loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar, loader=self._loader, fail_on_undefined=True, convert_bare=False)
  File "/usr/local/lib/python2.7/dist-packages/ansible/utils/listify.py", line 34, in listify_lookup_plugin_terms
    terms = templar.template(terms.strip(), convert_bare=convert_bare, fail_on_undefined=fail_on_undefined)
  File "/usr/local/lib/python2.7/dist-packages/ansible/template/__init__.py", line 437, in template
    disable_lookups=disable_lookups,
  File "/usr/local/lib/python2.7/dist-packages/ansible/template/__init__.py", line 659, in do_template
    res = j2_concat(rf)
  File "<template>", line 10, in root
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/filter/json_query.py", line 40, in json_query
    return jmespath.search(expr, data)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/__init__.py", line 12, in search
    return parser.Parser().parse(expression).search(data, options=options)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/parser.py", line 509, in search
    result = interpreter.visit(self.parsed, value)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/visitor.py", line 94, in visit
    return method(node, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/visitor.py", line 180, in visit_filter_projection
    if self._is_true(self.visit(comparator_node, element)):
  File "/usr/local/lib/python2.7/dist-packages/jmespath/visitor.py", line 94, in visit
    return method(node, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/visitor.py", line 171, in visit_function_expression
    return self._functions.call_function(node['value'], resolved_args)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/functions.py", line 80, in call_function
    self._validate_arguments(resolved_args, signature, function_name)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/functions.py", line 91, in _validate_arguments
    return self._type_check(args, signature, function_name)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/functions.py", line 98, in _type_check
    function_name)
  File "/usr/local/lib/python2.7/dist-packages/jmespath/functions.py", line 113, in _type_check_single
    self._convert_to_jmespath_type(actual_typename), types)
JMESPathTypeError: In function contains(), invalid type for value: irb.1111, expected one of: ['array', 'string'], received: "unknown"

fatal: [localhost]: FAILED! => {
    "failed": true,
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}
EXPECTED RESULTS

Something along the lines of (from http://jmespath.org):

[
  {
    "interface": "irb.1111",
    "address": "10.0.1.1/29"
  },
  {
    "interface": "irb.1114",
    "address": "10.0.2.1/29"
  },
  {
    "interface": "irb.1114",
    "address": "fe80::b/112"
  }
]
ACTUAL RESULTS

See above for the traceback that occured.

$ ansible-playbook jsonquery.yml
 [WARNING]: Host file not found: /etc/ansible/hosts                                                    
                                                                                                       
 [WARNING]: provided hosts list is empty, only localhost is available                                  
                                                                                                       
                                                                                                       
PLAY [json_query testing] *****************************************************************************
                                                                                                       
TASK [debug] ******************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: JMESPa
thTypeError: In function contains(), invalid type for value: irb.1111, expected one of: ['array', 'stri
ng'], received: "unknown"                                                                              
fatal: [localhost]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "
stdout": ""}                                                                                           

PLAY RECAP ********************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1                            
31 Answers

✔️Accepted Answer

The problem is related to the fact that Ansible uses own types for strings: AnsibleUnicode and AnsibleUnsafeText.
And as long as jmespath library has very strict type-checking, it fails to accept this types as string literals.

There was a clever hint posted here: #20379 (comment)
It is used to alter jmespath's typemap to accept Ansible custom types.

I was going to file an issue, but pasting my repro-code here instead:

- hosts: localhost
  gather_facts: no
  vars:
    qry1: "[?contains(foo, 'bar')]"
    qry2: "[?contains(item, '1')].cmd"
    json1:
      - foo: bar
      - foo: test
  tasks:
    - name: This fails because type(foo) = AnsibleUnicode
      debug:
        msg: "{{ json1 | json_query(qry1) }}"
      ignore_errors: yes

    - name: This works
      debug:
        msg: "{{ json1 | to_json | from_json | json_query(qry1) }}"

    - name: This generates registered facts that gets SafeWrapped
      command: echo {{ item }}
      with_sequence: count=2
      register: cmd_res
      changed_when: no

    - name: This fails because type(foo) = AnsibleUnsafeText
      debug:
        msg: "{{ cmd_res.results | json_query(qry2) }}"
      ignore_errors: yes

    - debug:
        msg: "{{ cmd_res.results | to_json | from_json | json_query(qry2) }}"

Other Answers:

| to_json | from_json workaround works for me with join function. Thx @berlic

Just two years later, this issue is still present in ansible 2.8.1.
At least a note in documentation would be great, as it would save an hour of troubleshooting to every single person hitting this road block.

Is there anyone who can/know how to submit an improvement for ansible doc?

I think if noone's going to fix this, can someone at least put a note in the json_query documentation about using to_json | from_json when using JMESPath functions/methods?

I have a good news and a bad news.
I have a fix/workaround but.. I don't know why it works

to fix, in json_query.py add import ast then add data = ast.literal_eval(str(data)) in json_query()

It convert the original data, which is a "dict" and when printed, looks good, to a string and convert it back to dict.

I'm not understanding the fix but it works.. I'll not make a PR has I have no explanation and also because the fix may have to be done elsewhere.
At the end may be the issue come from Jinja. I assume json_query() is called from jinja, could it be that Jinja manipulate 'data' in a way who change it's internal representation for python and produce this ?

I attach my json_query.py patched version. @bdlamprecht @Imran-ibm you may want to try it
json_query.zip

Related Issues:

143
ansible module_stdout: "/bin/sh: 1: /usr/bin/python: not found\r\n",
Just use ansible_python_interpreter=/usr/bin/python3 in ur inventory file ansible -m ping -u ubuntu ...
88
ansible error in cryptography setup command: Invalid environment marker: python_version < '3'
I had the same problem in Debian Jessie This is what I did to get it working for me: After this I wa...
88
ansible OSX crash complaining of operation in progress in another thread when fork() was called
This is apparently due to some new security changes made in High Sierra that are breaking lots of Py...
72
ansible Failed to connect to the host via ssh: Permission denied (publickey,password)
Good It's a bit hard to debug when you specify all in your command I have this error I use Debian St...
71
ansible Describe how to use "postgresql_user" properly with ansible >= 2.1.0.0
I managed to get this temporarily working with pipelining per task and becoming postgres user: Hopef...
54
ansible ansible unable to find boto: boto required for this module
@stevenscg still working me with this in my inventory file: Let me know if that does anything for yo...
43
ansible why is ansible's default output not more human readable... stilll?
Ansible 2.4+ has built-in support for human-readable results: Temporarily by setting ANSIBLE_STDOUT_...
37
ansible Reboot and Wait for
An update of the docs and/or the support article to use the preferred full YAML format for tasks wou...
37
ansible ERROR! Timeout (12s) waiting for privilege escalation prompt:
Just as a note I switched the connection over to paramiko and the issue went away and the playbook r...
33
ansible Failed to import docker-py for docker_container module
docker-py is just the name of the project It installs a python package named docker ...
32
ansible json_query filter fails when using the functions "contains", "starts_with", others
The problem is related to the fact that Ansible uses own types for strings: AnsibleUnicode and Ansib...
31
ansible feature: controlling ignore-errors output
From a UX perspective it seems reasonable to give visual distinction between explicitly ignored erro...
30
ansible Support specifying collections in git repositories in requirements.yml
This has become much more frustrating lately SUMMARY When I develop collections I like to store them...
29
ansible SSH works, but ansible throws unreachable error
This happende all of a sudden when I upgraded Ansible ISSUE TYPE Bug Report ANSIBLE VERSION CONFIGUR...
29
ansible Ansible evaluates with_items for tasks in blocks skipped by the block when condition
For anyone who finds this in future the way to have this work without the warning is to use with_ite...
29
ansible "template error while templating string: Missing end of comment tag" error
EDIT: When unsafe characters are defined in vars follow @inossidabile's recommendation to use !unsaf...
24
ansible ansible-galaxy should download dependencies in meta/main.yml
I heavily work with dependencies and meta/main.yml and it would be great to spare the necessity to m...
24
ansible Add an option lock_wait to the apt module
This should integrate with systemd ISSUE TYPE Feature Idea This is a copy of the issue on the old re...
23
ansible Windows 10/WSL: Ansible cannot read ansible.cfg from NTFS mounts
I think I found a solution for 2.6.1 and so on.. SUMMARY Ansible 2.6.1 added #42070 which makes Ansi...
20
ansible Anisble does not allow handling of "host unreachable" errors
Does anyone else agree we need to revisit how we are handling unreachable errors? We have a use case...
20
ansible delegate_to not propagated to include_role
I would say this is a huge issue If Ansible would have raised an error for combination of delegate_t...
18
ansible shared connection closed
Same for me on macOS: ISSUE TYPE Bug Report COMPONENT NAME Script module ANSIBLE VERSION CONFIGURATI...
16
ansible Handle omit value in task attributes (like environment or become_user)
I too am interested in something similar to this In my use case we use the same playbook for multipl...
15
ansible FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}
This happened to me after my internet connect dropped while running a playbook I fixed it by running...
14
ansible apt_key module ignores the proxy environment
I'm using this as a workaround: ISSUE TYPE Bug Report COMPONENT NAME apt_key ANSIBLE VERSION SUMMARY...
13
ansible Failure in apt. "Please install python-apt", but it is installed
I ran into this issue using the local connection mode -c local using ansible from a virtualenv ...
12
ansible [mac os x] ansible-galaxy: "unexpected Exception: name must be a byte string" when installing from requirements file
Upgrading urllib3 solved this problem for me: sudo pip install --upgrade urllib3 ...
12
ansible Support apt-mark hold
Full working example for reference from Ubuntu 16.04 and docker: From @scottnonnenberg on September ...
12
ansible Single Vault Encrypted value not decrypted in jinja2 pipeline
It still not work for password_hash It need to add string before using password_hash ...
12
ansible file touch always 'changed' - [was: need a separate touch module]
FYI: In Ansible 2.7 was added access_time and modification_time so you can use that to avoid change ...
12
ansible podman support (podman_container)
I am working on the following modules for inclusion in TripleO: podman_image podman_container I also...
11
ansible Inventory script does not work with assumed roles from the command line
For me the fix was to set AWS_SECURITY_TOKEN to the same value as AWS_SESSION_TOKEN ...
11
ansible failed to transfer file to ~/.ansible/tmp/ansible-tmp-xxx/setup.py: [Errno 2] No such file or directory
Same issue here with 2.2.1 (ok with 2.2.0) ISSUE TYPE Bug Report COMPONENT NAME ansible-playbook set...
11
ansible mysql_user broken in 2.7.1 when using /root/.my.cnf
Ok I found it It was a discussion on #ansible-devel on October 2nd SUMMARY When upgrading from 2.7.0...
8
ansible k8s module throwing 'This module requires the OpenShift Python client. Try pip install openshift'
So in my case it was an annoying Requests-related exception (actually just a RequestsDependencyWarni...
6
ansible synchronize: rsync_opts broken/changed in ansible 2.3.0
rsync cmd: BAD (ansible 2.3.0) GOOD (ansible 2.2.2.0) ISSUE TYPE Bug Report COMPONENT NAME synchroni...
5
ansible Issues in template module
Maybe you can add -K option for ansible-playbook command I fixed this problem in my case. ...
4
ansible (P1) nxos* modules timeout sending long running command for transport == cli
@mikewiebe One possible way is: provider: {{ connection | combine({'timeout': 400}) }} ...
3
ansible windows 8.1 .net 3.5 installation: raw, win_chocolatey, win_webpicmd
I ran into this on Server 2012 The easiest solution I found was this: Per this MSDN page Edited for ...
3
ansible pywinrm fails to authenticate from centos 7 host to windows 2012 R2 client
Had the same issue Fixed by uninstalling pyOpenSSL completely (cleaning folders like @darioems sugge...
3
ansible HaProxy drain mode 'bool' object is not callable error
@alikins I looked into the issue today ISSUE TYPE Bug Report COMPONENT NAME HaProxy Module ANSIBLE V...
3
ansible Add possibility to set up several ips for hostname in module ipa_dnsrecord
Are you thinking a format something like: ISSUE TYPE Feature Idea COMPONENT NAME ipa_dnsrecord ANSIB...
82
drupal vm Composer install fails without proper swap
or you can create a swap file sudo fallocate -l 2G /swapfile sudo chmod 600 /swapfile sudo mkswap /s...
77
kubespray After the certificate expires how use kubespray to renew certificate
@kerOssinas you are right the upgrade-cluster.yml of Kubespray will also rotate the certificates ...
32
kubespray Current install documentation is incorrect and does not work due to inventory script changes
@elfiii good luck. The install/usage documentation here: https://github.com/kubernetes-sigs/kubespra...
31
ansible elasticsearch Permissions on elasticsearch.keystore prevent Elasticsearch from starting
This entire problem is being caused by an incorrect mixing of static read-only configuration (elasti...
29
drupal vm Failing to install Drupal on macOS High Sierra - NFS filesystem issues
@ajhoddinott OMG That works thank you! For explicit instructions on Mac OS High Sierra open the app ...
28
kubespray etcd cluster is unavailable or misconfigured: connection refused
Run on master nodes: Run no all nodes: btw SELinux is working fine i did not had to do any adjustmen...
22
kubespray Unable to add new master/etcd node to cluster
You should be able to In the past we managed to replace all nodes in the cluster: master etcd and wo...
21
ansible lint Re-evaluate E0010 - Package installs should not use latest
The official Ansible yum module docs prominently recommend using state=latest with name=* to update ...