Monthly Archives: October 2016

Custom Elasticsearch template with custom field mapping

Disclaimer: I am writing this post mostly for me to remember how and what I did to solve wrong field mapping in elasticsearch.

Quote from Elasticsearch Dynamic Mapping documentation page

When Elasticsearch encounters a previously unknown field in a document, it uses dynamic mapping to determine the datatype for the field and automatically adds the new field to the type mapping.

Sometimes this is the desired behavior and sometimes it isn't. Perhaps you don't know what fields will be added to your documents later, but you want them to be indexed automatically. Perhaps you just want to ignore them. Or - especially if you are using Elasticsearch as a primary data store - perhaps you want unknown fields to throw an exception to alert you to the problem.

I had this annoying problem with a field that was mapped as date when it was supposed to be a string.

It is worth mentioning that existing type and field mappings cannot be updated because that would mean invalidating already indexed documents and the right way would be to create a new index with the correct mappings and reindex the data into that index.

Ok, cool, but how do I tell elasticsearch to map that particular field to string? Well, we need to create a template that will automatically be applied when new indices are created. The template could include both settings and mappings, and a simple pattern template that controls whether the template should be applied to the new index.

Let's get to work!

What is the full path of my field (stats_dates in this case)?
[codesyntax lang="bash"]

curl -s -XGET http://localhost:9200/logstash-2016.10.05/_mappings | jq 'path(recurse(if type|. == "array" or . =="object" then .[] else empty end))'

[/codesyntax]

[
...
  "logstash-2016.10.05",
  "mappings",
  "hemlock",
  "properties",
  "stats_dates",
  "type"
]
...

Ok, so the full path is mappings.hemlock.properties.stats_dates.type. Now that we the full path, let's create the template.
[codesyntax lang="bash"]

curl -s -XPUT http://localhost:9200/_template/logstash-stats_dates -d '

{
  "order": 0,
  "template": "logstash-*",
  "settings": {
  },
  "mappings": {
    "hemlock": {
      "properties": {
        "stats_dates": {
          "type": "string"
        }
      }
    }
  },
  "aliases": {
  }
}

'

[/codesyntax]

Check if the template was applied to the new index:

[codesyntax lang="bash"]

curl -s http://localhost:9200/logstash-2016.10.06/_mappings | jq '.[].mappings.hemlock.properties.stats_dates'

[/codesyntax]

{
  "type": "string"
}