Elasticsearch mapping - different data types in same field

Question

I am trying to to create a mapping that will allow me to have a document looking like this:

{
    "created_at" : "2014-11-13T07:51:17+0000",
    "updated_at" : "2014-11-14T12:31:17+0000",
    "account_id" : 42,
    "attributes" : [
    {
        "name" : "firstname",
        "value" : "Morten",
        "field_type" : "string"
    },
    {
        "name" : "lastname",
        "value" : "Hauberg",
        "field_type" : "string"
    },
    {
        "name" : "dob",
        "value" : "1987-02-17T00:00:00+0000",
        "field_type" : "datetime"
    }
]

}

And the attributes array must be of type nested, and dynamic, so i can add more objects to the array and index it by the field_type value.

Is this even possible?

I have been looking at the dynamic_templates. Can i use that?

Answer 1

No - you cannot have different datatypes for the same field within the same type.

e.g. the field index/type/value can not be both a string and a date.

---

A dynamic template can be used to set the datatype and analyzer based on the format of the field name

For example: set all fields with field names ending in "_dt" to type datetime.

But this won't help in your scenario, once the datatype is set you can't change it.

Answer 2

You actually can index multiple datatypes into the same field using a multi-field mapping and the ignore_malformed parameter, if you are willing to query the specific field type if you want to do type specific queries (like comparisons).

This will allow elasticsearch to populate the fields that are pertinent for each input, and ignore the others. It also means you don’t need to do anything in your indexing code to deal with the different types.

For example, for a field called user_input that you want to be able to do date or integer range queries over if that is what the user has entered, or a regular text search if the user has entered a string, you could do something like the following:

PUT multiple_datatypes
{
  "mappings": {
    "_doc": {
      "properties": {
        "user_input": {
          "type": "text",
          "fields": {
            "numeric": {
              "type": "double",
              "ignore_malformed": true
            },
            "date": {
              "type": "date",
              "ignore_malformed": true
            }
          }
        }
      }
    }
  }
}

We can then add a few documents with different user inputs:

PUT multiple_datatypes/_doc/1
{
  "user_input": "hello"
}

PUT multiple_datatypes/_doc/2
{
  "user_input": "2017-02-12"
}

PUT multiple_datatypes/_doc/3
{
  "user_input": 5
}

And when you search for these, and have ranges and other type-specific queries work as expected:

// Returns only document 2
GET multiple_datatypes/_search
{
  "query": {
    "range": {
      "user_input.date": {
        "gte": "2017-01-01"
      }
    }
  }
}

// Returns only document 3
GET multiple_datatypes/_search
{
  "query": {
    "range": {
      "user_input.numeric": {
        "lte": 9
      }
    }
  }
}

// Returns only document 1
GET multiple_datatypes/_search
{
  "query": {
    "term": {
      "user_input": {
        "value": "hello"
      }
    }
  }
}

I wrote about this as a blog post here