Skip to content

CONCEPT Cited by 1 source

Nested-sort path requirement

Definition

When sorting Elasticsearch search results on a nested field (a field inside an Elasticsearch nested type), the sort clause must include a nested.path sub-clause naming the path to the nested parent document. This is documented both in ES 7.17 and ES 8 reference material as a requirement, but ES 7.x did not enforce it — it accepted sort clauses on nested fields with no path sub-clause and returned results. ES 8.x does enforce it: such queries now fail with:

{
  "type": "query_shard_exception",
  "reason": "it is mandatory to set the [nested] context on the nested sort field: [trace.origami.timestamp]."
}

The two sort-clause forms

Incomplete (accepted in 7.x, rejected in 8.x):

"sort": [
  {
    "trace.origami.timestamp": {
      "order": "desc"
    }
  }
]

Correct (accepted in both 7.x and 8.x):

"sort": [
  {
    "trace.origami.timestamp": {
      "order": "desc",
      "nested": {
        "path": "trace"
      }
    }
  }
]

Why ES 7.x accepted the incomplete form

Confirmed on the Elastic discussion board as a bug in ES 7: the query-shard-level validation check for the nested.path field was missing. The docs always said the path was required; the implementation did not enforce it. ES 8 fixed the bug, which therefore appears as a breaking change from the perspective of any query that happened to rely on the 7.x laxness.

Seen in

  • sources/2023-11-19-zalando-migrating-from-elasticsearch-7-to-8 — Canonical wiki instance. Zalando's Origami e2e integration tests against ES 8 began failing with this exact query_shard_exception. The fix — adding nested: { path: "trace" } to the sort clauses — was mechanical but had to be done exhaustively across all sort clauses on nested fields. Zalando surfaced the bug on the Elastic forum, and Elastic confirmed it was a known fixed bug, not a policy change.

Generalisation

This is the canonical instance of undocumented-behavior drift on version upgrade: the spec / docs said one thing, the ES 7.x implementation was more permissive, ES 8 aligned with the spec, and queries that relied on the implementation's laxness broke. Any major-version upgrade of a query engine risks this class of regression. The pre-upgrade discipline is:

Last updated · 501 distilled / 1,218 read