Make flattened synthetic source concatenate object keys on scalar/object mismatch #129600

parkertimmins · 2025-06-18T03:42:23Z

There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source.

This fixes the issue by replacing the problematic object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by .. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.

For example if the following flattened field values is indexed:

{
   "a": {
       "b": 5
       "b": {
          "c": 10,
          "d": { "e": 15 }
       }
   }
}

The following synthetic source will be produced:

{
   "a": {
      "b": 5
      "b.c": 10,
      "b.d.e": 15
   }
}

Fixes #122936

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java

kkrik-es · 2025-06-18T08:35:53Z

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java

+            next = nextValue == null ? KeyValue.EMPTY : new KeyValue(nextValue);
+
+            var startPrefix = curr.prefix.diff(openObjects);
+            if (startPrefix.prefix.isEmpty() == false && startPrefix.prefix.getFirst().equals(lastScalarSingleLeaf)) {


What if the conflict doesn't happen on the first part of the prefix, e.g.

field { path { to: 10 to { foo: bar } } }

Would this be caught here?

Yep, so this will become:

field { path { to: 10 to.foo: bar } }

When it get to the first key/value field.path.to|10 it will take the else block and traverse down into the object, adding field and path to the openObject context. When it reaches the key value field.path.to.foo|bar that object will still be open, and seeing that lastScalarSingleLeaf has a value of to, and that to is the first token in the startPrexix (to.foo), it will make a concatenated path.

(Updated a test to this situation to verify it)

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java

parkertimmins · 2025-06-18T14:43:45Z

.../java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelperTests.java

@@ -103,7 +103,7 @@ public void testSingleObject() throws IOException {

        // THEN
        assertEquals(
-            "{\"a\":\"value_a\",\"a\":{\"b\":\"value_b\",\"b\":{\"c\":\"value_c\"},\"d\":\"value_d\"}}",
+            "{\"a\":\"value_a\",\"a.b\":\"value_b\",\"a.b.c\":\"value_c\",\"a.d\":\"value_d\"}",


Interestingly, there was a test which had a scalar/object mismatch. But it produced duplicate keys. When the xcontent was converted to jsont these duplicate keys originally threw an error, but now just drop the duplicates. (Something must have changed in xcontent stuff since the issue was opened to cause this change from an error to deduplication)

elasticsearchmachine · 2025-06-18T19:48:56Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2025-06-18T19:59:01Z

Hi @parkertimmins, I've created a changelog YAML for you.

lkts

Can we document this? Otherwise LGTM

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java

lkts · 2025-06-18T20:51:37Z

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java

+                // In the open object, there is a leaf with a scalar value, which is also the first
+                // part of the current path. Instead of traversing down into the path and building objects,
+                // combine the path into a single leaf and add it as a field.
+                if (curr.pathEquals(next) == false) {


Can you explain what this if does to me?

This if is where we add fields to the output. Specifically, fields that need to be specified with the concatenated path due to a scalar/object mismatch. The similar if in the else block below serves the same purpose. The if is necessary to handle multiple values for the same field. For example if we have the following key/value pairs (where I'm using | instead of \0 to separate key and value):

foo|5 foo.bar|10 foo.bar|20 baz|99

Assume we've already written out foo|5. When we go through the loop for foo.bar|10, we'll add 10 to the values array (which will be empty) at the beginning of the loop. But when we get to this if statement, we'll see that the next pair (foo.bar|20) has the same path foo.bar. Because of this we won't enter the if and won't write out the field yet. In the next loop iteration, curr is now foo.bar|20. We'll add 20 to values, making it [10, 20]. When we get to this if statement, we see that next path is baz. Since this not equal to foo.bar we know that we have seen all values for foo.bar. So we enter the if statement and write out all values for the field.

This is a good example, consider adding it to the comment above, or splitting some of that before the first branch.

Thank you @parkertimmins, i think it's worth a comment too. Alternatively it sounds like we could restructure this to first read all the values for a field (move the if into a separate loop above this one) and then write them but this is not critical of course.

github-actions · 2025-06-19T03:31:53Z

🔍 Preview links for changed docs:

docs/reference/elasticsearch/mapping-reference/flattened.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

kkrik-es · 2025-06-19T08:19:32Z

docs/reference/elasticsearch/mapping-reference/flattened.md

+`"bar"` has both a scalar value `"10"`, and an object value of `{ "baz": "20" }`.
+
+With synthetic source, objects with such duplicate fields will appear
+differently in `_source`. For example, if the above field has synthetic


More accurately: if the field is defined in an index configured with synthetic source.

kkrik-es · 2025-06-19T08:22:04Z

docs/reference/elasticsearch/mapping-reference/flattened.md

+This is because, when constructing the source, the key `"foo.bar"` is eagerly expanding into the key `"foo"` with an object value.
+Then `"bar"` is added to the object with the scalar value of `"10"`. Then, because another
+`"bar"` cannot be added with an object value, `"bar"` and `"baz"` are collapsed
+in the flat key `"bar.baz"` with a scalar value of `"20"`


I'd probably skip this paragraph, and maybe mention above:

... to produce a valid JSON output.

+1, let's not go into the details

kkrik-es · 2025-06-19T08:38:32Z

server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java

+        var syntheticSource = syntheticSource(mapper, b -> {
+            b.startObject("field");
+            {
+                b.field("key1.key2", "foo");


Maybe add also: `b.field("key1", "baz");

kkrik-es · 2025-06-19T08:40:00Z

.../java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelperTests.java

+        final FlattenedFieldSyntheticWriterHelper writer = new FlattenedFieldSyntheticWriterHelper(new SortedSetSortedKeyedValues(dv));
+        final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+        final XContentBuilder builder = new XContentBuilder(XContentType.JSON.xContent(), baos);
+        final List<byte[]> bytes = List.of("a.b.c" + '\0' + "10", "a.b.c.d" + '\0' + "20")


Let's also add something for a and a.b.

kkrik-es · 2025-06-19T08:40:36Z

server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java

+        });
+        assertThat(syntheticSource, equalTo("""
+            {"field":{"key1":{"key2":"foo","key2.key3":"bar"}}}"""));
+    }


Shall we also add an example with arrays?

kkrik-es

Nice, some minor comments to address but otherwise it looks good.

lkts · 2025-06-19T16:32:56Z

docs/reference/elasticsearch/mapping-reference/flattened.md

@@ -364,3 +364,42 @@ Will become (note the nested objects instead of the "flattened" array):
  }
 }
 ```
+
+Flattened fields allow for a duplicate key to contain both an object and a scalar value.


Suggested change

Flattened fields allow for a duplicate key to contain both an object and a scalar value.

Flattened fields allow for a key to contain both an object and a scalar value.

lkts · 2025-06-19T16:39:21Z

docs/reference/elasticsearch/mapping-reference/flattened.md

+This is because, when constructing the source, the key `"foo.bar"` is eagerly expanding into the key `"foo"` with an object value.
+Then `"bar"` is added to the object with the scalar value of `"10"`. Then, because another
+`"bar"` cannot be added with an object value, `"bar"` and `"baz"` are collapsed
+in the flat key `"bar.baz"` with a scalar value of `"20"`


+1, let's not go into the details

parkertimmins added 5 commits June 17, 2025 16:43

Initial working version

438dc85

some cleanup

e9cbdc0

move some stuff around

dd88ff7

revert some changes

8ef6d51

cleanup tests

529a510

elasticsearchmachine added the v9.1.0 label Jun 18, 2025

[CI] Auto commit changes from spotless

d0a3365

kkrik-es reviewed Jun 18, 2025

View reviewed changes

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java Outdated Show resolved Hide resolved

kkrik-es reviewed Jun 18, 2025

View reviewed changes

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java Outdated Show resolved Hide resolved

kkrik-es reviewed Jun 18, 2025

View reviewed changes

.../main/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldSyntheticWriterHelper.java Outdated Show resolved Hide resolved

updates from feedback

bd571fd

parkertimmins commented Jun 18, 2025

View reviewed changes

parkertimmins added 2 commits June 18, 2025 11:00

Merge branch 'main' into parker/flattened-scalar-object-mismatch

e893e73

add a few mapper tests

55480d4

parkertimmins marked this pull request as ready for review June 18, 2025 19:32

parkertimmins requested review from kkrik-es and lkts June 18, 2025 19:33

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jun 18, 2025

parkertimmins added auto-backport Automatically create backport pull requests when merged :StorageEngine/Mapping The storage related side of mappings labels Jun 18, 2025

elasticsearchmachine added the Team:StorageEngine label Jun 18, 2025

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Jun 18, 2025

parkertimmins added needs:triage Requires assignment of a team area label v8.19.0 v9.0.4 v8.18.4 and removed Team:StorageEngine needs:triage Requires assignment of a team area label labels Jun 18, 2025

elasticsearchmachine added the Team:StorageEngine label Jun 18, 2025

parkertimmins added >bug and removed Team:StorageEngine labels Jun 18, 2025

elasticsearchmachine added the Team:StorageEngine label Jun 18, 2025

Update docs/changelog/129600.yaml

2ac1c55

github-actions bot deployed to docs-preview June 18, 2025 19:59 View deployment

lkts approved these changes Jun 18, 2025

View reviewed changes

updates from review

c699bfd

github-actions bot deployed to docs-preview June 18, 2025 22:24 View deployment

update docs

408393c

github-actions bot deployed to docs-preview June 19, 2025 03:32 View deployment

kkrik-es reviewed Jun 19, 2025

View reviewed changes

kkrik-es approved these changes Jun 19, 2025

View reviewed changes

lkts approved these changes Jun 19, 2025

View reviewed changes

	Flattened fields allow for a duplicate key to contain both an object and a scalar value.
	Flattened fields allow for a key to contain both an object and a scalar value.

Make flattened synthetic source concatenate object keys on scalar/object mismatch #129600

Are you sure you want to change the base?

Make flattened synthetic source concatenate object keys on scalar/object mismatch #129600

Conversation

parkertimmins commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

lkts left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lkts Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lkts Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkrik-es left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

parkertimmins commented Jun 18, 2025 •

edited

Loading

lkts Jun 18, 2025 •

edited

Loading

lkts Jun 19, 2025 •

edited

Loading