Cannot use custom token filters with Elastic.Clients.Elasticsearch v9

[J-ohn]

Client version: Elastic.Clients.Elasticsearch v9
Elasticsearch version: 9
.NET runtime: .NET 9
Operating system: Windows

Problem Description

The v9 Elasticsearch .NET client does not allow defining custom or plugin-provided token filters (for example, from an installed Elasticsearch plugin).

When attempting to create an index with a custom token filter such as skroutz_stem_greek, the following exception is thrown:

System.Text.Json.JsonException: Variant 'skroutz_stem_greek' is not supported for type 'ITokenFilter'.
at Elastic.Clients.Elasticsearch.Analysis.ITokenFilterConverter.Write(Utf8JsonWriter writer, ITokenFilter value, JsonSerializerOptions options)

This occurs because the client uses a hardcoded switch statement inside ITokenFilterConverter that supports only a predefined set of token filter types. There is no support for custom or unknown types, despite Elasticsearch allowing them via plugins.

Steps to Reproduce

Install a plugin in Elasticsearch that registers a custom token filter named skroutz_stem_greek.

In .NET, define a class:

  public class SkroutzStemmerTokenFilter : ITokenFilter
   {
    public string Type => "skroutz_stem_greek";
   }

Attempt to use it in a CreateIndexRequest:

    request.Settings.Analysis.TokenFilters["greek_stemmer"] = new SkroutzStemmerTokenFilter();

Execute the request and observe the exception.

Expected Behavior

The client should allow the use of custom ITokenFilter implementations without throwing exceptions. Ideally, if a type is not recognized in the switch, it should be serialized based on its Type property and any other supplied fields.

Current Workaround

The only current workaround is to bypass the high-level API and use the low-level transport client with manually constructed JSON.

Request

Please provide a mechanism to support custom or plugin-defined token filters, such as:

A fallback branch in ITokenFilterConverter

A way to register additional token filter types

Treating unknown filters as passthrough JSON objects based on the Type property

[flobernd]

Hi @J-ohn

Thanks for reporting!

This mainly looks like a duplicate of #8500 - just for serialization instead of deserialization.

The workaround mentioned in that linked issue should work for your case as well (if you adapt for the "write" case).

As mentioned in the other issue, this part of the client will be improved soon.