Support new @elastic/elasticsearch TypeScript type definitions

[fuglu]

I'm submitting a...

[ ] Regression 
[ ] Bug report
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

Currently @nestjs/elasticsearch only supports the legacy typings of @elastic/elasticsearch.

Expected behavior

It would be great if we could support the new type definitions without waiting for the next major version of @nestjs/elasticsearch and @elastic/elasticsearch.

The different type definitions are documented over here: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/typescript.html#typescript

Currently the client exposes two type definitions, the legacy one, which is the default and the new one, which will be the default in the next major

Minimal reproduction of the problem with instructions

The current ElasticsearchService class extends the Client class of @elastic/elasticsearch which exports the legacy type definitions. To use the new type definitions we have to import the new typings like this:

import { Client as NewClient } from '@elastic/elasticsearch/api/new'

// <snip>

export class ElasticsearchService extends NewClient {

// <snip>

This is documented over here: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/typescript.html#_how_to_migrate_to_the_new_type_definitions

But this would be a breaking change and is not an option. However, it would be great if we could provide another method to support the new type definitions in a minor release.

Of course you could always do something like this but native support would be sweet.

const result = await (this.elasticsearchService as unknown as NewClient).get({
  id: "my-id",
  index: "my-index"
});

What is the motivation / use case for changing the behavior?

The new type definition is more advanced compared to the legacy one. In the legacy type definitions you were expected to configure via generics both request and response bodies. The new type definitions comes with a complete type definition for every Elasticsearch endpoint.

It would be great if we could support the new types. I could create a pull request, but at this time I am not sure what the best solution would be. Maybe generics could be used, maybe a second ElasticsearchService class is the solution. I am open to suggestions here and would need some help or guidance from experienced Nest developers.

[kamilmysliwiec]

Since we're about to publish a new major release, I'm totally fine with breaking changes that are required by the underlying tools. In this case, we wouldn't have to support the legacy typings. Would you like to create a PR to address this issue?

[fuglu]

I'm stuck but maybe someone has an idea.

The problem

To implement the new type definitions the // @ts-expect-error @elastic/elasticsearch annotation which is documented here has to be in place.

@Injectable()
// @ts-expect-error @elastic/elasticsearch
export class ElasticsearchService extends Client implements NewTypes {
// <snip>
}

However, the annotation is deleted by the TypeScript compiler when building the library. This is explained in microsoft/TypeScript#38628.

Some things I tried

I tried to get around this by using this trick but unfortunately JSDoc seems to be invalid for class declarations.
The comment will be included in the *.d.ts file when /** @ts-expect-error @elastic/elasticsearch */ is added to a class property but not on the class declaration. So no luck with this hack.

Next I tried to omit missing properties on the Client class by using:

@Injectable()
export class ElasticsearchService extends Client implements Omit<NewTypes, 'dataFrameTransformDeprecated'> {
// <snip>
}

This just yields more type incompatibilities and I gave up on this approach.
I even tried to add some of the missing properties (dataFrameTransformDeprecated) to the ElasticsearchClient subclass but this just provoked the same errors.

Then I tried to just force the type conversion by doing this:

@Injectable()
class ElasticsearchService_Old extends Client {
  constructor(
    @Optional() @Inject(ELASTICSEARCH_MODULE_OPTIONS) options: ClientOptions
  ) {
    super(options);
  }
}

export const ElasticsearchService = ElasticsearchService_Old as unknown as NewTypes;

But this results in Type 'Client' is missing the following properties from type 'ExistingProvider<any>': provide, useExisting ts(2322) in the ElasticsearchModule.

Other (hacky) ideas

Does anyone know if you can force TypeScript classes into a different type when extending them? Something like this would be great:

@Injectable()
export class ElasticsearchService extends (Client as NewTypes) {
// <snip>
}

Or maybe we could use some kind of intermediate interface or class and force the type conversion there.

Maybe version 8.0.0 of @elastic/elasticsearch ships in the near future. Using the canary version is probably a bad idea.

Test setup

I have a simple Nest application in place to test the library once the errors have been resolved. You can use it like this:

# clone the Nest elasticsearch client library
git clone https://github.com/einsenundnullen/elasticsearch.git
cd elasticsearch

#  checkout the upgrade-elasticsearch-type-definitions feature branch
git checkout upgrade-elasticsearch-type-definitions

# build the library
npm run build

# create a link to the library
npm link

# now checkout the Nest example application
git clone https://github.com/fuglu/nestjs-elasticsearch-playground.git
cd nestjs-elasticsearch-playground

# link the elasticsearch client
npm link @nestjs/elasticsearch

# start an elasticsearch server using docker-compose
docker-compose up

# start the Nest application
npm run debug

# trigger some elasticsearch operations and test the library
curl localhost:3000

[kamilmysliwiec]

Thanks @fuglu! Based on your research, it seems that the elasticsearch type definitions are simply not "production ready" at this point. These are issues that should be solved at their package level first.

[fuglu]

Seems fair. Putting more effort into the issue doesn't seem to make sense to me now and I would rather wait for the next major release. I was thinking if it would make sense to create an issue at @elastic/elasticsearch to explain our use case there and let them know. Maybe I will do that later.

I'll close this issue here for now.

[tomups]

Is the prospect for this better now that the official @elastic/elasticsearch v8.0.0 is out? :)

[kamilmysliwiec]

Would you like to create a PR to address this issue?

[tomups]

Fortunately the PR is already there by the automation:

#748

You just need to require the higher version of Elasticsearch as dependency. Only the new types are available now in that version by default, so nothing else to do!

But this is a breaking change, so would need a new major version of this package...

Not sure what would be the best way to name it, because as #580 (comment) says, having the current version be 8.0.0 while listing @elastic/[email protected] as a dependency is very confusing.

Now to make @elastic/[email protected] a dependency, this package would have to be versioned as 9.0.0 to show it's a breaking change, so the confusion would keep going on and on with every major Elastic release...

[kamilmysliwiec]

You just need to require the higher version of Elasticsearch as dependency. Only the new types are available now in that version by default, so nothing else to do!

@thomrick Does it mean there are no code changes needed to make this package compatible with the latest version? If so, I'd say setting the peer dependency constraint to "^7.4.0 || ^8.0.0" should be sufficient and in this case, we'd not have to trigger a new major release.

[tomups]

Yes, I tested force installing @elastic/[email protected] and could use the new types with no code changes to this package.

So changing the peer dependency to accept both v7 and v8 should indeed be enough. Then the @elastic/elasticsearch version would be defined by what the user installs, and this package can be compatible with v7 and v8 as-is, without breaking changes 👍🏼