[ML] Inference request count telemetry per node

[jonathan-buttner]

WIP

This PR wires up the telemetry code to record inference request counts by model id (if it is defined).

The inference section of the telemetry looks like this now:

"inference": {
        "available": true,
        "enabled": true,
        "models": [
            {
                "service": "cohere",
                "task_type": "RERANK",
                "count": 1
            },
            {
                "service": "cohere",
                "task_type": "TEXT_EMBEDDING",
                "count": 1
            },
            {
                "service": "openai",
                "task_type": "TEXT_EMBEDDING",
                "count": 1
            }
        ],
        "requests": [
            {
                "service": "cohere",
                "task_type": "rerank",
                "count": 1,
                "model_id": "rerank-english-v3.0"
            },
            {
                "service": "cohere",
                "task_type": "text_embedding",
                "count": 1,
                "model_id": "embed-english-v3.0"
            },
            {
                "service": "openai",
                "task_type": "text_embedding",
                "count": 1,
                "model_id": "text-embedding-3-small"
            }
        ]
    },

TODOs

• Spinning up a cluster and testing that the apm metering works as expected
• Making the changes in the telemetry repo to index the inference usage in a new index

[jonathan-buttner]

@elasticmachine merge upstream

[prwhelan]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[prwhelan]

@elasticmachine update branch

[prwhelan]

Tested with APM:

{
  "_index": ".ds-metrics-apm.app.elasticsearch-default-2024.07.22-000001",
    ...
    "data_stream": {
      "dataset": "apm.app.elasticsearch",
      "namespace": "default",
      "type": "metrics"
    },
    "es": {
      "inference": {
        "requests": {
          "count": {
            "total": 8
          }
        }
      }
    },
    ...
    "labels": {
      "model_id": ".elser_model_2_linux-x86_64",
      "otel_instrumentation_scope_name": "elasticsearch",
      "service": "elser",
      "task_type": "sparse_embedding"
    },
    ...
  }
}