EQL: case sensitivity in ES EQL string functions

[astefan]

At the moment, the documented string functions in eql (https://eql.readthedocs.io/en/latest/query-guide/functions.html) have an inconsistent behavior when it comes to case sensitivity. All of them, with the exception of between function deal with strings in a case insensitive way.

Case insensitive functions

• arrayContains
• endsWith
• indexOf
• match
• startsWith
• stringContains
• wildcard

Case sensitive functions

• between which has two additional parameters, one of which configures it to be case sensitive or not (default is false)

This issue is for deciding on a path forward regarding the case sensitivity aspect in ES EQL string functions.

Proposals:

1. ES EQL should mirror exactly the functionality in EQL

• con: inconsistent implicit functions behavior
• pro: the two implementations are fully compatible

2. ES EQL should handle both cases of case sensitive and insensitive functionality by giving the users the power of choice through an additional functionality: LOWER and UPPER functions that should perform lowercasing/uppercasing. These two functions, combined with existent functions will help with case insensitive behavior. For example: file where startsWith(lower(file_name), lower('expLORER.exe'))

• cons:
  • a less user-friendly behavior, because users need to use an additional function for the same behavior they get in EQL
  • ES EQL and EQL will diverge in functionality
• pro: a more powerful functionality for users, since they now the possibility of matching exactly strings in their functions

3. This is a variant of first option where, by default, the behavior of ES EQL will be the same as EQL, but offering users the possibility of changing this behavior through a session-level parameter (either fully case sensitive or fully case insensitive):

• con: cannot use case insensitive and case sensitive functionality in the same query
• pro: the two implementations are fully compatible

These are just some ideas and we don't have to choose one or another. The list is open for discussion, suggestions are welcomed.

[elasticmachine]

Pinging @elastic/es-ql (:Query Languages/EQL)

[astefan]

Following our discussion about this:

• we decided to add a request parameter that should enable/disable case sensitivity in a query. The default (if the parameter is not specified) should be case insensitive in line with the current behavior in the EQL endpoint.
• the between function, if its use is minimal, should have the optional parameter (case sensitive) removed altogether and behave like the rest of the string functions.

Follow up items:

• @rw-access check if it's feasible to remove the optional parameter from between function, considering its current usage in the EQL endpoint.
• @tsg would you mind weighing in on the requirements for case sensitive/insensitive functionality in SIEM and if you have any pro/con comments regarding the proposal above?
  • note that at the moment eql endpoint has all string functions case insensitive except between function that has an optional parameter for this, the default being to be case insensitive.

[tsg]

Thanks for the ping. I like the behaviour of being case insensitive by default, in most security use cases that's usually what people need. If adding case-sensitivity via the request parameter is not a lot of effort, that seems like a good idea, it will allow for other use-cases for EQL.

[rw-access]

Yes, we can remove the case sensitive parameter from this function.
The function uses case sensitivity when finding the matching substrings left and right within source. But it always returns the the substring between those in its original case. If we remove that parameter, we'll have to figure out how to perform case-(in)sensitive matching while returning the original string. It's not as simple as between(lower(source), lower(left), lower(right)).

When greedy=false, that makes the function roughly equivalent to

substring(source, // notice the original casing
          indexOf(lower(source), lower(left)),
          indexOf(lower(source), lower(right), indexOf(lower(source), lower(right), lower(left)))
          )