fix Invalid JSON desearilaztion beyond buffer end

[ayavilevich]

fix for #183

[Copilot]

Pull Request Overview

This PR addresses issue #183 by correcting JSON deserialization beyond the buffer end.

• Updated deserializeJson call for ArduinoJson v6 and non-v6 implementations, passing request->contentLength() as an additional boundary argument.

need to think about it again...

[mathieucarbou]

The fix is not as simple as relying on request->contentLength());.

What is happening if you run:

curl -v -X POST -H 'Content-Type: application/json' -d '5' -H 'Content-Length: 3' http://127.0.0.1:8080/json2

[ayavilevich]

The fix is not as simple as relying on request->contentLength());.

What is happening if you run:

curl -v -X POST -H 'Content-Type: application/json' -d '5' -H 'Content-Length: 3' http://127.0.0.1:8080/json2

If you send 1 byte but specify in headers that you are sending 3 then the server will keep on waiting for 2 more bytes. Eventually it will timeout and handleBody won't be called.

This is what curl will say:

curl -v -X POST -H "Content-Type: application/json" -d "5" -H "Content-Length: 3" http://192.168.4.1/json2
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying xxxxx
* Connected to xxxx port 80
> POST /json2 HTTP/1.1
> Host: xxxx
> User-Agent: curl/8.9.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 3
>
* upload completely sent off: 1 bytes
* Empty reply from server
* shutting down connection #0
curl: (52) Empty reply from server

An edge case in the other direction where you send 5 bytes but claim 3 in the headers will result in 2 bytes being ignored.

curl:

C:\Users\arik>curl -v -X POST -H "Content-Type: application/json" -d "55555" -H "Content-Length: 3" http://192.168.4.1/json2
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying xxxx
* Connected to xxxx port 80
> POST /json2 HTTP/1.1
> Host: xxxx
> User-Agent: curl/8.9.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 3
>
* upload completely sent off: 5 bytes
< HTTP/1.1 200 OK
< connection: close
< accept-ranges: none
< content-length: 3
< content-type: text/plain
<
int* shutting down connection #0

where ESP will say

12:00:45.874 > Json:
12:00:45.875 > 555
12:00:45.876 > Got an int

So 555 and not 55555. Expected.

So it seems that it works however you have a point. There are two different _contentLength here (which seem to be in sync as far as I can check) one in AsyncCallbackJsonWebHandler and one in AsyncWebServerRequest.
The one used to allocate the buffer is the local one to AsyncCallbackJsonWebHandler and not the one used in the proposed fix. It might be safer/better to use the local _contentLength directly. Do you agree? LMK and I will update.

[ayavilevich]

The one used to allocate the buffer is the local one to AsyncCallbackJsonWebHandler and not the one used in the proposed fix. It might be safer/better to use the local _contentLength directly. Do you agree? LMK and I will update.

Actually, after further inspection, I take it back.
We have to remove the local _contentLength as it doesn't make sense. A single handler can handle many requests, even concurrently. So having a global length for all requests is a mistake.
So the original proposal is fine. Maybe we can add some additional checks. More work is needed to determine if there are cases whee it might be needed.

[mathieucarbou]

The one used to allocate the buffer is the local one to AsyncCallbackJsonWebHandler and not the one used in the proposed fix. It might be safer/better to use the local _contentLength directly. Do you agree? LMK and I will update.

Actually, after further inspection, I take it back. We have to remove the local _contentLength as it doesn't make sense. A single handler can handle many requests, even concurrently. So having a global length for all requests is a mistake. So the original proposal is fine. Maybe we can add some additional checks. More work is needed to determine if there are cases whee it might be needed.

I agree!

[mathieucarbou]

Do you want to see how this can be improved ?

[ayavilevich]

Do you want to see how this can be improved ?

yes, I will update the PR.

[mathieucarbou]

PR looks better! Just a few things to fix.
If you want to go further in code cleanup, like using the index instead of the tmpObject to make it a little more readable, go ahead!

[Copilot]

Pull Request Overview

This PR addresses an issue with invalid JSON deserialization beyond the buffer end by improving buffer allocation and JSON parsing logic. Key changes include:

• Adding a new _tempSize member to track allocated buffer size.
• Removing the unused _contentLength variable in the JSON handler.
• Adjusting JSON parsing logic to ensure proper null termination and buffer bounds checking.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/WebRequest.cpp	Added _tempSize to the constructor initializer list.
src/ESPAsyncWebServer.h	Declared _tempSize member to support buffer size tracking.
src/AsyncJson.h	Removed the unused _contentLength member.
src/AsyncJson.cpp	Modified JSON parsing and buffer handling logic for safety.

Comments suppressed due to low confidence (1)

src/AsyncJson.cpp:174

• Since the buffer is allocated with an extra byte (i.e., malloc(total + 1)), consider renaming or updating _tempSize to reflect the actual allocated size to avoid potential off-by-one misunderstandings in future code changes.

request->_tempSize = total;  // store the size of allocation we made into _tempSize

[ayavilevich]

If you want to go further in code cleanup, like using the index instead of the tmpObject to make it a little more readable, go ahead!

What is "the index"? How will it replace tmpObject?

[ayavilevich]

@mathieucarbou I have responded to your questions. I need some further guidance.

[mathieucarbou]

If you want to go further in code cleanup, like using the index instead of the tmpObject to make it a little more readable, go ahead!

What is "the index"? How will it replace tmpObject?

You can look at the examples in the project. In the end, the code will work both ways, but usually when determining if this is the first time the callback is called, it is done with if(!index) {}. Index tells the position (or count) of the data already received which can be used as the position where to do a buffer copy.

[ayavilevich]

You can look at the examples in the project. In the end, the code will work both ways, but usually when determining if this is the first time the callback is called, it is done with if(!index) {}. Index tells the position (or count) of the data already received which can be used as the position where to do a buffer copy.

I would rather check the pointer itself as an indication to whether it was allocated rather than use "index" for that.

@mathieucarbou would you accept a solution that is based on a struct with the following members

size_t length
uint8_t buffer[1]

and a size of "sizeof(size_t) + total + 1" ?

I think I could do it robustly this way and without adding a new field ("_tempSize") to the request object.

[mathieucarbou]

You can look at the examples in the project. In the end, the code will work both ways, but usually when determining if this is the first time the callback is called, it is done with if(!index) {}. Index tells the position (or count) of the data already received which can be used as the position where to do a buffer copy.

I would rather check the pointer itself as an indication to whether it was allocated rather than use "index" for that.

@mathieucarbou would you accept a solution that is based on a struct with the following members

size_t length
uint8_t buffer[1]

and a size of "sizeof(size_t) + total + 1" ?

I think I could do it robustly this way and without adding a new field ("_tempSize") to the request object.

yes! you can also just have only one uint8_t pointer that you allocate like you said with sizeof(size_t) + total + 1 instead of a structure and you extract the size_t from the first bytes. No structure needed.

[mathieucarbou]

@ayavilevich : I added more comments to this PR and decided ti push a cleanup commit to complete it.
Could you please have a look and test on your side ?
I tested all use cases added to the Json.ino examples.
I am waiting for your feedback, then will merge.
Thank you 👍

[Copilot]

Pull Request Overview

This PR fixes an issue where JSON deserialization could access memory beyond its buffer end, by removing the redundant _contentLength member and updating type casts and buffer handling in deserialization.

• Removed the _contentLength member from AsyncJson.h
• Updated deserialization casts from (uint8_t *) to (const char *) in AsyncJson.cpp
• Adjusted handleBody logic and enhanced example documentation to illustrate edge cases

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/AsyncJson.h	Removed the redundant _contentLength field to streamline content length validation.
src/AsyncJson.cpp	Updated deserialization casts and buffer handling to prevent memory overruns.
examples/Json/Json.ino	Added additional Serial output and documented edge cases for JSON payload handling.

[ayavilevich]

@ayavilevich : I added more comments to this PR and decided ti push a cleanup commit to complete it. Could you please have a look and test on your side ? I tested all use cases added to the Json.ino examples. I am waiting for your feedback, then will merge. Thank you 👍

@mathieucarbou Hey, looks fine. I tested it on my end with some test cases and wasn't able to find any issues. Can you merge this?

[mathieucarbou]

@ayavilevich : I added more comments to this PR and decided ti push a cleanup commit to complete it. Could you please have a look and test on your side ? I tested all use cases added to the Json.ino examples. I am waiting for your feedback, then will merge. Thank you 👍

@mathieucarbou Hey, looks fine. I tested it on my end with some test cases and wasn't able to find any issues. Can you merge this?

merged!

I am waiting for the LIbretiny PR to be updated and merged to issue a new release