Performance Monitoring too many events in queue breaks feature entirely

[jakehockey10]

Operating System

Pop OS

Environment (if applicable)

Angular, Chrome

Firebase SDK Version

9

Firebase SDK Product(s)

Performance

Project Tooling

I'm using Angular 19 with @angular/fire

Detailed Problem Description

I was attempting to use the Performance Monitoring feature, and from the start, it wouldn't send any of the events that are automatically tracked. It also wouldn't send any of my custom traces (I had one for testing when I realized the built in monitoring wasn't working).

Eventually, after a couple of days on StackOverflow and GitHub, researching the issue, I found the source code for this in my Chrome DevTools, found the spot where the request via navigator.sendBeacon and, if necessary, fetch, will post the events to the logging endpoint. Unfortunately, if an error is encountered, the catch block doesn't capture the error in a variable, so I wasn't sure what the error was.

Then I ran across another issue in another repository where the navigator.sendBeacon call was returning false, and one suggested that it might be because the request body is too large. Sure enough, I set a breakpoint on the call to the endpoint, and there was hundreds of events in the request body. So I removed all but one of them in the debugger, and let it roll. Events started streaming in finally.

I'm not sure how the events accumulated this much without any of them being sent to the logging endpoint, as this was my first few days trying to use the feature. But now events are streaming in. I'm not sure if this will happen again or what caused it, but I wanted to report this issue in hopes that handling the request body being too large might be something worth handling in the firebase sdk itself as I don't seem to see a way to get a hold of these events in my code.

Steps and code to reproduce issue

• Use Angular 19
• Use the app without Performance
• register Performance and make sure it is initialized
• somehow get hundreds of events to accumulate without them being sent, successfully, to the endpoint, beyond the point where the request body is too large for navigator.sendBeacon and fetch

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[jakehockey10]

After seemingly fixing the issue, I reloaded the page several more times, and started seeing the issue again:

Performance: Tries left: 2.
Performance: Tries left: 1.
Performance: Tries left: 0.

Sure enough, the number of events in the request body is back up to 195 log events. They all seem to have the same timestamp on them as well (right now):

This is unfortunately happening when refreshing the page and just sitting on the Profile Page of my appilcation:

It looks like there is a mechanism to grab only some events from the queue, and using a constant called MAX_EVENT_COUNT_PER_REQUEST set to 1000 with this concern in mind. However, my request body only consists of 195 events, and this is still preventing the events from being sent out. I would imagine that this many events is not supposed to happen, and I'm not sure what is causing so many events to be sent for seemingly the same thing.

[hsubox76]

I think you are right about this being a request size issue. I was able to reproduce the issue with this snippet (plain JS, no angular needed):

const app = initializeApp(firebaseConfig);
const perf = getPerformance(app);
for (let i = 0; i < 100; i++) {
  const t = trace(perf, "CUSTOM_TRACE_" + i);
  t.start();
  t.stop();
}

I decreased MAX_EVENT_COUNT_PER_REQUEST to 20, and it was fine. I walked through the code and it seems like both navigator.sendBeacon and fetch find the payload size too large (I was sending 126 events). I'm not sure what the exact number should be but I guess it's usually 64KB in most browsers. It looks like my 126 events come up to about 66094 bytes, so 1000 seems way too high for a limit. Maybe something changed in Chrome at some point about the format of an event object?

I hand-edited MAX_EVENT_COUNT_PER_REQUEST to 100 and it correctly sent everything in 2 chunks, one with 100 events and one with 26.

I actually think it's not that safe for the splitting logic to depend on event count when it should probably depend on byte size - either that or we could just try to keep the event count well below the max. I'll see if I can talk to someone on the Performance Monitoring team and see how they want to approach it.

Edit: This is the logic that would have to be changed:
https://github.com/firebase/firebase-js-sdk/blob/main/packages/performance/src/services/transport_service.ts#L91

[jakehockey10]

Just out of curiosity, I'd love to learn more about what hundreds of individual event objects represent in a seemingly short period of time or if some of these events are redundant. I'm unfamiliar with the details about what goes into an individual event. But maybe there is some documentation I haven't discovered yet

[raymondlam]

Just out of curiosity, I'd love to learn more about what hundreds of individual event objects represent in a seemingly short period of time or if some of these events are redundant. I'm unfamiliar with the details about what goes into an individual event. But maybe there is some documentation I haven't discovered yet

Firebase Performance will automatically capture each network request and PerformanceMeasure as a separate events. Some frameworks like React do automatically create a number of PerformanceMeasures which we would capture.

[jakehockey10]

@raymondlam that makes sense but I still would imagine that the number of events would be less than hundreds in a short period of time while minimizing user interaction and page navigation. Without navigating away from my profile page, it seemed to have many events pertaining to visiting the profile page. What fields on these events tell me what the even represents so I can further assess whether these events are redundant or not?

[raymondlam]

@jakehockey10 You can take a look at the trace_metric.name or network_request_metric.url (depending on if it's a trace or a network request).

If you breakpoint the same line of code as above, you can run the following to see the traces and network request information we collect.

data.log_event.map(e => JSON.parse(e.source_extension_json_proto3).trace_metric || JSON.parse(e.source_extension_json_proto3).network_request_metric)

The application_info.web_app_info.page_url represents the url of the page and is used on our Firebase dashboard so you can filter by the page on which the network request/trace event occurred. If you are looking only at the profile page, it will likely be the same across all events.

If you could provide a snippet of duplicate trace_metric or network_request_metric, it would be really helpful to debug.

[jakehockey10]

Thanks for the suggestion and the snippet! I took what you gave me here and investigated all the URLs. Looks like ~60% of the events that have a url property are to do with pulling Vite cached chunk files:

Maybe this is by design, but I noticed that a lot of them are for sections of the app that I haven't visited yet.

The rest of the urls were seemingly legitimate events representing the various requests being made by the Angular framework for things like chunks that are needed, styling, Firestore, etc.

I realize that this is a tool meant to be used in production so this might not be as much of a problem there if this many events are due to running a locally running ng serve. But I'm wondering if there is something I can do to filter out events I know are only from a local environment?

It also does seem like the limits have request size have changed recently. It seems that this limit might be currently 64 KiB which is pretty small. Today, the number of events amount to ~120KiB:

[jakehockey10]

I think you are right about this being a request size issue. I was able to reproduce the issue with this snippet (plain JS, no angular needed):

const app = initializeApp(firebaseConfig);
const perf = getPerformance(app);
for (let i = 0; i < 100; i++) {
  const t = trace(perf, "CUSTOM_TRACE_" + i);
  t.start();
  t.stop();
}

I decreased MAX_EVENT_COUNT_PER_REQUEST to 20, and it was fine. I walked through the code and it seems like both navigator.sendBeacon and fetch find the payload size too large (I was sending 126 events). I'm not sure what the exact number should be but I guess it's usually 64KB in most browsers. It looks like my 126 events come up to about 66094 bytes, so 1000 seems way too high for a limit. Maybe something changed in Chrome at some point about the format of an event object?

I hand-edited MAX_EVENT_COUNT_PER_REQUEST to 100 and it correctly sent everything in 2 chunks, one with 100 events and one with 26.

I actually think it's not that safe for the splitting logic to depend on event count when it should probably depend on byte size - either that or we could just try to keep the event count well below the max. I'll see if I can talk to someone on the Performance Monitoring team and see how they want to approach it.

Edit: This is the logic that would have to be changed: https://github.com/firebase/firebase-js-sdk/blob/main/packages/performance/src/services/transport_service.ts#L91

Looks like 100 is right around the threshold for me. This is my quick test: