How is the Resource Unit normalization for CSOM requests calculated to determine when app should be throttled?

[digiofficerobin]

What type of issue is this?

Question

What SharePoint development model, framework, SDK or API is this about?

SharePoint CSOM

Target SharePoint environment

SharePoint Online

What browser(s) / client(s) have you tested

• 💥 Internet Explorer
• 💥 Microsoft Edge
• 💥 Google Chrome
• 💥 FireFox
• 💥 Safari
• mobile (iOS/iPadOS)
• mobile (Android)
• not applicable
• other (enter in the "Additional environment details" area below)

Additional environment details

SharePoint online CSOM / REST API

Issue description

In chapter Application Throttling there is the following text:

SharePoint provides various APIs. Different APIs have different costs depending on the complexity of the API. The cost of APIs is normalized by SharePoint and expressed by resource units. Application’s limits are also defined using resource units.

We want to rate limit on the client side to avoid throttling by the server. Therefore we need to know how much each request will cost or has cost.

In particular we want to know Resource Unit of the following CSOM queries/actions:

1. Context.Load(siteUsers, us => us.Include(u => u.Title, u => u.Id, u => u.LoginName, u => u.PrincipalType, u => u.Email, u => u.UserPrincipalName, u => u.AadObjectId));
2. var refUserAsPrincipal = rootWeb.EnsureUser(logOnName);
3. SecurableObjectSharePoint.Context.Load(refUserAsPrincipal, _user => _user.Email, _user => _user.LoginName);
4. SecurableObjectSharePoint.ResetRoleInheritance();
5. SecurableObjectSharePoint.BreakRoleInheritance(true, true);
6. SecurableObjectSharePoint.Context.Load(SecurableObjectSharePoint, s => s.HasUniqueRoleAssignments);
7. _spfile = ClientContext.Web.GetFileByUrl(AbsoluteUrl);
8. ClientContext.Load(SPFile, f => f.TimeLastModified, f => f.Exists);
9. ClientContext.Load(SPFile, f => f.Exists);
10. ClientContext.Load(SPFile, f => f.Exists, f => f.TimeLastModified, f => f.MajorVersion, f => f.MinorVersion, f => f.Length, f => f.ListItemAllFields.Id, f => f.ListItemAllFields.ParentList.ForceCheckout, f => f.CheckOutType, f => f.CheckedOutByUser.UserPrincipalName, f => f.LockedByUser.UserPrincipalName, f => f.ListItemAllFields.HasUniqueRoleAssignments);
11. ClientContext.Load(SPFile, f => f.ListItemAllFields.RoleAssignments.Include(rs => rs.Member, rs => rs.RoleDefinitionBindings.Include(rdb => rdb.RoleTypeKind, rdb => rdb.Name)));
12. ClientContext.Load(member, u => u.AadObjectId);
13. ClientContext.Load(SPFile, f => f.Exists, f => f.ListItemAllFields.ParentList.Fields.Include(fld => fld.InternalName, fld => fld.FieldTypeKind), f => f.ListItemAllFields);
14. ClientContext.Load(web, s => s.ServerRelativeUrl);
15. ClientContext.Load(checkedoutedFile, f => f.Where(fi => fi.ServerRelativePath.DecodedUrl == fileRelativeUrl).Include(fi => fi.CheckedOutBy.Email));
16. SPFile.CheckIn(comment, (CheckinType)checkinType);
17. SPFile.CheckOut();
18. SPFile.UndoCheckOut();
19. SPFile.DeleteObject();
20. var mstream = SPFile.OpenBinaryStream();
21. ClientContext.Load(SPFile, r => r.ListItemAllFields.HasUniqueRoleAssignments);
22. ClientContext.Load(SPFile, r => r.ListItemAllFields.ParentList.ParentWeb.RoleDefinitions.Where(rd => !(rd.RoleTypeKind == RoleType.Guest || rd.RoleTypeKind == RoleType.RestrictedGuest)).Include(rd => rd.Name, rd => rd.BasePermissions, rd => rd.RoleTypeKind));
23. ClientContext.Load(SPFile, r => r.ListItemAllFields.RoleAssignments.Include(ra => ra.Member.LoginName, ra => ra.Member.PrincipalType, ra => ra.Member.Id, ra => ra.RoleDefinitionBindings.Include(rdb => rdb.Id, rdb => rdb.RoleTypeKind, rdb => rdb.Name)));
24. ClientContext.Load(SPFile, _file => _file.Exists, _file => _file.ListItemAllFields.ParentList.Fields.Include(_field => _field.InternalName, _field => _field.ReadOnlyField, _field => _field.FieldTypeKind), _file => _file.ListItemAllFields.ContentType.Name);
25. var uploadFile = folder.Files.Add(newfile); ClientContext.Load(uploadFile);
26. bytesUploaded = uploadFile.StartUpload(uploadId, startStream);
27. uploadFile = uploadFile.FinishUpload(uploadId, fileoffset, finishStream);
28. bytesUploaded = uploadFile.ContinueUpload(uploadId, fileoffset, continueStream);
29. uploadFile.CancelUpload(uploadId);
30. site.Context.Load(oWeb, w => w.Title, w => w.Language, w => w.Url, w => w.Navigation.UseShared, w => w.Description, w => w.AllProperties, w => w.HasUniqueRoleAssignments, w => w.ServerRelativeUrl, w => w.Id);
31. site.Context.Load(oWeb, w => w.RoleAssignments.Include(ra => ra.Member.LoginName, ra => ra.Member.Id, ra => ra.RoleDefinitionBindings.Include(rdb => rdb.Id, rdb => rdb.RoleTypeKind, rdb => rdb.Name)));
32. site.Context.Load(oWeb, r => r.RoleDefinitions.Where(rd => !(rd.RoleTypeKind == RoleType.Guest || rd.RoleTypeKind == RoleType.RestrictedGuest)).Include(rd => rd.Id, rd => rd.Name, rd => rd.BasePermissions, rd => rd.RoleTypeKind));

We want also know the points for more complex calls/actions, such as creating or updating lists, contenttypes, lists, site columns.

We now have made estimations what each call will probably costs based on the Graph API call costs table with values 1,2 or 5

If there is a way to return the actual resource unit costs of each action/query in the response headers, that would be awesome. We then can implement rate limiting on the client side with actual values.

The Metered API charges on price per 1000 requests (thus not on points). We therefore cannot use the metered (pay-per-use) API to determine the points per request.

Can you assist us to clarify the Resource Unit costs of the CSOM operations/queries?

[Ashlesha-MSFT]

Hello @digiofficerobin,
Thank you for bringing this issue to our attention. We will look into it and get back to you shortly.

[Tanddant]

@digiofficerobin - You might find some of your answers in this repo

https://github.com/OneDrive/samples/tree/master/scenarios/throttling-ratelimit-handling

While it doesn't log the "cost" of a request, you could look at your rate-limit remaining, and do the math yourself to some extend.

But it also shows you how to implement Rate-limit handling and have your code "back off" before it becomes a real problem

This is also built into the latest version of PnP Framework: pnp/pnpframework@eec5fb2

[digiofficerobin]

@Tanddant While it doesn't log the "cost" of a request, you could look at your rate-limit remaining, and do the math yourself to some extend.
In theory that is a good idea. Unfortunately this is not possible in practice, because rate limit headers are most of the time not returned. See issue #10069 . And video's https://www.youtube.com/playlist?list=PLnX8DsYQ38gOUhzNBG0UnPHL7NqKBobX1 .

And if they return, then I need to use multiple threads to create enough requests which makes it difficult to calculate the consumed quota. It would have been better if the consumed points are returned with every response.

By the way load testing is not allowed You're running a load-testing script on SharePoint Online and you get throttled. Load testing isn't allowed on SharePoint Online. according to this article: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#common-throttling-scenarios-in-sharepoint-online

But it also shows you how to implement Rate-limit handling and have your code "back off" before it becomes a real problem
We have implemented this, but it won't work if no Rate-Limit headers are returned. And even if we get the response headers and wait the RateLimit-Reset time in seconds, then the quota isn't reset. See issue #10070

[Tanddant]

@digiofficerobin
I agree, the whole system feels "half baked" - but I'll say with some minor exception handling, and the latest version of PnP Framework, I managed to do 400k "heavy" CSOM requests (Versions and Permissions primarily) against a "pretty bad tenant", that's constantly being throttled due to another vendor not knowing what they're doing - without my code ever breaking.

I did see some of the other random errors that sometimes pops up, that isn't strictly speaking throttling, such as PnP Framework timeouts - but with the following setup I managed to get through

• SocketException
  • Where the the ErrorCode was
    • 503
    • 504
    • 429
  • Or the Message was
    • An existing connection was forcibly closed by the remote host.
• TaskCanceledException where
  • Message is A task was canceled. and the StackTrace contained PnP.Framework (when PnP Framework times out)
• ServerException
  • Where ServerErrorTypeName was
    • Microsoft.Office.Server.Search.SubstrateSearch.QueryTimeoutException
    • Microsoft.Office.Server.Search.Query.InternalQueryErrorException
    • Microsoft.SharePoint.SPQueryThrottledException
  • Or the Message was
    • Timeout
• WebException
  • Where the StatusCode was
    • 503
    • 504
    • 429

I simply wrote a generic wrapper for my CSOM request, and then catch those types of exceptions, and then retry - and while it's not perfect, it actually worked wonders, and my code just kept crunching for almost two days until it completed successfully

[digiofficerobin]

@Tanddant Thanks for your solution. We have similar exception handling and retry logic for these types of errors.

Our app includes both background services and end-user (GUI) services, which share the same app registration and service account (ROPC flow). When the background application is throttled for a few minutes, all end-users experience delays, timeouts, hangs, exceptions, etc. This is a significant issue.

To address this, we opted for a softer solution that allows us to smoothly delay requests from the client side during high demand periods, preventing us from exhausting our quota too quickly. We have implemented this solution, but we need to understand the costs of each request.

As a surrogate for real costs, we have estimated the costs as follows:

public class QuotaPointConstants
{
	/// <summary>
	/// Single item query, such as get item.
	/// Delta with a token
	/// </summary>
	public const double One = 1.0;
	/// <summary>
	/// Multi item query, such as list children, except delta with a token.
	/// Create, update, delete and upload
	/// </summary>
	public const double Two = 2.0;
	/// <summary>
	/// All permission resource operations, including $expand=permissions
	/// </summary>
	public const double Five = 5.0;
	public const double ResetRoleInheritance = Five;
	public const double BreakRoleInheritance = Five;
	public const double CompleteRoleAssignmentReplication = Five;
	public const double CompleteRoleDefinitionReplication = Five;
	public const double EnsureListItemProperties = Two;
	public const double LoadItemProperties = Two;
	public const double LoadSiteProperties = Two;
	public const double FindSubWeb = Two;
	public const double FindSubWebIncludeSecurity = Five;
	public const double FindListByUrl = Two;
	public const double FindListByUrlIncludeContentTypes = Five;
	public const double FindListByUrlIncludeSecurity = Five;
	public const double FindSPListTemplateByInternalName = Two;
	public const double EnsureContentTypeOnList = Five;
	public const double EnsureContentTypeOnListAddFieldToList = Two;
	public const double EnsureContentTypeOnSiteCollection = Five;
	public const double EnsureContentTypeOnSiteCollectionSaveChangesInContentType = Two;
	public const double DecodeClaim = One;
	public const double EnsureSiteCollectionUpdateWeb = Two;
	public const double EnsureSiteColumns = Five;
	public const double EnsureSiteAddSiteCollection = Five;
	public const double EnsureListAdd = Five;
	public const double EnsureListDeleteDefaultContenttype = Two;
	public const double EnsureListUpdateExistingList = Two;
	public const double EnsureFolderLoad = Two;
	public const double EnsureFolderLoadIncludeSecurity = Five;
	public const double EnsureFolderAdd = Two + Two;
	public const double EnsureFolderAddWithSecurity = Five;
	public const double LoadUsersFromSite = Two;
	public const double EnsureUserWithLoad = Two + One;
	public const double EnsureSiteColumnOnSiteCollection = Five;
	public const double GetFileInfo = Two;
	public const double GetRoleAssignments = Five;
	public const double GetFileProperties = Two;
	public const double GetCheckoutStatusInfo = Two;
	public const double CheckIn = Two;
	public const double CheckOut = Two;
	public const double UndoCheckOut = Two;
	public const double DeleteFile = Two;
	public const double OpenBinary = One;
	public const double LoadFileInfoByAbsoluteUrl = Two + One;
	public const double LoadFileWithSecurity = Five;
	public const double LoadItemPropertiesWithSiteContentTypes = Two;
	public const double AddRegularFile = Two;
	public const double AddSlicedUploadStart = Two;
	public const double AddSlicedUploadFinish = Two;
	public const double AddSlicedUploadContinue = Two;
	public const double AddSlicedUploadCancel = Two;
	public const double DoesUserHasPermission = Five;
	public const double GetSiteCollection = Two;
	public const double GetSiteCollectionWithSecurity = Five;
	public const double LoadSiteIsCurrentUserSiteAdmin = One;
	public const double GetClientContext = Two;
}

We would like to know if these estimates reflect the real costs and how we can better smooth out the delays between requests. It really matters if a request costs 1, 2, or even 5 points.

Additionally, we need to account for the number of requests a service account can make. For years, we have used the ROPC flow (delegated permissions) because it is easy to set up with CSOM and provides fine-grained control over service account access. The ClientCredentials flow, which uses application permissions, offers an all-or-nothing approach on a site with Sites.Selected permission. For CSOM, requests require client certificates (client secrets are not allowed).

Since September 2024, we have experienced increased throttling at our customers using the ROPC flow. The article Avoid getting throttled or blocked in SharePoint Online was updated in March with more details about actual limits, which is helpful but not sufficient.

We suspect that since September 2024, the ROPC flow is now treated as a user and thus subject to 'User Throttling,' whereas previously it was only subject to 'Application Throttling.' If someone can confirm or deny this, it would greatly aid our issue research. Despite searching for release notes and the SharePoint Online roadmap, we could not find an official statement about this change.

The difference between User Throttling and Application Throttling is significant. A user can make 3000 requests (not points) per 5 minutes, which translates to only 600 requests per minute. An application has no request limit but operates with Resource Units (points).

If this change occurred around September 2024 in SharePoint Online, it would explain the issues we have been experiencing with our customers since that time.

[Ashlesha-MSFT]

@digiofficerobin,
Your detailed breakdown of CSOM request costs and the challenges around opaque throttling logic highlights a real gap in SharePoint Online’s developer experience.
Since SharePoint currently doesn't expose resource unit consumption per request—making proactive rate limiting difficult—it would be valuable to raise a feature request to Microsoft.
You can submit this through below link
https://feedbackportal.microsoft.com/feedback/forum/06735c62-321c-ec11-b6e7-0022481f8472

[digiofficerobin]

@Ashlesha-MSFT Thanks for your response.

I've added the feature request to the provided feedback poral: https://feedbackportal.microsoft.com/feedback/idea/84324290-6d27-f011-9d48-000d3a0d77f1

[digiofficerobin]

Video explaining the importance of this issue: https://youtu.be/alSYB1Su41g

[digiofficerobin]

Any updates on this question?