High number of allocated objects calling sqs.DeleteMessage

[clarkmcc]

The following is a screenshot from an alloc_objects profile representing 36,153 calls to sqs.DeleteMessage in 2m which resulted in over 16,067,498 allocated objects by just the smithy-go library alone which is over 400 allocations per call. We're pushing against a significant amount of GC pressure and want to understand whether this can be improved, so here are my questions:

1. Is there a possibility to reduce optimizations/add pooling, etc to the smithy-go library?
2. In the immediate term, are there any knobs we can mess with to shrink this? For example, I noticed that almost 1M allocations came from github.com/aws/smithy-go/transport/http.withMetrics, but we don't care about those metrics, so if we can disable them and save 6% of the allocations that are happening, that would be a big win.

I'm happy to provide more info, or even share a profile privately.

[lucix-aws]

Am I interpreting this correctly that the overwhelming majority of these allocations come from the middleware system?

[clarkmcc]

@lucix-aws yes, it does seem like that is the case.

[lucix-aws]

I may not have been entirely correct. Looking at this again I do want to point out something. Middleware operates by creating a giant nested function call. The Initialize step wraps the Serialize step, which wraps Build, and so on. So, if this is just saying that a lot of allocations are coming from Initialize, that's expected, because the entire operation call actually takes place within the call stack of the initialize middleware.

[lucix-aws]

So middleware may not be the overwhelming cause of the issue in the sense that the creation of the middleware stack itself doesn't appear to be a huge problem. It is making it harder to actually find allocation hotpaths though, since all of the logic is sort of obfuscated by the middleware stack.

[clarkmcc]

So middleware may not be the overwhelming cause of the issue

I don't know that that is entirely true is it? I understand that due to how the call stack works, even non-middleware allocations (such as making the HTTP request itself) are going to appear under middleware in the flamegraph, however, if you actually look at the percentage of allocations that come from middleware itself in the form of many small, compounding allocations, we're still looking at the vast majority aren't we?

[lucix-aws]

Yes but literally everything happens in middleware. My original question/assumption was that these allocations were coming from the overhead of setting up a middleware stack, but that's not true. All that's saying is that "more allocations happen outside of the HTTP client than not", which I can believe.

[lucix-aws]

the overwhelming majority of these allocations come from the middleware system

They do, but only because the operation entirely happens within the middleware system. We don't seem to have an abnormal amount of allocations due to the system, is what I mean.

[lucix-aws]

All that aside, I can't really do anything with this. I don't know that 400 allocations per call is an issue, I have nothing to compare that to.

Serialization and deserialization are probably the biggest sources. Serialization should be pretty efficient, deserialization could probably be improved. If you could produce a more definitive listing of allocation hotpaths within the middleware stack, especially in those two areas, I'd be interested in seeing that.

[clarkmcc]

Here's what I'm seeing in my profile just to put some more concrete numbers behind things. Also, if it would be helpful, feel free to email me and I'll send you this profile (clarkm@adobe.com).

• HTTP Request + Response Deserialization: 11.3%
• Serialization: 2.1%

This tells me that 86% of the allocations are coming from stuff that isn't serialization, deserialization, or the HTTP/net areas of the code. Without going into any specifics, this just feels like a lot. I would've expected the majority of allocations to be in these core serde/http areas, but for some reason they're everywhere else.

I'm going to go through each layer and see if I can find a few examples of allocations that I feel are unnecessary but I wanted to quickly respond with some stats about how from a gut reaction something feels off.

[lucix-aws]

Note that request signing and endpoint resolution will also be major contributors.

[clarkmcc]

A couple of initial thoughts:

• signer.(*httpSigner).buildCanonicalHeaders represents 3% of allocations, mostly in heap allocating http.Headers and strings.Builder for each request. Seems like these could be pooled.
• When profiling a single round-trip request, I found that 10% of the 573 allocations by the library happen in httptrace.WithClientTrace. Can I turn off tracing?
• When profiling a single-round-trip request, 26 allocations come in the signer.buildAuthorizationHeader from having to re-grow the string builder because it's too small. Interestingly, I only see one Grow call in the code, so that makes me think buildAuthorizationHeader is being called many times.
• middleware.(*FinalizeStep).HandleMiddleware takes 3% of allocations to (I think?) re-order the middleware?
• retry.(*Attempt).HandleFinalize 2% of allocations are spent on cloning the request. Could I disable retries?
• sqs.(*withAnonymous).ResolveAuthSchemes I'm sure I don't understand all the ins and outs here, but why can't all auth-related things happen per client rather than per request?
• resolveEndpointV2Middleware takes another 1% also wondering why this is happening per request?
• sqs.timeOperationMetric this is outside the scope of this project I think, but 10.8% allocations on multiple middleware are spent on these metrics.

The biggest concern to me is there's ton of abstraction, layering, and interfacing in here which seems to always make optimizations difficult and I can't help but think if our team just hand-rolls an API client based on the API spec that we'd have massively reduced allocations. It seems like many of the allocations are happening because this library can't optimize and amortize across middlewares (for example, it's calling withStackValue multiple times across multiple layers when it could be just once).

Ultimately my question is whether there's interest from maintainers to take a hard look at all the work that's being done, determine what (if any) is wasteful, or amortizable. I have a suspicion that this would require probably some fundamental re-thinking of the library based on what I'm seeing here.

[lucix-aws]

SDK performance/scalability is an infinitely complex topic with a ton of nuance, and I don't have anywhere near enough time to get into a detailed discussion about it or address all of your points above, but here's the gist:

We are always interested in performance improvements, but the SDKs at large are also written with reliability (e.g. retries) and customizability (e.g. middleware, auth resolution, endpoint resolution) in mind. I am sure we can do some of these things better than we are doing now, but there is a certain level of minimum business requirement that is always going to convey overhead.

To put it bluntly, we are never going to live in a world where our sqs.DeleteMessage outperforms a handwritten binding that does the absolute bare minimum logic required to make that call for your specific needs / use cases. So, if performance is significant enough of a concern for you, you may wish to investigate making direct API calls yourself.

[EDIT] concerning middleware -- I'm personally not its biggest fan as a design abstraction. Indeed it is the cause of many performance inflations as you have inferred. Unfortunately there's absolutely nothing we can do about that since it's part of the public API. Removing it would require going to a v3 SDK, which we have no present plans of doing.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.