AMQP 0.9.1 - require publisher confirmations via settings/configuration

[lukebakken]

RabbitMQ series

4.2.x

Operating system (distribution) used

Linux

How is RabbitMQ deployed?

Other

What would you like to suggest for a future version of RabbitMQ?

Overview

Considering the importance of using publisher confirmations in AMQP 0.9.1, it would be great if there were a way to enforce their use. Here's the general idea:

• RabbitMQ is configured to require publisher confirmations (more on that below)
• Application sends basic.publish
• RabbitMQ accepts method, checks that confirms are required, and checks channel for confirms
• If confirms not enabled on the channel, close the channel with an error (406?)

Configuring

Requiring publisher confirmations could be configured via some or all of these methods:

• Globally via rabbitmq.conf
• Per-vhost
• Per-exchange (via policy / arguments)

Thoughts?

[kjnilsson]

Would the motivation be to try to force users into "the pit of success" or something else? If so this needs to be done for all (protocols) and not just the legacy one.

[lukebakken]

Would the motivation be to try to force users into "the pit of success" or something else?

Yep!

[lhoguin]

What's the difference between accepting publishes without confirms and the client being forced to get confirms but ignoring them? Knowledge of them? Something else?

[michaelklishin]

Some clients can automagically handle confirms in a mostly reasonable way. The .NET client 7.x already does that, we will likely spend some time on expanding this opinionated feature to more clients.

[lhoguin]

Cool, didn't know that. In that case it's the kind of setting that we should aim to enable by default at some point, safe by default. Even if only in 5.0 or later.

[stefanmoser]

@kjnilsson Yes, "the pit of success" is a great way to say it. Since publisher confirms are the only back pressure mechanism that the server has to the client, they're essential for protecting the server from being overwhelmed.

[stefanmoser]

Good to know. Is it different in AMQP 0.9.1 and AMQP 1.0 too?

[the-mikedavis]

Yeah we probably don't need to require confirms in 1.0 since it has a non-optional flow control mechanism https://www.rabbitmq.com/blog/2024/09/02/amqp-flow-control

[lukebakken]

Yes I suggested AMQP 0.9.1 as it is the worst offender 😉 ...but I wouldn't be surprised if all protocol users are "surprised" that confirmations are available. receipt header in SEND in the STOMP protocol, for instance.

[stefanmoser]

@the-mikedavis We should run some tests with each protocol to see the difference in how they behave under extreme load.

[kjnilsson]

Good to know. Is it different in AMQP 0.9.1 and AMQP 1.0 too?

AMQP 0.9.1 (legacy) and AMQP 1.0 (AMQP) are very different in this regard. AMQP (1.0) is the recommended protocol going forward really, it is a superior transfer protocol.

[michaelklishin]

Combined with automagical handling of publishing confirms, like in the .NET client starting with 7.0, this feature makes sense.

However, I'm afraid the penalty hit of this setting on the hot code path will be meaningful, and it will take a long time to update all clients to support automagical publisher confirm handling. Right now we have little mileage on the .NET client 7.x in the grand scheme of things.

[the-mikedavis]

Requiring publisher confirms is a means to exert backpressure on clients which flood the broker. We should also consider if we could provide demand-balance (backpressure or even load shedding) aside from confirms. In protocols where flow-control isn't built in (like non-legacy AMQP or MQTT) maybe we could temporarily stop reading from the connection TCP socket like the alarm mechanism when a throughput threshold is exceeded, or drop publishes which do not require confirms.