diff --git a/docs/user-manual/amqp-address-federation.adoc b/docs/user-manual/amqp-address-federation.adoc new file mode 100644 index 00000000000..0344af2fc98 --- /dev/null +++ b/docs/user-manual/amqp-address-federation.adoc @@ -0,0 +1,147 @@ += Address Federation +:idprefix: +:idseparator: - +:docinfo: shared + +Address federation can be thought of as full multicast over the connected brokers. +Every message sent to address on `Broker-1` will be delivered to every queue bound to that address on that broker, but also will be delivered to the matching address on `Broker-2`, `Broker-3` ... `Broker-N` and all the queues bound to that address. + +Address federation creates consumers that read messages from the matching address on the remote peer when demand is detected on a local address that matches a specific federation address policy. +On the remote peer a queue binding is created on the federated address which will store and forward messages sent to that address back to the initiating broker. +Once a message arrives at the federation address consumer on the initiating broker it is then published into the local address which routes the message into each of the queues bound to that address. + +Address federation can be configured to operate in a single direction between peers or can configured to operate in both directions between peers allowing message to flow in both directions delivering messages to consumers on either broker that were sent to the matching address on the opposing broker instance. + +image::images/amqp-federation-address-concept.png[] + +== Common Address Federation Topologies + +The sections that follow discuss some common address federation topologies and some configuration considerations specific to those topologies. + +=== Two way Address Federation + +This is a common case where consumers on either broker should receive messages sent from producers both on the local broker and on the remote broker. +Messages only cross the federation connection if a consumer exists on the remote address otherwise messages do not leave the local broker. + +image::images/amqp-federation-address-bi-directional.png[] + +This configuration can be setup by adding a federation policy on both broker instances that configure the address to federate and a connection URI pointing to the opposing broker. + +[,xml] +---- + + + + + + + + + +---- + +It is also possible to configure this on only one of the two brokers using a remote address federation policy in the broker connection configuration. + +[,xml] +---- + + + + + + + + + + + + +---- + +In either of these two configurations the federation implementation will protect itself from looping messages so unlike other implementations you do not need to configure a `max-hops` setting. + +=== Three Broker Mesh + +An expansion of the previous configuration is a three broker mesh where each broker creates an address federation for the same address on each of the other two brokers when local demand is present. +In this scenario any message published on an address on one of the brokers will be sent to consumers on that address that are active on the other two brokers. + +image::images/amqp-federation-address-mesh.png[] + +Because this configuration now creates a loop with more than two brokers the configuration must add a `max-hops` value to ensure that the messages sent to one of the brokers does not propagate beyond the two closest neighbors in the loop otherwise the messages would loop forever amongst the three brokers. + +[,xml] +---- + + + + + + + + + + + + + + + + +---- + +By configuring each of the brokers with a `max-hops` value as above the message will be stopped from moving beyond the neighboring broker that receives it and no message looping occurs. + +=== Address Federation Ring + +Another less common configuration is a good demonstration of a case where using the `max-hops` settings is critical for preventing an infinite loop of message federation. +This is also known as a ring and involves creating an address federation configuration that connects to a neighboring broker amongst three or more broker instances. +In this scenario the configuration must set a `max-hops` value of N-1 where N is the number of brokers in the ring. + +image::images/amqp-federation-address-ring.png[] + +To achieve this configuration each broker would need to add a broker connection for its neighbor as follows. +Note the `max-hops=4` in this case since there are five brokers in the diagram, this ensures a federated message never cross back through the originating broker and thus cannot create an message loop. + +[,xml] +---- + + + + + + + + + + +---- + +=== Hub and Spoke Address Federation + +Another configuration that allows consumers on brokers to read messages produced on any other broker peer is a hub and spoke configuration. +In this configuration spoke brokers connect to a central hub and message traffic between the spokes passes through that hub broker which centralizes communication and allows the spokes to only need to know about one other broker, the hub broker. +When demand exists on any of the spokes they create a federation consumer link to the hub, this in turn causes the hub to create federation consumer links to each of the spokes where the address exists regardless of the presence of a producer on those spokes in order to ensure that when a producer arrives the messages that it sends reach the other spokes with attached consumers (this is shown on diagram at `spoke 3`). + +image::images/amqp-federation-address-hub-spoke.png[] + +This is configured on the spoke brokers much the same as dual address federation where a local and remote policy is configured both sides to listen for demand on the given address and federate messages when demand is detected. +This setup allows the hub to now sense demand and federate from the spoke as needed and since each spoke mirrors this configuration the message can travel in either direction based on which spoke a consumer has subscribed to the federated address and which spoke a producer sends to the address. +There is no need to configure a `max-hops` value in this configuration since the federation implementation prevents a loop in this case in the same manner it does for dual federation configurations. + +[,xml] +---- + + + + + + + + + + + + +---- + +While this is not the only configuration that can create this setup it does represent a simple and flexible configuration where spoke can be added without making configuration changes on the hub. diff --git a/docs/user-manual/amqp-broker-connections.adoc b/docs/user-manual/amqp-broker-connections.adoc index 057b812054a..cbdbfda4e31 100644 --- a/docs/user-manual/amqp-broker-connections.adoc +++ b/docs/user-manual/amqp-broker-connections.adoc @@ -45,7 +45,8 @@ The default value is `5000`. reconnect-attempts:: default is -1 meaning infinite auto-start:: -Should the broker connection start automatically with the broker. Default is `true`. +Should the broker connection start automatically with the broker. +Default is `true`. If `false` it is necessary to call a management operation to start it. NOTE: The connection URI options for transport settings, such as enabling and configuring TLS, are common with other broker connector URIs. @@ -111,6 +112,7 @@ To specify more than 2 elements use a comma separated list after the first pair, ---- + .Broker Connection - Reconnecting and Failover image::images/broker-connection-failover.jpg[] @@ -118,8 +120,7 @@ The previous example portrays a case of connection failure towards ServerA. The system would try to connect to ServerA, BackupA, and BackupB until it successfully connects to one of these nodes. WARNING: For failover on mirrored brokers, all nodes in the list must be high-available (HA) failover nodes. -Specifying independents nodes in the mirror list can cause consistency issues between the message -sent to the mirror and the acknowledgements received on the mirror. +Specifying independents nodes in the mirror list can cause consistency issues between the message sent to the mirror and the acknowledgements received on the mirror. == Mirroring @@ -174,6 +175,7 @@ Examples: ==== * Address exclusion will always take precedence over address inclusion. * Address matching on mirror elements is prefix-based and does not support wild-card matching. + ==== sync:: If set to `true` any client blocking operation will be held until the mirror has confirmed receiving the operation. @@ -257,6 +259,7 @@ If messages are sent on DC1 (_DataCenter1_) these will be automatically transfer Message acknowledgements received on DC2 will be automatically related back to DC1. The only exception to that rule would be if there were already consumers with pending messages on any server, where a mirrored acknowledgement will not prevent the message being consumed by both consumers. + [NOTE] ==== It is recommended to not have active consumers on both servers. @@ -429,7 +432,7 @@ IMPORTANT: To avoid confusion it is recommended that `address` `name` and `queue == Federation Broker federation allows the local broker to create remote receivers for addresses or queues that have local demand. -Conversely, the broker connection can send federation configuration to the remote broker causing it to create receivers on the local broker based on remote demand on an address or queue over this same connection. +Conversely, the broker connection can send connection scoped federation configuration to the remote broker causing it to create receivers on the local broker based on remote demand on an address or queue over this same connection. Add a `` element within the `` element to configure federation to the broker instance. The `` contains all the configuration for authentication and reconnection handling. @@ -461,101 +464,30 @@ The broker connection federation configuration consists of one or more policies ---- -=== Local and remote address federation +=== Address federation -Local or Remote address federation configures the local or remote broker to watch for demand on addresses. -When demand exists it will create a consumer on the matching address on the opposing broker. -Because the consumers are created on addresses on the opposing broker, the authentication credentials supplied to the broker connection must have sufficient access to read from (local address federation) or write to the matching address (remote address federation) on the opposing broker. +Address federation can be thought of as full multicast over a set of loosely coupled brokers. +Every message sent to address on `Broker-1` will be delivered to every queue bound to that address on that broker, but also will be delivered to the matching address on `Broker-2`, `Broker-3` ... `Broker-N` and all the queues bound to that address. -An example of address federation configuration is shown below. -Local and remote address policies have identical configuration parameters, only the policy element names are different. +.Address Federation +image::images/amqp-federation-address-concept.png[] -[,xml] ----- - - - - - - - - - - ----- +For further details please see the xref:amqp-address-federation.adoc#address-federation[Address Federation] documentation. -name:: -The name of the policy, these names should be unique within a broker connection's federation policy elements. -include:: -The address-match pattern to use to match included addresses, multiple of these can be set. -If none are set all addresses are matched. -exclude:: -The address-match pattern to use to match excluded addresses, multiple of these can be set, or it can be omitted if no excludes are needed. -max-hops:: -The maximum number of hops that a message can perform to be federated. -Default value is `0` and will work for most simple federation deployments. -However, in certain topologies a higher value may be required to prevent looping. -auto-delete:: -For address federation, a durable queue is created on the broker from which messages are being federated. -Set this property to `true` to mark the queue for automatic deletion once the initiating broker disconnects and the delay and message count parameters are met. -This is a useful option if you want to automate the cleanup of dynamically-created queues. -The default value is `false`, which means that the queue is not automatically deleted. -auto-delete-delay:: -The amount of time in milliseconds after the initiating broker has disconnected before the created queue can be eligible for `auto-delete`. -The default value is `0` when option is omitted. -auto-delete-message-count:: -After the initiating broker has disconnected, the maximum number of messages allowed in the remote queue for the queue to be eligible for automatic deletion. -The default value is `0`. -enable-divert-bindings:: -Setting to `true` enables divert bindings to be listened-to for demand. -If a divert binding with an address matches the included addresses for the address policy, any queue bindings that match the forwarding address of the divert creates demand. The default value is `false`. +=== Queue federation -=== Local and remote queue federation +Queue federation offers a means of load balancing message queues across multiple broker instances. +Messages sent to a queue on `Broker-1` will be consumed and sent to the matching queue on `Broker-2` if there is no local consumer available to consume the message. -Local or Remote queue federation configures the local or remote broker to watch for demand on queues and when demand exists it will create a consumer on the matching queue on the opposing broker. -Because the consumers are created on queues on the opposing broker, the authentication credentials supplied to the broker connection must have sufficient access to read from (local queue federation) or write to the matching queue (remote queue federation) on the opposing broker. +.Queue Federation +image::images/amqp-federation-queue-concept.png[] -An example of queue federation configuration is shown below. -Local and remote queue policies have identical configuration parameters, only the policy element names are different. - -[,xml] ----- - - - - - - - - - - ----- - -name:: -The name of the policy. -These names should be unique within a broker connection's federation policy elements. -include:: -The queue-match pattern to use to match included queues. -Multiple of these can be set. -If none are set all queues are matched. -exclude:: -The queue-match pattern to use to match excluded queues. -Multiple of these can be set, or it can be omitted if no excludes are needed. -priority-adjustment:: -When federation consumers are created this value can be used to ensure that those federation consumers have a lower priority value than other local consumers on the same queue. -The default is `-1`. -include-federated:: -Controls if consumers on a queue which come from federation instances should be counted when observing a queue for demand, by default this value is `false` and federation consumers are not counted. +For further details please see the xref:amqp-queue-federation.adoc#queue-federation[Queue Federation] documentation. == Bridges -AMQP Bridges allows the local broker to create remote receivers for addresses or queues that have local demand and match configured policy constraints. Conversely the broker can create remote senders for addresses or queues that exist on the local broker and match configured policy constraints. +AMQP Bridges allows the local broker to create remote receivers for addresses or queues that have local demand and match configured policy constraints. +Conversely the broker can create remote senders for addresses or queues that exist on the local broker and match configured policy constraints. Add a `` element within the `` element to configure AMQP bridge to the broker instance. The `` contains all the configuration for authentication and reconnection handling. @@ -589,7 +521,9 @@ The broker connection bridge configuration consists of one or more policies that === Bridging from remote addresses and queues -Bridging from a remote address or queue involves monitoring a local address or queue for demand and reacting when demand is added or removed. When demand exists the bridge will create a receiver on the matching address or queue on the opposing AMQP peer. Because the receivers are created on addresses and queues on the opposing peer, the authentication credentials supplied to the broker connection must have sufficient access to the bridged address or queue on the remote peer in order to consume messages from them. +Bridging from a remote address or queue involves monitoring a local address or queue for demand and reacting when demand is added or removed. +When demand exists the bridge will create a receiver on the matching address or queue on the opposing AMQP peer. +Because the receivers are created on addresses and queues on the opposing peer, the authentication credentials supplied to the broker connection must have sufficient access to the bridged address or queue on the remote peer in order to consume messages from them. An example of AMQP address and queue bridge from configurations are shown below. @@ -613,41 +547,54 @@ An example of AMQP address and queue bridge from configurations are shown below. The set of common policy attributes for bridge from address and queue policies: name:: -The name of the policy. These names should be unique within a broker connection's bridge policy elements. +The name of the policy. +These names should be unique within a broker connection's bridge policy elements. priority:: A configured priority value which if set is applied remote receiver created by this bridge policy, this value is added to the receiver link properties and if supported will influence the priority assigned on the remote. priority-adjustment:: -When bridge receivers are created this value can be used to ensure that those bridge receivers have a lower priority value than other receivers on the same address or queue. The default is `-1`. +When bridge receivers are created this value can be used to ensure that those bridge receivers have a lower priority value than other receivers on the same address or queue. +The default is `-1`. includeDivertBindings:: Should the bridge from address policy include divert bindings in the checks that control if a remote receiver is created to bridge from an address. filter:: Optional filter string that is validated as a JMS selector string and is applied to the receiver that is bridging messages from and address or queue on the remote peer. remoteAddress:: -An optional override of address that is set as the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. The default action is to assign either the address or queue name depending on whether the policy is bridging an address or queue. +An optional override of address that is set as the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. +The default action is to assign either the address or queue name depending on whether the policy is bridging an address or queue. remoteAddressPrefix:: -An optional address prefix that is set on the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. The default value is `null` and no prefix is applied to the source address. +An optional address prefix that is set on the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. +The default value is `null` and no prefix is applied to the source address. remoteAddressSuffix:: -An optional address suffix that is set on the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. The default value is `null` and no suffix is applied to the source address. +An optional address suffix that is set on the address in the Source configuration of the AMQP receiver that is bridging messages from the remote peer. +The default value is `null` and no suffix is applied to the source address. remoteTerminusCapabilities:: -Optional set of remote source capabilities that are set on the receiver when bridging an address or queue from the remote peer. The capabilities are configured as a comma delimited list whose default value is `null`. +Optional set of remote source capabilities that are set on the receiver when bridging an address or queue from the remote peer. +The capabilities are configured as a comma delimited list whose default value is `null`. Policy attributes that apply to AMQP bridge from address policies only: enable-divert-bindings:: -Instructs the receive from address policy to look at diverts that matching address would forward to and check for demand on those diverts when processing demand on the matched address. By default this option is `false` and diverts are not checked. +Instructs the receive from address policy to look at diverts that matching address would forward to and check for demand on those diverts when processing demand on the matched address. +By default this option is `false` and diverts are not checked. use-durable-subscriptions:: -Instructs the receive from address policy that it should use an JMS over AMQP mapped durable subscription to create the remote receiver which is meant to result in a durable address subscription being created for receivers on addresses that have local demand. By adding a durable subscription the remote will accumulate messages when the broker connection if down and will recover stored messages if the remote server is shutdown and restarted. By default this option is set to `false` to avoid unwanted accumulation of remote messages and if enabled intervention may be needed from remote management to remove old subscriptions or clear space as needed. +Instructs the receive from address policy that it should use an JMS over AMQP mapped durable subscription to create the remote receiver which is meant to result in a durable address subscription being created for receivers on addresses that have local demand. +By adding a durable subscription the remote will accumulate messages when the broker connection if down and will recover stored messages if the remote server is shutdown and restarted. +By default this option is set to `false` to avoid unwanted accumulation of remote messages and if enabled intervention may be needed from remote management to remove old subscriptions or clear space as needed. Configuration policy elements of which more than one entry per policy is allowed: include:: -The address-match or queue-match pattern to use to match included addresses and queues. Multiple of these can be set but if none is set then no matches would be made. +The address-match or queue-match pattern to use to match included addresses and queues. +Multiple of these can be set but if none is set then no matches would be made. exclude:: -The address-match or queue-match pattern to use to match excluded addresses and queues. Multiple of these can be set, or it can be omitted if no excludes are needed. +The address-match or queue-match pattern to use to match excluded addresses and queues. +Multiple of these can be set, or it can be omitted if no excludes are needed. === Bridging to remote addresses and queues -Bridging to a remote address or queue involves monitoring for the existence of a local address or queue amd reacting when one is added or removed. When the target address or queue exists the bridge will create a sender on the matching address or queue on the opposing AMQP peer. Because the senders are created on addresses and queues on the opposing peer, the authentication credentials supplied to the broker connection must have sufficient access to the bridged address or queue on the remote peer in order to produce messages to them. +Bridging to a remote address or queue involves monitoring for the existence of a local address or queue amd reacting when one is added or removed. +When the target address or queue exists the bridge will create a sender on the matching address or queue on the opposing AMQP peer. +Because the senders are created on addresses and queues on the opposing peer, the authentication credentials supplied to the broker connection must have sufficient access to the bridged address or queue on the remote peer in order to produce messages to them. An example of AMQP address and queue bridge to configurations are shown below. @@ -671,33 +618,42 @@ An example of AMQP address and queue bridge to configurations are shown below. The set of common policy attributes for bridge to address and queue policies: name:: -The name of the policy. These names should be unique within a broker connection's bridge policy elements. +The name of the policy. +These names should be unique within a broker connection's bridge policy elements. priority:: A configured priority value which if set is applied local receiver created by this bridge policy, this value is added to the receiver link properties and if supported will influence the priority assigned on the local server. priority-adjustment:: -When bridge senders are created this value can be used to ensure that those bridge senders have a lower priority value than other senders on the same address or queue. The default is `-1`. +When bridge senders are created this value can be used to ensure that those bridge senders have a lower priority value than other senders on the same address or queue. +The default is `-1`. filter:: Optional filter string that is validated as a JMS selector string and is applied to the sender that is bridging messages to an address or queue on the remote peer. remoteAddress:: -An optional override of address that is set as the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. The default action is to assign either the address or queue name depending on whether the policy is bridging an address or queue. +An optional override of address that is set as the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. +The default action is to assign either the address or queue name depending on whether the policy is bridging an address or queue. remoteAddressPrefix:: -An optional address prefix that is set on the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. The default value is `null` and no prefix is applied to the target address. +An optional address prefix that is set on the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. +The default value is `null` and no prefix is applied to the target address. remoteAddressSuffix:: -An optional address suffix that is set on the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. The default value is `null` and no suffix is applied to the target address. +An optional address suffix that is set on the address in the Target configuration of the AMQP sender that is bridging messages to the remote peer. +The default value is `null` and no suffix is applied to the target address. remoteTerminusCapabilities:: -Optional set of remote target capabilities that are set on the sender when bridging an address or queue to the remote peer. The capabilities are configured as a comma delimited list whose default value is `null`. +Optional set of remote target capabilities that are set on the sender when bridging an address or queue to the remote peer. +The capabilities are configured as a comma delimited list whose default value is `null`. Policy attributes that apply to AMQP bridge to address policies only: use-durable-subscriptions:: -Instructs the send to address policy that bindings created on local addresses that match the configured policy includes should create a durable address binding which will be reloaded on server restart and will store messages that were sent to the address while the broker connection is down. The option is `false` by default and a volatile binding is created on the local address that is deleted whenever the broker connection is down so that messages are not retained. +Instructs the send to address policy that bindings created on local addresses that match the configured policy includes should create a durable address binding which will be reloaded on server restart and will store messages that were sent to the address while the broker connection is down. +The option is `false` by default and a volatile binding is created on the local address that is deleted whenever the broker connection is down so that messages are not retained. Configuration policy elements of which more than one entry per policy is allowed: include:: -The address-match or queue-match pattern to use to match included addresses and queues. Multiple of these can be set but if none is set then no matches would be made. +The address-match or queue-match pattern to use to match included addresses and queues. +Multiple of these can be set but if none is set then no matches would be made. exclude:: -The address-match or queue-match pattern to use to match excluded addresses and queues. Multiple of these can be set, or it can be omitted if no excludes are needed. +The address-match or queue-match pattern to use to match excluded addresses and queues. +Multiple of these can be set, or it can be omitted if no excludes are needed. === Examples diff --git a/docs/user-manual/amqp-federation-configuration-glossary.adoc b/docs/user-manual/amqp-federation-configuration-glossary.adoc new file mode 100644 index 00000000000..d7b48350440 --- /dev/null +++ b/docs/user-manual/amqp-federation-configuration-glossary.adoc @@ -0,0 +1,258 @@ += AMQP Federation Configuration Glossary +:idprefix: +:idseparator: - +:docinfo: shared + +Federation configuration occurs in a broker connection element where a set of policies defines the desired address or queue federation and the locality of that federation (local or remote). +A single federation instance is supported per broker connection element however each federation instance can contain multiple address and queue policy elements. +Address and queue policy configuration exist for both the local and the remote side of the federation over the broker connection. +An example of such a federation configuration element follows which contains both local and remote policy elements. + +[,xml] +---- + + + + + + + + + + + + + + + + + + + + +---- + +Local or Remote federation policies configure the local or remote broker to perform the federation operation over the defined connection but otherwise use the same configuration attributes and properties. +The only difference is whether the federation occurs locally meaning local demand triggers consumers to be created towards the remote broker, or if remote demand triggers consumers to be created towards the local broker (local meaning the one that carries the configuration). +For this reason the remainder of this documentation will use local federation policies when showing configuration examples for simplicity. + +Federation configuration can contain multiple policies of the same type (address or queue) along with a mix of types as seen in the configuration example above. +This allows for policies with differing configurations for federating different addresses and queues within the same broker connection. +It is important to note that the policy names must be unique within the federation configuration. +The following configuration example demonstrates the mix of policy configurations within a single federation configuration. + +[,xml] +---- + + + + + + + + + + + + + + +---- + +It is also supported to have only remote federation policies within a broker connection federation element. +In this case the policy is sent across the connection to the remote broker and federation occurs on the remote for the life of that broker connection. + +[,xml] +---- + + + + + + + + + + +---- + +=== Address federation configuration elements + +[,xml] +---- + + + + + + + + + + + + + +---- + +The address policy element can be configured with the following set of attributes, or if the value is omitted the stated default value will be applied to the deployed policy. + +name:: +The name of the policy, these names should be unique within a broker connection's federation policy elements. +max-hops:: +The maximum number of hops that a message can perform to be federated. +Default value is `0` and will work for most federation deployments. +However, in certain topologies where a loop is configured a higher value may be required to prevent infinite looping of messages. +auto-delete:: +For address federation, a durable queue is created on the broker from which messages are being federated. +Set this property to `true` to mark the queue for automatic deletion once the initiating broker disconnects and the delay and message count parameters are met. +This is a useful option if you want to automate the cleanup of dynamically-created queues. +The default value is `false`, which means that the queue is not automatically deleted. +auto-delete-delay:: +The amount of time in milliseconds after the initiating broker has disconnected before the created queue can be eligible for `auto-delete`. +The default value is `0` when option is omitted. +auto-delete-message-count:: +After the initiating broker has disconnected, the maximum number of messages allowed in the remote queue for the queue to be eligible for automatic deletion. +The default value is `0`. +enable-divert-bindings:: +Setting to `true` enables divert bindings to be listened-to for demand. +If a divert binding with an address matches the included addresses for the address policy, any queue bindings that match the forwarding address of the divert creates demand. +The default value is `false`. + +The following set of elements can be configured within the address policy to control the behavior of the deployed policy. +For the policy to actually match any addresses on the target broker an include must be provided at a minimum. + +include:: +The address-match pattern to use to match included addresses, multiple of these can be set per policy. +exclude:: +The address-match pattern to use to match excluded addresses, multiple of these can be set, or it can be omitted if no excludes are needed. +property:: +A key / value property setting that allows for configuration options to be set either at the root federation level or within each federation policy that affects some defined behavior of the policy implementation. + +=== Queue federation configuration elements + +[,xml] +---- + + + + + + + + + + + + +---- + +The queue policy element can be configured with the following set of attributes, or if the value is omitted the stated default value will be applied to the deployed policy. + +name:: +The name of the policy. +These names should be unique within a broker connection's federation policy elements. +priority-adjustment:: +When federation consumers are created this value can be used to ensure that those federation consumers have a lower priority value than other local consumers on the same queue. +The default is `-1`. +include-federated:: +Controls if consumers on a queue which come from federation instances should be counted when observing a queue for demand, by default this value is `false` and federation consumers are not counted as local demand. + +The following set of elements can be configured within the queue policy to control the behavior of the deployed policy. + +include:: +The queue-match pattern to use to match included queues and enforce that a matched queue is also bound to a specific address. +Multiple of the include elements can be set per policy, but if none are set all queues would be matched unless there is an exclude configured. +exclude:: +The queue-match pattern to use to match excluded queues that the policy should ignore. +Multiple of these can be set, or it can be omitted if no excludes are needed. +property:: +A key / value property setting that allows for configuration options to be set either at the root federation level or within each federation policy that affects some defined behavior of the policy implementation. + +=== Federation policy property configuration index + +The following properties can be set either at the federation element level or set per-policy in order to apply different behaviors to different policies within the same federation configuration. +When remote address or queue policies are configured the value of these properties are sent to the remote along with the policies to ensure both sides are operating with the same configurations. +Certain value can also be set in the broker connection URI when noted if not set in any of the available locations the configuration default value is used. + +|=== +| Name | Type | Description | Default + +| attach-timeout +| Integer +| Controls the timeout value for a federation receiver link attach to complete before the attach attempt is considered to have failed. The value is configured in seconds. +| 30 (Seconds) + +| amqpCredits +| Integer +| Defines the amount of credits to batch to an AMQP receiver link and the top up limit when sending more credit once the credits are determined to be running low. This value will be read from the broker connection URI if one was specified there and not set in the federation policy.. +| 1000 (broker default) + +| amqpLowCredits +| Integer +| A low water mark for receiver credits that indicates more should be sent to top it up to the original credit batch size. This value will be read from the broker connection URI if one was specified there and not overridden in the federation policy. +| 300 (broker default) + +| amqpPullConsumerCredits +| Integer +| Defines the amount of credits to batch to an AMQP receiver link and the top up value when sending more credit once the broker has capacity available for them. In order for this value to take affect the `amqpCredits` value must be set to zero. If `amqpCredits` ist set to zero and this option is not configured the default value is used. +| 100 + +| minLargeMessageSize +| Integer +| Used to configure the value to use when considering if a message is a large message for federation consumers. This value will be read from the broker connection URI if one was specified there and not overridden in the federation policy. +| Broker default + +| ignoreAddressBindingFilters +| Boolean +| Used to configure if federation address consumers should filter messages using the filters defined on individual matching address bindings. This value can be used to reduce the amount of messages that are federated for an address if the local consumers use filters but must be enabled with care as it can easily lead to duplicate messages being dispatched from the remote as each unique address binding filter creates a remote consumer with that matching filter. If any filters overlap in the messages they allow to pass or there are bindings without a filter then the remote will be sending the same message to multiple federation consumers in order to provide each group of bindings with their matching messages. +| true + +| ignoreQueueConsumerFilters +| Boolean +| Used to configure if federation queue consumers should filter messages using the filters defined on individual queue subscriptions. This can be used to prevent multiple subscriptions on the same queue based on local demand with differing subscription filters but does imply that message that don't match those filters would be federated to the local broker. +| false + +| ignoreQueueConsumerPriorities +| Boolean +| Used to control if federation queue consumers should apply a consumer priority offset based on the subscription priority or should use a singular priority offset based on policy configuration. This can be used to prevent multiple subscriptions on the same queue based on local demand with differing consumer priorities but does imply that care needs to be taken to ensure remote consumers would still normally have a higher priority value than the configured default priority offset. This value is off by default as that is the safest and simplest to manage as a single offset value is set and all federation consumer will be know to be using that value. +| true + +| amqpUseModifiedForTransientDeliveryErrors +| Boolean +| Used to configure how a federation receiver should respond to delivery errors indicating that an address is full and cannot accept messages at this time. By default we will send Modified outcomes with the delivery failed value set to true such that the remote will deliver the message again after incrementing the delivery count of the message. Disabling this causes the Rejected outcome to be sent instead which is a terminal outcome which results in message being dropped and possibly sent to a DLA on is configured. +| true + +| amqpLinkQuiesceTimeout +| Integer +| Defines the time in milliseconds that a receiver will wait before considering a pending quiesce timeout to have failed and should close the link. This option can be used to override the value specified on the connector URI to allow federation to operate with a different default. This timeout applies to AMQP links that are being drained due to a transient delivery error such as the address being full and the address policy is set to send a failed response to the remote sender. If the drain on transient delivery errors option is set to false then this option is ignored. +| Broker default (10 minutes) + +| amqpDrainOnTransientDeliveryErrors +| Boolean +| Used to control if a federation receiver should drain the link credit when a transient delivery error such as the address being full occurs. This option can be used to override the value specified on the connector URI to allow federation to operate with a different default. +| Broker default (true). + +| receiverQuiesceTimeout +| Integer +| When a federation receiver link is being drained due to removal of local demand this timeout value enforces a maximum wait for drain and processing of in-flight messages before the link is forcibly terminated with the assumption that the remote is no longer responding. +| 60s + +| addressReceiverIdleTimeout +| Integer +| When a federation address receiver link has been successfully drained after demand was removed from the federated resource, this value controls how long the link can remain in an attached but idle state before it is closed. This can allow for quicker restoration of federated messages if the demand on a queue is removed and re-added quickly as the link is drained of credit but not immediately closed. For address receivers the default is kept short to prevent holding broadcast messages on the remote for a receiver which might not return in a timely manner. +| 5s + +| queueReceiverIdleTimeout +| Integer +| When a federation queue receiver link has been successfully drained after demand was removed from the federated resource, this value controls how long the link can remain in an attached but idle state before it is closed. This can allow for quicker restoration of federated messages if the demand on a queue is removed and re-added quickly as the link is drained of credit but not immediately closed. +| 60s diff --git a/docs/user-manual/amqp-queue-federation.adoc b/docs/user-manual/amqp-queue-federation.adoc new file mode 100644 index 00000000000..8a42c5dcbba --- /dev/null +++ b/docs/user-manual/amqp-queue-federation.adoc @@ -0,0 +1,120 @@ += Queue Federation +:idprefix: +:idseparator: - +:docinfo: shared + +Queue federation offers a means of load balancing message queues across multiple broker instances. +Messages sent to a queue on `Broker-1` can be consumed and sent to the matching queue on `Broker-2` via a federation consumer if there is no local consumer available to consume the message. + +Queue federation creates consumers that consume messages from the matching queue on the remote peer when demand is detached on the local queue that matches a specific queue federation policy. +The federation consumer acts as an addition consumer on the target remote queue competing with consumers on that remote queue for messages. +Federation queue consumers can be configured to always have lower priority than the local queue consumers and also can be configured to only pull messages from the remote queue if there is capacity to handle those messages on the local broker. + +image::images/amqp-federation-queue-concept.png[] + +Queue federation can be configured operate in a single direction between peers or can configured to operate in both directions between peers allowing message to flow in both directions delivering messages to consumers on either broker that were sent to the matching queue on the opposing broker instance. +When configured for dual federation messages can migrate back and forth between brokers depending on capacity and available local consumers, this back and forth message flow can be reduced by ensuring that the federation policy uses a pull when space is available configuration which will only pull a batch of messages from the remote if the local queue does not already have a backlog. + +== Common Queue Federation Topologies + +The sections that follow discuss some common queue federation topologies and some configuration considerations specific to those topologies. + +=== Simple Queue Load Balance + +The simplest configuration of Queue federation involves a two broker setup where `broker-1` creates a federation consumer to `broker-2` when demand is present on a federated queue. +The messages sent by a producer on `broker-2` are shared between any local consumer and the federation consumer. + +image::images/amqp-federation-queue-simple-lb.png[] + +This configuration can be setup by adding a federation policy on `broker-1` that configure the queue to federate and a connection URI pointing to the opposing broker. +Note that this configuration adds properties to configure the federation queue consumer to pull messages as opposed to the default which allows messages to flow regardless of local queue backlogs, this helps to prevent pulling messages from the remote broker unless the local broker has a shortage which helps to prevent starving consumers directly attached to the remote broker. + +[,xml] +---- + + + + + + + + + + + +---- + +While this configuration provides basic sharing of messages between brokers it can result in messages being stuck on `broker-1` if the consumer goes down as messages cannot flow back to `broker-2` to be be read by a consumer there which is why the next topology is often deployed. + +=== Bi-directional Queue Federation + +In a bi-directional queue federation configuration both brokers are configured to federate a given queue between them allowing messages to flow in either direction. +This configuration allows message that have been federated to one broker to flow back to the other if local demand is removed which can prevent stuck messages. + +image::images/amqp-federation-queue-bi-directional.png[] + +This configuration can be setup by adding a broker connection element to each of the broker instances with the same local queue federation policy or could be defined on only one broker using both a local and remote queue federation policy as shown below. + +[,xml] +---- + + + + + + + + + + + + + + +---- + +This configuration would result in only a single connection between the two brokers. +Each policy will be configured by the shared properties of the federation element to use a pull consumer that batches ten messages at a time whenever the target queue has no pending messages already waiting to be read. + +=== Queue Federation with Intermediary Brokers + +In some use cases a federated queue may need to cross through an intermediary broker before reaching a consumer. +For example consider a three broker chain where messages produced on `broker-1` must pass through `broker-2` before reaching a consumer on `broker-3`. + +image::images/amqp-federation-queue-with-hop.png[] + +To accomplish this the federation queue policy for the intermediary broker must have additional configuration to allow for federation of messages when no local demand is present but a federation consumer is. +Normally federation queue policies do no consider a federation queue binding on a local queue as demand to avoid creating loops. +To disable that behavior the policy supports the attribute `include-federated` which defaults to `false`. +The XML configuration for `broker-2` is below that employs this option. + +[,xml] +---- + + + + + + + + + +---- + +The configuration on `broker-3` looks essentially the same as that on `broker-2` however the `include-federated` attribute can be omitted. + +[,xml] +---- + + + + + + + + + +---- + +With this configuration in place messages produced on `broker-1` will be federated to `broker-2` and then on to `broker-3` whenever a consumer is attached to `broker-3`. +It should be noted however that in a configuration such as this that messages can be left on `broker-2` if they were arriving there at the same time that the consumer on `broker-3` had gone away and the federation link between `broker-3` and `broker-2` was gearing down. \ No newline at end of file diff --git a/docs/user-manual/images/amqp-federation-address-bi-directional.png b/docs/user-manual/images/amqp-federation-address-bi-directional.png new file mode 100644 index 00000000000..47a43061d88 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-address-bi-directional.png differ diff --git a/docs/user-manual/images/amqp-federation-address-concept.png b/docs/user-manual/images/amqp-federation-address-concept.png new file mode 100644 index 00000000000..0982a93566d Binary files /dev/null and b/docs/user-manual/images/amqp-federation-address-concept.png differ diff --git a/docs/user-manual/images/amqp-federation-address-hub-spoke.png b/docs/user-manual/images/amqp-federation-address-hub-spoke.png new file mode 100644 index 00000000000..ccb1f826a87 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-address-hub-spoke.png differ diff --git a/docs/user-manual/images/amqp-federation-address-mesh.png b/docs/user-manual/images/amqp-federation-address-mesh.png new file mode 100644 index 00000000000..38316b678fd Binary files /dev/null and b/docs/user-manual/images/amqp-federation-address-mesh.png differ diff --git a/docs/user-manual/images/amqp-federation-address-ring.png b/docs/user-manual/images/amqp-federation-address-ring.png new file mode 100644 index 00000000000..5cdc9649a28 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-address-ring.png differ diff --git a/docs/user-manual/images/amqp-federation-queue-bi-directional.png b/docs/user-manual/images/amqp-federation-queue-bi-directional.png new file mode 100644 index 00000000000..003714454d1 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-queue-bi-directional.png differ diff --git a/docs/user-manual/images/amqp-federation-queue-concept.png b/docs/user-manual/images/amqp-federation-queue-concept.png new file mode 100644 index 00000000000..091c01f2884 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-queue-concept.png differ diff --git a/docs/user-manual/images/amqp-federation-queue-simple-lb.png b/docs/user-manual/images/amqp-federation-queue-simple-lb.png new file mode 100644 index 00000000000..7f199c9f4fa Binary files /dev/null and b/docs/user-manual/images/amqp-federation-queue-simple-lb.png differ diff --git a/docs/user-manual/images/amqp-federation-queue-with-hop.png b/docs/user-manual/images/amqp-federation-queue-with-hop.png new file mode 100644 index 00000000000..ab7f7d76c93 Binary files /dev/null and b/docs/user-manual/images/amqp-federation-queue-with-hop.png differ diff --git a/docs/user-manual/index.adoc b/docs/user-manual/index.adoc index c7f1605d393..dee1ebd7735 100644 --- a/docs/user-manual/index.adoc +++ b/docs/user-manual/index.adoc @@ -72,8 +72,8 @@ image::images/activemq-logo.png[align="center"] ** xref:amqp-broker-connections.adoc#amqp-server-connections[Server Connections] ** xref:amqp-broker-connections.adoc#mirroring[Mirroring] ** xref:amqp-broker-connections.adoc#federation[AMQP Federation] -*** xref:amqp-broker-connections.adoc#local-and-remote-address-federation[Address Federation] -*** xref:amqp-broker-connections.adoc#local-and-remote-queue-federation[Queue Federation] +*** xref:amqp-broker-connections.adoc#address-federation[Address Federation] +*** xref:amqp-broker-connections.adoc#queue-federation[Queue Federation] ** xref:amqp-broker-connections.adoc#bridges[AMQP Bridges] *** xref:amqp-broker-connections.adoc#bridging-from-remote-addresses-and-queues[Bridging From a remote] *** xref:amqp-broker-connections.adoc#bridging-to-remote-addresses-and-queues[Bridging To a remote]