Summary
This article explains how to setup a geographically distributed federation of OpenAMQ brokers and how to define dataflows between individual geographical locations.
Introduction
Imagine you are running a company in London. There's a lot of users and services there connected to OpenAMQ broker. Some time later you decide to expand and establish a branch in Paris.
While there are only few employees/services there, each can connect to the broker in London in a standard manner. However, as your Paris branch grows, the volume of data transferred between the offices grows. The message that's delivered to 100 clients in Paris actually passes the WAN 100 times!
Moreover, you have management and security issues. You have to take care of every computer in Paris and ensure it can create a connection to London-based broker.
At this point, what you want is to install a separate OpenAMQ broker in Paris and pass a subset of messages from London to Paris and vice versa. The users in Paris would then connect to the local broker via LAN facing little administration problems. Moreover, each message would be delivered from London once only and afterwards it'll be redistributed to Paris-based employees/services over the LAN.
This type of messaging architecture is called "broker federation". It is implemented in OpenAMQ using a component named MTA (message transfer agent). The picture below shows the problem (upper part of the picture) and how it can be solved (lower part):
Setup
MTA connections cannot be setup at runtime. You have to define them in your configuration file, meaning that broker will take note of them only on the startup.
To forward messages from London to Paris add the following section to you amq_server.cfg configuration file on the Paris-based box :
<cluster_mta
name = "market-data-lse"
vhost = "/"
host = "openamq-broker-london-001"
login = "peering"
mode = "1"
/>
The name field specifies the name of the exchange whose messages are to be pulled from the federated broker. vhost specified the virtual host to use, host is the name of the federated OpenAMQ broker - as we are editing the configuration file in Paris, we specify the name of the London-based box. login specifies login to use when connecting to the federated broker. mode specifies the way to pass messages between the locations - we'll examine this property in a moment. Note that there can be multiple MTAs defined for a single broker.
This setup is sufficient to pass messages from London to Paris. Note that no specific setup has to be made in London. Actually, London broker doesn't even have to be restarted. Also note that:
- Each message is passed over the WAN once only, even if there are 100 subscribers for it in Paris.
- If there are no subscribers for a message on the remote location (Paris) the message won't be passed over the WAN at all.
So, when there are no users connected to the Paris broker (say at night) the link between London and Paris is completely idle (with the insignificant exception of heartbeats) irrespectively of what amount of messages is being passed in London.
At the moment, the messages are passed in single direction only - from London to Paris. Now, how can we make the message sharing work in both ways?
The obvious solution is to edit the configuration file in London the same way as we did in Paris:
<cluster_mta
name = "market-data-euronext-paris"
vhost = "/"
host = "openamq-broker-paris-001"
login = "peering"
mode = "1"
/>
This is the preferred way to interconnect two brokers. However, there may be several reasons that can make editing London configuration file impossible:
- The London broker may be run in 24/7 mode with no way to restart it for the new configuration file to take effect.
- The London broker may belong to a different company. In that case you have no privileges to configure their broker.
- The broker in London may be AMQP compliant, but not OpenAMQ itself. In that case there is no MTA mechanism available there.
In one of the above is true, you can still pass data from Paris to London, although in a less efficient way. You can add a new MTA to the Paris broker, specifying mode property to have value of 2. This switches on message-push mode rather than message-pull mode that we've been dealing with so far. The configuration file in Paris would thus contain two MTA definitions:
<cluster_mta
name = "market-data-lse"
vhost = "/"
host = "openamq-broker-london-001"
login = "peering"
mode = "1"
/>
<cluster_mta
name = "market-data-euronext-paris"
vhost = "/"
host = "openamq-broker-london-001"
login = "peering"
mode = "2"
/>
The drawback of message-push mode (2) is that it passes *all* the messages to the remote broker. So, even if there is no subscriber for a message in London, the message will be still passed from Paris to London and then dropped by the London broker. Still, if there are multiple consumers for the message in London, it will be passed once only over the WAN.
There are two more transfer modes you can use:
Message-push-else mode (3) passes only those messages that have no consumers on the local broker to the remote one. The idea is that on SOA-based system (have a look at a tutorial here) you would like to request a work to be done by a local service, however, if the service is not available locally, the request will be forwarded to the remote broker. So, for example, if order management system in Paris fails, the traders in Paris would failover to the order management service running in London.
Message-push-and-pull mode (4) passes messages on particular exchange from London to Paris and vice versa. You cannot achieve this behaviour by combining message-push and message-pull MTAs as every message would be circulated endlessly between Paris and London. However, note that this mode is not particularly efficient. For best performance create two exchanges (market-data-lse and market-data-euronext-paris) instead of single one (market-data) and pass messages only in appropriate directions.
Extending the federation
As your company grows you establish branches in Berlin and Hong-Kong and connect them to the London in the same way as Paris broker is. At that moment you decide to expand to mainland China and establish branch offices in Beijing and Shanghai. It happens that the connection from London is slow, low-bandwidth or too expensive. Therefore you would like brokers in your mainland offices to speak to the broker on your Hong-Kong site rather than directly to the broker in London. Have a look at the diagram bellow. Note that messages are delivered in a slower manner - there are 2 network hops involved to pass a message from London to Shanghai - however, the lower intercontinental bandwidth usage etc. may still pay off.
You can define various advanced setups for your company. Recall that setup of MTA is done on exchange level. Therefore, if your traffic is reasonably split across several exchanges, you can define message distribution network for each exchange separately.
Say, there is a trading application that need lowest latency possible: You should establish direct links between individual locations to eliminate excessive network hops.
However, there is also a post-trade reporting service that doesn't need to be that quick: You can pass the messages via a hierarchical tree-like broker federation.
The both distribution schemes can be used at the same time using same broker instances provided that message traffic is clearly separated into "trading" and "reporting" exchanges. This kind of setup can be seen on the diagram bellow. "Trading" MTA connections are represented by full lines, "reporting" connections are represented by dashed lines (Hong-Kong and Paris report directly to London, Berlin reports indirectly via Paris):
Comments
Author
- Martin Sustrik <moc.qmtsaf|kirtsus#moc.qmtsaf|kirtsus>