Aggligator

One connection, many links

How the links are driven

Splitting one stream over several links is easy to get wrong, and it goes wrong in one particular way: a slow link holds up everything that was sent over the fast ones. Three mechanisms keep that from happening, and none of them needs to measure how much bandwidth a link has.

The split is discovered, not calculated

Nothing decides what share each link should carry. The next packet simply goes to a link that has room for it. A fast link finishes sooner, has room again sooner and ends up carrying more; a slow one carries less, without anyone working out how much less. No estimate of a link's throughput takes part in the decision, so there is no estimate to be wrong when a link changes speed — which, on Wi-Fi and mobile networks, is constantly.

Waiting data is measured, and acted on

Data that arrives out of order has to wait for the gap in front of it, and that waiting data is what a bad link really costs you. The sender counts it directly: every packet the far end has received but cannot pass on yet. When that backlog passes a third of the buffer, the link holding the oldest missing packet has its allowance trimmed; past three quarters, the allowance is halved. Allowances grow only while data is waiting and every link is already full, so a link is granted more only when it has proven it is the bottleneck.

If a link fails to acknowledge what it was sent, everything still in flight on it is put back in order and sent again over a different link — never over the one that just failed it.

A link too slow to help is set aside

Bandwidth cannot buy back delay: past a point, a slow link costs more in waiting than it contributes in throughput. Each link's round-trip time is therefore measured against the fastest link of the connection.

The factors follow from link_max_ping_spread, which defaults to 5, and are relative to the fastest link of the moment, so a connection on which everything is slow keeps all of its links. Bonding a fast line with a much slower one on purpose — fibre and a mobile connection, say — means raising that factor, or setting an absolute max_ping instead.

Every knob and its default →

Usage

The client connects to a host and gets one stream. How many links that stream is made of is decided by how many ways there are to reach the host: every local interface is tried against every address the name resolves to, and links appear and disappear from then on without anybody asking.

use aggligator_transport_tcp::simple::tcp_connect;
use tokio::io::AsyncWriteExt;

// Every route to `server` becomes a link.
let mut stream = tcp_connect(["server"], 5900).await?;

// From here on it is just a stream.
stream.write_all(b"hello").await?;

use aggligator_transport_tcp::simple::tcp_server;

tcp_server(
    SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 5900),
    |mut stream| async move {
        // One task per connection, links and all.
    },
).await?;

Links do not have to be of one kind. A Connector takes as many transports as you have and puts all of their links into the same connection, so a device can be reached over the network and over its cable at once.

let mut connector = Connector::new();
connector.add(TcpConnector::new(["server"], 5900).await?);
connector.add(UsbConnector::new(|dev, _| dev.vendor_id() == 0x1209)?);

let stream = connector.channel().unwrap().await?.into_stream();

How the tools below do it →

Read on