How to eliminate one very common source of lag Yesterday, I didn’t really have time to go over the network improvement that I posted, and explain it in more detail. Here’s the dope, starting with a little background.

Most data connections across the Internet are asymmetrical. That is, one side or the other is sending more data than the other side. If data is coming into your machine, your machine is sending back packets saying “Yep, got that. Got that too…” and so on, letting the sender know that you’re keeping up and nothing got mangled or lost in transit.

Now, you know what it is like to be in a queue. You stand around waiting to get to the front and get served. It can take an awfully long time sometimes. Even if the server is very fast, and it only takes a very short time for each person, if you’re 30 or 40 people back in the queue, you know you’ll be standing there for a while, right?

So, your cable-modem (or DSL modem) also has a queue, only it is for data packets that your machine is sending for all the Internet data connections that it’s chattering away on at any given moment. It’s not uncommon for a single machine to maintain many dozens of connections, and to send dozens of packets per second.

The modem, however, has a speed limit. Usually the connection between your computer and the modem is very fast. 100Mbps isn’t uncommon. At that speed, your computer can shovel ten megabytes of data into the modem every second. The modem, however, is like one of those reduced-speed-limits at road construction zones. If your upload-speed is 256Kbps, then it can only push 25 kilobytes per second up the wire to your ISP.

If you’re sending data to the modem faster than that, the packets get put into a queue, and they wait. If the queue fills up, the modem will throw the excess away, and your machine will resend them later on. Nevertheless, most modern modems have some very large queues, and even though it takes a tiny slice of time to send one packet of data, there could be a hundred or a thousand packets waiting to be sent.

If there’s a hundred packets already in the queue, then any data you send now sits in the queue until its turn comes and only then will it be sent up the wire to your ISP and make it out onto the rest of the Internet. If it takes 10 milliseconds to send a packet, and the queue is 100 packets long, that’s a whole second. If the queue is 1000 packets long, it’s more like ten seconds before the packet even has a chance to leave your house, let alone travel across the country.

When you’re gaming online, or using a virtual environment like Second Life, that’s one of the most direct causes of lag.

That’s mushiness you can feel. That’s rubber-banding and delays. That’s some connections hogging your Internet link while others starve. That’s downloads and uploads taking longer than they rightly should, and remote interactivity just generally turning to jello.

How do you get rid of that? The answer is simple enough: Don’t fill up the queue on the modem.

How do you do that? Well, that’s a little trickier.

Many of you probably have your Windows PC connected more or less directly to your modem. Maybe you have an Ethernet hub or switch in there, especially if the rest of the family use the connection too. I’m afraid I can’t help you out, if that’s the case. I’m not sure Windows can even handle this sort of thing.

Personally, I think it is unwise to connect a PC running Windows to the public Internet like that. Some of you think the same, and have a Linux box acting as a router and firewall between your desktop system and the modem. Or you’re running a Linux desktop system connected directly to the modem. In either one of those scenarios, you’re in luck.

Mac OSX users, I’m not sure what to tell you. If you’ve got a Linux firewall/router between you and the modem, this will work for you. Maybe it’s possible to do this in OSX, but I’ve no idea.

Okay, now the gritty technical bits…

What you need:

  • A Linux network interface that is connected directly to your modem, whether it’s a desktop machine, or a router or whatever.
  • A reasonably modern Linux kernel (2.2 or later should suffice). It should be built with the various packet schedulers. find /lib/modules | fgrep sch_tbf See any hits under your current kernel version? If you do, you’re good to go.
  • iproute (grab it as a package, or build it yourself, whatever you prefer to do)

Got everything? Right. Here we go.

Now, every network interface has a ‘queuing discipline’ (or qdisc). That controls how packets that are to be sent through the interface are handled, and in what order.

The default is pfifo_fast. Unless you’ve done anything wondrous and magical, it will be behaving as a big dumb buffer 1000 packets long:

# ip link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

The first packet to be sent to the interface will be the first to go on the wire. Followed by the second, and so on. pfifo_fast can be made to do tricks with packet-priorities and reordering, but like I said, unless you set that up, it probably isn’t.

Now, how do we stop the data we’re sending through this interface from exceeding the modem’s transmission speed, and causing packets to queue in the modem’s copious internal buffer?

If only there was some way we could tell our network interface not to send any faster than the modem can transmit data to the ISP.

… Wait! We can do that!

What we’re going to do is change the queuing discipline on our network interface and restrict how fast it is willing to send packets.

Now, you’re immediately thinking that if we throttle that rate back, packets will pile up on the network interface instead of the modem, and we’re back to where we started, right?

Not quite. We’re going to take care of that too.

Now, the first thing to know is your transmission speed. It’s probably 128K or 256K or 512K or something like that. Those numbers are in kilobits per second (Kbps). In practice, the modem might not actually be able to send at 100% of the nominal speed, though, so let’s experiment a little.

Let’s assume your connection is 256Kbps. Now, we’ll start by assuming your modem can handle about 85% of its rated speed. That’s 220Kbps. Assuming that eth0 is the network interface that we’re talking about, this is the command:

tc qdisc add dev eth0 root tbf rate 220kbit latency 50ms burst 1540

You’ll need to be root (or use sudo) to do this.

It takes immediate effect, and you should be able to notice the improvement almost immediately, especially when many data connections are hogging your internet link at once.

You might need to tweak the 220 figure up or down slightly to get optimal performance. If your modem has a different transmission speed, start with 85% of that. My 512Kbps connection works fine with a rate of 500Kbit.

Before we go any further, I’ll tell you how to get rid of it – should you feel the need.

Either:

  • Reboot, or…
  • tc qdisc del dev eth0 root will set the qdisc back to the default parameters

The change won’t persist unless you keep adding it in manually, or wire it into a script that runs when the interface comes up. If you’ve gotten this far, you’re probably okay with that.

Now, that fixes the problem of packets accumulated in the modem’s buffers, but how does it stop packets accumulating in the network interface’s buffers on your linux box?

That’s the “latency 50ms” parameter. The tbf (token bucket filter) qdisc will only send packets up to the specified rate (with a little extra allowed for by the ‘burst 1540’ parameter). Excess packets will be queued – but not for very long. Since the qdisc knows exactly the rate of the interface, and knows how many bytes it has queued, it can calculate how long the last packet in the queue must wait before it gets transmitted.

“latency 50ms” tells tbf to simply drop any packet that would have to wait on the queue for more than 50 milliseconds.

I know, right? Actually discarding packets seems like a bad thing.

However, it actually isn’t in this case.

Modern Internet communications protocols (and by ‘modern’, I mean about 1980 onwards – that’s a good three decades now), are designed to compensate for dropped packets – and indeed, it’s dropped packets that alert the protocols to network congestion and trigger behaviours that result in better sharing and overall throughput.

Plus, it’s heaps better dropping packets on a local 50ms delay than having them dropped off of the tail end of a many-seconds-long buffer in your modem, or on some router you’ve never heard of on the other side of the world, believe me.

You’ll find that 50ms latency value improves the performance of your ssh and telnet connections, and quite a bit of other interactive traffic.

Summary: If you can, try it. You might not believe just how well this works until you do. I’m getting a heap more out of my Internet connection than I was this time two days ago.

Of course, if you’ve got a Linux router at home, you could do all sorts of other gimmicks like creating different priority groups for different kinds of Internet traffic, ensuring that what should be fast stays fast, even when you’re downloading episodes of Jersey Shore in the background.

Though if you are downloading and watching Jersey Shore, there’s probably no hope for you.

Tags: , , , , , , , , , , ,

Categories: How To, Internet, Technology.



Got a news tip or a press-release? Send it to [email protected].
Read previous post:
Close