Under ideal circumstances the Internet’s TCP protocol finds the best effective throughput between the two connection endpoints.
Unfortunately, it turns out that numerous ‘improvements’ in routers have actually made conditions significantly less than ideal, leading to a situation where TCP isn’t able to achieve optimal throughput and – under real-world conditions – winds up operating in a very substandard sort of way.
The problem is – essentially – the packet buffering on routers.
Some buffering is absolutely necessary. The router may not be able to handle a packet at the instant it comes in, or there may be no time on the wire at the instant a packet is to be transmitted. Without at least a little buffering, packets would be discarded almost constantly, and that’s bad.
No, the problem is too much buffering.
TCP starts out relatively modestly, and based on the time it takes to move packets from end-to-end, steadily achieves an optimal rate, adjusting that rate to avoid congestion.
Unfortunately, large packet buffers mean that when congestion actually starts to occur somewhere along the route, TCP is prevented from finding out that this is occuring. It keeps shovelling packets onto the wire, and those start to accumulate in buffers as the load rises on intermediate links. Neither end of the TCP connection is immediately aware that this is going on; that packets are being held in-transit.
TCP continues to shovel packets out, increasing the latency at key routers along the way, until suddenly TCP realises that things have slowed way the heck down, and compensates by backing off to a lower rate. Packets in buffers are slowly cleared, and the latency from end-to-end is returned to normal. The TCP endpoints however, remain unaware of this for some time, but when they do, it starts to ramp up again.
This results in overall throughput on the TCP connection rising up towards a peak, reaching it, and then declining savagely to a mere fraction. This cycle then repeats itself, again and again.
With less buffering, various TCP connections would share links along the route in a more equitable fashion, responding more perspicaciously to actual latencies and link utilisations.
Instead, TCP connections end up behaving like feeding pigs at a trough, or the shoppers at the opening of Black Friday sales, because they’re being denied key feedback that they were designed to have. This sort of sawtooth congestion pattern then causes additional problems for other protocols (like UDP).
The ‘improvement’ of large buffers actually makes things worse – a not uncommon result in a wide variety of queuing problems.
What is there to be done? Probably nothing. Large buffers are a default configuration for most routers, and it’s rarely your own router that causes the problem (at least not for you). It’s the ones at your ISP, and at your ISP’s ISP and so on right across the world. Without a substantive push to discover more optimal buffering capacities, and then to implement them across tens of thousands – or hundreds of thousands – of routers, the situation is unlikely to improve at all.
Further reading: Whose house is of glasse, must not throw stones at another.
The adoption of Large Buffers is (IMO) a side-effect of human nature and not technical evaluation. We’re all too familiar with the ‘If a little helps, a lot helps more” mentality, and thus we see Network Managers applying this overly general rule in an effort to ramp up their throughput. Sadly, I don’t think we’ll see a rapid end to the Human Nature factor, but blogs like yours may help educate those that have glanced at their configuration and haven’t really LOOKED at the problem. Thank you for pulling it into the light a bit more.
Tateru, contrarily to what you say here, the problem with excessive buffer size is most of the time on the end equipment (your ADSL MODEM), and using QoS and traffic shaping on your end (so that your ADSL MODEM never receives packets from your computer or receive packets from Internet at faster outbound/inbound rates than the ones it can afford) is the solution to the ‘hiccups’ you describe and also to delayed ACK packets (that, without proper QoS, get queued together with normal traffic in the huge ADSL MODEM queue, thus lagging further traffic for protocols such as telnet, instant messaging, etc).
However, while it is theoretically easy to do (or at least under Linux), it can in reality prove harder with modern “Internet boxes” that many ISPs provide as a manner of an ADSL MODEM and that are actually routers that often already use QoS by themselves to prioritize VoIP (phone) and broadcast TV traffic: in this case, creating QoS rules that won’t conflict with the built-in rules implemented (in a total, opaque way for the user) in the “box” can become quite hard…
QoS tagging doesn’t get you very far when your provider strips them straight off of any packets that the modem receives from you, it’s true.