Lone Android dev 'almost brought down T-Mobile' T-Mobile USA told the FCC that having an open network caused it severe overloading and network degradation, thanks to one incompetent Android developer. The developer in question created an Instant Messaging application that drove traffic up by 1,200 per cent in one instance. As it became more popular, T-Mobile was obliged to …
"it seems likely the application was repeatedly setting up and tearing down IP connections, using each one for a very small amount of data and thus generating more signalling load than the base stations could manage."
Isn't that EXACTLY the problem that iPhones had that caused the problems on AT&T and O2?
This has nothing to do with net neutrality. Looking at wired infra that would be like saying that net neutrality forbids NOCs to counteract DDoS attacks. Net neutrality is there to protect users, if a dev writes an app that kills the network (on purpose or by accident) net neutrality should not forbid the NOC from taking action.
Of course this is just my 2cts, the laws over in Usonia are probably being written to exactly the effect that T-Mobile is afraid of.
One of the (many) complaints about Lemon 4 was dropping calls, separate and apart from it's Grip of Death problems.
Several test facilities have attributed dropped calls to this technical slight of hand fom Apple in a feeble attempt to claim it had lengthened battery life.
This is similar to saying a 40 watt light powered from a car battery lasts as long as an LED array from the same car battery.
But, hey, Apple claims to have some of the best RF guys in the world ... one thing for sure they introduced unique sets of problems!
"Looking at wired infra that would be like saying that net neutrality forbids NOCs to counteract DDoS attacks"
the way the fanatic fringe - ie the part that gets the publicity - of the net neutrality brigade are phrasing things, that's exactly what it would mean.
I agree the circumstances in which ISPs can discriminate traffic need to be locked down, but the tubthumpers (again, only the very fringe, but the people who get quotes) are going on about total neutrality. I want ISPs to be able to:
- deal with (D)DOSs and faulty apps like in the article
- prioritise based on tcp/ip headers (icmp vs udp vs tcp, for example)
- to further be able to prioritse based on traffic type (real-time VOIP and gaming vs web pages vs bittorrent/ftp/binary newsgroups)
...all of which make sense, but we're in danger of throwing out the quality of service baby with the sneaky underhand revenue generation bathwater
An incompetent developer is No excuse for being exempt from net neutrality.
It seams that no one at T-Mobile have ever seen: The Fallacies of Distributed Computing.
So here they are:
1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.
What I've seen of new programmers in the last 10 or so years is that many have no clue how a computer or network really works, nor do they care. Far too many recently "educated" programmers were brought up developing on PCs where their actions affect no one but themselves and resources are far beyond what they need for their application. Sloppy, inefficient coding is all they know. Then they get in the real world and suddenly efficiency and size matter and resources aren't free.
They should all be forced to develop on 6809 with 16kb of RAM until they learn to stop being stupid.
Being a dev - I can assure you that we try to learn as much as we can about the environment into which we deploy. But in the present day and age, that environment is so large and diverse that things just happen unintentionally. That is not incompetence. Lack of experience maybe - but you only get experience from making mistakes. Don't confuse the two.
I'd estimate that the developer was partially to blame - repeated setting up / tearing down of connections is surely going to be inefficient. But what sort of OS is it that, when faced with critical network constraints like this, does nothing to stop an app from just going ahead and doing it?
Surely then the answer is not just to fix the app. The OS is presenting an opportunity for a denial of service attack. Write a popular app, give it a command channel, then blackmail the network operators by threatening to activate the nasty payload on millions of phones that does what this app accidently did.
If freely sending and receiving packets is that damaging to their network, why hasn't the network operator provided a custom tcp/ip stack that queues pending packets, and bursts them on a schedule that isn't damaging to their network? Or uses some sort of heuristic that starts to queue and burst if it detects overloading, but allows quasi-real-time operation otherwise. Unless T-mobile is claiming this 'incompetent' Android developer's app hacked the hardware to send stuff over the antenna directly, presumably he's just using the standard sockets API provided. Seems to me traffic management should be done a layer or two down if it requires special attention (as evidently it does).
Personally I doubt T-mobile or any other operator working on their own could fix a problem in relation to technology they implement but didn't themselves develop. This seems more like a standards issue to me. Perhaps the standards developers didn't foresee this potential mobile application usage pattern. Given the difficulty and cost of transitioning from 2-3-4G and the time taken, or of industry-wide retrofitting of existing platforms, maybe this is one that will have to be patched in less optimal but more feasible places, i.e. within the applications which exercise this platform bug.
This one isn't an issue of network neutrality as far as I am concerned, more a question of pragmatism and muddling through.
Is another reason why Devs have to be cautious about making open internet connections and leaving them open.
I am sure that there was improvements that could be made in the development but this sounds like it is down to a number of things that eventually meant that a bit of poor code could cause such a bad effect.
"This kind of thing is increasingly a problem as smartphones try to push battery life by shutting down data connections as quickly as possible, while the applications treat data connectivity as if it had no overhead - and thus make no attempt to aggregate traffic. Desktop applications can treat connectivity like that, resulting in a critical difference that T-Mobile reckons warrants an industry-wide exception to net neutrality."
Hello? It's bad practice even in desktop and server applications to blast through zillions of connections, although you're more likely to see the consequences if you're doing server development (which is why various "ooh so shiny we are" desktop developers should at least familiarise themselves with the work of their server-developing counterparts).
And an "industry-wide exception to net neutrality"? Yet another power-grab by corporations when they suddenly have to step up and do some work rather than dangle their customers upside down. You don't see T-Mobile and chums complaining when some unfortunate punter roams onto their network from abroad and starts firing up those data connections.
Dont promise users all-you-can-eat bandwidth if you can't provide it.
And the solution isn't to create a different and artificial legal framework, that is the domain of the MPAA/RIAA etc. The solution is better development. Provide devkits that encourage/enforce efficient use of network resources.
But don't promise (and sell) something you can't provide, then try to change the law so that you don't have to.
They said what? A demonstrably broken application was causing what amounted to a DDOS on the T-Mobile network, in circumstances which were not really normal intended operation? And T-Mobile see those circumstances as an excuse to support their case for routinely abandoning net neutrality in normal intended operation?
Sorry T-Mobile but you can't be much more transparently wrong than that.
"it seems likely the application was repeatedly setting up and tearing down IP connections, using each one for a very small amount of data and thus generating more signalling load than the base stations could manage."
1. There is no such thing as an IP connection
2. The network shouldn't see TCP connection setup and teardown, that's for the endpoints to worry about.
It looks like T-Mobile have a horrible layering problem. It's probably NAT.
NAT is bad.
This is nothing to do with network neutrality, it's to do with network stupidity.
You need to learn a little about the difference between connection-oriented networks (GSM, UMTS, mobiles in general) and connectionless networks (Ethernet being the classic modern example).
I know a little, just enough to recognise that this issue has nothing to do with NAT.
Others who know more will have to supply more details of what it *is* to do with - but it *is* to do with setting up and clearing down a connection - but closer to RF/cell level than to IP level.
...is that if they hadn't traced the cause of the problem and got it dealt with, their next move would probably be to identify, limit, and block the disruptive activity.
But if net neutrality was the law, imposed on their contracts with customers, they might not be able legally to do that.
I see their point - but as a mobile network user myself, I don't want to have my own use of online content filtered or throttled.
that what they are referring to here is the additional signalling in transmission network between the radio network controller and the base station. Ife the application was clearing the internet connection and thus its radio resources then immediately setting them back up, requiring additional signalling to re-establish the radio resources. The signalling channel becomes overloaded, then other requests are delay/lost then repeated leading to a cascade effect. Also T-Mobile has no control over the stack in their handsets or network its 3GPP.
If the application is popular enough then it would become as someone said earlier a DDOS.
Still no excuse for T-Mo bashing net neutrality.
Having witnessed the gradual introduction and widespread adoption of mobile phones, through the years, I grew up with the given that mobile phones are not mission critical devices and are prone to failure, be it device failure (battery etc) or reception/network failure.
I suppose this article is a reminder to reflect on this.
That's the first thing.
The second: Interestingly T-mobile is not telling us exactly what happened. One would think that if it was coding that was so blindingly bad and detrimental they would issue some sort of advisory. Maybe they have - can't find it as of now. Because of this, I would wager we cannot really comment on whether the programmer did something to be frowned upon or not.
But I am highly suspicious that T-mobile was offering much more than it could provide and as such is blaming said programmer for making assumptions on its capability to provide.
Clearly something needs to be done, but I am not sure that it isn't on the part of the provider.
If this is a tower connectivity issue, then it looks like a connect/reconnect problem rather than net neutrality. Had the user stayed connected, there wouldn't be a problem, right? This looks like a radio network issue, not an IP issue. How on earth would traffic management prevent a phone from switching its radio on and off?
How about just saying, "You may not rapidly and repeatedly connect and disconnect from our network. Otherwise will we send you a text message telling you that we will block your data connection for 2 minutes." If you want to be friendly, you could add the port numbers of the data being seen around that time, to help the user can identify the application causing the problems.
Or you could hold/cache the radio data at the tower with the expectation that the user might connect to the same tower more than once, even if they have shut down the radio connection.
It would be interesting to know if other network operators have the same problem.
> This looks like a radio network issue, not an IP issue. How on earth would traffic
> management prevent a phone from switching its radio on and off?
As suggested above somewhere, the stack on the phone itself could detect this problem and deal with it, with no basestation intervention. Seems like a good idea to me.
Base stations already have the ability to order handsets to reduce xmit power, to prevent more-local handsets from shouting down less-local handsets. It would presumably not be too difficult to use this feature to cut off offending handsets.
Sorry, What did you say?
You want to bring down T-Mobile's network?
Well...
There's an App For That!
--CEASE AND DESIST ORDER 1ZILLION --
Dear Mr Gangsta
The use of that phrase ("There's an App For That!") is forbidden in the perfect world; but the internet is good enough for me.
Regards, Masterful Leader,
Steve
T-Mobe's shitty network aside, this is what happens if you get a load of neophyte devs with no experience other than a bit of hobbyist Java coding and let them distribute mobile apps to millions of people without adult supervision.
This may be an acceptable price to pay for all the open, I couldn't possibly comment.
Net Neutrality treats traffic the same no matter whether the source is Verizon, Google, etc, and no matter whether they're doing email, movies, etc.
Quotas limit the total bandwidth given to any particular subscriber.
A simple per-subscriber quota would solve the problem T-Mobile encountered. No need to throw out Net Neutrality and start cutting special deals with media companies, just to stop an errant messaging app.
It seems their network is going to be downright saturated not far from now:
http://www.androidauthority.com/index.php/2010/10/19/mytouch-4g-video-shows-off-some-features/
If video conferencing is coming, and if simultaneous data downloads are possible, and if Android root kits allow owners to tether their phones without the disincentivizing pay-more-to-tether options, then won't TM's network see more congestion and shakeups?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
