T-Mobile JavaScript comment stripper breaks websites Attempts by T-Mobile to speed up mobile data connections are breaking websites. The bug intermittently affects mobile device users and PC users using tethered connections. It is caused by "optimisations" to the sites' Javascript code made on the fly, in attempt to optimise the amount of data received. Instead of stripping out …
I think you are confusing opera mini with opera - perhaps because they have similar names or are made by the same company or because they look a bit the same.
Opera mini is fast web-browsing and low data charges.
With that, who cares what browser it is. It's not like any of the other web-browsers are normal either.
They all work pretty much the same on a mobile phone.
Opera Mini is a fast web browser but strictly speaking it is not a browser rather a thin client. A server is browsing on behalf of the client and sending it page data.
Also, there is the rather fundamental point that you are allowing Opera to browse on your behalf. They literally screenscrape everything you browse whether it is over http or https to feed to the client. They know where you're going, how long you stayed there, what links you followed, what sites you have accounts on, how often you check your email, your banks, how often you browse, where your focus falls most on a page. They're in the same business as Google, sucking up personal data to monetize you.
On the flip side, it can be useful if you feel like browsing a site without revealing who you are. To the site, the request comes from opera, not your phone so it is hard to figure out who is making the request.
But not for AJAX heavy sites. The thing tries its best but the reality is it's a thin client serving munged content from a server side renderer. There are limits on how it can behave. From playing around with a few ajax sites it appears to deliver snapshots of content in response to taps which is better than nothing but it wouldn't account for a lot of things sites could do. For example a site which has a ticker, or pops up glasspanes, or otherwise does out of band things isn't going to work too well with Opera Mini.
from the user to do this?
It seems to me that if they want to save data by messing with the content then they should check that the user doesn't mind first?
although to be fair, they shouldn't do it at all in my opinion. If I want to see something it's not up to them to decide if there are bits in there I don't need.
As I said - these are web pages being requested and in this context potential code re-use is not an issue. If users do use source code direct from web pages delivered as content then it is the user's responsibility to ensure that they are complying with any IP provisions and in the absence of an assertion they cannot assume that any code is free from such.
There is no obligation on ISPs to redistribute licence assertions which are included in sections which are, by design, not intended to be delivered as part of the response - indeed the GPL could, if it wanted to, include such a provision but it does not and one can only assume that it does not include such a provision (easily delivereable) because the people who wrote it don't want to include such a provision - it is not as if code optimisers are exactly new.
"... deal in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute ..."
"The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software."
Release the hounds!
... a la Phorm? How dare they tamper with, forge and falsify people's private communications? It doesn't matter whether the motive is advertising or "optimising", no carrier has the right to tamper with the data we ask them to convey any more than they have the right to change the words or voices in our phone calls.
In that case, you will also want them to also transmit all the silence in calls to you too (taking more data and meaning higher call charges for you)?
You will probably want them to turn off all the background noise removers, and echo cancellation that actually make mobile conversations so much better?
Possibly you could also sue them because any freqencies above 4kHz (or 8kHz or higher depending on codec) are also stripped to reduce bandwidth requirements and lower transmission costs.
If you want them to just be a dumb data carrier, I'd suggest used encrypted VPN so that they don't do anything to what you send/receive.
You're going to need more than regular expressions to parse something like javascript properly. Besides which the code will probably be broken up into blocks as it traverses the network and since regexps can't maintain state each they'd work on each block individually. Not much use if your comment or string starts in one block and ends in another.
Writing a simple stream parser on the other hand to do this would be a piece of piss.
Given that you would want to exclude the matches that occur within a string, which will be delimited with either single or double quotes (which JavaScript allows IIRC), possibly at some arbitrary point away from the match in question, the processing cost of the pattern match may be more than you expect. I seem to recall an example given in The Camel Book of a regex using lookaheads which would take longer than the lifetime of the universe to complete.
...it was for a little homebrew interpreter based on Multi User Forth. Stack-based reverse polish is a lot easier to parse than conventional languages, plus your programs end up looking like Yoda made them (whether this is a bonus or not is up to your own preference).
Anyway, the script file was converted into objects via two stages. In the first stage, I'd detect and objectify strings using a variable called in_string, which could be either a single-quote, double-quote or nil. Okay so there was a little more complexity for detecting escaped quote characters within strings, but not all that much. If you're not in a string (if in_string is nil) and you come across a quote character, fill in_string with the character and continue. You're now in a string (and therefore code comment stripping should not apply) until you recieve another matching, non-escaped, quote character. Detecting escapes is really simple stuff (at least compared to parsing and organising multiple nested IF statements, which was eventually solved using a depth counter and object-in-object-in-object recursion), but I wouldn't want to do T-Mobile's job for them for free.
Simple enough algorithm, and could be applied to stripping comments from web pages "properly" in just one pass. That is unless the web page has some really funky way of using quotes within <script> blocks. As for my choice of writing a language interpreter in Ruby, well.. I did it because I could. That's my excuse.
: main
"Hello, World" "#test_channel" say
time "The time is " swap concat "#test_channel" say
2 2 + 4 = IF
"I'm working fine." "#test_channel" say
ELSE
"Something's not quite right." "#test_channel" say
THEN
;
Yeah, I suppose writing a language interpreter in Ruby was a minor crime compared to the language itself.
To those seeing illegal interceptions every time someone processes a file.
Code comments are not content - you didn't request them and they were never intended to be delivered to you as part of the response - they are not a part of the data you asked to be conveyed.
The only problem here is TMobile's inadequate knowledge of how to achieve what they wanted to do.
They should really have left it alone - it is a matter for web site ownesr to optimise. If their pages are slow they will lose hits.
But copyright notices are, especially if the code in question is under the GPL. If I remember rightly, removing the GPL copyright notice from a GPL copyrighted file violates the terms of the GPL licence. Given that someone receiving the page may well look at the source, think "that's a useful piece of code", not see a copyright notice, and appropriate it, I would ahve thought that in this situation, T-Mobile would be liable for the breach of the GPL. (IANAL)
A web site with some GPL'd JS is running as a program, not source code. So copyright messages are meaningless in what it delivers to a browser to execute. I also don't see how it would be the site's fault if something subsequently stripped off the notice if it left the site with the notice intact.
I doubt even the FSF at their most obstinate could reasonably argue that it's a proxy's job to honour comments inside a piece of code or HTML. They might however recommend that if there is a proxy that a user be allowed to choose to circumvent it if they wish.
A more serious question is what liabilities you open yourself up to if you run GPL'd JS on your site. After all you're not "linking" GPL'd code to other content of your site, but it is all being executed in the same JS runtime instance. Therefore you may inadvertently end up tainting your entire site this way. The answer therefore would be to avoid GPL'd JS like the plague. Not that there is much need to use it anyway since the major AJAX libs use another licence, probably for this reason.
@DrXym - May all be well and good arguing that it's not distributing source code except for the fact that when I say "I'd like the source code (as you definitely can with webservices and AGPL)" they also strip it. And when you run "wget http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.js" to fork it and make aweseomeQuery 4.0 instead they strip that too.
So no, it's clearly not harmless from the perspective of the licenses
There is no obligation to deliver human readable JS in your site even if it's under the GPL. At the point that a browser requests and executes it it can be reasonably regarded as a "program" and munged into any form you like, e.g. through a minifier. The GPL requires you to release the "source", which is the input to the program. The site site can provide a link to the source (i.e. the unminified code) somewhere else.
Also, it's not the site's fault if they do honour the GPL and some bloody proxy screws it up. They've met their obligations and that's all that matters.
As I said a more pressing issue is if you minified your JS with the GPL'd code, have you tainted your site. And even if the JS files are separate and running through the same JS instances, does that qualify as a program? What about if the JS comes from one domain and the site is on another. You can see how fraught this is. Which is why AJAX tools generally do not licence with the GPL or dual licence.
"A more serious question is what liabilities you open yourself up to if you run GPL'd JS on your site. After all you're not "linking" GPL'd code to other content of your site, but it is all being executed in the same JS runtime instance."
Yeah cos the GPL is so contagious that it'll spread to the other JS. FFS next you'll be telling me I've tainted every process on my system because they are running on the same core as a GPL'd piece of code.
The answer therefore would be to _READ UP ON LICENSES_ rather than believing that shit that gets spouted (about any license for that matter).
"A web site with some GPL'd JS is running as a program, not source code." Yes, but how is that JS delivered to the browser? Does the server send a binary or the source? Ah yes.... the source. So if someone uses "View Source"?
"I also don't see how it would be the site's fault if something subsequently stripped off the notice if it left the site with the notice intact."
You're quite right, not the sites fault. However, it would be pretty shit if you did "view source" looked at some JS which carried no copyright notice (having been stripped by TMob) and decided "hey I can use it" (pretty stupid choice IMO, but it happens) and then got shafted.
Comments are there for a reason, they may be useless to a lot of users, but less so that a page that doesn't run because someone cocked up removing them
I sent a request to www.foo.bar asking it for the contents of index.html, not to T-Mobile asking them for their summary of it. If that content includes comments, I wanted them; what I wanted is the *exact* response from the www.foo.bar server, and not a paraphrase from someone else who has no right to even be reading my communications. You can't change that just by using the euphemism "processes a file" to describe what's going on; I didn't ASK anyone to apply any process to a file, and particularly not some third-party who has no business doing anything with it whatsoever.
(Half the jokes in the TDWTF's front-page articles are embedded in the source as html comments, it would ruin the fun if they were taken out.)
Surely not. Clearing your browser cache once on a decent network again will remove the offending broken js magiacally fixing these permanently broken websites.
So more like T-Mobile has broken websites temporarily until you clear your cache or it expires?
For 99% of the population who've never even heard of a cache, yet alone worked out how to clear it yes. For the 1% of nerds who the first thing they think of is "ooh I'll clear the cache" no.
I wouldn't be surprised if they're modifying the cache expiry dates whilst doing all this too. Lots of evil proxies have tried similar in the past.
If T-Mobile didn't compress images to the point where they look awful on a modern smartphone with a decent display, and didn't try this type of stuff, we might actually see decent latency:
Just now, on T-Mobile, HSDPA full signal strength:
---www.bbc.net.uk ping statistics ----
3 packets transmitted, 3 packets received, 0% packet loss time 2797ms
min = 421.298ms
avg = 489.761ms
max = 539.252 ms
mdev = 49.991 ms
I was under the impression that all the UK mobile ISPs did this (granted T-Mobile screwed it up) - compressing javascript/css down, dropping it into the page inline, compressing images (and adding a JS file to your pages so that you can say 'gimme the original version of this' if you need to) and so on. While it does save some bandwidth, the speed-constrained part of the transfer will be completed faster thus improving the end-user experience.
At least this was the experience i had with my o2 3g cellular modem dongle. I wouldn't have even noticed that they were fudging with things except on a desktop some of the graphics look visibly lower quality, and there's a tooltip on them saying press (key combination) to load the original.
When done correctly, it's a useful feature, especially when you have a particularly weak signal, as long as it's done correctly, and as long as you aren't trying to do code debugging etc.
I used to have this problem on AT&T's network. I'd download an image from my webstite and then diff it with a copy on my local machine and while the were supposed to be the same, and the timestamp in the html was the same, the saved file was always smaller. turns out one of the AT&T's updates to the windows client enabled automatic compression.
I don't see why AT&T wants to buy T-Mobile, they know how to fuck up already.
Not content with attempting to slice your 3 gig cap to 500 megabytes unilaterally with no warning or compensation, and not content to applying such utterly overaggressive JPEG compression to images that they become almost unviewable, they now break page code too?
Anyone got any recommendations regarding phone providers? I'm not renewing my T-Mobile contract.
Don't know about smartphones but for dongles I prefer 3. T-mob mess compress images, Vodafone are expensive. 3 do block access to mail servers unless you use an alternative port, Vodafone don't but seem to block content to innocuous sites too often. Looging into a three account from Linux is easier than the others as it dosn't require user ids and passwords.
They do some attractive PAYG Internet options. Even on this old SIM I can use £5 of my balance and get 2GB for 30 days. Newer SIMs do a £15 30-day offer where you get a few hundred minutes, a boatload of texts and as much data as you can cram down the line (allegedly).
Only complaint I have is the awful censoring that happens until you either give them credit card details or walk into a shop and loudly proclaim that you want to watch grumble flicks on your phone (or just view EncyclopediaDramatica), while at the same time giving them a bunch of details that they don't need when you're visibly old enough to make your own decisions without having to present a card to anybody. That's another subject though, and a crappy "feature" that all the mobile networks seem to have.
T-Mobile might want to read the Copyright Designs and Patents Act or the Computer Misuse Act more carefully.
More carefully, than, for example Phorm did.
They have no right to make adaptations of a creative work without licence.
Under the Computer Misuse Act they are interfering with the operation of a computer without authorisation.
Personally, I'd recommend they stop what they are doing immediately, and stop being so stupid.
Even the most conscientious developer is unlikely to have more than 10% comment/code ratio.
Javascript files of significant size are normally processed through a code compressor before they are deployed to public web sites (this is certainly true of the big libraries like JQuery and ExtJS), so there won't be any comments left in them.**
So T-Mobile is removing maybe 5kb from a typical site. Big deal!
**The authors of the JS compression tools seem to have managed to do what T-Mobile has failed at.
You can get around t-mobiles javascript issues using an add-on for firefox called Modify Headers, download and install the add-on and then add the following two options:
Action Name Value
Add Pragma no-cache
Add Cache-Control no-cache
Then select "Enable All"
This will also stop images from being compressed, I had to do this on my Mac as the Mac version of the Web'n'Walk software that comes with my USB stick doesn't include a facility to disable the compression, unlike the windows version, which does.
All of the mobile networks in the UK implement this compression to varying levels on mobile networks, but T-mobiles does seem most aggressive.
That explains the horrible image quality I get on my smartphone. I can understand doing it, but why not allow the option to 'view original' if the user wants to, because maybe it's important?
And I guess that explains why some websites have been acting crankily on my device, not responding to form events, etc.
I hope they switch to a very simple and fast stream parser, which is what should be being used for this type of modification.
I visit broken sites all the time. Either because of ad blockers or script blockers. Or course any site that requires Apple Quicktime will broke because I refuse to have Quicktime installed. So why should I expect full internet from a tethered EDGE mobile connection. I might complain with a 4G connection though.
I have in the past used my t-mobile phone as a tethered modem. I find it slow and unreliable, that is until you go to an encrypted site. As soon as you hit one of those and you're bypassing the "transparent" proxy, everything speeds up.
If I have to use one I just open a Citrix session to work and browse out from there.
I bumped into a similar issue with Vodafone a few years ago. There were two solutions suggested by vodafone (who eventually admitted they "optimised" HTTP):
1) Tell them to leave your site alone via a well hidden web page.
2) Include a custom HTTP header field that told their stripper to s*d off
Option 1 didn't do anything (well not in the few days I tried it for). Option 2 however did work fine.
So you can have an 'optimising' APN if you want these sort of shenanigans, or a vanilla APN normally. They also squash image files and the like.
Most 3G smartphone and tether users are well advised to take the latter option. This sort of thing has been happening for at least 10 years, BTW.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
