
Kittens?
Pirate lord or prodigious porn pumper – how much did YOU download this month? The majority of us will be on unlimited connections and such a question won't matter outside of mobile data. But if you consider yourself a bit of a download demon, note that Australia's Openreach cognate nbn™ has drilled down into the nation's …
This post has been deleted by its author
You don't have an alternative ISP, do you?
Comcast keep threatening caps here, but we have FIOS fiber here, and the minute they implement caps, I'm calling Verizon and moving the RJ45 to the ONT.
Perhaps that's why they keep upping our data rate? We're up to 100Mb/s now. Still too expensive, but FIOS is the same. Neither has implemented caps in Massachusetts yet.
This post has been deleted by its author
It's a little more complicated than that because you need to find out whether nbn's numbers include the encapsulation overhead (and find out what the encapsulation is) and then decide if you want to include the encapsulation bits in your numbers. But thereabouts, yes.
You could easily do that by downloading a Linux ISO over torrent and then forgetting about it and seeding for the rest of the month. In my case, it was Linux Mandrake 7 while I was out at work, way back when my 512KB ADSL link was more than my entire workplace had at its disposal. Came home to find myself a "supernode" or whatever the term is. That's put me off peer-to-peer and torrents ever since.
I started downloading an Astronomical database and it was nearly a day before I realised the indicator was still stuck at 0% and it dawned I didnt want it after all.
I have a friend who downloads and uploads Tb on his 1G connection but he is a real astrophysicist even it he is retired.
The nbn report says "data consumed", but this report, and almost all the commentards are turning that in to "data downloaded". It's fare more likely that he's downloading lots and uploading even more. So he's using roughly 894 GB a day - lets say he's an avid scene torrenter, downloads 6 TV shows episodes in 1080p each day, at 3GB each, and 3 movies (each 12GB) that's just a share ratio of 16:1, that's "only" 1620GB downloaded per month.
That would depend on the quality they would be downloading.
Those movies, if remuxes instead would move up to ~50GB a piece, 4K instead of 1080p would then push it between 50 and 100GB each.
TV shows, if a 'collector' / hoarder (blue-ray remux) would come in at 6 - 10GB a piece again. so a season could set you back 200GB.
Plus only 6 TV episodes, 1.5 tv series a month? would be more like 4-5 tv series at 4 episodes a month, so 20 episodes.
My BitTorrent client has been seeding several flavors and versions of Linux ISOs, and Kiwix Wikipedia full snapshots in two languages. There's 95 MB/sec (big B) of available bandwidth yet the stats say only 24 TB uploaded after several months of running.
Hitting 26TB in a month would require far more rare and interesting torrents.
You could easily do that by downloading a Linux ISO over torrent and then forgetting about it and seeding for the rest of the month.
I don't believe so. The story was about the amount of data downloaded and not the amount uploaded. You would still generate quite a bit of traffic, but in the opposite direction of what's indicated here.
This post has been deleted by its author
This post has been deleted by its author
3's 'unlimited' 4G broadband does say if you hit 1TB in a month they'll check for 'commercial use' so it's not such a crazy suggestion. Well, if you ignore the extremely optimistic description of the service as broadband! There are believable claims of being checked (and cleared) for hitting up to 1.3TB on it.
If Virgin had put a similar clause in their cable contracts there would have been fun&games every month in my last gig. Wish they had, might have noticed their commercial service is cheaper and faster!
This is the biggest load of bullsh*t I've heard from an Network supplier, since ISP's branded Firefox public enemy no1 for DNS over HTTPs...
If its being routed across their network, they have a substantial amount of information, including
1) What URLs are being requested (even with HTTPS requests, you can get the domain from the packet)
2) What DNS requests are being made
3) Source address (or at least the last NAT'ing) and destination
The only time they would not have any information, is if the end-user was tunnelling their traffic via a VPN.
Could he be hosting a Tor Node ?
Alternatively a CIA Safe House used for ensuring the safety of Australia (cough) ?
A Pablo Escobarian supercomputer used to pull all the details of his coca-dollar banking transactions with the various governments mafias of the world ?
What's is very surprising though is the quantity of hard disk space that one would require should this been a download session. Not everyone has 27 TB of space "available" on a home computer....
I do. I have a nice little four bay NAS with four 8 TB disks. It backs up all the other computers on my home network, including laptops, tablets, and cell phones. I currently have several 2 to 5 TB drives connected to various computers by SATA6, USB3, and Thunderbolt, mostly holding my music collection, assembled over 25 years in many cases by me converting from vinyl or tape or CD to high-quality digital (that is, not low-end MP3) files, and my movie (no, not porn. Mostly not porn.) collection, again dumped from VHS or DVD to high-quality digital files and also assembled over 25 years. Once all the collections are safely on the NAS I’ll get another, to back up the first, and repurpose the local storage, most of which is getting a bit elderly just now. One 3TB drive is starting to have SMART errors and will be removed from service this weekend.
It took me 25 years to assemble enough data to justify this level of storage. He did it in 30 days. No comment.
Suggestion: go to ZFS rather than a hot backup.
Reason: "self-healing" -- 3way comparisons fix intra-file media failures.
Particularly with media files' ultra"tight" compression, losing a single bit intrafile can and will screw the entire file. Exacerbated risk from the physical size of the file. I speak from bitter experience, having had some disks with precious memories unexpectedly x-rayed. You can see where in each photo file the bits were flipped: perfect up to that point, carnage thereafter.
See example of single-bit flips here: http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ .
The article itself is an excellent read. Short, sharp, to the point.
Saying "We cannot see any content access or information sent over the internet." doesn't mean it's true, it could be a cover for "we don't want to say".
Also, slightly unrelated, and I'm not sure about Oz data protection laws, but even if they knew it would surely be illegal to share such information.
It could well be true. As I understand it the NBN is essentially a link between the customer and the ISP. As such it's likely to be based on low level links and virtual circuits using MPLS or similar and the carrier is moving abstract data between the two sites and whether it is IP or something else makes no difference to the carrier. IP headers, packets, segments etc aren't pertinent to the carrier's role, they are simply part of the data to be moved. Any logging, blocking etc based on the content is managed by the ISP at the other end of the link who has knowledge of what structure is imposed on the data sent.
You can't see the URL.
You *may* be able to *infer* the domain, but that's not given (it depends on SNI and other protocols, not to mention a dearth of other secure websites on the same IP).
You *can* see the destination IP, but that's obvious.
You can't see DNS requests if they're using a number of secure DNS services (not least things like DNSCrypt etc.).
You can see source addresses but that's little help at all.
And when they say "we cannot" it doesn't mean "we couldn't". It means "we're not allowed, and our customers would throw a fit and sue us into oblivion if we routinely did that". ISPs and back-end providers aren't even allowed to do as much as any government black-box, for example, themselves.
I can't just dig into my workplaces finance database and change numbers. It's easily *possible*, no doubt, as I have full and total access to the software, administrative rights, and the underlying storage. But I *can't* do it, legally. Nor can the ISP.
And good luck anaylsing terabytes of data for even a single customer like that. Even a torrent client running 24/7 could do what he's done, and you'd have basically no way of knowing what it was that he was torrenting if it was done on a private secure tracker, for example.
Hell, he could be VPNing into his other house or a rented server and using that for everything (or even be misconfigured and accidentally his default route!).
If you're worried about surveillance, use the technology that stops it.
If you're not worried, why do you care?
If they revealed any info on this, you'd be all over them for snooping on the poor guy, even if they dug into it out of curiosity.
Because they say that can't do that (which doesn't mean it's a physical impossibility, it means they can't do that and stay within their contract with the guy / the ISP), you're complaining?
...and yet they know that the biggest use is streaming video. How do they know that if not through some packet / IP analysis?
I get the not allowed to / unable to look at specifics but they obviously have some knowledge of the traffic across their infrastructure unless they're just making best guess statements...
I get the not allowed to / unable to look at specifics but they obviously have some knowledge of the traffic across their infrastructure unless they're just making best guess statements...
If nbn provides an IP (aka Internet) service, it could infer that by looking at utilisation across peering connections to the likes of Google/Netflix/Amazon etc.
Problem with diving deeper is a combination of technical & legal, ie data protection. Doing traffic analysis to log per-session or per-protocol is expensive, and operationally unnecessary.. Which was/is one of the challenges with the EU & UK's data retention rules. ISP's generally don't care which URLs are the most popular, except for peering analysis, and even then it's generally done on destination IP. Looking at DNS requests or URLs is a DPI function that routers generally don't do well.. Plus the cost of logging and analysing that traffic.
Sampling top IP destinations is a fairly normal trick, especially looking at transit connections to see what traffic you could take off paid transit links and shift to 'free' peering.. But care needs to be taken to steer clear of data protection rules.
You assume they are logging every bit of information possible.
Guess what? To an ISP, we aren't that interesting. Sure, they have to log some stuff by law, but to think they want to spy on all their users is tinfoil country.
I run the email-server my brother uses. On more than one occasion he's told me something, and said "But you know that already, it was in my email"
Of course, I wouldn't invade his privacy like that, but my reply? "Mate, you're not that interesting."
P.S. There's nothing technically stopping your neighbour from pointing covert cctv cameras at your windows, doesn't mean he does. Feel free to open those curtains!
Exactly this!
My workplace uses SSL interception (via a man-in-the-middle certificate on all authorised devices). I can, in theory, see absolutely anything that happens through our connection, not to mention have administrator access to everything else, including finance, HR, etc.
Though it's *always* made clear that the connection is monitored to all staff, we don't stop them going to and booking flights or whatever they need to do, so long as it doesn't interfere with their job.
The amount of people who assume I must just be sitting there reading everyone's email, reading every file they make, and looking at every website they log into makes me think three things:
- They're doing something they shouldn't be.
- They themselves would be snooping if *they* could.
- They think I don't have a life.
Honestly, you're barely worth the log-space for the basics, let alone any deep analysis, and even that's basically because it's required in my industry.
Guess what? To an ISP, we aren't that interesting. Sure, they have to log some stuff by law, but to think they want to spy on all their users is tinfoil country.
Both when I did SysOpy-stuff and when I had a repair shop I had people who thought I'd be sifting through all their data. Most quickly groked that you really don't have the time or the inclination to be going through it. A very small few were always concerned we would still snoop (and told variations of "if you're that worried, go elsewhere, I have way too much to do and don't want to waste more time with this talk").
ISPs, even small ones, have so much stuff passing through them that trying to retain anything beyond the most basic of accounting data quickly becomes overwhelming. Just try and keep up with all the material posted here to El Reg and you'll get an idea of the information volumes even small ISPs face. And try and catalogue (even automatically) all of that material so there's some semblance of usefulness....
(that's why I post so much - so theystop monitoring me or go bankrupt with all the disks they'd need! :) )
26.8TB in a month seems steep to stash on consumer hardware
Once upon a time, but not maybe so much any more? A boggo 5 bay nas could easily be filled with 12TB disks now and provide nearly 50TB of storage at raid5. I mean, ok, if he's doing that _every_ month then there's probably something more substantial as his backing store, but I can't believe how much storage it's possible to squeeze into a desktop device these days. Not wanting to come over all "in my day" or anything, but for those of us raised on 400k and 800k floppies that number is just utterly insane.
I miss tape. Punch tape, that is, not magnetic. Clearing everyone's chairs out of the way before walking across the room at a steady pace to load a program. Cheap stickers that came loose leaving an entire coil to unbundle itself in your pocket. Spilling tea on it and having to loop it over the radiator to dry out before seeing if it would still load without snagging. But at least all your old program versions could end up as computer-themed Xmas decorations!
When I was a young lad we carved our computer programs into rocks with other, harder rocks. Couldn't believe my eyes the day Grog walked in with his fancy new shiny rock. Called it "metal". Said it would revolutionize the world. We all threw our computer programs at Grog as we laughed him out of the cave.
...and of course the little punched-out circles gradually accumulating in a plastic bag.
Come the hot weather, empty the results into the fresh air intake of the boss's car. When he starts it and the climate control goes into overdrive, presto! Instant snow globe.[1]
There are other uses[2] but that was funniest.
[1] Handily, when punched tape was around, pollen filters weren't.
[2] Someone I know tipped a whole black dustbin liner full over the balcony in an old skool cinema during a playing of "The Rocky Horror Picture Show". The air didn't clear before the film finished.
Come the hot weather, empty the results into the fresh air intake of the boss's car.
The bin of debris did indeed find many uses. One bagful went into the air vents of a mate's car at his wedding, which was far cheaper than buying the traditional packet of cornflakes.
I get routinely freaked out by the fact that i have a number of 'spare' 128gb USB sticks. The other day I had to carry 8tb of HDD from our small onsite fire safe, to our considerably larger offsite filing cabinet. It made me a little uncomfortable. And don't even get me started on MicroSD cards.
This is just a theory, but, if as a example, Local Cattle Station wanted to run a ISP on a Remote Homestead to allow all the Workers there to use that as their Satelite/Landline/Cable connection point to Internet, on their own VPN Server, it would appear if that Data was going to 1 user, depending on set-up of Network ?
Yes, agreed, that's a fair bit of storage, but it isn't ridiculous - it's less than 2 HDDs worth of stuff these days.
Being the responsible IT people we all are, we know that you should always have an off site backup.
Given the number of fires that Australia has had recently, I'd be tempted to test my restore too.
We have a 500G monthly allowance. We have a couple of holiday cottages and rarely get near using it all unless a games machine appears on the list of devices. I tend to check when teenagers are seen around and very occasionally turn on QOS to limit it a bit. Did have an old lady whose phone seemed to have been infested and had to put that on choke.
Ah, the days when I used to go into university to use their (I think) 1Gbps line back in the late 1990's.
Me, a travelcard, and a bunch of ZIP disks and floppy disks, with a spanned zip archive and an extreme knowledge of PKZIP command line parameters.
Then go home, spent the evening unzipping them all and hoping you didn't get a dodgy disk. Was literally faster than the only dialup available to me at the time.
Best bit was when they kept publishing who was using up their resources and because they didn't have the equipment to monitor such a connection, they basically equated home-folder-storage-size-used with who-is-using-up-our-line. I escaped any scrutiny for years by downloading everything, filling up the home storage, moving it all off onto disks, deleting it, and avoiding the script that ran at a well-known time (it was in the logs that they emailled the major culprits as evidence!) so that my home folder was empty at that time.
I filled a huge thick book full of CD-Rs with all that data (no, not that kind!), still have them to this day.
I used to use my university's labs to download early versions of Linux onto 1.44MB floppies, starting at around 20ish floppies in the early days to 40ish by the time I left university.
Unfortunately, the labs at the time were filled with Macs (the non-compsci 'general' labs that is, the compsci ones had Sun workstations), which could write to a PC-format 1.44 drive, but it'd take many times longer than Mac-formatted floppies. Something like 10-minutes per disk.
So, I'd spend all day downloading then copying to 20-40 floppies, take them home, and often 80% of the way through the installation there'd be a corrupt floppy. Grrrr.
So I'd have to go back, rinse-repeat the entire process - until I had the lightbulb moment of leaving the download on the Mac instead of deleting it, and only having to re-copy the failed disk. But this had its problems too, as it was a local copy on the Mac, so I'd hang about, looking all suspicious, until whoever was using 'my' Mac would leave and rushing over to grab it before anyone else did, hoping n-one had needed the space themselves and deleted it all.
That's a metric fuckton of data so I thought I would see what mine is and so far this month for 10.5 days I have according to my router 439.96gb received so based on that my estimated monthly total would be 1,626gb.
Maybe a new el reg measure is needed? 26.8TB = 1 RB Rogerbyte as it can mean both pirate or porn.
As if any user in Australia ever hits 100MBps for even 5 minutes. Let alone for an entire month! I call fake!/
That's roughly what I was thinking.
Might be the best testimonial for that particular ISP ever - they should be putting it on their advertising flyers.
I can barely get a 15 minute youtube video (SD mind you) to play without stuttering and spinning circles every 8.5 seconds my end.
I thought I was a little excessive, when I went looking for a particular pr0n clip (and found it), but it was contained in an archive of 50GB of material.
So naturally I downloaded the lot. It took a while on a 512Kbps ADSL line..
I ended up deleting most of it later on. 26TB has to be more material than you can humanly view in a month.
Perhaps the user is downloading a load of illegal streaming services, and rebroadcasting them in their house? Otherwise, scientific datasets as others have said.
... or anyone downloading their Steam library on a new PC.
Or just the fecking updates every time I bother to actually play it... "Hmm, havn't tried this in a few weeks, maybe I'll play something different" [5 mins later] "Sick of waiting, I'll go do something else" [few hours later] Oh yeah, that was downloading updates. Well maybe next weekend I'll give it a whirl.."[Week+5 minutes later] "Sick of waiting....."
It doesn't negate the significant amount of data for that month, but many of our (Aussie) 'Unlimited' plans do actually come with a variety of provisos in the ultra fine print such as "Your data speed will [or MAY] drop once you reach bigly amounts [though what an actual 'bigly amount' is isn't actually defined] or if we think you are being naughty [doing commercially stuff] or running a Mega replacement site or etc etc etc"
I seem to recall a bod taking either Telstra or Optus to court over the definition issue because, well, isn't unlimited actually unlimited, m'lud?
Either way, still a bigly amount.