We will find you and we will sue you, Twitter tells 4 mystery alleged data-scrapers Elon Musk's X Corp, lately the parent company of Twitter, is suing four unknown individuals for scraping data from the bird site, claiming that the parties unjustly enriched themselves on the back of Twitter data. X Corp filed the suit [PDF] in Dallas County District Court in Texas, and is seeking more than $1 million in …
So in other words, Twitter has nobody left who knows how to block four IP addresses. And these look they're all from Linode, by the way.
Seriously, any competent network setup would warn you about this happening instantly and optionally block it. But those people all left.
You know Discworld is something I’ve never read. I just googled “Discworld Luggage” in hope of getting your reference, alas I didn’t find it, but I did find this little snippet “The Luggage said nothing, but louder this time.” This line alone has made me wonder what I’m missing.
Audiobook or paperback?
"Audiobook or paperback?"
Entirely up to you. I prefer books, but I drive a lot so have both read the books (NOT while driving!!) and listened to the audiobooks on the road. And there's no real "reading order" either except where some stories may be related or direct sequels, and even then , it usually doesn't matter that much. Personally, I prefer publication order
As for the Luggage, Try here for a more in depth description.
Oh, and there's been a few Discworld live action TV miniseries as well as a couple of hard to find animations.
Elon hasn't added that 127 guy to the case yet 'cos Musk is running scared: that is one l33t haxor and every time they add his IP to a blocklist he retaliates by crashing a part of their system! So far, when they unblock him, he fixes most of the damage but what could he do if they push too hard and annoy him?
To be fair, the lawsuit is about scraping efforts/attempts that already happened. You can't really predict in advance which IP address will be launching a scaping effort unless you want to block the entire Internet. I do have to say though, that would make the job of the firewall admin very easy. It'd be one of the cleanest rules tables anyone has ever seen!
That all said, this is pretty standard Twitler stupidity. He's become increasingly isolated and paranoid, to the point where he needs a bodyguard to follow him to the bathroom at Twitter. Whatever cocktail of drugs he's got himself on, they clearly have not been having the desired effect(s).
The suit also suggests that they scraped millions of pages, and unless they spread those requests out, that probably means thousands per minute. Network analysis can be set to detect thousands of requests from one IP address in a minute and block that address, probably temporarily. It does a pretty good job against heavy scrapers unless they spread out their requests or their IP addresses.
...probably means thousands per minute...
Just how difficult is it to limit one IP address to a maximum amount of requests per minute - enforcing a few seconds of delay between requests would not impact meatsack users who can only type so fast but would seriously limit bots and scrapers.
It is very easy, but that's less often done than detecting the first burst of requests and blocking entirely for a while. The reason is the same one as the reason Twitter's view limits aren't good. If a user scrolls through a set of data which hasn't been entirely retrieved, the frontend may issue a few requests quickly to catch up with the speed of their scrolling, common if they don't care about a lot of the content and are skimming to find something they do like. If they put in a one-second delay between requests, that process could become jumpy or broken, whereas if they check how many requests were issued in the last thirty seconds, they will be more easily able to tell between a user doing something quickly which the app turned into a burst of requests and someone running an automatic script which isn't going to stop any time soon.
Either way, it's the kind of thing that a system which acts at scale either does routinely or has made an explicit decision not to do. It shouldn't be causing Twitter too much difficulty now.
"Just how difficult is it to limit one IP address to a maximum amount of requests per minute - enforcing a few seconds of delay between requests would not impact meatsack users who can only type so fast but would seriously limit bots and scrapers."
That would depend on how many requests one 'interaction' produced. A few seconds between each call/request could be very painful to the meatware user if doing something like clicking on a hashtag caused a whole series of requests to need to be sent.
That all said, this is pretty standard Twitler stupidity. He's become increasingly isolated and paranoid, to the point where he needs a bodyguard to follow him to the bathroom at Twitter. Whatever cocktail of drugs he's got himself on, they clearly have not been having the desired effect(s).
The next Howard Hughes?
"The next Howard Hughes?"
No, the present-day Howard Hughes.
And Cybertruck could be considered to be the equivalent of the 'Spruce Goose', if it weren't for the facts that the 'Spruce Goose' was a practical machine, with a practical end use in mind; and was amenable to mass production.
Also, if the traffic from four IP addresses can put a strain on Twitter servers, then their infrastructure is on even worse state than I thought. Even if these IP addresses are on optic fibers, I'd expect the total traffic they can download to be insignificant for Twitter.
They wouldn't have to bother. They'll have their own servers at the data centres that automatically sift through everything with much more meta data than scraping offers. I presume this mechanism is still in place on the basis that Musk hasn't had any unfortunate "accidents".
Whatever the actual truth of the Hunter Biden laptop "story", the fact that it was censored on the back of a polite but ambiguous request should tell you all you need to know about how in bed these companies are (or, at the very least, were) with the 3 letter agencies. It goes well beyond PRISM.
According to the suit, bots are "flooding Twitter’s sign-up page with automated requests".
That rather suggests that bots that were previously scraping Musk's barrel with relatively little impact are now creating large numbers of accounts in order to get around the recently-imposed reading limits. In other words, Musk's latest genius idea (making Twitter largely inaccessible to its actual users) might simply have made his problem worse.
Perhaps he should be grateful something is still consuming the advertising.
> Perhaps he should be grateful something is still consuming the advertising.
So should he pivot over who he sells the advertising to? Otherwise, it won't remain relevant to the readers of his site.
"Do you suffer from blocked IP packets? Is *your* connection timing out? You need to try Nagle's Ointment!"
"You can't surf without Cerf's Solution, The Best A Bot Can Get".
"Otherwise, it won't remain relevant to the readers of his site"
Since when has targeted advertising actually been relevant to anyone? The best they can offer us ",ooo, this looks a bit like what you've just been looking at, but it isn't really that close" or "people have been buying this too" or "people in<insert your general location> have been going crazy for this"
And there are still folk who wonder why people use Adblock etc.
Since when has targeted advertising actually been relevant to anyone?
Dislike or block enough ads on Twitter and the remaining ads you get vary between ridiculous and hilarious. Lately I've been getting ads for small pet shops across the southern US states and engineering industries in India. I live in the UK.
"nobody realised he meant he was planning to turn a profit simply by suing as many people as possible and cutting staff rather than actually improving the platform or the business."
So, something like those adult entertainment companies that used their videos to bait the trap so they could demand settlements from people that DL'd their videos. The video was only made to be able to generate the infringement notices, not make a profit on its own.
For those four IP addresses respond with tweets replaced by a random sequence of words. Wait a few days and ask Google where some of those random sequences turn up. Try some large language models - perhaps something like ChatGPT will be able to correctly complete the start of the fake tweets.
Looks like more lawyers work for Twitter than programmers. If only they had someone working there able to pay bills and sort out redundancy payments.
I see another person who didn't read the terms of service. Does this part mean anything to you:
8.2 You retain all your ownership, copyright and other interests and rights in your comments but by posting any comments on our Website you grant us a non-exclusive irrevocable and royalty free worldwide licence to use, modify, alter, edit copy, reproduce, display, make compilations of and distribute such comments throughout our Website.
Twitter will have something similar. Yes, the effects of "I have a license to use the thing you wrote because you granted it to me" and "I own the thing you wrote" are similar, but the concepts are not at all the same. The former applies to most social media.
"8.2 You retain all your ownership, copyright and other interests and rights in your comments but by posting any comments on our Website you grant us a non-exclusive irrevocable and royalty free worldwide licence to use, modify, alter, edit copy, reproduce, display, make compilations of and distribute such comments throughout our Website."
This sort of clause means that once you've uploaded the comments, your ability to make any money from them is severely diminished. Not my comments, obviously, as they're worthless. But somebody like InstaPintaTwitFace is in a far better position to monetize the content users post to their service. By granting them an unrestricted license in perpetuity, you give away far more than you get in return.
From: https://twitter.com/en/tos
Your Rights and Grant of Rights in the ContentYou retain your rights to any Content you submit, post or display on or through the Services. What’s yours is yours — you own your Content (and your incorporated audio, photos and videos are considered part of the Content).
But hey, let's not let the truth get in the way of a paranoid tinfoil hatter.
More generally, do you really think if I wrote and posted a brilliant short story on here, or some great mathematical proof I'd discovered, "The Register" could claim it as their own and lock me out of any decision to publish commercially?
Why did you only cut and paste the first sentence? Here's the paragraph directly following the sentence you posted from https://twitter.com/en/tos :-
By submitting, posting or displaying Content on or through the Services, you grant us a worldwide, non-exclusive, royalty-free license (with the right to sublicense) to use, copy, reproduce, process, adapt, modify, publish, transmit, display and distribute such Content in any and all media or distribution methods now known or later developed (for clarity, these rights include, for example, curating, transforming, and translating). This license authorizes us to make your Content available to the rest of the world and to let others do the same. You agree that this license includes the right for us to provide, promote, and improve the Services and to make Content submitted to or through the Services available to other companies, organizations or individuals for the syndication, broadcast, distribution, Retweet, promotion or publication of such Content on other media and services, subject to our terms and conditions for such Content use. Such additional uses by us, or other companies, organizations or individuals, is made with no compensation paid to you with respect to the Content that you submit, post, transmit or otherwise make available through the Services as the use of the Services by you is hereby agreed as being sufficient compensation for the Content and grant of rights herein.
"By submitting, posting or displaying Content on or through the Services, you grant us "
Phones Sheridan, about the only thing you left out is "without credit". Most services have that in their EULA so they don't have any obligation to credit you for the work. Another clause that will be included is that you indemnify them against legal action for all of what you post. Even if you post your own original work and they get a letter from an attorney with claims it was pirated, they could settle with a payment and require you to pay them back for the expense without any other notification or participation. That's not how it's handled, but that indemnification clause puts you on the hook if they decide to do it that way. It's a trap my attorney has expensively explained to me. I didn't have it happen, it's just that talking to a blood sucking lawyer isn't cheap but in this case, learning how to break down a contract's terms was a good investment.
This post has been deleted by its author
"More generally, do you really think if I wrote and posted a brilliant short story on here, or some great mathematical proof I'd discovered, "The Register" could claim it as their own and lock me out of any decision to publish commercially?"
They wouldn't be claiming it as their own original creation and can't lock you out of making money from it, but they don't have to pay you and you've granted them permission to sell it on for as much as they like with no credit or royalties, forever. Welcome to Social Media.
Musk may be inadvertently helping us all here.
If this "unjust enrichment" idea is allowed to stand in this case, we can make exactly the same argument for companies collecting and using data about us.
We'll have to show that there's some added cost or burden from their data collection. Not sure if the use of cookies could meet that, but requiring logins (as, to take an example purely at random, Twitter is currently trying to do) certainly would.
Reading the linked court PDF, I suspect the “unjust enrichment” verdict is itself unsafe.
It would seem a vendor can supply a DB based application and charge extra for a data export tool, if the customer doesn’t purchase the data export tool and simply uses a little nous and DB vendor supplied data extraction tools to extract their data then this qualifies as “unjust enrichment”…
host twitter.com
twitter.com has address 104.244.42.193
[+ some stuff about mx records which I don't care about]
whois 104.244.42.193
NetRange: 104.244.40.0 - 104.244.47.255
CIDR: 104.244.40.0/21
NetName: TWITTER-NETWORK
NetHandle: NET-104-244-40-0-1
Parent: NET104 (NET-104-0-0-0-0)
NetType: Direct Allocation
OriginAS: AS13414
Organization: Twitter Inc. (TWITT)
RegDate: 2014-12-08
Updated: 2020-06-28
Ref: https://rdap.arin.net/registry/ip/104.244.40.0
OrgName: Twitter Inc.
OrgId: TWITT
Address: 1355 Market Street
Address: Suite 900
City: San Francisco
StateProv: CA
PostalCode: 94103
Country: US
RegDate: 2010-03-08
Updated: 2023-04-07
Ref: https://rdap.arin.net/registry/entity/TWITT
What's your point? They have a lot more than one IP address for their website, including all the backend stuff. It is still possible that they have resources in Akamai as part of their diversification efforts to prevent single points of failure. Not guaranteed, as I think they would still have someone who would catch it if this were actually their own, but just demonstrating that their main domain's A record goes to an address they've directly allocated says nothing about where all the rest of their systems are.
Among other things, we know that they have a lot of systems on AWS, since that's one of the suppliers they tried not to pay, so that suggests that some of their internal systems may use AWS IP addresses.
FFS, we know he filed suit in Texas because the state is well known to give favorable rulings to companies with dubious legal claims. And the 5th Circuit has been one of the most conservative and partisan courts in the US for decades. It's long been damn the torpedoes if the law happens to disagree with political ideology.
Second, we also know that the only reason he gives a shit is because he just started his own AI company and plans to use Twitter data to train it. Well, maybe that and the fact that he fired all the people maintaining the servers which have been a slow moving train wreck ever since. He even went so far as to deliberately reduce capacity by shutting down a data center or two.
I genuinely believe the world would be a better place if Twitler suffered a massive heart attack or stroke and either died or was forced to step away from all his business concerns. It's a rare person who meets that criteria, but Twitler has done it many times over.
Sure. Not saying there aren't plenty of other people who deserve it more, but Twitler suffering from Sudden Russian Death Syndrome or his heart finally giving out after years of heavy cocaine abuse would have a pretty positive impact on the world.
Twitter would likely cease to exist. Tesla would be able to finally address the systemic racism and sexual assaults going on in its plants, not to mention quality issues and maybe they'd be able to finally address FSD in a sane way. Neuralink would probably fold, which is likely for the best given some of the things we know about it and the fact that they're way behind pretty much everyone else. Boring might manage to carry on, but who knows or cares since their only reason for being seems to be to toss a spanner in the works of the bidding process for various transit projects. Sure, it may not stop a war, like Putin's death could, but it would improve the lives of many people if Twitler died. Ideally of natural causes, freak accident with his private jet, or maybe in a FSD related traffic accident for a twist of irony.
As the reign of terror continues after the French Revolution[Twitter takeover]...
"They seek him[them] here, they seek him[them] there
Those Frenchies[Twits] seek him[them] everywhere
Is he[Are they] in heaven or is he[are they] in hell?
That[The] damned elusive Pimpernel[data-scrapers]"
With thanks and apologies to Baroness Orczy
I have to wonder what the point of scraping twitter would even be!? You're not going to get cogent or constructive conversation. Or correct spelling, grammar or even full sentences.
Basically creating an LLM from Twitter would get you the equivalent of a 3 year olds speech, full of racist and hate filled vitriol.
Who on earth would want that?
Waaah! Someone profited by using my product and it wasn’t me! Waaaaaaahhhh!
Twitter was built with the idea of profiting from as many people as possible making as many requests (as possible) to display as many cleverly disguised ads (as possible). Whether someone else makes a wad of cash off the back of their service (even at their expense) is irrelevant. They don’t get to dictate the user agent people use to access their stupid website, nor do they own the copyright to user content, so they have no exclusive right to profit from said content.
There is no technical reason Twitter needs to operate on the World Wide Web as a free-for-all. The company could very easily write native apps for each platform with per-version API keys, require users to be 18-or-over, charge a token amount (e.g. 89p/month) on direct debit to ensure people are real, and turn verification into a cheap annual service (e.g. £35/year) where photographic identity documents and the account telephone number is matched to real-world selfies on a regular basis. Perhaps for one neat payment of £3.99/month, all of the above could be lumped together, but with a nice supporter badge and the ability to vote on upcoming changes to the service. Everything would still be somewhat ad-driven, but the company could dial it down a lot and charge a higher CPM as advertisers would know real humans are guaranteed to see their copy without issues.
Unless companies like these start moving away from an outdated, legacy web-based method of serving their customers, they will continue to fail.
On the other hand, he does have a point regarding website data scraping. According to the article, under current US law, if you publish something on the web, it's legal for anyone to then scrape it. What they use it for afterwards might be a legal grey area, possibly even directly illegal. But the general thrust is that your data can be legally taken, even if it impacts on the performance of your servers to the detriment of the intended users. This applies even if the data isn't your, but user generated too. This seems to go against the spirit of the internet and the web in a big way. It's already bitten service providers offering "unlimited use"[*] contracts that then get abused by some users downloading at full speed 24/7 such that they had to add "fair use" clauses when the numbers of people taking the "unlimited" term literally grew too high.. Same could be said of an "All You Can Eat" buffet. Some people will go over the the top, abuse the system, and fill up multiple plates and waste much of the food.
When we have grey areas in the law, there will always be some fuckwit who pushes the boundaries because they need a hard line with a sign saying "Do Not Cross" before they stop, they just can't seem to cope with the "reasonable doubt" of a fuzzy area, almost always to the detriment of the normal, reasonable people who can see that grey area "boundary" and treat it as a sensible warning, keeping clear and only pushing it gently now and then.
Musk may be a twat, but even a stopped clock is correct twice per day.
[*] lets not get into the argument about "unlimited data" and lack of provision for obvious and expected increases in usage by service providers.
I disagree about the scraping permission being contrary to the spirit of the internet. It certainly would be if the precedent stated that blocking scrapers wasn't permitted, but that's not the case. The reason I want that to remain is that allowing sites to make scraping a legal offense makes it too easy to attack people for accessing a resource they made public. If YouTube could go to a court and tell them that, because I used youtube-dl instead of a browser to view a video, I've committed a crime (decided on the basis of "it's a crime if you do what I don't want you to do and I don't even have to decide beforehand what counts and what doesn't"), I think that's a lot more contrary to the spirit of the internet.
Nothing prevents site owners from looking for bots using a lot of different measures and blocking them by technical means. Is it annoying that it may become necessary to do so, yes. However, it's the same point as the all you can download internet plans; the ISPs were not forced to provide those or to sell them at the price they do, and if someone has a dozen terabytes they want to download, they have purchased a service which claims that they can do so. The ISP can add a fair use provision, technical limitations, or change their business model to prevent that person causing problems, but they shouldn't be allowed to decide that they want to make that action retroactively criminal.
Yes, thanks, you raise a good point with your example of using youtube-dl (or the improved fork yt-dlp). Laws tend to be badly written until there's some case law and judges decisions to help define it, and we all know how poorly written many laws are. After all, if they were well written and clear, we'd not not need so many lawyers :-)
"Waaah! Someone profited by using my product and it wasn’t me! Waaaaaaahhhh!"
Which is what the whole Cambridge Analytica kerfuffle was about. FB wasn't paid for that data set and that's an issue as it's where FB makes the bulk of its revenue. The media has made is sound like some nefarious actors hacking the core of FB when it was just about monies paid or not paid.
"The company could very easily write native apps for each platform with per-version API keys, require users to be 18-or-over, charge a token amount (e.g. 89p/month) on direct debit to ensure people are real, and turn verification into a cheap annual service (e.g. £35/year) where photographic identity documents and the account telephone number is matched to real-world selfies on a regular basis."
How long would it take before a document was leaked/stolen that connected account names with real names, phone numbers, payment details, etc?
There's no way in London Under that I'd ever send some social media site a copy of my government ID, a real telephone number and a valid bank card number. I imagine there are legions of pseudonymic accounts where people don't/can't divulge their real names. They could be killed for their views. I also expect that plenty of accounts are from government TLA's and politician's offices where they don't want any connection of the comments as they pretend to be somebody else. Wouldn't it be so useful to get a copy of those user names the next time Elon does a mass purge and somebody accidentally posts it someplace public?
John Does 1-4 Respond, "Good Luck; I'm Behind 7 Proxies!" https://i.kym-cdn.com/photos/images/newsfeed/000/131/714/7proxies.PNG
It will be interesting to see whether or not those John Does went cheap and skipped the seven proxies part. Of course, if the Does were smarter, they'd have parallelized the whole thing behind a shedload of IP addresses, and not hit the Twitter servers so hard that they noticably overloaded them.
"It will be interesting to see whether or not those John Does went cheap and skipped the seven proxies part. Of course, if the Does were smarter, they'd have parallelized the whole thing behind a shedload of IP addresses, and not hit the Twitter servers so hard that they noticably overloaded them."
It a tactic to be weighed. If you can sneak bottles of beer out the door of the store one at a time with no problem, maybe you do that. If it's the same chances you'd get caught and blocked whether you nick one or a whole case, you might find it better to go for the whole case. It reminds me of a person that went into a store with a pallet jack as bold as brass and left with a whole load of beer in one go. All it took was a generic uniform, a cap and a clipboard and nobody asked any questions. In the US it's often the case that the distributor brings in the new and takes away the old (if anything is past its best-by date) rather than it arriving on company trucks and being handled by store staff. The exploit worked once and now staff and managers keep an eye out for that sort of thing with distributors required to check in when they arrive and provide an accounting of anything being removed prior to it be taken from the store.
Today is a good day.
Anything that makes him unhappy and even better causes him to lose $$$$ makes me very happy.
No matter how many Tesla bots there are on this site, until I can buy a spare part for my Tesla he can go pound sand. I'm not talking something major or safety critical but the dog chewed on some internal trim and I'd like to replace it but God impersonator Elon SKUM has decreed that only Tesla Service people can do this and I'm not waiting 3 months to even get an appointment.
I found the bit of trim at a scrapyard and fitted it myself in under 20 mins but the fact that he and his droids won't sell me a part has turned me off his computer on wheels for good. Next time I will NOT be buying a Tesla.
The case is a waste of time – malice can't be demonstrated and "fair use" doesn't exist on the internet – but the court was chosen because it routinely sides with plaintiffs in the most ridiculous of cases and the aim may simply to tie up companies legally up and force them to settle out of court.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2023
