Posts by SJG 53 publicly visible posts • joined 6 Apr 2010
You can never design a data warehouse up front and then build it. By definition, you don't know the data well enough... the point of the warehouse us to explore and mine data you don't understand, and to run queries you don't know you need.
The fault here lies with the underlying procurement approach which drove the waterfall development method. It should have been an agile project, delivering small increments of business value.
Fully agree. "Agile" should be seen as one aspect of software engineering, not the complete story. To take this to the extreme, I would be very reluctant to get on a fly-by-wire plane whose control software had been driven primarily by user feedback and developed without a detailed specification. Software engineering is a complex and broad topic, let's not dumb it down into agile is good, everything else is bad.
"We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one we intend to win, and the others, too."
https://en.wikipedia.org/wiki/We_choose_to_go_to_the_Moon?wprov=sfla1
Let's assume that BA have lost a data centre. The process of switching hundreds, maybe thousands of sustenance to a secondary site is extraordinarily complex. Assuming that everything has been replicated accurately (probably not) then you've also got a variety of RPO recovery points dependent on the type and criticality of system. BA have mainframes, various types of RDBMS and storage systems that may be extremely difficult to get back to a consistent transaction point.
I know of no companies who routinely switch off an entire data centre to see whether their systems run after failover. Thus BA and most big companies who find themselves in this position will likely be running never fully tested recovery procedures and recovery code.
The weak point of any true DR capability is the difficulty of synchronising multiple, independent transactional systems which may have failed at subtly different times.
Encryption, Compression, Data Aggregation, Search - all these could be hardware accelerated as hardware acceleration has greatest benefit where there are very simple, and well-defined algorithms that can be parallelised across a dataset.
None of these are particularly useful on a mobile or desktop device, but on the server-side they can accelerate many data-centric tasks ten-fold or more.
At the moment, all those thousand-node x86 bigdata clusters out there take multiple clock cycles to apply an operation to a single data item - SIMD allows multiple data per clock cycle, but hardware or silicon based acceleration provide the opportunity to apply an operation to an entire dataset with the CPU hardly being involved at all.
This doesn't make any real sense. I guess it would work for unencrypted file data, but most really useful data is either in a database and likely unreadable (and if sensitive also encrypted, so definitely unreadable) or these days in Hadoop where there's usually no storage box - it's direct attach disk to the hadoop data node.
Seems a database with a trigger on the table or an event filter for streaming data would
pretty much cover this use case without any of the complexity.
The MS Dblin case seems simple to me.
A US court has decided that Microsoft must provide information that it holds about a US citizen. The court doesn't care where that data is held. This is, in fact, right and proper.
However, Microsoft has the relevant data held in Dublin, and under Irish law Microsoft can't release data about an individual (whatever their nationality) without the data owner's permission or an Irish court order. This is also right and proper.
So, this is courts and jurisdictions working quite correctly. The underlying problem is that there is no international agreement about what should happen in this situation. Microsoft either breaks Irish law, or it commits contempt of court in the US for not providing the data.
If you think of an analogy, say Microsoft had printouts of the data but no electronic copy, the situation would actually be exactly the same, so this is not just a technology problem. This is a very similar issue (but not exactly the same) to the Swiss banks providing lists of offshore account holders who may be avoiding tax.
Just to be clear, if this was an Irish company, and Irish citizen with the data held in Ireland, and the original court action was happening in an Irish court, then the data would already have been handed over to the court through an Irish court order.
So at the core, this isn't about an individuals protection per se, it's about what should happen when the jurisdiction of a national court has an international impact and where there is no clear legal framework to decide what decision should be taken.
Loans are designed to be repaid. The fact of non-repayment can only be established over time. We can hardly blame the revenue for not realising that there was no intent to repay. I'm pretty sure that no-one explicitly stated that the loan would never be repaid when getting the scheme signed off by the revenue. Similarly it would only be after a number of tax returns showing no repayments that the HMRC could justifiably become concerned.
I've been a contractor myself during which I passed an IR35 investigation from HMRC, but I made a very conscious decision to avoid this type of scheme as they are all challengable. Where any corporate structure is put in place where the primary purpose of that structure is to reduce the tax burden, then it's challengable under current tax law.
A limited company's primary purpose is almost always to maximise profits to pay dividends (excluding not-for-profits etc). For the board of a company to make a decision to 'loan' an employee a tax free sum with the understanding that the sum will never be repaid is clearly against the companies interest. When this is compounded that the board member making the decision is the same person as the employee receiving then there's a conflict of interest - there's no 'arms-length' decision making here. So in corporate governance terms this is very dodgy from the start.
It's also the case that employee loans are perfectly fine - many banks offer them to staff at reduced rates so no issues there per se - the loan will get repaid eventually. The issue arises if the loan does not get repaid.
... and here's the key point about why the revenue are perfectly within their right to take this approach...
At the time the loan is given, everything is hunky-dory. The revenue signs the scheme off - as a loan with the expectaion that it will be repaid. The issue can then justifiably be 'discovered' by the revenue when they find out that the loan has not been repaid, and that fact can only become evident after some time has passed. Unless the scheme includes specific notification to the revenue that the employee had no intent to EVER pay the loan back (something I doubt), then the revenue are well within their rights to investigate when it becomes apparent that no payments are being made.
Of course, there's an easy way to get out of this. Pay back the loan, then the Revenue have no cause to complain. Borrow the money or remortgage your house, but pay back the loan. Then pay yourself the way that you ought to have done in the first place - i.e. through salary or dividends. You'll still need to pay tax on that income, but it will probably be less than the revenue is asking for.
BTW, the clue's in the name - it's a loan - loans get repaid unless the recipient goes bankrupt. So that's the choice.
I've encountered a few problems along the way that adding some level of monitoring made a particular problem go away. IIRC, first time I saw this was on an ICL mainframe in their Quickbuild Application Master 4GL where just referencing a variable altered the namespace so that the correct variable was then subsequently picked up. Probably sometime around 1990 IIRC.
I think you'll find that The Big Bang Theory progressed rather quickly to the point where there are just as many female as male scientists. Interestingly the actress playing Amy the neuroscientist is actually a neuroscientist in real life https://en.wikipedia.org/wiki/Mayim_Bialik.
Before you start correcting people, it would be worthwhile reading at least one page on the subject.
See the Apache Hadoop home page for some clarity : https://hadoop.apache.org/
A direct cut and paste from that page shows that most of what you mention are all classed as Hadoop.
Hadoop Common: The common utilities that support the other Hadoop modules.
Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
Hadoop YARN: A framework for job scheduling and cluster resource management.
Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
The exception would be machine-learning which is Apache Mahout.
Also all 10 related top level Apache projects are not unique to Hortonworks, they're pretty much the standard whichever stack you adopt (e.g. Cloudera, etc).
I really have no idea where you are going with your Docker comparison - docker is a method of packaging, delivering and running distributed apps, none of which Hadoop does, and Hadoop is, at its core a MPP platform, something which Docker doesn't do.
Try reading this (http://www.theregister.co.uk/2014/01/31/eu_us_data_transfer_to_be_suspended_if_united_states_does_not_fix_probs/)
EU law does not allow the export of EU citizen data to the US. A US jurisdiction may issue a subpeona against a US company to provide data that may be residing in the EU (e.g. see what is happening in Ireland with Microsoft at the moment) but that subpeona (or any other US legal instrument) does not give any legal basis to that data being provided to any US body - i.e. a US subpeona does not give the legal right to a US company to break EU (or any member state) law. The company holding the data would be breaking EU or local country law if the US company were to comply with the subpeona. Technically the company may be in breach of US law by not complying, but that's their problem.
Of course, what actually happens is that the US body (aka NSA) simply takes the data anyway without any legal basis (often with the aid of GCHQ of course). In this situation, the fact of who is the ultimate owner of a company or in which country the company is registered is totally irrelevant.
In other words if you think your data is safe from the prying eyes of multiple governments in any company or country these days, think again.
You're missing pretty much the entire way networks work ....
MAC codes are not visible outside the LAN. Similarly, even IP addresses do not stay constant even on an apparently fixed broadband connection as IP addresses are often re-allocated. Some 'broadband' cable services are also provided via a shared IP address which serves multiple physical locations. And don't get me started on any connections to the internet via mobile 3G or 4G, that's another Pandora's box entirely.
If it were that easy that everyone would be doing it already. DDOS attacks are notoriously difficult to stop - go try googling 'prevent DDOS attack' and you'll find lots of information about how corporate security and hackers are in a constant escalating battle.
Substantial use of the word 'could'. It's OK to have some innovative thinking sometimes, but there are far too many guesses in here.
I'm not sure what problem 'server SANs' are really solving. From this article it seems that they are unable to deliver data fast enough to servers because of the network time involved in the data transfer, and so they would seem to be unsuitable for their core purpose - supplying data to apps. Suggesting that data then has to be cached in the server increases complexity just to cover the inadequacy of the chosen storage platform - chosing a fit for purpose platform is a much better solution.
I'm not a fan of big arrays, but this seems to be a retrograde step even for them. They are built for online disk replacement, multi-tiered caching, fast(ish) FC connectivity etc etc. moving to an estate of generic 2 CPU servers with a bunch of disks attached to each somehow seems a retrograde step, and IMHO for certain a maintenence nightmare.
This glosses over the real storage challenge - how the hell do we server up data fast enough to the massive amounts of CPU throughput we are seeing on even 2 CPU servers. There's no way that even a pair of 10GbE connections is going to be enough, so unless we rethink the core datacentre network (and physically upgrade) then server SAN's seem to be a solution looking for a problem.
If only life were so simple. The storage vendors have for as long as I can remember, refused to make any significant optimisations for database. The long term fixation on block or file protocols have limited storage to the dark ages. Most databases throw away most of the data retrieved from storage because the database only want a few rows (a couple of hundred bytes) out of each I/O which is typically thousands of bytes. Add to that the fact that when a 'select' statement filters a small number of rows out of a very large dataset, then massive amounts of data are passed from the array to the database only to be immediately discarded as the filter kicks in.
The recent(ish) innovations by first Netezza (now IBM), then Oracle (Exadata) and Hadoop are all examples where standard storage protocols have been discarded and a fundamentally different approach - one in which a significant amount of the database processing is pushed down to the storage layer so that less data has to pass over the network/fabric. All three of those technologies have the common approach of running much of the work locally within a storage component - i.e. at PCIe speeds rather than network/FC speeds (Netezza originally used a FPGA directly attached to a disk)
Of course, this is not a new idea - in the 1960s ICL invented the CAFS system (http://en.wikipedia.org/wiki/Content_Addressable_File_Store) which is fundamentally the same idea.
Most database systems today are I/O bound, not CPU bound. When these boxes have a max memory bandwith of 136GB/s, then if they're running a database or any IO intensive app then they need much more bandwith than PCIe SSDs can deliver to keep those CPUs busy. There's not much point in having most of the 36 cores just hanging around doing nothing while they wait for IO.
When we look at IO bound systems generally, then the bottleneck is getting the data from the disks/flash to the CPU fast enough. We have IOs galore on the back end and cheap CPUs with enormous capacity. Unfortunately most systems still connect with antiquated protocols that were originally designed for spinning disk and those connections will often be the bottleneck in overall system performance. NVMe has been designed specifically for the high performance gained from flash based storage.
Poor article and even poorer comments.
NVMe is an industry standard initiated by over 90 companies (so its not just Oracle). The first chip sets were available in 2012 (so no-one could have done this 5 years ago), and there was a re-badge in 2014 with 65 companies involved to NVM Express and it's not just PCI Express SSD - in fact it was explicitly defined to address the issues that high throughput flash experiences in PCI Express.
The already released hardware products are by Samsung and Intel : http://www.nvmexpress.org/products/. It is already supported in Windows, Linux, Solaris, BSD et al.
"The NVM Express, Inc. is directed by a thirteen-member board of directors selected by the promoter group, which includes Avago Technologies, Cisco, Dell, EMC, HGST, Intel, Micron, NetApp, Oracle, PMC, Samsung, SanDisk and Seagate."
It's poor journalism and commentry when people can't even be bothered to look at Wikipedia or find the relevant standards website.
"Shop Direct generates 1.3PB of storage data each day"
Seeing as the LHC only generates 25PB per year (http://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid), I think we're a few orders of magnitude out here.
also Pearl should be Perl
One day The Reg will get someone who knows a little technology to proof read these articles. There's no sign of it yet.
This article (http://www.iplayerconverter.co.uk/articles/recording-tv-radio-in-the-uk.aspx) has a fairly strong argument that saving iPlayer programmes for later personal use is explicitly legal, and in fact there is an argument that the BBC is preventing us from enjoying rights that have been granted to us.
We shouldn't get hung about technology here - the BBC provides programming which we are allowed to save and watch later. Whether we choose to save a recording on tape, CD, mp3 is not relevant to legality. Ignoring iPlayer for the moment, I can still record from a direct broadcast and then save to mp3 today, it's just too time consuming.
Grabbing a recording via iplayer (as long as I'm not circumventing encryption) for my own later use is just fine. Unfortunately, the BBC seems to be trying to make this harder for us all. Like many others who have commented here, I time-shift radio programmes for listening in the car or while on a train - something the BBC still doesn't offer.
Unfortunately, the 'obsolete' service is not so obsolete when both my panasonic TV and sony bravia bluray player have lost iPlayer in this debacle.
If you're going to knock your competitors you should check your facts first.
Oracle HCC was an Exadata only feature until September 2011 (http://www.oracle.com/us/corporate/press/508020). The purchase of Sun was completed 18 months earlier in January 2010 (http://en.wikipedia.org/wiki/Sun_acquisition_by_Oracle). So your statement is factually incorrect.
Oracle HCC is not simple block compression, it uses advanced database structure aware algorithms that can re-organise data at logical units way larger than the physical block level to increase compression efficiency. The algorithms for this do not exist in today's disk controllers.
You might be thinking about Oracle advanced compression which is still, and always has been, supported on any hardware platform that can run an Oracle database.
Disclosure: I check my facts before posting and I'm also an Oracle employee.
Oddly enough, this really is about service. Understanding which sites get the most distinct users over a month can only be done with some data being retained through the month, otherwise the carrier cannot differentiate between a very small number of users with very high usage or far more distinct users occasionally accessing a site.
At the moment, in most EU countries it's against the various privacy laws to use data collected for one purpose for another purpose - without explicit user consent - hence the Phorm/BT scandal many years ago. You'll be quite surprised by the lengths most telcos go to to protect privacy data. It's the government security authorities that ride rough-shod over our privacy, and also force companies like Vodafone to collude with them and then use legal means to prevent the telcos from going public.
Vodafone takes the privacy of its customer data very seriously.
Your level of self interest is astonishing. Probably means another 500 jobs lot being lost from the high tech sector in the local Sony factory. In this area, Sony has been one of the best employers from almost any angle for the past 20 years - good conditions, good pay and they have show a strong loyalty to the workforce.
Shame on you.
The rules on the export of personal data (of which the content of an email between individuals would be an example) are clear and covered under the European data protection laws. It is an offence to 'export' data without the recipient being designated a data processor and the relevant contracts already being in place. See the Reg article here : http://www.theregister.co.uk/2010/05/19/eea_personal_data/. It is also very clear that it is the location of the data controller that is the key - and data controllers must be in the local EU jurisdiction.
There is clearly no blanket exclusion to this that would allow companies whose ultimate ownership is outside the UK to export personal data, even where the legal authorities in the requesting legal jurisdiction demanded it. In fact there are specific agreements in place with the US for the sharing of airline passenger information with US Homeland Security.
The implications of allowing, for example, any foreign jurisdiction to force the export of criminal history, medical history, DNA information, financial transactions would be quite substantial. As someone pointed out, much (probably most) of the UK government's data is held in data centers owned and run by US companies such as IBM, HP etc, so allowing this would mean that an arbitrary US court could request pretty much anything they wanted.
There are so many new (and often untested in court) laws around data privacy and security that this is likely to take some time to play out. It's even possible that it would be a serious criminal offence (aka computer misuse act) for a US based Microsoft employee to access private data held in Ireland against the local regulations - this would be considered the equivalent of computer hacking.
My understanding is that if Microsoft were to fulfil the request from the US court, then (assuming the data is personal, and is related to a EU citizen) then the registered data controller for that data in Ireland would have failed in their duty to protect the data and so Microsoft Ireland could also be fined.
If the data is pertaining to a US citizen located in the US then there is probably no breach, however if the individual purchased a service from the local service provider (i.e. the US based individual directly contracted for the service with, in this case, an Irish provider) then the law starts to become a little murky. I have no idea what the situation is if the US citizen happens to be in the US ...
All this, of course, depends on the T&Cs of the service that was purchased, and of course we all always read that small print, don't we ....
In the latest analysis, Oracle is the 13th largest contributor to the linux kernel with 1.3% of changes. Notable by not appearing in the top 30 despite their business models being substantially reliant on Linux are :
Amazon
HP
EMC/VMware
SAP
Yahoo
... and none of the bigdata startups who are all totally reliant on Linux.
Four years ago I switched my laptop to a Mac after various Windows machines since Windows 2000. My previous employer (last year) mandated I use a brand new Lenovo on Windows 7. I hated every minute and my new enlightened employer allows me to use my four year old Macbook Pro. They gave me a brand new Lenovo but it sits unopened in its box - I prefer the four year old Mac to a brand new Lenovo on W7.
At home I mainly run Linux Mint (I may go back to Ubunu in the next version) and it's my development platform of choice. I despise the Apple lock-in and I have Android phones and tablets and I haven't used itunes for many years since I left my iPod on a plane.
I also built my 14 year old daughter a windows based games machine about a year ago; much faster that our family shared iMac. Yet my wife and daughter still argue over who gets the iMac.
So, for all those semi-religious apple haters/lovers, please give it a rest. It's your money, go buy whatever you want, but please stop telling everyone else that they're idiots.
Looks like they read your post - Oracle now trading above yesterday's close, so the 5% fall in after market trading now regained.
Now, i just wish I'd had a $15 million to buy after the fall - looks like someone bought about half a million at the low price, taking a profit now making a cool $700k in 12 hours.
... and just as in teenage sex, the new breed of young data scientists will also eventually realise that they have spent a long time fumbling in the dark re-discovering the things that have been common practice for some time.
However, I do wish they'd catch up a little quicker. Maybe one day they'll realise that:
* there's no such thing as 'unstructured data' (if it's unstructured then it might as well be white noise)
* it's daft to call a whole family of databases"NoSQL" then spend the next 3 years building SQL on top
* taking data out of a relational database and putting somewhere else doesn't make the data "non-relational"
* just because the security cameras in you HQ record more data every day than your customer ordering system, it doesn't mean they hold more value
* when it's harder to merge data in your database than it is using vlookup in excel then you probably don't have the right database for analytics
I guess I shouldn't really complain; after the SOA guys spent 10 years spending all the CIO's budget on integration projects that never seem to deliver anything more than a pile of questionable documentation, perhaps it's the turn of us data guys. Hopefully there'll be some budget leftover for some real data analysis after the "Bigdata" teenagers have finished spending on all those blue (or should it be white) elephants.
My 602P is still working well and sits happily in the top drawer of my desk.
I probably learned more about the basics of programming from the 602p than my early days with Basic. It's assembler like commands with a total of just 512 available 'steps' taught me to be a very lean programmer. At least it did have a GOSUB which started me thinking about subroutines very early on.
Together with a few friends, we managed to write some games, even an adventure game which ran in a 9x9 grid and an interactive game based on 'hitting' numbers which progressed across the screen.
It's a fantastic pity that today's super-powerful smartphones just don't provide the same type of easy opportunity for learning the basics of programming. I spent many hours of free lessons programming on the 602P, I'm not sure that anyone really does that with a smartphone.
There is, however, a nice simulation of the 602P that's available for android, so it is possible to simulate the environment, but it's just not quite the same.
Here's the link : https://play.google.com/store/apps/details?id=net.sourceforge.uiq3.fx602p
I have been using Yahoo Mail for my personal email since soon after it's launch. I use it almost exclusively by IMAP, very occasionally webmail and do use it every day for a least a few emails.
I don't recall ever having experienced or noticed an outage, I'm entirely happy with IMAP, and spam doesn't seem significantly worse than any other mail system. I just checked via email in firefox and all my emails appeared in less than a second.
I kinda feel left out.
... and what about in a car. Many manufacturers (even including BMW) still sell cars with only FM. Adding DAB at home is fairly easy - buy a set and plug it in. With the integrated radios in most cars this just isn't feasible.
There's even a safety aspect of this - moving from the integrated radios and steering wheel controls to a new DAB radio fixed somewhere under the dashboard is clearly a real backwards step.
Minecraft works fine on Mint 13, including all the mods I've tried so far. You'll need a reasonable video card and a reasonable CPU though, my 256Mb integrated intel video is certainly not enough. Build in a few popular mods in minecraft and the Java process can easily grow beyond 1Gb - e.g. the popular Tekkit set of mods really needs 2Gb just to get started.
I suspect that whatever OS you put on there, it's not going to be fast enough in Minecraft to keep a 12 year old happy.
I have both LoveFilm and Netflix and so have access to probably the two largest streaming libraries in the UK. However, if I were in the US, I get a whole load more on those same services - especially with more recent episodes of long running TV series.
For me, the problem isn't so much that I want to avoid paying (I do pay - see above) it's that I just don't understand why in this world of globalisation I can't buy a digital product that is available elsewhere in the world. I would even be willing to pay extra for more recent versions of these series - but someone somewhere has decided that my money is not good enough, and I have to wait at least a year before I am allowed to buy the product.
Can someone explain to me how it's better for the media companies to not sell me something I want to buy?
... and on another point, I travel in Europe a lot, but if I'm in a hotel in another EU country both Netflix and Lovefilm are blocked .... but I've paid and it's a single free market .... how does that work then ?
I just wonder how long it will take before we're being asked to switch off our watches too.
I have a Pebble watch that I packed in my checked-in baggage. When I switched on my phone after landing my phone immediately connected to my Pebble still in the hold. Ooops. Impressive bluetooth range of the pebble though :)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
