Re: Pandas and Anaconda
Apparently all the cool kids are using Polar or Arrow. But it's great we have all of these cool tools.
54 publicly visible posts • joined 31 Jan 2011
Yep, excel wasn't designed to work with large data sets. Don't get me wrong it's still a great tool.
I had some python experience from some sysadmin tasks. I was given a spreadsheet sheet with approx 500k records and needed to identify the duplicate records based on different data fields. Even the 64bit version of excel would lock up.
A day later a 14 line script with pandas identified the duplicate records and outputed the common records to another file so we could investigate and reconcile the records. Took it about 5 mins to spit out the output.
At some point there will be legal mandates for energy recovery for Data Centres. I'm aware that there are already schemes where this piped out to the local municipality to heat local home's.
However building facilities that are more efficient in recycling the energy is going to be a challenge in both terms of design and cost. This would indicate the need for more liquid cooling to be able to capture the exhaust heat. What's not going to decrease is our appitite for compute power.
Its the standard youll be more attractive to the opposite sex if you use x product. Its an advertising staple. The lynx ads were a lot more blatent and the diet coke ad bordered on porn. Its tongue in cheek humour people need to get over it. George Carlin was dead right
“Political correctness is America's newest form of intolerance, and it is especially pernicious because it comes disguised as tolerance. It presents itself as fairness, yet attempts to restrict and control people's language with strict codes and rigid rules. I'm not sure that's the way to fight discrimination. I'm not sure silencing people or forcing them to alter their speech is the best method for solving problems that go much deeper than speech.”
I used both. Nothing is perfect. the UNIX shells were the inspiration for powershell. It's comes down to best tool for the job. I like powershell. Recent changes to decouple it from the version of Windows are welcome. I learned a lot from sed awk and Perl back in the day. Fact is that with desired state computing admins,spend less and less time logged into servers. I've played with nano. it's seriously retro interface reminded me of DOS. take everything on its merits.
There was a policy of decentralization in setting up government departments in Ireland when the office was set up. I always get the feeling that we have a data protection commissioner because we need to be compliant with EU law not because its a good idea to have someone watching the space.
The location is probably also coupled with the data protection officer asking some important and awkward questions of various governments down through the years. Typically our watchdogs are toothless tigers but the data protection commissioner has bitten some big important arses in the last few years. Its no coincidence that the funding for the department has remained static even though the workload has increased dramatically.
Obviously because the Irish are best workforce in the world :)
Actually it stinks because indigenous industry/business don't get any where near the same amount of relief. The Irish economy did not do so well, it collapsed after the construction bubble as manufacturing jobs went after the tax reliefs were up. This was masked by the move from manufacturing to construction. in the jobs market.
Setting up indigenous sustainable industry is key to the long term success. The fact is that Ireland is an export lead economy because its population is relatively small. You can hardly call a debt ration of 120% to GDP a success. We had our chance to build something for the future and we blew it. We should have been concentrating of renewable energy,moving up agriculture value chain and developing our own IT industry. Instead we built apartment blocks in fields.
Give the Irish credit, we were more reckless than anyone else with perhaps the exception of Greece. We are till paying for it and unfortunately so will our children.However I think the Scots have a better chance of being financially prudent as there is culturally a more responsible attitude to money.
Its all a bit Monty Python as in "what did the Romans do for us?"
What ever the Scots do decide I hope there isn't any bitterness afterwards.
Fab facilities are often described as 200mm, 300mm or 450mm referring to the wafer size. The actual process that may run in these facilities may be different nm process as fab are often refitted. The equipment or tools as they are known in the trade are normally designed for a specific wafer size. Generally speaking the larger the wafer the more chips you can produce per wafer leading to higher volumes.
Try the ISE its not too bad, its not eclipse but its pretty good to start out with.
http://technet.microsoft.com/en-us/library/dd315244.aspx
I tend to use use notepad++ for heavy editing cos it has some nice macro capability but the ise is good for knocking out scripts.
If you are going to admin Windows you need to have some handle on powershell.
if you are feeling particularly masochistic you could always configure VI to work with powershell, you loose the ability to run scripts directly but if you comfortable in vi then perhaps worth a shot
http://juliankay.com/development/setting-up-vim-to-work-with-powershell/
Theres also an eclipse module but I haven't used that
http://marketplace.eclipse.org/category/free-tagging/powershell
Perhaps windows 8.1 will iron out the kinks in the original 8.0 release. I was the canary for my department as I pretty much told everyone to hold off till 8.1 or install classic shell.
My main bug bear is windows 2012 interface being the same as a tablet to me that was a major fail.Yes I know we are supposed to be using power shell to configure our servers but sometimes you have to log into the box.
Windows 8.0 was the first desktop i couldn't grok within a few hours and Ive used everything from windows for work groups to hp ux to mac os to gnome kde and unity including all of the variants of windows. I really felt that with windows 7 that Ms had finally nailed the desktop . Its really bad when your desktop UI makes you feel annoyed and stupid and costs you time.
I get that the tablets needed a different UI to support touch but why did they have to foist this on desktop users. At home I'm booting linux mint with cairo dock and its just lovely.
Intel arent MS even if they have been closely associated in the past. the difference here i would imagine is that Intel will provide APIs for anyone to build applications on top of. The interface will depend on the app developers. Also the Konnect works pretty well and that's an MS device.
Keyboards aren't natural so there is a movement to design a more natural interface for computers. they are a tool and tools evolve.A lot of people slagged off the Kinnect but I was quite impressed with it. My non geek friends love it. Same for the WII , the interfaces were so much more intuitive for the novice. When your 70 year old mother and 8 year old daughter can enjoy a computer game in three minutes of getting their hands on the device thats what this is about. We've moved on from the all people need to be mechanics to drive cars phase to all people can drive cars but we only need mechanics to service them model.
Also if they manage to crack using this technology to replace pin numbers and passwords securely I'll be delighted. Yes i know there are password managers but its still a pain in the ass and not very secure.
Bluetooth is still the connectivity medium that many devices that this chip is intended to work with use. You could say the same for wifi, a chip these days without these connectivity options is going to fail. Wearables at the moment is dominated by Bluetooth connectivity and this chip is aimed at that and other markets.
Yep good admins are paranoid. Triumph over experience over optimisim. But tape is still cheaper than disk and more portable. SAN fails , databases get corrupted and you dont ever find out how good your BCP is till you have to use it. Tape isnt fasionable but it still has its place.
When you start getting into large data sets its quicker to recall the tape offsite than to wait for the data to be replicated from offsite even with dedicated links and networks . Most large enterprises use disk to disk to tape . Tape is not sexy but it is very effective in many circumstances. If you need to offsite large amounts of data tape is still the most sensible option. Sneaker net is still very much alive and well. As someone who started his career in IT as a tape monkey swapping out DAT tapes at 3am in the morning and hoping I got the second tape in the drive before the job timed out , I have no rose tinted view of tape backups. However it does seem to fashionable to bash tape backups when they are still the best tool for the job in many circumstances.
The fact is that these people made several serious threats. They made them on very public medium and now people are surprised that this has resulted in prosecution. This has nothing to do with feminism and everything to do with you cant just say what you want without consequences. Just because its twitter doesn't mean you can threaten people. Theres a difference between disagreeing with someones position on a topic and threatening to rape them . I'm going to rape you seems to be the new Godwins law http://en.wikipedia.org/wiki/Godwin%27s_law.
Every story i read about the Xbox one makes it less appealing. I don't know why but I just assumed that the old xbox games would play on the new xbox . This seems to be the standard on other consoles and makes sense as theres a considerable investment in games from the previous generation of console. The ps4 and Wii u play the previous generation console games without any fiddling about.
When Apple pulled the plug on its server range it was a pretty clear indication that they were not actively pursuing the enterprise market. Apples main focus is the consumer market. Managing 1000s of clients is not a trivial task and most enterprises have deployed third party tool in order to help them manage be it windows, Linux or apple clients. Google has the resources to roll its own solutions but many IT shops are struggling with this as they don't have the internal resources or expertise to manage that many apple clients. with the rise of BYOD the management overhead for IT shops has increased dramatically and many are not convinced about productivity increases from people using their pet hardware/os.
Puppet/Chef/cfengine are tools for managing configuration. This is something that system centre actually doesn't do very well . System centre does a great job of monitoring the event log and bouncing out the events to an alerting system but working with the management packs required for any thing that is not written by MS is a pain in the ass.
What we are seeing is a move to manifest driven configuration for large deployments. Trying to do the same with GPOs is horrible. Puppet does have its draw backs trying to manage manifests is a pita they suggest using a svn/git to manage these in large deployments .
I
The fact that the announcement was hosted by ARM holdings undermines the message somewhat. Your going to see ARM in the DC to what extent is unknown as previously stated there has to be a compelling business case to deploy in large numbers. The tool chains for Linux on X86 are more developed for DC workloads. Arm is going to push Intel in the DC space but Atom is going to start eating ARMs market share in mobile. When we talk about ARm which arm chips are we talking about and how compatible are they with each other. ARMs memory management is still poor and anyone who runs a large visualised workload will tell you its memory you run out of long before cpu. Having said all of that Shuttleworth does have one thing right the way apps are written for the cloud is very different from the traditional X86 and ARM architecture apps. Ironically Intel is now one of the biggest contributors to open source software and this will help maintain its dominance. Having two strong competitors in the DC space can only be good for the consumer.
I'm not defending MS , if it was my business being affected Id be pretty pissed. I dont think even MS PR department could spin this one out a success story. My point was that the recovery time is unacceptable an that adequate recovery procedures weren't in place or it would be sorted. MS have again shot themselves in the foot.
Linux does not = Reliability any more than Windows = Reliability. Any sys admin worth their salt will tell you that things are a lot more complex than that. The issue is that they cant recover quickly form an outage and that is usually more to do with their operational procedures and safeguards than the particular OS. However its hurting their customers and thats bad for business.
No system is perfect. There is this idea that the cloud is magic and infallible. Its still hardware and software and many meshed systems its just sitting in someone else's facility. There are very few cloud providers that haven't had some issue that impacted customers. This happens in the enterprise and unless it impacts external customers you dont get to hear about it. Enterprises dont air their dirty linen in public unless they have to.
There is a cultural thing where the new cloud web 2.0 kids seem to think that just because you stamp cloud on a service that you forget all of the lesson learned by operational teams over the last 30 years. The basis tenants of keeping things running remain the same. As much fun as it is to poke fun at MS, Amazon, Rack space , IBM or HP it underlines the issue that no large environment is free from outages.
The best operational teams understand this and have good plans in place for recovery. They assume something will break and have factored this into their business and operational process so when something does go wrong they can recover quickly. The issue here is not that something went bang its that its taken them over a week to fix it.
Got one at Christmas, great device , easy on the eye, fantastic battery life. Use Calibre to manage my library and is even compatible with overdrive/Adobe digital edition (local library e access). E ink seems to be much easier on the eye. It also displays the cover of the last book you were reading in sleep mode. Its slim light and fits in my coat pocket. It just works, has wfi for connecting to e book and magazine sites. Colour is not a need for me so Im happy with my kobo. The build quality is pretty solid and the interface is pretty simple too / Also nice to see some diversity in the Market the Kindle is also a good product but prefer not to be tied to Amazon.
The desktop train has left the station. Canonical has realised this and Unity is an attempt to create a UI for the next wave of devices, tablets, smart TVs etc. I dont use Unity as the ui drove me nuts but I'm not the target market. I've used Ubuntu since 6.04 and I loved it. It felt like home. However things change and there are other choices.
Id like to see Canonical becoming a profitable venture, there's a lot of really good people there doing something they are passionate about in technology. Yes the Amazon search lens was a cock up but at least they put their hands up. Canonical owes me nothing. This sense of entitlement that people have to something they haven't invested development time or money into amuses me. The thing about Linux is that you have choice its a key differentiator over Ms windows and Apple.
It finally got one Powershell. Its not perfect, but its a whole lot better than Vbscript or batch files. But then again Bash scripting and Perl aren't either so its the normal case of having to live/adapt to the limits of the tools you have. The tool kits are getting better as a whole. The fact is that as a developer you are going to have to know several programming languages, old systems have to be maintained and new systems requiring new features need to be written. I'm a sys admin and I have to be familiar with several scripting and programming languages to support the business. It comes down to the best tool for the job at hand.
My concern is that a lot of the University's have become vendor training centres rather than really teaching computer science and programming. I came from a Physics background so maybe my education that taught how to analyse and solve problems and test ideas left me unhindered by any only one way type dogma. If you work in IT you are going to realise that you are in an industry that requires life long learning. If this isnt something you are prepared to do then your options become increasingly limited as time progresses.
@h3, I think the difference here is that you are looking at this objectively. I use both *Nix and Windows each have their advantages and disadvantages. Theres still a few things you cant do in powershell but they are getting fewer with each release. Having said that my 12 year old bash and perl scripts still work. Its really about having the right tools for the job. I find it ironic that we are still having the my OS is better than your OS when most users don't care because they are busy getting things done . It the equivalent of metric vs imperial tools sets in the mechanical world. saving said that planting adds in a local search is reprehensible, the fact is that search is becoming less useful as a result.
I never thought Id have to say this but it seems that MS is being honest with users than Canonical. I get that Canonical has to make money . The software store was definitely a step in the right direction where Canonical could make a cut of sales from the distro that they create and maintain. I even bought some software and as software transactions go it was the most painless experience , better than google play or MS download.
Ubuntu One was another step in the right direction where you paid for a service if you wanted more space, having all of your DRM free media files synced on any device is quite cool.
However I have to draw the line when my desktop becomes a SEO vehicle for a company such as Amazon which has given nothing back to the free software community. If MS did this everyone would be screaming blue murder, just because a company is in the open source business doesn't not give it carte blanch to abuse the position.
Lots of companies use Foxconn components in their products. Foxconn replaced AMP in the motherboard connector business about 15 years ago and up scaled from there. Motherboard production is a low margin business so any reduction on cost is sought to remain competitive. You want cheap consumer products then the biggest factor in most manufacture is labour. If you want to purchase products that have been made by companies that treat their workforce well then be prepared to pay a premium for it. There are many reasons for not buying Apple but in this case if you were to veto every purchase of electronic equipped based on Foxconn components then you would find yourself with few choices.
In China workers are having to fight for rights that have been won in the Europe and the US. Expect this to continue for some time. These rights are under threat everywhere in the global economy . Once these rights get established in China expect companies to set up elsewhere where labour is cheap and they can operate as they please.
The whole idea of patents was to protect the inventor. Now it seems the idea is to cripple your competition and prevent sales of similar competing products. The system is now penalising invention and thats only bad for customers. I doubt very much that any US judge is going to find against Apple and ban imports. Its a mess ,the consumer is going to suffer and perhaps the Chinese workforce who will have to bear the brunt of the inevitable cost cutting to pay for the patent wars.
Cutting a profitable business that allows them to vertically integrate in the enterprise, very forward thinking. There are significant economies of scale involved in the production of hardware. Wall street , slash and burn for next quarters profits! Thats what got HP into trouble in the first place. There was a time when HP concentrated on execution and R&D. They need to get back to that.
@mark65 Well said. It would seems that we are going back to the dot com days when VCs handed over stupid amounts of cash for businesses that would never work. Seems that banks and investors never learn. There have been very few tech IPOs where the value has remained. Creating and driving a successfully new business is hard that's why so many fail. I know of several companies that were just run till the VC ran out and the staff move on to the next one. They are all of the "koolade" variety, all flashy website and no plan or ability to execute consistently.
Mail clients aren't exciting. They aren't very web 2.0 but they are very useful. Having a local store of your emails especially in business is vital. Web mail is useful for short stints but if you are in an environment where you write more than 5 emails a day of more than a few paragraphs then a dedicated mail client is the way to go.
The fact is that the all of the Linux email clients have been inferior to Outlook. Outlook is one of the few products that MS nailed. It is very difficult to give up the integrated messaging, calender, contacts and collaboration once you get used to it. It does cost a lot but your paying for the productivity boast. Large enterprises need this kind of capability and are willing to pay for it.
I use Thunderbird on my Linux Mint and Ubuntu set ups , I even installed it for my Mother. She loves it because it works and it makes email simple she has all of her email contacts in once place and the interface is pretty easy to understand. I would be sad to see it loose the little support Thunderbird gets from Mozilla but its their product and they are entitled to do this if they feel their resources would be better spent elsewhere. I wonder if the the libre office team might take up the baton. The office productivity suites on Linux really do need a lot of love and a polished functional email client is a must.
Linux isn't perfect but then neither is any other operating system but for the most part its good enough. The by-product of this work maybe more fundamental in that it promoted open source as a method of development. It laid the foundation for collaborative programming projects using the internet and harnessing the distributed talents and resources of volunteers globally.
This article seems to be more of a sales pitch for Amazon than a look at the impact that the cloud is having on Enterprises. Cloud really meaning hosted in someone else's data centre. The key point that being missed is lack of trust and business knowledge that keeps services in house. Most large enterprises have what is essentially an internal cloud which they have control over.
Samsung are one the largest Chip foundries in the world. They havent been number two in Volume since 2002 by accident. There are not many other companies that have anywhere near the same silicon maufacturing capability. Intel is the largest in terms of volume. Chip process design and manufacture are expensive businesses requiring massive R&D and Capital investment. The idea that a Fab can easily ramp up to take the kind of volume for ipads/iphones volume is a misnomer. Samsung have both the LCD and phone chip market sown up. Even with two years notice its hard to ramp up silcon production , it usually means new process/equipment.
There has to be a way for inventors to monitise their inventions or innovation ends. The patent system has become a trough for the legal profession.
Canonical are now more focussed on mobile and tablet devices. this is driving UI changes. Canonical own Ubuntu and if thats what they want to do its up to them. There a great Ubuntu community and I hope they don't get sidelined as Canonical goes for more commercial Success.
I've moved to Linux Mint it may be that after 20 odd years of using a computer with a menu system I cant really adapt to one without it. Linux Mint seem to be taking a more measured approach as the UI will evolve. Most users are content consumers not content creators and the ratio is getting lager in favour of content consumers. I might go back to Ubuntu in a year when the glitches are ironed out