* Posts by sdunga

6 publicly visible posts • joined 13 Feb 2016

Should we all consolidate databases for the storage benefits? Reg vultures deploy DevOps, zoos, haircuts

sdunga

Multi-vector issue

As a person working most of my life in service providers that manage server and DBs for customers I do recognize valid points in almost all points of view.

In any case a trend I see over the years is that developers and DB admins and now the new devops generation and automation requirements work more and more in high level with little understanding how a computer works. Wellcome to the cloud era.

The examples and issues are so many I would have to right a book, but I will give few examples.

Devops and dB admins when specing a serve 80% of the time brute force resources and almost always ask for way more than needed, and in a virtual by default era, this brings some issues. For example in VMware environments, a very common and known problem with DB servers is with CPU scheduling and monster VMs in general. The more CPU cores that aren’t in use in reality, the worse the performance can be, and I lost the counts of app and dB owners complaints about cpu performance I resolved by simply cutting the CPU count in half or reorganizing the cores to align better with the physical server specs.

Another common problem already mentioned is the quantity of copies DB admins use. They often think the best way to backup is to do a local dump every now and then and then replicate it to a second location or 3rd, usually another DB server they control elsewhere. This of course on top of the DB replication. Some even use VM snapshots on top. This practice drives to an uncontrolled flow of that and when you actually need these copies, like in ramsomeware attacks, all these copies are gone. On top the data is almost not dedupable or compressible for storage systems. And interestingly create a constant abusive load of sequencial data that often affects SSD performeance, that is now the default for DB. Now when I say affect performance I don’t mean that to a point DB admins notice, because many DBs aren’t that intensive and the brute force some times compensate. But to illustrate as per this week I have a case in hands of a customer complaining about performance during certain periods of the night that last for 8 hours, and when the teams checked they were creating loads that exceed 400 MB/S that is damm good for SSD storage. To prespective aws EBS SSD won’t provide you more than 250. The ticket still says they have a performance problem. Don’t ask me what they are doing in terms of tasks to generate such traffic for a DB that isn’t bigger than 300 GB. The traffic they generate is enough rewrite their fill DB every few minutes in a loop for hours.

The above example leads to the other problem that is related with automation and reporting. The amount of loads that are now consistently competing with DB resources without any planning, between backups, replications, indexing, dB updates and reads for constant automation flows and reporting are often driving DB servers to unexplainable loads. There are 2 sides to this problem, first is with the abusive use of programmatic queries on the DB side in SQL DBs, with excessive joins, rather than just take the raw data and do that elsewhere. The number of long running queries I see this days with queries as big as full poetries is becoming surreal. The PostgreSQL folks are particularly good illustration of this, for some reason.

App designers and dB architects are becoming increasingly bad in using cached architectures and not caring at all and pushing businesses to brute force resources that when I get involved in average I can reduced 50% in computing and storage and increasing yet the performance by some %. There are several reason for this, one and often not accounted for is poor documentation of code and high rotation on the developers, that rarely will bother to look at existing code and tables and do everything yet again from scratch, because of course they know better. The increasing cases of duplicated loads because of this effect is mind blowing. The amount of code and scripts all over the place that is deprecated but still running is getting in some organizations to quite worrying Levels.

The other limiting factor on resource management is again cloud related, the procurement process was eliminated for the most as the installation process. This is driving to the notion of limitless and loose of cost control, the devops guys can just spin another 100 servers at the change of a value in an api call, so why to bother to think if they really need 100.

I could keep writing examples and issues I see daily, but in general the cloud generation and all the abstraction layers in the middle created a wasteful resource generation of IT professionals and when you cross that to the known ego problem that most IT professionals have that they know better and what was done before is crap is helping even less. And of course the very human nature of laziness will ensure there aren’t cleanups being done on old scripts, apps and whatever more that isn’t used anymore.

The sad thing is that today we have so many tools and open source projects that can do great optimizations, from cache to queuing, but because the professionals that implement them don’t really understand the capabilities and install them with default configs not even bothering to understand how those tools actually work and how they can contribute to improve things. Many applications that have the right stack of technology, but because of this they become just and yet again a wasteful resource. Redis and such things are the prime example, very good tools that people thing that just because they are their and out of the box they will do miracles, then the outcome is devs saying that redis creates more problems with caching and that isn’t true, they just blame it for every bad code they do. This is a real phenomenon I see, redis is very often used to explain strange behaviors, when in reality is just bad code or app not being adapted to run in redis or distributed environments. Again because who writes and designs the code doesn’t understand the basics of the computers that are still there as they were in the 80s and still provide the same challenges, just in a much bigger scale.

Telia engineer error to blame for massive net outage

sdunga

Re: Thus is revealed...

Majority of these services are provisioned separate and along years, so in reality there is little holistic view over everything and dependencies and inter dependencies.... Not to mention that half of the telcos bought another number of telcos and there are a lot of skeletons hidden in the closets.

In general these big networks are everything less well controlled, no matter what the telco is, prove is the successive amount of big outages, no matter how big the name is, they all had their own episodes in the last 2 years.

Reality is plain simple and hard, all protocols Internet is based in are old, very old and were never meant to scale out, along the years people have been implementing a number of best practices and workaround to try to make inefficient protocols work to the new demands. IPv4, ven TCP, DNS, SMTP.... all completely inadequate for today needs, but all the baseline of everything we call internet.

Even the IPv6, that is almost not implemented at all, is dated somewhere in 2003, that is close to 15 years ago. BGP that is the big engine of all of this, is dated somewhere in first half of the 90s.

To make it more interesting very few big telcos really have any automation in place in the cores, because the networks are so big and have been built over the years with all kind of hidden stuff, no one dares to apply the blind law of automation and risk an outage. Automation is normaly implemented in the edge in customer services, even there due to the complex nature of many, there is no automation possible. Normally things like DSL, cable and this kind of services is where you find the automation piece.

Trumping free trade: Say 'King of Bankruptcy' Ross does end up in charge of US commerce

sdunga

Blah blah blah

In theory all good, in practice there aren't qualified people to work in many of these areas globally. Every single company has too many vacancies open where ever. Not a single hiring manager will tell you a different story. Now the interesting part. Many of these companies promising to create jobs in US know they can't fill them, so either they will just open them and never fill them or bring people from India or alikes. I know that SAP as an example that just said they are creating thousands of us jobs, are relocating by the tens people working in Bangalore campus to San Francisco for half of the local market price. And again this will not benefit US, regular American will still be unemployed, they will be sponsoring immigration, salaries will level down and in the end the money will be sent abroad back to the original countries. Really don't think these guys have an exact idea of what they are doing and in the process will give tax benefits for nothing. Reality is that US doesn't have enough native population with the skills required and global market is the only able to give an answer for the demand and even like that all companies are struggling. Final note India got a bad reputation because people thought IT engineers grown from trees and in the end they were pulling anyone from the street and quality was obviously bad, same happening in east Europe countries like Bulgaria, same happened in china, same will happen in US, they will start offering low quality to satisfy quantity demand.

Here's how police arrested Lauri Love – and what happened next

sdunga

Re: Time to stop this

Not as simple... Legally speaking in this things it applies the legislation of both countries, because the data was store effectively in US, so to data it applies the US law, even if whoever is in another country.

Put into a perspective, when a drug cartel executes someone in any country being based in another, should it be judged based on Colombian or Mexican law, if 30 people were killed (by hypothesis) same applies with terrorism acts.

Depending on the findings 99 year may or not be exaggerated, recently several cases of cyber crime have been known like the attack of a water supply in some city and changing the chemicals on the water supply, potentially this can kill or injure thousands. And I can keep giving examples that shows the complexity of these issues.

One thing is for sure, whoever goes into this path knows or should know the risk, I didn't saw him denying he did something, and such a encryption, is a bit obvious the guy may have something to hide. The guy may not even have bad intention on his acts and the majority of the hackers do not have them, otherwise the world would have already exploded and went to a chaos. Still he knew the risks, these are clever guys, so...

Most likely there may be some abuses as well from authorities and excesses, US are well known for that and UK are no different, just less competent. Regardless the funny part on this, is that this proves that authorities had a very week case and aren't prepared for acting in these situations, the UPS delivery thing arrest is a joke from the movies, and after all the theatre, 3 years after they still have nothing... This is the real sad story.

Big data lakes? Too many ponds, that’s the problem

sdunga

Re: @sdunga Not as easy as it looks

You just pointed some of the reasons isn't as simple as it seems I pointed:

1 - Most people (business deciders) don't know what they are doing or asking, big data appears as a trendy thing and all people want it, regardless if they need it or not.

2 - Getting the right guys for the job... that is the biggest of the problems. Worldwide there is a huge deficit of experienced engineers/analysts across areas... there are too many position for what the market was ready and companies endup to close the positions with anyone from the street because if they don't they loose the headcount for the project (welcome to enterprise world). Try to hire 10 good people in the area that mark all the point you wish and let me know how long you took to hire them.

3 - Even if you are ok on point 1 and 2, you often have to run parallel platforms, migrate things, fine tune etc... That takes time and resources, both human and material. Resources mean money flying out of the bank account and depending how complex the enviroment, until you manage to get everything done, quite easily is a 1 to 2 years project, best of options, most businesses rotate their management every year and that is highly disruptive for long term projects and often generates frictions in the business.

And I could point another number of reason, key thing from my comment is simple, big data is awesome, the same a ferrari or a bugati veron is, but do I need it?

Maybe a VW Golf does the job and I will spend way less money.

Business is all about budget efficiency, like it or not, so if you can't deliver results before 12 months.... most likely you are screwed. Remember that you leave in a world that finds normal to fire the founder of the company just because he doesn't care about immediate financial results and IT history is full of examples.

Bring the alcohol and the white boards and good luck :)

sdunga

Not as easy as it looks

Big data as a start is actually a mirage and that is the really reason why the companies that bet in this horse are having issues.

Most of the companies refered with closet solutions they don't have big data requirements, implementing very big solutions for several customers, the majority of them doesn't even gets DBs, or whatever may be bigger than 1TB and that isn't big data, just name them, Oracle, SAP, etc, etc.

Big data systems and concept actually where thought for a really small set of enteties that actually need that level of efficient, lets say google, big scientific projects from pharmaceitical companies, to CERN, ESA, NASA, etc.

The data that is really growing a lot in all companies are other type of sta, like email systems, share drives for PDFs, words, images.... For these purposes Big data concept is useless.

Additionally one think that doesn't make it so easy is often enough legal requirements, that for example multinational companies that are usually the ones with more data in the enterprise world, they have all type of issues, like finance documents to be stored for 10 years, in some countries travel expenses and other sensitive data from employees for example can't leave their country... this will mean that you can't actually centralized things in one system alone often, most likely several smaller implementations, hopefully all standardized. Finally but not least... containers, yes very good for apps and a bunch of stuff, but lets face it, finance, HR, CRM and generally ERPs will always be dominance of the big sharks and containers very often aren't a solution.

In resume, there aren't magical receipts, learn your environment 1st, check legal requirements and actually how much data you have to consider and few TBs these days are big data, and be ready to take commitments, there aren't perfect solutions, always something will be compromised, you just need to decide what is really important you and your company.