IT god exposed as false idol by quirks of Java – until he laid his hands on the server Sometimes, IT professionals can appear as gods to users. Sometimes their mere presence can cause problems to miraculously disappear. In today's Java-based tale, a reader recalls the all-too-brief moment when he became a database deity. Welcome to Who, Me? Our story comes from "Bob" (not his name) and is set amid the fallout …
One Java product I supported ran on sun / oracle Xeon boxes with 64GB+ of ram. When the red hat based os was 32 bit the Java app could only support 4GB and there was a config file that ensured no more than 4GB was allocated.
That file persisted to the 64bit implementations, upping the Java memory to use more of the available ram became the go to troubleshooting step, yes I got them to amend the file in later versions of the product.
In the days of spinning disks, we had a small team of superior beings that descended from Olympus itself. Each knew just enough about the others responsibilities that a sick day or vacation wasn't an issue. The department ran efficiently and under budget, most years anyway. We ran proprietary Linux software solutions, developed in-house, that drove company production, orders, reporting and updated accounts flawlessly. Think ERP levels of integration at 1/20th the cost. We broke the 100 million gross income barrier with that system when efficiencies allowed us to take on an additional contract worth 14 million. Glory days.
After our senior hands-on VP retired, Finance took over the department and things went south fast. Accountants, with IT experience limited to Excel on Mac, began telling us how things worked "in the real world." We lost people quickly and I was the last to go of the original bunch. Ten years later, I popped in to say "Hi" to a few folks. The IT offices had been broken down and the server space gone. with everything outsourced across a 50MB Internet connection (and 180 employees using it daily). Found a single PFY IT consultant playing Tetris and waiting for something to break. The network was slow (The IT consultants were adding 5 port 10MB hubs on hubs on hubs to expand the network.) Production had been capped at below the levels we had originally established, because the new shiny ERP solution they bought was not utilized properly. The accountants got it working then stopped optimizing as it would take them over budget. The staff were not happy, but finance was. They were saving thousands at the cost of losing millions in new business they simply couldn't handle. The CEO, who was CFO during my tenure and made the dumb calls, was retiring under a cloud and no one wanted to take over their position. I heard he made 300K in bonuses his last year.
Penny wise, Pound foolish.
A few hands-on expert Gods into making trillions is much nicer, Clausewitz 4.0.
Are worlds now ready for that and/or those .... although try to stop them at your peril for they are extremely dangerous and highly volatile and do not suffer the actions of useless fools resultant from the machinations of useful tools.
The programmers aren't any good, but better ones “are too expensive”. How do we fix it? Let's change the language and put them in a baby cage. That'll do! Did it?
Most problems can be fixed by hanging the management from trees. Unfortunately, the management does not approve simple solutions.
Brutal reality is that very few programmers are actually fully informed and good at what they do.
You could arrogantly assume you only hire those, or you could find a safe playpen for the people you do hire so that they can't hurt themselves so easily.
Good management assesses the risks, applies a level of self awareness about their recruiting practices and salary offers, and playpen it is.
My guess - disabling the page file so that none of the pages can be swapped out.
Modern Unix/Linux systems allow you to lock pages into memory so that they cannot be swapped out, but I don't think that Solaris/SunOS had that capability way back then (but I could be wrong - its a long, long while since I've done any work on Solaris/SunOS).
Sort of, but what you would call a memory leak in C or C++ isn't really relevant in Java, precisely because of the garbage collector. The programmer isn't responsible for de-allocating the memory that was allocated to an object when it goes out of scope, as they would have been in C or C++. Programming is therefore done in a style which would cause big memory leaks in C or C++. The garbage collector is what frees this memory.
You can still allocate memory in Java that the garbage collector can't touch, but it's also no longer used for anything worthwhile. For instance, badly written ThreadLocal code can pollute a web server's worker threads with useless data. Certain hot-reload solutions could leave previous versions of class definitions in memory as well (at least prior to the introduction of metaspace, I'm not sure how often that happens nowadays). I would still call these memory leaks, even if the programmer is not responsible for directly deallocating memory in Java.
This is how default GC used to work before we switched to G1 - it would keep growing memory used until it gets to the max and then do a large GC pause.
If the application had more memory than it needed the heap usage graph would look like you have a memory leak - it would grow for a day (but then go down after scaring you).
G1 is collecting all the time so the graph looks like a series of small peaks.
If this was relatively early days for Java (which it sounds like, from the way the story was written) then the fancy garbage collection algorithms might not have been available yet, perhaps? The "Concurrent Mark Sweep" collector arrived around 2002-2004ish and the "G1" collector landed around mid 2009-ish.
It does sound a lot like someone gave a value for the '-Xmx' argument that included the swap size instead of correctly giving a value that would fit comfortably inside physical RAM.
This post has been deleted by its author
GC languages just get rid of the trivial cases. Getting rid of trivial cases is nice for productivity, and I do prefer using a GC language for that reason.
However, many assume that memory management in GC languages is easier. That's a bad assumption, because trivial cases are never where the difficulty is. I find I get memory leaks about as often in my .NET projects as I do in my C++ projects. They are just different. Maybe in one case, it turns out a dialog box subscribes to an event in the main window and never unsubscribes; in another, maybe I moved a block around and an early return is now before the free() call.
The "-Xmx" flag set the GC max heap size. Like e.g. "java foo.jar -Xmx 4096M" will tell it to avoid making the GC heap bigger than 4GB. You always want to configure this to be a bit less than the memory in the box, because you want to leave some RAM left over for the operating system kernel, filesystem cache, TCP buffers and so on.
If you configure it to make the JVM heap bigger than physical memory then the heap will extend into swap and then the above noted performance disaster happens when the GC needs to access the parts of the heap that are in the swap file. On mid/early 00s era hardware, touching a swap file page might take about 10-20 milliseconds.
Java will usually use only slightly more memory than the GC heap. Barring some edge cases, such as if you use FFI to call malloc() and have memory allocations outside the GC heap, or occasionally JVM bugs. :)
IANAJP (I am not a java programmer) but logic dictates that if you have a server dedicated to a single task, then you want to do as @RichardBarrell says but first load the server with all the admin tools you are likely to need (possibly including a simple mail server to message the admin in case of problems.) Check how much memory remains (without swap) and size -Xmx as appropriate, leaving an extra gig or so for some other tool you forgot. Turning off swap as indicated further above without doing this will result in the dreaded OOMkiller killiing off the very tools you'll need to find out what went wrong, and possibly lead to an even more dramatic panic.
Much as we all hate Larry's Corp, there is a useful article at oracle.com/technical-resources/articles/it-infrastructure/dev-oom-killer.html . Some may not be relevant because of systemd, but it is a good place to start.
unless he wasn't being serious. Why would you not care about people being canned?
""Never once did we spill a tear as another admin person was delivered a pink slip," boasted Bob. "No, these were demonstrations of just how mighty we were. We were the gods of efficiency.""
Having spent several years implementing a certain HR system it seemed only fair and natural justice to be getting revenge on those b-----ds.
OTOH I've been on a materials management project where due to the amount of up-front data capture needed in Goods In we actually needed more admins in that team, who were given more time to process each shipment and still saved time and effort overall - and freed up a lot of other people to do the manufacturing stuff they liked instead of the parts management that was universally hated. A rare win-win.
My takeaway from this was that when "efficiency drives" are pushed by finance or HR, they never look at putting their own house in order first.
And of course we liberated the occaisional really good admin to become trainers or even implementation consultants themselves. Thus completing the circle.
My job sometimes involves optimising business activities in a way that reduces the number of people needed.
It's very easy not to care: If they're any good, we can employ them in another role or they'll find a new one easily enough. If they're not any good, we don't want them anyway. On top of that, good staff want to do good work, not sit around bored, so if there's no work for them, we're hurting them by keeping them on.
Most of my work is about reducing the rate at which we need to bring in new staff, rather than sacking any, but it's business not charity. People need to contribute, and those that would turn up for their wage and not care aren't welcome.
I think you will find a problem with " People need to contribute, and those that would turn up for their wage and not care aren't welcome."
A lot of jobs are not interesting / exciting but people do them because they need to pay their bills.
Virtually nobody will get excited about flipping burgers as an obvious example.
If you think people are excited / enthusiastic / caring about low wage, tedious jobs, then you have been hoodwinked (as a key skill in those jobs is to pretend to be motivated & enthusiastic when people of authority are around)
.. and if a job is interesting, all the motivation in the world does not make somebody good (and the converse is true)
"On top of that, good staff want to do good work, not sit around bored, so if there's no work for them, we're hurting them by keeping them on."
.. Ever hear of burnout?
Most staff benefit from a bit of down time - and the good ones will not be bored but work on side projects that often benefit the company.
There may be lots wrong with Google, but their giving employees formalized time for their side projects is one of the few things they can be praised for.
As a sysadmin, my career is the pursuit of efficiency: lower costs for better outcomes. My job, and those of my coworkers, are major costs and will be sacrificed. Survivors adapt to change. High performers eliminate their own jobs, then justify taking someone else's. There is satisfaction in performing well.
It's still less corrosive than sales, where every coworker is a rival.
Improve amount of work a staff member can do - better software = quicker working.
Company 1 - make someone redundant
Company 2 - make spare person a checker - catch errors in orders
Company 3 - ride the improvements to get more work
Often 2 becomes 3
1s are a waste of time
have you forgotten the four basic unspoken rules?
- programmers only look at and think about code...
- technicians only look at and think about hardware...
- if hardware is the problem, don't expect a programmer to fix it unless you are looking for them to (blindly) (try to) code around the problematic hardware...
- if a program is the problem, don't expect a technician to fix it unless you are looking for them to (blindly) (try to) add faster CPU/GPU, more memory, and/or more storage...
there are more but these are the main ones that form the base everything else is built on...
> have you forgotten the four basic unspoken rules?
You've forgotten the most basic rule underlying all of these:
"It worked on my dev and/or test platform"
The problem tends to be that systems generally are built and tested using relatively small data-sets. And as data is poured in on live, things can often slow down.
Worse, the impact can often be exponential, especially if you have developers fond of using heavy levels of abstraction, since each layer of abstraction will consume a chunk of resources - and may lock things at the data-level, to boot!
> And no one had thought to look at the server performance stats before then?
Once upon a time, there was a web-based system provided by a major telecomms company, to a major financial company, which had little offices dotted around the world, each of which was hooked into this system via relatively slow wired connections.
And there were a lot of complaints about the performance of said system.
I wasn't actually involved in that project in a technical role, but one day, when looking at something, I fired up the browser devtools, and realised two things
First, the pages often included drop-downs which had hundreds if not thousands of options in them. And these lists of options were generally built from templates which included a lot of whitespace - to the tune of hundreds of extra characters per option. And this whitespace was invisible because when rendered as HTML, the browser just ignored them.
However, all this extra whitespace massively bulked out the size of the page - often from a few hundred kilobytes to several megabytes.
Secondly, the web servers didn't have any compression options turned on. Which meant that these megabytes of spurious whitespace were choking up the relatively low-bandwidth connections.
So, in the first instance, turning on mod_compress (or whatever the equivalent was at the time), shrunk the amount of data being transferred by about 85%, even with the whitespace still being in situ.
And then, stripping the whitespace from the templates reduced amount of data being delivered, further reducing the data-transfer sizes and reducing the amount of CPU and memory being consumed by the browser.
As such, we went from dealing with 5+mb per page-load to ~30kb of data being transferred, which then decompressed to about 150kb within the browser. Which may not have fixed all of the performance issues, but did make a significant difference!
However, if I hadn't stumbled over this by accident, the issue would have persisted for a lot longer!
Why would I seek to be a "monster" when I've already done the 50-60 hour grind in my youth?
You can keep your "monster" jobs; I'll settle for the *part* *time* work I do nowadays to pay the bills without chasing fame, glory, or fancy anything any more. Chasing "things" cost me a dear amount of money over the decades that ended up being worth nothing because it was all obsolete and got tossed, including a CAD$60,000 CD and DVD collection that has been replaced by a cigarette-pack sized external drive of their content.
Besides, no matter how much I learn nowadays, it will not change the fact I forgot 9 times as much as I know. The same is true of anyone in the tech industry for any length of time.
Things grow old fast. It is rare that you can ride a technology "wave" more than 5-10 years; only once or twice was I able to stretch a tool/technology longer than that before there was no market for people with skills in it.
My *nix skills, however, go all the way back to BSD 4.2 and AT&T SVR-4 (genuine, not a vendor adapted version.;) That is the one thing that has retained its value.
The same goes for laptops. Once upon a time, slowdowns and freezes could be diagnosed as swap hammer by the frantic twittering of the hard drive. Now, with a SSD, nada.
That wouldn't be so bad, but the modern fashion is to denude latops of all external lights except the manufacturer's logo, so there's not even an access light to clue you in!
Heat is the new clue to a laptop in difficulty. Either the fan going bananas or the keyboard getting noticeably warm, especially true if it really is on your lap, or you pick it up. I once had a Dell with chronic overheating, so much so that it would shutdown every hour or so when it tripped the thermal protection. Very embarrassing when it started happening on a customer gig, and I had to ask them if I could use the freezer in their kitchen area for 20 minutes before I could get back to work.
One of the things I particularly like about the Erlang runtime is the way it does garbage collection. A large scale Erlang application can easily have thousands, or tens of thousands of separate 'processes'. They are typically small. Each Erlang process is responsible for its own heap memory; there is zero sharing of memory between processes. When a process needs to GC, it does it, and without interfering in what any other process is doing. So your app never grinds to a halt for the One Big GC.
I've programmed Erlang. Sure it can handle threads like crazy - because it is a functional language that allows no variable manipulation, makes the simplest of calculations into list traversals, and only supports float, int, and string types.
i.e. If you want to hamstring your creativity with some telco's perverse idea of how to program a switch, have at it.
Don't even DREAM of using it for anything else. Run screaming from any idot who suggests it.
I spent three years in Erlang hell. There are many things it flat out can NOT do; and the last of them killed the project. All thanks to some predecessor who decided that because HE loved Erlang, he was going to make a "corporate direction" decision.
I managed an inspirational moment once - and it was a mechanical part of the system I was working on! I was developing software to process ground-sounding radar data in real-time (in the late 80s - transputers were the big new thing!). The antenna had a complex system to scan it sideways behind a moving vehicle, and the mechanical engineer had devised a nice system using stepping motors etc. Suddenly, he was made aware that the antenna was several times heavier than expected, and the system with all the motors was going to be too heavy to hang off the back of a van. All hands on deck time! But I sat down and worked out the movements of the linkage, and realized that two consecutive joints simply rotated by the same amount in the opposite direction, and a simple belt running between the two would achieve the same result as a clever system of motors. This reduced the weight sufficiently that all was well. I felt very clever for a while!
I remember eating humble pie when I was in support. A customer had a major problem. It could only be caused by a major setup problem with the customer's system, or code like
X = 0;
If (x> 0) then cause problem;
else x =1;
I told the customer that I staked my reputation that the code was good, I had spent hours looking at the small number of lines of code it was impossible for the value to change ... etc , it must be the customer's fault.
Someone overheard the call to the customer and came round for a chat.
The chat was along the lines of ... if you have two threads running this code - what happens?
We took a white board and crawled through the lines of code until there was an ah -ha moment. If thread 1 got here, and thread 2 got there.... then there is the problem.
We developed a fix which worked, and I sent them a picture of me with a venison pie from the local supermarket ( and an explanation of 'umble pie)
In future I used words like "very unlikely" rather than "impossible"
X = 0;If (x> 0) then cause problem;
else x =1;
Did you notice that you're testing an uninitialised variable and not using the one you did initialise.
Sorry, I know it's just a mis-typed example but these things jump out at me from years of troubleshooting code!
"I've also fed paper tape to a PDP-11 to boot it"
You were spoilt! One of my first close encounters with a computer was with a PDP-8 (admittedly already long in the tooth then). Before you could load a program with the paper tape reader you first had to enter a boot program using 12 toggle switches on the front panel, inputting one 12-bit word at a time. Though the system did have a light pen so you could play a moon lander game with it.
Usually the thrashing -- human or machine -- is caused by something obvious that the developers have never met before. The last time I achieved deification in the workplace was when the applications developers had run into a bug communicating with the firmware in a new piece of equipment. I wasn't concerned in that development so I was only dimly aware of the weeks long kerfuffle. It eventually got resolved (or more accurately, worked around). Shortly afterwards one of the developers asked me casually whether I had any idea why their block oriented file transfer mechanism should always lock up when the size of the transfer was a particular size. The answer was obvious; the transfer was broken into blocks and eventually they'd hit a file size that was an exact multiple of the block size, leading to a zero length data block. The recipient of the transfer would be decrrementing the block byte count towards zero, not a smart move if its starting with zero. Its actually a rookie mistake, but then most of these problems are. I'm used to this kind of problem so I was able to give an immediate answer resulting in my deification, at least temporarily until people remembered that "There's nothing new under the sun -- if you're stuck on a problem ask about, there's always someone out there who's met it before".
As for garbage collection, heaps and page files, all applications programmers know that these are zero cost system components that always work flawlessly. (Same with system cleanup after an abnormal program termination -- all resources used by that program are flawlessly identified and returned to where they belong.)
Seen it many times. Server get slower and slower, fails, recovers to full speed, then the cycle repeats. It's often caused by that one coder that isn't explicitly closing external resources - files, database connections, HTTP responses, etc. They run out and the server dies...but the error messages use just enough memory to invoke GC and it magically recovers. The knee-jerk reaction to add memory makes it much worse.
Not Java, but 'C++'. Had a single customer who had things blowing up with a divide by zero error in a single function. No idea how they managed to provoke this, and never will, 'cos fixed it with a simple test for the guilty parameter being zero, and a quick return to caller with a suitable return code.
No-one else ever reported anything of the kind, and to this day, I have no idea which combination of circumstances might cause this. Anyway, the problem went away and we, and the customer, are happy. Sometimes you just have to take the easy win when you can.
When my trouser waist band was about 6" smaller, SQL databases were very much the thing. Oh how we laughed when we found that that the OS (with swap file?) needed to be on its own smallish disk; the database was on a separate disk, and had been pre-allocated its own section covering most of the disk; the transaction/log file was on its own smallish and fast/expensive disk; and RAID was just becoming a thing; we generally got this stuff right (even if you were using Oracle). Structural integrity mattered, and almost everything involving data manipulation was "correctly" done by the database - Things "usually worked" (tm).
Then programmers got involved - Client Server and three tier systems - Dividing the presentation, application logic and data processing layers across client and server devices. Maybe it's a coincidence, but isn't that when everything started going to hell in a rapidly descending handcart? I may have been one of the guilty for a short while by incorporating C into stuff, to avoid some Oracle licencing fees (Reports and Forms!), but I started with FORTRAN, then learnt, Dartmouth Basic, SQL, and only came to C later - So I was a hopeless case anyway.
Haven't we just replaced this stuff with 2 apparent layers - The web and the data? Sorry folks - Don't mess around with important data with Java, JavaScript, and your 10 favourite frameworks, just learn SQL like civilized people... :-)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
