What's it like to work for a genius and Olympic archer who's mates with Richard Branson? Welcome back to On-Call, our Friday glimpse at readers' tales of being asked to fix the ridiculous. This week, reader “Alien8” shared a story about “One of the worst jobs I ever had” doing “IT support at a fulfilment house.” The big problem here was his boss, the IT manager, who Alien8 says just made stuff up all the time. …
It's actually a perfectly decent tool for the jobs it was designed to do. It's also capable of doing far more than almost anyone who ends up using it, which means without an understanding of ER, they mash the keyboard until they come up with something that looks pretty whilst having no concept of their data, and without even validating that the system does what it was intended to do.
I figured here we'd know better than to blame the software for the users limitations.
That aside, it's often a reasonable marker for something that needs starting all over again ('please add a 'show history' button, it must have rounded corners' - yep had that one on a system with no audit functionality), or something that a finance person hacked together, and has become business critical and now has to service multiple sites. Or they just link every single table from the real enterprise system and munge a load of inner joins for meaningless, but pretty reports (killing network performance as every record crosses the network multiple times per report/query, and for good measure, scatters a handful of locks in its path).
It's a shame, as it does what it was designed to do perfectly decently, especially in the late 90's when it ran Toyota's website for a while.
For what it was doing it wasn't a bad tool, it was the implementation of it that sucked.
It used an ODBC connection to create a faux SQL connection to take a dump of the entire fulfilment system. By the time it was taking 8 hours to run the database tables were hitting 2Gb each (this was around 2004 so was hitting the maximum size Access could handle at the time before corrupting everything). It quickly grew into 3 separate databases to hold the data and a 4th database to do the actual work, at which point it got handed to me. Turned out all the views were pulling everything from the ODBC connection, then pulling everything into a single query which was the filtered for the final report. I rewrote the ODBC view adding where clauses to only pull the data we actually required (resulting in a single database of a few hundred Mb) and then added where clauses to each view accessing that data so all the filtering was done before it hit the final report view.
From 8 hours it went down to 2, most of that now in the SQL of the ODBC connection.
I worked at a company that used an Access front end to a MSSQL backend for an order processing system.
Every couple of months the system was required to create about 5 million records in one table with sequential serial numbers for gift vouchers - an identity column was no use as there were four series of serials, for different denominations of vouchers.
The guy who created the system did this using loops in access - get the next serial number, create a record, add one to the serial number and repeat until the correct number of records had been created, then do the same again for each of the other three denominations.
This routine usually tool at least 12 hours to run, so was left to run overnight. Often it failed part way through. I replaced it with a SQL stored procedure that used a tally table (http://www.sqlservercentral.com/articles/T-SQL/67899/) - the new routine ran in 15 minutes
I saw it used in a Regional Health Authority (who constantly had financial troubles, but were handling $500million+ each year using this thing, which also held details on half a million patients) and to my eternal shame it got pulled into my company as the accounting system.
One of the company directors was "mates" with the "authors" of this monstrosity and rather than spending the "exorbitant" sum of $500 for a networked version of Quickbooks, insisted that we use the accounting system from hell on the basis that his "mate" saw it as a simple conversion from the existing larger setup (I was outvoted by the other director, who happened to be non-technical and his wife)
6 years later, after the company books were in a complete shambles (said director and his "mate" at the RHA were the only one able to understand the accounting system. Even Inland Revenue gave up trying to audit it) I finally managed to wrest the system out of his control and spent 6 weeks feeding everything into.... Quickbooks (with the aid of a couple of friends).
We very quickly established that rather than turning a profit every year, the company had been making a staggering loss and to cap it off had been trading insolvent for the previous 18 months.
Time to call in the receiver - who was very understanding and had seen this kind of thing several times before, but that didn't make it any less stressful.
Lesson: When someone pulls this kind of stunt, resign or get as far away from the financials as possible. When the inevitable happens, shit will fall on you (as a director) from a great height even if you're not involved in the money stuff, but at least you won't run the risk of a nervous breakdown trying to cope with it all.
Yes. Everyone knows that Excel is all the database anyone would ever need.
Oh God! The flashbacks! The nested ifs and vlookups. The brackets! The horror!
...whimper...
I remember the day I took over a "database" responsible for running a £20m budget. On Excel. It took me about 2 weeks with a whiteboard and a lot of paper and highlighter pen ink to work out what it did. A day to change the colour scheme from purple on brown, black on hideous flourescent green and the like. My eyes! Another 2 weeks to re-write it so it actually worked properly. Another week to test that I hadn't screwed it up. A week to put the data back in correctly. And a few weeks to fail to persuade my boss to buy some proper software to do it, or to get someone competent to make it into a proper database, using something a touch more robust than Excel.
I think it was a 200MB file, extensively linked to 3 other equally huge Excel abominations - and I know that some other people linked their stuff to it across the network. It was the sort of file you hit open on, then wandered off to make a coffee. Back in the 90s, when 1GB of RAM was something that happened to other people.
It worked though, but it scared me. Infernal audit were called in, and after two weeks pronounced that they couldn't find a single mistake. I suspect they just couldn't untangle the spaghetti and were bluffing...
"I remember the day I took over a "database" responsible for running a £20m budget. On Excel."
My first "IT" job. I was a trainee engineer for a semiconductor firm. The engineering budget was somewhere in the region of £40M per day and I was given the budget variance report to run. It was an Excel spreadsheet that ran on an old 486 laptop in Excel 5 and took 20 minutes to run. The report was run at exactly 10:30 every morning (ready for the 11:30 engineering meeting) which just happened to coincide with the day shift's breakfast break (so tea/coffee and full english was consumed while the report ran).
Unfortunately for me I took over in the first week of April which meant having to rewrite the report for the new financial year. This being the first Excel report I'd ever used. And most of it being VBA driven.
Somehow I managed to decode 4 pages of VBA code over the course of a weekend and got the report out without any mishap on my first full day on the job.
I also worked at a company (~£10M turnover) where the head of finance had a 2GB Access (plus numerous copies for roll-back) 'database' that pulled in data from SQL so he could then run his own reports (via Excel of course).
Eventually for equally stupid reasons they moved to using SAP, which in the circumstances was as overpowered as the Access system was underpowered, but it did at least include a budget for VMWare and a SAN which I enjoyed spending.
Impressive. I'm also an Engineer, and have had similar experiences. Isn't it interesting how us Engineers like nothing more than working weekends, and evenings, for free (or rather, just the buzz of solving a problem, which, of course, is priceless)? I don't encounter many admin people who say 'no problem. I'll take it home and have it sorted by Monday morning'.
Yes, I am a Grumpy Old Engineer.
Some years ago, I was working in one of about a dozen different teams which were improving Information Assurance throughout a UK department of state. In order to track the performance of these teams, and hence the Department, the central organization devised a monitoring tool, which they were pleased to call a 'dashboard', implemented in ... Excel.
So, the teams sent in their performance measures to the centre, where they were entered centrally into the spreadsheet, which was then published.
So far, so good, and this went on swimmingly for many months, until one of my team members looked hard at the formulae underlying the pretty graphs and pie charts... It turned out that at some point in the dashboard's history [1] somebody had inserted a row into a "table" and put all the values off by one, so that reported values for target X were contributing to the charts for target X+1. Executive summary: Borked and meaningless. And this for a product that was meant to be tracking Information Assurance!
Irony overload, you might think, but that would leave you nowhere to go when you heard about the response from the central Information Assurance team. They acknowledged the fault, but declined to fix it, because "it would make the previous reports look different, and they had already been published to the Secretary of State".
If there's a moral, it's to have training for Excel operators in the use of the rather excellent but (IME) underused Auditing Toolbar, and then to audit its use!
[1] version control? No, that would have been a good idea, wouldn't it?
How soon they forget...
"How The London Whale Debacle Is Partly The Result Of An Error Using Excel"
http://www.businessinsider.com/excel-partly-to-blame-for-trading-loss-2013-2
"Over at The Baseline Scenario, law professor James Kwak, says that what has been generally under-reported about the London Whale debacle is how badly Excel failed as a financial modeling program."
And yet in all my work in the US and UK financial system (not my primary focus, but substantial and ongoing) Excel remains the primary tool for most levels of financial analysis, reporting, and modeling - before you jump to the massive back box applications like Hyperion.
Every project has the absolute requirement to manage tens of thousands of large (20MB+) excel spreadsheets which are all linked in a gordian knot of bullshit.
Equally scary is the fact that when "real" databases are used they are all managed by the lowest possible cost offshore teams who have 100% turnover every 3-6 months. All have full dbo / sys access from across the seas, so goodness only knows where the data ends up.
The more industries I work in, the more I realize that the entire economic system of the developed world is a house of cards.
I once worked for a director who used a spreadsheet as a word processor. Literally, just typed memos or whatever in at one line per cell. And complained about how much work it was to manually re-word wrap everything if she had to make a change. She thought we should write a macro or something to do that for her.
And no, we never did get her to use Word, although she did eventually graduate to writing everything in her email client.
Reminds me of the story of the bloke who wrote (part of) Windows 95. Article was on here a few years ago but I've never been able to find it since.
He started at MS back in the old Windows 3 days and for fun created a 6 line program in MS Basic. And then couldn't understand why it wouldn't work.
New developer "It won't work"
Boss "It's a bug in the code"
ND "It's 6 lines"
B "Not your code, in MS Basic"
6 months later he's debugged MS Basic and fixed the bug, calls everyone in to show them and then exclaims "What kind of idiot writes a piece of shit like this?"
Bill walks out the office and his boss tells him "That idiot was Bill". Amazingly he wasn't fired, which was lucky as he fixed the memory issue with Windows 95 iirc.
"what a steaming pile of manure the "eMailer" was, are you?"
I wasn't, but I told the NZ Sugar wannabe that it was (they sold it there as the "iPhone" a decade before Apple thought up the name), especially given they wanted ISPs to stump up 20k for support software that would only work on SunOs 4.1.3 (hardly justifiable, given the poor user experience it offered with a price tag approaching a week's salary. IIRC I called it a "very slow, extremely expensive shiny toy which can't even display web pages properly")
They weren't happy and harrumphed a lot. I eventually saw the thing on sale in Dick Smith Electronics where it sold for several years through the 1990s - right up to the point where they all stopped working one night.
It turned out that Sugar-wannabe had managed to get the demonstrator devices running by installing the control software on a sparcstation in a student lab at Victoria University (unauthorised). One day the sparcstation failed to switch on and ended up in a skip shortly afterwards as it was a decade old.
Yup, all those commercially sold units (a few thousand) were using the software on that Sparc, not something purpose-installed by some gullible ISP.
Yup, all the user data was held on the sparcstation - without any backups.
Users weren't happy at losing their mail and other data.
Dick Smith Electronics wasn't happy, as they were court-ordered to 100% refund all buyers and Sugar-wannabe had slung his hook, so they ate the costs.
The fate of the drives in the sparc remain unknown - they weren't in the thing when it was recovered from the skip so it's unknown if they were erased or not.
Mrs. Archer once described her husband Jeffrey as suffering from "narrative inaccuracy". Presumably this guy was a Lord Archer, but on an Olympic scale of telling porkies.
Incidentally, Spike Milligan describes a similar person in his war diaries. That ended much better because when the shit hit the fan the senior officers told the storyteller that he was delusional.
Well, his IT manager should have become a politician instead.
That would have gotten him to use his innate ability to adlib lies wholesale while arguing about weapons of mass destruction in the commons, lying about supporting terrorism and intentionally destabilizing whole regions so that tens of millions go homeless, are murdered or die on the long "trek of the exiles".
Any likelihood to any characters known or imaginary is purely coincidental. Me coat.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
