CEO insisted his email was on server that had been offline for years Welcome again to On-Call, The Register’s Friday column in which readers share tales of tricky tech support tasks. This week meet “Sam” who “used to work for one of those zombie web companies that limped along after the dotcom bubble burst because no-one had told it that it should be dead.” The company no longer had a proper …
I work somewhere like the authors org.
I'd guess that there's one business unit that has a solid underpinnings that keeps the rest of the business afloat and isn't entirely reliant on little things like functional IT infrastucture. As such, you don't need developers, network engineers, sysops etc - I mean, the part that makes money never failed, so anything else is just a cost.
Anon, because you know why.
I mean, the part that makes money never failed, so anything else is just a cost.
And this is something that's bugged me for most of my working life - with very few exceptions[1] IT is seen a net loss in financial terms rather than a cost-of-business expense.
I mean, it's all very well having sales droids in shiny suits going out pressing the flesh, but if it wasn't for IT, they wouldn't be able to sell anything!
Strangely enough, very few sales types seem to see it in those terms. Or accountants. Or Finance Directors..
[1] And all the exceptions seem to have been companies whose main product is IT.
with very few exceptions[1] IT is seen a net loss in financial terms
I worked in local radio in the 1990s as an "engineer". I ended up "doing IT" because almost nobody else knew where the "on" buttons were, but bear in mind that when I started the entire IT assets of this 35-ish employee company consisted of one '286 running DOS and WordPerfect, connected to a Laserjet, one '386DX with Windows 3.0 for the boss's secretary (and her Laserjet) a '386SX that my boss managed to budget for himself, and a multi-user minicomputer thingy running half a dozen VT52 terminals for booking.
Anwyay, besides the point. Just before I started that job my boss and my predecessor had (apparently) taken part in a company-wide transition to a "cost centre" format. Each department cross-charged each other department.
The way my boss told it, the only department in the entire company that made a profit was engineering, and the experiment was quietly dropped :-)
M.
I have a conscious and worry about some things that aren't done right, risks taken and corners cut.
Then I visit someone else's IT function and I don't feel so bad ..... :)
Properly done IT is actually quite expensive and entrepreneur types like to get by on less money by taking calculated risks. We do our best with what we have, but on a wing and a prayer sometimes.
We're still running a Laserjet here at home, a 6MP. It is connected to a JetDirect box which talks nicely to the router, via enet and it is then on the LAN enabling me to print wirelessly from Mac laptops and iMac to the wife's Dell but not her work one because she isn't allowed to install the drivers . . .
We've had it for decades and it seems pretty bulletproof. Bought second hand originally too. Absolute value.
Via of course the magic of CUPS and Gutenprint as HP no longer supports the native driver. Sometimes I absolutely adore the interwebnets and the geeks who produce things like Gutenprint as a service for others.
And this is something that's bugged me for most of my working life - with very few exceptions[1] IT is seen a net loss in financial terms rather than a cost-of-business expense.
[1] And all the exceptions seem to have been companies whose main product is IT.
So what's IBM's excuse?
"And this is something that's bugged me for most of my working life - with very few exceptions[1] IT is seen a net loss in financial terms rather than a cost-of-business expense."
Yes indeed. Worked at a manufacturer that completely saw IT as a cost. In their viewpoint - the Plants made product and sales droids sold product. IT was just a necessary evil that cost a lot of money. While that might have still been true as recently as the mid 90's when you could fall back to paper, but fast forward to today... The entire company would collapse during any outage in the data center, no matter how short. Trucks couldn't leave the plant with product, payroll couldn't process, vendors weren't paid, health benefits weren't managed, etc etc. There were a number of times I (as IT) had to scramble to fix those lowly processes like payroll that nobody thought about, but everyone likes the money in their account for some reason. :) Where things are today - no IT, no ability to do business.
--- Data transfers going in and out of the Database servers is the centre of their universe (and every other company anymore).
* Bills of Lading, Metallurgical reports, etc etc for product coming in/and out of plants
* Invoices coming in and out
* Bank Transfers
* Health Insurance claims
* Payroll
* Export and import documentation
* and the list goes on...
The actual 4 critical departments in this instance for day to day operation? Mfg Plants (who make the widgets for sale), Accounting, HR and IT. Everyone ELSE is effectively overhead (ala - optional if the above doesn't happen) all the way to the C-Suite. The problem is most of the stuff going on behind the scenes that gets them a paycheck is f'ing magic and few of the corp drones (including the regular IT staff that did desktop support) gets into how their sausage is made. :) Lot of effort goes into making systems *look* really simple while also being reliable.
HR day to day?
I didn't know they were active quite a bit until I managed their automated data transfers. Yes, (in the US anyway) someone has to process the healthcare stuff, approvals, and such which require more care and feeding than just monthly or biweekly. Presumably you like your dentist or doctor being paid so they continue to welcome you as a patient, or your coverage actually being in place so the insurance card is valid? :) Another one of those things that most people don't know that is going on behind the scenes.... They weren't just sitting on their butts when it wasn't payroll time. :)
Ahh yes the Company Boss that happily shows off the new Company Jaguar saloon he gave himself to drive the four miles to work and then got into a fainting fit when you tell him he needs to spend £600 to replace the 10 year old Vista heap that the business runs off mostly.
"Have you got something second hand or cheaper?"
Trend change (maybe!)
The trend might just might be changing when it comes to paying attention to IT. There seems to be a dearth of Techies coming through and the costs in replacing a Techy is increasingly expensive.
I'm just about to leave one job and they are offering me a pay rise. My thoughts are why am I not on this salary in the first place, so I wont be accepting it.
I should feign at my modesty but I'm not going to since I know I can do a good job, willing to learn on the job, and put in a little extra effort to get things done. I work beside a mix of Techies i.e. some are only turning up for their pensions and other are there to do a good job.
Management have introduced a retention bonus but that's only for Microsoft tech. I use Oracle and our team was not included but management are well aware how disgruntled the team is.
Its an unfortunate situation that people have to leave before other people realise their worth.
As I say that trend might just be changing and the other Oracle developers might be getting a retention bonus.
I'm just about to leave one job and they are offering me a pay rise. My thoughts are why am I not on this salary in the first place, so I wont be accepting it.
I learned back in the '70's to never, ever accept a counter offer from your employer no matter how good it is. They will use it as a wedge to treat you unmercifully and also put you at the top of the layoff list. I wasn't the one who accepted the counter offer, it was friend and within two years, he was not his old self but rather beaten down.
the other classic is staying up all night to get a fix/patch/install but being denied a late start because its company policy.
That was the final time i worked out of hours on something that couldnt be prevented,the bosses soon started to ask me why there was a planned server outage during the working week and I happily brought up all the times I had requested funds for a redundant clustered set of servers so we could do rolling updates with no downtime (testing be damned, a lab setup was a pipedream of accounting).
Or accept the counter offer, but if you stay a while be aware that future yearly increment / performance raises will be less than what may otherwise have been offered, until you reach the point where 2/3% yearlies would've beaten the one off 10% a few years ago to stay.
Same with R&D. Their costs were just losses to a company where I was in R&D. The salesmen were the golden children getting huge bonuses for selling what we created. Eventually, they cut expenses, including R&D, to increase the bottom line. I left for greener pastures. It only took a couple of years before there was no bottom line. No one wanted to buy outdated, obsolete products, and the company folded.
This is why a good CEO is worth big bucks.
Some folks seem to want to keep every email they've ever received, for ever, just in case. Others (mia culpe) hate having cluttered computers and delete them asap - though sometimes it means not having the email you do need. I had to force myself to be disciplined and keep some where I might need them for evidence/reference later. .But also to have filing rules that put the blessed things into appropriate folders. The e-mail hoarders, at least the ones I had to deal with, just let them lie where they fell.In a very long in-box folder.
I am that hoarder, but it's not because I want every email, it's just that I can't be bothered to sort them out. I have two types of email, read one and unread ones.
I am thinking I should really delete all my work ones (>10 years) as I was once told at an Oil & Gas firm that we should delete everything after 3 years of the project ending so that our own email records couldn't be used to sue us in the future.
I am a hoarder too, not for OCD reasons but because I have a lousy memory. 2 years down the line I will be damn sure there was a reason why I did X, X being not the obvious way to do something. Someone will want to simplify my solution and I'll just know that will break something. Will I be able to remember? Not a hope.
I am a hoarder too, not for OCD reasons but because I have a lousy memory. 2 years down the line I will be damn sure there was a reason why I did X, X being not the obvious way to do something. Someone will want to simplify my solution and I'll just know that will break something. Will I be able to remember? Not a hope
Surely that's all written down in your documentation? Yes, I've put a quid in the swear jar.
On the one hand, it feels a bit lazy and untidy. On the other, falling storage costs have outstripped the time/effort required to be more proactive about it.
I seem to recall Facebook never actually bother deleting photos or videos - they just de-index them from your profile because the cost/effort of purging a given object from their various storage systems and caches is greater than just orphaning them on the disk until that disk eventually fails and gets cycled out (they may also be keeping it shadow-linked for analytics, but that's another discussion). The hyperscalers are be applying such logic to media, and the same applies to e-mail for the rest of us.
Surely that's all written down in your documentation? Yes, I've put a quid in the swear jar.
Wherever possible yes. The most common instance where it is not possible is on a drawing, there's no change history on those[1]. Sometimes a document referenced on the drawing closes the loophole but not always.
[1] By which I mean a detailed list of changes in each issue with reference to the requirement for each change. Documents can contain that but drawings just have a list of issue dates.
I've got every email I've ever received or sent. They are filed in a way that is meaningful to me, and yes the archive is encrypted[0]. Why? I dunno. It started back when the concept of email was still somewhat magical, and each one seemed important enough to keep. Now I just save 'em out of reflex. It's probably a pointless waste of disk space ... but then again, maybe a multiple-greats grandchild will find the archive an interesting view into my life.
[0] Should be fairly easy for anyone to decrypt any year now.
"I was once told at an Oil & Gas firm that we should delete everything after 3 years of the project ending so that our own email records couldn't be used to sue us in the future."
The alternative view is to keep them so that they can be used in your defence in the future.
The choice made tells you a lot about the company.
"The choice made tells you a lot about the company."
My employer recently moved to a 90-day email retention policy. Hmm. Thankfully I use the client's email system instead, and they have a much more reasonable 15 months policy. There's still a LOT of emails I save to disk for future reference.
Because all civil litigation attorneys act in good faith, and all civil jurors are sensible, level headed, and well able to evaluate evidence.
Do operable fire extinguishers at a company tell you that the company has good sense about things that might start fires, or bad?
I have every email I've ever sent or received, ever, back to... pre-2000 certainly, probably a few years before that if I dig around in my old Hotmail archives.
Because... well, email is so small compared to everything else that there's no reason to delete them. Additionally, having a HUGE stock of source data means that an email from an old friend won't be put into spam by my Bayesian-filtering mail client. Plus... wow, I mean, does it save your backside. "That's NOT what we agreed, and I can prove it", say I, with instant narrow-down searches of nearly two decades of email in fractions of a second.
In work, I make things work the same way. People tell me they are deleting their old email. Why? What are you hiding? Did you get the "you're running low on space" email? No? You know why? Because it's set ludicrously high - the point at which our 1000+ users would start to fill up the server's storage - and nobody ever reaches it despite having well-advertised, public addresses that thousands of customers and suppliers use to contact us all the time. I think in 4 years, that email went out twice, and both times it was "Yeah, you have a lot of email, but it's nothing in server terms. Don't worry about it." and I doubled their warning threshold.
Because literally "server storage for email" divided by "number of users" gives me a potential average mailbox size in the 10's or 100's or gigabytes. And pretty much Exchange says that the average mailbox is less than 2Gb. So I can afford for some heavy users because almost nobody else is using the storage available. Their roaming profile, however, could well be in the 10-50Gb size, even WITHOUT the documents folder (which is redirected to a network share).
My email retention policy? Don't delete email. No point. You're going out of your way to permanently remove data that might be useful to you for no real reason... And sucking down even a 10Gb mailbox to a new machine is going to take... hold on... 1Gbps network... 8Gbits in a Gbyte... 80 seconds? That's just not worth worrying about for a one-off only on the machines that you've NEVER used before.
And I don't worry about how users organise their own inbox. Why worry? It's up to them, hinders only them, and I have Powershell search tools that basically ignore folder structure anyway if I need to find something in them.
To be honest, I have EVERY mailbox ever made on my system still present too. 4 years, and I have 80+ users come and go every year. Why do I keep them? Because that involves zero effort and provides safety (users do sometimes return, sometimes we need to pull things from old mailboxes, etc.) and costs nothing. If I was getting tight on space, I'd just archive the oldest of them off the live systems. I still wouldn't delete them, though, just put them somewhere else. And we have backups and archives and retention policies and all the normal so even users deleting every email (rather than just throwing it into a folder to get it out of the root inbox) makes no practical difference to the size of the storage required for the mailboxes.
IT people whose systems are so underspecced, and who have so much time on their hands that they worry about a handful of old email for a few dozen users who keep everything? They really need to re-evaluate their systems and working practices. If you're on a budget, just archive mailboxes once a year and throw them on a cheap NAS, who cares? At least you'll still have everything.
But literally my users are instructed "Don't delete anything, there's no point. And especially not email, which is so tiny as to be pointless". You desperately need storage (unlikely, we don't quota either)? Then a video is the size of a thousand photos, which are each the size of a thousand documents. So long as you're not emailling around video directly (you idiots), don't worry about it.
"well, email is so small compared to everything else that there's no reason to delete them"
An interesting approach. I assume you have set your clients to not have local mail caching turned on then? I work in a "certain environment" where we are subject to DPAR and FOIA and, as such, email retention is a subject of some discourse. Our end users also use a range of devices to access mail and calendars, many of which are shared with support staff of various kinds. As such, we are caught between either providing email with no local caching, which can lead to asynchronous abberations (appt. in one calendar may not immediately appear in an instance of that calendar on another device, especially if its a sodding iPhone) or turning caching on in which case we're looking at ever expanding .OSTs and, ultimately, complaints that the device is slowing down over time (yeah, your profile is expanding...). Add bloody OneDrive/SharePoint synchronization into that mix and its fun and games all day.
These days, my approach is to try an persuade everyone to use o365 portal. That makes it M$'s problem.
every email I've ever sent or received, ever, back to... pre-2000 certainly
My home email server only has emails back to ~1998 - because that's when I had a major disc-crash and had to rebuild the server from scratch.
But I keep my inbox relatively small and archive/discard stuff every so often.
Just the other week my new boss told me that she hates email and would prefer everyone to talk to each other. Well she would wouldn't she? All the more reason to follow up any conversations with thoso who must be obeyed with a "Thank you the recent chat. Can you just confirm my understanding that ..... " type email.
My advice: never delete email and always back it up. I have been able to help myself out of tricky situations ie prove what was actually said too many times by emails that others would have deleted long ago.
Just say No.
When I worked in the cubicle farms before, one of my Friday afternoon rituals was reviewing what has to be done by somebody-more-capable, then pitching an email in that guy's direction so that he would have to double check what his tasks were for Monday. Chances are, if it's not in MY Inbox, it's on someone else's todo list early next week.
But literally my users are instructed "Don't delete anything, there's no point. And especially not email, which is so tiny as to be pointless".
And not forgetting, of course, that in many jurisdictions there is a corporate responsibility to retain all emails up to some age threshold whether the user has deleted them from their own mailbox or not.
Some folks seem to want to keep every email they've ever received, for ever, just in case. Others (mia culpe) hate having cluttered computers and delete them asap
There is a third category. My dear old Mum had the habit of using the Deleted Items folder as an online archive. Once an email had been read, she would delete it. When she needed to refer back to it, she would go into the deleted items folder and dig it out.
That sort of worked, until the mail client (I can't remember which one she was using - she insisted on having something slightly obscure, the same as she'd used on her office PC when she was working) decided to do some admin itself, and purged some older emails from her deleted items folder.
In the end, I solved the problem by hiding the deleted items folder from view, and creating a new regular folder called "Deleted".
About 20 years ago whilst working on an email migration we had one of our customer's senior staff doing the same thing - using the Deleted folder as an archive.
He wasn't best pleased when he encountered the empty Deleted folder of the new email client; cue one of those 'difficult' conversations explaining that using the Deleted folder as an archive probably wasn't the best idea...
"using the Deleted folder as an archive"
I've seen a contractor fired for "helpfully" emptying a user's Deleted Items without asking first.
Any sympathy I had for the guy was quickly quashed when I heard that, to avoid any blame, he'd told the upset user that he was me before leaving site. Twat.
Funny you should say that.
In my old firm, around 20 years ago, we were frantically busy over a weekend and trying to calibrate some equipment over and above the normal spec. It was like herding cats because as soon as one parameter was what we wanted another one would drift out.
The test gear output results on a continuous 'till roll'. My boss - whose temper could go nuclear in one second flat - had it happily spooling into a convenient waste basket below the bench, and took himself off for breakfast.
While he was away, in came the cleaner...
Luckily the results were retrieved, but I thought we would need at least one ambulance before the day was out, if only for the onlookers who were going into shock!
To be fair, modern operating systems and email clients have degraded the meaning of "delete". When I first started using computers, if I deleted a file, it was deleted (okay, the actual bytes probably had not been overwritten, but the linkages in the allocation table were gone). Same thing with an email. Now, they are simply moved to another location and can be retrieved as needed.
We, here, understand that this is to help us recover from mistakes, which are far too common. Users, well I tend to refer to them as muggles for good reason.
"There is a third category. My dear old Mum had the habit of using the Deleted Items folder as an online archive. Once an email had been read, she would delete it. When she needed to refer back to it, she would go into the deleted items folder and dig it out."
Oh so much this! I've seen SO many customer do this with their email, and they can never grasp why it's a bad idea. Especially fun when you migrate email between systems, and select not to export/import the deleted items... then once finished get asked why their email is now missing!
"There is a third category. My dear old Mum had the habit of using the Deleted Items folder as an online archive. Once an email had been read, she would delete it. When she needed to refer back to it, she would go into the deleted items folder and dig it out."
My brother uses the "drafts" folder.
No offence intended, but if you KNOW that some users use their Delete box as long term storage and you actively choose NOT to migrate their "deleted" mail, you are acting in an unprofessional manner. Worse still, instead of building your working relationship with those users, you are tearing it down.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
