* Posts by Stoneshop

5954 publicly visible posts • joined 8 Oct 2009

NSA leaker bust gets weirder: Senator claims hacking is wider than leak revealed

Stoneshop
Big Brother

Re: curiouser and curiouser

There shouldn't be anything to defend.

No? How about the voter registration database, used to print out the list of who is entitled to vote in each district? Say, your name is Tuttle, but changing one single character will have you listed as Buttle. Voting is not going to happen for you until bureaucracy has been wrestled into submission and corrected the entry three years from now, or you have given in, requested a name change which has finally been granted[0]. Or you have two digits transposed in your address, or your date of birth. Those 'errors' will cause delays and people being turned away at the voting station,

The actual voting totals don't need tampering if you take this route.

[0] the initial problem will have been detected and corrected two days before the name change is to officially go into effect.

Stoneshop

Re: It could be worse ...

sending scanned images of printed pages to a journalist using her work computer!

Nope. She printed out a classified document at work, smuggled it out, sent it (anonymously) to The Intercept, who then proceeded to show it, identifying microdots and all, to a NSA contractor to verify. That contractor duly reported this to his superiors, who went to work locating the document and identifying who had accessed and printed it on that particular printer. Postmark on the envelope and using her work computer to send a message regarding a podcast to The Intercept using gmail was just icing on the cake.

Consultancy titan EY to shift jobs to Indian outsourcer TCS

Stoneshop

we are in the process of transforming how we deliver IT services

Upgrading from 1200 to 2400 baud modems, are they?

First-day-on-the-job dev: I accidentally nuked production database, was instantly fired

Stoneshop
Facepalm

Re: @wolfetone

"Connect to database CastleAnthrax

Username: Tim?

Password: g0aW4y||1sh4lLtauntua2ndtiem"

Or even simply "Generated by setup tool"; even better if that would cause a syntax error on at least one of the input fields if someone brainfarted and put in that string.

The biggest British Airways IT meltdown WTF: 200 systems in the critical path?

Stoneshop
Facepalm

Re: Sunny when it is working

I know that they said outsourcing is not to blame but getting rid of people that know how things work or are held together is a dangerous risk. How many companies know nothing about what all the boxes do let alone how they are all dependant on each other?

Another factor that I see happening is feature sprawl, add-ons often being introduced as 'nice to have', with a low priority to fix if broken. Problem is, even if those features keep being handled at low prio[0], each of those features adds to the knowledge the first and second line support have to have at the ready, as well as simply adding to the workload as such. Having to not just physically but also mentally switch from one environment to another if a more urgent problem comes in and you have to suspend or hand off the first problem because you're the one who best understands the second one is another matter.

[0] and often they don't, because the additional info they provide allows for instance faster handling of processes, smoother workflow, better overview, etcetera, and after a while people balk at having to do without them. So even when they''re still officially low prio, call handling often bumps them to medium or even high because "people can't work". Oh yes they can; how about remembering the workflow that doesn't rely on those add-ons? The workflow they were trained in?

BA IT systems failure: Uninterruptible Power Supply was interrupted

Stoneshop

Re: Isn't restore 101

Why aren't the different components not integrated?

Hysterical raisins, for a large part. Security comes into play too: systems that have outside connections in whatever form are not allowed to even communicate with the main clusters directly, let alone have processes running on those clusters. And there is stuff like data conversion systems supplied by third parties, hardware and software, therefore not integrated as as matter of fact.

Furthermore, you don't have your monitoring system integrated with whatever you're monitoring, do you? Or your storage management system integrated with the systems you're providing storage to?

Stoneshop

Re: Isn't restore 101

Good effort, but why didn't you have a plan prepared in advance?

Good question. Next question please.

What it comes down to, because that particular information just wasn't there, and basically the best thing to do in the time available was distilling it from a connectivity matrix, combined with noting whether systems were essential, auxiliary or 'meh, can wait'.

We have detailed info on all systems, including how to start them up from zero. Like for the main VMS cluster basically: 1) network, 2) storage management, 3) storage shelves and controllers, 4) the cluster nodes themselves, but only few of those documents describe the how and why of interaction with other systems. That info tends to be in another class of documents, not the system operation manuals. There are sections that describe what has to be done to neighboring systems in case of a total system shutdown, but that assumes you can log in to those systems to shut down the affected comms channels and such. With the power cut having done so for you, site-wide, there was a certain 'fingers crossed' involved, but the hardware proved surprisingly robust (two minor errors over the entire site, AFAIR), and the software only needed minimal prodding to get the essential bits working again.

Stoneshop
Facepalm

Re: If it got interrupted...

Usually near the door in each DC hall...

But not so near that they can be mistaken for a door opener button by the dimmest of dimwits. At chest/shoulder height and at least a few steps away from the door appears to me the most sensible location.

That said, I've seen a visitor who shouldn't have had access to the computer room in the first place look around, totally fail to see the conveniently located, hip-height blue button at least as large as a BRB next to the exit door, and killed the computer room because a Big Red Button high up the wall and well away from the exit is obviously the one to push to open the door for you.

Unfortunately, tar, feathers and railroad rails are not common inventory items in today's business environment; rackmount rails are too short and flimsy for carrying a person.

Stoneshop

Re: Isn't restore 101

Quite.

We had a rather unscheduled event once, where the fire brigade threw the Big Red Switch in the outside feed. During the time the cleanup was done (mopping up the water and ventilating the building) we worked out the startup sequence for the stuff present: network gear and standalone servers that wouldn't care about connectivity, servers that would need network or else their network configuration would be totally bonkers, and servers that would need to see particular other servers, otherwise their would be in a bind with the best way to recover being rebooting once the other end became available.

Energy was told to switch off all local circuit breakers before restoring power to each of the computer rooms, so that we could switch off all systems before the racks got powered.

With that crib sheet things went as good as flawless.

Stoneshop
Thumb Up

Re: Disaster Recovery anyone?

it is not a DRP, but an NMP (Not My Problem)

A SEP, actually.

Stoneshop
FAIL

Re: If it got interrupted...

I bet he was called in and told to pull the plug as a consequence of the system grinding slowly to a halt yet not switching over to secondary.

If you really want to force a failover that way, you do so by shutting down the small number of systems that would cause the monitoring system to detect a "critical services in DC1 down, let's switch to DC2". If you can't log in to those systems because of system or network load you connect to their ILO/DRAC/whatever, which is on a separate network, and just kill those machines. If the monitoring system itself has gone gaga because of the problems, you restart that, then pull the rug out from under those essential systems. Or you cut connectivity between DC1 and the outside world (including DC2), triggering DC2 to become live, because that would be a failure mode that the failover should be able to cope with.

You. Do. Not. Push. The Big. Red. Button. To. Do. So.

Ever.

Stoneshop
Devil

From The Meaning Of Liff

AIRD OF SLEAT (n. archaic)

Ancient Scottish curse placed from afar on the stretch of land now occupied by Heathrow Airport.

It's clearly working.

Stoneshop

Re: A data or application problem most likely

The article quotes someone as saying a data problem is easier to fix than a hardware one. No idea where you got that total bullshit from. It depends on the circumstances. Even if you had to replace some hardware, that can generally be done faster than trying to fix a set of applications with corrupt or otherwise invalid files that are all trying to talk to one another.

Indeed. And even if half your hardware is fried, it should be possible to bring up the other half with a reduced set of applications in a way that core functionality can be restored. And your DR plan should have tables of what machines can be reallocated to other tasks in a case like that.

Corrupted data is another matter entirely. Can you fix it by rebuilding a few database indexes or zeroing some data fields, do you need to restore a backup or is the 3rd line support tiger team huddled over their monitors amid mountains of empty coffee cups, alternately muttering lines of logging or code, and obscenities?

Stoneshop

Re: A data or application problem most likely

2. The primary DC has backup/UPS power - why doesn't that work? The article suggests *maybe* the main power and backup were applied simultaneously causing the servers to use 480V. Fair enough.

That (getting 480V ed into the racks) suggests a more than grave wiring error that would have caused one or more of 1) seriously frying the output side of the UPS, 2) seriously frying the generator, 3) causing an almighty bang, 4) causing parts leaving their position at high velocity, 5) the electrician(s) that did the wiring leave their place of employment at high velocity, and 6) one or more electrical certification agencies not previously involved in certifying and testing this setup taking a long, hard look at the entire process from commissioning the installation to the aforementioned result.

3. How does (2) affect what happens at the secondary DC? Why does exactly the same thing happen on a redundant system which is designed to mitigate against such problems occurring at one DC?

Did it? Or was the second DC karking the result of the primary DC splaffing corrupted data as it went down, and thus corrupting the failover?

Walmart workers invited to shuttle packages

Stoneshop
Boffin

Re: SlaveMart

But they don't because they know the shoppers are savvy enough to know it doesn't matter whether you come to them or they come to you: gas gets used either way.

Not quite, if the employee, driving home anyway, has to make a detour to deliver a package, only the extra distance and the petrol used for that detour should be counted against the delivery

Utah fights man's attempt to marry laptop

Stoneshop
Coat

Bigger is not always better

Well, the 8 (not 10) inch floppy would still be, well, floppy; that 3.5 inch one is much stiffer.

Maybe he should fit his laptop with one of those JizJaz drives.

Boffins find evidence of strange uranium-producing bacteria lurking underground

Stoneshop

Re: yes the Uranium is being used as metabolic fuel for the bacteria.

"Another example of evolution in action."

Yes, but in which direction?

Nuclear powered bacteria.

If these had been discovered in the 1950s, when people expected everything to become powered by atomic energy ...

Pai guy not too privacy shy, says your caller ID can't block IP, so anons go bye

Stoneshop

It is based on the false premise that police lives somehow matter more than the lives of the rest of us.

As such I agree with you, but as police and first responders have a certain responsibility towards the safety and wellbeing of the community (putting aside the question of whether that responsibility is always carried out correctly), they should be able to expect the community taking responsibility for them being able to perform their duties.

And I don't think 'Blue Alerts' are the right way to express that responsibility.

Elon to dump Trump over climate bump

Stoneshop

Re: Coal stocks fell today

Hmm, sounds like socialism, didn't think that was popular in the US.

That would be if the guvmint was to be paying unemployment benefits. Instead, Bombastic Bob is claiming he'll dump the bill on a private person he dislikes, and probably will go on to sue[0] him if he refuses to pay.

[0] increasing employment for landsharks in the process.

Stoneshop
Holmes

Re: Coal stocks fell today

I shall bill you for the living expenses of the miners and others that lose their jobs as a result of it. what, not ready to TAKE RESPONSIBILITY for what you want to happen?

80200 jobs in mining, 60460 jobs in coal-fired power plants, and a little over 4000 in construction of new plants. With only a small part actually involved in mining, and possibly unemployable elsewhere if all coal-related activities were to cease.

Stoneshop

Re: Dumping Paris is a good thing

It gives China a free pass until 2030.

Which, even now, they appear to be using only sparingly and reluctantly. Witness them boosting renewables like hell.

China is finding it hard to attract the highly skilled people from other countries it still needs noting that those are rather turned off by the prospect of not being able to see across the street, and requiring breathing gear when venturing outside

Stoneshop

The idea of the podgy unstable finger hovering over the red button takes me back the the fear I had when I was a teenager.

"he best way to keep Mr. Trump off Twitter, advisers said, is to keep him busy. During his foreign trip, he was occupied 12 to 15 hours a day, seldom left alone to fulminate over the Russian investigation and given less unstructured time to watch television — although he did tune in to CNN International and fumed privately that it was even more hostile to him than the domestic network."

I'm not sure if that's a better tactic than just having him on Twitter all his waking hours, finishing with a satisfying covfefe while he zones out on the couch, exhausted, Unfortunately, there are no adults in the White House that can take up the slack in actually making policy, but keeping the Orange Baboon from anything more dangerous than a blunt spoon may still be a good thing.

Hmm, even a phone with rounded corners may well be rather dangerous in his tiny hands.

Stoneshop
Boffin

Re: Surprised?

I think you need to go read some Svante Arrhenius papers.

Which can be simply dismissed, Arrhenius being some ancient furriner doing science.

I also doubt that Bombastic Bob would be able to make it past the introductory paragraph.

Identity management outfit OneLogin sugar coats impact of attack

Stoneshop
Pirate

I would personally suggest adding some steps on to the guide.

These combined would actually supersede step 1 as listed.

Step 12 wold be something like, ahem, strongly discouraging anyone you know to even think of using OneLogin, with step 13 being the activation of a set of landsharks to go and find if OneLogin's usage policies regarding immunity against claims for damage are as solid as their security.

Security company finds unsecured bucket of US military images on AWS

Stoneshop
Trollface

Re: There's a hole in me bucket,

Please tell me more about the hole in your bucket.

Boffins spot 'faceless fish' in strange alien environment

Stoneshop
Boffin

Re: The Deep Ones...

I think they wouldn't be reporting anything, tbh.

Unless the Deep Ones have kept up with technology and are now able to summon a global EMP, it would still take a not infinitesimally small time interval between disturbance and total worldwide annihilation, in which case at least some news covfefe will get out.

How the Facebook money funnel is shaping British elections

Stoneshop

Re: If ad slinging

Tell me the truth, give me the numbers, be honest about your motives. Thats all I ask of politicians.

You want a pony to go with that?

Much-hyped Ara Blackphone LeEco Essential handset introduced

Stoneshop

and the wallet is too empty to buy a replacement

Five quid buys you a BT headset receiver that you can plug a wired headphone into. Available at your favourite Chinese Tat Bazaar.

BA's 'global IT system failure' was due to 'power surge'

Stoneshop

Re: Comment from a Times article.

Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time.

True.

That means, when you restart them, components like memory chips and network cards fail.

Nonsense; only if you power-cycle them Just rebooting without power cycling doesn't matter to memory or network cards. Processors and fans may be working closer to full load while booting compared to average load, and with it the PSUs will be working harder, but your standard data centre gear can cope with that.

Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers

Switching PSUs have the habit of drawing more current from the mains as the voltage drops. Which will cause the voltage to drop even more, etc., until they blow a fuse or a circuit breaker trips. But as this lightens the load on the entire feed, it's really quite hard to get a DC to go down this way.

- this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.

Any operational monitoring tool will immediately call out the systems that it can't connect to; the harder part will be getting the hardware jocks to replace/fix all affected gear.

Stoneshop
Holmes

Re: Really a power failure?

The idea was to ensure people couldn't casually plug uncertified equipment into those sockets.

Cleaners will only notice they can't plug their vacuum in after unplugging that which must never be unplugged.

WannaLaugh? Funsters port WannaCrypt to Commodore, Cisco, Nintendo and Tesla

Stoneshop
Devil

Re: They won't stop

IMHO it is better to only allow updates to be done by a dealer.

And the update file won't get infected between the manufacturer and the dealer? Or at the manufacturer itself?

US laptops-on-planes ban may extend to flights from ALL nations

Stoneshop
Facepalm

Domestic flights

Also it's kinda crazy that they're not talking about banning hand luggage laptops on domestic flights.

Especially since those tend to carry a greater proportion of domestic[0] passengers, which is what the terrrists are said to be aiming for.

[0] though not necessarily domesticated.

'Major incident' at Capita data centre: Multiple services still knackered

Stoneshop

Re: Probably got their own staff to install the back up generators

Isn't the refilling done by the tanker driver who delivers the stuff?

As the Germans say 'Jein' (contraction of yes and no): first someone[0], having been notified by Facilities that the tank is running low, has to call the supplier for delivery, then with the tanker arriving someone[1] has to unlock[2] the gate/hatch/trap door to the tank neck.

[0] from Finance, or Contract Manglement[3]

[1] from Security[3]

[2] you don't really want someone peeing down the filler neck, or dropping sand or sugar in.

[3] in extremely enlightened cases these responsibilities will have been delegated to Facilities as well.

Stoneshop
Flame

Re: Probably got their own staff to install the back up generators

Heck, just kick out the mains CB and let the genny take over (for 30 minutes each week)

Ingredients: one power grid with regular shortish (30 minutes or less) outages, one computer room floor with various systems, one UPS powering the entire floor running at ~15% capacity, one diesel genny. Due to the regular power dips, we were quite sure the UPS and diesel were functioning as intended; fuel was replenished as needed. Then came the day that the power consumption of the computer room doubled due to an invasion of about 45 racks full of gear. And then came the next power dip. Which made the UPS (powering the computer room; the generator was hooked up so that it basically kept the batteries charged) suddenly work quite a bit harder. And longer; for a number of reasons. Which caused the temperature in the UPS room rise quite a bit more than previously. Environmental monitoring went yellow, and several pagers went off, and Facilities managed to keep the UPS from shutting down through the judicious use of fans scrounged from a number of offices.

Moral of this story: cooling is important too, not just for the computer room, but also for the UPS room.

Sysadmin finds insecure printer, remotely prints 'Fix Me!' notice

Stoneshop

Scream-tracing

A little over a decade ago I was contracting with a large software supplier/bodyshop[0] that was closing down one of their branch offices. Which involved moving most of the systems in that office (let's call it 'E') to the one where I was orking (let's call it 'N'). After some culling and rearranging about 53 racks had to be moved from E to N, but initially N had only space for 18. So we had a look around to see what could be culled from N. This promised to be quite worthwhile, because the floor was littered with gear for customer projects that had long been delivered, patched, upgraded, patched some more and declared finished. With the project teams long disbanded, reassigned to other projects, split off into separate ventures and members having left the company or even this earthly plane. Documentation was either stored in a filing cabinet in a disused lavatory, or buried in soft peat for three months and recycled as firelighters. So I called a meeting of all department heads that might have some equipment in use on the floor in N, handed them a document template and a pile of stick-on envelopes[1], and the notice that any system not labeled two weeks from that date would be subject to gravitationally motivated impact tests in the car park.

Of course we did not do so right away after those two weeks, instead simply unplugging any network cabling[2] from those systems. And one sub-department came and wailed bitterly that they could not access their test rig, explaining that their lack of labeling was because of the department they were part of not passing on the meeting request. We then reconnected their gear, and there was much rejoicing.

(this action netted twelve racks of orphan systems that nevertheless had been running, consuming power and cooling, for several years)

[0] they clearly had insufficient bodies available with the skills required for their own operations.

[1] none of this faffing about with a shared database, which would have taken weeks to set up, deploy, get everyone to add their data (for which they probably would have needed to visit the computer room floor), after which we would have needed to match that data with the systems ourselves anyway.

[2] deemed safer than powering off; quite a few systems were expected to suffer Spontaneous Loss of Magic Smoke in case they needed to be powered back on.

Stoneshop

Re: See the printer?

I'm guessing here, but he physically saw it, remembered make and model

Not necessarily. If you can access the web interface, you're usually presented with this info, and much more such as page count and ink/toner levels, in one of the maintenance pages.

Juno's first data causing boffins to rewrite the text books on Jupiter

Stoneshop

Re: 6MB

https://www.nasa.gov/press-release/nasa-s-juno-mission-to-remain-in-current-orbit-at-jupiter

"NASA’s Juno mission to Jupiter, which has been in orbit around the gas giant since July 4, 2016, will remain in its current 53-day orbit for the remainder of the mission."

Stoneshop

I believed Jupiter threw bolts, not balls...

By Jove, you're right.

Stoneshop

Ethernet

Even 10B5 has a length limit of 500m (3.615 brontosauri), which would require adding a rather large pile of repeaters, and powering them.

They probably went for ADSL, and subcontracted to BT for the copper.

Stoneshop
Boffin

Re: 6MB

325 bits/second is 28Mbit/day, or 38Mbyte per 11 days.

It's roughly the speed of my first modem, and three times as fast as an ASR33. The BBC B could easily cope with that datastream, you'd just need about four days worth of audio cassettes to save the data from a single orbit.

Google wants to track your phone and credit card through meatspace

Stoneshop

They'd know which cashpoint I visited

Your bank knows, Google can only infer your visit from your Android collecting location data and the time you spend near an ATM.

And if you're in Europe, your bank is in for some unpleasant regulatory action if they even think of using this info unless it's directly related to their own business.

Stoneshop
Facepalm

Categories

I bought a bread bin. It holds a loaf and some muffins (oven bottom to avoid the whole roll/barmcake/barm/stoaty debate) so why oh why do they keep sending me emails about bread bins?

I've mentioned it before, but sellers need to categorise their stuff into three categories, roughly: regular purchases, durable goods and occasional stuff.

Regular is everything you buy at least once every two weeks or so, like food. You may want to see ads for that if you're interested in knowing what's on special, but usually you have your favourite brands and advertising is rarely effective in changing that. Durable goods are what you buy once every five years at most, and if you've just bought one of that category you are EXTREMELY UNLIKELY to be interested in a second one anytime soon; at best related items like paper or toner if you've just acquired a laser printer. The occasional stuff is probably where the richest pickings for sellers/ad-slingers are, so they might want to concentrate on that.

Stoneshop
Big Brother

Re: get ready for “what's your e-mail address?” from counter staff

Everybody should start answering "abuse@google.com" to that.

IT firms guilty of blasting customers with soul-numbing canned music

Stoneshop
Holmes

Re: Suggestions for tech firms' hold music

Uber: Elvis Costello - Watching The Detectives

Stoneshop
Big Brother

Re: Suggestions for tech firms' hold music

Google: Every Breath You Take - The Police

Stoneshop
Devil

Re: Suggestions for tech firms' hold music

Kraftwerk - Der Telefonanruf

Bankrupt school ITT pleads 'don't let Microsoft wipe our cloud data!'

Stoneshop
Facepalm

Once again

"The Cloud" is shown to be someone else's computers. Also, they didn't do their backups to on-premises systems, if at all.

Clearly, they were betting on the wrong horse.

New York Attorney General settles with Bluetooth lock maker over insecurity claims

Stoneshop
Pirate

Re: Is IoT things developed by non IT people?

while in the second case, we can hope security is part of their very nature.

Maybe you still have that hope; I don't.

Vegemite tries to hijack Qantas name-our-planes competition

Stoneshop

T&C

"The entry must be:free from any claims, inc. trademark claims, by other parties."

Vegemite Co.: "Hey, we suggested it, you think we'd go make a trademark claim afterwards? Wouln't be very sporting, now."

No nudity please, we're killing ourselves: Advice to Facebook mods leaks

Stoneshop
Thumb Up

Videos of violent deaths "can help create awareness"

So a livestream of The Zuck's brain being throttled by his own major intestine should be OK then.