Posts by Wzrd1 2268 publicly visible posts • joined 7 Dec 2012
While I can understand even the DoD can outsource some administrative tasks, I can't understand how they think to be able to support worldwide combat operations through a commercial system, where AWS will try to 'maximize shareholders value', aka profits.
Combat operations are not conducted via the unclassified networks, they're planned, communicated about and documented on classified networks that are still DoD operated.
The worst network problems are usually intermittent issues that cause repeated failovers and churn in the routing process
Which is when the management software marks that carrier/route as deficient and switches to the properly functioning, reliable carrier, triggers an alert to be ignored by management.
But, I'd have digitally signed e-mails notifying management of single point of failure mode now, due to the primary carrier becoming unreliable. Something a plaintiff would find fascinating in any discovery case. And an absolute defense, as I'd re-warn management and the superior in the case of non-response. And preserve a "shut the hell up" response, proving a lack of due care and due diligence.
Giving me quite a legal shield, while the highest levels of management have their pantsless sacks hovering millimeters from the fan blade.
The failure in this case was of the optical layer.
Had something very similar with a bank's home office. The. Entire. Network. Was. Down. Hard.
Flooded with traffic. Set a sniffer, went out and enjoyed a smoke, came back and read jibberish on a copper network. But, I did get enough fragments to have the MAC and traced it to one cheap, off brand NIC. It was mangling packets in just the proper way to be broadcast to each and every switch.
Padded the time, of course, since they should have had packet examination switched on in the core, which would've only dropped a small segment, if that.
So, what we really have is, yet another case of "critical services" having a single point of failure.
Because, public safety is number one...
Indicated via a raised third digit.
> "Hang on," said Frank, and, because he had the terminal window open at the time, he typed in the command to disable OSPF... on a router to which he was connected remotely.
When working at a US military base abroad, our higher echelon was the only level permitted to work on our firewall or tier 1 router. With predictable results, twice per year.
Sudden service outage.
Telephone rings, sheepish voice from a two hour flight away notifying us that he banjaxed the router/firewall configuration and he'd walk us through the steps to break into the otherwise centrally (from their site, of course) access controlled setup.
No need, with having to do this every other quarter, we could break into our own equipment drunk, sleep deprived and kicked in the head by a camel. Which is precisely what we thought of the intelligence of the caller.
Besides, we already hard coded our logons into the equipment the first time, they entirely missed that fact and we fixed the idiot's error far faster than it would have taken to bypass loading the configuration. We also had taken the liberty of connecting a console cable up to a well secured, highly specific server that those idiots lacked access to and never had to even bother going near the service room.
Although, we were seriously tempted to block all traffic on SSH from their network segment...
Instead, I made certain to mention the incident in our theater wide information security teleconference and our weekly Command and Staff meeting, so that the General got a good chuckle. Along with, well, everyone that was on the line. Some, coworkers of said rocket scientist.
All remembered, before I went into information assurance, I was our installation's BOFH.
As for The Princess clients, we've all had them. All suddenly have a Special Project that absolutely cannot be interrupted for even a microsecond, when it comes time for an assigned service outage for maintenance and patching.
"Absolutely impossible for this quarter, we're in the midst of a major effort here and are busy 24/7!"
"Oh? Well, your services and systems were patched yesterday, while the lot of you were at lunch and your systems globally idle. That's why your systems weren't locked, but had to be logged into when you came back from lunch."
Only had to intervene with three systems and extend the service timeout a bit on the server, as the "enhancement" made the blasted thing a bit slow on starting.
At least, that's what my script e-mailed me to report after patching and our hell desk never had so much as a blip from The Princesses.
Fine. I will see you in court.
I will seek 473 quintillion dollars in punitive damages. That's eight teen zeros if your vocabulary is lacking and I will liquidate your company and your client company and use your desk as a tool bench for car repair.
Juries love David v Goliath stories to adjudicate. But, my desire would be clear to a jurist, liquidate the companies involved that created the massive injury and properly punish them.
Because, their assertion is that they're not responsible for their own contractors contract ignoring errors and that's equal to, "It's not my fault that my gun went off while I was raping your daughter and now she is dead, it's the gun manufacturor's fault for it functioning properly".
At least, by the time I and my legal team are done telling and framing it.
'...and the implicit subtext of "touching it will break something, but nobody's sure how or why"'
And I'm the unmitigated prick who will do precisely that, on a scheduled downtime interval, to ascertain if it's still necessary and if so, precisely why, include it into a manual and log, then test the next damnable taped over switch or remove the tape after testing and change management review.
I worked in a change management lacking environment once. One day, I was happily tooling along with some patches to be manually applied via dos prompt and suddenly, my prompt promptly disappeared.
I turned to the shop supervisor and asked, "I just lost my dos prompt and can't raise it again, what did you do?".
It turned out that there was a change order and we were exempted via that order, so he helpfully tested it upon us, the BOFH mk2's.
"Stop it!" at rather high volume was projected by a voice heard by approximately 1000 men without amplification.
He subsequently rose to management, but retained access rights by chicanery and telephone calls to his office frequently revolved around "Stop it!", raising a rueful laugh in the building.
By that point, I had become in charge of information security, he kept trying to do admin things without warning, creating hell, or havoc, your choice.
Laughably, due to manpower shortages, which were infamous, I retained my network god access and worked some issues alongside the tier 2 and 3 types.
I happened to be working at one point on a modest change in a specific network and noticed that my computer logon script wasn't what I saved and worse, as I slapped a tail program on it, changes were incremental and eventually, a loop was introduced. I reached for my telephone and speed dialed one of the few entries on my speed dial.
"Carl, I'm working on the ZZZZZZZ network logon script and noticed some odd changes."
"Woo-hoo, I was working on that computer logon script, as it's part of a change order and it needs to be done. I accidentally created a loop and I'll fix it once I reboot and log on with my network cable disconnected."
"Carl, you're no longer a tech or administrator, you're a manager. STOP IT AND MANAGE", generating open laughter in the office.
I then advised him that I had the ticket, fixed his foul up and the order is marked compliant, so Stop It!.
Generating sidesplitting hilarity in the office.
Full background, that was a US military installation that is part of a command abroad, in a war theater.
I also was asked, by the installation commander how I was to enforce a SOP (Standard Operating Procedure) mandated and frequently ignored, with problematic results, a weekly mandatory reboot.
Knowing that tasking order, I had a script ready to go, which implemented active directory changes to our OU, a secondary SCCM job to push a script that did the same and a more retro approach, where the DNS and WINS server were queried and what was common and not responded to by SCCM or AD got poked the script, which performed the changes at system level.
I simply said, the network command refuses to enforce the order, so, as computers are your responsibility, what do you want to do?
He said, reboot at seven days.
Went to my office, did so. One push on the enter key was all that was needed, once logged in.
About a month later, he was briefing his General and his computer rebooted in the middle of his presentation and he related his orders.
Which he did, which much open laughter, relate to me.
"But also, how did it get to the point of having the printer stripped to components (i.e. who authorised it?) before having someone point out the work around?"
That suggests that stickers were available in that specific era, which is iffy, save toward the end, when printers finally sang their swan song and the "printing" was stored in a log file.
And when PM very often required a manual teardown to remove paper dust and small shreds. And where the masked off contact should have been observed during the tear down, as inspection is a part of the process.
Oops, I think I just gave away my age...
He flew it there thinking there may have been a road accident.
WTF? Was it going there to airlift the injured to hospital?
What a planet.
And I've long considered going through the aggravation and nuisance of purchasing a drone, launching it and reconnoitering the route to work.
Fortunately, I'm not in the UK, where doing so could turn into a felony.
The venerated NASA kept NT4 for quite a few years after its drop dead date, even paying to have service pack 7 created and heavens knows what else after.
Some things work only under specific environments. One then protects those vulnerable environments from the evils of the intertubes or other potential external threats.
And I'm the guy who was BOFH mkII, who navigated successfully through the 2007 cyberattack against USCENTCOM, which nailed the majority of servers in the entire extended domain. Because, we followed both US DoD standards on configuration *and* industry best practices, trivially defeating the blasted USB crapware.
Precisely zero infections on my network that I was responsible for, although I treated systems that alerted as infected and were purged and reimaged, just to teach the end user a lesson on proper caution.
"...but I'll wait for the iFixit breakdown."
Do you mean like Microshaft's bollocks of 1802/1803 cloistersmurf, then the goat rope of 1809?
Seriously, I suggested in our staff meeting that the 1802 was beta tested by interns, staff suggested that beta testing was performed by the interns children.
I'll not bother going into our organization's costs on the more ancient patch level, 1809, well, it followed an earlier pattern that we observed, Microbrain fouling older drivers.
Only one version survives the earlier patch, so I suspect it was a test case, which its fix patch fouled up.
In this idiotic environment, yes.
Assuming the court allows such lunacy.
More worrisome, the trumpism making its way that far down into the rank and file to actually get this filed with a court.
I just had a foreign born, US Army war veteran employee report to me some support among junior civilian staff of Idiot in Chief's view that a Constitutional amendment could be overruled via an executive order.
Our dislike rule by fiat is nearly equal to the UK's notion on the entire subject, the difference is, the populace is armed.
And considering other issues that are also recent, alarming.
Especially since, we're all working under the US DoD.
Fortunately, the rank and file military are entirely Constitutionally driven.
I'm actually growing alarmed.
“In their statement, Google points to its AI principles as the reason for this decision (principles that are themselves a response to internal dissent). The truth is that the project was stopped by the thousands of workers who demanded a say in what they build.”
No, that big NOFORN tag halted things.
You're going to end up with a judgement that trades security against lens quality. And so forth.
So, we're going to end up with a judgement of a security system being superior because it has an on/off switch accessible from the street.
Got ya!
I'll stick with my original assessment the first time I read their drivel, buy anything not listed by them. They're as bad as Consumer Reports, who reported that RCA VCR's were superior to Hitachi VCR's, despite the fact that Hitachi made them for RCA and component for component, were identical.
Which should give an indication on my views and experience with bullshitting rating disservices and sites.
For the life of me I can't see why China would be that fussed about getting access to what TV show is being played on a random four letter TV station.
Well, FEDRAMP is hosted largely on AWS, so that isn't exactly a random four letter TV station, it's the primary access point for civilian US federal agencies cloud presence.
As for bloke on the street targeting, I know a number of peers, as well as myself who could be targeted, due to the PRC hack of OPM and the downloading of our security clearance investigation files.
One upside of that is, now, we can get a security clearance in China.
From the course views of chip locations and traces, it's probable these are CMOS wedge devices, pre-pre-execution environment for the BMC, root kitting it at a hardware level, before the BMC CMOS gets loaded. That bypasses checksums, signing, etc.
But overall being a huge national security issue and potential shock to the US Cloud industry, its almost certain big tech would have been told to keep quiet and indemnified from any markets issues.
And a fairly large segment of civilian US Government agencies and projects operate on AWS platforms. Such as the compromised more than once NFIP, aka NVITS NFIP Virtual Information Technology System (NVITS) (NFIP being National Flood Insurance Program). I could mention more, but at a severe cost, due to an NDA. As NFIP is going to close down, due to a lack of congressional funding, that NDA is entirely moot, as is flood insurance in the US.
But Win10 is such a pile of garbage that I can't be bothered to investigate. There are far more obnoxious issues to worry about, such as the machine's failure to get its updates from our corporate SCCM servers (instead it continues to insist on pulling them directly from Microsoft), and its failure to keep my default application choices across reboots.
Indeed, we blocked and DNS poisoned the Windows Update site, pull our updates from SCCM and fix SCCM client weekly...
What's "Outlook"?
Microsoft Outlook, also known as Microsoft Outrage. E-mail client that borks itself and its profile at random times, in really exotic ways.
Typically, after receiving a "feature upgrade".
Does anyone remember NT4 service pack 6? Followed by service pack 6a. Followed by a string of hotfixes...
Bleh, I'm the BOFH MKII, turned infosec technical.
My security robots are armed with laser wielding sharks and the room that you thought was the server room is actually a tank to be instantly filled with water and yes, the robots are water tight to 190 meters. Working on getting greater depth of performance and performance depth from them. The sharks are fine on their own.
For security reference, for this non-subject, I'll offer affidavits from SAS and SBS team members, who know me and served together during mutual bacon saving missions. I'm also available via FVEY references.
Conditions, property rights in ten acres or so of fine forest, with a modest residence hidden within and a modest access road with a collapsible upon authorized vehicle approach of a "road closed" sign. ;)
Negotiable down to five acres, with no blasted neighbors. And a firearms license, with arms and ammunition to be stored at the local constable's office and constables will be authorized to familiarize themselves to their heart's content - in a sporting manner.
And of course, full NHS health care access. ;)
Yes, I know. That last is a joke, the other conditions, negotiable somewhat. I am an excellent competition marksman, but securing the things are damnably expensive for civilized nations life, so I'll palm it off on the constabulary. Military would also, on their free time, be permitted to utilize said firearms.
The road close sign, a joke, but would be welcome.
Yes, I've had some legendary bad neighbors in my time, peace and quiet is all that I want now, save at work, where it actually is WWIII with cyber warfare.
I'll supply my own batmobile, which likely will suspiciously appear to be a caravan.
The reason such chips haven't been developed is because despite offering Hollywood-blockbuster levels of security, the first time one ACTUALLY fried itself, and some moronic user puts on their Daily Mail sadface with a headline about how they "lost" £1,500 simply because the entered the wrong passcode (or their darling brat did) and it's game over.
I own several Ironkey devices, which do precisely that. They're also designed to brick if cut into.
Modern Windows apps also should store passwords etc in secure storage provided by the OS, which is encrypted.
Rather like writing down the combination to the secure safe, then storing it inside of said safe.
I've actually witnessed someone do just that. I acidly corrected the individual and told them to use the other secure safe.
I say secure safe within a specific context, as it has very specific ratings and itself is inside of a secure facility, inside of a specially rated vault that has 24/7 monitoring via multiple methods.
This policy arose from an incident where an executive left his machine on, and it was infected with malware. Those behind the attack had the whole weekend to surf out or internal network.
We have 24/7 monitoring of the network and systems logs via two layers of monitoring. And a host based IPS system. And 24/7 on call staff to respond to any incident.
The few times I saw a network pwned, it was due to a lack of a system administrator following policy and either not performing the proper baseline configuration or using found USB mass storage devices on the servers and due to the misconfiguration, autorun installed the malware.
They received punishing paid overtime and were named company heroes for working all of that overtime to fix what they fouled up. Until they promptly reinfected everything, precisely the same way in which they did the first time. The DoD was not amused that time.
Their advice was, if a computer gets compromised and it has UEFI, shred it. Don't bother trying to do a clean install, because you can never be 100% sure they haven't slipped something into the UEFI. You can't just throw out the old drives and put new ones in any more. Likewise, even updating the UEFI isn't a 100% guarantee.
Understanding the UEFI system, it's simple enough to reset to factory defaults, flash the BIOS to factory as well and wipe the hard drive. Have yet to have a system retain nastiness once I got my mitts on it.
The script deletes all partitions, creates a single full drive partition, formats it, deletes that partition, resets BIOS to factory defaults, flashes the BIOS, resets it again, then creates new partitions, copies base files, reboots and does hash testing on the files, then goes on for installation.
Even the NSA was impressed.
We disable sleep and always have. Hibernate can be attacked using a different method.
I either lock the machine and leave it running or shut it down. Either way, it comes home with me.
Where someone stealing it is unlikely, as they have to get past the security robots, laser wielding sharks, elevators with dubiously reprogrammed controllers, the hallway of flamethrowers, followed by a liquid nitrogen moat. All, while the BOFH MKII watching and waiting.
The problem isn't the technology, it's the shitty way people like you choose to employ it, were warned by us and we were ignored and it blew up the enterprise.
Then, you fail by trying to blame us and we have tons of notes and memos that proved we had grave reservations over "implementation X", which subsequently blew up.
Thank you!
Proved our points, yet again, Pointy Haired Boss moron.
Single point of failure, bad, especially when it's a fucking cluster. Shared traffic amongst clusters as super groups, good, negotiating constantly for roles.
Adds overhead, but adds reliability.
So, how many bowls of stupid flakes to you enjoy in the morning, with moron milk liberally applied to them?
Plan for failure modes and recovery, not trust the shit actually always fucking works.
If the price of gas didn't go up, then Texas didn't have unusual weather.
An interesting phenomenon frequently reported and related in the Philadelphia area, as the entire Delaware River is lousy with refineries from coast to Philly and beyond.
Somehow, our gasoline prices soar, due to a Houston problem.
Yet the very same company complains over ACA cost sharing.
"What could possibly go wrong?"
Once, on a US military installation in the middle of a massive desert, we had a flood.
Even I, a paranoid contingency planner type, didn't plan of a flood.
The area, high calcium carbonate compact layer (old natural concrete from ancient coral shoals), composing the "bedrock" of the area, six to eighteen inches below the topsoil.
A 3/4 inch water pipe ruptured, spilling out heaven knows how many gallons of water.
End result, a manhole filled with water - our primary telephone trunk occupying manhole.
Network, being optical and continuous, operated normally.
Fine day of no telephone ringing.
Turned out, all telephones on the mid range military installation were entirely out.
Oddly, my IP based secure telephone never rang to report a major outage. Which was far outside of policy.
And noted in the subsequent staff meeting.
The policy being, use *every* form of communication when losing primary communication.
I literally look for smoke signals, when outside.
Morons failed entirely to do anything other than, initially, try a web interface. Failed, halt.
Two used a telephone, failing that, used a secure telephone communications network that was optically based.
Which utterly ruined a very fine day for me.
Turned it from maintenance to recovery in a New York minute, which is measured in Planc units.
I want to bring this to meetings with senior executives... We need a Wikipedia or some easily accessible public link to all the Cloud outages. Its just too easy to skip over intermittent failures otherwise... Or try to explain them or excuse them away....
Seriously, have you ever heard of Google? It's a really cool site, found at https://www.googlel.com. There, you can search for such failures being mentioned. Even get a count of reports. Even look for scholarly articles on scholar.google.com.
Hint: You'd never get to work for me, if HR hired, you, *they* would get fired.
Outside of a specific use case (being able to quickly change the scale of a deployment), I honestly can't think of a solid advantage to using the cloud. It looks more expensive and less reliable on the whole.
Not really, it's how it's implemented. Implementation in a deficient way, where a region shuts down or errors disabling the cluster, bad. Implementation where such a thing springs up nodes to replace them, good. Rejoin them once they're back online, unfuck the clusterfuck of changes between the two at one's leisure, which is milliseconds at worst in a modern environment.
Seriously, I have 64 processors, with cores added, per brick, got two dozen of them in two cinder blocks in the rack.
I have, actually, more off site.
Not figuring out how to balance on site and off site, bad planning of the entire debacle to be.
And I plan on outage of a major sort, at the most vulnerable point, just to avoid a problem and worse, unscheduled overtime.
Overtime is for real emergencies or for work that I have scripts to do, while I surf the web.
Cloudy, with a chance of meatballs.
As usual.
Same old, same old for root cause, over reliance on a geographical location with "preferred" links elected by the controllers, which then become unavailable and wreak havoc on the entire network until communications are restored and the flood of changes are propagated (OK, massively Goobered down version there).
Noticed it at work, noticed a roll back scheduled and erroring, due to it being rushed through change management.
BOFH MK-IV, occupying PFY's position currently. Little could be more dangerous than that. ;)
End result, Outrage, erm, Outlook hangs and is slow in updating. Same with Skype for MonkeyBusiness.
I'm blaming the HCU just outside of our star system for it. The bored Heavy Offensive Unit, the size of our moonish, was bored and introduced an intentional error to see the monkeys scatter about and throw scat at one another.
Or a programming team made a massive, glaring error, which they see what they think is there and not what is actually there in the code. Been there, done that, wore out that tee shirt. Back before said programmers were born.
Blaming an entire technology convergence is just idiotic. Over-reliance on a still immature technology, possibly yes, but we're stuck with what the vendor offers.
Ever look at Microsoft's certification portfolio? Looks like it was written by sales.
Seriously, find me a pathway that can find someone properly trained to find a resultant set of policy in a corporate environment of even moderate membership!
While I manage to figure out RSOP in my head to a minimum of 18 sets and frequently manage far beyond that.
Did a correction on a errored RSOP on New Year's Eve, when all of the on call couldn't figure out the debacle.
The debacle originated on making a significant change just before a major farking holiday! Worse, by junior staff and even worse, being rubber stamped at the end of the fragging day!
Fucking morons.
Took me 20 minutes, being quite well bathed in ethanol and logging in, due to being made on call, when I was solidly off call, looking at the cloistersmurf from hell that that team crafted (I strongly suspect that they began their celebrations and libations a full week before, considering the nature of changes and level), unfouled the mess in my head and ran a RSOP on it, got bored and applied it to fix the issue before I got the damned results.
Called out dead on the second, recovered on the third. Purely out of spite and besides, an old and long missed coworker, back in my ancient carpentry days, called out dead for the holiday weekend to excellent effect.
I'd do that, but I live in the US.
But, I do have an upside. If they try to search my iPhone, it's protected and I'd have it wiped immediately *and* give the TSA a butt rash. The phone is a US DoD provided phone.
I land, connect to the internet on my DoD provided computer and send a wipe signal to our specialized software.
And coming to think of it, I'll store the PIN for the device on a classified network. Tell the TSA agent that the PIN is classified, which it then would be and if he or she insists, have them arrested for espionage.
Which laughably, is literally the law in the US!
Microsoft has been hacked, Sony has been hacked. Apple has been hacked. Everyone has been hacked at some point - and some don’t even realise it (because the hacker who hacked was serious, competent and not doing it for the lulz).
Do you mean like the US OPM hack?
Well, an upside to that is, I can now get a security clearance in China.
Would they stick to that policy, or "finally" agree that they "should" help get the cops into an iThing to do their bit to help fight dangerous criminals?
Do you mean like when they, during the whole court order thing, suddenly allowed generic fingerprint scanner modules to be installed on the iPhone and somehow, a security researcher found a way to get one that always read false positive the very next day?
Remarkable coincidence, if you believe in coincidences, which I do not.
Or that super sweet DoD deal for thousands of iPhones that came shortly after? Got a DoD issue iPhone sitting right next to me now, as I'm on call.
Dammit.
"The reference to the extension has been removed from the blog post as part of the investigative process."
Because, security by obscurity is actually an effective thing. No, it's security by obscenity.
Disabling or removing the applet is proper, removing the post is not.
Information security meeting is tomorrow, this will indeed be brought up and I strongly suspect, Firefox will no longer be on our entire network. Which is a rather significant number of users.
I always thought that hot fusion worked well - or at least it appeared to be doing ok this morning before it clouded over.
Cold fusion works 100% of the time. Not well, most certainly not gainfully!
https://en.wikipedia.org/wiki/Neutron_generator#/media/File:Neutristor_in_its_simplest_form.JPG
Aka, a neutron generator. Generated neutrons, generates a large electric bill, heat, not so much.
So, fusion does occur and indeed, at room temperature. Poorly, with massive losses. Tweaks lower the loss, increase neutron creation, at a cost of additional energy input at room temperature and ambient pressure.
What gets done now is high pressure, insanely high pressure and hence, temperature isn't extremely relevant there, save if one is conducting quantum level calculations.
"Cops want this tech cool give it to them but. When it fucks up the cops have strict liability. The police and their superior are finally liable . This can not be discharged. Cops can by charged with false arrest"
Largely false in the US.
Oops, we fucked up is the excuse and it's excused.
Eventually, the POTUS will get shot by police, "by mistake" and they'll end up excused.
The US has a massive gap between federal and state systems. Police are local, not quite state, not quite not. Federal officers, like Secret Service and FBI are federal.
Each have their own Constitutional boundaries.
So, it's entirely possible for a local police officer to conduct the behavior I suggested in the US.
God Save The Queen if such a system misidentified her, as US police don't think, as they have a "war on crime" mentality.
With them literally thinking that they're soldiers in a literal war.
I know, I know enough of them.
Not very mentally flexible at all.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
