I wonder why, as I read that Windows was successfully patched in November.
Azure VMs borked following Meltdown patch, er, meltdown
Microsoft Azure customers are reporting problems with their virtual machines, which are struggling to come back online after being updated with the Meltdown processor patch. The firm told one customer in a direct message on Twitter: "We are currently investigating alerts affecting Virtual Machines in West Europe. You should be …
COMMENTS
-
-
-
Friday 5th January 2018 22:43 GMT Anonymous Coward
Azure was down but it doesnt say it was
Notice the Azure Twitter has a bunch of questions and Azure is moving those conversations to private? I have contacts in North America that were down as well as Europe. Yet azure status site and history shows nothing. https://azure.microsoft.com/en-us/status/history/
-
-
Thursday 4th January 2018 20:16 GMT rmason
like mnany I suspect.
Like many others i'm up patching tonight.
mixed CentOS and windows environment, haven't had anything fail to come back up *yet*.
Worth noting for others though;
Our benchmarking stuff is showing a 20%+ (ish)slowdown on postgres (SQL) on centOS.
I've had tests at everything from -7% on some boxes, to the -20% mark. Bad times.
-
-
Thursday 4th January 2018 22:45 GMT Pax
Re: like mnany I suspect.
This is part of what organisations should be doing in assessing the risk to their organisation on machines which don't run user interactive sessions, and "adequate protection" is deemed to be in place.
Can your business application/users cope with the performance hit, and can you scale your resources accordingly to mitigate a performance hit?
If you are in public/hybrid cloud, what is the cost in horizontally scaling resources to mitigate performance hits.
-
Friday 5th January 2018 00:26 GMT Maventi
Re: like mnany I suspect.
> This is part of what organisations should be doing in assessing the risk to their organisation on machines which don't run user interactive sessions, and "adequate protection" is deemed to be in place.
Good call.
> If you are in public/hybrid cloud...
Then it's probably best to take the performance hit as you never know who else might be sharing your compute node with potential access to your own host's memory.
-
-
Friday 5th January 2018 03:14 GMT Adam 1
Re: like mnany I suspect.
> if it doesn't run code from the outside and if you don't want the performance hit - then why patch?
In the case of Azure, you are not going to be running on bare metal. You are going to be on a VM guest, so the important question is whether your kernel's data can be read from collocated VMs belonging to other customers. I am personally unclear on whether patching the VM host is sufficient or whether both host and any guest need both be patched.
But yes, if you have a machine which is air gapped with a performance critical workload then you are one of the lucky few.
I am sure we won't be waiting too long for this to be exploited via JavaScript or a PDF/docx/xlsx file with some macro. That is going to suck big time.
-
-
Friday 5th January 2018 15:48 GMT Alistair
Re: like mnany I suspect.
We're seeing almost 60% of a single core being eaten by our networking -- this is on a hadoop data node. 14 cores, but still likely to cause some imbalance long run.
We're trying some tweaking to see if we can make it better. (closed system testing in a sandbox, no way I'm putting this in prod yet)
-
Monday 8th January 2018 12:17 GMT rmason
Re: like mnany I suspect.
To try and answer some questions:
I didn't patch everything that night, this was a "suck it and see" test on playground servers running copies of both our internal stuff, and the stuff we sell. We will be patching everything though, regardless of performance (they must then be "fixed" somehow).
Like most places we are hybrid, some AWS and azure, some on prem stuff, it's all getting patched, that is a clear directive from "on high".
I Imagine the future fix for performance will involve upgrades both to physical and cloud stuff, and improvements by our devs to the product.
I'm not touching prod stuff in terms of the product, because of the variance in issues seen on our SQL driven stuff, but the windows stuff that's on prem (DCs and FS basically) is all patched up (as far the update produced so far) and done.
-
-
-
Friday 5th January 2018 10:34 GMT Tezfair
might not be just VMs...
Customer server installed something last night and since then hourly throws this error...
----------------------------------------------------------------------------------------------
File Server Resource Manager Service error: Unexpected error.
Error-specific details:
Error: GetVolumeInformation, 0x80310017, The data drive specified is not set to automatically unlock on the current computer and cannot be unlocked automatically.
----------------------------------------------------------------------------------------------
It only started appearing in the logs after an update. They do have a bitlocked backup, but that has opened ok and the other drives are fine.
-
Friday 5th January 2018 13:20 GMT TheVogon
Re: might not be just VMs...
That's nothing to do with this update.
Removable data drives must have either a password or a smart card unlock method in addition to the automatic unlock method. Automatic unlocking cannot be directly specified by policy settings.
To configure a BitLocker-protected fixed or removable data drive to automatically unlock, follow these steps:
1. Click Start, click Computer, and then right-click the BitLocker-protected fixed or removable data drive that you want to automatically unlock.
2. Click Manage BitLocker, click Automatically unlock this drive on this computer.
-
-
-
Monday 8th January 2018 14:24 GMT Anonymous Coward
Remembering Snowden....
Yup.....Meltdown and Spectre are both causing trouble.
But how long before we hear about ALL THE OTHER ZERO DAY PROBLEMS which the NSA, GCHQ, the Russians, the Chinese, and all the other bad actors have squirreled away against the day that Meltdown and Spectre end up patched everywhere?
-
Saturday 13th January 2018 18:15 GMT portyman
cloud and hyperconverged
In the rush to all things new and "flashy" in the assumption its cheaper a lot of people have been burned.
many years a go someone said, don't put any data into the cloud you don't want other to see. We had cloud providers saying how secure it was but strangely went very quiet when we asked if they would guarantee that with some money should data leak out, Never heard a salesman stumble of words so quickly.
Public cloud will never be secure, every security feature that has been created, has been got around at some point, or will be. Things are too complicated these days making it almost impossible to make things secure.
Hyper Converged was touted as the solution to most data centre issues, Spectre has shown that sharing your data and compute on the san node is suicide for security. I hated the idea anyway due to the state of software testing these days never mind trying to get different vendors to fix issue.