nav search
Data Center Software Security DevOps Business Personal Tech Science Emergent Tech Bootnotes
BOFH
Lectures

back to article
Xen Project wants permission to reveal fewer vulnerabilities

Lack of disclosure isn't going to win many, if any, friends, especially when it comes to a hypervisor that's been riddled with vulnerabilities as of late. It just makes me wonder what they're trying to hide and for what reason.

Transparency is not the enemy, especially not in an open source software project.

Overall, it sounds like a really good reason to stay the hell away from Xen, as if we needed more.

7
5
Silver badge

Nice link, pity you didn't read it

Now try looking at the KVM entries that are also in that list, or using KVM as a keyword.

2
0
gwd

It's funny that you go on about how transparency won't hurt you, and then point to the list of disclosed vulnerabilities as evidence that you should "stay the hell away from Xen".

Transparency about vulnerabilities actually has a massive cost for any software project, for exactly that reason: people assume that if they don't know about vulnerabilities that there aren't any. So the project which is least transparent will be *perceived* as being more secure, and the project that is the most transparent is *perceived* as being less secure.

The Xen project is one of the few projects which is brave enough to be transparent about our vulnerabilities. The reason you don't see a long list of KVM, Hyper-V, or VMWare vulnerabilities isn't that they don't exist; they do. It's that they don't have the same level of reporting as Xen. We know it affects our reputation, but we think our users' security is actually more important. But most projects look at comments like yours and say, "There's no way we're going to be that transparent." Your attitude actually makes transparency much less attractive to projects, which in turn makes your own security worse.

The proposal we're asking feedback for isn't going to affect the vast majority of vulnerabilities. Late last year we had a bug in the handling of the PSE bit on a 4th-level pagetable (XSA-176). We looked at the code of Linux, NetBSD, FreeBSD, and OpenBSD, as well as the behavior of Windows, and none of them ever set that bit (and so wouldn't be vulnerable). But we issued an advisory anyway to be on the safe side. We've had other similar bugs since. The question we're asking our community is, "Do you want us to issue advisories like that, where nobody actually seems to be affected?"

Good luck getting that kind of consideration from any other hypervisor.

4
0

"Lack of disclosure isn't going to win many, if any, friends, especially when it comes to a hypervisor that's been riddled with vulnerabilities as of late. It just makes me wonder what they're trying to hide and for what reason."

You may wonder that, but the reality looks quite different. If you look at the facts (aka https://www.cvedetails.com/vendor/6276/XEN.html), you will find that the issues reported have been going down in the last two years (2014: 44, 2016: 28). This has happened in parallel with the project actively taking measures to find more bugs. Also, our security team is bigger in terms of members participating than at any time in the past.

If you look at Linux (https://www.cvedetails.com/vendor/33/Linux.html) and QEMU (https://www.cvedetails.com/vendor/7506/Qemu.html) the opposite is true. The big difference is the media attention that Xen Project vulnerabilities get compared to other projects: so yes, it looks as if we had more vulnerabilities than others, when in reality we are actually doing OK.

And it is not because we issue fewer CVE numbers or handle fewer issues. When you look at the data, you will also find that the average CVSS score of the issues we handled has reduced as well (the average score used to be around 5.1 - 5.4), but last year it was around 3.3.

"Transparency is not the enemy, especially not in an open source software project."

I do believe that we are one of the most transparent projects on how we handle security issues. This is why we made a public proposal to get feedback. Also, it is worth mentioning that there is a trade-off to transparency: every time we issue an XSA, which could be a vulnerability in some theoretical circumstances, but may not really be one, we are creating work for our down streams and users. We were criticised about this in the past, and this proposal is trying to address some of this alongside some other issues we have come across since we revised our policy last.

It is of course also good that El Reg is giving the proposal visibility. If you have an opinion, feel free to vote on the El Reg survey, but I did want to point out that a reply to our proposal on xen-devel at lists dot xenproject dot org is more helpful. You can also use the Reply button at https://www.mail-archive.com/xen-devel@lists.xen.org/msg96571.html (but make sure you CC xen-devel at lists dot xenproject dot org)

6
0
Silver badge

The real reason

I think his comments state very admirably why they want to do this. It's all about cost. Nothing to do with building a decent hypervisor or removing vulnerabilities. They simply don't like spending the money. I also like the reference to costs for Xen 'users'. Given that this is pretty much just cloud providers, that speaks plenty as well.

The most worrying statement he makes is not reporting if “no known combination of software in which the vulnerability can be exploited.”

This basically says that if they can't think of a way of exploiting a bug, then don't worry about it. This is exactly the sort of thinking that compromise security hugely. Just because you can't think of a way of exploiting it, doesn't mean nobody else can. This is giving hackers etc. a pretty free reign if carried out.

If this goes through, I wonder if anybody would reconsider their use of Xen based cloud solutions? Just how safe and confortable do you feel with your applications and services sitting under Xen after a statement like this? Knowing companies, I doubt if many will care. Until someone gets caught of course.......

3
2
gwd

Re: The real reason

"This basically says that if they can't think of a way of exploiting a bug, then don't worry about it."

No. If you read the full text under number 4, it goes into more detail about what this is about. XSA-176 was a bug in the way L4 pagetable PSE bits were handled. But when we looked, there wasn't a single operating system (Linux, *BSD, Windows, &c) that used the L4 PSE bits -- so nobody was actually vulnerable. We issued an advisory anyway, because the policy wasn't clear and we wanted to err on the safe side.

This is basically us saying to the community, "OK, do you want us to issue advisories like that, or shall we not bother you with them?"

"If this goes through, I wonder if anybody would reconsider their use of Xen based cloud solutions?"

Or, they could, you know, go and ask for the policy to be different. The blog post is a request for people's opinions, not a diktat. Even the proposed change itself says, "These guidelines are not meant to be set in stone; if they do not fit your needs as a user, please raise the issue on xen-devel." So if this passes as-is and you want to come change it later, you're still encouraged to come and ask. Good luck getting that kind of an offer from Microsoft or VMWare, or from KVM (which doesn't issue advisories of any type).

4
1
Silver badge

Re: The real reason

@gwd.

"No. If you read the full text under number 4, it goes into more detail about what this is about. XSA-176 was a bug in the way L4 pagetable PSE bits were handled. But when we looked, there wasn't a single operating system (Linux, *BSD, Windows, &c) that used the L4 PSE bits -- so nobody was actually vulnerable. We issued an advisory anyway, because the policy wasn't clear and we wanted to err on the safe side."

That's not what I said at all and I did read the full text. It's you that has failed to understand my point. You cannot possibly know if anyone is vulnerable or not as you don't necessarily know about the whole x86 ecosystem. You can check the big operating systems and variants etc., but there are all sorts of operating systems and variants out there you don't necessarily know about. So, how can you say nothing is vulnerable. Given that the Linux kernel can be modified and rebuilt by people, why could somebody not be making use of this in their variant? So, you can't know nothing is vulnerable. That was my point.

"Or, they could, you know, go and ask for the policy to be different. The blog post is a request for people's opinions, not a diktat."

That's why I said 'If this goes through'. I never said it was a diktat or that it was guaranteed to go through. I merely pondered whether anyone would choose to reconsider their use of Xen based cloud solutions IF this went through.

It seems Xen are getting very sensitive about this given the employees posting on here, who are misinterpreting peoples posts and seem to be incredbily defensive. I made a couple of reasonable points and get attacked for it...........me thinks Xen are too sensitive on this.

0
2

Yet another reason to

Use KVM

1
3
gwd

Re: Yet another reason to

KVM has a much simpler advisory policy: They don't issue any advisories at all. If you're a cloud provider, you get to find out about security issues in KVM by watching the morning news. Or by paying RedHat. Or by looking at every single commit that goes in upstream. But in any case you don't hear before it's public, so you better hope you notice before the bad guys notice.

With Xen, any cloud provider can put their e-mail address on a list and hear about vulnerabilities two weeks before they're publicly disclosed, so you can patch your systems before the world at large knows about it. And you don't pay a dime. You won't get that from any other hypervisor.

3
0
Anonymous Coward

Re: Yet another reason to

But but but KVM doesn't even have a security process, so what is your point?

What am I missing here? Or am I terribly outdated?

2
0
Silver badge

Re: Yet another reason to

@gwd.

The biggest problem Xen has is not it's security policy or anything like that. The issue is that Xen is pretty much only used by a few extremely large cloud providers who pretty much own Xen because of this. As a widespread hypervisor used by many, it has failed. I'm not being judgemental about this, or saying why or anything. Just that the user base is a few very large companies. People simply don't run Xen at home for instance.

Therefore, peoples perception of Xen is very much clouded by these other companies and peoples perception of them. Let's list a couple of them.....Oracle and IBM for instance. Now, they're really not liked at all and this reflects on Xen. Knowing that these companies have huge influence on Xen means that people therefore question Xens decisions, directions etc. as they're not really seen as independent anymore. These other companies slash costs whenever possible, even to the detriment of their users (your statements also talks about reducing costs!!) and all sorts of disliked tactics.

This is the issue for Xen.

0
1
Silver badge

Re: Yet another reason to

'People simply don't run Xen at home for instance'

Most people might not, that doesn't mean all. I do, so do others. KVM has the advantage that it's easier to install as part of a distro, its vfio infrastructure is excellent, and it supports more architectures. On the downside it's less well integrated than Xen, effectively Linux only, and pretty much necessitates external libvirt based management.

KVM feels very Linux : a set of sometimes bleeding edge components that sort of hang together, where some bits work better than others. It's also used by some large companies including, err.. IBM - remember that KVM supports S370.

Personally the criticism I've seen of Xen has come from its longevity and complexity, borne in its unique paravirtualisation origins. There's other disadvantages : complexity of pci passthrough, different levels of functionality on the various dom0 platforms (not really Xen's problem), a variety of management interfaces that are upgraded more quickly than they should be (xm was deprecated and removed in favour of xl before the more obscure parts of xm were entirely replicated in xl), and (on Linux dom0) a set of kernel options to create a successful dom0, that can easily get missed (base it on an existing working config and it's trivial).

Little of the above applies, of course, if XenServer is downloaded. It's turnkey, freely available, and comes with management tools, not dissimilar to ESXi.

2
0

Re: Yet another reason to

Sort of like how RedHat pretty much "owns" KVM? Yes, I know it's a Linux Foundation project now, but RedHat pretty much runs the show, sadly.

And yes, I *DO* run Xen at home, thank you very much!

1
0

Re: Yet another reason to

I pretty much agree exactly with BinkyTheMagicPaperclip. I think the biggest problem with Xen right now is that the code is moving somewhat too fast. Eg, PVH became HVMlite which became PVHv2 (or something along those lines). When the target is moving so fast, mistakes happen. On the flip-side, I think once PVHv2 is done and most people can completely skip the QEMU madness (except for Windows guests and such), the attack surface suddenly shrinks a lot. Right now with stub domains, however, it's still more secure than KVM.

I completely agree on KVM being a spaghetti web of different things patched together. Ironically I daresay the best KVM distribution around right now isn't even Linux-based, it's SmartOS. SmartOS hides a lot of the nastiness and makes KVM a lot easier to work with, and also more secure by isolating them off into Zones.

For Xen I've mainly used XenServer or NetBSD as a dom0, both have worked very well, although NetBSD doesn't support PVH mode currently (FreeBSD can run as a dom0 now too). For KVM, I've stuck with SmartOS. I've even added a lot of FreeBSD bhyve into the mix, which is very similar to KVM, except it doesn't use QEMU AT ALL. They all have their strengths and weaknesses. Xen's live patching is definitely a huge plus in its favor, however.

2
0

XenServer

"It's turnkey, freely available, and comes with management tools, not dissimilar to ESXi."

And as far as I know still relies on crufty old EXT3 as file system for local file storage, with all the drawbacks that come with it.

Oh, and building a local ISO repository is still torture.

XenServer is nice if you have large external storage arrays but on servers with local storage (pretty much the default for small and medium size businesses) it's nowhere near ESXi.

0
1
Silver badge

“Issuing advisories has a cost”

Wrong, wrong, wrong.

Writing bad code has a cost. The cost of putting it right later.

Not fixing things has a cost. The cost of lost customer confidence when things break.

Keeping quiet has a cost. The cost of lost customer trust.

And if you're cutting corners on bug-fixing, where are other corners being cut ?

1
2
Silver badge

Please stop this ill informed and click bait 'journalism'

Look what you didn't quote from the mailing list post, that I've only just read :

'every advisory has the risk that it will be picked up and blown out of proportion by the media'

It's looking very much like Thereg is deliberately targeting Xen, possibly because the team create nicely formatted advisories and spend the time to explain issues.

There are no recent articles complaining about KVM, despite the fact it also has security issues, and I remember few reports of VMWare problems.

Xen is not perfect by any means, but put this in perspective, please. I've read the post now, and the discussion is not entirely unreasonable and is a starting point, not a policy. Try marc.info, it's easier to read : http://marc.info/?t=148353353000001&r=1&w=2

There is NO link to security statements on linux-kvm.org. There is on xenproject.org.

6
0
(Written by Reg staff)

Re: Please stop this ill informed and click bait 'journalism'

A few of things to address here.

The Reg is not targeting Xen. But when we see nasty bugs in anything we think readers are likely to use, we report them. When a stream of nasty bugs appears in the same code, we report the ongoing issues. Look at our reporting on Google Cloud outages.

I missed the comments about the media because I based this report on George Dunlap's blog post and posts to xen-devel in recent days. I glanced at the Jan 5 thread that contains the comments about media, but realised this month's thread was more recent and therefore directed readers to it.

If Xen thinks we're going overboard with our coverage of its bugs, the folks there know where to find us. Indeed, George Dunlap has posted here a few times on this story alone!

On the charge of clickbait, I've never seen a project, commercial or open source, make a suggestion like this. That makes it newsworthy. And believe me there's about a zillion ways to headline this story to make it really scary and negative. I can imagine others going with "Amazon's core open source cloud engine wants to hide security problems". THAT's clickbait. IMHO this story quite soberly reports the blog post, quotes liberally from it, shows readers where they can learn more, tries to tell a story about the debate

0
1
Silver badge

Re: Please stop this ill informed and click bait 'journalism'

I'm sure you are reporting issues, but being a virtualisation desk, it's certainly possible to be more proactive about covering all products. What I'd expect from a journalism site is not just the prepackaging of a public blog post, but searching out details of information that is not immediately obvious, such as the issues in KVM.

Whilst I accept that you have not claimed that KVM is any better, it is an unfortunate part of human psychology that people will believe that the thing not being actively criticised (technology, politics, whatever) is an improvement over the thing being criticised. Additionally, a number of options in the poll could be considered leading questions.

There's little criticism of projects like Qubes, who keep complaining that they have to handle Xen issues in their OS, when as far as I can see (please, either you or Xen people correct me) they contribute very little upstream. In the time when they've been complaining, FreeBSD had implemented their own KVM like hypervisor (bhyve) from scratch.

On that note it would be nice to see some coverage of bhyve other than in FreeBSD release announcements, as from what I can see it's really quite usable now. The occasional vmm (OpenBSD) update would be nice too, but that is currently in a state of flux, and mostly only usable for running OpenBSD inside itself.

1
0
gwd

Re: Please stop this ill informed and click bait 'journalism'

FWIW Simon, most of us in the XenProject thought the story itself was unbiased and informative, and we appreciate the "signal boost" to get more people to weigh in on the discussion.

0
0
Bronze badge

Maybe a cup of tea will help

People use "issuance of advisories" as a proxy for "taking security seriously" but they are definitely not the same. However, the onus is on the Xen project team to prove that. It's not enough to say, "well, of course we do!" They need to show by their actions that this is the case. For example: a security audit of the code; participation in CTFs/bounty programs; partnership with major cloud folks to review Xen security. And so on.

By way of comparison, the OpenBSD team are constantly fixing issues with potential security impact but they do not issue CVE advisories on each one: however, their active reputation (in no small part earned by activities such as the above) still leads them to be highly rated in the security space.

0
1

Re: Maybe a cup of tea will help

"However, the onus is on the Xen project team to prove that. It's not enough to say, "well, of course we do!" They need to show by their actions that this is the case. For example: a security audit of the code; participation in CTFs/bounty programs; partnership with major cloud folks to review Xen security. And so on."

Thank you for the suggestions. On the topic of bug bounties, basically every large vendor and contributor to the Xen Project runs their own bug bounty program as can be seen on https://firebounty.com. There are also regular security audits of sections of the code by various vendors. The last batch of XSAs in late Nov came out of such an audit. We are also integrating fuzzing into our CI infrastructure, etc.

But maybe the project does need to take more of an active role in CTFs/bounty programs and such things and/or communicate better what we do.

1
0
Anonymous Coward

@ Xen Project Team

"The Xen Project is asking if it can disclose fewer bugs. If no operating systems are vulnerable to a bug, no advisory will be issued."

First thing first, I'm no pro on your development, but as open source, that's a really terrible approach. When your team cherry pick bugs to be disclose, you're basically hiding issues and creating a trust issue with your client and other supporters. It'll take one disregarded vulnerability disclosed by users as Xen being incompetent for the project trip to the bottom of the hill.

I will first assume that the current security advisories included every single bug. If your team problem is about issuing security advisories cost, then put the spending in the right place and tell us about it.

Your team noted this, "All security issues are bugs, but not all bugs are security issues.", which is exactly the approach you team should move forward. Like other big companies and team, they have bug logs, fix and hot-fix. They categorize the types in to priority, and Xen team should do the same. Xen team should log all bugs, but note high priority security issues bugs proceed to the security advisories. It is a great way to present that your team understand the number of issues and priority including resource management and being competent.

But no, your team should never try to hide bugs. We will know at one point and that would be the end of Xen development.

1
1

Re: @ Xen Project Team

> I will first assume that the current security advisories included every single bug

Currently security advisories are only issued for bugs that have been identified as security vulnerabilities which is in code that is not experimental or in preview. Whether the answer to your question is yes or no, depends on what you mean by every single bug.

> When your team cherry pick bugs to be disclose, you're basically hiding issues and creating a

> trust issue with your client and other supporters

That would certainly be true, if we changed the policy without consultation, or with consultation and lack of consensus and just introduced different criteria.

The proposal does not suggest that we would not fix bugs. What it does suggest is that we would fix issues for some security bugs without pre-disclosure and without issuing an XSA. That is not unusual approach: for example OpenStack does this (see https://security.openstack.org/vmt-process.html#incident-report-taxonomy - it only issues an advisory for some issues and a security note for others, using a more lightweight process)

> But no, your team should never try to hide bugs.

Well, that is not what we were asking or trying to do. Although I admit that it may have come across like it.

1
0

kvm.

xen is a dying ..why not kill if off once and for all.

0
1

Re: kvm.

er.... AWS?

0
0

Re: kvm.

er.. AWS is dying?

0
0

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

The Register - Independent news and views for the tech community. Part of Situation Publishing