Now that we know AoE is ATA Over Ethernet, a primer
Now that we know AoE is ATA over Ethernet, isn't broadly supported, lacks enterprise features, isn't generally supported and generally isn't Enterprise Ready, let's talk about the other two. Let's have a primer for the gallery.
Fiber channel (FC) is the grandaddy of shared storage tech. It once was the fastest available transport method (back at 1Gbps or 2Gbps). Its reliable packet delivery offers a firm foundation on which to build your enterprise tech. People who build out FC care more about reliability than anything else, so they usually populate two HBAs, each of which provides a link to at least two FC switches - themselves connected in a mesh network, with each storage appliance (SAN) likewise at least dually connected, through dual SAN controllers to drawers all the way down to redundant connections to individual drives. The idea is that you can wipe out one entire path of controller, fiber, switch, fiber, controller, fiber - and still not lose your connection to an individual drive. The drives are then striped and/or mirrored for additional redundancy at the media level, and in some cases can even be striped or mirrored across SANs that are geographically separated for the ultimate in storage reliability. All of the connections (or nearly all - some vendors cheat) involve interconnections using Fiber Channel protocols that guarantee reliable end-to-end and in-order delivery of packets. Typically the connections between boxes are fiber-optic.
Because it's rarer than common server hard drives and networking, FC is expensive. It involves a large number of patented and licensed technologies. It has garnered a certain high-end storage following, and it's well deserved. Many of the storage technologies that will follow in this discussion arrive from the need to maximize the benefit of very expensive FC disk, or to work around the idea that bringing your storage offline "just isn't done". Price really isn't much of a consideration with FC folks. You might pay $3,000 for a 600GB 15K rpm 4Gbit FC Hard drive, for example if you were looking for the IBM 59Y5460. FC now can use interconnects in the market that are 8Gbit/second between servers and switches or switches and storage, and these links can be aggregated for arbitrary amounts of bandwidth.
FC is a stable market. It's got reliable some % growth year over year in business, but it's not sexy, it's not new. Most people who aren't FC shops aren't looking to sign up for that drill.
iSCSI is a different technology invented by Microsoft - one of the few things they've made standard that I'm in favor of. It's possible to borrow some of the same ideas from FC and build in the same path redundancy as Fiber Channel, all the way to the drive - but this is a fairly recent development. iSCSI goes over Ethernet, which succeeded in the networking market despite the fact - perhaps because of the fact - that it DOES NOT guarantee either reliable end-to-end delivery nor in-order packets. Ethernet now gives commonly available links that run 10Gbits/second but with packet overhead it's a wash because it amounts to 8Gbit/sec actual bandwidth. The technology in iSCSI that allows for divergent paths to the disk is called Multipath I/O (MPIO). It requires special drivers in the OS, special configurations and recent versions. It requires validation testing, and frankly the technology is still a little bit fresh for environments where human life and safety are at stake. Like FC, connections between boxes can be aggregated for arbitrary bandwidth.
iSCSI works off of the principle that the aged SCSI bus protocol required that information be organized into packets so they could pass over the parallel SCSI bus. On this bus there were multiple drives and they needed to cooperate with the controller to avoid crosstalk. iSCSI essentially encapsulates the packets for transport over Ethernet, and adds a few features. We don't use the parallel SCSI any more, but its organization can still be useful. The client side driver and the storage device software negotiate reliable end-to-end delivery and in-order execution of writes and reads across the unreliable Ethernet connection using their own intelligence. Using modern iSCSI capable Host Bus Adapters rather than standard Ethernet adapters on the server side allows for boot-from iSCSI SAN as well as offloading of the processing overhead from the CPU. Enterprise iSCSI devices can still be expensive. A Dual-Port Serial Attached SCSI (SAS) 6Gbps (6G) drives at 600GB lists today for $809. That's not cheap, but it's better. You can also buy a cheaper and slower dual-port 6G SAS drive with a short warranty from that vendor that has 2TB for $949, which is coming somewhat closer to consumer SATA technologies and prices. They're cheaper because these drives are sold in vastly larger quantities as directly attached storage in millions of servers. The Dual-Port thing is an important part: it allows for backplanes that provide independent links to independent SAN controllers to complete the last leg of a redundant connection all the way from the server bus to the drive. Because it's more common and so leverages economies of scale, Ethernet switching generally costs less - but we're talking about 10Gbit Ethernet here and it's not as widely deployed as it might be so this is still not a small business use case yet. 10Gbit Ethernet can go over copper to the top-of-rack, or a few racks over - and save a lot of money doing it - but it's not an end-of-row solution. If you need 10Gbps Ethernet for more than a few meters, you're going to buy the expensive fiber GBICs. Most folks are still exploring 10Gbps Ethernet, and frankly if you can isolate this cost center as closely as possible to the servers, that's a good thing. We're not ready to go 10Gbit to the desktop yet. Because the abstractions used to maximize the return on investment for FC SANs are just software, the software has been ported to iSCSI SANs to provide the same amplifications. This includes things like synchronous replication, asynchronous replication, clustering, virtual volumes, thing provisioning, snapshots, clones and so on (the SAN Features).
There are even companies sell devices to insert into your FC + iSCSI network that serve to take the block storage that you have whether it's iSCSI or FC, ignore its special features, and provide all of the SAN Features so that you can continue to use the underlying storage using your preference of iSCSI or FC while you migrate from one vendor or technology to another without doing the (gasp! Forbidden thought!) unforgivable of bringing your storage offline. This is called Storage Virtualization.
iSCSI is a growth market. It's growing multiple x per year. Some say 10x.
Convergence: And then we have Converged Enhanced Ethernet (CEE), DataCenter Ethernet (DCE) or FCoE (Fiber Channel over Ethernet). These are all names for the same thing - at least they became the same thing when the standard was announced a few months ago. The multiplicity of names was from vendors attempting to take ownership of the symbol space leading up to the standard. At the moment this is a technology that exists between the server and the first switch that it encounters. Using this technology, which I'm going to call FCoE to be simple, the server has a FCoE Host Bus Adapter that provides a reliable and in-order packet to the switch. All of the HBAs I've seen operate at 10Gbps and have two ports for a net 20GBps. Typically each port can be configured to be presented to the server's I/O bus as MULTIPLE separate hardware devices which are either Ethernet or Fiber Channel. With FCoE you can do it either way, or do it one way now and migrate to another way later. You get to choose. You can adjust the fraction of bandwidth allocated to storage vs networking based on your need. This is a powerful choice.
The trick with convergence is that it ends at the first switch. There's no switching or meshing standard for FCoE yet. The first switch MUST break out the virtual connections into Ethernet or Fiber Channel, and pass them to their respective networks from there. Instead of simplifying things at this point it just adds a third network. But this can be useful for some things, depending on your needs. The Cisco Nexus 5000 is typical of the device you would use as a top-of-rack switch for this sort of application, but it's one of several.
There are FCoE HBAs you can use that connect with this first switch using a copper connection. This can save a lot of money, as Fiber GBICs (the module that converts the electrical signal to light and back again) cost a good deal of money and you need a pair of them to terminate both ends of a fiber connection. The Copper cable includes integrated GBics on each end and can be had for a couple hundred dollars. Fiber GBics generally start at about $800 each, or $1600 for each path. Remember, you need pairs of paths.
And then there are blade servers, some of which now include a pair of 10Gbit FCoE HBAs on the motherboard by default and avoid all the expense and connections, at least for sets of up to 32 servers at a time. The blade server chassis can provide the first-hop FCoE switch, and a lot of the complexity of the problem goes away, as does the need for HBAs, the vast majority of the GBICs, almost all of the cables. This is VERY important. I was taught nearly three decades ago in IT that if there's a problem, 90% of the time it's the cables, and my experience has held true to that lesson.
OK, that's enough of a primer. If this post passes El Reg moderation I'll discuss current solutions in this space.