Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking

The Journey to Convergence
Stuart Miniman, Office of the CTO

The iSCSI Story

  • Transport SCSI over standard Ethernet
  • Reliability through TCP
  • SCSI has limited distance, iSCSI extended the distance

Non-Ethernet Convergence Options

  • Infiniband
    • Used broadly for High Performance Computing (HPC) environment
    • Low cost and ultra-low latency geared for server to server cluster
    • Separate use from general network (Ethernet) or storage (FC or Ethernet)
  • PCIe
    • Extension of the server bus to an I/O aggregation box
    • Not a standard, small players
    • Still using Etherhet and FC network and storage from an aggregation box

Maturation of 10Gb Ethernet

  • Allows replacement of n x 1Gb with must smaller number of 10Gb adapters
  • Single network allows for easier mobility for virtualization/cloud deployments
  • Simplifies server, network and storage infrastructure


  • 40Gb and 100Gb Ethernet (IEEE) standards will be completed in June 2010
  • 16Gb FC (T11) standard is targeted for completion at the end of 2010
  • 32Gb is in the works
  • Server Adoption of FC ~3+ years, of Ethernet ~5+ years

Protocols and Standards
Fibre Channel over Ethernet

  • Developed by T11, International Committee for Information Technology Standards (INCITS) T11 Fibre channel Interfaces Technical Committee
  • FC-BB-5 standard ratified in June 2009

Converged Enhanced Ethernet

  • Developed by IEEE Data Center Bridging (DCB) Task Group
  • Commonly referred to as Lossless Ethernet
  • IEEE standards targeting ratification mid-2010

iSCSI and FCoE Framing

  • iSCSI is SCSI functionality transported using TCP/IP for delivery and routing
  • FCoE is FC frames encapsulated in Layer 2 Ethernet frames over Lossless Ethernet

FCoE Frame Formats

  • Ethernet frames give a 1:1 encapsulation of FC frames, no segmenting FC frames across multiple Ethernet frames


  • Next step
  • Not required for multi-hope FCoE or other current deployments
  • Likely to support point-to-point configuration which allows two FCoE devices to communicate without going through an FCF (or switch)
  • Today initiator cannot talk directly to a target without a switch in between. FC-BB-6 is investigating this.

Lossless Ethernet

  • IEEE 802.1 Data Center Bridging (DCB) is the standards task group
  • Converged Enhanced Ethernet (CEE) is the industry consensus term
  • Link level enhancements (Priority Flow Control, Enhanced Transmission Selection, Data Center Bridging Exchange Protocol) are shipping in products today
    • PAUSE and Priority Flow Control
      • Classic 802.3x PAUSE is rarely implemented since it stops all traffic
      • New PAUSE known as PFC that can halt traffic according to priority tag while allowing traffic at other priority levels to continue. This creates lossless virtual lanes.
    • Enhanced Transmission Selection
      • Maintain low latency treatment of certain traffic classes
    • Data Center Bridging Exchange Protocol
      • Auto-negotiation for devices as they determine the link parameters
  • The CEE cloud or DCB-enabled LAN is only for the portion of your network that requires lossless Ethernet

Beyond Link Level

  • End-to-end
  • Congestion notification
    • IEEE 802.1Qau ratified
    • Allows a switch to notify attached ports to slow down transmission due to heavy traffic
  • Layer 2 multipathing
    • IETF TRILL – Transparent Interconnection of Lots of Links
    • Used with STP to provide more efficient bridging and bandwidth aggregation
    • Focuses on bridging capability that will increase bandwidth by allowing and aggregating multiple network paths

Solution Evolution

  • iSCSI was >15% of revenue ($1.8B in 2009) and >20% capacity in SAN market in 2009
  • 10Gb iSCSI solutions are available
  • iSCSI natively routable (IP)
  • iSCSI solutions are much smaller scale than FC


  • FCoE with direct attach of server to Converged Network Switch at top of rack or end of row
  • Tightly controlled in 2009
  • First solutions are with FcoE aware Ethernet switch (FIP snooping)
  • Blade > FCoE switch > FC switch > Storage

Rack Area Network (RAN)

  • Cisco UCS
  • Vblock
  • VCE (virtual computing environment)


  • End of year, you will see FCoE native in CX and DMX

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s