End-to-End Communications with Advanced Fiber Optic Technologies

By TE Connectivity

More Data = More Energy Consumption

The emergence of faster data rates to keep pace with the demands for data poses a number of technical challenges. While this is not a new or radical thought, the radical conversation on how to combat these challenges—specifically when it comes to the resources, like energy—is imperative to fuel this data deluge. Among the problems that became apparent even at 10 Gbps is the power consumption of copper-interconnect signals within systems—and the problem is compounded at 25 Gbps data rates. Power consumed by the networking equipment alone significantly contributes to the overall energy consumption problem of data centers that is estimated at 30 Billion watts worldwide. ¹

The energy use of data centers is already staggeringly large and estimated to be more than 2% of total U.S. electricity consumption. Networking equipment consumes about 50% of a typical data center’s energy. Air movement and cooling equipment consume about 37%, transformers and uninterruptible power supplies account for 10% and lighting and other items take another 3%.²

The U.S. Department of Energy recognized the impact of massive data processing and storage many years ago. In addition to the recently updated Version 2.0 ENERGY STAR specification for Computer Servers that took effect on December 16, 2013, Version 1.0 ENERGY STAR specification for Data Center Storage was also finalized and took effect on December 2, 2013. Purchasing ENERGY STAR rated equipment should significantly lower a data center’s energy consumption and improve the bottom line by reducing energy costs. Acquiring an ENERGY STAR rating should help server and storage equipment suppliers differentiate and increase the sales of their products. More efficient data transmission can play a role in obtaining the ENERGY STAR rating.

Concerns for higher efficiency in data centers go beyond government organizations. For example, the Open Compute Project initiated by Facebook targets greater efficiency in servers and data centers. This type of industry interest places even further demands on more-efficient data communications that can benefit from the use of fiber optic technology. For high-speed communication beyond the first few meters, optics is a field-proven, energy efficient transport mechanism. Figure 1 shows the relative energy efficiency of optical and electrical links.

With core networking doubling every 18 months or so and server I/O density doubling approximately every 24 months (source: IEEE802.org), delaying the inevitable transition to higher-speed data transmission capability could prove costly for many companies.

Figure 1: Comparison of energy efficiency for electrical versus optical links.

Coolbit Optical Engine Product Technology

Advancing the capabilities of fiber optics to accommodate the demands for 25 Gb/s and beyond is the challenge that has been met by TE Connectivity’s (TE) advanced Coolbit optical engine. This engine satisfies high-density and high-bandwidth requirements while running at about two-thirds the power of conventional solutions.

The solution is based on the fully vertically-integrated fabrication of the Coolbit optical engine at TE’s facility in Stockholm, Sweden. This development process begins with the semiconductor fabrication of the VCSEL and Photodiode ICs; moves to the automated wafer assembly of the VCSEL, photodiode and other ICs; and ends with operational testing of the wafer that results in the final engine.

Using in-house, automated wafer-level assembly, TE’s Coolbit optical engines are manufactured to semiconductor quality and reliability levels with passive self-alignment of VCSEL, photodiodes and ICs. These optical engines are optimized to perform at the emerging requirements for 25 Gb/s performance

More and Faster Data

"Big Data," a term that encompasses many of today’s technology buzzwords, including the Internet of Things (IoT), machine to machine (M2M) and wireless communication, cloud computing and more, is driving the need for more bandwidth. But the demand for more bandwidth is nothing new. So what could compel a transition to fiber optics for both inter- and intrasystem high-speed communications that has been predicted for over twenty years?

Video: Take a video tour of our Coolbit Optical Engine fabrication process.

LightCounting, a market research firm focusing on high-speed interconnects, reports in "Embedded Optical Modules" (May 2012): "At 10 Gbps signaling, the problems were manageable but as the industry transitions to 25G signaling, significant signal losses and issues are surfacing. As speeds increase to 25GHz, the number of signal compensating electronics needed is skyrocketing along with costs."

As a result, an emerging product segment has been created that allows system designers to embed optical transceiver technologies inside computer and communication systems. Embedding high-speed optical transceiver technologies, known as mid board optics (MBO), onto traditional server line cards or switch fabrics allows system architects to achieve:

• Higher input/output densities
• Systems that are not bound by copper interconnect lengths
• More power-efficient systems

MBO inside systems mitigate the added electrical losses encountered at the 25 Gbps signaling rate. Figure 2 shows a comparison of faceplate density when optics move off the faceplate and onto the system’s printed circuit board. The top image shows maximum pluggable I/O density, based upon 400 Gbps CDFP active optical cable assemblies. The bottom shows the benefit if I/O is based upon optical connectivity. The optical solution results in substantially higher electrical I/O density while eliminating the cooling problem at the faceplate.

Figure 2: Faceplate airflow as well as power and heat requirements provide a driving force for intra-chassis MBO that delivers increased signaling density and front panel capacity.

The top design in Figure 2 has 22 CDFP MSA x 400 Gbs = 8.8Tbs* (*bidirectional calculation) faceplate capacity. This can be extended to 22 CDFP MSA x 640 Gbs = 14.1 Tbs*. Changing from front panel pluggable modules to an MBO coupled to optical front panel connector approach achieves the same data capacity, but preserves precious front panel space for additonal functionality (a demonstration of the MBO approach was presented at the OFC 2013 tradeshow; see Figure 3.)

Figure 3: TE's mid board optics demonstration from OFC 2013 tradeshow.

The mid-board optics modules shown in Figure 3 and 4 provide the key to eliminating the faceplate density problems and the attendant heat management problems. TE’s MBO module is a 12-channel transceiver capable of transmitting and receiving 300 Gb/s. The electrical interface is provided through a land grid array (LGA) socket on the optical module side and a ball grid array (BGA) on the host board, and allows modules to be placed on a 1-inch grid. The high-speed inputs are AC-coupled to a floating input termination. On the Rx side, the incoming optical signal is converted to a current by the PIN diode. The output stage is current-mode-logic and provides 50 Ohm back-terminations.

Figure 4: Mid-board optics modules shown here provide the key to a high-density interface.

Figure 5 shows the system level impact of optically enabled system design, enabling high multiterabit systems: horizontal line cards with 28 Tb/s of IO connected to onboard MBOs, managed fiber solutions on the line cards via optical flex circuits, and an optical backplane capable of more than 900 Tb/s of interchassis interconnectivity.

Figure 5: Passive optical backplane development for intra-chassis optical interconnectivity.

Faster and Denser Pluggable Optical Module

Active optical cable assemblies (AOCs) embed the high-speed optics (Coolbit optical engines) behind two transceiver ends and deliver an electrical interconnection to the other system electronics (Figure 6). This design enables very-high-speed and high- aggregate data rate links at costs significantly below those of separate transceivers and fibers. AOCs offer the benefits of optical with the ease-of-use of copper.

Figure 6: CDFP module with four parallel bidirectional 25.78 Gbps channels.

TE offers two core AOC solution sets: CDFP and QSFP28 form factor. Shown in Figure 6, the CDFP supports 400 Gbps biEnd directional data transport in a 16x25 Gb/s configuration. It is a front/rear panel pluggable module that consumes less than 6 Watts per end—with all signal integrity features enabled. This provides the highest density per panel area compared to other competing form factor products resulting in some of the lowest power per gigabits and the highest gigabits per square inch, at 444 Gb/in ².

TE offers two core AOC solution sets: CDFP and QSFP28 form factor. Shown in Figure 6, the CDFP supports 400 Gbps biEnd directional data transport in a 16x25 Gb/s configuration. It is a front/rear panel pluggable module that consumes less than 6 Watts per end—with all signal integrity features enabled. This provides the highest density per panel area compared to other competing form factor products resulting in some of the lowest power per gigabits and the highest gigabits per square inch, at 444 Gb/in ².

The QSFP28 supports 100 Gbps bidirectional data transport in a 4x25Gb/s configuration. It is a front panel/rear panel pluggable module that represents an IO density of 240 Gb/in2, while consuming less than 1.5 Watts per end with all the signal integrity features enabled (channel equalizer, transceiver and receiver clock data recovery). A QSFP28 pluggable transceiver and interface to industry-standard MPO connections are also available. Both CDFP and QSFP28 AOCs and the QSFP28 transceiver devices are built with TE’s Coolbit optical engine technology.

Figure 7: QSFP28 Transceiver

Figure 8: QSFP28 AOC

In addition to having a higher data density and longer transmission distances, a fiber optic link results in significant size and weight savings. Fiber interconnects are about one-fortieth the weight of twinax cabling and consume one-thirtieth the packaging volume per unit length of twinax. Coupling these advantages with large available bandwidth (MM >400GHz & SM >10 THz) and fiber solutions represents multigenerational future-proof architectures. Fiber can accommodate future-generation technology insertion of faster optical transceiver and more data per fiber via wavelength division multiplexing (WDM).

System-level thermal advantages take two forms. First, the lower power consumption needed for optical transmission (power/bit-meter) is the primary thermal advantage, a direct result of the Coolbit optical engines. Less power dissipation results in less power needed in the conversion to drive the optics; and less heat that needs to be extracted by system cooling mechanisms. Second, the high-density packaging and low cross-section of optical fiber produces less obstruction to system cooling, so cooling schemes are more efficient compared to solutions that encounter airflow restriction with high-density copper cables.

Industry Leading Power Efficiency

Power consumption is critical to high-speed data transmission. For the same performance in terms of data transmission, lower power consumption reduces the cooling load level that end customers need to provide, which reduces their energy bill and overall expenses. With some of the lowest-power-consumption products in the industry, TE’s fiber optics solutions help minimize these energy costs. For example, the target power consumption on the CDFP product is 6 Watts, for the QSFP28 at 100 Gbps it is 1.5 Watts and for the mid-board module it is 4.5 Watts. For comparison, MSA suppliers are currently targeting the emerging standard for the CDFP product for a maximum of 8 Watts. Additionally, the resulting lower energy consumption could help achieve an ENERGY STAR rating for further product differentiation.

TE also produces 10 and 14 Gbps VCSEL and PIN devices, enabling a QSFP transceiver and AOC portfolio including InfiniBand double data rate (DDR), fourteen data rate (FDR) and enhanced data rate (EDR) products.

Demonstrated Performance

Key to the performance of all the new module products is the 25 Gbps VCSEL and PIN devices in the Coolbit optical engine. The performance of 25G optical components was measured using a bit error rate (BER) tester and digital communication analyzer in the test set-up shown in Figure 9.

Figure 9: The measurement set-up used to verify the performance capability and signal integrity of the zQSFP module.

The resulting eye diagrams and bath-tub curves are shown in Figure 10. They show little change at 25°C and only slight degradation at 85°C.

Figure 10: Eye diagrams of 25.78 Gbps data transmission over the entire range of 25°C to 85°C of module case temperature using OM3 fiber.

Figure 11 shows 25 Gbps optical components performance measured to BER < 10-15="" for="" 114="" m="" of="" optical="" cable="" at="" 25="" and="">

Figure 11: The 25.78 Gbps bath-tub curves for data transmitted through 114m of optical fiber with no CDR at 25°C and 85°C.

The benefits of fiber optics inside systems are offset by critical challenges that have been barriers to widespread adoption. The first and main challenge for optics is that the acquisition cost of optical solutions is higher than copper solutions, with the optical transceiver being the dominant cost element of an optical link. TE’s vertically integrated value stream combined with its wafer scale fabrication of Coolbit optical engines can help reduce the up-front costs. Several studies have shown the total cost of ownership of optical systems to be lower than comparable copper-based systems; these studies mostly refered to large-scale supercomputing systems.3 There are strong similarities between supercomputer sites and the hyperscale data center.

The second challenge is the ease of use of optical connectors versus tried and proven plug and play electrical connectors. This next section will cover this second challenge in detail and dive into passive fiber technologies.

The Passive Story

In order to establish a connection, optical fiber is typically terminated into a ferrule that provides precise mechanical registration of the fiber end face. The fiber end face is then polished and inspected to exacting standards. The fiber ferrule assemblies are then built into a connector housing/packaging. The tolerance and precision in optical connector manufacturing is well beyond that of electrical connectors. This precision assures minimal signal loss and low back-reflected signal. To achieve good optical connector quality, the end face must be cleaned and inspected before physically mating in the field. The predominant optical connection mechanism is a physical contact that holds the two butt-coupled fiber end faces (held in ferrule) together under spring tension. Although physical contact connectors have been widely deployed, users have expressed interest for a "better optical" connector system. Some of the attributes users’ desire include:

• Better immunity to dust, dirt, end face contamination
• Less cleaning and inspection
• Lower mating force
• Lower cost per connection

All of these enhancements result in lower operating cost and total cost of ownership.

TE has addressed these issues by designing a highly reliable and easy to use expanded beam interconnect. Shown in Figure 12, an expanded beam interconnect delivers on the improvements users expect in a better optical connector. A lens takes the light traveling through the fiber and expands the beam to a larger area. (See Figure 12.) As a result, a speck of dust at the interface has a significantly reduced effect on the amount of light transmitting from a lensed connector, providing much greater immunity to dust and dirt, producing a more stable optical performance over a lifetime with less need for cleaning, replacement and system downtime.

Figure 12: Collimating expanded beam.

Because there is no physical contact at the end face, expanded beam interconnects do have slightly higher insertion loss. Expanded beam interconnects are more tolerant to alignment tolerances and have 5x lower mating force required. Expanded beam ferrules are precision micro-molded devices that incorporate a collimating lens profile. This ease of manufacturing follows into the cable assembly process by eliminating end face polishing and end face interferometry, resulting in a simplified manufacturing process that can be automated to reduce system cost.

System Connectivity

In addition to a comprehensive portfolio of fiber optic connectivity components and new high-performance active components, TE is developing high-density connectors focused on in-the-box, box-to-box, and rack-to-rack links. These are enabled on the face plate by cost-effective MT-compatible connectors and in the mid-plane/back-plane by blind-mate connectors. To achieve the vision of making optics as easy to use as copper, these connectors are designed to be low-maintenance. Problems that are solved using the systems approach and the newly introduced active components are:

1. Increased data density (at front panel) bit/rate per channel for a 19-in. rack
2. Reduced power consumption that reduces cooling load level required: for the same performance - less power
3. Reduced system cost essential to make the concept design acceptable

With the array of technologies required to handle an end-to-end system design that moves data from outside the box to inside and then across the inside of the box, many system houses would be significantly challenged to produce this type of intra-chassis interconnectivity.

How it is Done

As demand on bandwidth expands, the fiber optic architecture becomes more complicated and the increased density can become disorganized. An automated lower cost fiber routing interconnect system, which manages by automation the complex fiber routing structure, provides the solution. Optical flex circuits transfer the ownership of the complicated fiber routing from the installer to the engineering design house, producing less complex installation, decreasing routing errors, providing a space savings, and ultimately producing a system cost savings with a poke-a-yoke design. The combination of expanded beam ferrules polishing and automating the fiber routing simplifies the assembly and installation process of the connector system.

The connector system delivers improved packaging density with low service requirements based on the non-contact, expanded beam design. When optical fibers are connected to an optical engine, the devices receiving light are very small, requiring precise placement of the fiber coupling element to the substrate that has the optical detector on it. With the lens design, the system is far more tolerant of manufacturing variances. Figure 13 shows TE pioneering expanded beam ferrule design (TELLMI). The TELLMI product design inlcudes the following attributes and innovations:

• Single-piece design integrated precision V-groves to position the fiber
• Recessed lens array to provide protection for lens damage
• Laser cleaved fiber for a more consistent optical path
• Ultraviolet (UV)-curable adhesive for reduced termination time

Figure 13: Features of TE Connectivity's TELLMI connector.

Figure 14 below shows two mated-lens interconnects. Note the alignment (guide) pins and sockets for easy insertion.

Figure 14: The lens array in the TELLMI connector can handle twelve separate fibers.

In summary, the advantages of expanded beam design include:

• No physical contact
• Dust insensitivity
• Greatly reduced mating force
• Highly automatable
• Need for less cleaning
• Less susceptibility to physical end face damage
• Interoperable with existing MT Format

Figure 15 shows a lensed connector’s tolerance of dust and dirt and the impact of a simple yet effective cleaning mechanism. When the expanded beam design approach is applied to socket-able mid board optics, the MBOs become pluggable modules that are field-replaceable inside the box. As a result, the lensed connector technology overcomes both of the disadvantages of the optical backplane.

Figure 15: Dust performance of TELLMI product lens interconnect.

The Future Proofing Approach: TE's Optical End to End Integration

With the inherent performance advantages of fiber optics, the key to optimum systems-level solutions involves the appropriate integration of active and passive technologies. TE is in the unique position to deliver optimum solutions with its active optical transceivers, passive fiber connectivity and traditional copper interconnect products and expertise.

As an alternative to users becoming masters of all the technology aspects of fiber optic systems themselves, with its system-level signal integrity design expertise, TE brings end-to-end system capability to high-speed data transmission (Figure 16). Users now have options to make the transition from copper to fiber that is necessary to keep pace with market needs.

Figure 16: TE Data Communications provides end-to-end connectivity with the appropriate mix of copper and fiber technologies.

As one of the few vertically integrated companies, TE can enable the complete end-to-end solution without a handoff in the middle. TE’s comprehensive roadmap for active and passive optical products supports all major form factors for the 25G generation.

Developed and manufactured in house, Coolbit 25G VCSEL and PIN based MM transceivers deliver exceptional performance today and provide a strong foundation for next-level integration. With fiber optics and copper-based components from TE, partial implementation—fiber optic/copper hybrid solutions— are possible. As a result, upgrading to fiber optics performance is not an all-or-nothing decision and there is a migration path from all copper to copper-fiber hybrid to all fiber connectivity.

As Dale Murray at LightCounting observes, "Most people will admit that there is a point at which optics just makes more sense. For each application, it is a matter of what is the timing?"

If you are ready to evaluate your options, TE is ready and can provide end-to-end communication solutions for optimized performance at increasingly higher data rates.

Footnotes

¹New York Times - Data Centers Waste Vast Amounts of Energy Belying Industry Image

²EPA Creates Energy Star Spec for Data Center Storage Equipment (10/7/13) http://www.energymanagertoday.com/epa-creates-energy-star-spec-for-data-center-storage-equipment-095953/

²Measuring Power in your Data Center: The Roadmap to your PUE and Carbon Footprint https://www.bicsi.org/pdf/Regions/ireland/BICSI%20Presentation%20-%20Raritan.pdf

³Alan Benner - High-Bandwidth Integrated Optics for Server Applications

References

LightCounting Group (www.lightcounting.com)

Marek Chaciński, Nicolae Chitica, Stefan Molin, Nenad Lalic, and Olof Sahlén, "25.78Gbps Data Transmission with 850nm Multimode VCSEL Packaged in QSFP Form Factor Module," Optical Fiber Communication Conference

Anaheim, California United States, March 17-21, 2013, http://www.opticsinfobase.org/abstract.cfm?uri=OFC-2013-OW1B.1