WO2016037262A1

WO2016037262A1 - Low latency optically distributed dynamic optical interconnection networks

Info

Publication number: WO2016037262A1
Application number: PCT/CA2015/000486
Authority: WO
Inventors: Yunqu Liu; Kin-Wai LEONG
Original assignee: Viscore Technologies Inc.
Priority date: 2014-09-09
Filing date: 2015-09-09
Publication date: 2016-03-17
Also published as: WO2016037262A8

Abstract

Within data centers the ratio between intra-data center traffic to external traffic can be as high as a 1000:1 on a single simple request. Within data center's 90% of the traffic inside data centers is intra-cluster. The prior art Folded Clos topology deployed scales cabling complexity as a quadratic function of the number of nodes. Accordingly, it would be beneficial for new fiber optic interconnection architectures to address the traditional hierarchal time-division multiplexed (TDM) routing and interconnection and provide reduced latency, increased flexibility, lower cost, lower power consumption, and provide interconnections exploiting N x M x D Gb/s photonic interconnects wherein N channels are provided each carrying M wavelength division signals at D Gb/s.

Description

LOW LATENCY OPTICALLY DISTRIBUTED DYNAMIC OPTICAL INTERCONNECTION NETWORKS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of U.S. Provisional Patent Application 62/055,962 filed September 26, 2014 entitled "Low Latency Optically Distributed Dynamic Optical Interconnection Networks", and U.S. Provisional Patent Application 62/047,689 filed September 9, 2014 entitled "Methods and Systems for Distributed Dynamic Optical Interconnection Networks with Optical Low Latency Signaling", the entire contents of both being included by reference.

FIELD OF THE INVENTION

[001] This invention relates to optical interconnection networks and more particularly to distributed optical switch networks with cyclic wavelength dependent routing elements and an optically dedicated distributed low latency signaling.

BACKGROUND OF THE INVENTION

[002] Data centers are facilities that store and distribute the data on the Internet. With an estimated 14 trillion web pages on over 750 million websites, data centers contain a lot of data. Further, with almost three billion Internet users accessing these websites, including a growing amount of high bandwidth video, there is a massive amount of data being uploaded and downloaded every second on the Internet. At present the compound annual growth rate (CAGR) for global IP traffic between users is between 40% based upon Cisco's analysis (see http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/ white_paper_c l l-481360_ns827_Networking_Solutions_White_Paper.html) and 50% based upon the University of Minnesota's Minnesota Internet Traffic Studies (MINTS) analysis. By 2016 this user traffic is expected to exceed 100 exabytes per month, or over 42,000 gigabytes per second. However, peak demand will be considerably higher with projections of over 600 million users streaming Internet high-definition video simultaneously at these times. All of this data flowing into and out of these data centers will generally be the result of data transfers between data centers and within data centers so that these overall IP traffic flows must, in reality, be multiplied many times to establish the total IP traffic flows.

[003] A data center is filled with tall racks of electronics surrounded by cable racks where data is typically stored on big, fast hard drives. Servers are computers that take requests and move the data using fast switches to access the right hard drives and either write or read the data to the hard drives. In mid-2013 Microsoft stated it had itself over 1 million servers. Connected to these servers are routers that connect the servers to the Internet and therein the user and / or other data centers.

[004] At the same time as requiring an effective yet scalable way of interconnecting data centers and warehouse scale computers (WSCs), both internally and to each other, operators must provide a significant portion of data center and WSC applications free of charge to users and consumers, e.g. Internet browsing, searching, etc. Accordingly, data center operators must meet exponentially increasing demands for bandwidth without dramatically increasing the cost and power of the infrastructure. At the same time consumers' expectations of download / upload speeds and latency in accessing content provide additional pressure.

[005] According to Facebook™, see for example Farrington et al in "Facebook's Data Center Network Architecture" (IEEE Optical Interconnects Conference, 2013 available at http://nathanfarringtonxom/presentations/facebook-optics-oidal3-slides.pptx), there can be as high as a 1000: 1 ratio between intra-data center traffic to external traffic over the Internet based on a single simple request. Within data center's 90% of the traffic inside data centers is intra-cluster. Further, Farrington notes that whilst a Folded Clos topology provides the best economics at the largest scales the cabling complexity becomes a daunting problem as it is quadratic function of the number of nodes. Farrington notes that the issue of reducing the cabling complexity of Folded Clos topologies is an industry-wide problem worth solving.

[006] Accordingly, it would be beneficial for new fiber optic interconnection architectures to address the traditional hierarchal time-division multiplexed (TDM) routing and interconnection and provide reduced latency, increased flexibility, lower cost, lower power consumption, and provide interconnections exploiting N x M x D Gb/s photonic interconnects wherein N channels are provided each carrying M wavelength division signals at D Gb/s. [007] Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying figures.

SUMMARY OF THE INVENTION

[008] It is an object of the present invention to address limitations within the prior art relating to optical interconnection networks and more particularly to distributed optical switch networks with cyclic wavelength dependent routing elements and an optically dedicated distributed low latency signaling.

[009] In accordance with an embodiment of the invention there is provided an optical network comprising:

a plurality of planes of nodes;

a wavelength dependent data distribution layer; and

a passive broadcast control distribution layer, wherein

the nodes within a plane and the equivalent nodes within each plane are both connected by a switch exploiting the wavelength dependent data distribution layer and the passive broadcast control distribution layer.

[0010] In accordance with an embodiment of the invention there is provided a switch exploiting a wavelength dependent data distribution layer and a passive broadcast control distribution layer.

[0011] In accordance with an embodiment of the invention there is provided a switch exploiting a wavelength dependent data distribution layer and a passive broadcast control distribution layer wherein the transmitters connected to the wavelength dependent data distribution layer are high speed wavelength tunable sources providing 2R or 3R functionality.

[0012] In accordance with an embodiment of the invention there is provided a network employing a fast tunable optical laser source in combination with a passive wavelength dependent distributed optical switch with discrete optical hyperedge signaling.

[0013] In accordance with an embodiment of the invention there is provided a reconfigurable 2R or 3R optically tunable laser source connected via strictly non-blocking wavelength router connected so as to provide one of a distributed VLB switch, a complete graph switch, and a perfect difference graph switch. [0014] Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments of the present invention will now be described, by way of example only, with reference to the attached Figures, wherein:

[0016] Figure 1A depicts data center network connections according to the prior art using two-tier leaf-spine architectures;

[0017] Figure IB depicts a chordal interconnection pattern for a ring network according to the prior art for use within the interconnection of servers and data centers;;

[0018] Figure 2 depicts schematically the optical elements of a node within a 12 fiber chordal ring architecture according to the prior art of US 2012/0,321,309 entitled "Optical Architecture and Channel Plan Employing Multi-Fiber Configurations for Data Center Network Switching";

[0019] Figure 3 depicts schematically a system overview of an Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention;

[0020] Figures 4 and 5 A depict schematically the operating basis of a wavelength dependent interconnection networking element (AWGR) forming part of a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention;

[0021] Figure 5B depicts schematically the operating basis of a wavelength independent interconnection networking element forming part of a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention;

[0022] Figure 6 depicts schematically the result of combining AWGR based Fast Tunable Laser Source Switch (FTLSS) and AWGR based Dynamic Reconfigurable Graph Data center Interconnection Network (DRGDIN) to provide Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) based networks with a discrete Optical Hyperedge Signaling Panel (OPHYS1P) for connecting data centers in a three-dimensional architecture;

[0023] Figures 7A and 7B depict 2R and 3R regenerators exploiting semiconductor based picosecond tunable wavelength converters; [0024] Figure 8 depicts an exemplary schematic of opto-electronic reducer as a receiver structure which offer contention free for a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention; and

[0025] Figure 9 depicts an exemplary schematic of single receiver structure with a contention control push back for a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention.

[0026] Figure 10 depicts an exemplary schematic of multiple but not fully populated optoelectronic reducers as receivers with a contention push back for a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention;

[0027] Figures 1 1 A to 1 1C depicts schematically alternate data center interconnections in a three-dimensional architecture according to an embodiment of the invention by combining AWGR based Fast Tunable Laser Source Switch (FTLSS) and AWGR based Dynamic Reconfigurable Graph Data center Interconnection Network (DRGD1N) to provide Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) based networks with a discrete Optical Hyperedge Signaling Panel (OPHYSIP) and in-plane torus networks.

DETAILED DESCRIPTION

[0028] The present invention is directed to optical interconnection networks and more particularly to distributed optical switch networks with cyclic wavelength dependent routing elements and an optically dedicated distributed low latency signaling.

[0029] The ensuing description provides exemplary embodiment(s) only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiment(s) will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It being understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0030] The majority of large data centers, including cloud scale data centers, exploit networks today that are designed around a two-tier leaf/spine Ethernet aggregation topology leveraging very high-density switches. Servers first connect to leaf switches and then leaf switches connect to spine switches. Each leaf switch must connect to every spine switch in order to ensure that the network is never oversubscribed at any location beyond the chosen oversubscription threshold. By using such a network topology, and leveraging an equal cost multi-path protocol (ECMP), it is then possible to have an equal amount of bandwidth across the aggregated path between the upstream and downstream thereby providing a non-blocking network architecture via multiple aggregated link. It would be evident that the number of uplinks on the leaf switches limits the number of spine switches to which they can connect whilst the number of downlinks on the spine switches then limits the number of leaf switches that can be part of the overall network.

[0031] Consequently, the number of computer servers that can be added to two-tier leaf/spine data center network architecture is a direct function of the number of uplinks on the leaf switches. If a fully non-blocking topology is provided then the leaf switches are required to have as many uplinks as downlink interfaces to computer servers. Nowadays, 10 Gbps is the default speed of network interfaces of data center servers and hence, with the number of servers required to support the growth of Hybrid/Multi-Cloud services etc. requiring much larger and more centralized data centers, it has become challenging to design non-blocking and cost-effective data center networking fabrics. A combination of a Public and a Private cloud forms a Hybrid Cloud. The combination of multiple Public Cloud services forms a Multi-Cloud. The combination of a Hybrid Cloud and a Multi-Cloud forms a Hybrid/Multi- Cloud.

[0032] Within a two-tier leaf/spine topology, an oversubscription ratio is defined as the ratio of downlink ports to uplink ports when all ports are of equal speed. In the case of servers with 10 Gbps interfaces, and considering these as part of a 3: 1 oversubscribed architecture, then 40 Gbps of uplink bandwidth to the spine switches is necessary for every 12 servers. Today, cloud scale data center operators are accepting the constraints of a 3: 1 oversubscribed two- tier leaf/spine topology (see for example http://www.ieee802.org/3/400GSG/public/13_07/issenhuth _400_01_0713.pdf) due to the much higher costs of implementing non-blocking fabrics.

[0033] The 3: 1 threshold is generally seen as a maximum allowable level of oversubscription and is carefully understood and managed by the data center operators. Accordingly, referring to Figure 1A there is depicted a 3: 1 oversubscribed leaf/spine/core architecture supporting communications within and between a pair of data centers, Data center A 1 10 and Data center B 120. The computer infrastructure generally consists of servers 130 interconnected at 10 Gbps to Top of Rack (ToR) Ethernet switches that act as first level aggregation, the leaf switches 140. These ToR leaf switches 140 then uplink at 40 Gbps into end of row (EoR) Ethernet switches, which act as the spine switches 150 of the leaf/spine topology. As an example, with a 48-port ToR switch of 10 Gbps per port, ensuring a maximum 3 : 1 oversubscription ratio requires that the ToR switches have 16 uplink ports at 10 Gbps or alternatively, 4 ports at 40 Gbps. Then in order to enable connectivity across data centers, the spine switches then connect at 100 Gbps to core routers 160, which then in turn interconnect to optical core infrastructure made up metro/long-haul DWDM/ROADMs transport platforms.

[0034] Whilst, this leaf/spine/core architecture is the most pervasive manner of providing any-to-any connectivity with a maximum amount of bisection bandwidth within and across data centers it is not without its limitations. One such limitation is latency due to the requirement to route by at least one leaf switch 140 or more typically via two leaf switches and two or more spline switches 150 and / or core routers 160 according to the dimensions of the data center, the uplink capacity, downlink capacity, location(s) of the servers being accessed, etc. Accordingly, within the prior art alternative architectures have been proposed such as chordal networks and spline ring networks. Considering the former then a 32 node chordal ring network is depicted in Figure lb wherein each spline switch is addressed from another spline switch by the selection of the wavelength upon which the data is transmitted. Accordingly, there the number of spline switches / core switches traversed may be reduced through Dense Wavelength Division Multiplexing (DWDM) based chordal ring architectures as depicted in Figure I B as rather than routing data through multiple spline and / or core switches the data routed from a node based upon wavelength wherein the N'^h wavelength denotes the N'^h node around the ring.

[0035] A node of a spline ring network according to the prior art is depicted in Figure 2 after Barry et al in US 2012/0,321 ,309 entitled "Optical Architecture and Channel Plan Employing Multi-Fiber Configurations for Data Center Network Switching." Within this architecture Plexxi Inc. implement a 12 fibre ring, 6 East and 6 West, each with a Coarse WDM channel plan. At each node one fiber in each direction is dedicated to adjacent node communications with 4 wavelengths. At each node a second fiber terminates in each direction with 8 wavelengths wherein these have been added in pairs at nodes N - 2; N - 3; N - 4; N - 5 where N is the current node. Each node can therefore add either one or both wavelengths on the appropriate fiber to allow to be sent directly to nodes N + 2; N + 3; N + 4; N + 5 . [0036] Within other prior art developments to address the drawbacks within two-tier leaf- spine networks have included the addition of direct connectivity between spine switches rather than requiring routing via a core router and the provisioning of increased connectivity between leaf switches and spine switches such that each leaf switch is connected to multiple spine switches. However, within data center inter-connection networking scenarios these approaches maintain centralized switching functionality requiring extra network links be traversed, commonly referred to as increasing the number of hops, which in turn increase latency, increase cost and increase power consumption. Three key factors that cloud data storage providers and data center operators are seeking to lower. Accordingly, it would be evident that removal of one layer of switching would yield substantial benefits overall for the data center network (DCN). Further, it would be beneficial to reverse the current situation wherein design methodologies aimed at Internet switching scenarios consider centralized switching the natural deployment scenario.

[0037] Referring to Figure 3 there is depicted schematically a system overview of a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention. Accordingly there are depicted first to third servers 31 OA to 3 I OC, representing a plurality of servers, which are each connected to a Data Layer fully interconnected network 340 and a Control Plane fully interconnected network 330. Accordingly, each of the first to third servers 31 OA to 310C is bi-directionally coupled to the Data Layer fully interconnected network 340 and bi-directionally coupled to the Control Plane fully interconnected network 330. As depicted and described below in respect of Figures 4 and 5A the Data Layer fully interconnected network 340 exploits a wavelength dependent routing core and as depicted and described below in respect of Figure 5B the Control Plane fully interconnected network 330. Accordingly, as the Data Layer fully interconnected network 340 connects each input to every output each server of the first to third servers 51 OA to 5 IOC respectively is coupled to every other server of the first to third servers 51 OA to 5 IOC respectively. By implementing the Data Layer fully interconnected network 340 as a non-centralized switched architecture, i.e. a physically distributed switch, then latency is reduced, cost is reduced, and power consumption reduced. In some embodiments of the invention where the Data Layer fully interconnected network 340 is a passive optical component the power consumption is zero. [0038] Similarly as the Control Plane fully interconnected network 330 connects each input to every output each server of the first to third servers 31 OA to 3 IOC respectively is coupled to every other server of the first to third servers 31 OA to 3 IOC respectively. Again, by implementing the Control Plane fully interconnected network 330 s a non-switched architecture, latency is reduced, cost is reduced, and power consumption reduced. In some embodiments of the invention where the Control Plane fully interconnected network 330 is a passive optical component the power consumption is zero.

[0039] Referring to Figure 4 there is depicted a schematic of a Data Layer fully interconnected network 340 employing a cyclic N x N array waveguide grating router (AWGR) and its subsequent exploitation in Figure 5A within a Physically Distributed Optical Switch (PDOS) allowing tunable switching and dynamic reconfiguration. Within Figure 4 N = 4 such that the AWGR 400 has 4 input ports 41 OA to 410D and four output ports 420 A to 420D respectively. Considering first input port 41 OA then this receives one or more optical signals with wavelengths

such that these are routed to the first to fourth output ports 420A to 420D respectively, i.e.

_(420D) .

However, due to the operation of the AWGR 400 employing a waveguide phased array in conjunction with launch and collection free propagating zones if the same 4 optical wavelengths are now coupled to the second input port 410B, and we denoted these as λ ; then due to the adjusted launch condition into the waveguide phased array from the second input waveguide relative to the first input waveguide then whilst we get output signals on all 4 output ports 420A to 420D their wavelengths as launched from the second input port 420B are now

_(4205); Λ;; _(420C); , J420D) . As such a signal X launched into the second input port 410B is coupled to the first output port 420A just as a signal Λ, launched into the first input port 41 OA is coupled to the first output port 420A.

[0040] In a similar manner the third and fourth input ports are mapped as

_(420£>) and λ _{42§Α); λ J420S); /l2 _(420C); /l3 _(420 ) respectively. Accordingly, as depicted in Figure 5A by tuning the wavelength of a signal coupled to an input port of the first to fourth input ports 41 OA to 410D, e.g. one of first to fourth transmitters 51 OA to 510D, to one of the four wavelengths X ; X X ; X (where x represents the input port) supported by the AWGR 400 then the AWGR 400 will route the optical signal to the predetermined output port of the first to fourth output ports 420A to 420D and therein to the appropriate receiver of first to fourth receivers 520A to 520D respectively. It would be evident therefore that the AWGR 400 supports via optically tunable transmitters the equivalent of a larger switching based non- blocking interconnection between the transmitters and receivers. The AWGR 400 removes through its inherent wavelength routing characteristics between different input ports and the common output port array substantial optical interconnection complexity, e.g. input and output arrays of 1 :N and N: l optical switches and a perfect shuffle interconnection of complexity N² .

[0041] Within other embodiments of the invention an input / output port may be configured with N Tx/Rx pairs whilst another sub-grouping may be configured with 2N Tx/Rx and yet another sub-grouping N I K , i.e. K - 2 , Tx/Rx per port. Within the configurations employing 2N Tx/Rx per port the HYSPDOS thereby leverages its functionality as a Virtual Load Bearing (VLB) switch and the HYSPDOS offers data centers a technology route to reduced hop count, reduced latency switching, and higher bandwidth than deployed commercial 2-tier spine-leaf centralized switching.

[0042] Accordingly, with a wavelength agile transmitter data can be routed to the appropriate receiver by simply changing the wavelength of the transmitter. If the wavelength agile transmitter supports fast sub-nanosecond tuning (switching) then the resulting optical network supports dynamic reconfiguration at the packet level. If the wavelength agile transmitter support multiple wavelengths simultaneously then the optical network supports unicast (one-to-one) routing as well as broadcast (one-to-many) distribution. One such embodiment is a WDM array of laser based transmitters such that the same data signal can be modulated onto multiple WDM wavelengths and hence routed to the appropriate receivers.

[0043] The Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention depicted in Figure 3 in addition to the Data Layer fully interconnected network 340 employs a Control Plane fully interconnected network 330. As the Data Layer fully interconnected network 340 supports potentially rapid dynamic reconfiguration and both unicast / broadcast routing the Control Plane fully interconnected network 330 within the embodiments of the invention depicted here are based upon broadcast signaling methodologies. However, it would be evident that in other embodiments of the invention the signaling information may be transported on a Control Plane fully interconnected network 330 that is essentially a replica of the Data Layer fully interconnected network 340 or with other network topologies supporting unicast and / or broadcast signaling communications. Referring to Figure 5B there are depicted first and second embodiments 500 and 550 respectively for a wavelength independent interconnection networking element. First embodiment 500 is a passive distribution network formed from 3dB couplers 510 in ranks 520 which are interconnected through perfect shuffle networks 530. Second embodiment 550 is a multimode interference interferometer (MMI) star coupler providing the same 8x8 broadcast capabilities.

[0044] Within the prior art whilst optical switching has been explored by industrial and academic researchers relatively few examples of deployment exist apart from protection switching and slow speed reconfigurable add-drop multiplexers (ROADMs). A large number of active switching technologies from micro-electro-mechanical system based mirrors (MEMS mirrors) to semiconductor waveguide switching have been considered. However, passive fabric based optical switching, such as HYSPDOS has been less researched although the potential of N x N coupler based and AWGR based switches have been identified, see for example McGill University in "TED PAPER" and Viscore Technologies Inc. in U.S. Provisional Patent Application 61/729,872 entitled "Methods and Devices for Passive Optical Switching."

[0045] However, switch designers and researchers have focused to the exploitation of the AWGR switch within centralized switch designs which connect the AWGR router with line cards of a large electronic switch or router, see for example I. Keslassy et al. in "Scaling Internet Routers Using Optics" (Proc. SIGCOMM'03, pp.189-200). This is partly because the distributed switch without an effective distributed-control is not practical. A factor within the logic leading to such designs has been the unavailability of commercially manufactured fast wavelength tunable components such that in many instances wavelength reconfiguration is abolished together with the requirement for tunable components and the designs have accordingly tended to rely upon architectures such as complete graph load balanced switches exploiting an AWGR central switch design. One such example being R. Zhang-Shen and Ν. McKeown in "Designing a Fault- Tolerant Network Using Valiant Load-Balancing" (Proc. 27^th IEEE Conf. Computer Communications, INFOCOMM 2008).

[0046] The inventors have established an optical networking architecture, the Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS), which exploits AWGR elements with Fast Tunable Laser Source Switch (FASTLASS) to provide AWGR Dynamic Reconfigurable Graph Data center interconnection networks (ADRG-DINs) with discrete optical hyperedge signaling panels (DOEH-SPs). Accordingly, a plurality of data centers which are denoted as Data center A 640 and Data center B 660, are depicted in schematic 600 as arranged upon a plurality of tiers, Tier 1 to Tier 9, and having associated with each Storage 650, representing the internal memory storage within a data center. The data centers within a Tier are interconnected via first POXN (Hyperedge/AWGR) A 610 and second POXN B 620 whilst data centers across the Tiers are interconnected via third POXN C 630. Each of the POXN is a HYSPDOS providing Optical Hyperedge Signaling, via the Control Layer data interconnection network 330 within the HYSPDOS, for control together with AWGR based Data Layer data interconnection network 340 within the HYSPDOS.

[0047] The inventors have considered as the basis of optical transmitters within the wavelength agile routing networks enabled by the AWGR devices within the Data Layer data interconnection network 340 of HYSPDOS that exploit semiconductor based picosecond tunable wavelength converters (TWCs) such as depicted in Figures 7A and 7B. Referring initially to Figure 7A there is depicted a so-called 2R regenerators exploiting a semiconductor based picosecond tunable wavelength converter, 2R regenerators in optical domain being considered to wavelength convert and optically amplify. As depicted the optical wavelength converter converts an input signal of wavelength λΐ into an output signal of wavelength λ2. As depicted the optical wavelength converter includes a saturable absorber switch (SATABS) 720 which is coupled to the optical input via circulator 710. Also coupled to the SATABS 720 is an optical emitter 730 operating at λ2 providing a signal at wavelength X2 to the SATABS 720 together with the input signal at wavelength λΐ . Optical emitter 730 may be a tunable laser operable over the absorption region of STABS 720 or an array of lasers each operable at a different wavelength within the operating window of the SATABS 720. In other embodiments SATABS 720 may be coupled to a broadband light source with tunable filter, e.g. EDFA, supercontinuum light source, LED, etc. Within different embodiments the optical signals from optical emitter 730 and input signal are comparable powers whilst in other embodiments of the invention the optical emitter 730 is a high power signal or the input optical signal is high power or made high power via an optical amplifier, not shown for clarity. The SATABS 720 generates an output signal at λ2 which is coupled via the optical circulator 710 to the output via filter 740. Optionally, filter 740 is a band filter to limit noise within the optical network or it may be a tunable optical filter to increase isolation of the input wavelength λΐ at the output.

[0048] As SATABS 720 comprises a non-linear absorbing medium then under predetermined conditions, e.g. relatively low intensity light is incident upon the non-linear absorbing medium, then it is highly absorbing. However, upon illumination by a high intensity beam, non-linear absorbing medium saturates, becoming less absorbing. An incident optical beam having an associated wavelength within the absorption region of non-linear absorbing medium can saturate it (making it less absorbing) over its entire absorption range. Thus, it is possible for a high intensity optical beam of wavelength λΐ to switch another optical beam having a wavelength λ2 given that both wavelengths fall within the absorption band of nonlinear absorbing medium. Alternatively, a low intensity optical beam of wavelength λΐ may switch another optical beam having a wavelength λ2 Wavelength λΐ can be either greater or smaller than wavelength λ2 as what is important is the optical power level. In this manner the 2R wavelength converter can achieve both "up" conversion and "down" conversion functions, where "up" conversion refers to a conversion from a low energy photon (i.e., long wavelength photon) to a high energy photon (i.e. short wavelength photon) and "down" conversion refers to the opposite.

[0049] Referring to Figure 7B there is depicted a so-called 3R regenerator wherein in addition to wavelength conversion and optical amplification (i.e., higher output power at converted (output) wavelength than input power at input wavelength) the 3R regenerator retimes (and / or reshapes) the optical signal. Accordingly, as depicted the 3R regenerator similarly comprises input port, output port, circulator 710, SATABS 720, and filter 740. However, now the optical source at λ2 is now modulated with a clock signal such that the optical signal coupled to the SATABS 720 at X2 is digital rather than CW such that now the SATABS 720 will only be transparent when both optical signals meet the appropriate condition. Accordingly, the emitted signal at λ2 is now retimed and reshaped when compared to the input signal at λΐ . According to embodiments of the invention the SATABS 720 may be a semiconductor optical amplifier (SOA).

[0050] It would be evident to one skilled in the art that the Data Layer data interconnection network 340 as described and discussed employing an AWGR router is strictly non-blocking in that there always exists a connection between a given input port and a given output port irrespective of the other connections mapped for the other input / output ports. However, it is possible that absent appropriate control for two transmitters to be accessing the same output port simultaneously. Within the electrical switching domain this is referred to as contention control. Accordingly, referring to Figure 8 there is depicted an exemplary structure of the receiver side of a HYSPDOS according to an embodiment of the invention. As depicted each optical receiver 51 OX coupled to the Data Layer fully interconnected network 340 is now replaced by a Contention Reducer 800X which is then coupled to electrical Receiver 860X.

[0051] Each Contention Reducer 800X as depicted by fourth Contention Reducer 800D comprises an optical demultiplexer (DMUX) 820 that separates the optical signal(s) from fourth port 420D into the discrete N wavelengths. Each optical wavelength is then coupled to an avalanched photodiode (APD) / photodetector (PD) with transimpedance amplifier (TIA) combination, depicted as an N element opto-electronic converter array 830 which provides each serial electrical signal to series to parallel (S2P) converters / buffers 840. The parallel data from the S2P converter / buffer 840 is then coupled to shuffle logic 845. Shuffle logic 845 allows any suitable combination of the electrical data signals output from the N S2P converter / buffer 840 to be provided to the receiver 860D via electrical multiplexer (MUX) 850. Optionally, shuffle logic 845 may be omitted from a Contention Reducer such as Contention Reducer 800D as may optionally the buffer functionality within S2P converter / buffer 840.

[0052] The shuffle logic 845 and buffer functionality may be implemented according to an embodiment of the invention as a silicon field programmable gate array (FPGA) or application specific integrated circuit (ASIC) which is connected to the N channel array of APD(PD)/TIA. This silicon FPGA / ASIC chip has limited size of buffer for each APD(PD)/TIA. When multiple transmitters are sending packets of data to the same receiver the DMUX 820 will separate the multiple wavelengths from the output port of the AWGR router and couple them to different APD(PD)/TIA elements allowing the Contention Reducer to determine automatically which packets are being transmitted and from which transmitters. Based upon this information, the Contention Reducer will send a signal back over the Control Plane fully interconnected network 330, depicted as star coupler 810, to all of the transmitters involved in transmission. This reverse signalling via the Control Plane fully interconnected network suppresses the transmitters that are contending with the selected channel on the receiver, in this instance fourth receiver 860D. The selection of the transmitter by a receiver may be established upon a range of conditions including, but not limited to, maintaining an already transmitting transmitter, randomly picking an active transmitter, and cycling active transmitters. The selected transmitter is given continuous sending privilege for a predetermined period of time, e.g. 3/tf or 10/zs .

[0053] Now referring to Figure 9 there is depicted an exemplary schematic of receiver structure without contention reducer for the Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention. In this embodiment of the invention the transmitters, first to fourth transmitters 51 OA to 510D, are again connected via Data Layer fully interconnected network 340 to first to fourth receivers 520A to 520D respectively. However, in this instance there is no Contention Reducer and the optical signal at the receiver is tapped such that a portion of the signal is coupled back via Control Plane fully interconnected network 330, depicted as star coupler 810, to all of the transmitters wherein the transmitter(s) detects the contention signal and stops transmitting and they restart transmitting at a predetermined offset of a plurality of predetermined offsets or according to a time stamp added / present within the signal routed back through the Control Plane fully interconnected network 330. The time stamp may be added by a modulator within the tapped feedback path prior to the Control Plane fully interconnected network 330. Whilst contention performance and throughput of the design depicted in Figure 9 may not be as high as that depicted in Figure 8 it is a more cost effective approach.

[0054] Optionally, a wider bandwidth DMUX at the receiver side may be employed and each transmitter operates upon a subset number of the available transmitter ports. By combining outputs to a single receiver non-blocking operation may still be implemented whilst reducing the contention frequency. Such a scenario is depicted in Figure 10 wherein multiple but not fully populated opto-electronic reducers operate as discussed in respect of Figure 8 with a contention push back for a Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) according to an embodiment of the invention. Accordingly, as depicted each of the first to fourth Contention Reducers 1000A to 1000D have S2P converters / buffers 840 but their number, M , is now less than the number of channels, N , supported by the Data Layer fully interconnected network 340.

[0055] Some algorithms may be introduced within the HYSPDOS depicted in Figure 10 on the transmitting side, first to fourth transmitters 51 OA to 510D respectively, in order to reduce the contention possibility to even lower level. Since Data Layer fully connected layer 340 allows multiple transmitters to connect to one receiver side PD/TIA and associated silicon logic, then multiple transmitters from a group, e.g. all transmitters or a subset of the transmitters, have until they receive a contention signal back from a receiver via the Control Layer fully connected network 330 have no means of determining that their transmission will yield contention for the PD/TIA/Logic of a receiving side receiver, e.g. first to fourth receivers 520A to 520D. To reduce the possibility of contention, the transmitters may be grouped and transmitters within the same group may employee different prioritization for sending data between groups and hence using the same wavelength to transmitters within another group. For instance, a Txl in Group A set Group A as one priority whereas the Tx2 in Group A set the priority for Group B at a different level. Hence, when the transmitters Txl in Group A have data for sending upon the same wavelength they choose this first Group differently, hence reducing the possibility of contention to transmitters within the same group.

[0056] Another method of avoiding contention in the same group is to employee a SOA- TWC such as depicted in Figures 7A and 7B respectively. Accordingly, in a first group, Group A, if Tx 1 is transmitting, the SOA-TWC associated with Tx2 will convert the Tx2 transmitted wavelength to the first wavelength of a second group, Group B, the SOA-TWC associated with Tx3 will convert the Tx3 transmitter wavelength to the first wavelength of a third group, Group C. This method also can significantly reduce the contention ratio and with appropriate rules the contention may theoretically be reduced to zero, i.e. the HYSPDOS operates contention free.

[0057] Now referring to Figures 1 1A to 1 1C there are depicted firs to third schematics 1 100A to 1 100C respectively in respect of alternate data center interconnections in a three- dimensional architecture according to an embodiment of the invention established by combining AWGR based Fast Tunable Laser Source Switch (FTLSS) and AWGR based Dynamic Reconfigurable Graph Data center Interconnection Network (DRGDIN) to provide Hyperedge Signaled Physically Distributed Optical Switch (HYSPDOS) based networks with a discrete Optical Hyperedge Signaling Panel (OPHYSIP) and in-plane torus networks.

[0058] Referring to first schematic 1 100A a plurality of data centers which are denoted as Data center A 1140 and Data center B 1160 are configured into a plurality of tiers, depicted in first schematic 1100A as Tier 1 to Tier 8, and having associated with each Storage 1150, representing the internal memory storage within a data center. The data centers may be visualized as a rectangular array even where their physical locations are not. The data centers at two mutually perpendicular edges of the Tier are interconnected via first Data POXN (Hyperedge/AWGR 1) 1110, second Data POXN (Hyperedge/AWGR 2) 1130, and first Control POXN (Hyperedge/POXN) 1 120 whilst data centers within a Tier are interconnected by first and second ring networks, Torus A 1170 and Torus B 1180 respectively, which provide virtual horizontal and vertical networks in a mesh network interconnecting the data centers upon a Tier. Each of the first and second Data POXN 1110 and 1130 together with the Control POXN 1 120 within a Tier are coupled through equivalent networks that connect all tiers along two mutually perpendicular faces of the rectangular prism formed by each tier of N x M data centers and the R tiers.

[0059] Figure 1 IB and second schematic 1 100B depict a single Tier whilst in Figure 1 1C with third schematic a routing architecture is depicted according to an embodiment of the invention. Accordingly, a first Data Center A (DCA 1) 1 145 A rather than routing to a first Data Center B (DCB 1) 1 155A via either second Data Center A (DCA 2) 1 145B and third Data Centers A (DCA 3) 1 145C or fourth Data Center A (DCA 4) 1 145D and second Data Center B (DCB 2) 1 155B routes in a manner to cross as many Data Centers as possible. In Figure 1 1C in third schematic 1 lOOC this routing is a spiral passing all data centers. It would be evident that instead of data centers these elements in first to third schematics 1100A to 1100C represent servers and / or server racks within a single data center. Optionally, routing may be to pass as many Data Centers as possible without looping back or it may be set to pass all other nodes. Other routing rules may be established including, but not limited to, shortest path.

[0060] It would be further evident that in addition to the connections between tiers along two edges that the tiers may be interconnected on additional edges or that additional tier-tier interconnections may be provided throughout the "body" of the rectangular prism virtual construction. Optionally, in addition to the linear bus networks shown connecting the edges of the tiers and the ring networks within a tier these networks as well as the additional "through" networks may be ring networks, torus networks, mesh networks, and what the inventors refer to as "cube" networks. "Cube" networks are those connecting elements on multiple tiers which within a representation such as that depicted in Figure 6 or Figures 11A to 1 1C would be represented in three dimensions (3D) rather than one dimension (ID) or two dimensions (2D) within a tier or between tiers. Such "cube" networks may include, for example, a network diagonally through the tiers from one corner to another or from one edge across multiple tiers to another edge along multiple tiers or single tier.

[0061] According to embodiments of the invention an AWGR distributed optical switching methodology is presented wherein routing and channel selection is distributed to the edge of the switching fabric leaving a purely passive interconnect in the core. Further, contention control and signaling via a second optical control plane exploiting a similarly purely passive core are presented and described. Also, this disclosure provides a method to expand the total size of networks constructed from the two distributed switch design.

[0062] According to embodiments of the invention where the transmitters connected to ports of the data layer fully interconnected network employ novel sub-nanosecond fast tunable light sources, e.g. SOA based tunable wavelength converters (TWCs), of 2R or 3R functionality to transmit the data packets. Optionally, the TWCs may be replaced fast tunable lasers, broadband source with fast tunable laser or a fast receiver such as a fast tunable coherent receiver.

[0063] The disclosure allows designs for optical switching within data center interconnections that are physically distributed in a distance range of particular interest given today's large data centers, namely interconnections at the 1km and above range. In contrast to prior art short range optical interconnections and line card connections the current designs according to embodiments of the invention are possible as a result of the interconnection between the optical data layer and the sub-nanosecond distributed signaling panel, i.e. the Control Layer fully interconnected network, which is aligned with the distributed switch.. Further for any physically distributed switch, the contention control of mutual exclusivity is a significant challenge as control signals must traverse the distributed switch and may have, potentially, significant propagation delay, as travel from end to end has considerable propagation delay which is about 5 ? / &/?2 in optical fibre. Hence, where targeting sub- nanosecond line rates, the signaling panel is too slow. Accordingly, the inventors in Figures 8 and 9 present two methods to address this. One exploits content free operation of the AWGR router such that appropriate edge connection settings are always routed and the second exploits the zero skew performance of passive optical cross-connection networks.

[0064] Within the prior art Internet or telecom networking economy in order reduce the costs implementations with lower degree (TxRx ports per node) of the network have been typically preferred. However, for the rich-connection data center, the port count the nodes have is important as the total bandwidth is important as a node often needs to send high traffic demands to a large number of destinations. In addition, since the optical line rate per wavelength is limited by the optical physical links then current high bit rate ports already exploit multiple transmitters and multiple receivers, e.g. 100G and 160G networks exploit 10 and 16 channels of lOGb/s Tx/Rx ports already. Accordingly, to leverage the natural high degree (TxRx ports per node) count of optical data center interconnections the inventors exploit high speed reconfigurable AWGR based networking. Considering, a group of 2N configurations the performance of the interconnection network according to embodiments of the invention is better than an equivalent physical core switch. In the group of (N - 1) configuration, the complete graph performance is close to a central core switch; with less than half of the switch cost. Switches according to embodiments of the invention provide for a 2N configuration plus a central switch.

[0065] Generally within embodiments of the invention the total wavelength count per fiber is the ports limit of the passive optical cross connection and it also limits the total port number of AWGR based data layer switches. Hence, some methodology is needed to expand the total networking size of the disclosed distributed AWGR switch topologies. Accordingly, the inventors exploit what they refer to as the "Cartesian product" to expand the scale of the networks. The Cartesian product of switch connected nodes is topologically homomorphic with the Microsoft Butterfly-Cube (MSFT-BC). However, the commodity small electronic switch in the MSFT-BC is replaced by a passive optical AWGR based physically distributed (hundreds to thousands meters) switch. Hence, the disclosed disclosure design has lower diameter count (1 vs. 2 per dimension), zero power consumption, and better cost economy than MSFT-BC design.

[0066] The foregoing disclosure of the exemplary embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many variations and modifications of the embodiments described herein will be apparent to one of ordinary skill in the art in light of the above disclosure. The scope of the invention is to be defined only by the claims appended hereto, and by their equivalents.

[0067] Further, in describing representative embodiments of the present invention, the specification may have presented the method and/or process of the present invention as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. As one of ordinary skill in the art would appreciate, other sequences of steps may be possible. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, the claims directed to the method and/or process of the present invention should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the present invention.

Claims

CLAIMS What is claimed is:

1. An optical network comprising:

a plurality of planes of nodes;

a wavelength dependent data distribution layer; and

a passive broadcast control distribution layer, wherein

2. The optical network according to claim 1 , wherein

nodes within a plane of nodes are configured as a N x M array with N columns and M rows;

nodes within at least one of a row and a column within each of the plurality of planes of nodes are connected by a first wavelength dependent data distribution fabric of a plurality of first wavelength dependent data distribution fabrics forming the wavelength dependent data distribution layer; and

nodes within the other of the row and the column within each of the plurality of planes of nodes are connected by a second wavelength dependent data distribution fabric of a plurality of second wavelength dependent data distribution fabrics forming the wavelength dependent data distribution layer.

3. The optical network according to claim 1 , wherein

nodes within at least one of a row and a column within each of the plurality of planes of nodes are connected by a first passive broadcast control distribution fabric of a plurality of first passive broadcast control distribution fabrics forming the passive broadcast control distribution layer; and

nodes within the other of the row and the column within each of the plurality of planes of nodes are connected by a second passive broadcast control distribution fabric of a plurality of second passive broadcast control distribution fabrics forming the passive broadcast control distribution layer.

4. The optical network according to claim 1 , wherein

nodes within at least one of a row and a column within a plane of nodes of the plurality of planes of nodes are connected by a first optical fabric exploiting a torus configuration; and

nodes within the other of the row and the column within a plane of nodes of the plurality of planes of nodes are connected by a second optical fabric exploiting a torus configuration.

5. The optical network according to claim 1 , wherein

nodes within a plane of nodes are configured as a N x array with N columns and M rows; and

routing from within a node of a first row of nodes within a plane of nodes of the plurality of planes of nodes to another node within another a second row of nodes within a plane of nodes of the plurality of planes of nodes is undertaken on the basis that the routing within the N x M array is mapped as a spiral following nodes within one of a column and a row and then the other of another row and another column in an alternating sequence dependent upon the originating node.

6. A switch exploiting a wavelength dependent data distribution layer and a passive broadcast control distribution layer.

7. A switch exploiting a wavelength dependent data distribution layer and a passive broadcast control distribution layer wherein the transmitters connected to the wavelength dependent data distribution layer are high speed wavelength tunable sources providing 2R or 3R functionality.

8. A network employing a fast tunable optical laser source in combination with a passive wavelength dependent distributed optical switch with discrete optical hyperedge signaling.

9. A reconfigurable 2R or 3R optically tunable laser source connected via strictly non- blocking wavelength router connected so as to provide one of a distributed VLB switch, a complete graph switch, and a perfect difference graph switch.