CA2093355A1 - Parallel computer system - Google Patents

Abstract

A digital computer includes a plurality of processing elements, a command processor, a diagnostic processor and a communications network. The processing elements each performs data processing and data communications operations in connection with commands. The processing elements also performing diagnostic operations in response to diagnostic operation requests and providing diagnostic results in response thereto. The command processor generates commands for the processing elements, and also performs diagnostic operations in response to diagnostic operation requests and providing diagnostic results in response thereto. The diagnostic processor generates diagnostic requests. The communication network includes three elements, including a data router, a control network and a diagnostic network. The data router is connected to the processing elements for facilitating the transfer of data among them during a data communications operation. The control network is connected to the processing elements and the command processor for transferring commands from the command processor to the processing elements. The diagnostic network connected to the processing elements, the command processor and the diagnostic processor for transferring diagnostic requests from the diagnostic processor to the processing elements and the command processor and for transferring diagnostic results from the processing elements and the command processor to the diagnostic processor.

Description

wr) 92/06436 PCr/US91/07383

2~33~

PARALLEL COMPUTER SYSTEM
INCORPOR~TION BY REFERENCE
Guy E. Blelloch, Scan Primitives and_Parallel Vector Models, (Ph.D. Dissertation, Massachusetts Institute of Technology: 1988), incorporated herein by reference.
U.S. Patent Appn. Ser. No. 07/489,079, filed March 5, 1990, in the name of W. Daniel Hillis, et al., entitled Digital Clock Buffer Circuit Providing Controllable Delay, and assigned to the assigDee of tbe present application, incorporated herein by reference.
FIELD OF THE INVENTION
The invenfion relates generally to the field of digital computer systems, and more particularly to massively parallel computing systems. The invention particularly provides arrangements for controlling processors in a computing system having a large number of processors, for facilitating transfer of data among tbe processors and for facilitating diagnosis of faulty components in the computing system.
BACKGROUND OF THE INVENTION
A digital computer system generally comprises three basic elements, namely, a memory element, an input/output element and a processor element. The memory element stores information in addressable storage locations. This information includes data and instructions for processing the data. The processor element fetches inforrnation from the memory elemeDt, interprets the information as either an instruction or data, processes the data in accordance with the instructions, and returns the processed data to the memory element. The input/output element, under control of the processor element, also communicates with the memory element to transfer informatio4 including instructions and the data to be processed, to the memory, and to obtain processed data from the ~nemory.
Most modern computing systems are considered ~von Neumann~ machines, since they are generally constmcted according to a paradig,n attributed to lohn von Neumann. Von Neuinann machines are characterized by having a processing element, a global memory which stores all in~ormation in the system, and a program counter that identifies the location in the glabal memosy of the instruction being executed. The processing elernent executes one instruction at a time, tha~ is, the instruction identilled by the program counter. When the instruction is executed, the program counter is advanced to identify the location of the ne7a instruction to be processed. (In many modern systems, the program counter is actually advanced before the processor has fin~shed processing the current instruction.) Von Neumann systems are conceptually uncomplicated to design and program, since they do only one operation at a time. A number of advancemeDts have been made to the original von Neu~nann paradigm to permit the various parts of the system, most notably the various components of thc proccssor, to opcrate relatively independenlly and achieve a significant incsease in processing speed. One such advancement is pipelining of the various steps in executing an instructioo, including iustructior~ fetch, operation code decode (a typical instruction includes an operation code which identifies the operation to be performed, aDd in most cases one or more operar d speciifiers, which 6 ~ ~

identify the location in memory of the operands, or data, to be used in executing the instruction), operand fetcb, execution tthat is, performing the operation set forth in the operation code on tbe fetched operands), and storing of processed data, which steps are performed relatively independently by separate hardware in the processor. ID a pipelined processor, the processor's instruction fetch hardware may be fetching one instruction while other hardware is decodirlg the operation code of another instruction, fetching the operands of still another instruction, executing yet another instruction, and storing the processed data of a fifth instruction. Sirlce the five steps are performed sequentially, pipelining does not speed up processing of an individual instruction. However, since the processor begins processing of additional instructiorls before it has finished processing a current instruction, it can speed up processing of a series of instructions.
A pipelined processor is obviously much more complicated than a simple processor in a von Neumann system, as it requires not only the various circuits to perform each of the operations (in a simple von Neumann processor, many circuits could be used to perform several operations), but also control circuits to coordinate the activities of the various operational circuits. However, the speed-up of the system can be dramatic.
More recently, some processors have been provided vith execution hardware which includes multiple fuDctional units each being optimized to perfonn a certain type of mathematical operation.
For example, some processors have separate functional units for performing integer arithmetic and floadng point arithmetic, siDce they are processed very differently. Some processors have separate hardware functional units each of which performs one or only several types of mathematical operations, including addition, multiplication, and division operations, and other operations such as branch control and logical operations, all of which can be operating concurrently. This can be helpful in speeding up certain computations, most particularly those in which several fun~ional ur~its n~ay be used concurrently for performing parts of a single computation.
In a von Neumann processor, including those which incorporate pipelining or multiple fuDctioDal units ~or both, since both may be incorporated into a single processor), a single instruction stream operates on a single data stream. That is, each instruction operates on data to enable one calculation at a time. Such processors have been termed ~SISD,~ for single~ stNction/single-data.~ If a prograr~ requires a seglnent of a program to be used to operate on a number of diverse elements of data to produce a number of calculations, the program causes the processor to loop through that segment for each calculation. In some cases, in which the program segmeut is short or there are only a few data elements, the time required to perform such a calculation may not be unduly long.
However, for many types of such programs, SISD processors would requi re a very long time to ~`~
perform all of the calculations required. Accordingly, processors have been developed which incorporate a large number of processiDg elements all of which may operate concurrently on the same instruction stream, but with each processing element processing a separate data stream. These processors have been termed ~SIMD~ processors, for ~ingle-~nstruction/!nultiDle-data.~
SIMD processors are useful in a number of applications, such as image processing, signal processing~ artificial intelligence, database operations, and computer simulation of a nurnber of things, : .

WO 92/06436 PCr/US91/07383 20~3~5

-3-such as electronic circuits and fluid dyDamics. In irnage processi~g, each prOCeSSiDg element may 'oe used to perform processiDg on a pixel (-picture elementn) of the image to eDhance the overall irnage.
Irl signal processillg, the processors coDcurrerltly perform a number of the calculations required to perform such computations gs the "Fast Fourier trgnsform' of the data defming the signal. In artificial intelligence, the processors perform searches on extensive sule bases representing the stored knowledge of the particular application. Similarly, in database operations, the processors perfor~
searches on the data irl the database, and mgy also perfor~n sorti~g and other operations. In computer simulation of, for example, electronic circuits, each processor may represent one part of the circuit, and the processor's iterative computations indicgte the response of the pgrt to signals from other parts of the circoit. Similarly, in simulating fluid dynamics, wnich can be useful in a Dumber of applicatio~s such as weather predication and airplane design, each processor is associated with one poirJt in space, and the calculations provide information about various factors such as nuid flow, temperature, pressure and so fortn.
-- Typical SIMD systems include a SIMD array, which includes the array of processing elements ~
and a router network, a control processor and an input/output component. The iDput/output componeDt, under coDtrol of the con~rol processor, enables data to be transferred into ~he array for processing and receives processed data from the array for storage, display, and so forth. The control processor also controls the SIMD array, iteratively broadcasting instructions to the processing elements for execution in parallel. The router net vork enables the processing elements to communicate the results of a calculatioD to other processing elements for use in future ca~culalions.
Several routing networks have been used in SIMD arrays and others have been proposed. In one routing network, the processing elemeDts are interconnected in a matri~, or mesh, arrangement.
ID such an arrangement, each processing element is coYmected to, and communicates with, four "nearest neighbors" to form rows and columns defining the mesh. This arrangement can be somewhat slow if processing elements need to communicate among themselves at random. However, the arrangement is inexpensive and conceptually simple, and may suftlce for some types of processing, most notably image processing. The ~Massively Parallel Processorn manufactured by (;oodyear Aerospace Corporation is an example of a SIMD array having such a routing network.
ID another routing network, processing elements are interconnected in a cube or hypercube arrangement, having a selected number of dimensions, for transferriDg data, in the form of messages, among the processing elements. The arrangement is a "cube~ if it only bas three dimensions, and a "hypercube" if it has more than three dimensions. U. S. Patent No. 4,598,400, entitled Method and Apparatus For Routing Message Packets, issued July 1,1986 to W. Daniel Hillis, and assigned to the assignee of the present applicatioD, describes a system having a hypercube routing network. ID the system described in the '400 patelll, multiple processiDg elements are connected to a single routing node, and the routing nodes are interconnected in the hypercube.
Another routing arraDBement which has been proposed is a crossbar switch, through which each processing element can s:oMmunicate directly with any of the other processing elements. The .. . "..., .. , ,., , . . ;. ~ .
: : . . :, . . : , ,; : , ,, :, ., ~, :
. - . : :: . ;: .............. , . : , .. .. . .
:: . -. - : . :; . . -. . : ::-:; . : .

WO 92/06436 PCr/US91/07383 ~ 0 9 3 3 ~ ~ 4 crossbar switch provides the most efficient communica~ions of any of the routing networks proposed.
However, a crossbar switch also has the most connections and switcbung elements, and thus is the most expensive and also the most susceptible to failure due to broken cormections aDd faulty switching elements. Thus, crossbar switch arrangements are rarely used, except when the number of processiDg elements is fairly small, since the complexity of a crossbar switch increases with the square of the number of processing elements.
Yet another routing arrangement is an omega network, in which switching is performed through a number of serially-connected stages. Each stage has two inputs, each connected to the outputs of a prior stage or processing elements, has two outputs which may be connected to the inputs of a subsequent stage or processing elements. The ~Butterfly~ computer system manufactured by Bolt Beranek & Newman uses such a network.
SUMMAR~ OF THE INVENTION
The ir vention provides a new and improved parallel computer system.
In brief summary, the-new computer includes a plurality of processing elements, a command processor, a diagnostic processor and a communications network. The processing elements each pcrforms data processing and data communicatioDs operations in connection vith cornmands. The processing elements also performing diagnostic operations in response to diagnostic operation requests and providing diagnostic results in response thereto. The command processor generates colDmaDds for the processiDg elements, and also performs diagnostic operations iD response to diag~ostic operation requests and providing diagnostici results in response thereto. The diagllostic processor generates diagnostic requests. The communiution network includes three elements, including a data router9 a control network and a diagnostic network. The data router is connected to the processing elements for facilitating the transfer of data among them during a data communications operation. The control network is connected to the processirg elemeDts and the command processor for transferring commands from the commaDd processor to the processing elements. The diagnostic network connected to the processiDg elements, the command processor and the diagnostic processor for transferring diagnostic requests from the diagnostic processor to the processiDg el ments al~d the command processor and for transferring diagnostic results from the processing elements and the command processor to the diagnostic processor.
BRIEF DESCRIPI IOI`I OF THE DR~WING5 This invention is pointed out with particularity in the appended claims. The above and further advantages of this invention may be better understood by referring to the foUowing desiption taken in conjunction with the accompanying drawings, in which:
Fig. 1 is a genesal block diagram of a massively parallel computer system constructed in accordance with the~i invention;
Figs. 2A and 2B are block diagrams useful in understanding the structure and operation of the data router of the conlputer system of Fig. 1;
Fig. 3 is a diagram depicting the structure of message packets transferred over the data router;

- . . - , , . ,, ; . .

- : ,. . : ", -, .....
. . .. . . . , . ,, - . , . . :, .

WO 92/06436 PCr/US91/07383 2~933S~
Figs. 4A and 4B are block diagrams useful in understanding the structure and operation of the control sletwork of the computer system of Fig. 1;
Fig. 5 is a diagram depicting the structure of message packets transferred over the control network;
Figs. 6A through 6C are block diagrams useful in understanding the structure and operation of the diagnostic network of the computer system of Fig. 1;
Fig. 7 is a diagram depicting the structure of message packets transferred over the diagnostic net~vork;
Fig. 8 is a general blo k diagram of a processing element in the computer system depicted in Fig. 1;
Fig. 9A-1 comprises a general block diagram of a data router interface circuit useful in interfacing the processing element depicted in Fig. 8 to the data router of the computer system depicted in Fig. 1, Figs. 9A-2A and 9A-2B contain def~itior~s of registers in the data router interface and Figs. 9B-1 through 9D-7 comprise-logic diagrams of she data router iDterface;
Fig. 10A comprises a general block diagram of a control network interface circlut useful ;D
interfacing the processing element depicted in Fig. 8 to the control network of the computer system depicted in Fig. 1, Figs. 10A-1 contains a definitions of a register in the control network interface and Figs. 10B through 10G comprise logic diagrams of the control network interface;
Fig. 1~A is a geDeral block diagram of a data router node used in the data router described in connection with Figs. 2A and 2B, and Figs. 11B-1 through 11D comprise detailed block and logic diagrarns of the data router node;
Fig. 12A is a general block diagram of a control network node used in the control network described in connection with Figs. 4A and 4B, and Figs. 12B-1 through 12D-1 comprise detailed bloclc and logic diagra~ns of the control router node; and Fig. 13A is a general block diagram of a diagnostic network node used in the diagnostic nenvork described in comlecdorl with Fig. 6, and Flgs. ~7B-1 through 13C comprise detailed block and logic diagrams of the diagDosîic network node.
DETAILED DESCRIPIION OF AN lLLUSTRA~VE EMBODIMENT
1. General Description A. General Description Of Computer System Flg. 1 is a general block diagram of a massively parallel computer system 10 constructed in accordance with the invention. With reference to Fig. 1, system 10 includes a plurality of processing elements 11(0) through 11(N) (generally identified by reference numeral 11), scalar processors 12~0) through 12(M) (generally identified by reference numeral 12) and input/output processors 13(0) through 13(~;) (generally identified by reference numeral 13). Input/output units (not shown), such as, for exa~nple, disk and tape storage units, video display devices, printers and so forth may bc comlected to the input/output processors to supply information, including data and program commaDds, for processing by the processing elements 11 and scalar processors 12 in the system, and may also receive .. . : . : ., . :: : : ; , . , :: . ::: . . . : , i ~ .. ~

WO 92/06436 PCr/US91/07383 2 ~ 6-processed data for storage, display and printing. The scalar processors 12 may also be cormected to input/output units including, for exarnple, video display terminals which permit one or more operators to generally control system 10.
The system 10 further includes a control network 14, a data router 15 and a diagnostic network 16. The control network 14 permits one or more scalar processors 12 to broadcast program commands to the processing elements 11. The processing elements 11 execute the commands generally concurrently. The control network 14 also permit the processing elements 11 to transfer status information to the scalar processors 12. The control net vork 14 is also used by the processing elements 11 to perform selected types of arithmetic operations, termed nscan" and nreduce" operatiorls, as described below. The control network 14 may also be used to provide synchror~zation amoDg the processing elements 11.
The data router 15 transfers data among the processing elements 11, scalar processors 12 and input/output processors 13. In particular, under control of the scalar processors 12, the inpul/output processors 13 retrieve data to be processed from the input/output units and distributes it to the respective scalar processors 12 and processing elements 11. During processing7 the scalar processors 12 and processing eleme~ts 11 can transfer data among themselves over the data router 15. In addition, the processing elements 11 and scalar processors 12 can transfer processed data to the input/output processors 13. Under control of the scalar processors 12, the inputtoutput processors 13 can direct the processed data that they receive from the data router 15 to particular ones of the input/output units for storage, display, printiug, or the like.
The diagnostic network 16, under control of a diagnostic processor (not shown), facilitates testing of other portions of the system 10 to iden~ify, locate and diagnose defects. The diagnostic .
processor may comprise one or more of ,the scalar processo}s 12. In addition, the diagnostie network 16 may be used to establish selected operating coDditions in the other portions of the system 10 as describcd below.
The system 10 is synchronous, that is, all of its elements operate in accordance with a global SYS CLK system clock signal pro~lided by a clock circuit 17.
One particular embodiment of system 10 may include hundreds nr many thousands ofprocessing elements 11 operating oll a single problem in parallel under control of commands broadcast to them by the scalar processors 12. In that embodiment, the processing elements 11 operate ~n parallel orl the same command on their individual sets of data, thcreby forming a parallel computer system. In addition, the system 10 may be dynamically logically partitioDed, as described below, into multiple subsystems whicb may concurrently operate on separate problems or separate parts of a single problem. In that case, each partition includes at least one scalar processor 12 and a ph~rality of processing elements 11.
B. General Description Of Communications Networks 1. Data Router 15 Before proceeding to a detailed description of the system 10 and its various components, it would be helpful to generally describe the structures of the control network 14 and data router 15. The ... . . ... .. . . .

.: . . i . . . .. ..

WO 92/06436 PCr/US91/07383 7 2~333~3 data router 15 and control net vork 14 both transfer information in the form of message packets, whicb will be described in detail below in connection with Figs. 3 and 5, respectively. Figs. 2A and 2B depict the general structure of the data router 15 and Figs. 4A and 4B depict ,~he general structure of the control net~vork 14.
With reference to Fig. 2A, the data router 15 is generaDy tree-structured, having a phlrality of data router node groups 20(i,j) (~in and nj~ are integers) organized in a plurality of levels each identified by the index ";" in reference numeral 20(i,j). A data router node group 20(i,j) al each level "i" is connected to a selected number of data router node groups 20(i-1J) in the next lower level "i-l" to form a tree~ As will be described in detail below, the data router node groups 20(i,j) perform message s vitching operations to transfer data, in the form of data router message packets, among the processing elements 11, scalar processors 12 and input/output processors 13, which are coDectively identified as leaves 21(0) through 21(N) (generally identified by reference numeral 21). Each data router node group 20(1,j) in the lowest level is connected to one or more leaves 21. In the reference numeral 20(i,j), the index (j) uniquely identi,5es each of the data router node groups 20(i,j) at each level l7i.n In the data router 15 represented in Fig. 2A, the data router node group 20(M,0) at the highest level "M~ is terrned the ~physical rootn of the tree. At eacn level nin, each data router node group 20(i,j) is termed the "parent~ of data router node groups 20(i-1,j) connected thereto, and each data router node group 20(i-1,j) is termed a Rchild" of tne data router node group 20(i,j) to which it is cormected. It will be appreciated that the data router node group 20(~) ~bill also be a child of tne data router node group 20(i+ 1,j) connected t'nereto. In one pardcular em'oodiment, each data router Dode group 20(i,j) in a pardcular level "i" is corlDected to four child data router node groups 20(i-1,j); in that embodiment, the "fan-out~ of the tree, that is, the n~nber of children comlected to each parent, is four. It will 're appreciated from the foDowing that the fan-out need not be constant, but may vary from level to level and aiso a,nong data router r ode groups ZO(i,i) witbin the same level.
The structure of the data router 15 is further termed a "fat-treen, and will be particularly described in connecdon with Fsg. 2B. With reference to F~g. 2B, at least some of the data router node groups 20(i,j) includes at least one, and typically two or more data router nodes 22(iJ,k), wherein "k~ is sn inleger that uniquely idendfies each data router node within a data router node group 20(i,j). Each data router node æ(iJ,k) in a data router node group 20(i,j) is connected to a plurality of data router nodes æ(i+l,j,k) in level "i+l,~ ~vith the connections being established so that the data router nodes 22(i,j,k) in each data router node group 20(iJ) are connected to different ones of the data router nodes 22(i+ 1,j,k) in the data router node group 20(ij) in level ~i+l.r For example, in data router Dode group 20(1,0), data router node æ(l,0,0) is connected to data router nodes Z(2,0,0) and 22(2,0,1) of data router node group 20(2,0), and data router node 22(1,0,1) is comlected to data router nodes 22(2,0,2) aDd æ(2,0,3) of data rou~er node group 20(2,0).
In addition, each data router node æ(i,j,k) in a parent data router node group 20(i,j) is conDected to one data router node æ(i-l,j,k) in that parent's child data router node groups 20(i-1,j).
Accordingly, as shown in F;g. 2B, data router node (2,0,0) in data router node group 20(2,1) is . . , - :;: : ., :::. ;: ;, , : , ~
. . , . , , , , ~: , , ,~, ,. :
.. ... . .. . . ... , :.

WO 92/06~36 PCI /USgl/07383 2()9~/~CiS -8-connccted to one data router node 22(1,j,0), where njn equals 0,1, 2 aud 3, in each of the data router node groups 20(1,0) through 21(1,3).
It will be appreciated that the collection of data router nodes Z(i,j,k) from each leaf 21 to and including the data router nodes æ(m,O,k) in the root data router node group 20(M,0) essendally forms an inverted tree. Each leaf 21 effectively comprises the root of one inverted tree and the data router nodes 22(M,0,k) of the root data router node group 20(M,0) form all of the leaves of all of the inverted trees defined by ~he collection of leaves 21. The number of data router nodes 22(i,j,k) in each data router node group 20(i,j) at a particular level n;n in tbe tree defining data router 15 will be determined by the fan-out at each level from level "1" to level nj~ in the inverted tree. The fan-out at a particular level ";" is the number of data router nodes æ(i+ 1,j,k) at level nj+ ln to which each data router node 22(i,j,k) at level ";" is connected. Thus, for example, since data router node æ(1,0,0) of data router node group 20(1,0) in level n1n is counected to two data router nodes 22(2,0,0) and æ(2,0,1) of data router node groups 20(2,0) in level n2,n the fan-out from data router node 22(1,0,0) is two. In one particular embodiment, the fan-out from data router nodes 22(i,j,1c) at a particular level ";" is the same for the entire level, but it may differ from level to level as described below.
As noted above, the data router 15 transfers message packets among the processiDg elements 11, scalar processors 12 and input/output processors 13, all of which are represented by leaves 21.
Each connection shown in Fig. 2B between a leaf 21 and a data router node 2~(1,j,k) of level 1, which is represeDted by a line therebetween, actually represents two unidirectional data paths, one for transferring a message packet in each direction. Thus, for example, the connection between leaf 21(0) and data router node 22(1,0,0) of data router node group 20tl,0) represents two data paths. One data path is used by the leaf 21(0) to transrllit a message packet to the data router node æ(1,0,0) for deUvery to another leaf 21(x). The other data path is used by the data router node æ(1,0,0) to deliver message packets originating at other leaves 21 destined for the leaf 21(0).
Similarly, each connection between a data router node æ(i,j,k) of a level Itj~ and a data router node 22(i+1,j,k) of a level "ii1,n which is also represented in Fig. 2B by a line, represents two unidirectional data paths, one for traDsferrirlg a message packet in each direction. Thus, for example, the connection between data router node 22(1,0,0) of data router node group 20(1,0~ and data router node 22(2,0,0) represents two data paths, one used to transfer message packets from data router node 22(1,0,0) to data router node æ(2,0,0) and the other to transfer message packets in the opposite direction, that is, from data router node æ(2,0,0) to data router node 22(1,0,0).
Transfer of a message packet from one leaf 21(x) to another leaf 21(y) through the data router 15 message transfer proceeds in two general operations. ~;irst, the data router nodes 22(i,i,k) trar sfer the message packet firsl "up ~he tree," that is, to data router nodes ~n successively higher leveLs, until it reaches a selected maximum level determined in part by the separation between the source and destination leaves. After a message packet has reached the selected maximum level, the transfer continues "down the treen, during which the data router nodes 22(i,j,k) transfer the mes. age packet to data router nodes at successively lower levels until it is delivered to the destination leaf 21(y). As will ..,, , ., , . . - . .. " . :,;. .~ . :
. . , ; ,, ~ :
.:: , . . . . ..

WO 92/06436 PCI'/I)S~1107383 2a~33~3 .9 be clear fro~ the detailed description of the structure and operation of a data router node æ(i,i,k) in Figs. 11A through 11D below, the data router 15 can transfer a plurality of messages concurrently, any of the data router nodes æ(iJ,k) can direct messages up the tree and other messages down the tree at the same time.
Before proceeding further, it may be helpful to describe the structure of a message packel transferred over the data router lS. With reference to Fig. 3, a data router message packet 30 includes three general portions, including a message address portion 31, a message data portion 32, and a checksum portion 33, each comprising one or more nflits.~ In one embodiment, each flit comprises four bits, which are transferred in parallel over a data router coDnection, that is, between a leaf 21 and a data router node 22(iJ,k) or between two data router nodes æ(i,i,k).
The~message data portion 32 includes several elements, includiDg a length flit 34, a tag flit 35 and one or more data nits 36(0) through 36(N) (geDerally identified by reference numeral 36). The data flits 36 generally contain the actual message data being transferret over the data router 15, which may vary from packet to packet. The tag fit 35 contains control information which may be used by the destination leaf, identified herein by reference numeral 22(y), in processing the data. The contents o~
the length flit 34 are identify the number of flits in the message data portion 32, and may vary depending on the amount of data being transferred in a particular packet. In one particular embodiment, the contents of length flit 34 identify the mL~nber of thirty-two bit words in the data flits 36 of the message packet. In that embodiment, the number of data flits 36 in the inessage packet is eight tirnes the ~alue in the length flit 34.
The checksum portion 33 contains a value which is used in detecting errors in packet trarlsmission over the data router 15.
The data router 15 uses the contents of the message address portion 31 to determine the path to be traversed by the message packet 30 from the source leaf to the destination leaf. The message address pordon 31 includes a header 40, wbich identif,es the selected ma~imum level to which the message packet is to be transferred when going up the tree, and a down path identification portion 41 which identifies the path down the tree to the destination leaf 21(y) when going down the tree. When directing a message packet up the tree, a data router node 22(iJ,k) at level "i,~ randornly selects orle of the data router nodes 22(i+ 1,j,k) connected thereto in level ~i+ l" ~ data router node group 20(i~ lJ) to receive the message packet. Other than specifying the selected maxirnum height for the message packet, the packet does not other vise specify the particular path it is to take up the tree.
The down path identification portion 41 of message packet 30 defines the path the packet is to take down the tree &om the data router node group 20(i,j) at the selected maximuln level ~o the destination leaf 21(y). The down path identification portion includes one or more down patb ideDtit`ler fields 42(1) through 42(M) (generally itentified by reference nurlleral 42). The successive do vn path identifJer fîelds 42, beginning with field 42(M), are used by the data router nodes 22(i,j,k) at successively lower levels as they direct the packet downwardly in the tree.
The down path identifier field 42(i) for level ~;" identifies the child data router node group 20(i-1J) to which the parent data router node group 20(iJ) that receives the packet at le ~ is to .

- : . , . ". , . . ,: , ,, ~ , : :

WO 92/06436 P~r/US~l/073~3 2~9~5~ -lo-direct the message packet 30. It will be appreciated that the do vn path identifer fields 42 need not specifically identify one of the data router nodes æ(i-lJ,k) in the data router node group 20(i,~) at each level to which the message packet is to be directed, since the path down the tree is effectively a traversal of the inverted tree of which the destination leaf 21(y) is the root.
In one embodirnent, in which each parent data router node group 20(iJ) is connected to four child data router node groups 20(i-lJ) or four ieaves 21, each down path identifier field 42 comprises two bits that are binary encoded to identify one of the four children to which the message is to be directed. As indicated by Fig. 3, two fields 42 are packed into a single four-bit flit in the message packet 30. Since one down path identifier field 42 is used to at each level (i) in the downward traversaL
the number of down path identifier fields 42 required to define the do vnward path corresponds to the selected ma~num level in the path up the tree, which, in turn, corresponds to the contents of header ;~
40. During the downward traversai mode, the data router nodes æ(i,j,k) through which a message packet 30 passes decrement the contents of the header 40 and, aPter both down path identifier fields 42 contained in a flit have been used, discard the flit.- Thus, the leDgth and content of a message packet 30 may change as it is being passed down the tree.
It will be apprec ated that the addressing arrangement provided by the header 40 and down path identification portion 41 can be viewed as follows. The selected ma~mum height ~n header 40 effecdvely identifies the data router node group 20(iJ) which is the root of a sub-tree, preferably the smallest sub-tree, of the data router 15 that contains both the source leaf 21~x) and the desdnation leaf 21(y). On the other hand, the down path identifiution por~ion 41 details the e~act path from that root to the destinadon leaf 21(y).
The provision of ineasing numbers of data router nodes 22(iJ,k) in data router node groups 20(iJ) at higher levels in the data router ~ ereby resulting in a ~fat-tree" design, provides several advantages. In a massively parallel computer SIMD system, processing elements 11 ~ypically transfer messages during a message trarlsfer operation, initiated by commands from the scalar processors 12.
During a message transfer operation, a large number of processing elements 11 may transfer messages concurrently. If the data router 15 did not have increasing nwnbers of data router nodes 22(iJ,k) at higher levels to which the message packets 30 can be directed when going up the tree, the bandwidth of the data router 15, that is, the rate at which i~ can transfer message packets 30, would decrease at higher levels.
Since increasing numbers of data router nodes æ(ilj,k) are provided at higher levels in ~e ~fat-tree~ design, the reduction in band vidth at higher levels can be mir~imized or coDtroDed. As noted above, the fan-out of data router node groups 20(i,j), that is, the number of data router nodes æ(i+ 1,j,k) at level ~i+ 1" connected to each data router node 22(ij,k) at level "i" CaD vary from level to level, and can be selecletl lo maintain a desired minirnum bandwidth between the respective levels ~i"
and ";+ 1.~ Alternatively, the fan-outs from each level to the next higher level can be selected so that the entire data router 15 has a selected minimum bandv~dth.
~ urther, as noted above, each data router node æ(iJ,k) randomly selects the data router node 22(i+1J,k) in the nex~ higher level to which it directs a message packet 30 in the path up the tree.

:- , : .:. . . . .

. .~ : . . :
:~ .. , ,~ :........ .. .

WO 92/06436 P(~/US91/07383 20933~3 Accordingly, the message packets are randomly distributed through the highes levels of the tree, which minimizes the likelihood of bottlenecks and ma~amizes the bandwidth in the higher levels.
As sho vn in Figs. 2A and 2B, each data router node group 20(iJ), and in particular each data router node Z(iJ,k), in the data router 15 receives an AFD(iJ) all-fall-do vn (iJ) signal. The AFD(iJ) all-fall-down (iJ) signal is provided by the control net vnrk 14, as will be described below in connection with Figs. 4A and 4B, under control of the scalar processors 12 to initiate a context switch operation.
The AFD(i,i) all-fall-down (iJ) sigl~al, when asserted, enables the data router 15 to enter an all-fall-down mode, in which it guickly empties itself of message packets. In response to the AFD(i,i) aIl-fall-down (i,i) signaL the data router 15 directs all message packets 30 directly down the tree to the leaves 21, where they are stored until the context in which the message packets were generated is restored. At that point, the leaves 21 which receive such messages can transmit them over the data router lS, which will deliver them to the intended destinations.
In contrast to normal operation described above, in which the contents of the header 40 are decremented and flits contaiDing down path identifier fields 42 discarded as the message packet 30 is directed down the tree, when the AFD(i,j) all-fall-down ti,j) signal is asserted the contents of the header 40 are not decremented and no changes are made to the ilits containing the down path identifier fields 42. When the context is restored and the leaves 21 return the message packels to the data router 15, they will be delivered to the proper destination leaves. This can be seen from the following explanation.
ln the followiDg explanation, reference numerals 21(x) a~ld 21(y) will refer to the origsnal source and destination leaves, respectively, for a message packet 30 and reference numeral 21(x') will refer to the intersnediate storage leaf which receives and stores the message packet 30 while the context in which the data router message packet 30 was generated is bcirlg switched out. Flrst, for those message packets that are being transferred np the tree or that have reached the selected maxisnu n height when the AFD(iJ) all-fall-down (iJ) signal is asserted, the contents of the header 40 and down path identificatiors portion 41 are the sarne as when they were originally transmitted by the source leaf 21(x). Since the intennediate storage leaf 21(x') receives the message packet 30 it must be part of a suo-tree of the data router 15 that includes both the source leaf 21(x~ and tlse destination leaf 21(y).
Further, the sub-tree has the same root data router node group 20(iJ) that the message packet 30 would have reached had the AFD(i,j) all-fall-down (i,j) signal not been asserted. Accordingly, when the intermediate storage leaf 21(x') transmits the message packet over the data router 15, the packet will go up the tree alsd reach the same data router node group 20(ij~ that it would have reached if the AFD(ij) all-fall-down (iJ) signal had not been asserted, and from there will ~ollow the sarne downward path, defined by the down path identification portion 41, that it would have taken.
On the other hand, if a message packe~ is being transferred down the tree when the AFD(ij) all-fall-down (i,j) signal is asserted, prior to the signal's assertion the contents of the header field 40 are decremented as the message packet is passed from level to levcl. Accordingly, it will be appreciated that, when the ~nessage packet 30 is transmitted by the intermediate storage leaf 21(x'), in its path up WO 92/06436 PCr/USsl/07383 2~335~ -12-the tree it will go only to a data router node group 20(iJ) at the level indicated in the header field 40, which, in turn, corresponds to the data rou~er node group 20(iJ) which controlled the direction of transfer of the message packet 30 wben the AFD(iJ) all-fall-down tiJ) signal signal was asserted. It will be appreciated that the data router node group 20(i,~) that the message packet 30 reaches raay not be the root of a sub-tree that includes the source leaf 21(x). However, it will be the root of a sub-tree that includes botb the intermediate storage leaf 21(x'), since the message packet 30 was transferred ~`
frorn that data router node group 20(~i) to the interrnediate storage leaf 21(x'), and the destination leaf 21(y), since the message packet 30 could have been transferred from that data router node group 20(iJ) to the destination leaf had the AFD all-fall-down (iJ) signal not been asserted.As will be described in further detail below, each leaf 21 maintains a message counter that it increments when it tranmsits a message packet over the data router 15, and that it decrements when it receives a message packet from the data router L~. As noted above, the control network 14 performs selected arithmetic operations, whose results can be provided to the processing elements 11 aad scalar processors 12. By enabling the control ne~work 14 to perforrn selected arithraetic operations using the values of the message counters, the results can identify when all of the message packets that were transmitted over the data router 15 have been received by the leaves 21, thereby indicating that the data router 15 is empty. This can be used to indicate that a message transfer operation has been completed, or that the router 15 is empty as a result of the assertion AFD(iJ) all-fall-down (iJ) sig~al so that a context switch can occur.
2. Control Network 14 As Doted above, the control network 14 transfers program commands from the scalar processors 12 to the processing elements 11 and returns status iaformation to the scalar processors 12, and in addition performs selected types of arithmetic operations. The control network 14 will be generally described in connection with block diagrams depicted in Figs. 4A and 4B, and with Fig. 5, which depicts the structure of a control net~vork message packet.
With refereDce first to Figs. 4A and 4B, the control network 14, like the data router 15, is generally tree-structured, having a plurali~y of control network node groups 50(i,j) (~i" and "j~ are integers) organized in a plurality of le~els each identified by the index ~i~ in reference numeral 50(i,j).
In the reference numeral 50(i,j), the index (j) distinguishes the diverse control network node group 50(iJ) at each level "L~ The tree structure of the control network 14 is generally similar to that of the data router 15. In particular, each control network node group 50(iJ) corresponds to a data router node group 20ti,j) having the same values for indices ~jn and ~jn~ and connections among control network node groups 50(iJ) follow the sarae pattern as connections a~nong data router node groups 20(i,j). Each control network node group 50(1,j) in the lowest level is connected to oDe or more leaves ~1, in the same pattern as the coDnections in the data router 15.
Similar terminology will be used in describing the control network 14 as was used in describiDg the data router 15 above. In particular, ;D the control net vork 15 represeDted in Fig. 2A, the control net vork node group 50(M,0~ at the highest level ~M" is termed the "physical root~ of the tree. At each ; . . . : : . . .
- : :
. . , ~ :

- : ~: . , : . .; ,. . .
.- .
., , , . . ~ , .

WO 92/06436 PCr/US91 /07383 2~933~a level "i", eacb control network node group 50(iJ) is termed the nparent~ of control network node group 50(i-1,J) connected thereto, and each control network node group 50(i-lJ) is tenned a nchildn of the control network node group 50(iJ) to which it is connected. The control network node group 50(iJ) wili also be a chiid of the control network node group 50(it1J) connected thereto. In one particular embodimeDt, each control network node group 50(iJ) in a particular level t;n iS conDected to four child control network Dode groups 50(i-1J), in which case the nfan-outn of the tree, that is, the number of children cormected to each pareDt, is four. As indicated above in connection with the data router 15, the fan-out need not be corLstaDt, but may vary from level to level and also among control network node groups 50(iJ) within the same level.
The structnre of a control network Dodc: group 50(i,l), which is shown on Fig. 4B, differs from the structure of a data router node group 20(iJ). With reference to Fig. 4B, a control network node group 50(iJ) includes three control network nodes 51(i,j,1), where ln can have the values ~p~n ~Cl~ or "C2.~ Within a control network node group SO(iJ), the control Detwork Dodes are connected so that control network node 51(i j,P) is parent of child control network Dodes 51(i,j,CI) and 51(i,j,C2). It wili be apprecia~ed that parent control net vork node Sl(iJ,P) of control network node group 50(ij) is itself a child of a control network node 51(i+1,j,CI) or control network node 51(i+1J,C2) of a control netwosk node group 50(i+ 1,j) of the next higher level ~i + Ln Similarly, each chiid control net~,vork node 51(i,j,C) is a parent of either a leaf 21 or a control nelwork node Sl(i-lJ,P) of the next lower level "i-l.~
It should be noted that, in Figs. 4A and 4B, the indices ~jn for control network nodes Sl(iJ,I) in each level increase from left to right. In the following, for each pareDt control network node 51(i+1,j,1), the chiid control network node 51(i,j,1) connected thereto with the lower index "j" will be termed the "left" chiid, and the control network node 51(i,j,1) with ths higher index "j" wiii be terrned the "right" child.
The control network node group 50(i,j) thus contains two sub-levels of control network nodes 51(i,j,1), one defined by parent control network node 51(i,j,P), and the other defined by child control network nodes Sl(i,j,Cl) and 51(i,j,C2). This enables the control network node groups 50(i,j) to have the same connection pattern within the control network 14 as the corresponding data router node groups 20(i,j) within the data router 15, while at the same time providing a t~,vo-chiid/one-parent connection for the control network nodes 51(i,j,1) which simplifies performance of the arithmetic operations as described below.
As in the data router 15, each connection between control network nodes 51(i,j,1) depicted in Figs. 4A and 4B represents two unidirectional data paths, which transfer message packets in opposi~e direct;oDs between the respective nodes.
As noted above, the scaiar processors 12 use the control network 14 to broadcast command~ to Ihe processing eiements 11. In tbis operation, a scaiar processor 12 transmits a message packet, which will be described below in detaii in connection with Fig. 5, to the control network node Sl(lJ,C) to which it is connected. The control network nodes transfer the message packet up the tree to the root, which then transmits the message packet down the tree to its chiidren. As each control network node .,. .. , . .:, ,: .:, - . .

WO 92/06436 PCr/USg1/07383 209~5~ -14-receives such a downwardly-going message packet, it transmits it to all of its children ~til the packet is deliverèd to the leaves 21. The control network 14 effectively broadcasts the message packet, and thus the command, to all of the processing elements 11. It will be appreciated that the message packet will r also be received at leaves 21 comprising scalar processors 12 and input/output processors 13" but these processors can be configured to ignore the packet.
As also noted above, the system 10 can be partitioned so as to effectively constitute multiple hdependently-operable systems, each including at least one scalar processor 12 and one or more processing elements 11. In partitior~ing the system 10, the scalar processor 12 establishes a logical root in a control net vork node 51(i,i,1) in the control network 14 which differs frorn the control ne~work node 51(M,O,P) which constitutes the physical root. The logical root effectively comprises the root of a sub-tree whose leaves include the scalar- processor 12 and one or more other leaves ~1. If a control network node Sl(iJ,I) becomes a logical root, while it is a logical root its parent node 51(i+ 1J,I) in the control network 14 does not not transmit downwardly-going message packets thereto.
Each control network node Sl(iJ,I) includes a root flag 1407, which is described in detail ;D
connection with Figs. 12A below. When the root flag 1407 is set, the control net vork node 51(iJ,l) is a root of the control network 15. If the control ne~work node 51(i,j,1) is to be a physical root, the root flag 1407 may be set by appropriate conditioning of an input signal that controls the control network Dode. To estabUsh a control net York node 51(i~i"l) as a logical root, the scalar prc,cessor 12 transmits a control network message packet therefor up the tree comprising colltrol network 1~. The message packet includes a height Yalue identifying the level and sub-level at which the logical root is to be established. Each control network node 51(i~;"1) which receives the message packet determines whether the height va,ue corresponds to its level and su'o-level, and if not passes the message packet to the next control network notd 51(i,j,1) up the tree. When a control network node 51(iJ,I) determines that the height value in the message packet corresponds to its level and sub-level, it sets its root flag 1407 and begins operating as a logical root as described above. ID comlection with that, tne control network node 51(i,j,1) notifies its parent control ne~work node 51(i,j,1) that it s a logical root.
It will be appreciated that a control network node 51(i,j,1) operating as a logical root of a partition may receive a message packet that indicates that a control network node 51(i+x,j,m) at a higher level or sub-level is to operate as a logical root. A scalar proc~ssor 11 may issue such a message to, for example, increase the number of processing elements 11 or scalar processors 12 in the partition.
In that event, the control net vork node 51(i,j,~) stops operating as a logical root.
To simplify the following description, the term ~root node,~ which may appear with or without the reference Dumeral Sl(iJ,I), will be used to collectively refer to the physical root control network node 51(M,O,P), in situations in which the control network 14 is not partitioned, and to a control network node 51(i,j,1) comprising a logical root in situations in which the control network 14 is pastitioned. If the controi network 14 is partitioned, the logical root node functions for the other control network nodes 51(i,j,1) in the partition substantially in the same manner as the physical control network ncde 51(M,O,P) functions for the control network nodes Sl(iJ,I) in an unparti~ioned control WO 92~06436 PCr/US9I/û7383 2~9335~
net vork 14. Otherwise stated, the physical root node can be considered as the logical root node of a partition comprising the entire system 10.
As Doted above, the control network 14 also performs several types of arithmetic operations in response to control network message packets therefor, including scan and reduce operations. Scan operations are generally described in Guy E. Blelloch, Scan Primitives and Parallel VeçtQr Models, (Ph.D. Dissertation, Massachusetts lnstitute of Technology. 1988). In a scan operation initiated by processing elements 11 that are logically arranged in a particular ordering, such as with increasing indices ~i~ in refereDce numeral 11(i) (with indices increasing, for example, from left to right, as shown in Fig. 4B), the scan operation for a particular arithmetic operator N~ on items of data ~D(i)N
mam~ained by the processing element 11(i) produces at each of the successive processing elements 11 in the ordering the result NR(i)~:
R(i) = D(0) ~ D(1) ~ D(2) ~ .... ~ D(i-1), vith R(0) = 0 [Eqn. 1]In the scan operation, the arithmetic operator may consdtute a number of types of operators, including, for example, signed or unsigned addition, OR, XOR (exclusive-OR) and MAX, the latter referencing determination of a ma~num of a set of values.
To accommodate scan operations, each control network node 51(iJ,I) includes an up data processor 1421, a do vn data processor 1652, and a scan buffer 1410, all of which will be described below in connecîion vith Figs. 12A through 12D-1. To initiate a scan operation, the processing elements 11 transfer control network message packets therefor over the control net vork 14. The control net vork message packet provided by each processing element 11(i) includes that processing element's data item D(i).
With reference to Fig. 48, each coDtrol network node Sl(lJ,CI) and 51(1,~,C~), on receiving a message packet from the processing elements connected thereto, loads the data from the left processing element, that is, the processing element 11(i) with the index ~i~ being zero or an even number, into its scan buffer 1410. In addition, the up data processor 1421 of each control network node Sl(lJ,C) performs the arithmetic operation on the data to generate a resulL that corresponds to the combination of the data received from the two processing elements 11 connected thereto, combined according to the arithmetic operator being used in the scan operation. The control net vork node 51(1J,C) uses the value generated by the up data processor 1421 as data in a message packet, which it transmits to its parent.
Each control network node Sl(iJ,I), except for the root node, on receiving message packets from both its left and right children, performs the same series of operations. In particular, each control network node 51(iJ,I) at each sub-level up ~o the root node:
(a) stores in its scan buffer 1410 the data in the cs)ntrol network message packet that it receives from its left c'nild controi net vork node 51(i-1,j,1); it vill be appreciated that this value corresponds to the combination of the data from the processing elements in the sub-tree of the control network 14 whose root is the left child control network node 51(i-1J,I), combined according to the arithmetic operator being used in the scan operation, and WO 92/06436 PCl'lUS91/0738:s h 1~ ~ 3 3 ~
(b) performs, using its up data processor 1421 the opcration, defined by the arithmetic operator being used in the scan operatio4 in connection with data from both of its children to generate a value which it transmits in a message to its parent. It will be appreciated that this value corresponds to the combination of the data from the processing elemeints in both sub-trees of the control network 14 whose roots are both child control network network nodes 51(i-1J,I) connected thereto.
Thus, at the point at which all control network message packets have propagated up the control network tree, the scan buffer 1~10 at each control network node 51(i,i,1), other than the root node, contains a value corresponding to the data provided by the processmg elements 11 in the su~tree whose root is the node's left child, processed according to the scan operaiton~s aritbmetic operator.
The root node also receives message packets from both of its children contair~ing intennediate results for a scan operation, it transmits message packets down the tree. The root node receives, from each child, a value corresponding to tbe data provided by the processing elements 11 in the sub-tree whose root is the respective cbild, processed according to the scan operation's arithrnetic operator. It will be appreciated that the value received from the left child control network node corresponds to the ~
combination of the data from the processing elements in the sub-tree of the control network 14 whose root is that left child control network node, and the value received from the right control network node corresponds to the combination of the data from the processing elements in the sub-tree whose root is the right control network node, in both cases the data being combined according to the scan operation's arithrnetic opeirator.
To its left child, the root node transmits a message packet whose data has the value zero. To its right child, the root node transmits a packet whose data has the value received from the left child.
As noted above, that value corresponds to the combination of the data from the processigl~ elements in the sub-tree of the control network 14 whose root is that left child control network node, combined according to the scan operation's ari~hmetic operator.
When each control network node 51(i,j,1) below the root node receives a coutrol network message packet from its parent, it (a) uses the down data processor 1652 to generate a value corresponding to the value of the data received from the parent combined with the intermediate result stored in the nodes' scan buffer 1410 according to the ari~metic operator used in the particular scan operation, which it transmits in a control network message packet to its right child; it will be appreciated that this ~alue corresponds to the combination of the data from the processing elements 11 in all sub-trees of the control network 14 up to the one whose root is the left child of the control network node, combined according to the arithmetic operator being used in the scan operation, and (b) transmiOE a control network message packet to its left child whose data has the same value as that received from the parent; it will be appreciated that this value corresponds to the combination of the data from the processing elements irl all sub-trees of the control network 14 up to the one whose root is the left child of the parent of the control network node, combined according to the arithmetic operator being used in the scan operation.

, . . .
.

.. .. .. . . . . . .. . .. . ..

WO ~2/06436 PCr/US9l/07383 2~3353 Thus, the control network message packets transmitted by the control network rlodes 51(iJ,l) down the tree will propagate the zero value down the left side to the leh-most processing elemenl 11(0). The next processing element 11(1) will receive the combination, as defined by the arithmetic operator, of the zero value propagated from the root node and the value stored in the scan buffer 1410 of the control network node 51(1,0,Cl), which corresponds to the value of the data transmitted by the processing element 11(0).
The next processing element 11(2) will receive, as the left child connected to the control network node 51(1,0,C2) the vah~e stored in the scan buffer 1410 of ~he control net vork node 51(1,0,P), which, as noted above, corresponds to the combination, as defined by the scan operatio~'s arithmetic operator, of the data from the processing elements 11(0) and 11(1). The processing elment 11(3) will receive, as the right child, the combination of that value and the value in the scan buffer 1410 of control ne~work node 51(1,0,C2), which, as noted above, corresponds to the data provited by the processing element 11(2). Accordingly, the processing element 11(3) will recehe the combination, as defmed by the scan operation's arithmetic operator, of the data from processing elements 11(0), 11(1~ and 11(2).
It will be appreciated that the control network nodes 51 will combine the data provided to the successive processing elements 11 in the sub-tree of the root node's left child sirnilarly. Accordingly, each processiDg element 11(i) in that sub-tree vill receives a value corresponding to data from processing ~lements 11(i-1) through 11(0) combined ascording to the arithraetic operator of the pardcular scan operation.
The control network nodes 51 in the sub-tree of the root node's right child also combine the data in the control network message packet provided by their respective parents with the data in their respective scan buffer 1410 in a similar maDner. As noted above, the root node tr~smits to its right child a control network message packet including a value corresponding to the combination of the data provided by the processing elements 11 in the sub-tree defined by the root Dode's left child, combined according to the scan operation's arithmetic operator. It will be appreciated that the control network message packets transmitted by he control network nodes 51(iJ,I) in that sub-tree will propagate that value down the left side of the sub-tree to the left-most prOCessiDg element 11(i), so that that processing element 11(i) also receives a value corresponding to data from processing elements 11(i-1) through 11(0) combined according to the arithmetic operator of the particular scan operation. Since the control nehvork nodes 51(i,j,1) in that sub-tree operate ia a manner similar to those in the sub-tree defined by the root node's left child, each processing element 11(i) will receive a value corresponding to data from processing elements 11(i-1) throug,h 11(0) combined according to the arithmetic operator of the particuJar scan operation.
The control network 14 can also perform a backward scan operation, in which the scan direction is from right to left, that is, toward processing elements 11(i) of lower indices. In that case, each processiDg element 11(i) wiD receive a value corresponding to data from processing elements ll(i+ 1) through 11(N) (where ~N~ is the highest index) combined accordiDg to the arithmetic operator of the particular scan operation. In that operatioD, each control net vork node 51(i,j,1) interchanges WO 92/06~36 PCI/US91/07383 2~933~3~ -18- l control net~vork message packets that it receives at its input tertninals from its children, and also the control net vork message packet that it transmits through the outputs to its children, and otherwise operates similar to that above. This effectively interchallges the left and right childreD at each level, so sbat if the control network nodes 51 otherwise operate as described above, the scan &ection will be reversed.
In addition, the control network 14 can perfonn a segmented scan operation, in which the processing elements 11 of a partition may be divided into two or more segments. In each case, the first processing element 11(i) in the first segment is the first processing element 11(i) in the partition. The fust processing element ll(i) in each succeeding segment transmits a control network message packet in which a seg nent bit is set. Each control network node 51(i,j,1) also includes a segment flag IS61 (Fig. 12B-lG). Each control network node 51(i,j,1) operates as described above, except that in transmitting control network message packets up the control network tree:
(a) if it receives a control net vork message packet from its right child in which the segrnent bil is set, it transmits in a control network message packet ~o its parent data corresponding only to the data in the control nenvork message packet received from the right child; a~d (b) if it receives a coDtrol nehvork message packet from the left child in which the segment bit is set, it sets its segment fJag 1561, and sets the segment bit in the cootrol network message packet it ;
that transmits to its parent.
In either case, the control network node 51 buffers the data received from the left child control network node in its scan buffer 1410, in the same marmer as in an unsegmented scan operation as described above.
In cormection with control nehvork message packets that are traDsmitted down the control network tree, each control net vork node 51, if its segment flag 15ol is set, transmits lo its right child a control net~vork message packet whose data corresponds to the value stored in the scan buffer 1410.
The control network node 51 transmits to it left child a control net vork message packet whose data corresponds to the data from its pareDt, in the same manner as in an unsegmented scan operation as described above.
It will be appreciated that the first processing element 11(i) which is the first in each segment, other than the processing element 11(i) comprising the first in the partition, will not receive the value zero, as required in Eqn. 1 above. However, since those processing elements 11, in initiating the scan operation, transmitted control network message packets whose segment bits were set, they are awase that they are the first processing elernents 11(i) in their respective segments, and can interpre~ the value receiwd as zero.
In a reduce operation for a particular arithmetic operator ~N on items of data ~D(i)~
maintained by the processing elements 11(ij produces at all of the processing elements 11 the same result ~R~:
R = D(0) ~ D(1) ~ D(2) ~ .... ~ D(i) [Eqn. y In a reduce operation, the arithmetic operator may constitute a number of types of operators, including, for exa~nple, signed or unsigned addition, OR, XOR a~d determination of a ma~num.

,,, , . , , , ~ :
. : .

.: :: . - .. . . : . : : : . .:

WO 92/06436 PCr/US91/07383 2~933~

In performing a reduce operation, the processing elements 11 transfer message packets therefor over the control net vork 14. The message packet provided by each processing element 11(i) includes that processing element's data item D(i). With reference to Fig. 4B, each control network node 51(1,j,C), Oll receiving a message packet from the processing elements connected thereto, performs the operation specified by the mathematical operator to generate an intermediate result, which it transmits in a message packet to its parent Dode 51(1J,P).
This operation is repeated at successive parent nodes at higher levels in the tree comprising control network 14 uDtil the message packets reach the root node. When the root node receives message pac~ets from both of its children, it performs the operation specified by the mathematical operator on the data from its t vo children to generate a result value. The root node generates message paclsets whose data is the result value and transmits them to both of its children. Each of the control networ3c nodes 51(i,j,1) that receiYes such a message pa~cet repeats it to both of its childrerl, until they reach the processing elements 11, thereby broadcasting the result to all of the processing elements 11.
- As noted abovej the Ieaves 21(i) may comprise a processing element 11(i), a scalar processor 12(i) or an input/output processor 13(i). In the above description, only the processing elements 11(i) have been indicated as engaging in scan operations and rednce operations. It will be appreciated, however, that scalar processors 12(i) and input/output processors 13(i) may, along with processing elements 11(i), engage in such operations. Alternatively, the scalar processors 12(i) and input/output processors 13(i) may abstain from the scan and reduce operations. They may accomplish this either by transmitting control network message packets which contain data having a value of zero, or by transmitting a special type of control network message packet, described below as an abstain type, which the control net vork nodes Sl(iJ,I) may treat as containing data having the value zero.
As noted above, each processing element 11 maintains a message counter which counts data router message packets it transmits and recçiws over the data router 15. The processing elemeDt 11 increments the message counter when it transmits a data router message packet over f~e data router 15 a~d decrements the cou~ter when it receives a data router message packet over the data router 15 during a message transfer operation. 1~ will be appreciated that during a message transfer operation some processing elements 11 may transmit more data router message packets than they receive, and thus at the end of the message transfer operation the mess~ge counter will have a positi~e value. On the other hand, some processing elements 11 may ~eceive more data router rnessage packets than they transmit duriDg the message trans&r operation, in w~ich case the message counter wiU have a negative value at the end of the messae transfer operation.
The processing elements 11 use the con~rol network 14, in particular enabling a reduce operation, to determine when the data router ~5 is empty, that is, when the data router 15 has delivered all data ruuler message packets to processing elements 11. More specifically, each processing element 11, after it transmits all of its data router message packets for the message transfer operation, begins transmitting control net vork message packets specifying a reduce operation, with signed addition as the arithmetic operator. The data in each control network message packet is the current value of the 20933~ -20-processing element's message counter. The processing elemellts 11 iteratively t~ansmit such control net vork message packets until they receive a control net vork message packet whose data has the result value of zero. It will be appreciated that, at that point the processing elements 11 have coDectively received as many data router message packets as they traosmitted during the message transfer operation, and so the data router 15 will be empty of data router message pacl~ets.
Fig. S tepicts the structure of a controi network message packet 60 that is transferred over the control network 14. With reference to Fig. 5, the control net~vork message packet 60 has a fn~ed length of thirteen ~flicks.~ In one embodiment, each aick has five bits, vith the furst t velve flicks, identified as FLICK 0 through FLICK 11, including four packet information bits (labelled ~PKT INFO~ in Fig. S) and one tag bit. The packet informatil)n portioD of the first twelYe aicks comprise a packet header portion 61 and a packet data portion 62. The thirteeDth flick, namely FLICK 12 identified by reference numeral 63, contains a checksum used in error detection. The checlcsum is generated across all five bits of the successive flicks in the packet 60. The tag bits contain control information as described below. -The packet header portion 61 includes four fields, including a message type field 64, a packeltype field 65, a combine function type field 66 and a pattern field 67(0) and 67(1) (c ollectively identified by reference numeral 67). The packet data portion 62 includes eight four-bit data nibbles 70(0) through 70(7) (generaUy identified by reference numeral 70) aDd a four-bit nibble 71 containing global inforrQation.
The message type field 64 identifies the type of message contained in the message packet 60.
In one embodimeDt, a packet 60 can coDtain one of five different types of messages, including an SS
Csingle source) message, an MS ~ultiple source) message, an ABS abstain message, an IDLE message and an NPAC nil packet message. When a scalar processor 12 broadcasts a command to the processiDg elements 11 for processiug thereby, it uses a single source message packet to carry the command. In addition, a scalar processor 12 may also use single source message packets ~o broadcast other types of control information to one or more of the processing elements 11 or input/output processors 13, or to another scalar processor 12.
A single source message packet is passed by each control network Dode S1(i~i,l) which receives it up the control Detwork tree &om }~ode to node untii it reaches the root node. The root node transmits the single source message packet down the tree to its cbildren. Each control network node Sl(iJ,I), which receives a single source message packet from its parent transmits it down the tree to both its children, effectively broadcasting the packet to all of the processing elements 11 iD the partition.
Muldple source messages are used by the processing elements 11 to initiate scan and reduce operations as described above. Idle message packets are transmitted when a leaf ~1 or control network node Sl(iJ,I) has no other types of message packets to transmit. A leaf 21 transmits abstaihl mes~age packets to indicate that it is not participating in a scan or reduce operation. If a control network node Sl(iJ,I) receives idle or abstain message packets from both of its children, it may transmit a message . . ' " " ' '.' ' ' . ' " ' ' ' '' ' ;.,: , ;' ~ ;J ' ; ' ' ' ;, , '', ' ~ . , ., . ' ' ' ': ' . ' 1 .

, ' J~
WO 92/Q6436 PCI/US91/073~3 2~33~

packet of the same type to its pareDt. If a control network node 51(iJ,l) receives a multiple source message packet from one of its children aDd an abstain message packet from its other child, it does not thereafter wait for a multiple source message packet therefrom to use in the arithmetic operation specified in the multiple source message packet that it receives from the one child. Instead, the control network node Sl(iJ,I) forwards the multiple source message packet that it receives to its parent, and, if the abstain message packet came from its right child, stores the data from the message packet in its scan buffer 1410.
A message packet of the nil packet type, unlike message packets of other message types, is only one flick in length. In particular, a nil packet message comprises only the message type flick 64, the COnteDtS indicating that the message packet is of the nil packet type. A control network Dode 51(i~i,1) continually transmits messages of the nil packet type to its parent while it [that is, the control network node 51(iJ,l)] is a logical root of a partition, aDd the paren~ transmits message packets of the same type to that child. If the parent receives a multiple source message packet from its other child, it forwards it to its parent.
The packet type field 65, combine function type field 66 and a pattern field 67 contain further information about the inforrnation in the control network message packet 60.
In one particular embodiment, the processing elements 11 can operate in two operational modes, identified herein as ~supervisor" aDd ~user.~ If the rnessage type field 64 indicates that the control network message packet is a single source message packet, the packet type field 6~ can identify a message packet as a broadcast supervisor packet or a broadcast user packet. If the packet type field 65 indicates that the control Detwork message packet is a broadcast supervisor packet, it contains a command for execution by the processing elemeDts 11 in the supervisor mode. On the other hand, if the packet type field indicates that the control network message packet contains a broadcast user packet, it contains a command for exewtion by the processing elements 11 in the user rnode.
In addition, if the message type field 64 indicates that the control networlc message packet is a single source message packet, the packet type field 65 may indicate that the control network message packet is an interrupt packet. The interrupt packet may be used to initiate operations at particular ones of the processing elements 11. The operations and the particular ones of the processing elements 11 to perform them may be identified in the packet data portion 62.
Further, if the message type field 64 indicates that the control network message packet is a single source message packet, the packet type field 65 may indicate that ehe control net vork message packet contains configuration information which enables the establishmeDt or eliminatioD of a logical root at a pareicular control network node 51(i,i,1). If the packet type field identifies th~ message packee as containing configuration information, the first two flicks 70(0) and 70(1) of packet data portion 62 contain data specifying the level and su~level in control network 14 at which the logical root is to be established. The control network node 51(i,j,1) at that level and sub-level which receives the configuration message packet establishes itself as the logical root.
If the message type field o4 identifies the messge packet as a multiple source message packet, the packet type field 65 identifies the operation to be performed as a scan involv~ng data i~ a single , :, : .., - :: , : .

: , .: - :. :. , : : : , ,: ; ~ . , . : . . . . .
.:, . : .:, : . . : , ~

WO 92/06436 PCI'/US91/07383 2 ~ 9 ~ 22-packet or a pluraliq of paekets, or to perform an operation to deter nine whether the data router 15 is empty. The data to be used is eontained in data fields 70(0) through 70(7) (generally identified by referenee numeral 70) of the paeket data portion 62. If the paeket type field 65 identifies a sean operation involving data in a single paeket, the sean operation is limited to a data value having a single thirty-two bit word. However, if the paeket ~pe field identifies a scan operation involving data in a plurality of successively-transmitted paeket, which ~nD be identified as a ~multi-word scan,~ the scan operation involves data values of more than thirty-two bits, whieh are eontained in eontrol network message paekets o0 suecessively transrnitted by the proeessing elements II. In either ease, if the paeket type field 65 identifies the operation as a scan operation, the pattern field 67 further identifies it as either a scan forward or sean baek~vard operatioo or a reduce operation, and combiDe function type field 66 identifies the particular arithrnetie operator to be used in the operatioD.
As has been described above, control network message packets of the multiple source type may be used, with arithmetic operations, to determine whether the data router 15 is empty, using the contents of message counters maintained by the processing elements II as data. Sihnilar control ne~work message paekets may also be used to perform other eontrol operations using, for example, bits of the global information field 71. For example, the scalar processors 12 may need to be notified when aD of the proeessing elements I1 have finished exeeuting a partieular comrnand before they transmit a subsequent eommand. In that ease, eaeh proeessing element when it has finished exeeuting a command, may tra~msit a control network message packet 6Q of the multiple source type, indicating a reduce operation using the OR operator, with a particular bit in the global information field 71 being set. It ~ill be appreciated that, after all of the proeessing elements 11 have exeeuted the istruction and transmitted corresponding paekets, the root node u~ill as the result of the reduce operation, broadcast eontrol network message packets down the control nenvork tree in which~the bit will be set. When the scalar processor 12 receives the resulting control net~,vork message packet from the control network node Sl(lJ,I) conneeted thereto, it caD determine the condition of the bit and determine therefrom that the command has been executed.
Bits of the global information field 71 may also be used by the processiIIg elements 11. In processing certain commands from the scalar proeessors 12, the proeessing elements 11 sometimes may reach a point in processing a command at which they have to verify that all of the processing elements have reached the same point before they proeeed. To aeeomplish that, when each processing element has reached the particular processing poirlt it may transmit a control nenvork message packet as described above, that is, of the multiple seuree type, indieating a reduce operation using the OR
operator, vith a particular bit in the global information field 71 being set. When the processing elements 11 reeeive the resulting eontrol lletwork message packet from their respective eontrol ne~work nodes Sl(lJ,I~ connected thereto, they can determine there&om that aD of the proeessing elements Il have reached the required point in their processing of the cornmand~ and contiDue processing.
The tag bits of the suceessive flicks in a control network message packet 60 eontain various types of control and status information. Several of the tag bits coDtrol the flow of control ne~vork : : : :~ : : : ~ : :

2~93~5~
-2~
message pacl~ets through the control network 14. Fi-te tag bits comprise scan flo~v bits, generally identified by refererlce l~umerals 72(i) (~i~ is an integer from ~1- through ~5~). The control network nodes 51~iJ,I), processing elements 11 and scalar processors 12, as wll as any input/output processors 13 which transmit and receive control network message packets over the control networlc 14, use the scan flow bits to control the transfer of message packe~s between directly-connected components in the con~rol network 14.
Two ty bits, including a broadcast usçr flow bit ;'3 and a broadcast supervisor flow bit 74 are conditioned by the processing elements 11, scalar processors 12 and those input/output processors 13 which transmit control network message packets over the control Detwork 14, to indicate whether they are able to receive control network message packets contaiDing conerol informatiorl for the supervisor and user modes respectively. Each processing element 11, scalar processor 12 and input/output processor 13, respectively, conditions bits 73 and 74 in any control network message packets tbat it transmits to indicate whether it can receive sinsle source message packets having packet types, as indicated in packet type field 6S, of broadcast supervisor type and bsoadcast ~scr type, respectively.
Anot'ner tag bit tnat controls the control ne~work 14 is a fl~sh bit 75. When a control network node 51(i,j,1) receives a control no~vork message packet ùl which the ll~h bit 75 is set, it clears its scan buffer. l~is may be used to clear iD~ermediate results of a sc~n or reduee operation from ~he control network 14 during a context switc'n.
A soft error bit 76 is used by a cor~trol network node 51(i,j,1) to indicate that it has detected a software error from the contents of a control n~twork message pacl~et 60. For e~ample, if the coDtrol network node 51(i,j,1) deterrnioes that the coDtents of the packet type 6eld 65 to not identify one of the estab'~ished pac~et types for the messagç type ide~tified m message type field 65, the node may seS the soft error bit 76.
As descri'oed a~ove, the control ne~vork 14 performs seg~nented scan operations using data in message pac cets transmitted by the processillg elemer,ts 11. A segment bit 77, ~hen set, indicates lhal the control network message pacl~et 60 coolains dala for the uppcr end of a segmen~. A scan overflow bit 80, when set, indicates that the result of the ari~hmetic operation is larger than can be ` accommodatcd in the data fields 70 of the control network message pac~et 60. T~e sa~ overflow bit 80 may alsc. 'oe ~l~ed to io.1icate overflow during a reduce o~lseration. If the scan overflow bit 80 is set.
the operation can be repeated in a multi-word operation.

. -. . ................... .... ..... .... .. . .

, . .

W O 92/06436 PC~r/US91/073X3 ~3~

..... .. ...

F~ally, a contsol rletwork messagc pacl~et 60 includes an AFD all-fall-dowll bit 81. If a co~trol network node Sl(-J,I) recei~cs a co ltrol Detworl~ message packet 60 in whic~ the AFD all-fall-do~n bit 81 is set, it asserts a~ ADF(~,I) all-fall-down sigl~al. The AFD(i~,l) all fall-do7~ (i~i) sign~l ~orn each parent control network note $1(i,~,~) Is cormected to tbe data iouter l~odes æ(i,i,~) o~ ~e data router r~ode group 20(iJ) ~ g thc s~e ~dices ~i~ and 3. DiagEIostic Network 16 As ~oled above, t~e dia8nostic network 16, under conlrol of a diagnostic processor, faci3itates tcsting of other portions of the system 10 to idenlify, locate and diagnose defects. 1~ addi~;on, the dia~ostic ~etworlt 16 may be uscd to establisb selected operat.ing conditions i~ ~e o~cr portiorls sf thc system 10 as dcscribed bclow. Tbe general structure of the diagDostic ne~ork 16, and its cor~cctio~ to the other eleme~ts of the system 10, Will b~ described in co~ectio~l wi~ Fgs. 6A
~hroug~ 6C. Messages tra~sfes~ed over the dia~ostic ~etworl~ 16 will be tescr~bcd in coDn~io~ h !SlJBSTl~ SI~EE~

... . , , ,,.",.. . . ........ .. .. ...... ... ..... .
,. . . , .... .-. ., . . " , . .... .. ,. ,,~ .. ,;., , ., , . ;,. . .

"'' " ;.'''"'' ' ''"' " "''' ~; `" "':' ;" ''"' ",' " ', ~'' ':'', WO 92t06436 P(:~/US91/073~3 .~ 2~933~3 Fig. 7.
With reference to Figs. 6A through 6C, the diagnostic network 16 iDcludes a plurality of diagnostic network node generally identified by reference numeral 100(h,p"-1), where "h' and "pncomprise integers representing a height value and a pod-type value, and nr r comprises one or more integers which together comprise a rQot-leaf value. The various diagnostic net vork nodes 1oo(h~p~r-l) are connected in a tree-type structure which actually forrns a tree of trees as shown in the Figs. In pardcular, the diagnostic network 16 includes a high-order tree ideDtified as a height-decodiDg tree, as represented by the diagnos~ic network nodes 100(h,p,r-1) in the left-most eolurnns of the respective Fips. 6A through 6C. Each diagnostic network node 100(h,p,r-1) in the height decoding tree is idendfied by a reference numeral 100(h,0,0...0), wkere the value of nhn is associaied with a level in the data router 15 and control network 14. A diagDostic pr~cessor 101 is connected to the diagrlostic network node 100(h,0,0...0~ at the highest level of the height decoding tree.
The height decoding tree is essentiaDy a linear tree, that is, there is no fan-out frs)m level to level iD the height decodiDg tree. The height decoding tree essentiaDy forrns the backbone of other lower-level trees in the diagnostic network 16, including a pod-type decodiDg tree, represented by diagnostic network n~des 100(h,p,r-1) in the middle col~amn of Figs. 6A through 6C, and a root-leaf decoding tree represented by diagnostic Detwork nGde 100(h,p,r-1) in the right-hand colurnn of Fig. 6A
throu~ 6C. In pardcular, depeDdiDg from each diaBnostic network node 100(h,0,0...0) in the height decoding tree is a diagnostic network node 100(n,1,0...0), which comprises the pod-type decoding tree.
Although oDly one diagnostic network node 100(h,1,0...0) is shown in the pod-type decoding tree at each level, the diagnostic network 16 may include multiple decoding nodes connected in a tree structure. In that case, the diagDostic network Dode 100(h,1,0...0) will comprise the root of the pod-type decoding tree, and other diagnostic network nodes 100(h,p,0...0) will comprise i~termediate nodes and leaves of the pod-type deco~ng tree.
In additio,n, depending from diaE~nostic network nodes 100(n,1,0...0) in the pod-type decoding tree are diagDostic network nodes 100(h,p,r-1) comprising the root-leaf decoding tree. As shown ~
Figs. 6A through 6C, depending frorn each diagnostic network nods 100(h,1,0...0) in the pod-type decoding tree is oDe or more trees of diagnostic network nodes 100(h,p,r-1) in the root-leaf decoding tree. In the embodirnent depicted in Figs. 6A through 6C, each diagnostic network node 100(h,p,r-1) can accommodate a fan-out of two, and so if the pod-type decoding tree includes one diagnostic network node 100(h,1,0...0), the diagnostic network 16 at that level may include up to two root-leaf decoding trees, which may connect to diverse types of other compolents in the system 10. Each root-leaf decoding tree includes a root diagnostic Detwork node 100(h,p,r...0) connected to tbe pod-tnpe decodins tree, and extends to a plurality of leaf diagnostic network nodes 100(h,p,r-l) coDneaed to a par icular Iype of pods in tne system 10.
The portions of system 10 comprising npods" may depeDd m,?oD tne physical embodiment of the particular system. As depicted OD Fips. 6A throup,h 6C, t'ne data router nodes 22(i,j,k) may comprise oDe type of pod, the coDtrol network Dodes Sl(i,j,l) may comprise a second type of pod, and the leaves :... ,; . ~ . . ,: ., ,, ,. ,- : , .: , WO 92/06436 P~/VS~l/07383 2(1~33~ -25-21 may comprise a third type of pod. As shown in Fig. 6A, level "M," which corresponds to the root level of the control network 14 and data router 15, includes two root leaf decodinE~ trees. One root-leaf decoding tree comprises the diagnostic network nodes identified by reference numerals 100(M,1,1...0) through 100(M,1,r-l), which is connected to the pods of the data router nodes in the root data router node group 20(M,0). The other root-leaf decoding tree comprises the diagnostic ~etwork node identified by reference numeral 100(M,2,1...0), which is connected to the pod comprising tbe root control network node group 50(M,0).
Similarly, level ~M-1," which corresponds to one level below the root level of the coDtrol network 14 and dat~ router 15, also includes two root-leaf dec~ding trees. One root-leaf decoding tree comprises the diagnostic network r odes identified by referente numerals 100(M-1,1,1...0) through 100(M,1,r-l), which is cormected to the pods of the data router nodes in the data router node groups 20(M-1J), one level below the root level. The other root-leaf decoding tree comprises the diagnostic network nodes identified by reference numerals 100(M,2,10...0), 100(M,2,11...0), and 100(M,2,12...0) which are connected to the pods comprising the root control network node group 50(M,0). The other levels of the diagnostic network 16, down to level ~1,n which corresponds to the lowest levels in the control network 14 and data router 15, are similar, including two root-leaf decoding trees, one connected to pods comprising the data router node groups 20(iJ) and the other connected to pods comprising the control net vork node groups 50(ili).
As indicated above, the diagnostic net vork 16 also includes a level ~on connected to leaves 21 in the system 10. That level includes wlly one root-leaf decoding tree, comprising the diagnostic network nodes 100(0,1,1...0) through 100(0,1,r-l), all of which are cormected to leaves 21.
A "pod" may comprise an individual data router node 22(iJ,k), control nehvork llode 50(iJ,l) or leaf 21, or groups thereof. In one particular embodiment, a npodn is a nfield-replaceable unit," such as an entire circuit board, which is replaceable by field-service or ~naintenance persomlel. In that embodiment, the diagnostic Detwork 16 can diagDose and locate failures in such field-replaceable units.
It will be appreciated that, if a pod-t,vpe decoding tree at any particular level includes multiple diagnostic network nodes 100~h,p,0...0) organized in a tree structure, multiple the root-leaf decoding trees can be provided each depending from a node comprisih~g a leaf of the p~d-~pe decodiug tree.
Thus, for example, if a particular level in the diagnostic net vork 16 required three or four root-leaf decoding trees, each connected to pods of particular types, if the fan-out from each level to the uext iD
the pod-type decoding tree is two, the pod-type decoding tree would include at least three diagnostic net vork nodes 100(h,p,r-l), including a root Dode and two leaf nodes connected thereto. In that case, e.ach leaf node would be able to connect to t vo root-leaf decoding irees. It uill be appreciated that, if the fan-outs in each of the trees is different from two, the number of levels and number of nodes in each level within each tree may also differ from that specificaUy described herein. In one particular embodiment, fan-outs in particular diagnostic network nodes 100(h,p,r-l) of both two and eight are used, at different levels in the respective trees comprising the diagnostic network 16.
The diagnostic net vork nodes 100(h,p,r-l) are generally similar, and will be described ;D detail in cormection with Figs. 13A through L3C. In brief, each diagnostic network node 100(h,p,r-l) includes WO 92/06436 PCr/US91/07383 2~933~
.26 an address control portion, generally identified by reference numeral 102, and a data control portion, generally identified by reference nuMeral 103. The address control portion of diagnostic net vork node 100(M,0,0...0) receives address control signals from the diagnostic processor over a bus 104(P). The node uses the address control signals to establish address state in an address state store 105.
The address state maintained by the diagnostic network Dode 100(M,0,0...0) eDables it to transmit subsequently-received address control signals (a) to one child node, in this case node 100(M-1,0,0...0) over a bus 104(Cl), (b) to the other child node, in this case node 100(M,1,0...0) over a bus 104(C2), (c) to both child nodes over the same buses, or, alternatively, (d) to neither child node. The node's address control portion 102 includes Qags 106(C1) and 106(C2) each associated wilh a corresponding bus 104(Cl) and 104(C2). If the flag 106(C;) is set in response to the received address control signals, the node is enabled to thereafter transrnit the address control signals to the respective child node over a bus 104(C;), and other vise it is clear.
The diagnostic processor 101 controls the conditioning of each of the flags 10o(C;) in the state store 105 of diagnostic network node- 100(M,0,0...0) serially. After the address state has been established in the state store 105 of diagnostic network node 100(M,O,O...0), the node transmits the address control signals that it subsequently receives over bus 104(P) from the diagnostic processor 101 over ehe particular buses 104(Cj) whose flags 1Oo(Cj) are set. If both flags 106(Cj) are set, the diagnostic network node 100(M,0,0...0) transrnits the address control signals over both buses 104(C;) in parallel. The address control signals thereafter enable either or both of those nodes to condition the flags lQ6(Ci) in their respective address state stores 105, enabli3~g thern to thereafter transmit the address control signaJs received thereby to either or both of the diagnostic network nodes 100(h,p,r-l) connected thereto. This process continues until flags 106(Cj) are set in selected ones of the leaf diagnostic network nodes 100(h,p,r-l) ;D the root-leaf decoding tree. This process may be repeated a~y nu~nber of times to condition flags 106(Cj) in auy combirlation of the leaf diagnostic network nodes 100(h p,r~
The sequerlce of flags '06(Cj) that are set in the various diagnostic network nodes 100(h,p,r-l), from the root diagnostic network node 100(1,0,0...0) in the height decoding tree to the ieaf ~;agnostic network nodes 100(h,p,r-l) in the root-leaf decoding trees, essentially form paths from the diagnostic processor 101 to selected pods. The paths may be subsequel~tly used to carry diagnostic test data in parallel frorn the diagnostic processor to the selected pods, and to return test results.
After it has conditioned flags 106(Cj) in ~he various diagnostic network nodes 100(h,p,r-l), the diagnostic processor 101 may also retrieve the state from each of the diagnostic rlet vork nodes 100(h,p,r-l). After each flag lOo(Cj) is conditioned, the diagnostic net vork node 100(h,p,r-l) may transmit a signal representing its state its state over its bus 104(P), which is coupled up the tree to the diagnostic processor 101. If multiple flags are conditioned in diverse nodes in paralleL the diagnostic processor 101 transmits an expected address data signal7 which enable the nodes intermediate the originating nodes and the diagnostic processor to combine the signals representiDg the state of the respective flags in response to a control signal from the diagnostic processor 101.

. ' ' - ' ' ' ' . ' '' ". J '" " ' ~' . ' ' ' . ' , ~ , ', '' .... '' , ,. .. ' . ' WO 92/06436 PCr/US9l/0738 0 l~ s ~ -27-Thus, if the flags 106(Cj) whose conditions are being retrieved are to be set, resulting in asserted state signals, the diagnostic processor 101 rnay enable the interrnediate nodes to logically AND the flag state signals received from their child nodes. In that case, if an irlterrnediate node receives a negated state signal, indicating that the flag 106(C;) whose condition is received is, erroneously, not set, the node will provide a negated state signal, which wiD be propagated up the tree to the diagnostic processor 101. On the other hand, if the flags whose conditions are being retrieved are to be cleared, resulting in negated state signals, the diagnostic processor 101 may enable the interrnediate nodes to logicaDy OR the flag state signals received from their child nodes. ID that case, if an intermediate node re~eives an asserted state signal, indicatiDg that the nag 106~C;) whose condition is received is, erroneously, not clear, the node will provide an asscrted state sigDal, which will be propagated up the tree to the diagnostic processor 101.
After the diagnostic processor 101 has established the address states in the respective diagnostic network nodes 100~h,p,r-l) to selected pods, it may transmit a test data out signal and an expected test data control signal, which are received by the root di;ignostic network node 100(M,0,0...0), over a bus 110(P). The root diagnostic network node 100(M,0,0...0) transmits the received signals over respective buses 110(C1) and 110(C2), as determined by the states of the respective flags 10o(C;), and the other diagnostic net vorlc nodes do the same. Thus, the diagnostic net vork Dodes 100(b,p,r-1) couple the ~est data out sig~al arld expected test data control signal do vn the respective trees along paths defined by the set llags 106(Cj). At some point, at least solne of the leaf diagnosdc network nodes 100(h,p,r-l) wiD couple test data signals to the selected pods, and obtain test data out signals representing diagnostic test results.
The diagnostic network Dodes 100(h,p,r-l) will pass the test data out signals up the paths defined by the set flags 106(Cj), each node combining the test data out signals received from its children in response to the expected test data control signal in a manner similar to that desibed above in connection vith retrieval of the states of the respective flags. That is, if the test data out signal is expected to be asserted, the diagrlostic processor 101 may enable the nodes to logically AND the test data signals received from the pods or child nodes connected thereto. In that case, if an intermediate node receives an erroneous negated test data out signal, the Dode u~ill provide a negated test data out signal to its pasent, which will be propagated up the tree defining the diagnostic Detwork 16 to tbe diagnostic processor 101. On the other hand, if the test data out signal is expected to be negated, the diagnostic processor 101 may enable the interrnediate nodes to logically OR the test data out signals received from the pods os the child nodes connected thereto. In that case, if an iDtermediate node receives a~ erroneous asserted test data out signal, the node will provide an asserted test data out signal to its parent, which will be propagated up the tree to the diagnostic processor 101.
If the diagnostic processor 101 receives an elToneous test data out signal, it can thereafter repeat the operations in connection with subsets of tbe previously-selected pods to identif~ the one which provided the erroneous signal. In that operation, the diagnostic processor 101 establishes states of the address flags 106(Cj) in the diagnostic network nodes 100(h,p,r-l) to eslabiish paths therethrough 2~933$~
to a selected subset and repeats the test operation in cormection witb that subset. If the test data out signal indicates an erroneous result, the diagnostic processor 101 caD reduce the size of the subset and repeat the operation. If the test data out signal indicates a correct resu~t, on the other hand, the diagnostic processor 101 can repeat tb,e operation in connection with a different subset. In one embodiment, the diagnostic processor 101 performs a binary search operation, iteratively repeating the operation in connection with half of the pods selected during the previous iteration to locate the pod providing the erroneous test data out sigr al.
Although not shown in Figs. 6A through 6C, the diagnostic network 16 may include multiple diagnostic processors connected to various ones of the diagnostic network Dodes 100(h,p,r-1). Each diagnostic processor may selectively control the portions of the tree defi~g the diagnostic network 16 below the diagnostic network node 100(h,p,r-l) connected thereto. Alternatively, the diagnostic processors may selectively condition the diagnostic net vork nodes 100(h,p,r-1) connected thereto to receive signals from, and transmit signals to, their respective parent diagnostic network nodes lOO(h?p,r-l). The additional diagnostic processors may facilitate diverse diagnostic operations in various parts of the system 1û in concurrently.
In one specific embodiment, the interface between the leaf diagnostic network nodes 100(h,p,r-1) and the ~"ods comprises the interface defined by the Jo~t Test Action Group (~JTAG~), as described in IEEE Std. 1149.1 (hereinafter aJTAG specification~). In any event, the interface provides a serial scan chain CiTCUit in each pod. The serial scan chain circ Ut iD each pod may extend through a nu~nber of registers and other storage elements in the respective pods, and may be used to establish the states thereof to thereby establish selected operating conditions in the respective pods. For example, each data router nodes æ(ij,k) and control network nodes 51(iJ,I) uses beight signals identifying the respective levels, which may be provided by R register thereon that can be loaded through the serial scan chain circuit. These Dodes also use signals which indicate whether cor~ections to the respective parent or child nodes are enabled or disabled, which may also be provided by registers loaded tnrough the serial scan chain circuit.
Fig. 7 depicts the stmcture of a diagnostic message packet 120. Diagnostic message packets 120 differ from the data router message packets 30 and control network message packets 60, in that they are not generated by pods connected to the diagnostic network 16 for delivery through the diagnostic net vork 16 to other pods connected thereto. The diagnostic network message packets are generated by a diagnostic processor for delivery to the pods, whicn, in turn, generate response data for transmission to the diagnostic processor.
In any event, the diagnostic message packet 120 includes an address portion 121 and a test data portion 122. The addJess portion 121 conditions the respective address control portions 102 in the diagnostic nenvork nodes 100(h,p,r-1). The test da~ra portion 122 is represented by tne test data in and and test data out signals. aloDg ~vith the expected test data in signals, coupled through t'ne data control portions 103 of the respective diagnostic network nodes 10û(n,p,r-1). Depending on the location of the diagnostic processor generating the diagnostic message packet 120, the packet 120 rnay include three ,f~

~ o ~ r) ~i -29-sections in the address portion 121, including a height identification portion 123, a pod-type identification portion 124 and a root-ieaf identification portion 125. Each of thc portions 123 through 125 are used by diagnostic network nodes 100(h,p,r-1) in the respective height, pod-type and root-leaf decoding trees to condition the respective nags 106(C;) therein. It uill be appreciated that the length of the respective portions 1~3 through 125 will vary, dependi~, upon the nurn'oer of diagnostic network nodes lOO(h,p,r-l) whose flags 10o(C;) are to be conditioned, and the number of flags in each node.

.. . ..

,, .,: ., : : : : : : . :: ,, , : , : . ; , , :, .

~-: : .:: ' " ~ ' : . : ~ '. ., . .. :: :, .. :. :

WO 92/06436 P~r/US91/07383 20~33~a . Det~uled Desc~Lption Of Particular Circuits A. General The remainder of this specification will present details of circuits used in one embodiment to carrv out the invention as set forth in the claims. In the following, the detailed logic of a leaf Z1, in partic~llar details of connection of a leaf 21 to the control network 14 and data router 15 will be discussed in connection vith Figs. 8 through 10G. Thereafter, the detailed logic of a data router node æ(iJ,k) will be described in connection witb Figs. 1LA through 11D, the detailed logic of a control network node 51(tJ,I) will be described in connection with Figs. 12A through 12D-1, and the detailed logic of a diagnostic network node 100 will be described in connectioa vith Figs. 13A through 13C.
B. Leaf 21 1. General Fig. 8 is a general block diagram of a leaf 21, in pasticular, a processirg element 11 in the computer system 10 depicted in Fig. 1. Other types of leaves, including a scalar processor 12 and an input/output processor 13 are generally similar at a block diagram level, except as noted below. - - -With reference to Fig. 8, processing element 11 includes a processor 200, memoTy 201 andnetwork iuterface 202 all interconnected by a memory bus 203. The network interface 202 interfaces the proccssistg element 11 to the various cornmunication mechanisms 14,15 and 16 in system 10. In particular, the network interface 202 includes a control network interhce 204 (described below in more detaa in connection with Figs. 10A-1 through 10G) that receives (ejects) control network message packets 60 from the control network 14, and that transmits (injec~s) control network message packets 60 to the control network 14. Similarly, a data router interface 205 (described below in morc detail in connection with Figs. 9A-1 through 9D-7) receives (ejects) data router message packets 30 from t~e data router 15 and transmits (injects) data router message packets 30 to the data router 15, and a diagnostic network interface 206 receives diagnostic network message packets from the diagnostic network 16 and transmits diagnostic network results over the diagnosdc network 16. Fig. 14 clepicts the logic diagram of an interface circuit that may be used as the diagnostic net vork intesface 20S to interface the nehvork interface 202 to the diagnostic net vork 16.
As noted above, scalar processors 12 and input/outpui processors 13 are generally similar, at a block diagram level, to the processing element 11 depicted on Fig. 8. Scalar processors 12 may also include, for example, video display terminals (not shown) which may comprise consoles to allow control of the system 10 by an operator. In addition, scalar proccssors 12 may include such elements as, for example, magnetic disk storage subsystems (also not shown) to store progra ns ant data to be processed by the processor. It ~ill be appreciated that processing element 11 may also include such elements. As noted above, an input/output processor 13 will include interfaces to external data inpuljoulput and slorage devices, including frame buffers, magnetic disk storage devices, and other such elements that are well kno vn in the art.
The network interface 202 includes a clock buffer ~07 that receives the SYS CLK system clock signal from the clock circuit 17 and generates a NODE CLK node clock signal in response. In one - . .. . .

- ~ : , .,, , ": , "
, ~, ,. ;: . ,,. ., ; :-.;
,, , , . . . : 1. ' ' ' ' !, ....
. , . ' ' ' . . . '.
". '. ' ' '. . ' , ''~' ' ,', ~ ~ ., . ' WO 92/06436 PCr/US91/0738 ~ 0 9 ~ 31-particular embodiment, the clock buffer 207 comprises a buffer as described in U.S. Patent Appn. Ser.
No. 07/489,079, filed March 5, 1,90, in the name of W. Daniel Hillis, et al., entitled Digital Clock Buffer Circuit Providing Controllable Delay, and assigned to the assignee of the present application.
The network interface 202 uses the NODE CLK node clock signal to synchronize its operation with the control network 14, data router 15, and diagnostic network 16. The NODE CLK node clock sigDal may also be used in generating clock signals for controlling the other components of the processing element 11 shown in Fig. 8, but it will be appreciated that those components may alternatively be controlled by signals other than the NODE CLK node clock signal.
The memory bus 203 transfers address signals that defme a processing element address space.
The memory 201 includes a memory controller 208 and a plurality of memory banks generally identified by reference numeral 210, the memory banks 210 inclu&g a plurality of addressable storage locations within the processing element address space. In addition, the control network interface 2W
and data rouLer interface 205 include a plurality of registers, described in more detail below, which are also vithin the processing element address space.
The interfaces 204, 205 and 206 are connected through a bus 211 to a processing elemerlt interfau 2~!, which, in turn, is connected to the memory b~s 203. In response to receipt of control net vork message packets 60 from tbe control network 14 or diagnostic network message packets 30 from the data router 15, the processiDg element inierface 212 can interrupt the processor 200. In response to the interrupt, the processor 200 can, by reading appropriate registers in tbe respective interface 204 or 205, retrieve the contents of the packet from the network interface 202. The processor may store the retrieved packet contents in the memory 201.
In addition, the processor 200 can initiate transfer of a control net vork messagc packet 60 over the control network 14 or a data router Dessage parket 30 over the data router lS. In this operation, the processor 200 ~rarsmits packet information over bus 203 to particul~r registers in the network interface 202. The processing element interface 212, in respol~se to address signals Gver memory bus 203 identifying the registers, receives the paclcet information and loads it into the respective registers. Upon rec~iving the packet information, the respective interface 204 or 20S
initiates transmission of a message packet 60 or 30 over the respective contsol network 14 or data router 15.
The processor 200 executes the commands transmitted in control net vor!c message packets 16 over the control network 14 by the scalar processors 12 and received by the control network interface 204. In response to a command, the processor 200 processes one or more instructions, which are maintained in memory 201, which may enable the processor 200 to process data in the memory 20L In addition, the instructions may enable the processor 200 to transmit packet informa~ion to respective registers in the network interface 202 to initiate a transfer of a packet 30 or 60 over the respective data router 15 or control network 14, or to read information from respective registers ts~ thereby retrieve the received packet information.
2. Data Router Interface 205 . . . ': . ;, . , ' , ,,, . : ' ~ . ., ' . . ' .: "` . ,i ' 2~33~

i. General The details of data router interface 205 Will be described in connection with Figs. 9A-1 through 9D-7. With reference to Fig. 9A-1, the data router interface 205 hcludes a data router message injector portion 220, a message ejector portion 221 and an injector/ejector common con~rol/status portion 222, all connected to processing element interface bus 211. The data router message injector portion 220 injects data router message packets 30 over the data router lS; that is, it transmits data router message packets 3û to the data router nodes 22(1,j,0) and 22(1,j,1) connected thereto.
The data router message injector portion 220 includes two rnessage injector ports ideDtified as left message iDjector port 223(1) and right message injector port 223(r) for injecting message packets 30 into the data router lS. In the following, data router node 22(1,j,0~ is terrned the ~left" node, aDd data router node 22(1,j,1) is termed the ~right~ node; in that case, left message illjector port 223(1) is connected to transmit data router message packets to data router node æ(1,j,0) and right message injector port 223(r) is connected to transmit data router message packets to data router node 22(1,j,1).
Data router message iDjector portion 220 also hcludes an injector commoD control/status portion 224 that connects to, and controls certain operations of, both left and right message injector ports 223(1) and 223(r). For example, wherl the processor 200 initiates transmission of a data router message packet 30, it may specif,v that the message packet 30 be transmitted through either the left or the right message injector port 223(1) or 223(r). In that case, the data router inter~ace 205 will transmit the packet 30 through the specified port 223(1) or 223~r). AlSerDatively, the processor may not specifv the particular port æ3(1) or 223(r), in which case the injector common control/status portion 224 will select one of the ports 223(1) or æ3(r) to transmit the packet 30.
The rnessage ejector pordon æl receives and buffers data router message packets 30 from the data router lS. In addition, the message ejector portion æl may mitiate interruptiDg of the processor 200 OD receiving a new data router message packet, and it transmits the buffered packets over the prOcesSiDg element interSace bus 211 iD response to a retrieval request from the processor 200. The message ejector portion 221 includes a left message ejector port 225(1) and a right message ejector poTt æS(r) that are conrlected to receive data router message packets 30 from data router nodes æ (1,j,0) and æ(1,j,1), respecdvely.
Data router message ejector portion 221 also includes an ejector common control/status portion 226 that cormects to, and controls certaiD operations of, both left and right message ejector ports 225(1) and 225(r). For example, if both right and left ejector ports æS(I) and ~S(r) have received message packets 30 aDd the procewor 200 has requested that the message data be transmitted to it vithout identifying either the particular left or right ejector port 225(1) or 225(r), the ejector cornmon control/status portio~ 226 determines the order in which the ports 225(1) and æS~r) will transmit the packets over the procewing element interface bus 211.
To transmit a data router message packet 30 to the data router node 22(1,j,0) connected thereto, the left message injector port æ3(1), in synchrony with the NODE CLK node clock signal, iteratively transmits (L) IN FLIT left inject flit signals to transmit successive nitS of the packet 30 to the ,, , , ,, , ; :, , , ~: .

WO 9~/06436 PCr/US9l/07383 2~933~ -data router Dode æ~1J,0). The leh rnessage injeclor port æ3(1) may transmit w~ile the data router node 22(1,j,0) is asserting an (L) IN FLY left input fly signal; if the data router node æ(lJ,O) negates the (L) IN FLY left input ny signal the leR rnessage injector port 223(1) stops transmitting. The right message injector port 223(r) transmits similar (R) IN Fl,IT right inject nit signals to data router node æ(1J,1) in response to an asserted (R) IN FLY right input ny signal.
The left message ejector port 225(1), in synchrony with the NODE CLK node clock signal, iteratively receives (L) OU T FLIT left eject flit signals to for successive flits of the packet 30 from the data router node 22(1,j,0). The left message ejector port 225(1) may ena'ole the data router node æ(1J,0) to transmit by asserting an (L) OUT FLY left eject ny si~al; if the port 225(1) negates the (L) OUT FLY left eject ny s4~al the data router node 22(1,j,0) stops transmitting. The data router node æ(1,j,1) transmits similar (R) OUT Fl.IT right eject flit signals to rigbt message ejector port 225(r) in response to an asserted (R) OUT Fl.Y right eject fly signal.
Figs. 9A-2A and 9A-2B depict the registers in the control/status portions 222, 224 and 220 ~n the data router interface 205. Fig. 9A-2A depicts the details of a data router interface middle register -set 230 which is used by the processor 200 when it does not specify tne particular message injector port 223(1) or 223(r) to transmit a particular data router message packet 30, or the message ejec~or port 225(1) or 225(r) from which it jr, to receive a data router message packet 30. With reference to Fig. 9A-2A, register set 230 includes t.vo status and control registers, including a status register 231 and a private register 232, a receive register 233, and two transmit registers, namely, a ~send first" register 234 and a ~send" register 235.
The status register 231 includes a number of fields shown in Fig. 9A-2A. As described 'oelow in connection with Fig. 9B-1, eacn data router message injector port 223(1) and 223(r) includes a first-in frst-out buffer which buffers inforrnation from processor 200 from whicrl the pac~et 30 is generated. A
send space field 240 identifies the amount of space left in tne buffer in the particular port 223(1) or 223(r) that is currently selected t~s transmit the packet 30. The contents of the send space field 240 ar provided by the currently selected left or right data router rnessage injector port 223(1) or 223(r).
Two flags 241 and 242 indicate the status of the last reception aud transmission, respectively, of a data router message packet 30 through the currently selected port. If the last data router message packet 30 to be received can be successfully received, nag 241 is set, and if the last data router message packet 30 to be injected was successfully iDjected, flag 242 is set. The ~ags 241 and 242 are conditioned by message injector pordon 220 and me~sage ejector portion æ1, respectively.
A receive message length field 243 indicates the length of the data router message packet 30 received through the currently selectcd port, and a lellgth left field 244 identiies the amount of data in a data router message packet 30 currently being retrieved by the processor 200 tha~ is remaining ~o be retlieved. The con~en~s Qf the receive message length field 243 correspond to the contents of length field 34 (Fig. 3) of the data router message packet 30. The contents of a receive tag field 245 correspond to the conteDts of the tag field 3S of the same data router message packet 30. The leugth left field 244 is effectively provided by a counter into which the contents of length field 34 are loaded ~, : . : -,, ,, : :

::. :: . ~, . -:. - . ~ :
. .

WO 92/06436 P~tUS91/07383 2~933~

wher~ the processor 200 begins retrieving the Message packet 30, and which is decremented as the message data is transmitted to the processor 200. The contents of f,clds 243, 244 and 245 are provided by the message ejector portion æ1.
A send state field 246 and receive state field 247 identify the state of injection and ejection, respectively, of respective message packets 30 by the message ejectos portion 220 and message injector portion 221. The seDd state field 246, whose contents are provided by the message injector portion 220, indicates whether either or both of the left or right message injector ports 223(1) and 223(r) contain partially-injected data router message packets 30. Similarly, the receive state field 247, whose conteDts are provided by the message ejector portion æ1, indicates whether either or both of the left or right message ejector ports 225(1) and æ5(r) CODtiUn partially-ejected (that is, received) data router message packets 30.
Finally, a router done nag 248, whose contents are actually provided by the control network interface 204, iDdicates whether the router is empty following a message transfer operation. The condition of the router done nag 248 is derived from the reduce operation performed over the control net vork 14 to determine whether the data router 15 is empty as desaibed above. `
The private register 232 also includes a number of fields, comprising flags 250 through 256.
Several flags, which are included in the ejector common control/status portion 226, control the operation of the message ejector portion æL A receive interrupt enable flag 250, when set, enables the data router interface 205 to generate an interrupt for transmission by the network interface 202 to processor 200 when a data router rnessage packet 30 is received by the currently selected left or right message ejector port 225(1~ or æ5(r). A receive stop flag 2S2, when set by the processor 200, &ables reception of subsequent data router message packets 30 by the currently selected left or riBht rnessage ejector port 225(1) or 225(r). The currently selected port 225(1) or 225(r) stops receiving flits immediately upon the flag 252 being set. A receiver full nag 252, when set by the currently-selected ejector port 225(1) or 225(r), indicates that a buffer maintained by the currently-selected ejector port is .
full.
The private register 232 also iDcludes a lock flag 251, inclutet in the injector common control/status portion 224, that controls the operation of the message iDjeaor portion 220. The lock 9ag 251 enables or disables the currently selected left or right message injector port 223(1) or 223(r).
When set by processor 200, the currently selected left or right message iDjeaor port 223~1) or 223(r) i~ores subsequent transmissiorls from processor 200, and the 9ag 242 in status register 231 is cleared, indicating unsuccessful irljection of the data router message packet 30.
The private register 232 also includes three flags that control operation of the data router interface 205 in con~ection with the all-fall-down mode of the data router 15 as described above. A
received all-fall-down Qag 254, controlled by the control network interface 204, indicates that it has received a data router message packet 30 while the data router 15 is operating in all-fall-down mode, for which the leaf 21 is not the destination. An all-faU-down iDterrupt enable Qag 255, when set by processor 200, enables the network interface 202 to generate an interrupt request fol~ transmission to ~- - ; ""

WO 92/06436 PCr/US91/07383 2~3~ 35-the processor upon the setting of the r~ceived all-fall-down nag 254. Finally, an all-faU-down enable nag 256, when set by processor 200, enables the control network interface 204 to set the aU-fall-down bit 81 of the next control network message packet 60 that it transmits.
The remaining registers in the middle interface register set 230 are used to transmit and receive data router message packet information. A receive register 233 contains a number of words 260(0) through 260~N) representing the da~a in a data router nnessage packet 30 received through the currently selected left or right message ejeaor port 225(1) or 225(r). In reference nurneral 260(N), ~N~
is an inteBer related to the ma~mum amount of data that can be transmitted in a single data router message packet 30. The data stored in receive register 233 is from the data flits 36 of the received message packet 30. The receive register is represented by a single address in the address space of memory bus 203. The processor can retrieve the data from a message by iteratively using the address in a read operation over memory bus 203. It will be appreciated that the data router inter&ce 205 decrements the contents of the receive length left field 244 as the processor 200 accesses the receive register to retrieve the message data. ~
Two registers, namely, the send first register 234 and the send register 235 are provided to enable the processor to supply information used by the message injector portion to generate data router message packets 30 for injection into the data router 15. The send first register 234 includes fields 270 aDd 271 in which message length and message tag infonnation is loaded. The contents of fields 270 and 271 are copied into the message length and message tag fields 34 and 35 in a data router message packet 30.
The send first register 234 also includes a message address field 273 that is used to generate the contents of message address portion 31 of packet 30, ar,d an address mode field 272. The mes~,e address in field 273 can be an physical address, which specifically identifies the lleaf 21(y) to receive the message, or a relative address, which identifies a displacement from the leaf 21(i) ident~led by the partition base register 305, that is, the leaf 21(i) in the system that is the lowest-~dexed element in the partition, to the leaf 21(y) to receive the paclcet 30. The contents of the address mode field 2;'2 indicate whether the message address in field 273 is a physical address or a relative address.
The send register 235, like receive register 233, contains a number of words 280(0) through 280(N) representing the data in a data router message pacxet 30 to be transmitted through the currently selected left or right message injector port 223(1) or 223(r). In reference numeral 28,0(N), "N~
is an integer rela~ed to the ma~num amount of data ~hat can be transmitted in a s ngle data router message packet 30. The data stored in send register 235 is copied iDto the data tlits 36 of the transmitted message packet 30. The send regis~er is represented by a single address in tlle address space of memory bus 203. The processor can load data into the register by iteratively UsiJg the addres., in a write operation over memory bos 203.
As noted above, tne processor 200 ~3ses îhe data router interface middle register set 230 when it does not specify the particular message irljector port 223(1) or 223(r) to transmit a particular data router message packet 30. The data router interface 205 includes two additional register sets, ,. ::, . ., ., ,., . :-- . .

. . : . . .::. . ... .: .

WO 92/0643~ PCI/US91/07383 2~933~3 identified as a left and right interhce register sets 29U and 291 (shown on Fig. 9A-2B), respectively, which the processor 200 uses when specifics a leh or right message injector port 223(1) or æ3(r) to transmit a particular data router message packet 30, or a left or right message ejector port 225(1) or æ5(r) from which it will retrieve data router message packet data. Both left and right interface register sets 290 and 291 include respective status, private, receive, send first and send registers, identified by reference numerals 293-297 (left register set 290) and 3~304 (right registeT set 291).
The registers in register sets 290 and 291 have fields and flags that are substantially the same as those of respective registers 231-23S of the middle interface register set, except that the left and right interface status registers 293 and 300 do not have fields corresponding to seDd and receive state fields 246 and 247 or router done flag 248 of status register 231. In addition, left and right interface private registers 294 and 301 do not have fields corresponding to all-fall-dowD iDterrupt enable nag 255 or all- -fall-down enable flag 256 of private register 232.
The data router interface 205 also includes a set of registers 292 which contain information that it uses, along with the rnessage address information in field 273 of the send first register 234 of the middle interface register set 230 or corresponding fields of send first registers 296 or 303 of the respective left or right interface register set, in generating address information for the message address field 31 of a data router message packet 30 to be transmitted. As described above, the system 10 can be partitioned, and a partition base register 305 and partition size register 306 contain values identifying the base and size of the processing element's partition. ID particular, the partition base register 305 contains the index (i) of the leaf 21(i) in the system tha~ is the lowest-indexed element in the partition. In addition, the contents of the partition size register 306 identify the number of leaves 21 in the partition. A physical self address register 312 for a particular leaf 21(i) ideDtifies the leaPs own index ~i~ in the system 10, which comprises an address or other identifier that ur~iquely identifies the leaf 21 in the system.
The address information register set 292 also includes a set of registers 307 and 31~311 whicb are used with a chunlc table 327 (which is described in more detail below in cor~ection with Figs. 9B-1 and 9B-2) and that provides additional addressing information used in generating address information for the message address field 31 of a data router message packet 30 to be transmitted. The chunk table is maintained by the injector common control/status portion 224 of the injector por~ion 2~0. The churlk table pennits groups of leaves 21(i) to be to be substituted for other groups, without requinng the address provided by the processor 200 to reflect the substitution. This may facilita~e, for example, one or more groups of leaves 21(i) being made inaccessible, or ~mapped out,~ of the data router 15, which may be helpful if one or more leaves 21(i) in the group is defective. The chunk table contains information that is used in generating a portion of the message address for a data router message packet 30, permitting a message packet to be re-directed from the leaf 21(i) irl the original group to the leaf 21(i) of a group that is assigned as a substitute ~oup.
To accommodate that operation, as described below in more detail in connection vith FK. 9B-2, the chunk table includes a rnemory including a plurality of entries. Eacl~ entry comprises a pohlter, -- -, . , ~: : : , .
., ~ . : , . .
: , , , WO 92/06436 PCr/U~.~l/07383 20933~ ~37~
or bits comprising a portion of the address to be used as a substitute. If the address from the processor 200, in tbe particular send first reg~ster 23~'., 296 or 303, is a relat;ve address, a portion of it is used to access the chunk table, to obtain the portion to be used in generating the address for the rnessage address portion of the data router message packet 30. Tbe particular portion of the relative address used to address the chunk table depends on the size of the group to be mapped in and out.
A rhunk table address register 307 and chunk table data register 310 are used together to enable loading of the entries in the chunk table. The chunk table address in register 307 is used to identify the entry in tbe chuuk table into which the COrlteDtS of the cbuDk tablf data register 310 will be stored. Tbe contents of the cbunk size register 311 identify the uurnber of leaves 21(i) in a group which may be mapped in or out, whicb, in turn, deterrnines the particular bits of the relative address to be used in accessing the chunk table 327.
Finally, the registers maintained by the data router interface 205 include the previously-mentioned data router message counter 31.3. Data router message counter 313 is maintained by the injector/ejector cominon control/status portion 2æ. The message counter 313 is iDcremented to reflect the injection by data router message injector port 220 of a data router message packet over the data router 15 during a message transfer operation, ard decremented to refl~ct the ejection, by the data router message ejector port 221 of a data router mess~e packet 30 that it rece*es from the data router 15. The injector/ejector common control/status portion 2~ generates a CUR MSG. CNT
current message count signal which identifies the curreDt value of tne message co~mter 313, and which it provides to tbe control network interface 204 for use in generating a routcr done control network message as described above.
ii. Message Injector PoJtion 220 With this backgrourld, details of circuits comprising the left and right message injector ports 223(1) and 223(r), along with portions of the injector common control/status portion 224, in the message injector portion 220 ~ill be described in cormection with Flgs. 9B-1 through 9B-8. Similarly, details of circuits comprising the left and right message ejector ports 225(1) and 225(r), along with portions of the ejector common control/status portion 226, in the message ejector poltion æ1 will be described in cormection with Figs. 9C-A through 9C-7. ID addition, details of circuits comprising the injector/ejector commo~ control/status portion 222 will be described in connection with Figs. 9D-1 through 9D-7.
Since the circuits comprising the left and right ~nessage injector ports 223(1) and 223(r) are substandally similar, only orle (without reference to it being the left or right port) ~ill be described in cormection with Flgs. 9B-1 through 9B-8. Fig. 9B-1 tepicts a general block diagraln of a message injeaor port 223. With reference to Fig 9B-l, the message injector port 223 includes a series of stages thal receive irlfonnation from the processor 200 ~o be used in 8eneradng the message packet 30 ~o be injected, buffer the information, perform any Decessary address geDeration, divide the infor~ation into flits and, under control of the IN FLY injector fly signal, transmit the fli!s as successive IN FLIT
injected flit signals. In the following description, the information received from tbe processor 200 is in ,: , ; ~, : . , .~. : , . :

: ~ : . : : ,. , , . . :,:, ;: , WO 92t06436 P(~/US91/07383 20~33~a the form of words each having, for example, thirly two bits, and each flit comprises four bits trallsmitted in parallel.
Information from the processor 200, received by the processing element interface 212, is first loaded into a write stage 320. The write stage 320 is conne ted to the processing element interface bus 211, is loaded in response to LD CI RL load control signals from the injector common control/status portion 224. The write stage 320 operates as the input to an injector first-in first-out buffer (E~IFO) 321. When the processor 200 loads iDforrnation into the send first registers 234, 296 and 303, or the send registers 235, 29 7 aud 3~, it is essendally received by the write stage 320.
If the information from the processor 200 is addressed to a send first register, the write stage 320 generates a thirty-four bit address word in which the low order twenty bits comprise the address information, the middle bits comprise message packet length and tag information and the high-order bits comprise two address mode bits which it derives from the address mode field 272. In particular, if one of the address mode bits is set, the address information is an physical address, and if the other address mode bit is set the address informadon is a relative- address. In response to the suc~essive words thereafter addressed by the processor 200 to the send register, the write stage 320 generates successive thirty-four bit words in which the t vo high-order address mode bits have the value zero, with the thirty-two bits data appearing in the low-order portion. It will be appreciated that the ~ero address mode bits effecdvely identify the low-order thirty-two bits as comprising daea.
The output of write stage 320, INJ FIFO DATA (33:0) injector first-in first-out buffer data signals def~ning thirty-four bit words, are coupled to the data input terminals of FIFO 321. The ~IFO
321 stores successive words received thereby in thirty-four bit storage locations, ~hereby accommodating the thirq-four bit address words. The FIFO 321 loads INl FIFO DATA (33:0) sigr als in response to FIFO LD E~ load enable signals from the injector common control/status portion æ~
and the NODE CLK sigDal. ID addi~ion, FIFO 321 geDerates FIFO STATUS status sign~ls indicating whether it can accept additional words from S~e write state 320. The injector common control/status portion Z4 can use the FTFO STATUS signals in controlling tbe operation of the write stage 320 and flow of information, including addresses and data, to FIFO 321.
In addition, the injector FIFO 321 receives a FRAME signal and a FLUSH signal from the injector common control/status portion æ4. The FRAME signal, when asserted, indicates that address and data words for an entire data router message pacl~et 30 have been loaded into the injector FIFO 321. At that point, the message injector port 223 will genera~e a data router message packet 30 using the words. The FLUSH signal, when asserted, indicates that address and data words being loaded into the injector FIFO 321 are to be flushed, and that no data router message packet 30 is to be generated in response to thosc words. The in3ector common control/status portion 224 may assert the FLUSH signaL for e~sample, if it detects an error in colmection with receipt of information for the data router message packet 30 as provided by the processor 200.
Essentially, the injector FIFO 321 includes a plurality of storage locations that are sequentially loaded with INJ FIFO DATA signals defining a word from the write stage 320. In addition the injector .
: .',` ' . . . t ' -. ' `'~ ` : . . .
,. ' : . ' . , ' WO 92/06436 PCrtUS91/07383 2~3~ 39-FIE'O 321 irlcludes a store pointer to the next location to store a word, and a message pointer pointing to the first word for a data router message packet 30. If the injector common control/status portion 224 asserts the FRAME signal, the message pointer is advanced to point to ~he same location as the store pointer. In addition, the irjector FIFO 321 urill assert a MSG AVAlL message available signal.
On the other hand, if the injector common control/status portion 224 asserts the FLUSH signal, the store pointer is moved back to point to the same location as the message pointer, so that the previously-written words for the flushed data router message packet 30 can be overwritten.
In addition, the injector FIF'O 321 includes a read pointer that points to the Dext location to be read, and as long as the read pointer has no~ advanced to point to the same location as the message pointer, the injector ~IFO 321 asserts a FIFO NE not empty sigllal.
The FIFO 321 effectively operates as a buffer between processor 200 and the data router 15.
As indicated above, when the message ejector port æ3 begins transmitting a data router message packet 30, the data router node 21(1,j,k) cormected thereto expects to receive successive flits in the packet 30 in synchrony with-successive ticks of the SYS CLK system closk signaL as long as the node ~ -21(1,j,k) maintains the IN FLY injector fly signal asserted. Thus, it will be appreciated tbat transfers from the FIFO 321, which are directly controUed by a RD FIFO read FIFO signal from an output latch 3æ connected to the output of PIFO 321, are effectively controlled by the IN FLY injector fly signaL
and by the conditions of stages in the message injector port after the FIFO 321. Tbese stages include the output latch 322, an physical address computation stage 323, a physical address computation stage 324, a message adtress computation stage 325 and a transmitter stage 326. The transmitter stage 326, w~ile the data router node 22(1J,k) is asserting the IN FLY injector fly signaL receives the actual IN
FLIT injector flit signals that control transmission of the successive flits of the data router message packet 30.
In any event, the FIE~O 321, in response to the RD FIFO read FIFO signal from the output latch 322, transrnits INJ F~'O OUT (33:0) injector FIFO output signals def~ung a word. In response to an ADV AACS advance absolute address computation stage signal from the absolute address computation stage 323, which indicates that the stage 323 can accept another word from th~ FIFO 321, and the FIFO NE not empty signal from the FIFO 321, and in synchrony with the NODE CLK node clock signaJ, the output latch 322 latches the INJ FIFO OU T injector FIFO output signals from FIFO
321. The output latcn thereupon transmits the latched signals to absolute address computadon stage 323 as LAT FIFO Ol lT (33:0) latched FIFO output signals. The output latch 322 also asserts the RD
FIFO read F~O signal to enable the FIFO 321 to transmit to it INJ FIFO OI~T injector first-in first-out buffer output signals defining the next word stored in the FIFO 321.
The absolute address computation stage 323 receives the l.AT FIFO Olrr (33:0) latched F~FO output signals from the output latch 3æ~ determines whether the signals comprise data, an physical address or a relative address, and if the signals eomprise a relative address determine an physical address value. The absolute address computation stage 323, in response to an ADV PACS
advance pbysical address computation stage signal from the physical address computation stage 324 .,, ~ . : . , , :.. ~
, j, '' ' ' " ' "~ .:, ' '. ' ' ~ ."

WO 92/06436 P~/US91/07383 20~3~
and the NODE CLK node clock signal, transmits ABS ADRS/DATA OU r (33:0) absolute address or data output signals to the physical address computation stage 324. The ABS ADRS/DATA OUT
(33:0) absolute address or data output signals comprise an physical address or a physical address if the LAT FIFO OUT (33:0) latched FIFO output signals received from the output latch 3æ included an address, and data if the LAT FIFO OIJT (33:0) latched FIFO output sigDals comprised data.
If the LAT FIFO OUT (33:0) latched FIFO output signals comprise a relative address, the ;;~
absolute address computation stage 323 uses the chunk table 327 and the chunk size register 311, maintained by the injector common control/status portion æ4 (~Ig. 9A-1), in deterrmining the physical address. As will be described in more detail below in connec~ion with F~g. 9B-2, the absolute address computation stage 327 generates a CHUNK TABLE OFFSET t5:0) signals to identify a location irl the chunk table 3Z7. In response, the chunk table 327 transmits to the physical address computation stage 323 the contents of the identified location as CT ADRS chunk table address signals. The absolute address computation stage 323 uses the CT ADRS chunk table address signals in generating the physical addrcss.- The absolute address computation stage 323 uses the contents of the chunk size register 311 in generating the CHUNK TABLE OFFSET ~5:0) signals, and in using tbe CT ADRS
chunk table address signals in generating the physical address.
The absolute address computation stage 323 couples ABS ADRS/DATA OUT absolute address or data out sigllals to the physical address computation stage 324. The ABS ADRS/DATA
OUT signals may comprise either a physical address, a relative address or data. If the ABS
ADRS/DATA OUT sig~als comprise a physical address or data, the physical address computadon stage passes the signals as PHYS ADRS/DATA Ol.rr physical address or data ou~ siEnals. On Ihe other hand, if the ABS ADRS/DATA OUT signals comprise a relative address, the physical address computation stage 324 generates PHYS ADRS/DATA OUT signals repsesenting a physical address, which corresponds to the address of the destination leaf 21(y) relative to the beginn-D~ of the partition including the source leaf 21(x). The physical address computation stage 324 uses the physical address to determiDe a physical address, which is the address of the destination leaf 21(y) relative to the first leaf 21(0) in the system 10. In this operation, the stage 324 uses tJe contents of the partition base register 305. The physical address computation stage 324, in respo~lse to a Rl~ signal from the transmitter stage 326 and in synchrony with the NODE CLK signal, genesates PHYS ADRS/DATA
OUT (33:0) physical address or data output signals. The PHYS ADRS/DATA OUT (33:0) physical address or data output signals comprise the physical address if the ABS ADRS/DATA OUT (33:0) absolute address or data output signals cornprised an addsess, or data if lhe ABS AI~RS/DATA OVT
(33:0) signals comprised datL
The message address computation stage 325 receives the PHYS ADRS/DATA OUT (33:0)physical address or data output signals and generates an address for inclusion irl the message address pordon 31 of the data router message packet 30. The message address compu~atioD stage 325 uses the contents of tbe physical self address register 312 ~n this operatioD. The message address computation stage 325, also in response to the RUN signal from tbe transmitter stage 326 and in syncbrony ~nth the .:
,, ... . . :
- : . :- . .. ... :: . .

W~ 92/0~436 P(~/US91/07383 N~ si~ generates MSG OIJT (31:0) message out signals. The MSG OUT (31:0) messageout signals comprise the message ad&ress if the PHYS ADRS/DATA oU'r (33:0) physical addJess or data output signals comprise a physical address, and data if the PHYS ADRS/DATA OUT (33:û) signals comprise data. The high-order MSG OUT (32) signal iderltifies whether the remaining MSG
olrr (31:0) signals represent a message address or data.
Finally, the transmitter stage 326 receives the MSG OUT (32:0) message out signals and generates therefrom the IN FLIT inject flit signals, comprising successive four-bit nibbles from the MSG OUT (31:0) signals. The transmitter stage 326 transmits successive nibbles in synchrony with the NODE CLK node clock signal while the IN FLY injector fly signal is asserted. In addition, the transmitter stage 326 uses the IN FLY injector fly signal in generatinB the RUN signal that controls the message address computation stage 32~- and the physical address computation stage 324.
The tetails of various stages of address computation performed by stages 323 through 325 and of generating successive nibbles for ~ransmission as IN FLIT input flit signals wiD be described in connection with Figs. 9B-2 through 9B-3; - Fig. 9B-2 depicts details of the absolute address computation stage 323. With reference to Fig. 9B`", the absolute address computation stage 323 recei~es the LAT
FIFO OUT (33:0) latched FIFO out 5i nals from the output latch 322. The two high-order signals, namely, the LAT FIFO OUT (33:3~) signals, which identify whether the LAT FIFO OUT (31:0) signals comprise an address and, if so, the address mode, are coupled to a decoder 330 and one input of a multiplexer 331. lf the LAT FIFO OUT (33:32) signals identify the relative address mode, the decoder 330 asserts a REL relative si~al.
The absolute address computation stage includes an address/data latch 332 that latches the physical address or data, which are derived from the LAT FIFO OUT (31:0) latched FlFO output signals. The latch 332 latches three groups of signals in unison ~ response to the ADV AACS advance absolute address computation stage and NODE CLK node clock signals. O~e group, the LAT FIFO
OUT (31:203 signals, are coupled directly to the latch 332 from output latch 3æ. In one particular embodiment, if the low-order IAT FIFO OUT (19:0) signals comprise address signals, the LAT F~FO
OUT (31:20) signals include the length and tag information, and thus would be invariant in the computation. In additioD, a second group of sigcals latched by latch 332, namely, the low order LAT
FIFO OUT (1:0) latched ~IFO output signals, comprising the two low-order address signals, are also invariant in the computatîon, and thus are coupled directly to the lateh 332.
The third group of signals latched by latch 332 are not invariant in the computation of the physical address. The LAT FIFO OUT (19:2) signals are coupled to one input tenninal of a multiplexer 333, which is controDed by the REL relative signal &om decoder 330. If the REL relative signal is negated, which will occur if the I~T FlFO OUT (31:0) sigDaLs comprise an physical address or data, the multiplexer couples the L AT FIFO OUT (19:2) signals directly to the latch 332. When the ADV PACS advance physical address computation stage sigllal is as erted, an OR gate 340 is enesgized to assert an ADV EN advance enable signal, which, in turn, energizes one input terminal of an AND
gate 344.

WO 9~/n6436 PCI/US91/07383 2~335~

If the REL relative signal is negated, an inverter 339 energizes an OR gate 343,' which enables the other input terminal of AND gate 344, enabUrlg it to assert the ADV AACS advaDce absolute address computation stage signal. The ADV AACS signal enables latch 332, which, in response to the Dext tick of the NODE CLK node clock signal, latches the LAT FlFO OUT (31:20) signals, the output of multiplexer 333, which corresponds to the LAT FIFO OUT (19:2) signals, and the LAT PIFO OUT
(1:0) siguals. Accordingly, the latch 332 will latch the entire physical address or data provided by the output latch 322. The latch 332 transmits the latched signals to the physical address computation stage as the ABS ADRS/DATA OUT (31:0) absolute address data out signals.
Contemporaneously, an address mode latch 338 will latch the output of multiplexer 331. The multiplexer 331 selectively couples either the l.AT FIFO OUT (33:32) address mode sig~iials or NOP
(1:0) null operation signals to the input terminals of thie address mode latch 338 iQ response to a LD
NOP MOD load null operation mode si~al from an AND gate 345. As will be desc~ibed below ;D
more detail, the NOP (1:0) null operation signals comprise a code that identifies arl error condition in colmection with conversion from a relative address to an physic~ address. In this case, since the REL
relatiYe signal is negated, the AND gate 345 is de-eDergized and the LD NOP MOD load null operation mode signal is negated, enabling the multiplexer 331 to couple the LAT FIFO OUT (33:32) signals to the mode latch 338. The mode latch 338 tra~smits the latched signals to the physical address computadon stage as the ABS ADRS/DATA OUT (33:32) absolute address/data out sigllals.
On the other hand, if the REL relative signal is asserted, which will occur if the LAT FIFO
OUT (31:0) signals comprise a relative address, the multiplexer 333 couples CONV ABS ADRS (19:2) converted absolute address signals Io the latch 332. The CONV ABS ADRS (19:2) converted absolute address sigDals are p}ovided by a conversion circuit comprising a window extrac~or 334, the cbunk table 327, a window ins,erter 335 and a window identifier 336.
The window extractor 334 receives tbe LAT FIFO Olrr (17:2) signals arld WIN SEL window select signals and generates, i~ response thereto, the CHUNK TABLE OFFSET (5:0~ signals to address the churlk table 327. The WIN SEL wi3ldow select signals identify a set of six consecutive sig1~lals in the IAT FIFO OUT (17:2) signals for the window extractor to couple to the chu~k table 327 as the CHUNK TABLl~ OF~7SET (5:0) signals.
As noted abnve, the contents of chunk size re~ster 311 identify the level "i~ in the data router 15 at which each data router node group 20(iJ) defmes a chunk. In one particular emboclime~t, the lowest leYel in the data router 15 at which a chunk may be defined is level four, Yhich has at least two-hundred and fifty six consecutive leaves 21(i) each ideDtified by a data router address. In that embodiment, each chunk is defined by the encoding of the LAT FIFO OUT (7:2) signals. In the same embodiïnent, the highest level at which a churk may be defined is level nine, which has at least 256k (k=1024) consecu~ive leaves. In that case, each chunk is defined by the encoding of ihe LAT FIFO
OUT (17:12) signals.
As the chunk size, as indicated by the value storet in the chunk size register 311, ihlcreases, higher order ones of the LAT FIFO OUT (19:2) signals are used to access the chunlc table 327. Tbe ~ ". ,: .. ~ ~ ., . .,. ,: , ....

2 Q ~ 3 ~ ~ ~ 4~
window identifier 336 decodes the contents of the chunk size register to identify the beginning of six sequential ones of the LAT FIFO OUT (17:2~ signals to address the c~unk table 327. The WIN SEL
window select signals enable the window e~tractor 334 to select the particular ones of the LAT FIFO
OUT (19:2) signals to address the chunlc table 327, and transmit them to the chunk table 327 as the CHUNK TABLE OFFSET (5:0) signals. If, for example, the chunk size register 311 identifies the chunk size as being two-hundred and fifty si~; the w~ndow extractor 334 selects the LAT FIFO OUT
(7:2) signals, whereas if it identifies the chunk size as being 256k (k-1024) the window e~ractor 334 selects the LAT FTFO OUT (17:12) signals. For window sizes between two hundred and fifty six and 256k, incrementing by powers of ~four,~ the window extractor selects six consecutive ones of the l AT
FIFO OUT (9:4), LAT FIFO OUT (11:6), LAT FIFO OUT (13:8), and LAT FIFO OUT (15:10) signals, respectively.
When the physical address computation stage 324 asserts tbe ADV PACS advance physical address computation stage signal, the signal energizes au OR gate 340, which asserts an ADV EN
advance euable sigrlal. The ADV EN advance enable si~al~enables one input terrninal of au AND
gate 341. If the decoder 330 is asserting the REL relative signal, indicating that the LAT FIFO OllT
(31:0) signals represent a relative address, the AND gate 341 is energized to assert a READ CT read chunk table signal, which is coupled to a read enable terrniual of the chunk table 327. In response to the asserted READ CT read chunlc table signal, the ch~ able 3Z7 transmits the contents of the location identified by the CHUNIC TABLE OFFSET (5:0) signals as CT ADRS (5:0) chlmk table address signals to the vindow inserter 335, and asserts a CT VALID ch~,mk table valid sig~
The window inserter 335 essenLially perforrns the reverse operation as the window e~ractor.
The window inserter substitutes the six-bit CT ADRS (5:0) chunk tabie address sigDals into the I~T
F~O OUT (19:2) signals, into six sequential bit locatiolls represented by the I~T FIFO O~T (19:2) signals, and couples the result as CONV AP.S ADRS [19:2) conver~ied absolute address signals to the second input terminal of multiplexer 333. The window inserter substitutes the Cr AI~RS (5:0) chunk table address signals for the same order signals as were extracted by the window extraclior 334~ The multiple~ter 333, under control of the asserted REL relative signal from decoder 330, couples the CONV ABS ADRS (19:2) converted absolute address signals to the input terminal of latch 332.
The CT VALID chunk table valid sig~al, when asserted, indicates that the CI- ADRS (5:0) chunk table address signals represent a valid value. In one embodiment, delays of several tic}s of the NODE CLK signal may be required bet veen the time tbe CHI~N~C TAPLE OFFSEI' sig~lals are coupled to the input termirlals of the chunk table and the READ Cr read cburlk table signal is asserted, and the time the chunk table 3Z7 provides valid CT ADRS (5:0) signals. DuriDg tbat time, signals may nonetheless be latched in the latches 332 and 338. Tbe CT VALID signal, if negated, essentially enables signals representing a aull operation code to be latched in the mode latcb 338, which can be ~used by subsequent stages, if they progress tbereto, to indicate tnat the latched signals sho~ald be ignored. It will ~e appreciated ~hat, since ~e ch~mk table 327 is only ~lsed in connection with relative addresses, if the address is a physical address, or if the LAT FIFO OUT (33:0) signals represent data, the Cl' VALID signal ~ill be ignored and the null operatio~ code will not be used.

WO 92/06436 PCI /US~1 /07383 ~, 2~333~
More specifically, if the chunk table 327 transmits a negated CT VALID chunk table valid signal, an inverter 346 enables an input termina of AND gate 345. If (a) the REL signal is asserted, indicating that the IAT F~FO OUT (33:0) signals represent a relative address, and ~b) an ADV PACS
signal is asserted, which in turn enables an OR gate 340 to assen arl ADV EN advance enable signal, the AND gate 345 is energized to assert the LD NOP MOD load null operation mode signal. The asserted LD NOP MOD signal enables the multiplexer 331 to couple NOP null operation signals to the rnput terminal of mode latch 338, which latches them rnstead of the LAT FIFO OUT (33:32) sigDals.
AND gate 34~ rernains energized until the CT VALID signal i5 asserted.
Contemporarleously, an address rnode latch 338 will latch the output of multiplexer 331. The multiplexer 331 selectively couples either the LAT FIFO o~rr (33:32) address mode signals or NOP
(1:0) null operation signals to the addr~ss mode latch 338 in response to the LD NOP MOD load DUI]
operation mode signal from AND gate 345. Since the LD NOP MOD signal is asserted, the multiplexer 331 couples NOP null operation signals to the mode latch 338. If the absolute address computatioD stage 323 is asr,erting the ADV AACS advance enable signal, the mode latcb 338 latches - - - -the coupled NOP signals at the next tick of the NODE CLK signal. The mode latch 338 transmits the latched signals to the physical address computation stage as the ABS ADRS/DATA OUT (33:32) absolute address/data out signals.
On ~he other ha~d, when the churlk table 327 transmits an asserted CT VALID siBnal, inverter 346 disables AND agate 345 to negate the LD NOP MOD sigDal. In addition, the asserted cr VALID
signal enables one input terminal of an AND gate 342. The assertion of the REL relative signal enables the other input terminal of AND gate 342, which is energized to assert a REL ADV relative advance sigDal. The REL ADV relative advance signal energizes an OR gate 343, which, in turn, enables one input terminal of an AND gate 344. Since tbe asse~ed ADV EN advance enable signal is enabling the other input terJninal of AND gate 344, the AND gate 344 is energized, thereby asserting an ADV AACS advance absolute address eomputatioD stage signal. The asserted ADV AACS signal enables the latch 332, which latches the LAT FIFO OUT (31:20), CONV ABS ADRS (19:2) and l~T
F~FO OUT (1:0) signals at the next dck of the NODE CLK node clocls signal. The latch 332 transmits the latched signals to the physical address computatio~ stage as the ABS ADRS/DATA OUT (31:0) absolute address data out signals.
It will be appreciated tbat, since tbe ADV AACS advance absolute address computation stage signal is asserted, the output latch 322 uill latch the next word from the injector FIFO 321. Since the next word will be the next word of the message, not an address word, the LAT FIFO OUT (33:32) signals uill Dot indicate that the LAT FIFO Ol~T (31:0) sigllals represent a relative address. In that case the decoder 330 will negate the ~EL relative signal, enabling the multiplexer 333 to couple the LAT FIFO OIJ~ (19:2) signals to the input terminal of address/data latch 332.
The negated REL relative signal also enables an inverter 339 to energ~ze OR gate 343, which, in turn, eDables one input terminal of AND gate 344. If the ABS ADR NOP absolute address hUll operation signal from comparator 347 is not asserted, when the physical address computation stage 324 . . . .
.

, :

.. ...

WO 92/06436 PCr/US91/07383 ~ . , .
~S-n~a~3t~t~e~DV PACS advance physical address computation state signal, OR gate 340 is energized. In response the AND gate 344 asserts the ADV AACS advance absolute address computation stage signal, enabling the address/data latch 332 to latch the LAT FIFO OUT ~31:0) signals at the next tick of the NODE CLK signal.
In addition, since the REL relative signal is negated, AND gate 345 maintains the LD NOP
MOD load null operation signal negated, thereby enabling the multiplexer 331 to couple the LAT
FTFO OUT (33:32) signals to the input terminal of mode latch 338. The mode latch 338 latches these signals contemporaneously with the latching by the address/data latch 332 of the I~S FlFO OI~T
(31:0) signals. It will be appreciated that these operations will occur iteratively with eacb tick of the NODE CLX signal, while the ADV PACS advance physical address computation stage signal is set, enabling the data words to be iteratively latched in the latches 332 and 338 and thereby transmitted through the absolute address computation stage.
The absolute address computation stage 323 includes a comparator 347 that compares the ABS ADRS/~ATA OU r (33:32) absolute address/data out signals to SND ADRS NOP send address null operation signals. If the multiplexer 331 coupled the NOP null operation signals to be latched by the mode latch 338, the comparator will assert an ABS ADR NOP absolute address null operatioD
signal, which enables ;he OR gate 340 to assert the ADV EN advance enable signal. SiDce the REL
relative signal is negated, inverter 339, through OR gate 343 enables one input terminal of AND gate 344, which is energized to assert the ADV AACS advance absolute address computation stage signal.
As described above, the ADV AACS signal enables the latches 332 and 338 to latch the next word defined by the LAT FIFO OUT (33:0) signals provited by the ou~put lasch 3æ at the next tick of the NODE CLK signal, and enables the output latch 322 to supply the next word in the FIFO 321. This occurs iteratively vith successive ticks of the NODE CLK node clock signal until the next address word is latched in latches 332 arld 338, in which case the ABS ADR NOP absolute address null operation signal is negated. At that point, control over the sequencing through the absolute address computatior stage 323 returns to the ADV PACS advaDce physical address computation stage signal from the physical address computation stage 324.
As noted above, the absolute address computation stage 323 transmits ABS ADRS/DATA
OUT (33:0) absolute address/data output signals, in the form of successive words, to the physical address computation stage 324. In this operation, if the ABS ADRS/DATA OI~T (31:0) signals represent an address derived from sigr,als representillg a relative address, the physical address is obtained by adding the address value to the value in the partition base register 305. If the ABS
ADRS/DATA OUT (31:0) signals represent an address derived from signals representing an physical address, the address from the absolute address compu~ation stage is the physical address. In either case, ~he address mode, as indicated by the high order ABS ADRS/DATA our (33:32) signals, control the operations of the pllysical address computation stage. Fig. 9B-3 depicts a detailed block diagra~in of the physical address computation stage 324.
With reference to Fig. 9B-3, the physical address computation stage 324 rece;ves the ABS
ADRS/DATA OUT (33:0) signals aDd couples the two high order ABS ADRS/DATA Ol~ (33:32) ., .: , ,.. ,, .., ";, , . ,,. , ~ , ;.; , ~ , . .':; ' .;:: : . .:: , :, , , . . ~ :: . . . :~

:

2~933~ ~
signals to the input terminal of a decoder 350 aDd one iDpUt terminal of a mode latch 351. At this point, if the ABS ADRS/DATA oUr (31:0) signals represent an address, the two high-order ABS
ADRS/DATA OIJT (33:32) signals continue to indicate the address mode of the original address. If the ABS ADRS/DATA OI~T (33:32) signals identify the relative address mode, decoder 350 generates a REL ADRS MODE relative address mode signal.
Of the remaining ABS ADRS/DATA oU'r (31:0) absolute address/data out signals, the high-order ABS ADRS/DATA OUT (31:20) sigllals are coupled directly to the input terminal of a physical address latch 352. As noted above, if the ABS ADRS/DATA OUT (31:0) sig~als represeDt an address, the high-order ABS ADRS/DATA OUT (31:0) signals carry the rnessage length and tag signals for inclusion in fields 34 and 35 of a data router message pacl~et 30, and thus are invariant under conversion. The remainiDg ABS ADRS/DATA OUT (19:0) sigDals, comprising the address are coupled to one input terminal of a multiplexer 353 and oDe input tenninal of an adder 354. The second input termir al of adder 354 is provided by the partition base register 305.
- - The multiplexer 353 is controlled by the REL ADRS MODE relative address mode signal from the decoder 350. If the REL ADRS MODE relative address mode signal is negated, indicatirlg that the ABS ADRS/DATA OUT (19:0) signals represellt either an physical address or data, the multiplexer 353 couples them directly to the physical address latch 352. If the transmitter stage 326 is asserting the RUN signal, an OR gate 355 asserts the ADV PACS advance physical address computation stage signaL enabling the physical addrcss latch 352 and the mode latch 352 to latch the signals at their respective input terminals in synchrony with the next tick of the NGDE CLK signal. In this case, the physical address latch 3æ will latch the ABS ADRS/DATA OVT (31:20) signals directly from the absolute address computation stage, and the ABS ADRS/DATA OUT (19:0) s~als coupled thereto by the multiplexer 353.
On the other hand, if the decoder 350 is asserting the REL ADRS MODE relatiYe address mode signal, indicating that the ABS ADRS/DATA OUT (31:0) signals are derived from a relative address, the multiplexer 353 couples sigrlals output from the adder 354 to the physical address latch 352. These signals represent a value comprising the sum of the address defined by the current ABS
ADRS/DATA OUT (19:0) absolute address/data output sigDals and ~he contents of the partition base register. If the OR gate 355 is asserting ADV PACS advance physic~ address computation stage signal, the physical address latch 352 latches these signals, along with the ABS ADRS/DATA OUT
(31:0) signals, in synchrony vith the Dext tick of the NODE CLK signal. The mode latch 351 provides the high order PHYS ADRS/DATA OUT (33:32) physical address/data output signals, and the physical address latch 352 provides the low order PHYS ADRS/DATA OUT (31:0) signalsi, both of which are coupled to the message address computation stage 325.
The high-order PHYS ADRS/DATA OUT (33:32) physical address/data output signals are also coupled to an irlput terminal of a comparator 356. The comparator 356 also receives the SND
ADRS NOP send address null operatiorl signals. If the PHYS ADRS/DATA OUT (33:32) si~nals correspond to the SND ADRS NOP send address null operadon si~als, the comparator 351 geneJates ,, ~ , .. :, '~.
- :, : : ` . ' `~ . ' . ~ : . . ' :
:: ': : ' ' ' ' : ,',:; ', ' ,' ' ~' '' ' '' :: '. : . ' ,. ' '., ` "": " - ; ,, ' ' ''~ ' i ' .

2 ~ 7-a PHYS ADRS NOP physical address null operation signal. it will be appreciated that the comparator 356 asserts the PHYS ADRS NOP signal under the same circumstance as thc comparator 347 asserts the ABS ADRS NOP absolute address null opera~ion signal. That is, the comparator 3S6 asserts the PHYS ADRS NOP signal in response to the coincidence of (i) the ABS ADRS/DATA OUT (19:0) signals representing a relative address a~d (ii) the chunk table location 327 used by the absolute address computation stage 323 (Fig. 9B-2) was not valid.
The asserted PHYS ADRS NOP physical address null operation signal also enables the physical address computation stage to perform operations similar to those performed by the absolute address computation stage in response to the asserted ABS ADRS NOP signaL that is, sequencing through words of a message provided thereto by the absolute address computation stage until it receives a valid word. The asserted PHYS ADRS NOP physical address null operation signal enables the OR gate 355 to maintain the ADV PACS advance physical address COmpUtatiOQ stage sigDal asserted, enabling the latches 351 and 352 to iteratively latch the signals provided thereto in response to successive ticks of ,he NODE CLK signal, until the comparator 356 determines that the contents of the mode latch 3S1 are associated vith a valid address word.
The PHYS ADRS/DATA OUT (33:0) physical address data out signals are coupled to the message address computation stage 32S. If the high-order PHYS ADRS/DATA OUT (33:32) signals indicate that the remaining PHYS ADRS/DATA OUT (31:t') signals correspond to an address, the message address computation stage 32S (i) detennines the contents of the header field 4C, (ii) arranges the first thirty-two bit message word, which includes the header, the dGwn path identification portio~
41, the message length field 34 and the message tag field 35, and (iii) latches the message word for transmission to the transmitter stage. If the PHYS ADRS/DATA OUT (33:32) signals indicate that the remaining PHYS ADRS/DATA OUT (31:0) signals do not comprise an address, they comprise data that are latched for transmission to the transmitter stage.
A detailed block diagram of the message address computation stage is depicted in Fig. 9B-4.
With reference to 9B-4, the high-order PHYS ADRS/DATA oU'r (33:32) physical address/data out signals are coupled to two decoders 360 and 316. If the PHYS ADRS/DATA OUT (33:32) sig~lials indicate that the assoaated low order PHYS ADRS/DATA OUT (31:0) signals are derived frorn an address word including a physical address, decoder 360 asserts an IS PHY ADRS signal. On the other hand, if the PHYS ADRS/DATA OUT (33:32) signals indicate that the associated low order PHYS
ADRS/DATA OUT (31:0) signals are derived from an address word including a relative address, decoder 360 asserts an IS REL ADRS sigQal. Both the IS PHYS ADRS signal and IS REL ADRS
sigl~al are coupled to input terrninals of an OR gate 362, which generates a~ asserted IS ADRS is address signal when either the IS PHYS ADRS signal or the IS REL ADRS signal is asserted. Thus, the IS ADRS signal will be asserled if the PHYS ADRS/DATA OUT (31:0) signals represent an address word.
Selected ones of the low-order PHYS ADRS/DATA OUT (31:0) physical address/data out signals are coupled to respective input terminals of three multiplexers 363, 364, and 365. In parficular, ' ' , ' . ; ' :.: - , ", ''', ';''' ,.,., "
. ' ' ' , ' ' ' ' ', '; " '' . . ., ' ' ' . ' ' . . ' ' ' ' . , ' ' ' ' ' . ' ' ' ' . ' ' '' ''", ', '', . ", ', : ~ ' ~' ': ',' " ',. ...

WO 92/06436 PC~/U~1J~7383 2~933~3 the PHYS AORS/DATA OIJT (31:~) signals are coupled to one data input terminal of multiplexer 363, the PHYS ADRStDATA OUT (7:4) signals are coupled to tbe corresponding data input terminal of multiplexer 364, and the PHYS ADRS/DATA Ol)T (3:0) signals are coupled to the correspon&g data input lerminal of multiplexer 365. If the IS ADRS sig~al is negated, which occurs if the PHYS
ADRS/DATA OUT (31:0) signals represent data, the multiplexers 363, 364 and 365 couple the sunals at these data input terminals through their respective output terminals to the input terrninai of a staging register 366. If the transmitter stage 326 is asscr~ the RUN signaL the staging register latches these signals in response to the next tick of the NODE CLK signal. The latched signals are coupled to the transmitter stage 326 as the MSG OUT (31:0) signals.
The second data input terminals of multiplexers 364 and 365 are coupled to PHYS
ADRS/DATA OUT (27:24) signals and PHYS ADRS/DATA O~T (23:20) signals respectively.
These signals represent the message length and message tag, which are inserted into fields 34 and 35, respectively, of the data router message packet 30 (Fig. 3). The second data input terminal of multiplexer 363 is coupled to a message address portion genera~ing circuit 367 which, in response to the PHYS ADRS/DATA oU'r signals (19:0), generates MSG ADRS (23:0) signals, which represent the header 40 and down path identification portion 41 of the message address portion 31 of the data router message packet 30 to be transmitted. When the OR gate 362 asserts ~he IS ADRS signaL multiplexers 363, 364 and 365 couple the signals at their second data input terminais to the ir~put terminals of staging register 366. If the transmitter stage 326 is asserting the RUN signal, the staging register latches these signals in response to the next tick of the NOD~ CLK sigQal. In particular, the staging register 366 latches the message address portion in its bits thirty-one through eight, the length in bits seven through four, and the tag in bits three through æro. The latched signals are coupled to tke transmitter stage 326 as the MSG OUT (31:0) signals.
In addition, if the transmitter stage 326 is asserting the RUN signaL a register 368 latches the IS ADRS is address signal in response to the next tick of the NODF CLK signal. The latched signal is coupled to the trans~itter stage 326 as the MSG OUT (32:0) signal.
The general blocl~ diagram of the message address portion generating circuit 367 is depicted in F~g. 9B-4, and logic diagrams of several circuits therein are depicted in Figs. 9B4 through 9B-7. The message address portion generating circuit 367 iDcludes an exclusive-OR (~XOR~) gate 370, which receives the PHYS ADRS/DATA OUT (19:0) sig~lals aDd the contents of physical self address register 312, and generates REL ADRS (19:0) relatille address signals which comprises the bit-wise exclusive-OR of the input signals. That is, for e~nple, the XOR gate 370 generates the REL ADRS (19) signal as the exclusive-OR of the PHYS ADRS/DATA OUT (19) si~al and bit r~ neteen from the physical self adtress register 312. The REL ADRS (19:0) relative address sigDals have a binary-encoded value that represents the displacement from the source leaf 21(x) to the destination leaf 21(y).
The REL ADRS (19:0) relative address sigDals are coupled to input terminals of a header nibble calculation logic circuit 371 and a header nibble select logic circuit 372. The header nibble calculation logic 371 (Fig. 9B S) uses the REL ADRS (19:0) signals to deterlnine the level in data WO 92/06436 PCr/US91/073g3 20933`3~

router 15 to which the data ronter message packet 30 must be sent to reach its destination. As described above, the height is the level ~;" of the first data router node group 20(iJ) which the data router node packet 30 reaches in the npward path through the data router 15 that is the root of a su~
tree that includes both the source leaf 21(x) and destination leaf 21(y). The header nibble calculation logic 371 generates a four-bit nibble, comprismg HDR (3:0) header signals, that is binary-encoded to identify this level.
The header nibble select logic circuit 372 (Fig. 9B-o) determirles the four-bit nibble, iD the MSG ADRS (23:0) message address signals, into which the HDR (3:0) header signals will be inserted.
This will depend UpOD the number of flits in the down palh identification portion 41 of the message, which, iD turn, depends on the level ~i~ to which the data router node groups 20(iJ) transmit the message pacl~et 30 in the data router 15. Essentially, the beader nibble select logic circuit 372 identifies the first nibble, beginning with the most-signiflcant nibble, in the PHYS ADRS/DATA OUT (19:0) signals from the physical address computa~ion stage 324 (Fig. 9B-3) which do not all represent the value zero. The header is inserted ioto the message address signals just above the identified r~ibble.
The HDR (3:0) header signals aDd the signals from the header nibble select logic circuit 372 are coupled to a message address assembly logic circuit 373 (Fig. 9B-7). This circuit also receives the PHYS ADRS/DATA OUT (19:0) physical address/data out signals from the physical address computation stage 324 (Fig. 9B-3). The message address assembly logic circu-t 373 generates the MSG
ADRS (23:0) message address signals by inserting the HDR (3:0) signals into the four-bit nibble identified by the signals from the header nibble select logic circuit 372. It will be appreaated that the message address assembly logic circuit 373 esseDtially transmits the nibbles of the PHYS
ADRS/DATA OUT (19:0) signals which do not represent the value zero as low-order MSG ADRS
message address signals, and the HDR (3:0) headerlsignals in the next higher-order nibble.
Fy~ 9B-5 depicts the logic diagram of tbe header aibble calculation logic circuit 371. With reference to Flg. 9B-5, the circuit 3il includes a height identifier circuit 374 that generates HGT (10:1) height identifier signals in response to tbe REL ADRS (19 2) relative address signals, and a decoder 37~. The height identifier circuit 374 includes nine OR gates 376(2) ~hrough 376(10) Igenerally identified by reference nulneral 376(i)], each of which receives two of the REL ADRS (19:2) signals and generates a HGT (i) signal in resporlss- . If a HGT (i) signal is asserted, the message packet 30 will go to at least that level ~P to reach a data router node group 20(iJ) that is the root of the smallest su~
tree including both the source leaf 21(x) and the destination leaf 21(y). If a HGT (i') signal having a higher index ~ is asserted, the message packet 30 will have to go at least to that level ~i'."
Thus, OR gate 376(10) receives the REL ADRS (19) and REL ADRS (18) signals and generates the HGT (10) height signal in respoDse thereto. If either the REL ADRS (19) or REL
ADRS (18) signals are asserted, the root of the srnallest sub-tree of data router 15 including both the source leaf 21(x) and destination leaf 21(y) is at level 10. On the other hand, if5 for example, the REL
ADRS (19:14) signals are negated, and the REL ADRS (13) signal is asserted, the displacement between the source leaf 21~x) and the desti~ation leaf 21(y) is not as great, and so the root of the sub-, .. .. . . . . . . . . . .. .. . . .

WO 92/06436 PCr/US91/07383 20933~
so.tree including both leaves will be at a lower level. In that case, the root of the smallest suhtree including ooth the source and destination leaves 21(x) aDd 21(y) is at level 7. OR gate 376(7), which receives the REL ADRS (13) signaL asserts the HGT (7) height signal in response to the asserted RE
ADRS(13) signal.
Since the message packet 30 will always be transmitted to at least level 1 in the data router 15, a HGT (1) height signal is maintained at an asserted level.
It will be appreciated that, depending on the particular encoding of the REL ADRS (19:2) relative address signals, several of the HGT (10:2) signals may be asserted. In any case, the HGT (i) signal that identifies the level ~i' of the smallest sub-tree including both the source and destination leaves 21(x) and 21(y) will always 'oe asserted. The decoder 375 generates HDR (3:0) header signals that are binary-encoded with the largest index value ~iu of the asserted HGT (i) signals. Thus, for example, if the HGT (10) height signal is asserted, index (10) is represented by binary encoding (1010).
To accommodate that encoding, the HGT (10) height signal enables an OR gate 377 to assert the HDR (3) signaL which provides the high-order ~1~ in the binary encoding (1010). The high-order ~1~ is also asserted in response to the assertions of the HGT (9) signal and the HGT (8) signal, and so the OR gate 377 is also connected to OR gatcs 376(9) and 376(8).
Since the highest level in the embodiment of data router 15 disclosed herein is 10, if the high-order HDR (3) header sigllal is asserted, the second-order HDR (2) si~al is never asserted; otherwise the encoding of the HDR (3:0) signals could represent binary-encoded values greater than ten.
Aocordingly, if the HDR (3) header signal is asserted, an inverter 380 disables an AND gate 381 to maintain the HDR (2) signal at a negated level. Thus, the HDR (2) signal would be negated even if a HGT (i) signal is asserted that would otherwise result in assertion of the HDR (2) signal.
Continuing with the example in which the HCiT (10) signal is asserted, the asserted HGT (10) signal also enables an OR gate 382 to assert the HDR (1) signal, providing the third-order ~1~ in the binary encoding (1010). In addition, an inverter 383 disables an AND gate 384 to negate the HDR (0) signal and providing the low-order ~0.~ The inverter 383 maintains the HDR (0) signal at a negated state even if other HGT (i) signals are asserted that would otherwise reslalt iD assertion of the HDR (1) signal.
The decoder 375 includes a number of other gates that selectively enable and disable gates 377, 381, 382 and 384 to enable generadon of the appropriate HDR (3:0) header sigllals in response to the HGT (i) height signals. Since the operation of the decoder will be apparent to those skilled in the art, it will not be ~urther described herein.
The header Dibble select logic circuit 372 (Fig. 9B~) is depicted in Flg. 9B-6. With re~erence to Fig. 9B-6, the circuit 372 comprises a height-pair identifier circuit 385 and a decoder 386. The height-pair ident;fier circui~ iDcludes a set of OR gates 387(5) through 387(2) Igenerally identit;ed by reference numeral 387(i)~, each of which receives four of the REL ADRS (19:4) relative address signals. With reference to Fig. 9B-5 as well, it will be appreciated that each OR gate 387(i) corresponds to two OR gates 376(i); that is, each OR gate 387(i) is energized by the REL ADRS

.. . . , - .. -: ..: ~, ., , : ., :: : . . . . . .
- ~. .. -: :., ~ . . .......................... . .
- ~ - . .: : ;; :, :; :

20933~S

signals when either of two OR gates 376(i) is energized thereby. Thus, for example, if either of OR
gate 376(3) or 376(4) is energized to assert the HGT(4) OR HGT(3) signals, indicating that the message packet 30 will be transmitted to at least either of those levels in the data router 15, OR gate 387~2) will also be energized. Similarly, OR gate 387(3) is energi~ed when either the HGT (5) or HGT
(6) signal is asserted, OR gate 387(4) is energized when the HGT (7) or HGT (8) signal is asserted, and OR gate 387(5) is energized when the HGT (9) or HGT (10) signal is asserted.If an OR gate 387(i) is energized, the header 40 comprises flit nin of the message address portion 31 of the message pacl~et 30, counting from the first flit of the message address portion 31 which is identified as flit zero. If one of the REL ADRS (19:16) relative address signals are asserted, there are nine or ten down-path identifiers 42 requiring five flits in the down path identifier portion 41.
Thus, if any of the REL ADRS (19:16) signals is asserted, OR gate 387(5) asserts a HDR FLIT S
signal, indicating that the header 40 comprises flit n5n of the message address portion 31. OR gates 387(4) through 387(2) operate similarly in response to the other REL ADRS (15:4) signals.
- The decoder 386 determines the energized OR gate 387(i) vith the highest index value (i) and in response generates signals for controlling the message address assembly logic circuit 373 (Fig. 9B-7).
If the OR gate 387(5) is eDergized, the decoder 386 asserts a HDR Fl.IT 5 signal, indicating that the header 40 is in flit "5" of the message address portion 31. The decoder 386 includes three ANI) gates 390(2) through 390(4), each of which has one input termiDal connected to the ouîput of a respective OR gate 387(2) through 387(4). If the OR gate 387(4) is energized, one input terrninal of AND gate 390(4) is enabled. lf the OR gate 387(5) is not eDergized, an inverter 391 asserts a HDR NOT FLlT S
signals, energizing the AND gate to assert a HDR FLIT 4 signal indicating that the header 40 comprises flit "4" of the message address portion, and that there are four flits in the dowo path identificationportion.
Similarly, if OR gate 387(3) is ener~ized, one input tenninal of AND gate 390(3) is enabled. If both the OR gates 387(4) and 387(5) are disable~L a NOR gate 392 energizes the second input tenninal of AND gate 390(3), enabling it to assert the HDR FLIT 3 signal, iDdicating that the header 40 comprises flit ~3" of the message address portion 31 and that there are three flits in the down path identification portion 41. An inverter 393 inverts tbe output of NOR gate 392 to assert an HDR Fl,lT

4/5 signaL indicating that the header is in either flit "4" or "Sn. AND gate 390(2) operates sirnilarly in response to the energization of the OR gate 387(2) and a NOR gate 394. If the OR gate 387(2) is energized and OR gates 387(3) through 38~(5) disabled, AND gate 390~2) asserts a HDR FLlT 2 signaL indicating that the header 40 comprises nit "2" of the message address portion 31. If the NOR
gate 394 is not energized, an inverter 395 asserts a HDR FLIT 3/4/5 sigral indicating that the header comprises either flit ~3,^ ''4," or ~5."
Finally, if ail of the OR gates 387(2) through 387(5) are disabled, a set of inverters 396 enables an AND gate 397 to assert a HDR FLIT 1 signaL indicating that tbe header 40 is in flit ~1" of the message address portion 31, and that the down path identification portiotl 41 of the message packet includes one flit, namely, flit ~o." If the AND gate 397 is disabled, such that the HDR PLIT 1 signal is , ~, ,, , . , .. ... . : :-WO 92/06436 2 o ~ 3 3P~JUS91/07383 negated, an inverter 398 asserts a HDR FLIT 2/3/~/5 signal indicating that the header 40 is io either flit 2, 3,~ n4 n or nS ~
The message address assembly logic 373 (Fig. 9B-7) uses the signals from the header nibble select logic 372 to control the insertion of the HDR (3:0) header signals in the MSG ADRS (23:0) message address signals. With reference to Fig. 9~3-7, the message address assembly logic 373 receives the PHYS ADRS/DATA OUT (19:0) physical address/data out signals and directs four-bit nibbles to gate circuits 400(1) through 400(4) lgenerally identified by reference nurneral 401(i)] which control the selection of the nibble of the MSG ADRS message address signals onto which the HDR (3:0) header signals are transmitted. Since the message address portion 31 of a data router message packet 30 always includes at leas~ one flit in the down path identification portion 41, which is represented by the low-order nibble MSG ADRS (3:0) signals, the PHYS ADRS/DATA OUT (3:0) signals are transrnitted directly as MSG ADRS (3:0) message address signals.
Each gate circuit 400(1) through 400(4) includes a multiplexer 401(1) through 4~1(4) Igenerally identified by reference nurneral 401(i)] and an AND gate 402(1) ~hrough 402(4) [generally identified by reference numeral 402(i)]. In each gate circl~it 400(i), the multiplexer 401(i) couples either the HDR (3:0) signals or a nibble of the PHYS ADRS/DATA OUT coupled to its associated AND gate 402(i) over one pa~ticular nibble of the MSG ADRS message address signals, in response to the HDR ~ i) signal from the header r~ibble select logic 372. The respective AND gate 402(i) is controlled by the HDR FLIT i~ 1/.../5 signal from the header r~ibble select logi Thus, if, for example, the header rlibble select logic determines that the header comprises fGt "1" of the message address portion 372, the HDR FLIT 2/3/4/5 sigllal i~ negated, disabling the AND
gate 402(1). The HDR FLll` 1 signal is asserted, enabling the multiplexer 401(1) to transmit the HDR
(3:0) header signals as the MSG ADRS (7:4) message address signals. As Doted above, the mess~ge address assembiy logic transmits the PHYS ADRS/DATA OUT (3:0) physical address/data out signals as the MSG ADRS (3:0) message address signals.
The other HDR FLIT (i) signals are Degated, enabling the respective multiplexers 401(i) (i=2, 3 and 4) to couple the values at their "zero~ data input terminals as the respective MSG ADRS signals.
In that case, since the other HDR Fl,IT i+ 1/.../~ signals are also negated, the other AND gates 402(i) are disabled, so that the input terminals to multiplexers at tbe nzeron data input ter~niDals are all zero, thus ensuring that ~he tespective MSG ADRS message address signals transmitted thereby all represent the value zero. In addition, an AND gate 403, wbich receives the HDR (3:0) signals, is disabled by the negated HDR FLIT S signal, thereby ensunng that the MSG ADRS (Z3:20) message address signal~ all represent the value zero.
If, as another example, the header nibble select logic 372 is asser~ing the HDR FLIT (3) signal, indicating that the header 40 comprises flit ~3~ of the message address por~ion 31, tbe HDR
FLIT 2/3/4/5 and HDR FLlT 3/4/5 signals are asserted, and the HDR FLIT 4, HDR FLIT S and HDR Fl,IT 4/5 siguals are negated. In that case, the multiplexer 401(3) is enabled to transmit the HDR (3:0) header signals as the MSG ADRS (15:12) message address sigDals. Since the HDR FLIT

,.,. , . . , "

: . : , . . , , " . ,:;
. . .~, . .: , : ::: :" , :; . :, : ::::
,- . : ., ; ,, . : . .: . :,, :.. : . . .. . .

WO 92/06436 PCr/US91/07383 20Y3~ 5~
4, HDR FLIT S and HDR FLIT 4/5 signals are negated, AND gates 402(3) through 4(~2(5) are all &abled. As a result, the MSG AD~S (23:20) ~essage address signals are all negated, representing the value zero. Similarly, the negated HDR FI~T 4 signal enables the multiplexer 402(4) to couple the signals from AND gate 402(4) as the MSG ADRS (19:16) signal. Since the A~ gate 402(4) is disabled, the MSG ADRS (19:16) represeDt the value zero.
Continuing w,ith the example, since the HDR FLIT 1 and HDR PLIT 2 signals are also slegated, multiplexers 401(1) and 401(2) transmit the signals from their respective AND gates 402(1) and 402(V as the respective MSG ADRS signals. Since the AND gates are enab]ed by the asserted HDR FLIT 2/3/4/5 and HDR FLlT 3/4/5 signals, the MSG ADRS (7:4) and MSG ADRS (11:8) signals, frorn multiplexers 401(1) and 401(2) correspond to the PHYS ADRS/DATA OUT (7:4) and (11:8) signals, respectively. The message address assembly log;c operates similarly if the header is to be in other flits in the message address portion 31.
Returning to Fig. 9B-4, as noted above, the staging register 366 latches the signals representing the message address, length, tag and data in-response to the RUN signal from the transmitter stage 326 (Fig. 9B-1) and the NODE CLK signal, and contemporaneously the latch 368 latches the IS ADRS
signal. The latches trausmit the latched signals as the MSG OUT (32:0) message out signals, which are coupled to the transmitter stage 326. Fig. 9B-8 depicts a detailed logic diagram of the transrnitter stage 326.
With reference to Fig. 9B-8, the transmitter stage includes a buffer register 410 and a nibble counter 411. The buffer register latches the MSG OUT (32:0) rnessage out signals and transmits the high order bit as an ADRS WD address word signal, and the semaining bits as XMIT OUT (31:0) transmitter out signals. If the ADRS Wr) address word signal is asserted, the XMIT OUT (31:0) signals correspond to a message address.
The nibble counter 411 transmits NC (7:0) nibble count signals each of which is associated with one of the successive four-bit nibbles of the XMIT OUT (31:0) signals. When enabled by an asserted NC DEC EN nibble counter decrément enable signal from an AND gate 417, the nibble counter 411 decrements in synchrony with the NODE CL~C node clock signal, iteratively asserting the NC 7 through NC 0 nibble count signals. The AND gate 417 is energized while the FLY IN signal is asserted, if a normally-negated CHEC:K OUT signal is negated.
When the NC 0 nibble count signal is asserted, it enables the buffer register 410 to lalch the MSG OUT (32:0) message out signals from the message address computation stage 325 (Fig. 9B~) at the next tick of the NODE CLK signal. The NC 7 nibble count signal is transmitted to the message address computation stage 325 and the physical address compu~ation stage 324. When the NC 7 nibble count signal is asserted, which occurs after the current contents of the staging register 366 and latch 368 i~ the message address computation stage 325 have been latched in the buffer register 410, the NC 7 nibble count signal enables those stages to operate, to generate new MSG OUT (32:0) message out signals.
The NC ~7:0) nibble count signals also control two circuits io the transmitter stags 326. A nit selection circuit 413, in response to the NC (7:0) nibble count signals, selects successively lower-order WO 92/06436 PCr/US91/07383 2~)933~
-s4 four-bit nibbles of the XMrr OUT (31:0) transmitter out signals for transmission as tbe four-bit F~IT
IN signals. In addition, a start of message detect circuit 414 scans the nibbles of the XMlT OUT (31:0) signals, and, if the XMIT OUT (31:0) signals represent an address word, as indicated by an asserted ADRS WD address word signal, asserts an SOM start of message signal.
The transrnitter control circuit 412 uses the SOM 5tart of message signal to identify the first flit of a data router message packet 30. When the SOM signal is asserted, if the FLY IN signal is asserted it asserts an EN OUT enable out signal, which enables the nibble selection circuit 413 to begin transmitting nibbles in synchrony with ticks of the NODE CLK sigl~aL A norrnally-negated CHECK
OUT signal from the transmitter control 412 enables a multiplexer 41S to couple the output from the nibble selection circuit 413 as FLIT IN signals.
In addition, the transmitter control circuit 412 asserts a HEADER signal that resets ar~d enables a check flit generator 416. The check flit generator 416 receives LAT FL,IT t3:0) signals from the nibble selection circuit 413, that, at each tick of the NODE CLK signaL are representative of the flit then being transmitted, and generates error detection check bits i~ response thereto.
The transmitter control circuit 412 also receives the LAT FLIT (3:0) latched flit signals representative of the header 40 of the message packet 30 and deterrnines therefrom the number of flits in the message address portiou 31. After that mlmber of flits have been transmitted, the next flit corresponds to the data router message packet's message leDgth field 34. The traDsmitter control circuit 412 uses the ~ralue identified by that flit in deterrDining when the message data pcrtion 32 has been transm;tted. When the transmitter control arcnit 412 determines that the message data portion has been transmitted, it asserts the CHECK OUT signal, which enables the multiplexer 415 to couple CHECK Fl,IT (3:0) signals frorn the check nit generator as the FLIT IN signals. The asserted CHECK OUT signal also disables the nibble counter 411, inhibiting it from decrementing.
The start of message deteclion circuit 414 includes a pluralily of OR gates 420(3) through 420(7) Igenerally identified by reference numeral 420(i)]. Each OR gate 420(i) recei~res the signals from the "i-th" nibble of the XMIT 011T (31:0) transmitter output signals. If all of the sigllaLs of the nibble coupled to OR gate 420(i) are negated, the OR gate pruduces a ~e8ated output signal. On the other hand, if one of the signals of the nibble 5 asserted, the OR gate produces an asserted output signal.
The output of each OR gate 420(i) is connected to the input tenninal of one of AN~ gates 421~3) through 421(7). The other input terminal of each AND gate 423 (i) is connected to receive a corresponding one of the NC i nibble counter signals. Thus, the nibble counter 411 decrements, successively asserting the NC ~ through NC 3 nibble count signals, enabling the successive AND gates 421(7) through 421(3). If an OR gate 420(i) is energized by the assertion of at least one of the signaLs in the nibble coupled tnere~o, ~he associated AND gate m(i) is energized to assert a corresponding one of the SOM ~i' star~ of message (nibble "i') signals (~i~ is an integer from se~en to three, corresponding to the nibble). When the frst SOM "i~ signal LS assened7 an OR gate 4æ is energ~zed to assert an SOM DET start of message detect sig~al.

, :. , . : :: :. : : .
-: : - ., . . . , ,-: . :. . : ........... . ;, : .. :
:: . ., .: .::: . ~ . :: :. . . : :
::, . : . ~ :,-: ,.. , ~ : :: . .. .

WO 92/06436 P~tUS91/07383 .

2~9335~ -55-lt will be appreciated that the SOM DET start of message detect signal is asserted whenever a nibble of the XMIT OUT (31:12) transmitter out sigllals includes an asserted signaL in synchroDy with the correspon~ing NC ~i~ nibble count signal. The SOM DET start of mess ge detect signal is coupled to an input terminal of an AND gate 423. If the AND gate is enabled by asserted ADRS WD address word and asserted FLY IN signals, which occurs if the buffer register 410 contams the &st word of a new data router message packet 30 and the transmitter stage 326 is enab]ed to transmi~, the AND gate 42~ asserts the SOM start of message signal.
It will be appreciated that the start of message detection circuit 414 of the embodiment disclosed herein does not need elements corresponding to OR gates 420(i) or AND gates 421(i) for ~i~
cqual to zero, one or two. The associated nibbles, comprising XMIT OUT (11:0) transmitter out signals, if they are in a word which contains the f~st flit of a new data router message packet, would contain the message length and message tag fields 34 and 35 and first flit of the down path identifica2ion portion 41. Thus, the nibble containing the header, which must contain at least one asserted signaL must be iD the XMIT OUT (31:1V signals. -~
The nibble selection circuit 413 receives the rlibbles of the XMIT OUT (31:0) signals and selectively transmits them under control of the EN OUT enable out si~al from the transmitter control circuit 412. The four-bit Dibbles comprishg XMrr OUT (31:4) signals are coupled to associated data input terminals of a selector 424. Under control of the NC (7:0) nibble count signals, the selector 424 gates the signals from the corresponding ~i-th~ nibble to the input terminal of a fli~ buffer 426, which latches the selected signals in response to the next tick of the NODE CLK signal.
The low-order nibble, comprising the XMIT OUT (3:0) signais, is loaded into a buffer 425.
The buffer 425 is enabled in response to tbe NC 7 nibble count signal, and it latches the nibble in response to the next tick of the NODE CLK signal. It will be appreciated that the buffer 425 latches the low-order nibble contemporaneous with the transmission by the selector 424 of the bigh-order nibble, that is, the XMIT OUT (31:27) transmitter out signals. The seleaor 424 traDsmits the co~tents of buffer 425 to the input terminal of nit buffer 4~6 in response to the NC 0 nibble count signal. The latching of the low-order nibble in the buffer 425 perrnits the contents of the buf~er register 410 to be updated in response to the NC 0 nibble count signal, making the next word available for transmission immediately after transmission of the low-order nibble in buffer 425.
As noted above, the output of the selector 424 are latched iD the flit buffer 426 in response to the NODE CI~C signal. The output of flit buffer 426, namely, the l.AT FLIT (3:0) latched fiit sy~al~, are coupled to the check flit geDerator 416, the transmitter control circuit 412, and also to the input terminal of a gated driver 427. When the transmitter control circuit 412 asserts the EN OU T enable output signal, the gated driver 417 couples the LAT FLIT (3:0) latched flit signals to o~e set of da~a input terminals of mul/iplexer 415. As described above, if the transmitter control circuit 4~ is negating the CHECK OUT signal, the multiplexer 415 couples the contents of flit buffer 426 as the FLIT IN
sigDal to the data router Dode æ(iJ,k) connected thereto.
As described above, the data router interface 205 includes two message injector ports æ3(1) and 223(r). Tbe processor 200 (Fig. 8) controlling the processing element 11 can select one of the - : ,. , , :, . ~ ,~

~.: "

WO 92/06436 2 ~ 9 3 ~ 3 -s6 message injector ports to inject the data router message packet 30 into the data router 14, in which case the information used in forming the m~ssage paclcet 30 are coupled to the identifled message injector port 223. On the other halld, the processor 200 c~n initiate injection of a data router message pac~et 30 by referenc~g a "middle~ message injector port, ar d the injector common corltroltstatns portion 224 will select one of the left or right ports 223 to inject the message packet 30. Fig. 9B-9 depicts a target select circuit 426, in the i~njector common control/status portion 224, for accomplishing this.
With reference to 9B-9, the target select urcuit receives (L) FIFO FULL and (R) FIFO FULL
FIFO full status signals from the first-in first-out buffers 321 (Fig. 9B-1) in the left and right injector ports 223(1) and 223(r), respectively. The target select circuit 426 includes a set of AND g,ates that use these status signals, and pointer signals CUR TARGET L current target left and CUR TARGET R
current target right generated by a flip-flop 442 and inverter 443. The assertion or Degation of the CUR TARGET L and CUR TARGET R identify whether the left or right injector port will be selected to transmil a data router message packet 30 addressed by the processor 200 to the "middle"
message injector port. ~ ~ ~
AN~ gate 430 is energized by the coincident assertion of the (L) FIFO FULL signal and of the CUR TARGET L signal to assert a CUR L FULL current left/full sigDal. The assertion of the CUR L FULL signal indicates that the left injector port 223(1) is currently selected by the target select circuit 426, but that its FIFO 321 is full. Similarly, AND gate 431 is energized by the coincident assestion of the (R) FIFO FULL signal and of the CUR TARGET (R) signal to assert a CUR R
FULL cusrent sight/full signal. The assertion of the CUR R FlJLL signal indicates that the sight injector port 223(4) is currently selected by the target select urcuit, but that its FIFO 321 is full.
AND gate 432 is energized by the coincident assertion of the (L) FIFO FULL signal and the CUR TARGET R signai to assert a NEW R FULL L llew right/full left signal. The assertion of the MEW R FULL L signal indicates that the right injector port 223(r) is currently selected by the target selector circuit a~d that the FIFO 321 of the left injector port 223(1) is full. Similarly, AND gate 433 is energized by the coincident assertion of the (R) FlFO FVLL signal and the CUR TARGET L signal to assert a NEW L FULL R new left/full right signal. The assertion of the NEW L FULL R signal indicates that the left injector port 223(1) is currently selected by the target selector circuit 426 and that the FIFO 321 of the right injector port 223(r) is fnll. The NEW R FULL L and NEW L FULL R
signals are coupled to input terminals of an OR gate 435, whose output is complemented by an inverter 436 to generate a NEW TARG NOT FULL new target not full signal.
The CUR L FULL, CUR R FULL and NEW TARG NOT FULL signals are coupled to an OR gate 437, which generates a TOGGLE EN toggle enable signal. If an OR gate 440 is energized, which occurs when the processor 200 enables the data router interface 205 to inject a new data router message packet 30, i~ asserts a NEW MSG new message sigllai. If the TOGGLE~ EN and NEW MSG
signals are asserted, and AND gate 441 is energ~zed, which, in turn, enables the clock input tennirlal of flip-nop 442. The flip-flop 442 generates the CUR TARGET L current target left signal, which is coupled to AND gates 430 and 433. In addition, the CUR TARGET L signal is complemented by - . . . . ................ . .: :.,.
. : , - .;: . : . . : .. ,. ,. :: . . . ..

WO 92/06436 PCI/lJS91tO7383 , 2~9335~ -57-inverter 443 to generate the CUR TARGET R current target right signal is coupled to the AND gates 431 and 433. The CUR TARGET R signal is aLr7o coupled to the data input terrninal of flip-flop 442, so that, when the AND gate 441 is energized, the flip-flop 4~2 toggles its state. The CUR TARGET L
and CUR TARGET R signals are also coupled to circuitry (Dot shown) that controls loading of information into the write stages 320 of the respective left and right message injector ports 223(1) and 223(r), thus selecting the respective ports in response to the conditions of their respeclive buffers 321 and the port through which the last data router message packet 30 was being injected.
While the message injector port 223 has been described as iDcluding ciruits all of which operate in response to the NODE CLK si~,nal provided by the clock buffer 207, it will be appreciated that that will nor~nally require substandal portions of the network interface 202, and iD SOD~e cases the entire processing element 11 (Fig. 8) to operate in response to the NODE CLK signal. In many cases, it may be desirable to have most of the processing element 11, including most of the network ~nterhce 202, to operate in response to a processing element clock signal (not shown) which synchronizes most operations on the processing element 11, and only small portions of the nehvork interface 202 operate in response to the NODE CLK signal. In particular, it may be desirable to have at least the stages of the message injector ports 310 through 325 (Fig. 9B-1), that is, those stages in advance of the transmitter stage 326, operate in response to the processing element clock signal.
In that situation, the various circuits of the transmitter stage 326, as shown on Fig. 9B-8, will operate in response to the NODE CLK signal from .~he clock buffer 207. In additiorl, the RUN signal, generated by the nibble counter 411, will not be coupled directly to the message address computation stage 325 and physical address computation stage 324. Instead, the RUN signal will be directed to a synchronizer 444 that generates, in response to the RUN signal, a SYNCH RUN synchror~ized run signal that controls the message address computation stage 325 and physical address computation stage 324.
Fig. 9B-9 depicts details of the synchronizer 444. With reference to Fig. 9B-9, syDchronizer 444 includes a set-reset flip-flop 445 that is set in response to the assertion of the RUN signal. Flip-flop 445 includes direct se~ and direct reset terminals, ;dentified as ~S~ and "R~, respectiYely. When a signal applied to either terminal changes conditiorl from negated to asserted, the nip^flop is, respectively, set or reset.
Ths set flip-flop 445 energizes the data input terminal of a second flip-flop 446, which operates as a buffer. In response to the next tick of a PE CLK processi~g element clock signal, the flip-flop 446 is set, to assert a Bl~: RUN signal. The asserted BUF RUN signal, in turD, enables a state machine 447, which actually asserts the SYNCH RUN synchro ~ized run signal to control the message address computation stage 325 and physical address computation stage 324.
The state machine 447, which is clocked in response to the PE CLK signal, has a state diagram which is also shown OD Fig. 9B-9. In particular, the state machine 447 has three states, namely, an initial state, represented by the box labelled ~INIT", an iDtermediate state, represented by the box labelled ~MID," and a synchronized run assert state, represented by the box labelled ~SYNCH RUN,~
",.~

: .. ~ , :, , : , :, .

WO 92/06436 PCllUS91/07383 20933-~a in which it asserts the SYNCH RUN signal. Transitions betweeD states occur in synchronism with the PE CLK signal.
The state machine 447 is initially in the initial state. When the BUF RUN buffered run signal is asserted, the state machine sequeaces to the intermediate state at the next tick of the PE CLK signal.
Regardless of the condition of the BUF RUN signal, the state machine 447 se4uences to the synchronized run assert state at the next dck of the PE CLK signal. As noted above, the state machiDe 447 asserts the SYNCH RUN signal when in the synchronized run assert state. The assertioo of the SYNCH RUN signal, in addition to controlling the message address computation stage 325 and physical address computation s~age 324, also resets the nip-flop 445, enabling the fiip-flop 446 to be reset at the next tick of the PE CLK signal, to negate the BUF RUN signal. The next tick of the PE
CLX signal also sequeDces the state machine 447 to the initial state. SiDce at that poir t the BUF RUN
signal is negated, the state machine will remain in the initial state.
iii. Message Ejector Portion 221 With reference again to Fig. 9A-l, the data router interface 201 includes left and righ-t message ejector ports 225(1) and 225~r). Since the left and rigbt message ejector ports are generally similar, only one (v.~ithout reference to it being the leh or rig~t port~ will be described. Fig. 9C-1 depicts a general block diagram of a message ejector port 225, and Figs. 9C-2 through 9C-7 depict more detailed logic diagrams of selected elements depicted in Fig. 9C-1.
With reference to Fig. 9C-1, the message ejector port includes a flit receiver stage 450, an ejector flit first-in first-out buffer tI:lFO) 451, a message assembler 452 and a bus interface 453. The flit receiver stage is connected to the data router 15 and receives the successi~e four-bit OUT FLIT
signals from the data router node æ(l,j,k) connected thereto. The flit receiver stage also performs an error detectio~ operation, in which it verifies correct transmission of the data rs)uter message packet through the data router 15. The flit receiver stage 450 controls loading of nibbles representative of the su~cessively-received flits into the ejector first-in frst-out FIFO 451.
The message assembler 452 receives the successive four-bit nibbles from the FIFO 451 and assembles thirty-two bit words in response thereto. The thirty-two bit words are available to the processor 200 through the respective receive register 260, 295 or 302. Each of the successive thirty-two bit words receive in a message packet effectively has the same word organization as the successive words transmitted by the source leaf 21(x).
The bus interface 453 co~trols the transmission of data from data router message packets 30 to the processor. The bus interface 453 also has a fust-in first-out buffer to buffer data received from the data router 15 before it can be retrieved by the processor 200.
Fig. 9C-2 depicts a detailed logic of the flit receiver stage 450. With reference to Fig. 9C-2, the flit receiver stage includes an AND gate 460 that is enabled by a normally asserted FLOW EN Ilow enable sigDal from a receiver stage control circuit 461. If the FLOW EN flow enable signal is asserted, and if a FIFO ALMOST FULL signal from FIFO 451 is not asserted, the AND gate 461 asserts an OUT FLY signal that is transmitted to the data router node æ(iJ,k) connected thereto. The F~LOW

WO 92/06436 PClt~JS~1/07383 2093~ci -S9-EN flow enable signal may be conditioned by a register (not shown) that may be coDditioned by the diagnostic network 16 to effectively enable or disable the ejector port 225. The FIFO 451 asserts the FlFO ALMOST FULL signal when it is nearly full to regulate ilow of flit data thereto for storage.
The output terminal of AND gate 460 is also coupled to a delay line 461 so that, wherl the OUT FLY signal is asserted, a MSG FLOW message flow signal is asserted a brief time thereafter.
The MSG FLOW signal is coupled to a receiver stage control circuit 462.
The four-bit OUT F~IT (3:0) output flit signals frorn the cormected data router node 22(1,j,k) are received at input terminals of a latch 463 and latched in resporlse to successive ticks of the NODE
CLK signal. The contents of latch 463 are coupled as LAT FLIT ~3:0) latched flit signals to input terminals of the receiver stage control circuit 462, to a message check circuit 464, and to one set of input terminals of a multiplexer 465.
The receiver stage control circuit 462 rece*es the LAT FLIT (3:0) latched flit signals from the latch 463 in synchrony with the NODE CLK signal. If the immediately-preceding data router message packet 30 has been comple~ely received, and if a SOM` COND DET start-of-message condition detect signal from an OR gate 465 is asserted, the receiver stage colltrol circuit 462 asserts a HEADER
signal. The OR gate 466 generates the SOM COND DET start-of-message condition detec~ signal if at least one of the four OUT FLlT (3:0) output flit signals is asserted, indicating the start of a new data router message packet 30 if the immediately-preceding data router message packet has been completely received.
The asserted HEADER signal reseSs the message check circuit 464 and enables it to iDitiate a check operation in connection with the LAT FLIT (3:0) latched flit signals in synchrony with the NODE CLK signals, while it is eDabled the MSG FLOW message flow signal is asserted If the MSG
FLOW signal is negated, the message check arcuit 464 is disabled. The ou~put of the message check circuit 464 is coupled to a second set of data input tenninals of multiplexer 465.
The multiplexer 465 is controlled by a CHECK IN signal from the receiver stage control circuit 462. The CHECK IN signal, when negated, enables the multiplexer 465 ~o couple the LAT
FLIT (3:0) latched flit signals as RCV DATA (3:0) received data signals to the input terminal of ejector first-in first-out FIFO 451 (Fig. 9C-1). The HEADER signal frorn receiver stage control circuit 462 is also coupled to the input terrninal of FIFO 451 as a high-order RCV DATA ~4) signal. The HEADER signal, since it is asserted during ~he receipt of the first flit of a data router message packet 30 and negated otherwise, is used as a start of message indicator in successive stages of the message ejector port 225.
The receiver stage control circuit 462, while the MSG FLOW signal is asser~ed, asserts a WRlTE FIFO signal that enables the FIFO 451 to latch the successive five-bit RCV DATA (4:0) ..
receive data words, representing successively-received flits, in synchrony with the NODE CLK signal.
It will be appreciated that, if the FIFO 451 asserts the FIFO ALMOST FULL signal, which negates the MS& FLOW signal, the receiver stage control circuit 462 will negate the WRITE FIFO signal, disabling loading of additional data into FIFO 451.

... " ,.. .. . . ...

.. : , , , , . - ~; ;

", . ~, . . . .
:, ; , , ' '.. ! ,' ', I

WO 92/06436 PCr/US91/07383 2~33~

As indicated above, the receiver stage control circuit 462 also receives the LAT FLlT (3:0) latched flit si~als. The receiver stage control circuit specifically latches the signals of the header field 40 and the message length 34 of the data router message packet 30 and uses them to indicate when ~he receiver stage 30 has received all flits of a data router message packet 30. When the receiver stage control circuit determines that all of the flits have been received, it asserts the CHECK IN signaL
enabling the multiplexer 465 to couple CHECK VERIFY signals from the message check circuit 464, which indicate whether the data router message paeket was correctly transferred through the data router 15, as the RCV DATA (3:0) receive data signals. The receiver stage control circuit 462 may then, in response to the next assertion of the SOM COND DET start-of-message condition detect signal by the OR gate 466, determine that the OUT FLIT (3:0) signals represent the first flit of a new data router message packet, in which case it enables ~he operations as described above to oe repeated.
The first-in first-out FIFO 451 buffers the RCV DATA (4:0) receive data signals, represeDting five-bit words, from the flit receiver stage 450. In response to READ FIFO signals from the ejector message assembler 452, the FTFO 451 transmits the buffered words to the message assembler 452 as -EJ FIFO OUT t4:0) ejector first-in first-out buffer out signaLs.
The message assembler 452 receives successive words represented by the five-bit EJ FIFO
OUT (4:0) signals and forms, from the low-order four bits, thirty-two bit words. The mossage assembler 452 asseznbles the f~st thirty-tl,vo bit word from the contents of tbe header, message length and message tag fields 40, 34 and 35 of a received data router message pacl~et 30. ID addition, if the data router message packet 30 was received while the data router 15 is in all-fall-do~vn mode, the first word includes the conte~ts of the down path identification portion 41, as received by the ejedor port 225. The message assembler 452 forms each of the succeeding thirq-two bit words from the successive data flits 36. Each of the successive thirty-two bit words formed by the rnessage assembler 4æ fro~n the data flits 36 from a data router message packet 30 received by an ejector port 225 at a destination leaf 21(y) have the same contents as the successive thirty-two bit words from which data router message packet 30 was formed by the injector por~ 223 at the source leaf 21(x). Aceordingly, the binary-encoded values represented by the successive Ihirty-nvo bit words received by the ejector port 225 at the destination leaf 21(y) are the sa~ne as the binary-encoded values represented by the thirty-t vo bit words transmitted by the injector port æ3 at the sowce leaf 21(x).
In the message assembler 452, a header delay circuit 470 generates a normally-asserted ASSY
CTRL EN assembly control enable signal, which enables one ~put terminal of an AND gate 472. If a STALL signal from the bus interface 453 is not asserted, an inverter 471 asserts a RUN signal, which energizes an AND gate 472 to assert a CIRL EN control enable signal. If the first-~n first-out I:IFO
451 is not empty, it negates a FIFO EMPTY signal, which is complernented by an inverter 473 to enable an ANI~ gate 474. The asserted CI RL EN coDtrol enable signal ener~zes ~he AND gate 474 to assert a READ FIFO signal. While the READ FIFO signal is asserted, the FIFO 4$1 ~rans~its successive five-bit words, defined by the EJ FIFO OUT (4:0) signals, in synchrony vith the NODE
CLK signal.

. . ; ,:; ' ' , : . , ' ', : :-:'.,, , . , .. ' , ' ~ ,! .

, :`' : , ''~ ' ~" ~ ;

WO 92t06436 PCI'/US91/07383 2~33~ -61-The bus interface 453 ma~ntains the STALL sigslal in a negated condition if it can receive data from the message assemble~ 452. When the bus interface 453 cannot receive data from the message assembler 452, it asserts the STALL signal lo negate the RUN signal. The negated RUN signal disables the AND gate 472 to negate the CrRL EN signal, thereby enabling the AND gate 474 to negate the READ nFO sigral. In addition, if the FIFO 451 asserts the FIFO EMPI Y signal, inverter 473 disables the AND gate 474, enabli~g it to negate the READ FIFO signal. While the READ FIFO
signal is negated, transmissions from the FIFO 451 stop.
A multiplexer 475 has one data Input terminal connected to receive the five-bit word def~ed by the EJ FIFO OUT (4:0) signals from the first-in fisst-out FIFO 451. If the FIFO EMPI Y signal is asserted, the multiplexer 475 couples negated signals, having a binary-encoded value of zero, as SEL
DATA selected data signals to the inpu~ terrninal of a flit buffer 476, which latches them in response to successive tieks of the NODE CLK signal.
051 the other hand, if the ~FO EMm~ signal u negated, the multiplexer 475 couples the EJ
- FIFO OIJT (4:0) signals as SEL DATA selected data signals to the input terminal of a fGt buffer 476 and the high-order EJ FIFO OUT t4) sigslal as an SEL DATA (4~ (HEADER) signal to an assembly corltrol circuit 4Tl'. The SEL DATA (4) (HEADER) signal indicates whether the remaining El l:IFO
OI~T (3:0) si~nals comprise tbe header of the received data router message paclcet 30~ The assembly control cireuit 477, iri response to tbe CrRL EN control enable signal from AND gate 472, and in resporlse to successive tich of the NODE CLK signal, generates tirning control signals for controlling the other elements of the message assembler 452.
Each tick of the NODE CLK signal enables the buffer 476 to latch SEL DATA selected data signals defining a five-bit word as received from the FIFO 451. The buffer 476 transmits the latched signals as BUF SEL DATA (4:0) buffered selected data signals. If the BUF SEL DATA (3:0) s;gnals frorn flit buffer 476 represent the header field 40 of the data souter message packet 30, the high-order BUF SEL DATA (4) buffered selected data signal from buffer 476 is asserted, which enables an AND
gate 481. If the RUN signal is also asserted, the AND gate 481 is energized to enable a load enable input tesminal of the header deby ~rcuit 470. The header delay circuit 470 receives BUF SEL DATA
(3:0) signals in response to the next dck of the NODE CLK signal. It will be appreciated that at that point the BUF SEL DATA (3;0) signals buffered by flit buffer 476 represent the header field 40 of the message paclcet 30. At that point, the header delay circl~it 470 negates the ASSY CrRL El~ assembly control enable signal, which, as described above, results in negation of the REAl ) FIFO signal.
The header delay circuit 470 maint~uns the ASSY Cll~L EN assembly control enable signal negated for a nu~nber of ticks of the NODE CLK signal which depends on the binary-decoded value of the BUF SEL DATA (3:0) signals, to provide timing synchron~zation with the bus interface 453. The Dum~er of NODE CLK ticks that the ASSY CTRL EN signal remains negated is, in turn, related to the number of flits in the down path identification portion 41 of the received data router message packet 30, which, in turn, relates to the binary encoded value of the BUF SEL DATA (3:0) signals.
The assembly control circuit 477 generates timing and control signals that control the arran8ement of four-bit nibbles, represented by the successive low order BUF SEL DATA (3:0) WO 92/06436 PCr/USg1/07383 -62- 20~33~
buffered selected data signals into thirty-two bit words, which are assembled in an assembly synchronizer register 480.
The assembly synchronizing register 4B0 includes eight fow-bit nibbles identified by reference numerals 480(7) through 480 (0) [generally identified by reference numeral 480~ , a beginning-of~
message flag 486 and an all-fall-down mode nag 487. The beginn~r~r-of-message nag 486, when set, indicates that the word assembled in nibbles 480(i) comprise the first word from a data router message packet 30 being received. The all-fall-down mode nag 487, when set, indicates tbat the data router message packet 30 is being received while the data router 15 is in all-fall-down mode. The aD-fall-down mode flag, 487 is used to condition the received all-faD-down flag 254 of the appropriate private register 232, 294 or 301 (Figs. 9A-2A and 9A-2B).
In addition, the assembly control circuit 477 generates a VALID WORD signal, which is trar smitted to the bus Interface 453. If asserted, the VALID WORD signal indicates that the message assembler 452 has assembled a thirty-two bit word at register 480 for transmission to the bus interface 453. The bus interface 453 controls the latching of the thirty-two bit word by the register 480.
If the SEL DATA (4) (HEADER) signal from multiplexer 475 is asserted, the assembly control circuit 477 receives the latched BUF SEL DATA (3:0) signals definillg the header from the buffer 476. The assembly control circuit 477 uses the BUF SEL DATA (3:0) signals to determine whether the data router message packet 30 is being received while the data router 15 all-fall-down mode. If the data router 15 is not in all-faD-down mode, the data router node (1~ ) connected to the ejector port 225 will have deaemented the contents of the header field 40 of the data router message packet 30 to a condition in which all of the BUF SEL DATA (3:0) buffered selected data signals are asserted, representing the binary-encoded value ~fifteeD.~ In addition, the data router node (lJ,k) wiD
have discarded the last flit of the down path idensification portion 41, and so only the message data portion 32 remains after the header field 4û.
As a result, the assembly control circuit 417 generates ti ning and control signals that enable the BUF SEI, DATA (3:0) signals representing the header, length and tag fields 40, 34 and 35 to be latched as the first word in the assembly synchronizing register 480. Therea~er, the timing and control signals from the assembly control circuit 477 enable the assembly synchronizing register 480 to form successive thirty-two bit words from successive sets of eight four-bit words, each of the four-bit words being defined by the BUF SEL DAT4. (3:0) signals at successive ticks of the NODE CLK signal. The assembly control circuit 4J7 enables the assembly synchronizing register to form a number of thirty-two bit words, the number corresponding to the value in the length field 34. Thereaher, the assembly control circuit enables the assembly synchronizing register to form aDother word to accommodate BUF
SEL DATA (3:0) signa]s comprising the check bits as gcnerated by the message check circuit 464 (Flg.
9C-2).
On the other hand, if the assembly control arcuit 47 7 determines from the BUF SEL DATA
(3:0) signals that the data router message packet 30 being received while the data router 15 is ;D all-fall-down mode, the BUF SEL DATA (3:0) buffered selected data signals that define the header field 40 ~: ' ' ' ' . : `, ' ' . ,': . :. ` ' ' ' : ::; .' ;~ , . '~:, . ' ' ~. , , ,,, ' ' ' ' " ' :: '.`' ''. : ` , ` , ` :
~:' : , " ` ' '' ' ' . '. ''.: .. '' j '': ~ ' ' ' ,:
.~ "~: ' , ' ` '. , ` ' '', ' ' '' ' :' ' ':. ' ` .
.;: , :. ,` " , , " , : : ~ , , , :
',: , : : ', ` ' ,.. ~, - . ' `,: ' ':

2 ~ 9 3 3 ~ ~ -63-are not aU asserted. As noted above, when in all-faU down mode, the data router nodes 22(i,j,k) in the data rou~er 15 will maintain the contents of the header fields 40 of the data router message packets 30 at their respective values at the time the all-fall-down mode is initiated. In one particular embodiment, the nurnber of levels in the data router 15 is selected so that the btilary-encoded value of the BUF SEL
DATA (3:0) signals wiU be less than afifteen.n Thus, the assembly control circuit 477 can det~-rmine whether the data router 15 is in an all-faU-down mode by determining whether the binary-encoded value represented by the BUF SEL DATA (3:0) signals is "fifteen."
As also noted above, when in all-faU-down mode, data router message packets that the ejector port 225 receives include at least some flits comprising the down path identification portion 41. In that case, the assernbly control circuit 477 generates tiining and control signals that enable the BUF SEL
DATA (3:0) signals representing the down path identification portion 41, as well as those representing the header, length and tag fields 40, 34 and 35, to be latched as the first word in the assembly synchronizing register 480. Thereafter, the tirnirlg and control signals from the assembly control Qrc~it 477 enable the assembly synchronizing register 480 to form successive thirty-two bit words ii tbè same manrler as when Ihe data router 15 is not in all-fall-down mode.
With this background, the assembly control circuit 477 incllldes circuitry, which is described below in connection with Figs. 9C~ d 9C-5 that successively generates FLIT ~i~ OF WORD signals (ni" is an integer from seven to zero) that control coupling of the low-order BUF SEL DATA (3:0) signals for latching in particular nibbles of the assembly synchronizing segister 480; The BI~F SEL
DATA (3:0) buffered selected data signals are coupled to the input terminals of respective gates 481(7) through 481(0) Igenerally identified by refereDce numeral 481(i)] which are controlled by the respective FLIl` "i" OF WORD signals. The output terminals of the gates 481(i~ are connected to respective ones of buffers 482(7) through 482(0) Igenerally identified by reference numeral 482(i)] which latch and buffer the signals gated by the respective gates 481(i). Effeaively, each buffer 482(i) latches the signals gated thereto by the respective buffer 481(i), in response to an enabling signal comprising the r coincidence of the FLlT ~i" O~ WORD SIGNAl. and the NODE CLK signal, the enabling sig~al being delayed to accommodate delay of propagatit)n of the gated signals to the input terminals of the respective b~ffer 481(i).
The output terminals respective buffers 482(~) through 482(0) are connected to input terminals of respective gates 485(7) through 485(0) Igenerally identified by reference numeral 485(i)].
In response to an asserted LAT WORD latch word signal from the assembly control circuit 477, the gates 485(7) and 485(5) through 485(1) couple the signals directed thereto by the buffers 482(i) to respective nibbles 480(i) of the assembly sync~roniziDg register 480. The nibbles 480(i) all latch the signals in unison in response to the assertion of the WRlTE ASR write assembly svnchroDîzing register signal ~rom the bus interface 453.
The input signals to nibbles 480(0) and 480(6) of the assembly synchronizing register 4~ are provided by multiplexers 483 and ~4, respectively. The mu]tiplexer 483, under control of a DATA
NIB (O) data nibble "zero" signal from an AND gate 486, selectively couples the output signals frorn " . , , . . :. .. . ~ ,. . . .

WO 92/06436 P~/U~;~l/07383 ~g3355 either gate 481(0) or 485(0) to the nibble (0) of the assembly synchron~g register 480. Similarly, the multiplexer 484, under coDtrol of a TAG sigDal from the assembly control circuit 477, selectively couples the output signals from either the gate 481(6) or 485(6) to the nibble (6) of the assembly synchroniDng register 480.
The assembly control circuit 477 controls ~.he assembly synchron~g reg~ster 480, the gates 481(i), buffers 482(i) and gates 485(i), as well as muldplexers 4b3 and 484, as follows. When the SEL
DATA (4) (HEADER) signal is asserted, the assembly control circuit 477 latches the BUF SEL
DATA (3:0) buffered selected data signals comprising the header field 40 of the data router message packet 30 being received. The assembly control circuit 477 asserts the FLIT 7 OF WORD signal, enab'~ing the gate 481(7) to couple the BUF SEL DATA (3:0) signals to the input terminal of buffer 482(7), which latches them in response to the next NODE CLK signal.
As noted above, the BUF SEL DATA (3:0) signals correspond to the contents of the message packet's header field 40. If the signals indicate that the data router 15 is jD all-fall-down mode, the assembly control circuit 477 asserls a DNF LOOP down nit loop signal and enables a dowrl flit counter, described below in connection with Fig. 9C-5. The down flit counter generates count signals that cootrol generation of the FLlT "i~ OF WORD sigDals to enable gating and buffering of BUF SEL
DATA (3:0) signals representing the successive flits of the down path identification portion 41 iu the successive ones of buffers 482(i). In one particular embodirnent, in which data router rnessage packets ~' 30 have a ~na~nmutn of five flits in the down pa~ identification portion, the FLIT ~ia OF WORD signals enable the BUF SEL DATA (3:0) signals to be successively buffered in buffers 482(4) t'nrough 482(0).
If a particular message packet 30 has fewer than five nitS n the down path identification portion, the FLlT ~i~ OF WORD signals enable the buffered BUF SEL DATA (3:0) signals to 'oe pac~ed toward ~Lhe buffers 482(i) with lower indices ~i"; that is, if a data router message paclcet 30 has only "j~ flits (~j~
less than five) in t'ne down pat'n identification portion 41, îhe ass.eulbly control circuit 477 successively generates the FLIT ~ OF WORD through FLIT 0 OF WORD signals, to enable the BUF SEL
DATA (3:0) signals representative of those flits to be loaded into buffers 482a-1) through 482(0).
In addition, when the assembly control circuit 477 asserts the FLIT 0 OF WORD signal, a multiplexer 490 couples the DNF LOOP signal, which sets the all-fall-down mode flag 487.
After the BUF SEL DATA (3:0) signals represer tative of the town path itentification portion 41 have been buffered in appropriate ones of buffers 482(4) through 482(0), the assembly control circuit successiwly asserts a LEN length signal and a TAG signal, in sync~onism with successive ticl~s of the NODE CLK signal. Contemporaneous with its assertion of the LEN le~ signaL the assembly control circuit 477 also asserts the FLIT S OF WORD signal, enabling the BUF SEL DATA (3:0) sy~nals to be gated to and latched by buffer 482(5). At tha~ point the BUF SEL DATA (3:0) buffered seleaed da~a signals are representative of the length field 34 of the data router message pac~et 30. In addition, the assernbly control circuit 477 stores the length information from the BUF SEL D~TA
(3:0) signals for its later use in assembling thirty-two bit words from data nits 36.
Contemporaneous with its assertion of the 'rAG signal, the assembly control Qr~uit 477 asserts the FLIT 6 OF WORD signal, enabling gate 481(6) to couple the BUF SEL DATA (3:0) buffered -- - - . : . :
- - . .~ ~ , . . .
: .: . .::. : : . :: . . ,. :
.
: , :
. .: :: ,: . . :.: :: :;, , WO 92/06436 Pcr/us91/073g3 selected data signals to the ~nput tenninal of both buffer 482(6) and to one set of data input terminals of multiplexer 484. Since the TAG signal is asserted, the multiplexer 484 couples thc output of gate 481(6) directly to the input terminal of the respective nibble (6) of the assembly synchronizing register 4~0. ln addition, the TAG signal is coupled to one input terminal of a multiplexer 491, which controls the input to begim~iDg-of-message flag 486 of the assembly synchror~zmg register 480.
Contemporaneously with the assertion of the TAG sigDal, the assembly control circuit 47'7 asserts the I~T WORD latch word signaL which enables the gates 485(7) and 485($) throttg~ 485(1) to couple signals from buffers 482(7) and 482(5) through 482tl) to respective nibbles (7) and (5) through (1) of the assembly synchronizing register. In addition, the LAT WORD signal enables the gate 485(0) to couple signals from buffer 482(0) to the multiplexer 483. Si~lce the DATA NIB (0) data nibble signal is negated, multiplexer 4~3 couples the signals from gate 485(0) to nibble (0) of the assembly synchroniaDg register 480. ID addition, the assembly control circuit asserts the VALID WORD signal, to indicate to the bus interface that a word is available at the input terminals of the assembly - synchror~izing register 480. ~ ``
When the bus interface 453 can receive the word from the message assembler 452, it asserts the WRITE ASR write assembly synchronizing- register signal, which enables the assembly synchronizirig register 480 to latch the signals at its input terminals. The assembly synchronizing ~`
register 4~0 transmih its conteDts to the bus interface as RCV WORD (34:0) recei~ed word signals, comprising the thirty-two bits from nibbles 480(7) through 4R0(0), a BOM LAT beginning-of-message r latched signal, and an AFD LAT all-fall-down latched signal.
It ~ill be appreciated that multiplexer 484 is provided to reduce the amount of time required to direct the respecti-fe signals to all of the nibbles 480(i) of the assembly synchroninng register 480 when it is for~niDg the first word from a data router message packet 30. In particular, since the gate 482(6) is the last to be esabled for the word, the mulliplexer 484 ensures that the signals from gate 481(6) do sot have to be latched in buffer 482(6) before they are coupled to the input ~enninals of nibble 4~0(6). This can reduce the amount of time required to assemble signals at the input terminals of all of the sibbles 481(6) by appro~nately one tick of the NODE CLK signal.
After the assembly synchronizing register 480 has transmitted RCV WORD (34:0) signals defDing the first word of the data router message packet, ~he assembly control circuit generates the timing asd control sigDal to enable the assembly syDchroDiziDg register 480 to assemble one or more thirty two bit data words, the number corresponding to the previously-stored length irlformatiom In assembling each word, the assembly control c~rcuit 47 7 asserts a DATA LOOP signal and successively asserts the FLIT 7 OF WORD through FI,lT 1 OF WORD signals. In respor~se, the successive gates 481(7) through 481(0) couple the BUF SEL DATA (3:0) signals, which a~ successiYe ticks of the NODE CLK signal represent contents of successive data flits 36 for latching in the successive buffers 482(7) through 482(1).
When the FLIT 0 OF WORD signal is asserted, the coincidence of the assertions of that si~al and the asserted DATA LOOP signal enable an AND gate 492 to assert a DATA NIB (0) data nibble . . .
- :. . ' ,. . :

WO 92/06436 PCl/US91/07383 ~ 20~3~5~
signal, which enables moltiplexer 483 to couple the output signals from gate 4~.1(0) to the nibble 4~'.0(0) of the assembly synchronizing register 480. In addition, contemporarleous with ~he assertion of the FLlT O OF WORD signal, the assembly contJol circuit 477 asscrts the L~T WORD signal to enable ga~es 485(7) through 48S(1) to couple the contents of buffers 482(7) through 482(1) to respcctive nibbles 480(7) through 480(1) of the assembly synchronizing register 480. The assembly control circuit 477 also asserts the VALID WORD signal to notify the bus interfacc 453 ~at it may assert the WRITE
ASR write assembly synchronizing register signal to eDable the assembly synchronizing register 480 to latch the signals input thereto.
It will be appreciated that multiplexer 483 is prov~ded to reduce the amount of time reqls~red to direct the respective signals to all of the nibbles 480(i) of the assembly synchronizing register 480 when it is forming words from the data flits 36. In particular, since the gate 481(0) is the last to be enabled for a word, the multiplexer 483 ensures that the si~nals from gate ~.1(0) do not have to be Iatched in buffer 482(0) before they are coupled to the input terminals of nibble 480(0). This can reduce the amount of time reqoired to assemble signals at the input terminals of all of the nibbles 481(0) by approximately one tick of the NODE CLK signal.
The assembly control circuit 477 iteratively enables these operations to occur until the assembly synchronizing register 480 has formed data words from all of the data flits 32 m the data router message packet 30. Thereafter, the assembly control circuit enables the assembly synchro~g register 480 to form one last data word, which includes the check signals generated by the message check generator 464 (Fig. 9C-2), which are latched in the nibble 480(7) of the assembly synchroniz~
regis~er 480.
If the bus interface 4S2 is unable to accept a word from the assembly syDchsonizing register 480 when the assernbly control circuit 4r,' asserts the VALID WORD signal, it may assert the STALL
signal, which, as described above, stalls the ejeaor first-in first-out buffer from transmitting signals representing successive flits to the message assembler. Accordingly, the contents of the nit buffer 476 remain unchanged while the STALL signal is asserted. 1 hus, if the B'UlF SEL DATA (3:0) buffered selected data signals represe~t the tag field 35 if the message assembler 4æ is assembling the first word from the data router messag,e packet, or data flits 36 to be loaded ,n the lo~v-order nibble 480(0) of the assembly synchronizing register 480 if it is assemb'~ing a word from the data flits 36, the flit buffer 476 maint~uDs the signals whiie the STALL signal is asserted. Accs)rdingly, the s gnals will not have to be buffered in the respective buffers 482(6) or 482(2).
It will be appreciated that the first word formed in respollse to a new data router message packet 30 includes the contents of the header field 40 of a received data ;outer message packet 30. It will be appreciated that, the processor 200, when it loads the send first register 234, 2,6 or 303 to enabie Iransmission of a new data router message packet, does not supply a value for Ihe header field 40, since, as described above, the message ,njector port 223 generates the vaiue itself. However, the message assembler 452 keeps the vaiue of the header field 40 of a received data router message packet 30. If the message packet 30 was received when the data router 15 is in all-fa'll-down mode, the value ..... .. ... .. .. .. .... .. . ..
,, : :.. .- , ., , . .:

,, ' ' ' ' ' : ' ' . ~ ,,'. ' ''.' :' ' ': '' , : ; :l :~

WO 92/06436 P(-r/U591/073~3 2 0 9 3 3f~ d~i -67-of the header 1 40, w'nich is loaded into the nibble 480(7) during creation of the first word, is nsed to identify the ones of nibbles 480t4) throngh 480(0) which contain values from flits of the down path identification portion 41. If the value from the header field 40 were nGt kept, the contents of nibbles 480(4) through 480(0) wonld be zeroed or pnrged at some point before the down path identification portion 41 were loaded therein. The processor 200, when it retransrnits the data router message paclcet, can use the header infor nation stored in the nibble 480(7) to identify the n~nber of valid flits for the down path identification portion 41.
Figs. 9C-4 and 9C-S depict logic diagrams of some elements of the assembly control circuit 477. Fig. 9C 1 depicts state machine circuitry, that generates the LEN length, TAG, DNF down fli~ and DATA LOOP data loop siE~,nals. Fig. 9C-S depicts circ~aitry that ge~erates the 'rLlT ~i~ OF WORD
signals. With reference first to Fig. 9C 1, initially a flip-flop 500 is set, thereby asserting an IDLE ST
idle state signal. If the SEL DATA (4) (HEADER) signal is negated, an inverter 501 maintains an AND gate Sa1 in an energized condition, which, in turn, enables an OR gate 503 to assert an IDLE
signal,-which is latc'ned by tbe flip-flop ~ at each- tick of the NODE CLK signal. Th~.~., while the SEL
DATA (4) (HEADER3 signal is negated, the IDLE signal remains asserted, enabling Ihe flip-flop 500 to mainîain the IDLE ST idle state signal asserted.
The asserted IDLE ST idle state signal also energizes an OR gate 504, ~hich, in turn, enables one input terminal of an AND gate 505. If the SEL OATA (4) (HEADER) signal is asserted, the AND gate 505 is energized to assert a HEADER signal. In response to the next tick of the NODE
CLK signal, a flip-flop 506 is set to assert a HEADER ST header state signal. Contemporaneously, the asserted SEL DATA (4) (HEADER) signal enables inYerter 501 to disable AND gate 502"n turn enabhng the OR gate 503 to negate the IDLE s4~al. At the same tick of the NODE CLK signal, the fli~flop 500 is reset, negating the IDLE ST idle state signal.
The HEADER ST header state signal is asserted contemporaneously with the latching by the flit buffer 476 (Fig. 9C-3) of SEL DATA selected data signals corresponding to the contents of the header field 40 of the data router message packet 30 bei~g received. Turlling to Fig. 9C-5, the asserted HEADER ST signal energizes an OR gate 510 to assert aD OTHER (7) signal, that, in turn, enables an OR gate 511 to assert the FLIl` 7 OF WORD signal. As described above, this enables gate 481(7) to couplc the BUF SEL DATA (3:0) buffesed selected data signals, which at that point correspond to the coDtents of the header field 40, to the buffer 482(7).
Returning to Fig. 9C 4, it will be appreciated that the AND gate 505 will assert the HEADER
signal only while the SEL DATA (4) (HEADER) signal is asserted, which is the case only for one tick of ~he NODE CLK sigDal. When the SEL DATA (4) (HEADER) signal is negated, the AND gate 505 negates the HEADER signal, the flip-flop 506 is reset at the next tick of the NODE CLK signal, negating the HEADER ST header state sigDal. As a resuit, the OT~IER (73 signal generated by OR
gate 510 (Fig. 9C-5) is negated, thereby enabliDg the OR gate 511 to negate the FLIT 7 OF WORD
signal.
While the HEADER ST header state signal is asserted, it also enables one input terrninal of an AND gate 512 and of a second AND gate 531. A third AND gate 514 receives the BUF SEL

-: :.. ,.' ' , ' :

' ' :.' ': ' .. . .
.. : . . . .. .

WO 92/06436 PCl'/US91/07383 20~33~
68.
DATA (3:0) buffered selected data signals, which still correspond to the contents of the header field 40. The AND gate 514 effectively determines whether the ejector ps~rt's leaf 21 is the destination or if the message packet 30 is being received while the data router 15 is operating in an aD-fall-down mode.
As described above, if the data router 15 is not operating in an aD-faD-down mode, the BUF SEL DAT
(3:0) signals are all asserted, in which case the AND gate 514 is energized to assert a FLIT V= 15 flit value equals fifteen signal. On the other hand, if the data router 15 is operating in an aD-fall-down mode, the AND gate is disabled, negating the FLIT V = 15 signal.
If the FLIT V = 15 nit value equals fifteen signal is negated, the second input terminal of AND
gate 532 is disabled. However, the negated Fl,IT V= 15 signal enables an inverter 515 to, in turn, enable the second input terminal of AND gate 512. The coincidence of the asserted HE~DER ST
header state and FLIT V~ 15 signals energ-7es the AND gate 512, enabling it to assert a LOAD DNF
CTR load down flit counler signal. It wiD be appreciated that the AND gate 514 wiD assert the FLIT
V= 15 signal whenever all of the BUF SEL DATA (3:0) signals are asserted, which may also occur when signals represent other fields of the data router messzge packet 30. However, ~he AND gate 512 wiD assert the LOAD DNF CTR load down flit counter signal only when the HEADER ST header state signal is asserted, which wiD occur when the BUF SEL DA~A (3:0) signals represent the header field 40. The asserted LOAD DNF CTR load down flit counter signal enables an OR gate 513 to assert a DNF LOOP down flit loop signal. In response to the next tick of the NODE CLK signal, a nip-nOp 514 is set, enabling a DNF LOOP ST down nit loop state to be asserted. It will be appreciated that the HEADER ST header state signal wiLI at this point be negated.
At this point, a DNF Cl-R O down flit counter equals zero signal is asserled, which enables an inverter 525 to disable an AND gate 526. Returning to Fig. 9C-5, the BUF SEL DATA (3:0) buffered selected data signaLs, which at this point represent the header 40 of the data router message packet 30 being received, are also coupled to a decremeatation circuit 515. The decrementation circuit 515 receives the four BUF SEL DATA (3:0) signals and generates four signals representmg a binary-encoded value that is one less than the binary-encoded value of the BUF SEL DATA (3:0) signaLs, and transmits the three high-order signals as LD VAL (3:1) load value signals to the initial data input terminals of a counter 516. The asserted LD DNF CTR load down nit counter signal enables the counter 516 to load the LD VAL (3:1) signals as an initial count value.
The counter 516 transmits DNF CNT (3:0) down flit count signals, decrementing their binary-encoded value in response to successive ticlcs of the NODE CLK signal. The DNF CNT (3:0) s;gnals are coupled to an inverter 523 that couples the coDlplements of each of the signals to input terminals of an AND gate 524. If all of the DNF CNT (2:0) signals are negated, which occurs if the binary-encoded value equals zero, the AND gate 524 asserts a DNF CTR O down flit counter equals zero signal.
However, when the counter 516 generates DNF CNl (2:0) down flit CouDt signals hav~ng other binary-encoded values, the inverter 523 will disable the AND gate 524 to negate the DNF CI-R O down flit counter signal.
The negated DNF CTR O down nit counter sigaal is coupled to an inverter 525 (Fig. 9C~) which enables one input terminal of an AND gate 526. Since the other input terminal of AND gate 526 ,,, : . ... , , . :.

~ .: . . . ::, .. -, : : ::: ... -,.. .. . . :
- : . . . . : : .. ,: . : : ~ : : : ;. . :: : : , .
: . . . - . :~ .. ::: : . :: :. : , - :
. .. .
' . . :: . : :
.. . . . . .

WO 92/06q36 PCl'/US91/07383 ?~9~rr -69-is tk'en ena~f~d~b~ the asserted DNF LOOP ST down flit loop state signal, tJe AND gate 526 is energized. The energ~zed AND gate 526 enables the OR gate to maintain the DNF LOOP down flit loop signal in an asserted condition, which, in ~urn, causes the flip-flop 514 to remain set during su~ceeding ticks of the NODE CLK signal. This, in turn, maintains the DNF LOOP ST do vn flit loop state signal in an asserted condition. Thus, the DNF LOOP ST down flit loop state signal remains asserted until the DNF CIR 0 down flit counter equals zero signal is asserted, as described below, which causes the irlverter 525 to disable AND gate 526.
In addition, the DNF CNT (2:0) down fiit count sigl~als are coupled to a decoder 517 that asserts a respective one of eight DEC DNF CNT (7:0) decoded down nit co~mt signals assoaated with r the binary-encoded value of the DNF CNT (2:0) down flit count signals. The DEC DNF CNT (4:0) decoded down flit count signals are coupled to respective ones of flit ~i~ of word signal generating circuits, one of which, identified by reference numeral 520(i), is shown in Fig. 9C-S. Circuit 520(i) includes an AND gate 521 that is enabled by the asserted DNF LOOP ST down flit loop state signal from flip-flop 514 (Fig. 9C-4). When the dccoder 517 asserts the particular DEC DNF CNT (i) decoded down flit signal associated with the circuit 520(i), the ~ND gate 521(i) is energized to assert a DNF (i) OF WORD down nit ~i~ of word sigDal. This signal, wheD asserted, in turn energ~zes an OR
gate 522(i), enabling it to assert the FLlT ~i~ OF WORD signal.
In one embodiment, in which the down path identification portion 41 of a data routrr message packet 30 may have fNe flits, only the five low-order DEC DNF CNT (4:0) decoded down flit CouDt signals may be asserted. In addition, the assembly control circuit 47'1 includes five circuits 520(i), each associated with one of the DEC DNF CNT (4:0) signals. The DNF LOOP ST down flit loop state signal enables AND gates 521(i) in all of the circuits 520(i) in parallel. However, the decoder S17 asserts only one DEC DNF CNT ";~ decoded down flit signal at a ti~ne, which in turn determines the FJIT "i" OF WORD signal asserted. The FI~ OF WORD signals determine the ones of buffers 482(4) through 482~0) which will receive and latch the BUF SEL DATA (3:0) signals, at each of the successive tic~s of the NODE CLK signal, as described above.
At some point in the decrementing of counter 517, the binary-encoded value of the DNF CNT
(2:0) down flit count sigllals from counter 516 will equal zero. Wben that occurs, the signals are all negated, in which case the inverter 523 enables all input terminals of an AND gate 524, which is energized to assert the DNF CTR 0 down flit counter equals zero signal. The asserted DNF CTR 0 signal causes the inverter 525 (Fig. 9C4) to disable AND gate 525, and causing the OR gate 513 to negate the DNF LOOP down flit loop signaL The negated DNF LOOP signal causes the fli~flop 514 to 'oe reset at t'ne ne~a tick of the NODE CLK signal. At tnis point, if the data router message pacxet 30 'oeing received has any flits i tne down path identification portion 41, they have bee n latched in the respective b7lffers 482(4) through 482(0).
If the data router message packet 30 being received has flits in the do~.vn path iden~ication portioo 41, the asserted DNF LOOP ST down flit loop state signal also enables one input terrnina'l of an AND gate 527. When the DNP CTR 0 down flit co~mter equals zero sig~al is asserted as described , ,, . ., : , .:

WO 92/0~436 PCr/US9ltO73.3 2~33~
-7~
above, the AND gate 527 is energi~ed, which, in tum, energ~zes an OR gate 530 to assert a LEN leDgth signal. The asserted LEN length signal enables a flip-flop 531 to be set in resps~nse to the next tick of the NODE CLK sigoaL enabling the assertion of a LEN ST length state signal. It u~ill be appreciated that when the LEN ST length state siBnal is asserted, the BUF SEL DATA (3:0) buffered selected data signals correspond to the contents of the message length field 34 of the data router message packet being receiYed.
Alternatively, if the down path identification portion 41 of the data router message packet 30 is empty, when the HEADER ST header state signal is asserted the BUF SEL DATA (3:0) buffer selected data signals ~vill all be asserted. The asserted BUF SEL DATA (3:0) signals enable AND gate 514 to assert the Fl,IT V=15 nit value equals ffteen signal. The asserted FLIT V=15 signal causes inverter 515 to disable AND gate 512, wbich, in turn, prevents nip-flop 514 from being set. The disabled AND gate 512 maintains the LOAD DNF CTR load do vn flit counter signal iD a negated condition, keeping the counter 516 (Fi~,. 9C-5) from operating.
The asserted Fl.IT V = 15 flit value equals fi~teen signal also enables one iDput terminal of an AND gate 532. Since the fli~flop 506 at that point is asserting ~he HEADER ST header state sigDal, AND gate 532 is energized, which energizes the second input terminal of OR gate 530 to assert the LEN signal. The asserted LEN siE nal enables the flip-flop 531 to be set in response to tbe next tick of the NODE CLK signal, enabling the assertion of a LEN ST length state signal. It will be appreciated that the LEN ST length state signal is asserted one tick of the NODE CLK signal after assertion of the HEADER ST header state sigllal by flip-flop 506, indirating that the BUF SEL DATA (3:0) buffered selected data sigDals represent the message length field 35 one tick of the NOI:)E CLK signal after they represent the header field 40, as would be the case if the down path identification portion 41 of the data router message packet 30 being received is empty. Immediately after the flip-flop 531 has been set, the HEADER ST header state sigDal is negated, thereby enabling AND gate 532 and OR gate 320 to negate the LEN length signal.
At the next tick of the NODE Cl.K signal, the asserted LEN ST signal enables a flip-flop ~29 to be set to assert a TAG ST tag state signal. Contemporaneously, since the LEN length signal is negated, the flip-flop 531 is reset to negate the LEN ST length state signal. The asserted TAG ST tag state signal energizes an OR gate 532 to assert a DATA LOOP signal, which enables a fli~flop 533 to be set in respoDse to the next NOI)E CLK node clock signal to assert a DATA LOOP ST data loop state SigDaL 1D response to the same tick of the NODE CLK signal, the negated LEN ST length state signal enables the flip-flop 529 to be reset, negating the TAG ST tag state signal.
Accordingly, it will be appreciated that flip-flops 531, 529 and 533 are set in response to sequential ticks of the NODE CLK signal, to sequentiaDy assert the LEN ST length state, TAG ST tag state and DATA LOOP ST dala loop state signal. In addition, the flip-flops 531 and 529 are seset after being set for one tick, so that the LEN ST and TAG ST sigDals are only asserted for ooe tick of th NODE CLK signal. It will be appreciated that while the LEN ST and TAG ST signals are asserted, the BUlF SEL DATA (3:0) signals represent, consecutively, ~he contents of the length and tag fields 34 and 35 of the data router message packet 30 being received.

~: - :: ,,:: - : ' ; ' ., . ', ., , .
- ` ~ . . ,; , .: . . . :: . :
:

WO 92/06436 PCl /US91/07383 20~33~ -71-ID addition, a DATA CrR EQ 0 data counter equals zero siE~nal is initially negated, which, in turn, enables an inverter 538 to enable olle i~put terminal of an AND gate 539. Tbe asserted DATA
LOOP ST data loop state signal energizes the AND gate 539, which enables the OR gate 532 to maintain the DATA LOOP signal in an asserted condition. Accordingly, the fli~flop 533 will remain in a set condition, maintaining the DATA LOOP ST data loop state signal asserted. As will be described in more detail below, the DATA CIR EQ 0 signal is asserted when the BUF SEL DATA
(3:0) signals represent the last data flit 36 of the data router message packet 30 being received. At that pomt, the asserted DATA CTR EQ 0 signal tisables AND gate 539, enabling OR gate 532 to negate the DATA LOOP signal and causing the flip-flop 533 to be reset, ncgating the DATA LOOP ST
signal, in response to the next tick of the NODE CLK signal.
Returning to Fig. 9C-5, the LEN ST length state and TAG ST tag state signals are coupled to respective ones of OR gates 534 and 535, to enable them to assert the FLrr 5 OF WOl?.D and FLIT 6 OF WORD signals, respectively. This enables the BUF SEL DATA (3:0) buffered selected data signals, which sequentially correspond to the contents of the message length and message tag fields 34 and 35 to be directed to aDd latched by respective buffers 482(5) and 482(6).
The LEN ST length state signal is also directed to a data nit counter circuit (Fig. 9C-5~ as a LOAD DATA CTR load da~a counter signal. When the LOAD DATA CTR load data counter signal is asserted, it enables a four-bit bir ary word counter 536 to load the BUF SEL DATA (3:0) buffered selected data signals, which at this po~nt represent the contents of the message length field 34 of the data router message packet 30 being received. The asserted LOAD DATA CTR load data counter signal also energizes an OR gate 537, which, in turn, enables a three-bit flits-per-word binary counter 540 to load an initialization value.
As described above, the value iD the message length field 34 represents the number of thirty-t vo bit worts in the four-bit data flits 36 in the message packet 30. The initialization value loaded by flits-per-word counter 540 identifies the Dumber of data flits 36 in oach word enumerated by the message length field 34. S~nu the BUF SEL DATA (3:0) buffered selected data signals at successive ticks of the NODE CLK node clo~k signal, after representing the message tag field 35, represent she data flits 36, wunter 34 is initially disabled by the TAG ST tag state signal, ant then deemented in responso to successive ticks of the NODE CLK signal.
The flits-per-word counter 540 generates binarv-encoded FLIT/WRD CN-r (2:0~ flits-per-word count signals which are directed to a decoder 541. The decoder 541 generates DEC DATA FLIT
CNT (7:0) decoded data flit count signals, each of which is asserted in response to the associated binary value of the Fl,IT/WRD CNT (2:0) sigDals. The DEC Fl IT/WRD CNT (7:0) signals are used ih~ the geseration of the FLIT ~i" OF WORD si~als to enable the gates 481(i) alld buffers 482(i) to assemble the particular portions of a thirq-two bit data word. ID particular, each DEC ~Lrr/WRD CNT (i) signal is coupled to one input terminal of an AND gate 548(i) ("i~ is an integer between 7 and 0), all of which are enabled in parallel by the asserted DATA LOOP ST data loop state signal from the fGy-flop 533 (Flg. 9C4). While the DATA LOOP ST signal is asserted, as the DEC FLIT/WRD CNT (7) , . .. -. : . -., : .: : : ; .:: .: : : - :

WO 92/06436 PCr/US91/07383 20933~

through DF.C FLIT/WRD CNT (0) signals are sequentiaDy asserted, the AND gates 5~8(7) through ~48(0) are sequentially energized.
When AND gate 548(7) is energ~zed, it asserts a DATA FLIT 7 OF WOE~D signal to energize the OR gate 511, which, in turn, asserts the Fl.IT 7 OF WORD signal. WheD AND gate 487(6) is energized, it asserts a DATA FLIT 6 OF WORD signal to energize OR gate 535, which, in turn, asserts the FLIT 6 OF WORD signal. Similarly, when AND gate 548tS) is ener~zed, it asserts a DATA Fl.lT S OF WORD signal to energized OR gate 534, which, in turrl, asserts the FLIT S OF
WORD signal. In addition, each of the circuits 520(i) irlcludes an AND gate 548(i), which it energized when the corresponding DEC DATA FlIT CNT (i~ signal is asserted, enabling it to i~ert a DATA
FLlT ~i~ OF WORD signal. The asserted signal energizes the OR gate 522(i), which, in turn, asserts the FLlT "i'' OF WORD signal. Each of tbe Fl,IT ~i~ OF WORD signals enables the gates 481(i) and buffers 482(i~ to successively gate and latch BUF SEL DATA (3:0) buffered selected data signals representing successive data flits 36 to ~orm a thirty-two bit wosd, as described above.
The FLlT/WRD CNT (2:0) flits-per-word count signals are also directed to an inverter 542, which couples complemented signals to an AND 8ate 543. When the binary-encoded value of the FLIT/WRD CNT (2:0) signals is zero, all of the cornplemeDted signals are asserted and inverter 542 er ergizes the AND gate 543 to assert a FLIT/WRD CNT 0 flit-per-word count equals zero signal.
When the FLlT/WRD CNT 0 signal is asserted, the BUF SEL DATA (3:0) buffered selected data signals corresponds to the low-order nibble of a thirty-hvo bit data word. The asserted FLIT/WRD
CNT 0 sig~al euables one input terminal of an AND gate 544. If a WORD CNT 0 word count equals zero signal, which is generated in response to the cOrreDt value of the word counter 536, is not asserted, AND gate S44 is de-energized to negate the DATA CI-R EQ 0 data counter equal zero signal. As will be described in more detail below, when the WORD CNT 0 word count equals æro signal is asserted, the thirhy-two bit word currently being formed by the gates 481(i) and buffers 482(i) is the last in the data router message packet 30 beinB received.
The FLlT/WRD CNT 0 flits-per-word count equals zero signal is coupled to an input terminal of an AND gate 545. In addition, in resporse to the next ticlc of the NODE CLEC sigllal after asserdon of the FLlT/WRD CNT 0 signal, the AND gate 545 is energized to assert a WORD DOWN sigcal, which is used to form the asserted LAT WORD latch word and V~LID WORD signals. The asserted WORD DOWN signal energizes the OR gate 537, which, in turn, enables the flits-per-word counter 540 to reload. In addition, the asserted WORD DOWN signal enables the word counter 536 to decrement.
The word counter 536 generates binary-encoded WORD CNT (4:0) word courlt signals that identify the number of thirty-two bit data words that have been received. An inverter 546 couples complemented WORD CNT (4:0) signals to the input terminals cf an AND gate 547. If all ol the WORD CNT (4:0) signals are negated, which wiD occur when the BUF SEL DATA (3:0) buffered selected data signal represent the data flits 36 of the last thirty-two bit word in the data router message packet 30. When the FLIT/WRD CNT 0 flits-per-word count equals zero signal is also asserted, which " ` ` : . '. ", ' ,.' `" :'' ~ ' ' :

WO 92/06436 PCr/llS91/07383 2~933~ -73-will occur when BUF SEL DATA (3:0) signals represent the last data nit 36 of ~he last thirty-two bit word, the AND gate 544 is energized to assert the DATA CrR EQ 0 data counter equals zero signal.
Returning to Fig. 9C-4, the assertion of the DATA CrR EQ 0 data counler equals zero sigrlal enables inverter 538 to disable AND gate 539, which de-energizes OR gate 532 causing the DATA
LOOP signal to be negated. On the other hand, since the DATA LOOP ST data loop state signal is asserted, the assertion of the DATA CrR EQ 0 signal energizes an AND gate 550 to assert a CHECK
signal. In response to the next tick of the NODE CLK si~al, the negated DATA LOOP signal will enable the flip-flop 533 to ~e reset, negatirlg the DATA LOOP ST signaL and the flip-flop 551 to be set, asserting a CHECK ST signal. Since at this point, the DATA LOOP ST signal is negated, the AND gate 550 is disabled thereby negating the CHECK signal. The negated CHECK signal causes the flip-flop 551 to be reset in response to the next tick of the NODE CLK signal, thereby negating the CHECK ST signal. AccordiDgly, the CHECK ST signal is asserted for only one tick of the NODE
CL~C signal.
It t~ill be appreciated that the CHECK-ST signal is asserted when the BUF SEL DATA (3:0) signals represent the cbeck bits from the check genera~or 464 (Fig. 9C-2). The CHECK ST signal is coupled to OR gate 510 (Fig. 9C-S) which is energi~ed ~o assert the OTHER 7 signal. This signaL as described above, enables the OR gate 511 to assert the Fl,IT 7 OF WORD signaL enabling the gate 481(7) to direct the signals to the buffer 482(7), which latches them as described above.
The CHECK ST signal is also coupled to an AND gate 552. If the SEL DATA (4) (HEADER) signal is negated,.an inverter 553 enables one input terri inal of the AND gate 552. The assertion of the CHECK ST signal energees the AND gate, which, in turD, energizes the OR gate 503, enabling it to assert the IDLE signal. In response to the next tick of the NODE CLK signaL the flip-flop 500 is set to assert the IDLE ST idle state signal. At this point, the sequence of operations described above in connection with Figs. 9C4 and 9C-5 can be repeated.
If, on the other hand, the SEL DATA (4) (HEADER) signal is asserted, which can occur if the FIFO 451 is transmitting FIFO OUT (4:0) signals representing the first flit of a new data router message packet 30, at the same time the BUF SEL DATA (3:0) signals represen~ the last ilit of a current data router message packet 30, the inverter 553 maintains the AND gate 552 in a de-energized condition. If that occurs, the asserted CHECK ST signal energizes C)R gate 504, which enables one input terminal of AND gate 505. The asserted SEL DATA (4) (HEADER) signal energizes the AND
gate 505, enabling it to assert the HEADER siBnaL which sets fGp-flop 506 at the next tick of the NODE CLK signal, enabling the operations described above to be performed witbout requiring assertion of the IDLE ST signal. This allows the message assembler 452 to begiD processing ~e new message packet 30 one tic~ of the NODE CLK signal earlier than would be the case if the CHECK ST
signal only operated to enable setting of fli~flop 500.
Returning ~o hg. 9C-3, as described above, after the bus interface 4~3 has been ootified by the assertion of the VALID WORD signal that a new word is available for latcbing in the assembly synchronizing register 480, it may assert the U~.~ ASR write assembly synchroniz~ng register sig - ~ - . . , . , . - , .................. ,.. , ~ -.

WO 92/06436 PCl/US91/07383 20933~ ;

The asserted WRITE ASR signal enables the register q80 to latch the signals in the respective nibbles 480(i), in the beginning of message nag q86 and in the all-fall-down nag 48. The assembly syncluoDizing register 480 transmits the latched signals to the bus interface 453 as RCV WORD (34:0) signals, cornprising a include thuty-two bit data word as the RCV WORD (32:0) signals, a BOM LAT
latched beginn~ng-of-mesr,age signal as the RCV WORD (33) signal, aDd an AFD LAT latched aU-faU-down mode signal as the RCV WORD (34) signal.
Fig. gC-6 depicts a detailed logic diagram of the bus interface 453 in one embodiment of the data router interface 205. With reference to Fig. sc-6, the bus interface iDcludes t vo fust-in first-out buffers (FlFOs) 560 and 561, both of which are connected to receive selecsed portions of the RCV
WORD (34:0) received word signals from the assembly synchror~zing register 480, and a length store 562, which receives selected portions of the REV DATA (34:0) signals from a multiplexer 563.
FIFO 560 receives the length and tag portions, comprising the RCV WORD (28:22) signals from nibbles 480(5) and 4~0(6) respectively, and the AFD LAT all-faD-down latch sigoal from ll-faD-do vn flag 487 of the assembly syDchron~ziDg register 480. The FIFO 560 is used to control the contents of the received leDgtb field 253, received tag field 24~, and seceived all-fall-down mode field 254 of the status aDd private registers 293 and 294 of left interface register set 290, or of status and private registers 300 and 301 of right interface register set 291 (Fig. 9A-2B).
If (i) the FIFO 560 is not Q~serting a ST INF FIFO NR FULL status information buffer nearly full signal, which indicates whether the FIFO 560 is able to buffer additionl informasion, (ii) the FIFO
5S1 is not asserting a DA rNF FIFO NR FULL data information buffer nearly full signal, and (iii) a REC EN receive enable signal is asserted, a control circuit 564 g~nerates a WRT write signal to energize one input terminal of an AND gate 565. When the RCV WORD reoeive word signa]s represent the first word of a data router message packet 30, they have the length, tag and all-fall-down mode infor~nation. In that case, the BOM LAT latched beginning-of-message signal, which comprises the R.CV WORD (33) signal, is asserted, which energizes the second input ter~nisal of an AND gate 564, enabling it to assert a STA WRT EN status write eDable signal. WheD the STA WRT EN signal is asserted, the FIFO 560 latches the portion of the RCV DATA (34:0) signals representing the leng~h, tag and all-fall-down mode inforrnation at the next tick of the NODE CLK signal.The data first-in first-out FIFO 561 is used to buffer the RCV WORD (31:0) signals from the assembly synchronizing register 480. The FIFO 561 is used to control the contents of the receive registers 233, 295 and 302 of the middle, left and right register sets 230, 290 and 291 (Figs. 9-2A and 9~-2B).
The BOM LAT latched beginning-of-message signal and AFD LAT latched all-fall-down mode signal are used, along witb the WRT write enable signal from control circuit 564, to control buffering of tbe RCV WORD (32:0) received word signals, by FIFO 5S1. lf a normally-negated RCV
STOP receive stop signal is negated, a multiplexer So6 couples the RCV WORD (32:0) receive word signals to the data input tenninal of the FIFO 561. If the AFD LAT latched all-fall-down mode is negated indicating that the data router message packet 30 was not received vhile data router 15 was in 2~3~ -all-fall-down mode, the BOM LAT and WRT signals control storage of the RCV WORD (32:0) signals in the FIFO 561. ID that case, when the BOM LAT signal is asserted, which occurs when the RCV
WORD (32:0) receive word signals define the length, tag and all-fall-down mode ~Dformation, an inverter 578 m~uDtains a~ OR gate 567 in a de-energized condition, causing it to disable one input terminal of an AND gate 570.
The AND gate 570 remains disabled regardless of the assertion level of the WRT write enable signal, in turn disabling one tnput terminal of an OR gate 571. If a PEI BUS WRT DA FIFO nterface bus write data FIFO buffer signal is negated, which is the case unless the processor 200 is attempting to load the data words into the first-in first-out FIFO 561, the OR gate 571 remains disabled and negates a DA WRT EN data FIFO buffer write enable signal, iDhibhiDg the FIFO 561 from loading the RCV
WORD (32:0) signals.
For each succeeding word derived from the message packet 30, the BOM LAT latchedbeginning-of-message sigllal will be Degated. 11l that case, the inverter 568 will energize the OR gate 567, enabling it to enable one input terminal of AND gate 570. In response to the assertion of the WRT write enable signal from control circuit 564, the AND gate will be asserted, in turn energiang OR gate 571. The energized OR gate 571 asserts the DA WRT EN data write enable signals. While the DA WRT EN signal is assertcd, the data f~rst-in first-out FIFO 561 latches the next word of the message packet 30.
If the FIFO 561 becomes nearly full, it can assert the DA FlFO NR FI~LL nearly full sig~
which disables the control circ~ 564 from asserting the WRITE ASR write assembly synchroni7ing register signal, and inhibiting it from asserting ~he WRT signal in response to receipt of a VALID
WORD signal from the message assembler 452. ID addition, the control circuit 564 will assert the STALL signal to stall the message assembler 452 as described above.
It will be appreciated that the concurrent assertion of the 13OM I~T latched beginning-of-message signal and negatioll of the AFD LAT latched all-fall-down mode sigDal inhibits the data first-in first-out FIFO 561 from latching the furst word of a new message packet 30. This condition will occur when the message packet 30 is received while the data router 15 is not in all-faU-down mode, in which case the atdress informatioD, defned by the RCV WORD (31:28) and RCV WORD (19:0) receive word signals need not be retained. Further, the remaining information, namely, the length and tag information represented by the RCV WORD (27:20) receive word signals, as well as the AFD LAT
latched all-faD-dowD mode signal, are available ir, the FIFO 560 and the appropriate status register. By inhibiting the data first-in first-out FIFO 561 from latch~g the &st word assembled by the message assembler 452 from a data router message packet 30 that was received when the data router 15 is not in aD-fall-down mode, the number of words that the processor 200 must retrieve from the FIFO 561 for a par~icular message packet 30 is reduced. Otherwise stated, this enables the FIFO 561 to hold more words formed from data flits 36 of the received r~essage packets 30.
On the other hand, if the data router message packet 30 is being received while the data router 15 is in aD-fall-down mode, the first word of the RCV WORD (31:0) receive word signals is buffered in .

: : : . , . . , ,: : , ,: . , , . :. - . , . :: .: , : ,, : . .

WO 92/06436 PCl/lJS91/07383 2~33~

the FIFO 561. Enabling the word to be buffered in the nFo 561 permits the address irlfonnation, defined by the RCV WORD (31:28) and RCV WORD (19:0) signals to be retained, which will facilitate retransmission of the message packet 30 through the data router interface 205 as described above. In that case, the asserted AFD I~T latched all-fall-down mode signal enesgizes OR gate 567, which enables one input terminal of AND gate 570. The AND gate 570 is energized in response to the asserted WRT si~al as described above, which energizes OR gate 571 to assert the DA WRT EN data bufer write enable signal, also as desc~ibed above. It will be appreaated that the data FIFO 561 wiD
latch the RCV WORD (31:0) signals defining the first word of the data router message packet 30 at the same time the status FIFO 560 is latchirlg the RCV WORD (27:20) and RCV WORD (33) s~gnals defining the length, tag and aD-fall-down mode informadon for the same word. The data first-in first-out FIFO 561 buffers RCV WORD (32:0) signals defining succeeding words of th~ data router message packet 30 in the same manner as desibed above.
The contents of the length store 562 idendfy the nusnber of thirty-two bit data words of the data router message packet 30 remaining to be received and buffered in the FIFO 561. The asserted - -BOM LAT latched beginning-of-message signal also enables the multiplexer 563 to couple the portion of tbe RCV DATA (34:0) signals representing the length information to the input terminals of the length store 562. The asserted WRT write enable signal enables the length store 562 to latch the signals at its inpu~ terminals, which at this point identify the length of the data router message packet 30 being received.
The LEN CNT length count output signals from length store 562 are coupled to the input terminals of a decrementation circuit 572, which generates NXI LEN CNT next length count signals that have a binary-encoded value one less than the binary-encoded value of the LEN CNT length count signals. With successive words of the RCV WORD receive word signals, the BOM LAT latched beginning-of-message signal is asserted, enabling multiplexer 563 to couple the NXI 1 FN CN T next length count signals to the input terminal of length store 562, which latcbes them in response to the asserted WRT write enable signal. Since the successive assertions of the WRT write enable signal also control the loading of R(: V WORD (31:0) receive word signals defining successive words in the data router message packet being received, the length store is successively decremented for each word latched in the FlFO 561.
Further, since the value initially loaded into the length store 562 identifies ~he number of words in the data router message packet 30 being received, vhen ~he contents of the length store 562 go to zero, all of the data words will be in the P IFO 561. When that occurs, the LEN CNl length count signals are all negated, defning a binary-encoded value of zero. The LEN CNT leDgt~ count signals from length store 562 are coupled to a NOR gate 573, which generates an asserted I FN O
length equals zero signal when all of the LEN CNT signals are negated.
If the first-in first-out FIFO 560 is asserting the ST INF FIFO NE status information first-in first-out buffer not empty signal, it and the asserted LEN 0 length count equals zero signal energize an AND gate 57~, which, in turn, energizes an OR gate 575 to assert an NEW DR MSG new data router WO 92/06436 PCr/US91/07383 ;
209~3~ -n-message signal. The asserted NE DR MSG signal indicates that the data router interface 205 has received a new data router message packet 30 from the data router 15. The signal may be used to corldition the receive bit 241 of the corresponding status register 231, 293 or 300 tPigs. 9A-2A and 9A-2B), and may also enable the Detwork interface 202 to interrupt the processor 200, in turn enabling the processor 200 to retrieve the data router message packet 30.
In addition, the asserted NEW DR MSG new data router message signal enables one input terminal of an AND gate 576. When the other input terminal of AND gate 576 is also enabled by an EN RD ST FlFO enable read status first-in furst-out buffer signal, the AND gate 576 asserts a RD ST
FIFO read status first-in first-out buffer signal. The state of the EN RD ST FIFO signal is effeaively controlled by circuitry controlling the status register 231, 293 or 300, and ;s asserted by the mjector/ejector COmmoD control/status circuit 2æ when the processor 200 has finished retrieving a previously-received message packet 30 whose length, tag and all-fall-down mode were in the status register. At that point, the circuitry controlling the status register can enable the inforrnation from the FIFO 560 to be loaded into the appropriate fields of the status register, aDd so it asserts the EN RD ST
FIFO signal, enabling the AND gate to assert the RD ST FIFO signal.
Retrieving the contents of the data f~rst-in filst-out FIFO 561 is controlled by a READ DA
FIFO read data first-in first-out buffer signal. This signal is asserted by the ejector common control/status circuit 226 when the processor 200 is reading the respective receive register 233.
The bus interface 453 also includes several other facilities. First, it u~ill be appreciated that, at the point of a context switch operatioD the data first-in first-out FlFO Sol may contain data router message packet data. In the performing the context switch, it is Decessary to drain the contents of the data first-in first-out FlFO 561 so that data of message packets that were transferred iD one context are retrieved by the processor 200 and processed in that context, and not in some other context. The processor 200 can perform that operation by retrieving the data through the receive registers 260, 295 and 302. On the other hand, when the context is restored, it uill be necessary to load the data words that were in the FIFO 561 at the point of the context switch back into the FIFO 561. This will restore the context to the same condition as at the point of the context switch.
To accommodate that, the second input terrninal of multiplexer 566 is connected to receive data from the interface bus 211. In this con&tion, the RCV STOP receive stop signal is assested, enabling the multiplexer to couple the data from the bus 211 to the input terminal of the first-in first-out FIFO 561. In addition, the interface 212 asserts a PEI BUS WRT DA FIFO wnte data first- in first-out buffer signal that eDergizes OR gate 571 to assert the DA WRT EN data first-in first-out buffer write enable signal. This enables the first-in first-out FIFO 561 to load the data from the interface bus 211, enabling the processor 200 to restore the context to the point at which the context switch occurred.
It uill be appreciated that the processor 200 may need to load data into the first-in first-out buffer 562 for other reasons, as well. For example, the processor 200 may need to perform a loop-back test, in which it loads data into the FIFO 561, and reads it back to verify proper working of the data router interface 205. This facility permits it to do tbat.

" ~ ." "" - ~

WO 9~/0~436 PCI/US91/073B3 2~33rj~

Ano~her facility al]ows the processor 200 to control disabling of recéption of data router message packets 30 by the ejector port 225, while at the same time ensuring that the bus ejector port 225 receives a complete data router message packet even if the processor 200 disables reception while the port is rec~iving a message packet 30. As noted above, the processor 200 can disable the ejector port 225 from receiving data router message packets 30 by setting the receive stop bit 252 of the private register 233, or of corresponding bits of private registers 295 or 301 of the left and right register sets 2~0 and 291, respectively.
In particular, as noted above when the bus interface 453 has received aLI of the data words off a data router message packet 30, the contents of the length store 562 have the value zero. When that occurs, the LEN CNT length count signals are all negated to represent the binary-encoded value zero, causing the NOR gate 573 to assert the LEN 0 length equals zero signal. In addition to controlling retrievals from FIFO 560 through AND gate 574, the LEN 0 si~al enables one input terminal of an AND gate 577.
The second input terminal of AND gate 577 is coDtroLled by a REC STOP REQ receive stop - - - -request signal from a receive stop control circuit 58~, which is depicted in Fig. 9C-7. With reference to Fig. 9C-7, the receive stop control circuit 580 includes a flip-flop 581 which is set in response to an asserted SET RCV STOP REQ FF set receive stop request flip-flop signal from aD AND gate 582.
The AND gate 582 is energized to assert the SET RCV STOP REQ FF signal in resporLse to signaLs from the interface 212 (Fig. 8) that control setting of the receive stop bit of the respective private registers 233, 290 or 291.
The asserted REC STC)P REQ receive stop request signal energizes the second input terminal of AND gate 577. When the LEN 0 length equals zero sigDal is also asserted, the AND gate is energized to assert the REC STOP GIUNT receive s~op grant signal. An iDverter 586 &ables an AND gate 587, causing to negate the REC EN receive enable si~al to disable the control circuit 564 from asserting the WRT write enable signal, effectively disabling the bus interface 453 from receiving from the assembly synchror~ing register 480. Since the AND gate 577 is not energized until the LEN
O signal is asserted, the AND gate 587 is DOI disabled until all of the data words o~ a tata router message packet 30 that was being received when the SET RCV STOP REQ FF set receive stop request flip-flop signal was asserted9 have been loaded into the FIFO 561.
The REC STOP GRANT receive stop grant signal from AND gate 577 is also coupled to an input terminal of an AND gate 585 of the receive stop control circuit 580 (Fig. 9C-7). When the REC
STOP GRANT signals have been asserted by both the left and right ejector ports 225(1) and æ5(r), an AND gate 585 is energized to assert a SET REC STOP set receive stop signaL which reseh 9ip-9Op 581, which negates the REC STOP REQ receive stop request si~al, and sets a 9ip-flop 583 to assert a REC STOP receive s~op signal.
Returning to Fig. 9C-6, the negated REC STOP REQ receive stop request signal disables AND gate 577, causing it to negate the REC STOP GRANT receive stop grant signal. The inverter 586 complements this sigllal, thereby enabling one input terminal of AND gate 587. However, the : :: - . :. . ;: . .: . ... ,:; : .

,. . : .
. , , . , , :: . . ,, , ~ .:
: . .. ,,, ; .. ., :
:,, ' :- . . . ,,. ~ . ~ :.:: . , ,: :;::

WO 92/~6436 PCr/US~1/0738 20g33Si~ 79-asserted REC STOP receive stop signal from flip-flop 583 (Fig. 9C-7), through an inverter 590, causes tbe AND gate 587 to maintain the REC EN receive enable signal ;D a negated condition.
The negated REC STOP GRANT signal also disables AND gate 585, causing it to Degate the SET REC STOP set receive stop signal. The processor 200 can thereafter enable the respectiYe ejector port to resume receiving by clearing tbe receive stop bit of the respective private register. In doing so, and AND gate 584 is eDergized to assert a RESET REC STOP reset receive stop signal, cDabliDg the flip-flop 583 to be reset, thereby negating the REC STOP receive stop signal. ReturrnDg to Fig. 9C-o, the negated REC STOP receive stop signal is complemented by inverter 590 to enable the second input terminal of AND gate 587, thereby energizing it to assert the RFC EN receive enable signal.
Thereafter, the control circuit 564 caD resume assertiDg WRITE ASR write assembly syDchroDiziDg register signal and the WRT EN write eDable signal to enable receipt of data router message packets 30 to be resumed.
As with the description of the ~nessage injector port 223 as ~loted above, ;D the preceding description of the message ejector port 225, all of tbe circuits operate in respol~se to the NODE CLK
signal provided by the clock buffer 207, whereas it may be desirable to have most of the processing elemen~ 11, including most of the net vork irlterface 202, to operate irl response to a processiDg element clock signal (not sho vn) which synchronizes most operations on the processing elemenl I1, and only srnall portions of the network interface 202 operate in response to the NODE CLK signal. ID that situation, in one embodimeDt the flit receiver stage 450, ejector FIFO 451 and ejector message assembler 4S2 operate i~ response to the NODE CLK signal, whereas the bus interface circuit 453 operates in response to the PE CLK signal. In tha~ er~bodiment, the VALID WORD signal provided by the bus interface 453 is not coupled directly to the ejector message assembler 452, but instead is sync~ror~ized through a sy~chronizer circuit similar to the syrlchrorizer circuit 444 depicted jD Fig. 9B-9.

:: : , :,. ~ : ., : : . . , WO 92tO6436 PCI'/US91/07383 209335~
.8 iv. Status/Control Circn~try Figs. 9D-1 through 9D-7 depict details of circuitry controlling several fields of the status registers 231, 293 and 300 (Figs. 9A-2A and 9A-2B), and private registers 232, 294 and 301, as well as the message count register 313. Figs. 9D-1 through 9D-S depicts details of circuitry controlling loading of the staius and private registers in response to receipt of data router message pacl~ets 30 from the data router 15 and in response to loading by the processor 200 (Fig. 8) over the interface bus 211. It will be appreuated that loading of the respective re8~sters under cor~trol of the processor 20() enables the processor 200 to initialize them, and also to establish co~text in response to a context switch. Flg.
9D-1 depicts a detailed block diagram, and Figs. 9D-2 through 9D-5 depicts detailed circuitry, for controlling loading of, and reading from, the ~middle~ registers, irl a manner transparent to the processor 200.
With reference to Fig. 9D-1, the ILEFT] MSG STAT message status signals for a message received by the left message ejector port 225(1) are coupled from the status inforrnation FIFO 560 over a bus 600. In response to [LEFTl NEW STATUS signals from the bus interhce 453, indicating receipt of a new data router message packet 30 to be retrieved by the processor 200, the [LEFT] MSG STAT
signals are latched in various fields of the status register 293 and private register 294. In particular, a field comprising MSG [LEFT] STAT [REC LENGTHl signals, indicating the length of the received message 30, are latched by a latch 601, which provides the receive lerlgth field of the status reg~ster 293.
In addition, the [LEF'Il MSG STAT lREC LlENGTH] signals are coupled through a selection and decrementation circuit 602 and latched in a latch 603, which provides the receive length remaining field of the status register 293. Similarly, the [l FE;Tl MSG STAT [TAG~ sigDals, sepresentative of the tag field of the received message packet 30, are latched ~n a latch 604, which provides the receive tag field of the status register 294. FinaUy, the [LEFrl MSG STATj[AFD] signals, which indicate whether the message packet 30 is received while the data router 15 is in a31-fall-down mode and tbat the message packet 30 has not arrived at its destination leaf 21, are latched in a latch 605, which pro ~ides the received all-fall-down flag of the private register 294. A similar bus and set of latches (not sh~wn) are provided for the right port ejector 225(r).
The latches 601, 603, 604 and 605 can also be loaded from the interface b~s 211. In particular, the appropriate ones of the data lines carrying signals, identified on Fig. 9D-1 as the PEI BUS [LENl, PEI BUS [TAG], and PEI BUS [AFD], respective3y, are connected to the latches 601, 603, and 605, respecti~ely. In additio~, ~he data lines carrying signals identified as PEI BUS lLEN REM3 are connected to the selection and deernentation circuit 602. In any case, the respective signals can be latched by the various latches in response to ~vrite enable signals, generally identified as PEI BUS
WRT LDR REG write leh data router registers signals from the interface 212 ~der control of the processor 200. The interface 212, under control of the processor 200, can also enable tbe contents of the va~ious latches to be coupled onto the bus 211 in response to corresponding PEI BUS RD LDR
REG register read enabling signals, The selection and decrementation circuit 602 performs several functions. When either the [LEFT] MSG STAT signals or signals from bus 211 are being loaded into latch 603, it selectively . , . : , . : . :
.. . :, .:; : . : : :
. .. : . . , .: :, : . :, , , :
;: : ,: , , ~ ;: : - :
~:: , ,: : . :: . : , : : : ,::
- ; . ", :, : : . :: :

WO 92/0643~ PCr/US91/07383 2~933~
couples the signals from one or the other to the latch 603. ID addition, when the processor 200 is readirlg from the receive register 2959 it decrernents the binary-encoded value represented by the signals latched ;D the latch 603 and enables the result to be latched therein, to represent the length of the message packet 30 remainiDg to be read. In addition, if the [LEFTl MSG STAT IAFD] aU-faU-dowD signal is asserted9 when the latch 603 is initiaUy loaded, it incremerlts the value of the MSG STAT
ILENGl~] signals before coupling them to the latch 603. It wiD be appreciated that, if the leaf 21 is an intermediate destination for a data rouler message packet 30 while tbe data router 15 is in all-faD-down mode, the length of the packet 30 is actuaDy the length as represented by the contents of the message length field 34, which, in turn9 is represented by the binary-eDcoded value of the [LEFl'~ MSG
STAT [LENGTH~ sigDals, incremented by one. This accommodates the additional word constituting the header 40, down path identification portion 41, and the message length and tag ficlds 34 and 35, whish are retained by the port ejeaor æ5(1).
When the [LE}: Il MSG STAT message status signals are loaded into the latches 601, 604 and 605, they are also loaded into a-shadow register 610. In particular, the [LEFTl MSG STAT [AFD]
signal is loaded into a latch 611(1), the lLEFT] MSG STAT ILENGTH~ signals are loaded in~o a latch 612(1), and the [LEFTl MSG STAT [TAG~ signals are loaded into a latch 613(1). Irl additioD, a valid flag 614(1) is conditioned to indicate the status of the contents of the l~tches 611(1) through 613(1) in the shadow register 610(1). The valid flag 614(1), when set~ asserts a [LEE:l l VALID signal. A shadow register 610(r) is provided, ha~ring similar latches 611(r) through 613(r) ar d a similar valid nag 614(r) for the ~IG~j MSG STATUS signals from the right ejector port 225(r). When the latches 601, 604 and 605 are loaded from the interface bus 211, the corresponding latches of the shadow register 610(1 are also loaded therefrom.
The latched signals output from both shadow registers 610(1) and 610(r) are coupled to respective data input terrninal of a multiplexer 614, whose data output termir al is connected to latches providing the received leng~h field 243 and the received tag field 245 of the status register 231, the received all-fall-down field 254 of the private register 22, and to a selection and decrementation circlL t 615. The selection and decremeDtatioD circuit 615, like circuit 602, coDtrols selection and decre~nentadon of sigDals coupled to a latch 244, which provides the length remair~ing field 244 of the status register 231. The latches 243, 244, 245 and 254 are loaded in response to a [DR~ NEW STATUS
signal, which are generated in response to the ILEFrl NEW STATUS and [LEFI-l VALID signals, as weD as the ~IGHT] NEW STATUS and IRIGHTl VALID sigDals.
The multiplexer 614 is controlled by an EJ PORT ~rR lLEFTl ejector port pointer lleft]
signal generated by control circuitry described below in connection with Fig. 9D-6. When the EJ
PORT PTR ILEFT] signal is asserted, if the processor 200 reads the middle receive register 260 (l~Ig.
3A-2A) the left ejeaor por~ 225(1j provides the data. On the other hand, when the EJ PORT ~R
ILEFl'J signal is Degated, if the processor 200 reads the middle receive register 260 (Fig. 9A-2A) the right ejector port 225(r) pro~des the data. Accordingly, if ~he EJ PORT PTR [LEFT] signal is asserted, the multiplexer 614 will couple the contents of the left shadow register 610(1) to latches 243, :: .. . : .. : ::,, ,,.. .,.. . .. : . ..
. ~ .:, ~ : . . . . ~. ., . ., :
.. : . , ., ': ' ' ' . :. ' :': '', ' ', ~f WO 92/06436 PCl/US91/07383 -82- 20~33~
244, 245 and 254 for loading if the processor 200 reads from the middle receive register 260.
Otherwise, the multiplexer 614 will couple the contents of the right shadow register 610(r) thereto.
The contents of the latches 243, 244, 245 and 254 can also be read oDto, and provided by, respective lines of the bus 211 in the same manner as 3atches 601, 603, 604 and 605, as described above.
The interface 212, under control of the processor 200, may provide signals, generally ide~tified as PEI
BUS WRT DR REG and PEI BUS RD DR REG signals, to control writing to and reading from the respective latches.
Fig. 9D-2 depicts circuitry for controlling coupling of respective signals for storage in the latches 601, 604 and 605, and the selection and decrementation circuit 602 for controlling coupling of signals for storage in the latch 603. With refereDce to Fig. 9D-2, the lLE~ ~SG STAT IAFD] signal, from the left ejector port 225(1), is coupled to one data input terminal of a multiplexer 620(1). The PEI
BUS IAFD~ signal is also coupled to the other data input terrninal of multiplexer 620(1). If the ILEFT
NEW STATUS signal is asserted, the multiplexer 620(1) couples the ILEFl-] MSG STAT IAFD] sig~
from the status information FIFO 560 as a [LEFT] SEL [AFD] left selected all-f~ll-down signal, to the data input terminal of latch 605. the assertion of the [LEFT] NEW STATUS sig~al also energ~zes and OR gate 621(1) to assert a lLE~ LD [AFD] leh load all-fall-down signal, which clocks the latch 605, enabling it to latch the ILEFTl SEL lAFD] signal.
Cn the other ~and, if the ILEFl'] NEW STATUS signal is negated, the multiplexer 620(1) couples the PEI BUS IA~;D] signal to the data input termiual of latch 605. If a PEi BUS WRT LDR
PRVT REG wri~e left data router private register signal is asserted, the OR gate 621~1) is also energized to assert the [LEF'Il LD IAFI)] signal.
Similarly, the latches 604 and 601 are associated with multiplexers 622(1) and 623(1), respectively. Multiplexer 6æ(1) receives, at its data input terminals, the PEI BUS ITAG~ and the LE~I'l MSG STAT lTAG] signals, and, under control of the [LEE;Tl NEW STATVS sigDal, couples one of them as the ILEFl'] SEL ITAGl selected tag si~ nals to the data input terminal of latch ~4.
Multiplexer 623(1) receives at its data input terrninals, the PEI BUS ILENl and the ILEFI l MSG
STAT [LEN3 signa~;, and, under control of the ILEE;Tl NEW STATUS signal, couples one of them as the ILE~ SEL ILENI selected tag signals to the data input termillal of latch 601. The assertion of the [LEFT] NEW STATUS signal also energizes an OR gate 624(1) to assert a ILEFT] LD
lTAG/LEN] left load tag and length signal, which enables the latches 604 to latch the respective [LEFI] SEL ll'AG] and ILEFTl SEL lLENl signals. If the p~FTl NEW STATUS signal is negated, the multiplexers 622(1) and 623(1) couple the PEI BUS ITAGl and PEI BUS ~ signals to the data input terminals of latches 604 and 601. If a PE1 BUS WRT LDR STA REG vrite left data router status register signal is asserted, the OR gate 624(1) is also energized to assert the ILEFT] LD
iTAG/~ PNj signal.
As noted above, the contents of latches 601, 604 and 605 can also be coupled onto the interface bus 211. Associated with each latch is a gated driver 625(1), 626(1) and 627(1), which, when energized by a PF,I BUS RD LDR STA REG read left data router status register signal, or a PEI BUS RD LDR

, " . ., . ,., ,.-.,/ . . ~ ", , , . ,,, ,, ,, . ~

2 ~ ~ 3 3 t~ 5 -83-PRVT REG read left data router private register signal, couples the contents of the respective latches onto respective lines of the in~erface bus 211.
The selection and decrementation circl~it 602 includes gated drivers 630(1), 631(1) and 632(1) which receives signals from the !hree sources from which the length remaining latch 603 may be loaded. Gated driver 630(1) enables the length information &om the interface bus 211 to be loaded into the latch 603. Gated driver 630(1), under control of an asserted PEI BUS WRT LDR STA REG
signal, gates the PEI BUS ILEN] signals to an OR arcuit 624(1). If the PEI BUS WRT LDR STA
REG signal is asserted, the OR circuit 634(1) couples the PEI BUS [LENl signal as [LFFll SEL NXT
LEN REM left selected next length remaining s~gnal. The asserted PEI BUS WRT LDR STA REG
signal also energizes an OR gate 635(1) to assert a ILEFTl LD [LEN REM] left load length remair' ng latch signal, which enables the latch 603 to latch the [LEE;T] SEL NXT LEN REM sigDals.
The leDgth remaining information from the left message ejector port 225(1) is coupled through gated driver 631(1). The [LEFT] MSG STAT lLENl sigDals are coupled to one data input terrninal of a multiplexer 636(1). The signals are also coupled tb an incrementation circuit 637(1), which provides signals to tbe other data input ter~ninal of multiplexer 636(1) whose binary-encoded value is one greater than the binary-encoded Yalue of the lLEFT] MSG STAT IlLEN] signals. The incrementation provided by the circuit 637(1) accommodates the additional message ler~th of a data router message packet 30 received while the data router 15 is irl all-fall-down rnode and the leaf 21 is not the destination. The multiplexer 636(1) is controlled by the [LEFT] MSG STAT [AFDj s~gnal, to couple one of the signals at its input terminal to the gated driver 631(1). If the [LEFT] MSG STAT lAFD]
signal is negated, the multiplexer 636(1) couples the ILEFT~ MSG STAT ILEN] signals to the gated driver, and if the [LEFT] MSt; STAT IAFD] signal is asserted it couples the incremented signals thereto.
If the lLEFT] NEW STATUS signal is asserted, it enables the gated driver 631(1) to couple the signals selected by multiplexer 636(1) to OR circuit 634(1), which, in turn, couples them as the [LEFl'~ SEL NXI LEN REM signals to the latch 603. The asserted [LEFT] NEW STATUS signal also energues OR gate 635(1) to assert the lLEFT] LD lLEN REM~ signal, enabling the latch 603 to latch the [LEFT~ SEL NXI' LEN REM signals.
The [LEFT~ LAT lLEN REM~ left latched length remaining signals from latch 603 are coupled to a decrementation circuit 640(1), which generates DECR LEN REM decremented length remaining signals whose binary-encoded value is one less thaD the biDary-encoded value of the ILEFl']
LAT lLEN REM~ signals. The DECR LEN REM signals are coupled to gated driver 632(1). If the LEI~ NEW STATUS and PEI BUS WRT L~R STA REG sigDals are negated, and if a ZERO
LEFT signal is also negated, the gated driver 632(1) couples the DECR LEN REM signals to the OR
circuit 634(1), which couples them as the lLEFT] SEL NXI' LEN REM sigl~als to latch 603. When the PEI BUS RD LDR REC REG signal is asserted, indicating that the processor 200 is enabling the left ejector port's rece;ve register 295 ~o be read, the OR gate 635(1) is energizsd to assert the [LEFT] LI) [LEN REM] signal. The asserted [LEFT] LD lLEN REM] signal enables the latch 603 to latch the :- ; ., ,, .. . . ~

WO 92t06~136 PCr/US91/07383 .~ 2~933~
[IEFTl SEL NXI' LEN REM signals. Accordingly, gated driver 632(1) enables the contents of latch 603 to be decremented when the processor 200 retrieves the data router message packet 30 from the left receive register 295.
As with the other latches 601, 604, and 605, the contents of latch 603 can be retrieved under control of the processor 200. If the PEI BUS RD LDR STA REG signal is asserted, a gated driver 641(1) couples the ILEFT] LAT [LEN REM] signals onto the interface bus 211.
As noted above, gated driver 632(1) is also controlled by a ZERO LEFT signal. The ZERO
LEFT signal is generated by an AND gate 642(1), which is energi ~ed by the coincidence of the PEI ~-BUS RD DR STA REG and the EJ PORT PTR LEFT signals. When the ZERO LEFT signal isasserted, driver 632(1) is disabled, effectively negadng the signals coupled to OR circuit 634(1). The processing element interface 212 asserts the PEI BIJS RD DR STA REG when the processor 200 enables retrieval of the co~tents of the middle status register 231. With the coincident assertion of the EJ PORT PTR LE~T signal, the contents of the middle status registPr 231 is provided by the left status register 293. When the processor subsequently retrieves the data router message packet 30 by -retrievals from the middle receive register 233, the data router message packet 30 will be provided by the left receive ~egister 295. Since the data router message packet 30 whose length is represented by the contents of the latch 603 will be retrieved through the middle receive register 233, the latch o03 is zeroed, by the assertion of the ZERO LEFT signal, to prevent the processor 200 f~om also at~empting to retrieve the same ta~a router message packet 30 by retrieval requests directed to the left receive register 295.
Fig. 9D-3 depicts the left shadow register 610~1) and circuitry for coatrolling contents thereo As ~,vith latches 605, 604 and 601 (Fig. 9D-2), the data input termiDals of latches 613tl), 612tl) and 611(1) receive signals from multiplexers, identified by reference numerals 650(1), 651(1~ and 651(1), respectively. Under cont~ol of the PEI BUS WRT LDR STA REG signal from the processor element interface 2127 the multiplexer 650(1) selecdvely couples either the p,EF'rl MSG ST~T rrAG] sigllals or the PEI BUS [TAGl signals as ILEFI'I SEL SHAD TAG left selected shadow tag signals to the data input terminal of the latch 613(1) of shadow register 610(1). Similarly, the muldplexer 651~1) selectively couples either the ILEF'I'] MSG STAT ILENl signals or the PEI BUS ILEN] signals to the data input terminal of the latch 612(1) of shadow register 610(1). If either the PEI BUS WRT LDR STA REG
signal or the IIEFI 1 NEW STATUS sigl~al is asserted, an OR gate 653 is el~ergized to assert a [LEFT]
LD SHAD STA REG left load shadow status register signal, which enables the respective latches 612(1) and 613(1) to latch the signals at their data input termiuals.
Sir~ilarly, under control of the PEI BUS WRT LDR PRVT REG sig~al from the processor elemeut interface 212, the multiplexer 650(1) selectively couples ei~er the pEF T] MSG STAT ~
signal or ~he PEI BUS ~ signal as ILEFT] SEL SHAD AFD left selected shadow all-fall-down signal to the data input terminal of the latch 611(1) of shadow register 610(1). If either the PEI BUS
WRT LDR PRVT REG signal or the ~ NEW STATUS signal is asserted, au OR gate 654 is energized to assert a lLEFTl LD SHAD PRVT REG left load shadow private register siylal, which enables latch 611(1) to latch the signals at its data iuput termiual.

.:: . ,, , :, . . ~ . ... . .. . .
:: - .. : . , ,: ~,:, . - ,.

.. ., ........ . .... . : , . . ... ,~
: :. ,. .: , : . :.

2 ~ ~i3 ~-~so depicts circnitry for controlling left valid flag 614(1), which generates the LEFT
VALID signal. In particnlar, a multiplexer 660 receives a ILEFT] NXF VAL left next valid signal from an OR gate 661 and a WRT LDR STA VALID write left status register/valid signal from a mllltiplexer 662. The OR 8ate 661 asserts the ILEFTl NXT VAL si~al under two geDeral circumstances. In particular, if, when the left message ejector port æ5(1) is asserting the [LEPTl NEW STATUS signal indjcating receipt of a data router message packet 30 to be retrieved by the processor 200, the ILEFT]
VALID signal is negated by a reset valid flag 614{1), a VAL NEW STAT valid because of ne v status signal is asserted. The asserted VAL NEW STAT signal energizes OR gate 661 to assert the ~LEFT
NXI VAL signal.
In addition, if, while the EJ PORT PI R LEFT signal is asserted, the processor 200 initiates a retrieval operation through the middle receive register 233, which enables the processos element interface 212 to assert the PEI BUS RD DR REC REG read middle receive register sig~aL an AND
gate 664 is energized to assert a l:~R READ FRM LEFT signal. The asserdon of either the DR READ
FRM LEFT signal or the PEI BUS RD LDR REC REG, which indicates that the processor 200 is inidadng a retrieval operatios~ through the left receive register 295, enerE~izes the OR gate 665 to assert a LEFT RD signal. If the [I~FTl VALID signal is theD negated, an AND gate 666 is eDergized to assert a VAL RD LEFT valid on read from left signal, which also energized OR gate 661 to assert the p~ lrJ NXI' VAL left next valid signal. If the processor element interface 212 is not asserting the PEI
BUS WRT LDR STA REG, which occurs when the processor 200 is enabling the left status register 293 to be loaded, the multiplexer 660 couples the ILEFT] NXT VAL left next valid signal to the data input terrninal of valid flag 614(1), to condition the flag 614(1).
On the otber hand, if the PEI BUS WRT LDR STA REG signal is asserted, indicating tbat t~e processor 20a is enabling loadiDg of the left status register 2g3, tbe multiplexer 660 couples a WRT
LDR STA VALID write left status register valid signal to condition valid flag 614(1). The WRT LDR
STA VALID signal is provided by multiplexer 662. In particular, if the aU-fall-down latch 605 (Fig. 9D-2) is negating the lLEFT] LAT [AFDl signal, multiplexer 662 cs~uples an output signal from a comparator o70, which generates an asserted signal if tbe PEI BUS ~ signals and the PEI BUS
II~N REMl signals have the sa~ne binary-encoded value, as the WRT Ll)R ST~ VALID signal. On the other hand, if the all-faU-down latch 605 h asserting the lLEFI l LAT lAFD] signal, multiplexer 662 couples, as the WRT LDR STA YALID signals, an output signal from a comparator 671, which generates an asserted signal if the PEI BUS [LEN REMl signals have the same binary-ellcoded value as that of the PEI BUS lLENl signals, as incremeDted by oDe by an incrementation circuit 672.
T~e comparators 670 and 671 enable conditioning of the valid Qag 614(1) usiDg the contents of the receive length and receive length remainiDg fields of the left status reg~ster 293 as it is being loaded by the processor 200, which wiU occur, for exarnple, during a context switch operation. In particular, if the leDgth (as adjusted by the incrementatio~ c~Jcuit 67? if the [L~FT] LAT lAFDl signal ~s asserted) and length remaining values are the same, no part of the data router message packet 30 whose status is indicated by the length and length remaining fields in the status register wiU not have been retneved, :: , , .: . :, ,. : ::, ,: . , : , . . . . .
'- :: :'''::: . :,...... . .. . .
. . ~ . ~ ,,,. , .. : , . ,: , ... . . . . ..

-86- 20933~
and so the processor 200 could enable retrieval of the message packet 30 through either the left receive register 295 or the middle seceive register 233.
On the other hand, if the lengths indicated by the values of the length (as adjus~ed) and length remaining fields differ, the processor 200 wiU have partially retrieved the data router message packet 30 through the left receive register 295, and it wiU resume retrieval therethrough after the context has been switched. In that case, the WRT LDR STA VALID signal will be negated, which will enable the multiplexer 660 to, in turn, enable the valid flag 614(1) to be clear.
The ILEFT] VALID and [RIGHT] VALID signals are used by the circuitry for controUing the middle status and private registers 231 and 232, which is depicted in Figs. 9D-4 and 9D-5. Much of the circuitry depicted on Figs. 9D~ and 9D-5 is similar to that described above in cormection with Fig. 9D-3, and will not be described herein. In parlicular, multiplexers and gates for controUiDg the loading of the respective latches 254, 245, 243 and 244, where sirnilar, is identified by reference numerals having the tag ~(M)'` instead of ~ , and such circuitrv operates in substantiaUy the sarne way. Fig. 9D4 also inctudes circuitry for generating a [MIDl NEW STATUS signal. In addition, the circuitry depicted on -Figs. 9D-4 and 9D-5 includes multiplexers 6&0 through 684 which, undes control of the EJ PORT PTR
LEFT signal, selecting between the left and right shadow registers 610(1) and 610(r) for providing signals to the respectdve multiplexers 620(M), 622(M), 623(m) and 636(m), substituting for the ILEFT]
MSG STAT signals in the circuit depicted in Fig. 9D-2. The EJ PORT PrR LEFT signal, when asserted, enables retrievals initiated by the processor 200 ~rom the middle receive register 233 to be provided by the left message ejector port 22~(1). On the other ha~d, when the EJ PORT E~R L~T
signal is negated, the retrieYals initiated by the processor from the middle receive register 233 are provided by the right message ejector port 225(r).
The ~ NEW STATUS silçnal is generated by a muldplexer 690 and AND gates 691 and 692. If the [LEFI'l NEW STATUS signal is asserted, and if the valid flag 614(1) is asserting the lLE}:Tl VALID signal, ~ND gate 691 is energized to assert a lMlD] NEW lLEFT] middle new status frorn left signal. If the El PORT Prla LEFT ejector port pointer left signal is asserted the multiplexer 690 couples the [MID] NEW [LEFT] signai as the lMlD] NEW STATUS signal. If either the ll EFT]
NEW STATUS or the ~LE~I'l VAIID signals are negated when the EJ PORT l~R LEFT signal is asserted, the IMID] NEW STATUS signal will be negated.
Similarly, if the IRIGHTl NEW STATUS signal is-asserted, and if the valid flag 614(R) is assetting the IRIGHl'~ YALlD signal, AND gate 692 is energized to assert a IMID] NEW IRT] middle new sta~us from right signal. If the EJ PORT PTR LEFT ejector port pointer left signal is neeated, to identify the left message ejector port X~(r), the multiple~er 6g0 couples the IMID] NEW [RTl sigGal as the lMID] NEW STATUS signal. If either the IRIGHl'l NEW STATUS or the IRIG~I 3 VALID
sigDals are negated when the EJ PO~T PrR LEFT signal is negated, ~he IMI~] NEW STAllJS signal will also be negated.
Fg. 9D-6 depicts circuitry for controlling the condition of tbe EJ PORT PI~ LEPT signal.
Generally9 the EJ PORT Pl R LEFT signal is controlled in response to the receipt by the le~t aod right , , . , ,, ~:; , ,. : , : , WO 92/06436 PCr/US91 /07383 ~ ~ Q ~ . 87-messa~ge~Je~)e~o~ ~?orts 225(1) aDd 225(r) of a data router message packet 30 to be retrieved by the processor 200, by the processor 200 enabliDg retrieval of a data router messa3e packet 30 through either the left or right receive registers 295 al~d 303, and by the condition of ~he EJ PORT ~R LEFT
signal. In particular7 if, for example, the left message ejector port 225(1) has a new data router message packet 30 for retrieval by the processor 200, and the right message ejector port 225(r) does not, arld if further the EJ PORT PTR LEFT signal is asserted, the circuit enables the EJ PORT PrR LEFT to remain asserted. On the other hand, if both message ejector ports 225(1) and 225(r) have Dew data router message packets 30 for retrieval by the processor 200, the EJ PORT PTR LEFT is enabled to toggle.
Retrieval requests from the processor 200 aLso can change the condition of the ET PORT ~R
LEFT signal. In particular, if the processor enables a retrieval operation from the left receive register 295, while the EJ PORT PTR L FT signal is asserted, the circuit negates the EJ PORT PTR LEFT
sig~al. On the other hand, if the processor 200 enables a retrieval operation from the right receive register 302 while the EJ PaRT PIR LEPT signal is negated, the circuit asserts the EJ PORT PTR
LEFT signal.
More specifically, with reference to Fig. 9D-6, if at least one of the [LElFTl MSG STAT lLENl signals is asserted, an OR gate 700(1) is energized to assert a RCV MSG LEFT received message left signal. This ~ill occur if the ILEFT] MSG STAT [LENl signals from the left message ejector port 225(1) i~dicate receipt of a new data router message packet 30. If an OR gate 700(r), which receives the [RIG~I'l MSG STAT lLENl signals, generates a negated RCV MSG RT signaL indicating that the right message ejector port 225(r) does not ha~e a new data router messa~e packet 30, an AND gate 701(1) is energized, to assert a }~ LEFT REQ point left request. If the EJ PORT PTR LEFT signal is assereed, and if a periodically-asser~ed UPDATE PTR EN update ~pointer cnable sigal is also asserted, an AND gate 702(1) is energized to assert a TO LEFT signal. The TO LEFT signal, in turn, energizes an OR gate 703 to assert an EN NEXr enable next signal wnich is coupled to the data input terrninal of a flip-flop 707. The asserted UPDATE PrR EN sigual alss~ eDables an OR gate 708 to assert an UPDATE PTR update pointer signaL which enables the fli~flop 707 to be set in res~ponse to the asserted EN NEXT signal, thereby asserting the EJ PORT PTR LE~T signal.
Si~nilarly, if at least one the lRlGHT] MSC; STA ILEN] signal is asserted, and the ILEFT]
MSG STAT ILEN] signals are negated, an AND gate 701(R) asserts a PT RT REQ point right request. If the EJ PORT PTR LEFT signal is negated, AND gate 702(r) will be energ~zed to assert a TO RIGHl` signal, which disables OR gate 703 to negate the EN NEXr signal. When the IJPDATE
Pl'R signal is asserted, the flip-flop 707 will be reset to negate the EJ PORT PTR LEFT signal.
On the other hand, if both RCV MSG LEFI' and RCV MSG RT signals are asserted, indicating both left and right message ejector ports 225(1) and 225(r) have received data router message packets 30 for retrieval by the processor 200, an AND gate 704 asserts a BOTH REI:2 both request signal, which is coupled to one input terminal of an XOR exclusive OR gate 705. If the EJ
PORT PIR LEFT signal is asserted, the XOR gate 705 negates a CH IrrR REQ change pointer . . '.' ': .: :' , ;, , ' ',: ' .': . ' ' : . .; `: '''' ,. ' "~
:': ' ' ~ . ' ', ' ~: ' ' ' , ' ;. . ' ; . ' , , WO 92/06436 P~/US91/07383 2~3~
88.
request signaL which disables an A~D gate 706 to negate a TO NEXT signal. The nega~ed TO NEXI
signal disables OR gate 703 to negate the EN NEXT signal, which, in turn, enables the flip-flop 707 to be reset when the UPDATE PTR update pointer sigllal is next asserted. On the other hand, if the EJ
PORT PTR LEFT signal is negated when the BOTH REQ both requested signal is asserted, AND
gate 706 is energized to assert the TO NEXT signaL which enables OR gate 703. Accordingly, flip-flop 707 will be energized in response to the assertion of the UPDATE Pl'R signal.
As noted above, the EJ PORT PTR LEFT signal can be controlled in response to requests from the processor 200 initiating retrievals from the left and right receive registers 295 and 304. In particular, if the processing element interface 212 is asserting the PEI BUS RD LDR REC REG
signal, indicating that the processor 200 is requesling a retrieval fsom the left receive register 295, and if the EJ PORT PTR LEFT signal is asserted, an AND gate 710 is energized to assert a TO
RIGHT/FROM LDR REC REQ signal, which is complemented to disable the OR gate 703. The disabled OR gate 703 provides a negated EN NEXT signaL which resets the flip-flop 707 in response to the asserdon of the UPDATE }~I R signaL thereby negating the EJ POlRT PrR LEFT signal.
On the o~her hand, if the processing element interface 212 is asserting the PEI BUS RD RDR
REC REG signal, indicating tbat the processor 200 is requesting a rctrieval from the left receive register 295, arld if the EJ PORT PTR LEFT signal is negated, an AND gate 711 is energized to assert a TO LE~T/FROM RDR REC REQ signal, which is complemented to enable the OR gate 703. The enabled OR gate 703 provides an asserted EN NEXI signal, which sets the fli~flop 707 in response to the assertion of the UPDATE PrR sigllaL thereby asserting the El PORT Pll? LEFT signal.
Fg. 9D-7 depicts the message count register 313 (Fg. 9A-2B) a~d circl~itry for controlli~g the register 313. As noted above, and as will be described in detail below, the coDtents of the message count register 313 are incremented in response to the transmission by the le~f 21 of data router message packets 30, and decremented in response to the receipt thereo In addition, CllR MSG CNT
current message count sigllals, which represent the value contained in the message count register 313, are used in coDtrol network message packets 60 to determine when the data router 15 is empty.
With reference to Fig. 9D 7, the message count register 3~3 may be wr~tten with an initial value by the processor 200 through a rnultiplexer 720. In that operation, the processor 200 provides the initial value to the processor element interface 212, and enables it to transfer the value as PEI BUS
IMSG CNTl message CouDt signals and assert a PEI BUS WRT MSG C~T REG ~vrite message count register enable signal. In response to the PEI BUS WRT MSG CNT REG signal, the multiplexer 720 couples the PEI BUS lMSG CNTl signals as NXI' MSG CNT next message count s~als to data input terminals of the message count re~ster. The asserted PEI BUS WRT MSG CNT REG signal also energizes aD OR gate 721 to assert a MSG CNT LD message count load signal, enabling the message count register 313 to load the NXI MSG CNT signals.
IncrementatioD and decrementation of the message count value storcd in the message CoDt ;~
register 313 are accomplished by an incrementation circuit 722 and a decrementation circuit 723, respectfully, under control of a control circuit 724. The control circuit 724 generates an INC MSG

WO 92/06436 PCr/US91/07383 2~3~ 89-CNT EN increment message count enable signal to enable the message count value ;D the register 313 to be incremeDted, and a DEC MSG CNT EN decrement message count enable signal. Generally, the control circuit 724 asserts the INC MSG CNT EN signal in response to transmission of a data router message packet 30 by the lef~ and right message injec~os ports 223(1) and 223(r). In addition, the control cira~it asserts the DEC MSG CNT EN sigrlal in response to reception of a data router message packet 30 by the left and right message ejector ports 2~5(1) and 225(r).
The incrementation circuit 7æ receiws the CUR MSG (:NT curre~t message cou~t signals from the message count register 3L3 and generates INC MSG CNT incremented message count signals i n response. The INC MSG CNT signals have a binary-encoded value that is one greater than the binary-encoded value of the CUR MSG CNT signals. Similarly, the decremcnta~ioQ circuit 723 receives the CUR MSG CNT signals and generates DEC MSG CNT decremen~ed message count signals having a binary-encoded value that is one less than the binary-encoded value of the CUR MSG
CNT signals.
~ Both the INC MSG CNT signals and the DEC MSG CNT signals are coupled to data input tennirlals of a multiplexer 725, which is controlled oy the INC MSG CNT EN increment message cs~unt enable signal from the control circuit 724. If the INC MSG CNT EN sig~al is asserted, the multiplexer 725 couples the INC MSG CNT signals as ADJ MSG CNT adjusted message count signals to the second data input terminals of multiplexer 720. 5imilarly, iE the control circuit 724 is not asserting the INC MSG CNT EN signal, ~he multiplexer 725 couples the DEC MSG CNT signals from decrementation circuit 723 as the ADJ MSG CNT signals. ln either case, at this point the PEI BUS
WRT MSG CNT REG is negated, which enables the multiplexer 720 to couple the ADJ MSG CNT
signals as the NXI' MSG CNT signals to the message count regisler 313.
If the control circuit 724 is not also at that point assertiDg the DEC MSG CNT EN decrement message count enable signal, an XOR exclusiYe OR gate 726 ~s enes~zed to assert a CH MSG CNT
chanBe message count signal. The asserted CH MSG CNT si~nal energizes OR gate 721 to assert the MSG CNT LD signaL to enable the message count register 313 to load the NX~r MSG CNT signals from multiplexer 720. It will be appreciated that, if the control circuit 724 is asser~ing both the INC
MSG CNT EN signal and the DEC MSG CNT EN signal, the XOR gate 726 ~ill be de-energized to negate the CH MSG CNT signal. If the PEI BUS WRT MSG CNT REG signal is also negated, OR
gate 721 will be de-energized, negating the MSG CNT LD signal and disabling the n~essage count register 313 from loading the N~CI' MSG CNT signals. The XOR gatc ~26 is proYided since, if control circuit 724 is sirnultaneously asserting both the INC MSG CNT EN signal, to enable increme~tatio~ of the ~ralue in 313 message count register, and the DEC MSG CNT EN siE~aL to enable decrementation of the r~lue in the message count register 313, the value in the register 313 will remaihl uDchanged.
The control circuit 724 includes an incrementation enabling portion 730, which controls the INC MSG CNT EN signaL and a decrementation enabling portion 731, which coDtrols the DEC MSG
CNT EN signal. The incrementation enabling portion 730, generally, controls the INC MSG CNT E~
signal in response to XMIT MSG STAT ITAG] (x) signals (nx" referencing "L" or "Rn) representative of : . .,. . .,: : . : - . ~., .

~ ;; , ,' ''' ".'" '' " ,':''.."~., ",.: ,', ':. ''' :~ ' ;'' ' : :. . ' ,' : , . , .: .

WO 92/06436 PCrlUS91/07383 2~33~
-so thc message tag field of the data rou~er message packet 30 to be transmitted, the [xDR] MSG AVAIL
signal from the left or right message injector port 223(1) or 223(r) indicating that the packet is being transmitted, arld COUNT MASK signals provided by a count mask register (not shown).
As with other registers, the count mask register may be loaded by the processor 200 The COUNT MASK signals permit the message count value in the register 3L3 to be selectively adjusted based on the ~IIT MSG STAT [TAG] signals. of the message pacl~et 30 being transmitted or received, and so the processor 200 can control pardcular condidons, represerlted by the vanous eDcodings of the tag fields 35, under which transmission or reception of messages will affect the message couDt value in the message count register 313. In the incrementation enablhle, portion 730, the COUNT MASK
signals are directed to data input terminals of multiplexers 732(x), which is controlled by the XMIT
MSG STAT ITAG] (x).
In response, each multiplexer 732(x) couples one of the COUNT MASK signals, selected based on the encoding of the XMIT MSG STAT [TAG] (x) signals, as an EN INC (x) enable increment siglsal. Whether the EN INC (x) enable increment signal is asserted will be de~ermined in response to the particular assertion pattern of the COUNr MASK signals and the encoding of the XMIT MSG STAT lTAG] (x) signals. The SEJ, CNT (x) signal and xDR MSG AVAIL signal (~x~
referencmg ~L" and NR~) are coupied to an AND gate 733(x). If both signals are asserted, the AND
gate 733(x) is energized to assert an INC CNT (x) increment count signal, which energizes and OR
gate 734 to assert the INC MSG CNT EN increment message count enable signal. On the other hand, i f either the EN INC (x) signal, or the xl)R MSG AVAIL signal are negated, AND gate 733(x) will be de-energized to negate the INC CNT (x) signal. If both the AND gates 733(1) and 733(r) are negating their lNC CNT (L) and the INC CNT (R) signals, the OR gate 734 ~ill be de-energi~ed to negate the INC MSG CNT EN signal.
The decrementation enabling pordon 731 has similar muitiplexers 740(x) and AND gates 741(x) and m OR gate 742 for generating the DEC MSG CN T EN decrement message count enable signal. The multiplexers 740(x) are controlled by the [LEF rl MSG STAT [TAG] and IRIGHTl MSG
STAT lTAG] signals from the left and right message ejector portions 225(1) and 22~(r) respectively. In response to the lx] MSG STAT [TAG~ signals, the corresponding multiplexer 740(x) couples a selected one of the COUNT MASK signals as an EN DEC (x) enable decrement signal to the AND gate 741(x). AND gate 741(1) is also controlled by the ILEE;Tl NEW STATIJS signal from left message ejector port 22.S(I). If the EN DEC (L) and ILEFTl NEW STATUS signals are asserted, AND gate 741(1) is cnergized to assert a DEC CNT (L) decrement count signal, which energaes OR gate 742 to assert the DEC MSG CNT EN signal.
The AND gate 74i(r), on the other hand, is controlled by a GATED IRTl NEW STATUSSignai from a buffer circuit 750. Buffer circuit 750 buffers and delays the effect of the ass~rtion of the IRIG~I 1 NEW STATUS SigDal iD COnneCtiOn with AND gate 741(r) if both it and the [LEF-I-l NEW
STATUS sigllal are asserted simultaneously, indicating simultaneous receptioD of data router message packets 30 by both message ejector ports 225(1) and 225(r). If the IRIGHT] NEW STATUS signal is , . : . . , :, ................... ;. . .~ :, , ~ . .

- ,,. .: , :

WO 92/06436 PCT/US9l/07383 2 ~ ~ 3 3 r ~ -9l-asserted~but the aEF'l-] NEW STATUS sigDal is negated, an AND gate 751 is energized, which, in urn, energizes an OR gate 752 to assert the GA'rED [RT] NEW STATUS sig~
On the other hand, if both the IRIGHl`] NEW STAl IJS and the [LEFT] NEW STATUS
signals are asserted, AND gate 751 is de-energized, arld an AND gate 753 is energized. The energized AND gate 753 provides an asserted signal to the data input tenninal of a flip-flop 754, which is set at the next tick of the NODE CLK signal to assert a BUF [Rl'J NEW STAlIJS buffered right new status signal. The asserted BUF [RTj NEW STATUS signal energizes OR gate 752 to assert the GATED
lRT] NEW STATUS signal, which enables the AND gate 7~1(r). Thus, if both the [LEFT] NEW
STATUS and the [RIGHT] NEW STATUS signals are asserted simultaDeously, buffer circuit 750 delays the enablement of the AND gate 741(r), which controls the DEC CNT (R) signal, to ensure decrementation of the value in the message count register 313 in response to reception of data router message packets 30 by both message ejector ports æS(I) and 225(r).

: . ' ,' ' ,: :. . ' : ' . '.', " : ' ' ' './ ' ' ! ' ,.:, ' :
' ~ '' , '''. ,; ', . , ~ ' .' ' ' ' ' : " . ' ' " ' ,', ' ' '' . ' WO 92/06436 PCltUS91/07383 2 ~ 9 3 3 ~ 3 3 Control Network Interface 204 i General As noted a'oove, the control network interface 204 receives (ejects) control nehvork message packets 60 from the control net vork 14 and transmits (injects) control net vork message packets 60 to the control net vork 14 A general block diagram of control network interface 204 is shown in Flg 10A, and more detailed diagrams are shown in Figs 10B through 10G
Wlth reference to Flg 10A, the contTol network i~terface 204 includes a trarJsmit section 800 that transmits control nehvork message packets 60 over the control network 14, a receive section 801 that receives control network rnessage packcts 60 from ~he control nctwork 14, and a root control/loopback section 802, all of which are controlled by a common control section 803 and set of registers 804 The transmit sectioD 800 transmits, that is, irjects, control network message packets 60 over the control network 14 The receive section 801 receives, that is, ejects, control network message packets from the control net vork 14, in particular, from the control net~,vork node 50(1,j) (Flg 4A) connected thereto '' ' ' ~~~
The root control/loopback section 802 determines whether the control net~,vork interface 204 is a logical root of a sub~tree, and if so asserts a ROOT UP signal, which is received by the control net vorlc node 50(1,j) It will be appreciated that, if the control net vork interface 204 is asserdng the ROOT IJ~ signal, it is the root of a sub-tree that comprises only the single leaf 21 The common coDtrol secdon 803 maintains several coDtrol and status registers aDd effectively controls the operations of the interface 204 as will be describsd ~elow The registers 804 contain information which is used in generating control network message packets 60 The registers 804 can be written by the processor 200 to Iransmit some control information over the control network 14 iD
message packets 60 transmitted by the transmit section 800 Altematively, the registers 804 may be loaded with cortrol information which the reccive section ~01 obtained from control network messa~e packets 60 received thereby Like ths data router interface 205, the control Detworlc inter&ce 204 also rnakes use of a number of registers to enable the processor 200 to ir~itiate transfers of message packets 60 over the coDtrol network 14, and facilitate recepdon by the processDr 200 of data from con~ol net vork message r packets 60 that the control network irlterface 204 receives from the control networl~ 14 In particular, the processor may initiate transmissions over the control net vork 14, by loading information in a supervisor broadcast register set, a broadcast register set and a combine register set provided in the control network interface 204 All of the reg~ster sets are generally similar to the send first, send and receive registers 234, 235, and 233, respectively of the data router iDterface 205 (F~g 9A-2A), except that the first word of the send first register in the control network interface includes diverse fields, as will be described n conneciiorl with Fi8 10A-2 Since the send and receive registers in each of the supervisor broadcast, supervisor, and combine register sets are similar to the correspondingly-named registers as sho vn on Fg 9A-2A, ~ey will not be described further herein The processor 200 enables the control network interface 204 to ges~erate con~rol net vork message packes 60 in a mamler similar to that in which it enables the data router interface 205 to :: .,: ; . : . ... . .... . . .. .

- : :. ~ . . .. : . . , :

WO 92/06436 PCr/lJS91tl)7383 e~te3~t~router message packets 30. In parhc~ar, the proccssor 200 first trarlsmits infor~na~ion to the control network send first register. Thereafter, if the control ~etwork rnessage packet 60 is, for example, for a multi-word scan operation, requir~ng multiple control network message packets 60 each having a thirty-two bit word, the processor 200 can provide the additional words by loading them into the send register in the particular reg~ster set.
When the control network interface ~04 receives a control network message pac~et 6û from the control rletworl~ 14, it enables the processing elemeDt interface 212 to interrupt the processor 200, identifying the particular register set into which the informatior. from the message pacltet 60 was loaded. The processor 200 ca~ obtain the data received from a particular control network message packet 60 by retrieving the contents of the receive register in the particular register set identified in the interrupt.
The send first registers iQ the supervisor broadcast, broadcast and cornbiDe register sets all have the same general structure, which is shown in Fig. 10A-2. With reference to Fig. 10A-2, e~ch send ~ --- - - first register includes two words 806 and 807. A high-order address field 808 in the first word 806 contains an address value, in the address space defined for the memory bus 203 (Fig. 8) of the network interface 202 and, specifically, of the control network interface 204.
The remaining fields 819 through 829 of the first word 806 include inforrnation which the control network interface 204 uses in generating the packet header 61. A combine fuDction field 819, and pattern field 828 in the first word 806 of the send first register c~ntain information that is copied into fields 65, 66 and 67, respedively~ of the pacl~et header 61. The second word 807 of a send first register contains data to be transmitted in data fields 70 of the packet data 62. If the contents of length field 829 if the send first register i~dicates that a message is to include multiple thirty-two bit words, each thiny two bit word is sent in a control network message packet 60, with the contents of suessive data fields 70 being provided through the send register.
Returning to Flg. 10A-1, the transmit section includes three first-in first-Gut buffers, namely, a supervisor broadcast trans~it first-in first-out buffer (FIFO) 810, a broadcast transmit FIFO 811 and a combiDe transmit FIFO 812. Generally, the supervisor broadcast transmit FIFO 810 stores informatis)n used by the contrd network interface 204 in creating control network message pacl~ets 60 of the single source message type, while the processor 200 is in its supervisor operating mode.
Sirailarly, the broadcast transmit FIFO 811 stores information used by the control net vork interface 204 in creating control net vork message packets 60 of the single-source message type, while the processor 20û is in its user operating mode. The combine transmit FIFO 812 stores information used by the control net vork interface in creating control nehvork message packets 60 of the multiple-source message type, includiDg router done packets 60 which aTe used to determine if the data router 15 is empty. Information in the combine transmit FIFO 812 that enables creation of packets other than a router done packet includes the data that will be used in establishing the contents of the data nibbles 7û(i) (Flg 5) of the packet 60. However, information in the combiue transmit FIFO 812 that enables creation of router done packets does not include such data; the data is instead provided by the CUR
MSG CNT current message count sigDals from the data router interface 205.

.- : . , . . -~: .
,: , : ,. ~ , ,., . . , :: .
- ,. , - ,. . . , . :, -~ :- : , :

WO9~/06436 PCJ/US91/û7383 20933~

Each FIFO 810 through 812 has data input termiîlals that are conuected to the interface bus 211 and is loaded by the interface 212 with the contents of the send first and send registers of the respective supervisor broadcast, broadcast and combine register set wheu the processor 200 transfers information thereto. The interface 212 asserts a PUSH XMIT SBC FIFO push transrnit supervisor broadcast first-in first-out buffer signal, a PUSH XMIT BC FIFO push transmit broadcast first-in first-out buffer signal or a PUSH XMlT COM ~:IFO push transrnit combir e first-in first-out buffer s~gnal to enable the respecdve E IFO 810 through 812 to receive and store the information.Each FIFO 810 through 812 generates status signals "XXX~ XMIT FIFO FUIL transmitbuffer full signal and ~XXX~ XMIT ~IFO MT transmit buffer empty signal (~XXX" may comprise ~SBCA which idendfies the supervisor broadcast transmit FIFO 810, ~BC' which ideDtifies the broadcast transmit FIFO 811, or "COM" which identifies thc combine transmit FIFO 812) which indicate whether the respeaive buffer is nearly full or nearly empty. If a particular FIFO 810 through 812 is assertirlg its XMIT FIFO FULL signal, the interface 212 rejects attempts by the processor 200 to load information therein. ~ ~
Each E~IFO 810 through 812 also includes data output terminals that are connected to a transmit message select a~d assembler circuit 813. Under control o~ a LOAD NEW MSG load new message signal, circuit 813 receives the ~ XMIT FIFO Ml signals, determtnes whether any of them have in~ormation to be used in a control network message packet 60, and if so assembles a coDtrol network message packet 60. In assembling the message packet, the circuit 813 may also use the contents of registers 8û4 and CUR MSG CNT current message count signals from the data router interface 205 if the P~ 1 YPE XMlT signals indicate that the control network message packet 60 is a router done packet. r lhe transmit message select and assembler 813 couples forty-eight bit words representing the header 61 and data portion 62 of the assembled message packet 60 ~o a transmit message buffer 814, which latches it in response to the LOAD NEW MSG signal. In response to successive ticks of the NODE CLK signal, a flick generator iteratively selects ~our-bit nibbles from the transmit message buffer 814 and appends to each selected nibble a high-order bit comprising the tag bit. The flick generator 81S transrnits ~he result as FLICK Olrr (4:0) flicic output sigDals to the control network node SO(lJ) connected thereto, and also to a flick buffer 816 in the root control/loopback arcuit 802.
As it iteratively transmits the FLICK OUT (4:0) signals, the flick generator 815 generates a checksum, which it transmits as the thirteen~h flick of the control network message packet.
Contemporaneously with transmission of the last flick of the message packet 60, the flic3~ geDerator 815 asserts the LOAD NEW MSG load new rnessage sig~al to enable the transmit message buffer 814 to latch a ~ew word and the circuit 813 to generate a new cwltrol network rnessage packe~ 60 for lransmission.
As noted above, the control net vork interface 204 includes a set of registers 804 that provide information which may also be used in generating message packets. A globals reE;ister 820 contains global bits that can be used to perforrn a global operation as described above. A flush flag 821 can be .. . .. . .
, . - - ", ,: . . ~ ~ ,, , -- ~ - .: ,,' , : . , . ;
.~, . . . . .

-..

~a~3~s~ 95~
used to control the Qush bit 75 in the control network message packet 60; if set, the flush bit 75 enables the control network 14 to flush intermediate results of a scan operation. An interrupt register 82 ~ can be loaded with an interrupt value that can be transmitted in a single-source message packet of the iuterrupt type, to broadcast iDterrupt information to other leaves 21 in the partidon.
A configuration register 823 contains a value that can be used i~l a single-source message packet of the configuradon type to idendfies the level and sub-level at vhich the logical root is to be established for the partition; this informatioll is loaded into fli ks 70(0) and 70(1) of the packet data pordon 62. An all fall-do vn mode ~ag 824, which is derived from all-fall-down enable bit 256 of the private register 232 (F~ A-2A) is used to inidate an all-fall-dow~ operadon in the data router 15; the all-fall-down mode flag 824 is used to condition all-fall-down mode bit 81 of the control network message packet 81. Finally, a segment flag 825, which may be conditioned by the processor 200, i5 used in segment bit ~7 of a muld-source control network message packet 60 to idendfy the beginniDg of a segment in a segmented scan operation.
The receive secdon 80I includes a ~ick demultiplexer 830 that iteratively recei~es, at each dck of the NODE CLK signal, either the FLICK IN signals from the control network node 50(iJ) or the BUF FLICK OIJT buffered flick out signals from the root control/loopback circuit ~02. If the root control/loopback circuit g02 is asserting SEL XMlT select transmit section signal, generally indicat~ng that the co~trol network interface 204 is a logical root, the flick demultiplexer 830 selects the BUF
FLICK OUT sig~als, and otherwise it selects tbe FLlCK IN signals. The flicl~ demultiplexer 830 strips of~ the tag signals, some of which it buffers, and demultiplexes t~e other received signals to so that successively received signals are used to form successiYe nibbles of a forty-eight bit word. At the same time, the flick demuldplexer 830 maintains a rum~ing checlcsum of the signals received at each tick of t,he NODE CLK sigDal. The flicls geuerator uses the checksum to determine whether the control uetwork message pasket was correctly received, and, if so, it asser~s a LOAD RCVD MSG load received message packet signal.
The assertion of the LOAD RCVD MSG signal enables a received message buffer 831 to latch the word generated by the flick demultiplexer. In addition, the asserted LOAD RCVD MSG signal enables a receive message buffer and destination select circuit 832 to e7samine the word contained in thG received message buffer 831, and determine which of a supervisor receiver broadcast FIFO 833, a broadcast receiver FIFO 834, a combine receiver FlFO 835, or oDe of the registers 804, in which the word should be loaded.
Each FIFQ 833, 834 and 835 generates an nXI~Xn RCV FIFO NR FIJLL receive FIFO nearly filll signal ("X~ may comprise ~SBC which identitles the supervisor receiver FIFO 833, ~BC which identifies the broadcast receiver FIFO 834, or ~COM~ which identifies the combine receiver FIFO 835) which indicate whether the respective FIFO is nearly fuU. As wiU be desc~ibed below in connection with Fig. 10C, the ~XXX RCV FIFO NR FULL sigDal is used by the transmit message select and assembler 813 in generating several of the ~ag signals for the various ~icks. In addition, tbe flick demuldplexer 830 couples several of the tag signals which it receives to the flick generator to control transmission of control network message packets 60 thereby.

, . : . : :: : :

WO 92/06436 PCI`/US91/07383 2~3~
.96 ii. Transmit Section 800 With this background, details of various por~ions of the control nehvor~ interface 204 will oe described in connection with Figs. 10B tbough 10G. Fig. 10B depicts a detailed functional block diagram of the transmit message select and assembler circuit 813 and tbe traDsmit message buffer 814.
With reference to Fig. 10B, the transmit message select and assembler circ~ut 813 includes a message priority select circuit 840 that perforrns several functions in response to the assertion of the LOAD
NEW MSG load new message signal from the flick generator 815 eDable generation of a word to be loaded into the transmit message buffer 814.
First, the message priority select circuit 840 receives status signals from various sources of message informatio;l such as the various FlFOs 810, 811 and 812, and the interrupt and configuration registers 822 and 823. In particular, the message priority select circuit 840 receives from the FlFOs 810 through 812 the ~ XMlT FlFO MT (~ being SBC, BC and COM) supervisor broadcast, broadcast and combine transmit FIFO empty signals. If any of these signals are negated, the corresponding FIFO 810, 811 or 812 contains data to be transmitted in a control network message packet 60. In addition, the message priority selea circuil 840 Jeceives an IJNSENT CONFIG unsent configuration signal from the configuration register 823 and an UNSENT INT unsent interrupt signal from the interrupt register indicating that the processor 200 has loaded a new value theJein that is to be transmitted in a control ~et vork message packet 60.
In response to the assertion of the LOAD NEW MSG load new message signal from the flicl~
generator 8~5, the message priority select Qrc ut 840 uses tbese status signals to select one FIE~O as a control net vork message packet o0 information source in response to a predetermined prioritv to use in generating a control network message packet 60. Tbe circuit 840 obtains a word from the ~ata output terminals of the selected FIFO for use in creating,the control network message packet 60. In addition, the circuit 840 genesates POP ~XXX" FIFO (~ referencing SBC, BC and COM) first-in first-out buffer pop signal, which, when asserted, enables the respective FlFO 810, 811 and 812 to provide a new word at its data output tenniDals which may be used in creating a subsequent control network message packet 60. In particular, since the FlFOs 810 through 812 are first-in first-out buffers, the words provided in response to successive POP ~ IFO signals will be in the order in which they were stored in the respective FIFO.
The circuit 840 also generates a READ INTR REG read interrupt register signal and a RE~D CONFIG REG read configuration register signal, which, when asserted, eDables the respective interrupt register 822 or configuration register 823 to couple its co~tents to the circuit 813. When the respective register couplcs its contents to the circuit 813, it will negate its UNSENT CONFTG unseDt configuration or UNSENT INT unsent interrupt signal until it is again loaded by the processor 200.
The message priority l:ircuit 840 also geDerates MSG TYPE XMIT message type transmit signals and PP~ TYPE XMIT that are coupled to input terminals of the traDsmit message buffer 814, in particular to input terminals of nibbles 814(0) and 814(1), respectively. The signals uill be used in forming the contents of message type field 64 and packct type field 65, respectively. It vill be ,: . . i . . .
': , ' . , ~: . ,: - .
, . ,. :.
, :~ ~ , : ~ . . . .

WO 92/06436 PCr/US91 /07383 2~3~5~i ~ 97-appreciated that the particular encoding of the MSG TYPE XMIT and P~ TYPE ~IIT signals will reflect the particular information source identiGed by the particular one of the POP ~ FIFO
signals or READ INTR REG or READ CONFIG REG signal that it asserts.
Alternatively, it will be appreciated that all of the ~ XMIT nFO MT sigDals may be asserted, indicating that none of the FlFOs 810 through 812 contain information to be used in establishing control network message packets 60, and the UNSENT CONFIG or UNSENT INT signals may be negated when the flick generator 815 asserts the LOAD NEW MSG signal. When that occurs, the message priority select circuit 840 may generate MSG l'YPE XMIT message type transmit signals that identify a control network message packet o0 of the idle type, which it couples to nibble 814(0) of the transmit message buffer 814. In either case, the P~ TYPE XMIT pacl~et type transrnit signals from circuit 840 are all negated.
The transmit message select and assembler circuit 813 also includes several multiplexers 841, 842 and 843 which provide additional signals that are coupled to the transmit message buffer 814 along with the MSG TYPE XMIT and P~ TYPE XMIT sig~als. With refere~ce also to Fig. ~, the iK~
TYPE XMlT signals comprise only three s~gnals that form the low-order three bits of the flick ~ The high-order signal is obtained from a multiplexer 841, which, under control of a FLICK (1) SEL flick 1 select signal from message priority select circuit 840, selects either a INTR (4) signal representing the high-order bit from the interrupt register 822 or a high-order PAT(1) paltern signal from a word popped from the combine first-in first-out buffer 812. If the circuit 840 is asserting a FLICK (1) OE
flick 1 ou~put enable signal, the multiplexer 841 couples the selected signal to a high-order input terminal of the nibble 814(1) of transmit message buf~er 8~4. It will be appreciated that the F~,ICK ~1) SEL signal will be conditioned to select the PAT (1) signal only if the message priority select circuit 840 has selected the combine buffer 812 as the source of informatioll for the control network message packet 60 to be transmitted. Further, the FLICR (1) OE signal will not be asserted if the MSG TYPE
XMIT message type transmit 5ignals from circuit 840 contain the encoding identifyillg an idle message packet, to ensure that the contents of the respective fields of the message packet 60 will be zero.
Multiplexer 842 is used to select information that will be used in establishing the low-order four bits of flick 2 of the control Detwork message packet 60. Under control of a FLIC~C (2) SEL flick ~2" select signal from the circuit 840, the multiplexer 842 selects one of the TNTR (3:0) interFupt signa)s, or COMB FTN (2:0~ combine fuuction signals and a low-order PAT (0) pattern signal. If a FLICK (2) OE flick (2) output enable signal is asserted, the multiplexer 842 couples the selected signals to the input terminals of ribble 814(2) of the transmit message buffer 814. The INTR (3:0) interrupt si~oals are based on low-order bits contained in the interrupt register 8æ. The COMB FTN (2:0) signals ant PAT (0) signal are from the word popped from tbe combine buffer 812. It will be appreciated tbat the FLICK (2) SEL signal will be conditioned to selec~ the COMB FTN and PAT (0) s~g~lals only if the mcssage priority select circuit 840 has selected the combine buffer 812 as tbe source of information for the control network message packet 60 to be tr:mcmiîted. Further, the FLICK (2) OE signal ~ill not be asserted if the MSG TYPE XMIT message type transmit sigDals from circuit 840 contain the . . .: :. . .: . .: , . :, .: , . .: - . : : : - . ,, ::: :: ~ .:,, , .. , - . . ~ . .. . .

WO 92/06436 PCr/US91/07383 -98- 2~33'3~
encoding identifying an idle message packet, to ensure that the contents of the respective fields of the message packet 60 will be zero.
Finally, the multiplexer 843 selects information that will be used in establis~ing the contents of data nibbles 70(0) through 70(7) in the control network message packet 60 under control of DATA
SEL data select signals from the message priority select circuit 840. If the PKI TYPE XMIT signals identify a router done message the DATA SEL signals will enable the multiplexer 843 to select the CUR MSG CNT current message count signals from the data router interface 20S to the input terminals of nibbles 814(4) through 814(10) of the transmit message buffer 814. Otherwise, the particular source selected by multiplexer 843 in response to the DATA SEL data select sigDals will ``
correspond to the source identified by the one of the POP ~ FIFO signals, the READ INTR
REG signal or the READ CONFIG REG signal asserted by the message priority select circuit 840. If the message priority select circmt 840 is also asserting a DATA OE data output enable sigDal, the muldplexer 843 will couple the selected signals to respective input tenninals of transmit message buffer 814. It will be appreciated that th-e DATA OE signal will not be asserted if the l~fSG 1YPE XMIT
message type transrnit signals from circuit 840 contain the encoding identifying an idle message packet, to ensure that the contents of the respective fields of the message packet 60 will be zero.
In addition to these signals, GLOBAL signals representillg the conteDts of the globals re~sister 820 are coupled to respecdve input terminals of nibble 814(11) of the transmit message buffer 814, which latches the GLOBAL signals along with the other sigllals at its iDput terminals. In response to the next assertion of the LOAD NEW MSG load new message signal from the flick generator 815, the transmit message buffer 814 latches all of the signals at its input terminals. The latched sigDals are transmitted as CN MSG coDtrol network message signals to the flick generator 1~. The CN MSG
signals define forty-eight bits in t velve four-bit nibbles. In respoDse to thq same assertion of the LOAD NEW MSG signal, the message priority select circuit repeats the above-described operations to enable generadon of a new word to be loaded into the transmit message buffer in respoDse to the subsequent assertion of the LOAD NEW MSG signal.
The flick generator 815 successively selects nibbles from the message buffer 814, appends to each nibble a tag bit to form five-bit flicks and traDsmits them as FLICK OUT (4:0) signals at each of iwelve successive ticks of the NODE CLK signal. In addition, the flick generator maintains a rur ning checksum which it transmits as the thirteenth flick. ~ig. 10C depicts a detailed block diagram of tbe flick generator 815.
Ufith reference to hg. 10C, the flick generator 815 includes a transmit timiDg control circuit 850 that iteratively generates XMIT O transmit tlick æro through XMlT 12 transrnit nick 12 S~S, including binary-encoded XMIT 3-10(2:0) transmit flicks three through ten signals. The XMlT O
through XMlT 1~ signals enable other circuitry depicted on Fig 10C to transmit successiYe ones of the thirteen nicks comprising a control network message packet 60. The transmit tirning control circuit 850 generates the XMlT O through XMIT 12 sigDals in syDchronism vith successive ticks of the NOI:~E
CLK signal, while enabled by a negated CN STOP SEND control network stop sending sigDal and a , . , ~

. . .
.: , , . ~- . .. :

WO 92/06436 PCI lUS91/07383 2 0 !3 3 ~
Degated MSG FLOW NOK message flow not ok signal, with the proviso that once the transrnit tirning coDtrol circuit 850 has begin transmitting a control network message packet 60 and has sequenced from the XMIT 0 ~ransmit flick zero signal to the transmit flick one signal, it will conti~ue sequencir~g through the remaining XMlT 2 through XMIT 12 sigrlals, and stop when it recycles to the XMIT 0 transmit llick zero signal again.
Various ones of the CN MSG control network message signals, representing various ones of the nibbles in the trallsmit message buffer 814, are coupled to several circluts in the flick generator 815.
In particular, the transmit flick generator includes a flick selector circuit 844 that, generally, selects signals from successive nibbles of the transmit message buffer for transmission in successiYe flicXs. A
flow control circuit 845 rec~ives flow control signals from the receiver section 801 (described below in further detail in Fig. 10D) that control transmissioD of control network message packets by the flick generator 815. A data nibble order circuit 846 determines the order in wDich nibbles 814(4) through 814tlO) will be selected for transmission by the flicl~ selector circuit 844. Finally, a tag signal generator circuit 847 generates tag signals to be transrnitted as the high-order ~LICK OUT sigllal.
As noted above, the ~ow control circuit 84~ receives flow control signals from the receiver section 801 (described below in further detail in Fig. 10D) that control transmission of control net vork message packets 60 by the flick generator 815 in resporlse to selected tag signals ill control nenvork message pacl~et 60 received by the receiver section 801. The control network node sotlJ) connected ~o the control network interface 204 i~ each leaf 21 can control the flow o~ control network message packets 60 thereto from the same co~trol net~vork interface 204 by suitable conditioning of the scan flow bits 72(1) through 72(5), ~he broadcast user flow bit 73 and the broadcast supervisor flow bit 74.
For ex~unple, if the receiver section 801 receives a control network message packet 60 in which the broadcast supervisor flow bit 74 is clear, ~he flick generator 815 is disabled from transmitting c ontrol network message packets 60 frorn irLor~nation iD transmit message buffer 814 that originated from the supervisor broadcast transmit FIFO 810. Similarly, if the receiver seaion 801 receives a control network message paeket in which the broadcast user bit 73 is clear, the flick generator 815 is disabled from transtnitting control network message packets 60 from information in transmit message buffer 814 that originated from the broadcast buffer 811. Fmally, if the receiver section 801 receives a control network message packet 60 in which a scan flow bit 72(1~ through 72(5) is clear, the flick generator 815 is disabled from transmitting control network message packets 60 from information in traQsmit message buffer 815 that originated from the combme buffer 812.
In each case, the fliclc generator 815 will stall the transmit message select and assembly c~rcuit 813 and will remain disabled from transmitting a message packet 60 containing information or~ginating from the particular FIFO 810 through 812 undl the receiver sectioll 801 receives a control neh~ork message paclcet o0 in which the particular bit is set. However, if the transmit message buffer 814 contains i~ormadon that originated from a different one of FlFOs 810 through 812 th~ that from whieh it is disabled, the flick generator 815 may transmit a control network message packet 60 using that informadon.

. ; . :~ : , : :
. ,, : . ;;

. , ,. :. : ":: ~ . . . .

20933 j~
~oo.
More speciflcally, the flow control circuit 845 includes t vo decoders, namely, a message type decoder 861 and a packet type decoder 862 that receive the ones of the CN MSG (3:0) ar,d CN MSG
(7:0) control net vork message signals from nibbles 414(0) and 414(1) of transmit message buffer 814.
The CN MSG (3:0) signals received by decoder 861 comprise the ones of the CN MSG signals that contain the message type information. If the CN MSG (3:0) signals have the encodirlg for a multi- "
source message, the decoder 861 asserts a MULT SRC MSG multi-source message signaL and if they have the encoding for a single so~rce rnessage the decoder 861 asserts a SINGLE SRC MSG single source message signal.
The CN MSG (7:4) signals receiwd by decoder 862 comprise the ones of the CN MSG signals that contain the packet type information. In particular, ~e PKT TYPE signals identify whether the contents of the transmit message buffer 814 originated in any of the FlFOs 810 through 812, and, if so, which one. The decoder 862 generates three output signals, identified as COMB combine, SBC
supervisor broadcast and BC broadcast, which, if asserted, indicate that the contents of message buffer originated in the combine transmit FIFO 812, the supervisor broadcast transmit FIFO 810 or the broadcast transmit FIFO 811, respectively.
The SINGLE SRC MSG single source message and MULT SRC MSG multi-source message signals from decoder 861, and the COMB, SBC and BC signals from decoder 862 are coupled, along with a RCVD BC FLOW received broadcast buffer flow signal, a RCVD SBC FLOW received supervisor broadcast buffer flow signal and a RCVD SCAN FLOW received scan flow signal to a circuit comprising a plurality of AND gates 864 through 866 and inverters 870 through 875. The RCVD BC E:LOW, RCVD SBC Fl,OW and RCVD SCAN FLOW signals comprise the Qow control signals generated by the receive section 801 in response to the respective tag bits of control network message packets 60 received from the control network node 50(1J). If all of ~ese signals are a~serted, inverters 870 through 872 will disable AND gates 863 through 865, enabling, in turn, inverters 873 through 875 to energi~e AND gate 866 to assert a FLOW PERM flow permitted signal.
On the other hand, if one of the flow control signals from the receive section 801 is Degated, and if the signals from decoders 861 and 862 have appropriate conditions, the AND gate 866 uill be disabled and the FLOW PERM tlow permitted signal will be negated. If, for example, ~he RCVD
SCAN FLOW signal is negated, if the MULT SRC MSG multi-source message and COMB signals are asserted and inverter 872 will complement the ne8ated RCVD SCAN FLOW signal to energize the AND gate 865. The energized AND gate 865 will eDable inverter 875 to disable AND gate 866, causing the FLOW PERM flow permitted signal to be negated. The other AND gates 863 and 864, and respective inverters connected thereto, operate siJnilarly in response to the RCVD BC FLOW and RCVD SBC FLOW signals, respectively if decoder 861 is asserting the SINGLE SRC MSG single source message signal.
The FLOW PERM flow permitted sigDal is coupled to one input terminal of a multiplexer 867.
If the transmit timing control circuit 850 is asserting the XMIT n transmit flick zero signal, the multiplexer 867 couples the FLOW PERM signal to a flip-flop 868, whieh latches the signal at the next WO 92/06436 PCI'/U~i9l/07383 20933~rj -101-tick of the NODE CLK signal. If the Fl,OW PERM signal is asserted, the nip-nOp 868 asserts the MSG FLOW OK signal. The MSG Fl,OW OK signal is complemented by an inverter 809. Thus, if the FLOW PERM signal is not asserted, causing the llip-flop 868 to negate the MSG FLOW OK signal, inverter 869 asserts a MSG Fl OW NOK message flow not ok signal.
The MSG FI~OW OK signal frorn flip-flop 868 is coupled to the other input of multiplexer 867.
It will be appreciated that at subsequent ticks of the NODE CLK signal, the XMIT O transmit flick zero sigDal ~ilt be negated, eDabling the multiplexer 867 to couple the MSG FLOW OK signal back to the data input terminal of the flip-flop 868. Thus, tbe flip-flop 868 will maintain its condition at subsequent ticks of the NODE CLK signal as the transmit timing control circuit 850 asserts the XMIT
1 through X~T 12 signals. At the r ext assertion of the XMIT O si~al, the multiplexer 867 again couples the FLOW PERM flow permitted signal to the data input terminal of flip-flop 868, at which point it latches the condition of the signal at that point.
It will be appreciated that, if the decoder 861 is not asse ting the SINGLE SRC MSG single source message signal, the conditions of the RCVD BC FLOW and RCVD SBC FLOW sigcals u ill not affect the condition of the FLOW PERM flow permitted signaL and, thus, will not affect the condition of the MSG FLOW OK and MSG FLOW NOK signals. In addition, if the decoder ~61 is not asserting the MULT SRC MSG multi-sowce message sigDal, the condition of the RCVl~ SCAN Fl,OW signal will not affect the condition of the FLOW PERM flow permitted signal. Thus, if, for e7~ample, the MSG TYPE signals identify the idle or abstain message packet types, Deither the MULT SE~C MSG
nor the SINGLE SRC MSG signal will be asserted, and so the FLOW PERM signal will be asserted, as will the MSG FLOW OK signal.
The portion of the CN MSG control network message signals from nibble 814(2) of the transmit message buffer 814 is soupled to the data nibble order circuit 846. That circuit 846 determines the order in which the flick selector circuit 844 transmits signals from nibbles 81~(3) through 814(10) in the FLICK OUT signals. If the control network message packet 60 being transmitted is a multiple source message initiating certain types of arithmetic operations, such as addition, the Dibbles 70(0) through 70(7) of the packet data portion 62 will carry data of increasing significance. T~is permits the control network nodes 50 to properly generate car;ies from one nibble 70(i) to the ne~ nibble 70(i+1). On the other hand, in some operadons, such as detennination of a ma~num, the control net vork nodes 50 u~ill perfonn nibble-by-nibble comparisorls of data in nibbles 70(0) tnrough 70(7) of the multiple-source message packets. Accordingly, tbe successive nibbles 70(0) thro~ 70(7) ~ill carry tata of decreasi~g sienificaDce.
The bina y-encoded XMIT 3-10 (2:0) signals generated by tLe trans~nit tining control circuit 850 are coupled to an XOR (exclusive-OR) gate circuit 880. The XOR gate circuit 880 geDerates D
FLICK SEL (2:0) data llick select signa'~. representing the exclusive OR of each of the corresponding XMIT 3-10 (2:0) sigr als with a REV DATA reverse data signal, which is controlled by the data aibble order circuit 846. The sequence of D FLICK SEL (2:0) data flick select signalr. determines the sequence of nibbles 814(3) through 814(10) of transrnit message buffer 814, and thus the order of ,., ,.. . , ~; , :: , ,, .: . , - : , . ;. ,.- , . . .- .,; : . , WO 92/06436 Pcr/us91/o7383 2~3~
-104- r significance of the data traDsmitted in the successive mbbles 70(0) through 70(7) irl the control nehvork message packet 60. If the REV DATA signal is negated, the sequence of binary-encoded values of the D Fl.ICK SEL (2:0) signals corresponds to the sequeDce of binary-eDcoded values of the XMIT 3-10 (2:0) signals, and so contents of nibbles 814(3) through 814(10) are transmitted in nibbles 70(0) through 70(7), respectively.
On the other hand, if the REV DATA reverse data signal is asserted, the D FLICK SEL (2:0) data flick select signals correspond to the complement of the respective XMIT 3-10 (2:0) sigl~als. In that case, the sequence of the binar,v encoded values of the D FLICK SEL (2:0) signals is the reYerse of the sequence of binar,v eAcoded values of the XMIT 3-10 (2:0) signals, and so the nibbles 70(0) through 70(7) of the control network message packel 60 carry the contents of nibbles 814(10) through 814(3).
The data nibble order circuut 846 genera~es the REV DATA reverse data signal. Circuit 846 includes a decoder 881 that receives CN MSG (118) control network message signals from nibble 814(2) of the transmit message buffer 814. If these signals have the encodi~g to identify a ma~num arith~netic operation, the decoder asserts a MAYBE MAX signal, which enables one input of an AND
gate 882. If contemporaneously the decoders 861 and 862 are asserting the MULT SRC MSG multiple source message signal and COMB combiue sigDaL tbe control network message packet 60 generated from the contents of transmit message buffer 814 will enable a ma~amum arithmetic operation. When that occurs, ANI~ gate 882 is energ~7ed to assert a MAX signal. When the XMlT 2 transmit flick two signal is asserted, a multiplexer 883 couples the ~IAX signal to the data input terminal of a nip-tlop 884, which latches the MAX signal at the ne~ tick of the NODE CLK signal.
The nip-nop 884 geDerates the REV DATA reverse data sigDal. In addition to controlling the XOR gate 880, the REV DATA sigl~al is also coupled to the other data input terminal of multiplexer 883, which couples that signal to the data input terminal of the fli~flop 884 while the XMIT 2 transmit flick two signal is Dot asserted. It will be appreaated that the condition of the REV DATA reverse data signal subsequent to the time at which the XMlT 2 transmit flick two s}gnal is asserted will correspond to the conditior~ of the MAX signal at the time the XMlT 2 signal is asserted.
Accordingly, if any of the MULT SRC MS~ multiple source message, COMB combine and MAYBE MAX signals is negated, which indicates that the coDtrol Detwork message packet 60 being transmitted is not enabling a ma~rnum arithmetic operation, the AND gate 882 does not assert Ihe MAX signal ant so the REV DATA signal is not asserted. On the other hand, if all of the MULT
SRC MSG multiple source message, COMB combine and MAYBE MAX sig~als are asserted, indicating that the control network message packet 60 being transmitted is enabling a maxirnurn anthmetic operation, the AND gate 882 will assert the MAX signal and so the REV DATA sigllal will be asserted. As noted above, the asserted REV DATA signal enables the sequence of binary encoded values of the D FLICK SEL t2:0) data flick select signals generated by XOR gate 880 to have the reverse order than if the REV DATA signal is negated.
The flick selector circuit 844 receives the CN MSG (47:0) signals at input terminals of a group of multiplexers 851 through 856 and an AND gate 855. The output terminals of multiplexers 851 - - :, :. ~ ~ ,:: .. , WO 92/06436 PCI'/~ 9l/07383 20~33~ o~ ;
through 854 are coupled to respective input terminals of a flick select multiplexer 856. The multiplexers 851 through 854 and 856, along with AND gate 855, cooperate to select successive four-bit nibbles for tramsmission as successive flicks of a coDtrol network message packet o0.
In particular, CN MSG (3:0) control Detwork message signals identifying ~he message type, transmitted by nibble 814(0) of the transmit message buffer 814, are coupled to one set of input terminals of m~lltiplexer 851. The other set of irlput tenninals of multiplexer 851 rece*e IDLE signals representing the message type encoding for an control network message packet 60 of the idle type.
The multiplexer 851 is coDtrolled by the MSG FLOW NOK message flow not ok signal from the flow control circuit 845. If the MSG FLOW NOK signal is asserted, the multiplexer 851 couples the IDLE
signals to the respedive input termir als of the flick seled multiplexer 856. On the other hand, if the MSG FLOW NOK signal is Degated, the multiplexer 851 couples the CN MSG (3:0) signals to the same input terrminals of flick select multiplexer 856. ~`
The CN MSG (7:4) signals, which are transmitted by Dibble 814(1) of the transmit message buffer 814 and identify the packet type, are coupled to input terminals of AND Bate 855. If the message flow circuit 845 is asserting the MSG FLOW OK signaL the AND gate 855 gates CN MSG
(7:4) signals to one set of input terminals of tbe multiplexer 855. The other set of input tenninal receive the CN MSG (11:8) signals, which are transmitted by i~bble 814(2) of Ihe transmit message buffer. If the XMl'r 2 transrnit flick two signal is negated, the multiplexer 852 couples the signals from AND gate 855 to a set of input terminals of flick select multiplexer 856. On the other hand, if the XMIT 2 signal is asserted, the muldplexer 852 couples the CN MS~ ~11:8) signals thereto.
The multiplexer 853 has eight sets of iDput terminals, each receiving four CN MSG signals from one of nibbles 814(3) through 814(10) of the transmit message buffer. The multiplexer 853 is controlled by the D FLICK SEL (2:0) signals from XOR gate 880. The multiplexer 853 couples signals at selected sets of inputs, as identified by the D FIICK SEL (2:0) siE~als, to a set of input terminals of the flick select multiplexer 856. The D FLICK SEL (2:0) signals detennine the order in whicn the sigDals from the Dibbles 814(3) through 814(10) are coupled to the ~ick select multiplexer 856.
Fmally, multiplexer 854 has one set of input terminals that receiYe CN MSG (47:44) sign~s from the nibble 814(11) of the transmit message buffer 814. The CN MSG (47:44) signals represent the contents of the globals register 820, which, as shown in Fig. 5, are îransmitted in the last flick of a control network message packet 60 before the flick contau~ing the checksum. If ~he XMIT 11 transmit flick eleven signal is asserted, the multiplexer 854 couples the signals from this set of terminals to a fourth set of input terminals of the flick selector multiplexer 856. VVhen thc XMIT 11 signal is not asserted, the multiplexer 854 couples CHECK (3:0) signals from a checksum generator 857 to the same set of input terminals.
The flick select multiplexer 8~o is controDed by the XMlT O through XMIT 12 signaLs from transmit tirning control circuit 850. When tbe XMIT 0 signal is asserted, the flick select multiplexer 856 couples the signals from multiplexer 851 as tbe low-order FLlCK OUT (3:0) signals. If the MSC
FLOW NOK signal is negated, the flick includes the contents of nibble 814(0) of the tramsmit message .~

~ t) 9 3 3 5 ~
-10~
buffGr 814. However, if the MSG FLOW NOK siE~al is asserted, the flick includes the idle message packet type code.
The sequential assertion of the XMIT 1 and XMIT 2 signals enables the flick select multiplexer 856 to couple the signals from multiplexer 852 as the low-order FLICK OUT (3:0) signals for the next two flicks. If the MSG FLOW OK signal is asserted, the flicks include the contents of nibbles 814(1) and 814(2) of the transmit message buffer 814. If, however, the MSG F~OW OK signal is negated, the signals from multiplexer 851 while the XMrr 1 signal is asse~ed will all be negated, and so the FLICK OUT (3:0) signals at the point will also be negated.
The assertion of the XMIT ~10(2:0) signals aDd the XMIT DATA signal enables the flick select multiplexer 856 to couple signals from rnultiple~er 853 as the low-order FLICK OUT (3:0) signals for the next eight flicks. The flicks will include the contents of nibbles 814(3) through 814(10) of the transmit nibble buffer 814, with the order depeDding on the condition of the REV DATA signal.
Finally, the sequential assertion of the XMIT 11 and XMIT 12 signals enables the flick select multiplexer to couple the signals from multiplexer 854 as the low-order FLICK OUT (3:0) signals for the final two flicks. When the XMIT 11 signal is asserted, the flick includes the contents of the nibble 814(11) of the transmit messags buffer 814. On the other hand if the XMlT ~ signal is asserted, the flick includes the low-order CHECK (3:0) checksum signals from checksum generator 857.
The tag signal generator circuit 847 receives signals from a number of sour*s and, in response to each the XMIT (0) through XMIT (12) signals, selects one to couple as a SEL TA(3 selected tag signal. Th~ SEL TAG signal is transmi;ted to the control network node 50(1J) and to the checksu~n generator 857 as the FLICK OUT (4) signal. In particular, the tag signal generator cira~it 847 couples a COM RCV FIFO NR FULL combine receive first-in first-out buffer near full signaL from buffer 835 (Fig. 10A-1) as the SEL TAG signal in response to the XMIT 0 signal and encodings of the XMlT 3-10(2:0) signals having the values three, six, eight and ten. This signal provides the scan flow bits 72(1) throLgh 72(5) of the control net vork message packet 60.
The tag signal generator circuit 847 couples as the SEL TAG signal a signal representing the segment flag 825, the BC RCV FIFO NR FIJLL broadcast receive first-in first-out buffer nearly fuU
signal, and the SBC RCV FIFO NR FULL supervisor broadcast receive first-in first-out buffer nearly full sigDal in response, respectively, to the XMIT 1 and XMlT 2 signals and the encoding of the XMIT
3-10 (2:0) signals having the value one. These signals provide tbe seg~nent, broadcast user flow, and broadcast supervisor user flow bits 77, 73 and 74 in the control network message packet 60 (Flg. 5).
In addition, the circuit 847 couples as the SEL TAG signal a signal representmg the condition of the all-fall-do vn mode flag 824 and flush register 821 in response to the encodings of the XMlT 3-10 (2:0) signals representing the values two and four. These signals provide the all-fall-down mode bit 81 and flush bit 75 in the control network message packet 60 (Fig. 5).
Finally, tag signal generator circuit 847 couples as the SEL TAG selected tag signal the CHECK (4) high-order checksum signal from the checksum generator 857. This provides the high-order bit of the checksu~n flick 63 in the control network message packet 60 (Fig. 5).

: ~. ., , , : .
- . .- : .. ~: :::... , . .. . : :

2 0 !) 3 ;~s~i~ied above, the checksum generator 857 generates C~ECK (4:0) checksum signals representing a checksum value. The checksum generator 857 generates the checksum over the first t velve flicks of the control network rnessage packet 60. The checksum generator 857 is reset by the assertion of the XMlT O ~ransmit fliclc zero signal and updates the checksum value in response to each successive tick of the NODE CLK signal.
The transmit section 800 couples the FLICK OUT (4:0~ flick output signals to the control network node 50(1J) cor~nected thereto, and also to the root control/loopback section 802. As noted above, if the root control/loopback portion is asserting the ROOT UP signal, which occurs if the control net vork interface 204 is a logical root, it buffers the flicks definillg control net vork message packets 60 for transmission to the receive section 801.
iu. Receive Section 801 The receive section 801 includes several elements, including the flick demultiplexer 830 which receives the sequential flicks of a cor~trol net vork message packet o0 and assembles them in the receive .
message buffer 831. The destination control 832 determi~es one of several deslinations for the contents of the receive message buffer, including the FlFOs 833 through 835 and interrupt register sæ, and coupled the conten~s ~o the identified destination. Details of the flick demultiplexer 830 and receive message buffer 831 will be described in connection with Fig. 10D, and the destination control will be described in connection vith Fig. 10E.
Wi~ reference to Fig. 10D, the flick dem~Jltiplexer 830 includes a nick source multiplexer 8~0 that selects from among two sources of signals for con~ol network message packets 60 in response to a SEL XMlT select transmit section signal ~rom the root control/loopback section 802. In particular, if the SEL XMIT signal is negated, the flick source multiplexer 8g0 couples the FLICK IN (4:0) flick input signals, which it reoeives from the control network node 50(1J) connected thereto at one set of input terrninals, as RCVD FLICK (4:0) received flick signals. On the other hand, if the SEL XMIT
signal is asserted, which generaUy occurs when the control ne~vork ir terface 204 is a logical root, the multiplexer 890 couples the BUF FLICK OUT (4:0) buffered flick output signals from the flick buffer 816 as the RCVD FLICK (4:0) received flick sigDals.
In either case, the RCVD FLICK (4:0) received flick sigllals are coupled to several portions of the flick demultiplexer 830. In particular, the RCVD FLICK (4:0) signals are connected to a receive tiining generator circuit 891, a check circuit 892, a~d a flick distribution and latch circuit 893. The receive timing generator circuit 891, like the transmit timing control circujt 850, iteratively generates RCV 0 receive flick zero through RCV 12 reccive flick 12 signals. The RCV 0 through RCV 12 signals enable other circuitry depicted on Fig. 10D to receive low-ordcr RCVD FLICK (3:0) signals corresponding successive ones of the first twelve flicks of a control network message paclcet 60, and t latch the flicks in respective njbbles 831(0) through 831(11) of the receive ~essage buffer 831. In addition, the RCV 0 through RCV 12 signals enable other circuitry to latch successive high-order RCVD FLICK (4) sigDals representing the tag bit in each flick and to direct the latched signals to appropriate destinations.

WO 92t06436 2 ~ ~ ~ 3 ~ Gr/US91/07383 The receive ti~ing generator circuit 891 includes a receive tuning control circuit 894 tha~
actually generates ehe RCV 0 through RCV 12 timing signals. If all of the RCVD FLICK (4:0) received flick signals are negated, a NOR gate 895 enables one input of an AND gate 896. If the receive t~ning control circuit 894 is asserting the RCV 0 timing signaL and if all of the RCVD FLICK
(4:0) signals are negated, AND gate 896 is energ~zed, which, in turn, energizes an OR gate 897 to assert a RST TO ST 0 reset to state zero signal. The RST TO ST 0 signal is coupled to a reset termirlal of the receive timing control circuit 894 to enable the circuit 894 to continue asserting ~he RCV 0 si~al.
The first flick of a control net vork message pac!~et is identified by at least one of the RCVD
Fl,ICK (4:0) sigDals being asserted when the receive timing control circuit 894 is asserting the RCV 0 receive zero signal. When that occurs, the NOR gate 895 disables the AND gate 895, in turn disabling the OR gate 897 and negating the RST TO ST 0 reset to state zero sig~al. This eoables the timing control circuit to begin stepping through the RCV 0 through RCV 12 signals iD sy~chronism with successive ticks of the NOI:)E CLK signal. It w ill be appreciated that the flick demultiplexer 830 will receive RCVD FLICK (4:0) signals representing all thirteen flicks of the control network message packet 60 as the receive timing control circuit 894 is stepping through the RCV 0 through RCV 12 sigllals.
When the receive timing control circuit 894 asser~s the RCV 12 signaL that signal cnergizes the OR gate 897 to again assert the RST TO ST 0 reset to state zero signal, whicb enables the circuit 894 to reset and assert the RCV 0 signal at the next tick of the NODE CLK s~gnal. At this point, if the RCVD F~ICK (4:0) signals are all negated, indicating tbat the first flick of a new control nenvork message packet 60 is not then being received, the NOR gato 895 is energized and it and the asserted RCV 0 signal maintain the OR gate 897 in the energized condition. The receive ~iming control circuit 894 thus maintains the RCV 0 signal in the asserted conditioo until at least one of the RCVD Fl,ICK
(4:0) signals is asserted.
As with traDsmissions of data nibbles 70(0) through 70(7) in successive flicks by the flick generator 815 as described above, the order of numerical significance of the received nibbles 70(0) through 70(7) of a control network message packet received by the flick demultiplexer 830 will depend upon whether the control network message packet 60 is a multiple sourcc message with the results of a maximum arithmetic operation. If not, the RCV REV DATA sigrlal is Degated, whirh enables a series of multiplexers 900(3) through 900(10) to assert SEL RCV 3 through SEL RCV 10 timing signals in the same sequence as the RCV 3 through RCV 10 timing signals. On tbe other hand, if the control network message packet 60 being received is a mul~iple source message with the results of a maximum arithmetic operation, the RCV REV DATA signal is asserted, which enables the multiplexers 900(3) through 900(10) to assert the SEL RCV 3 through SEL RCV 10 signals in the reverse order as the RCV 3 through RCV 10 receive timiDg signals.
The check circuit 892 includes t vo elements. A checksum check circuit 902 iteratively receives the RCVD FLICK (4:0) signals representing successive flicks of a control network message packet 60.
The checksum check circuit 902 is reset in response to the RCV 0 signal ftorn receive timing control , , i, : : '' : ' ' ~: ~ : :

~ WO 92~06436 PCI /US91/07383 2QQ33~ 7-circuit 894, aDd receives the successive flicks at each tick of the NODE CLK sigDal thereafter. In response to the RCV 12 signal, which is asserted at the point at which the RCVD F7 ICK (4:0) signals represent the checksluin portion 63 of the received message packet 60, if the checksum check circuit 902 detennines that the message packet was properly received, it asserts the CHECK OK signal.
A protocol check circuit 903 iteratively receives the low-order RCVD FLICK (3:0) signals in response to the RCV 0 through RCV 2 signals, that is, thc signals representing the flicks comprising packet header 61 of a control network mes~age packet 60, and determines whether the encodings correspoDd to those that are permissible in the particular system 10. If the protocol cbeck circuit 903 delerrnines that the encodings are permissible, it asserts a PROT OK protocol ok signal. In addition, the protocol check circuit 903 determirles whether the message packet 60 being received is of the multiple source type and has the result of a maximum arithmetic operation, and if so, it asserts the RCV REV DATA receive reverse data signal, which is coupled to XOR gate 900.
The low-order RCVD FLICK (3:0) received flick signals are also coupled to data illpUt terminals of a series of multiplexers 904(0) through 904(11) [generaUy identified by reference numeral 904(i)]. Each of the multiplexers 904(0) through 904(2) and 904(11) is controlled by one of the respective RCV 0 througn RCV 2 and RCV 11 receive liming signals from receive timing generator circuit 891. When the respective RCV 0 through RCV 2 and RCV 11 receive timing sigDals are asserted, the respective multiplexer 904(i) couples the RCVD FLICK (3:0) sigDals to input tern~inals of a respective latch generaUy identified by reference numeral 905(i), which latch the signals at the next tick of the NODE CLK signal. If the respective RCV ~in receive timing signal is Degated, the multiplexers 904(i) coupJe the latched signals from the respective latches 905(i) baclc to their input ter~ninals, to enable the latches uill maintain their con&tions at subsequent ticks of the NODE CLK
signal.
The multiplexers 904(3) through 904(10), OD the other hand, are controlled by the SEL RCV 3 through SEL RCV 10 selected receive ti~iDg sigrals from multiplexers 900(3) through 900(10). When the SEL RCV 3 through SEL RCV 10 signals are asserted, multiplexers 904(3) through 904(10) couple the RCVD FLICK (3:0) signals to input terminals of a respective latch 905(3) through 905(10), whicb latch the signals at the next tick of the NODE CLK signal. Accordingly, if the protocol check circuit 903 is negating the RCV REV DATA receive reverse data signal, the RCVD Fl,ICK (3:0) signals received during sequeDtial assertions of the RCV 3 t~rough RCV 10 signals will be latched iDto the successive latches 905(3) through 905(10). On the other hand, if the protocol check circuit 903 is asserting the RCV REV DATA receive reverse data signaL the RCVD ~LICK ~3:0) signals received dunng sequential assertions of the RCV 3 through RCV 10 signals will be latched into the successive latches in the reverse order, that is, into latches gO5(10) through 905(3).
The CHECK OEC and PROT OK signals from check circuit 892, along ~vith the RCV 12 signal are coupled to a receive strobe enable circuit 907. If, during receipt of the RCVD PL~CK (4:0).signals representing successive tlicks of the control net vork message packet 60, check circnit 892 asserting both the CHECK OK and the PROT OK sigDals, the RCV 12 signal enables the receive strobe enable ' ' ' ,' ! . :'. , , ' ~ ' ' ' ' ~ ' ' ' ' , ' ' ; ' , ' ' "' . " . ' ' ' . i ~ ' ' WO 92/06436 PC~/US91/07383 2~35~

circuit 907 to assert a LOAD ~CVD MSG load received message sigDal, which enables the receive message buffer to latch the signals coupled to its input tenninials by the latches 905(i). The received message buffer iB31 transmits the latched signals as RCVD MSG (47:0) received message signals to the destination control circuit 832.
The flick demulfiplexer 815 also includes a number of tag signal latch circuits 910(i), some of which are also depicted on Fig. 10D. The index ~i~ in reference numeral 910(i) corresponds to the index ~i~ of the receive tirning signal RCV ~i~ that is asserted when the particular tag sigDal is being received. The tag signal latch circuits are all similar, and so o~y circuit 910t2) that generates the RCVD BC FLOW received broadcast flow signal, circuit 910(4) RCVD SBC FLOW received supervisor broadcast flow signal aDd circuit 910(0,3,6,8,10) that generates the RCVD SCAN FLOW
received scan flow signal, which are used to control Ihe traDsmit flick generator 815 (Fig. 10C), are depicted ;D Fig. 10D.
As shown in Fig. 10D, each tag signal latch circuit 910(i) includes a multiplexer 911(i), which receives at one input terminal the RCVD FLICK(4) received flick signal representing the tag bit of each flick. Each multiplexer 911(i) is controlled by the corresponding RCV ~i~ recei~e timing signal to couple the RCVD FLICK (4) signal to the data input tenninal of a flip-aop 9~ 2(i). The tag signal latch circuit 910(0,3,6,8,10) also includes an OR gate 913 that iasserts a RCV (0,3,6,8,10) signal when any of the RCV 0, RCV 3, RCV 6, RCV 8, or RCV 10 signals are asserted to then enable multiplexer 911(0,3,6,8,10) to couple the RCVD FLICK (4) sigDal to the data input terminal of flip-flop 912(0,3,6,8,10). With re~erence to Fig. 5, it will be appreciated that, when the RCV 0, RCV 3, RCV 6, RCY 8, and RCV 10 signals are asserted, the RCVD FLICK (4) signal representiDg the flic~s received at those points irl time will represeDt the scaD flow bits 72(1) through 72(5), which is latched in tag signal latch circuit 910(0,3,6,8,10).
Each fli~nop 912(i) is clocked in response to the NODE CLK signal. Thus, the tag signal as latched by each tag signal latch circuit 910(i~ is updated at the next tick of the NODE CL~C signal after the assertion of the RCV ~i~ signal. The output signal from each flip-flop 912(i) is ~oupled to the other data input terrninal of the associated multiplexer 911(i), wbich couples that signal back to the data input terminal of the flip-flop 912(i) wheD the RCV ~i~ receive timing signal is not asserted.
Accordingly, each tag signal latch circuit 910(i) is updated at the Dext tick of the NODE CLK signal following the associated RCV ai~ receive timing signiaL and maintai~s the state until the next update.
The destinadon control circuit 832, when an control network message packet 60 is received, identifies the pardcular FIFO 833, i834 or 835, or the one of registers 804 into which the contents of the receive message buffer 831 are to be stored, and enables the contellts to be stored therein. With reference to Fig. 10E, the destination control circuit 832, includes a destination decode circuit 914 which receives the RCVD MSG (3:0) and RCVD MSG (7:4~ received message signals f~om the receive message buffer 831. These signials reprcsent tbe contellts of, respectively, the message type field 64 and packet type field 65 of the received control network message packet 60.
ID response to tbe assertion of the LOAD RCVD MSG load received rnessage signal from tbe receive strobe enable circuit 907, the destination decode circuit 914 uses these RCVD MSG signals to - . - : :: ......... , . ... . , .. . , , ; . . ".: , ' ': : ~' , ' , : ' !.' ~; '. , WO 92/06436 PCr/U~;91/07383 2 ~ S -log-select for assertion one of a PUSH RCY BC FIFO push receive broadcast first-in first-out buffer sigDal, a PUSH RCV SBC FIFO push receive supervisor broadcast first-in first-out buffer signal, a PUSH RCV COM FIFO push receive combine first-in first-out buffer signal and a STORE INTR
REG store interrupt register signal. The asserted signal enables the contents of nibbles 831~3) through 831(10) of received message buffer 831, which correspond to ,~he contents of data nibbles 70(0) through 70(7) of the received message packet 60 aDd which are represented by the RCVD MSG (43:12) received message signals, to be stored in oDe of the FIFO 833 through 835 or in interrupt register 822.
In addition, the destination decode circuit 814 also generates a STORE GLOBALS signaJ that enables the globals register 820 to latch the RCVD MSG (47:44) received message signals. These signals represent the conteDts of Dibble 831(11) of received message buffer 831, which, in turn, correspond to the contents of the global information nibble 71 of the receive control network message packet 60.
iv. Root Control/Loopback Section 802 Figs. 10F and 10G depict detai~s of Ihe root control/loopback section 80 '. ID particular, Fig.
10F depicts a detailed logic diagrarn of the root flag control circuit 817. With reference to Fig. 10F, the root flag control circuit 817 includes three general portions, including a decoder circuit 920, a root establishment timing circuit 921 and a root delction timing circuit 922. The decodes circuit 920 determines whether the control network interface is transmitting a control network message packet 60 of the sirlgle source message type and of the configuration packet type, and, if so, determines whether the height value contained therein identifies the control network interface 204 or a control network slode 50(i,3) as a logical root.
If the decoder circuit 920 determines whether the control network interface 204 h transmitting a control network message packet 60 to establish the control network interface 204 as a log~ical root, and if the control network interface 204 is not then a logical root, the root establishment timing circuit 921 establishes the control Detwork interface 20~, as a logical root, and asserts the ROOT UP sigDal.
Thereafter, the receive section 801 is enabled to receive subsequent control network message packets 60 from the transmitter section 800 of the control netwosk inter&ce 204. On the other hand, if the decoder circuit 920 determines that the control Detwork interface is transmittiDg a control network message paclcet 60 to establish a control net vork node 50(i,j) as a logical root, and if the root establis~nent timing circuit 921 is currently asserting the ROOT UP sigDal, the root deledon timing circuit 9æ enables the root establishrnent tirning circuit 921 to negate the ROOT UP signal, so that the receive sectioD ~11 receive subsequent control network message packets 60 from the control Detwork node 50(1,j) connected thereto. The root establishrnent timing circuit 921 and the root deletion timing circuit gæ operate so as to ensure that the ROOT UP signal is asserted and negated at control network message packet boundaries, that is, after the receive section has finished receiving a coutrol network message packet 60 that it is then receiving. This ensures that the receive section 801 does Dot misinterpret the FLICK IN (4:n) signals representing successive flicks of a control network message packet 60 that it is receiving.

:: , . : , . :., : : : , ': ' . ; ' :: . . ! . , '' ;

If the CN MSG (3:0) signals indicate that the control network message packet is of the single source message type and the CN MSG (7:4) signals indicate that the packet is of the configuration type, and if the CN MSG (16:12) signals have have a binary-encoded value of zero, the control network interface 204 is to become a logical root. The decoder circuit 920 includes an inverter 923 that receives the CN
MSG (16:12) signals, representing the contents of the first nibble 70(0) (Fig. 5) and one bit of the second nibble 70(1) of the packet data portion 62 of the message packet 60 to be transmitted. The inverter 924 couples complements of these signals to input terminals of an AND gate 924. If all of the CN MSG (16:12) signals are negated, which will occur if they have the binary-encoded value of zero, the AND gate 924 asserts a ROOT HT 0 root height zero signal.
The decoder circuit 920 also includes an OR gate 928 that also receives the CN MSG (16:12) control network message signals and asserts a ROOT HT NE 0 root height not equal to zero signal if at least one of the CN MSG (16:12) signals is asserted. It will be appreciated that, if at least one of the CN MSG (16:12) signals is asserted, the binary-encoded value thereof is not zero. Accordingly, if the CN MSG (3:0) and CN MSG (7:4) signals indicate that the control network message packet 60 is of the single source message type and the configuration packet type, the control network message packet 60 is establishing a control network mode 50(i,j) as a logical root. If that occurs while the control network interface 204 is a logical root, the root flag control circuit 817 will delete, that is, disestablish, the root condition at the control network interface 204.
It will be appreciated the the CN MSG (16:12) signals may all be negated in connection with control network message packets 60 having message types other than single source or packet types other than configuration. Accordingly, the decoder circuit 920 includes a decoder 925 that receives the CN MSG (3:0) control network message signals and the CN MSG (7:4) signals, which identify the message type and the packet type, respectively. If the ROOT HT 0 signal is asserted, and the CN MSG
(3:0) and CN MSG (7:4) signals identify the single source message type and the configuration packet type, the decoder 925 asserts a CNI ROOT HT control network interface root height signal in response to the assertion of the XMIT 0 transmit flick zero timing signal.
The CNI ROOT HT signal is coupled to the root establishment timing circuit 921, specifically to an input terminal of an AND gate 927. The second input terminal of AND gate 927 is controlled by a LEAF ST leaf state signal from the root deletion timing circuit 922. If the LEAF ST signal is asserted, the control network interface 204 is not then a logical root. In that case, the asserted CNI
ROOT HT signal energizes AND gate 927, which, in turn energizes an OR gate 930, enabling it to assert a CNI BECOME ROOT control network interface become root signal. THE CNI BECOME
ROOT signal is coupled to the data input terminal of a flip-flop 9312, which is set in response to the next tick of the NODE CLK signal to assert a CNI BECOME ROOT ST control network interface become root state signal.
The CNI BECOME ROOT ST control network interface become root state signal is coupled to one input terminal of an AND gate 932. While the RCV 12 receive flick twelve signal is not asserted, the other input terminal of AND gate 932 is enabled, and the assertion of the CNI BECOME

WO 92/06436 - PCl/US91/07383 20933r~
ROOT ST signal causes the AND gate 932 to be energized. While the AND gate 932 remains energized, the OR gate 930 also remains energized to, in turn, maintain the CNI BECOME ROOT
signal in its asserted condition.
When the RCV 12 receive timing signal is next asserted, it and the asserted CNI BECOME
ROOT ST signal energize an AND gate 933, which, in turn, energizes an OR gate 934 to assert a WAIT FOR XMIT wait for transmit state signal. It will be appreciated that this assertion of the RCV
12 receive timing signal occurs when the receive section 801 is receiving the FLICK IN signals representing the last flick of the control network message packet it is then receiv~ng. The WAIT FOR
XMIT signal is coupled to the data input terminal of a flip-flop 935, which is set at the rlext tick of the NODE CLK signal to assert a WAIT FOR XMIT ST wait for transmit state signal. The assertion of the RCV 12 signal also disables AND gate 932, which, in turn, disables OR gate 930 to negate the CNI
BECOME ROOT signal. The negated CNI BECOME ROOT signal resets ~he flip~flop 931 at the same tick of the NODE CLK signal, which, in turn, negates the CNI BECOME ROOT ST signal.
With reference to Fig. 10G, which depicts circuitry in the flick buffer 816, the asserted WAIT
FOR XMIT ST wait for transmit state signal generated by flip-flop 935 eilergizes an OR gate 936 to assert the SEL XMIT select transmit signal. As described above in connection with Fig. 10D, the assertion of the SEL XMlT select transmit signal enables the multiplexer 890 to receive and use the BUF FLICK OUT buffered flick out signals from the tlick buffer 816, instead of the FLICR IN signals from the control network rlode 50(1,j) connected thereto. At this point, the BUF FLICK OU T signals to the receive section 801 are all negated, and so the flick buffer 816 is essentially providing the equivalent of control net,vork message packets of the r~il packet type, as described above. However, it ~ill be appreciated that, since this occurs in response to the RCV 12 signai, this operadon occurs only after the receive section has firushed receiving a complete control network message packet 60 from the cor~trol network node SO(lJ) cosnected thereto.
ReturDing to Fig. 10F, the WAIT FOR XMIT ST wait for transmit state signal from fli~flop 935 is coupled to two AND gates 537 and 940. If the WAl r FOR XMIT ST si~al is asserted, and if the XMIT 12 transmit timing signal is negated, which is ~he case before the transmit sec~ion 800 has finished transrnitting a control network message packet 60 it is cusrently transmitting9 the AND gate 937 is energized, which in turn energizes the OR gate 934 to maintain the WAIT FOR XMIT wait for transmit signal asserted. It will be appreciated that, at this po-nt, the control network message packet 60 being transmitted by the transmit section 880 is the configuratioD packet that identifies the control network interface 204 as the logical root.
Wben the transmit section 800 is transmitting the last flick of the control network message packet 60, it asserts the XMIT 12 signal. The coincidence of the asserted XMIT 12 sigDal and the asserted WAIT FOR XMIT ST wait for transmit state signal enesgize an AND gate 940, which, in turn, energ~zes an OR gate 941 to assert a ROOT signal. The ROOT signal is coupled to the data input terrninal of a flip-flop 942. At the next tick of the NODE Cl,K signal, flip-flop 942 is set, to assert a ROOT ST root state signal. A driver 943, connected to the data output terminal of the nipflop 942 ;.; ;. , , , ........... . :

:: : , . :: ~.: . ::. :: . ; :

WO 92/064~6 PCrlUS~l/07383 2~3~a receives the ROOT ST signal and couples it as the ROOT UP signal to the control network node SO(lJ) connected to ~be control networl~ interface ~04. The asserted XMIT 12 signal also disables AND gate 937, which, in turD, disables OR gate 934, causing it to negate tDe WAIT FOR XMlT signal.
The negation of the WAll FOR XMIT signal, in turn, causes the flip-flop 935 to he reset in response to the same tick of the NODE (`LK signal, negating the WAIT FOR ~IIT ST signal.
With reference again to Fig. 10G, the asserted ROOT ST signal maintairls the OR gate 936 in the energized condition, which, in turn, mainta~ns the SEL XMIT select traDSmit signal in the asserted condition. Accordingly, the multiplexer 890 in the flick demultiplexer 830 of the receive section 801 continues to couple the BUF E;LICK oUr buffered flick out signals from the flick buffer 816 as the RCVD FLICK (4:0) received flic~ sigllals. The asserted ROOT ST signal also energizes an OR gate 944 to assert a GAT~ FLICK signal, which enables a gate 945 to couple the FLICK OU r (4:0) signals from the transmit section 800 as GATED FLICK (4:0) signals to the data input tenninaLs of a buffer 946. The buffer 946 latches the GATED FLICK (4:0) signals in response to each tick of the NODE
CLK signaL and provides at its data output terrninals the BUF FLICK o~rr (4:0) buKered flick out signals that are coupled to the receive section 801.
It will be appreaated that this occurs as the transmitter section 800 is beginning transmission of a new control net~vork message packet 60. Accordingly, the receive section 801 will begin receiving a control network message packet 60, other than the packets of the nil packet type,:from the transmit section 800 at the first flick of the respective packet. As noted above, the receive section 8~1 can identify the first nick as the first flick for which tbe RCVD FLICK r,ignals are not aD negated.
Returaing to Fig. 10F, ar, noted above, if the transmit message buffer 814 contains a control network message packct 60 of the single source message type and configuration packet type which identifies a height other than zero as the root level, the decoder 925 asserts the CNI NE ROOT HT ;^`
control network iuterface not equal to root height signal, and otherwise negates tbe signal. If the CNI
NE ROOT HT signal is negated, an inverter 950 enables one input of an AND gate 947. If the flip-flop 942 is asserting the ROOT ST signal, the AND gate 947 is energized, which maintains OR gate 941 in an energized condidon and the ROOT signal asserted. The asserted ROOT rignal, in turn, maintains the hip-flop 942 set and the ROOT ST and ROOT UP sig~als ar,serted. While the ROOT ST signal is asse~ted, the eDergized OR gate 936 (Fig. 10G) maintains the SEL ~CMlT signal asserted and the gate 945 enabled so that the receive section 8~1 receives the ~;LICK OUT signals from the traDsmit section 800 representing the successive flicks of control network message packet.
When the decoder 925 asserts the CNI NE ROOT HT signaL the coincidence of that signal and the asserted ROOT ST signal energize an AND gate 951, which, iD turn, energizes an OR gate 952 to assert a LEAVE ROOT signal, which is coupled to the data input terminal of a flip-flop 953. The asserted LEAVE ROOT signal causes the flip-flop 953 to be set at the next ~ick of the NODE CLK
signal, enabling the flip-flop 953 to assert a LEAVE ROOT ST leave root state signal.
Contemporaneously, the asserted CNI NE~ ROOT HT signal causes inverter 950 to disable AND gate 947, which, in turn, disables the OR gate 941, causing it to negate tbe ROOT sigllal. The negated ROOT signal enables the nip nOp 942 to reset at the same tick of the NODE CLK signal.

r: ~ , ' ; ' ., :,' ' ' ', ', ~ . ' . ;
;

20~335~
The asserted LEAVE ROOT ST leave root state signal also maint~uns the OR gate 936 (Fig.
lOG) energized to assert the SEL XMlT select transmit signal and OR gate 944 energized to enable gate 945. Accordingly, while the LEAVE ROOT ST signal is asserted, the receive section 801 continues to receive and use BUF FLICK OUT (4:0) buffered flick out signals representing successive flicks of a control network message packet.
The LEAVE ROOT ST leave root state signal will remain asserted until the receive section 801 asserts the RCV 12 signal, indicating it has received BUF FLIC~C OU r (4:0) buffered flick out signals representing the last flick of a ~ontrol network message packet 60. In particular, prior to assertion of the RCV 1~ signaL the coincidence of the negated RCV 12 signal and the asserted LEAVE
ROOT ST leave root state signal maintain an AND gate 954 in an energized condition, which, in turn, matntains OR gate 952 energized to assert the LEAVE ROOT signal. As above, the while the LEAVE
ROOT signal remains asserted at successive ticks of the NODE CLK signal, the flip-flop 953 remains set to maintain the LEAVE ROOT ST signal asserted.
However, when the receive section 801 asserts the RCV 12 signal, the AND gate 954 is disabled, which, in turn, disables the OR gate 952, causing it to negate the LEAVE ROOT signal. The negated LEAVE ROOT signal causes the FLIP-FLOP 953 to be reset, which, in turn, negates the LEAVE ROOT ST signal. The negated LEAVE ROOT ST signal, in turn, disables OR gate 936, enabli~g it to negate the SEL XMl T select transmit signal. The negated SEL XMIT signal enables the multiplexer 8g0 of flick demultiplexer 830 to begin coupling the FLICK IN (4:0) signals from the control network node 50(1,i) connected thereto as the RCVD FLIC:K (4:0) received flidc signals. Prior to this point, while the control ~etwork interface 204 had been assertiDg the ROOT UP signal, the control network node SO(lJ) had been transmitting FLICK IN (4:0) si~als all of which were negated, essendally transmitdng control network message packets 60 of the nil packet type. After negation of the ROOT UP s;gnal, the control nehvork node 50(1,~) ~egins transmitting control network message packets 60 of other message types. As described a'oove, the flick demultiplexer 830 identifies the first flick of one of these control network message packet 60 as that is the first flick for which the RCVD
FLICK (4:0) sig~als are not all negated.
In addidon, the negation of the LEAVE ROOT ST leave root state stgnal also disables OR
gate 944, causing it to ne8ate the GATE FLICK signal to disable gate 945. Thereafter, until the GATE
FLICK signal is agairl asserted in response to assertion of the ROOT ST root state signaL the GATED
FLICK (4:0) signals latched by the buffer 946 will all be negated.
Retun~in8 to Fig. 10F, the assertion of the RCV 12 reuive tisning signal while the flip-flop 953 is asserting the LEAVE ROOT ST signal also energizes an AND gate 955. The energized AND gate 955, in turn, energizes asl OR gate 95o which, in turn, asserts a LEAF signal, wbich is coupled to the data input te~minal of a nip-nop 957. The flip-flop 957 is set at the De~a tick of the NODE CLK signal, enabling it to assert a LEAF ST signal. As noted above, thc LEAF ST sigDal controls one input tenninal of AND gate 927.
The asserted LEAF ST signal also enables one input terrninal of an AND gate 961, whose other input tenninal is controlled by the CNI EQ ROOT Hl` control network interface equals root : .: , , ,., , , ,: -;, ,: . , - ' " , ' '' ' '~.' . ''~ .,J ' ., ' ' '', '', " '. ' "

'' , ' '', 1 ~ ' ' . :

WO 92/064?36 PCT/US91/07383 2~33~
height sig~al through an inverter 960. While the LEAF ST signal is asserted and the C?all EQ ROOT
sIT sigDal is negated, the AND gate 961 remains energized, in turn enabling the OR gate to remain energized to maintain the LEAF signal asserted. While the LEAF signal is asserted, successive ticks of NODE CLK signal maintain the flip-flop 957 in the set condition, maintaining? in turn? the LEAF ST
signal asserted. With the LEAF ST signal in the asserted conditiofi, when she CNI EQ ROOT HT
signal is asserted, the AND gate 927 can be energized to initiate the sequence described above.
Assertion of the CNI EQ ROOT HT signal also, ~hrough invener 9~, disables AND gate 961, which? in turn, disables OR gate 956 causing the LEAF signal to be negated. The negation of the LEAF signal causes the tlip-tlop 957 to be reset, which, in turn? causes the LEAF ST signal to be ?uegated at the Dext tick of the NODE CLK signal. Accordingly, the AND gate 927 is disabled? by negation of the LEAF ST signal, on tick of the NODE CLK signal after the CNI EQ ROOT HT signal is asserted? and is not enabled again uDti] the LEAF ST signal is again asserted. It will be appreciated that the AND gate 927 is provided so that, if the control network interface 204 is already a logical root, which will be the case if the LEAF ST signal is negated, the rest of the root establishment tirning circuit 921 will be inhibited from sequeDcing if the CNI EQ ROOT HT signal again is asserted without interveniDg assertion of the CNI NE ROOT HT signal, as the sequenciDg may cause a momentary glitch or noise in the ROOT UP signa~. ~*

. '' ', ,~ ' ~' . , ' ' - ~ .

- . : / ., ` . . . .
: : .

WO 92/0~36 PCr/US91/07383 2 ~ 3at'a~ R30~ltçr Node 22 -115-1. General Fig. 1LA is a general block diagram of a data router node 22 used in the data router desaibed above, and Figs. 11B-1 through 11D comprise detailed block and logic diagrams of the data router node 22. With reference to Fig. 11A, the data router node æ includes a child interface 1001, a parent interface 1002 and a sw~tch 1003, all controlled by a node control circuit 1004. The data router node also rcludes a diagr4Ostdc network interface 1005, which provides an interface to the diagnostic ner.vork 16. In additior4, the data router node 22 includes a clock buffer 1008 that receives the SYS CLK system clock signal from the clock circlut 17 and generates a NODE CLK node clo k signal in response. In one particular embodimeDt, the clock buffer 1008 comprises a buffer as described iD aforemendoned Hillis, et al., Patent Appn. Ser. No. 07/4~9,079, filed March 5,1990, entitled Digital Clock Buffer Circuit Providir g ControUable Delay.
The child interface 1001 includes a set of child interface modules generally identifJed by reference numeral 1001(i) (nin being an integer). Each child interface module 1001(i) is connected to a child in the data router 15, which may comprise either a leaf 21, iD the case of a data router node 22(1J,k) in the first level of the data router 15, or a data router node 22(i-lJ,k) that forms part of a child data router node group 20(i-1J) in the case of data router nodes at higher levels. Each child interface module 1001(i) receives data router message packets 30 from the cnild connected thereto and couples them to the switch 1003. In addition, each child interface module 1001(i) receives data router message packets from the switch 1003 and couples them to the child connected thereto~ It vti44 'oe appreciated tnat the number of child interface rnodu4es 1001(i) in a c'nild interface 1001 of a particular data router node 22(i,j,k) ~wi44 general4y depend upon the farn-out from one level "i" to the next lower level ~i-l" in the data router lS. In the system 10 descri'oed herein, the fan-out is four, and tne clhi4d interface 1001 depicted in Fig. 1h~ includes four cni4d interface modules 1001(0) througlh 1001(3).
The parent interface 1002 also includes a set of parent interface modu4es general4y identified by reference nunmera4 1002(i) (ni" being an integer). In data router nodes Z(iJ,k) at levels below the level of the root data router node group 20(M,0), each parent interface module 1002(i) is cormected to a data router node æ(i+lJ,k) that fonns part of a parent data router node group 20(i+1J). In the case of data router nodes æ(M,O,k) in the root data router node group 20(M,0), those nodes do not require any parent interface modules 1002(i). Each parent interface module 1002(i) receives data router message packets 30 &om the parent, if any, connected thereto and couples them to the switch ;~
1003. In addition, each parent interface module 1002(i) receives data router message packets fJom the switch 1003 and couples them to the parent, if any, connected thereto. The number of parent interface modules 1002(i) in a parent interface 1002 of a particular data router node 22(iJ,k) wil4 generally depend upon the fan out &om one level ~i" to the next higher level ~i+1" in the fat-tree defining the data router 15. In the system 10 described herein, the fan-out at some levels is t vo ard at other levels is four. The parent interface depicted in Fig. 11A includes four parent iuterface modules 1002(0) through 1û02(3), which wi44 accommodate a fan-out of four.

, . - . , , . :. .: ,,, :
. . .: .: , : ; " , ~ , . :,.:, .

WO 92/06436 PC~/US91/07383 2~35~

lt will be appreciated that in other embodiments of system 10, the fan-out, both up and down the fat-tree, may be different, in which case different numbers of child interface modules 1001(i) and parent interface modules 1002(i) may be provided in a particular data router node 22(i,j,k). In addition, if, as is the case with one emboditnent, all of the circuitry for the data router node 22(i,j,k) is fabricated on a single integrated circuit chip, the circuit may include a number of child interface modules 1001(i) and parent interface modules 1002(i) corresponding to the maximum fan-outs in the system 10. In that case, particular ones of the child interface modules 1001(i) and parent interface modules 1002(i) that are not connected to child or parent data router nodes æ(iJ,k) may be disabled.
The s~,vitch 1003 receives data router message packets 30 from the child interface modules 1001(i) and the parent interface modules 1002(i). In the case of a data router rnessage packet from a child interface module 1001(i), the switch 1003 may transmit the message packet either to a child ..
interface module 1001(i) or to a parent interface module 1002(i). If the header field 40 of the data router message packet 30 contains a value that corresponds to the level of the data router node ætiJ,k), or if the AFD MODE signal indicates that the data router is in all-faD-down mode, the s vitch 1003 will direct the packe~ 30 to a child interface module 1001(i). Othenvise, the switch directs the message packet 30 to a parent interface module 1002(i). Alternatively, in the rase of a data router message packet 30 from a parent inter~ace module 1002(i), the switch 1003 wiD transmit the message packet 30 to a child interface module 1001(i).
The node control circuit 1004 receives selected sigDals from and generates various control sigr als in response. For example, the node control circuit 10~4 receives binary-encoded HEIGHT
(2:0) signals which identify the level of the data router node 22(iJ,k) and generates DECR HGT
decremented height signals which are binary eacoded to identify the ne~ lower leYel in the data router 15. As described above, as the data router nodes 22(iJ,k) transfer the data router message pacl~ets 30 down ihe data router 15, the nodes decrement the value contained in the header field 40, which identifies the le/el. ID that case, the D~CR HGT decremented height sigDals are used to form the contents of the header field 40, as described below.
ln addition, the node coDtrol circuit 1004 receives the AFD (iJ) all-fall-do vn (iJ) signal from the control network 14 and generates, i~ response thereto, an AFD MODE aD-fall-down mode signal which controls operations in the child interface modules 1001(i) and the parent interface modules 1002(i) as described below. The node control circuit also generates an EN enable signal, which enables the data router node 22(i,j,k) to operate, and P3:P0/C3:C0 DIS parent interface module/child interface module disable signals that disable selected ones of the child interface modules 1001(i) and parent interface modules 1002(i).
The node cor~trol circuit 1004 also generates a set of CHILD MAP signals which are coupled to the cniid interface modules 1001(i) and parent interface modules 1002(i) and are used to force the association of each of the modules 1001(i) and 1002(i) with a particular the child interface module lOOl(i) during while the AFD MODE all-fall-down mode signal is asserted. This forces the switch 103 to couple data router message packets 30 received from a particular source, a particular parent or child, to a particular child interface module 1001(i).

, :, ~
- - ; ~ , :
. .- . . : . ..
:

2~33~ -117-Finally, the node control circuit 10~4 also transmits selected error signals, represented by an OUT ERROR signal on Fig. 11A, to the diagnostic network 16 if it detects the presence of selected error conditions.
Each child interface module 1001(i) includes an input child circuit, generalJy identified by reference numeral 1006(i), and arl output child circuit generally identified by reference numeral 1007(i~. The input child circuit 1006(i) transmits a C~i~ IN FLY child ~i~ input ny sigDal to the child connected thereto, and receives C~i~ IN FLIT child ~i~ input flit signals, comprising four sigDals received in parallel. The Ci~ IN FLlT signals received at succcssive ticks of the NODE CLK signal represent four-bit flits of the data router message packet 30 from the child connected thereto.
The input cDild circuit 1006(i), in respoDse to the message address portion 31 of the message packet determines whether it is to be transmitted up the tree or down the tree defining the data router 15. If the input child circuit 1006(i) determines that the data router message packet 30 is to be transmitted up the tree, it enables the switch 1003 to direct the successive flits cornprising the pacl~et 30 to a parent interface module 1002(i) selected by the switch 1003 at random. On the other haud, if the input child circuit 1006(i) determines that the data routcr message packet 30 is to be transrnitted down the tree, it ideDtifes one of the child interface modules 1001(i) to which the swi~ch 1003 is to direct the packet 30. The swi~ch 1003 then directs the successive flits of the message packet 30 to the output child circuit 1007(i) of the ideutified c~ild irterface module 1001(i). :
Each output child circl~it 1007(i) receives a Ci~ OUT FLY child ~i~ output fly signal from the child cormected thereto, and transrDits thereto Cni~ OUT FLIT child "i" output flit signals, comprising ~our signals transmitted in parallel. The cni" IN FLIT child ";~ input flit signals traDsmitted at successive ticks of the NODE CLK signal represent four bit flits of the data router message packet 30 transmitted to the uhild cor,nected thereto.
Each parent interface module 1002(i) includes an iDput pareut circuit, geDeraDy identified by reference numeral 1010(i), and an output pareDt circuit generally identified by reference numeral lOll(i). The input pareDt circuit 1010(i) transmits a P~ IN FLY parent tj~ input fly siglaal to the parent connected thereto, and receives P'i~ IN FLIT parent ~i~ input nit signals, comprising four signals received in parallel. It ~ill be appreciated that the Cui~ O~lT FLY child ~i" output fly and Ci~ OUT
FLIT child ~i" output flit signals described above in connection vith an output child c*cuit 1007(i) in a data router node 22(i,i,k) at one level "i~ will correspond to the Pni" IN FLY parent ~i" input 9y and P~i~
IN FLIT parent "i~ input flit signals of input parent circuits 1010(i) ;D the child data rGUter Dodes 22(i-lJ,k) in the next lower level.
The il~pUt parent circuit 1010(i), in response to the message address portion il of the message packet, identifies one of the cbild iDterface modules 1001(i) to which the swi~ch 10a3 is to direct the packet 30. The switch 1003 then directs the successive nits of the message packet 30 to the output child circuit 1007~i) of the ide~tified child interface raodule 1001(i). It will be appreciated that, i~ the system 10 described herein, data router message packets 30, once they have been transmitted up the tree defined by the data router 15 to the level identified in the message packet 30 and have started down the WO 92/06436 PCi/US91/073R3 -118- 2~333~
tree, are not thereafter transmitted up the tree again. Accordingly, message packets 30 coupled by input parent circuits 1010(i) to switch 1003 are directed only to an output child circuit 1007(i), and not to an output parent circuit 1011(i).
The output parent circuits 1011(i) operate iD a similar manner as the output child circuits 1007(i). Each output parent circuit 1011(i) receives a P"P OUT FLY parent ~P output fly signal from the parent connected thereto, and transmits thereto P~P OUT FLIT parent ~P output flit signals, comprising four signals transmitted in parallel. The p~in IN FLIT parent "i~ input flit signals transmitted at successive ticks of the NODE CLK signal represent four bit flits of the data router message packet 30 transmitted to the parent cormected there~o. It wiD be appseciated that the CP IN
FLY child ~P input fly and CP IN Fll'r child ~i~ input flit signals described above in connection vith an input child circuit 100o(i) in a data router node 22(iJ,k) at one level ~;n will correspond to the p~;" OUT
FLY parent ~i~ output fly and p~;" OUT FLIT parent ~P output flit signals of output parent circuits lOll(i) in the parent data router nodes æ(i+ lJ,k) in the next higher level.
.. .. . . . ............................................... .
Thus, i~ will be recog~zed tha~ ~he inpu~ child ciscuit 1006(i), which receives the CP IN FLIT
child inpu~ flit signals representing nitS of data router message packets ~0, regulate the flow of flits thereto by meaDs of the CP IN FLY child input fly signal. When the input child circuit 1006(i) circuit negates the C~i" IN FLY signal while the child data router node 22(i-1,j,k) connected theseto is transmitting a data router message packet 30, it stops transmitting signals, which the input child circuit 1006(i) receives as the CnP IN FLlT signals, representing the nitS. The child data router node 22(i-1,j,k) effectively provides negated C"i' IN FLIT signals.
When the input child circuit 1006(i) again asserts the C"i" IN FLY signal, the child data router node æ(i-lJ,k) resumes tran~mitting signals, which the input child circui~ 10Q6(i) receives as the Ci~
IN FLIT signals, which, a~ successive ticks of the NODE CLK signal, represent successive nitS of the packet 30. Between data router message packets 30, the child data router node æ(iJ,k) negates the C"i" IN FllT signals, regardless of the condition of the C"P IN FLY signal. Accordingly, the input child circuit 1006(i) can identUy the first flit of a new data rou~er message packet 30 as the first tick of the NODE CLK signal following the end of a previous packe~ at which the CP IN FLIT signals are not all ne8ated.
Sirnilarly, the inpu~ paren~ circui~ 1010(i), which receives the P~P IN FLIT parent input flit signals representing nitS of data router message packets 30, regulate the flow of flits thereto by mean of the P~i~ IN FLY parent inpu~ fly sig~al. When the inpu~ parent circuit 1010(i) circuit negates the rP
IN FLY signal while the parent data router node ~(i+lJ,k) connected thereto is transmitting a data router message packet 30, the parent data router node 22(i + lJ,k) stops transmittiDg signals, which the input paren~ circui~ 1010(i) receives as the pnp IN FLIT signals, represen~ing ~he ~its. The parent data router node æ(i+ lJ,k) effectively provides negated P~P IN FLlT signals.
When the input parent circuit 1010(i) again asserts the P~P IN FLY signal, the parent data router node 22(i + lJ,k) resumes transmitting signals, which the input parent circuit 1010(i) receives as the }'~P IN FLlT signals, which, at successive ~icks of the NC)DE (::LK signal, represent successive flits , . ~, " ;, . . . . , , .

WO 92/06436 PCl`/US91/0738.~
2a933~'~

of the packet 30. Between data router message packets 30, ~he parent data router node æ(i+lJ,k) negates the P~i~ IN FLlT signals, regardUess of the conditiorl of the ri~ IN FLY signal. Accordingly, the input parent circu~t 1010(i) can identify the first flit of a tlew data router message packet 30 as the first tick of the NODE CL~C signal foUowing the end of a previous packet at which the P~i~ IN FLIT
signals are not aU negated.
The parent and child input circuits 1006(i) and 1010(i) are generally similar, as are ~he parent and child output circuits 1007(i) and 1011(i). The details of child input circuit 1006(0) will be described in connection with Figs. 11B through 11B-3A. The details of the switch 1003 wiU be described u connection with Figs. 11C-1 through 11C-6. Finally, the details of output child circuit 1007(0) will be described in connectioD with Fig. 11D.
2. Input Child Circuit 1006(0) i. General ~ ig. 11B depicts a general block diagrarn of input child circuit 1006(0). With reference to Fig.
11B, the input child circuit 1006(0) includes an interface circuit 1020, an input message control circuit 1021, an input message first-in first-out buffer (I:IFO) 1022, an output request identification FIFO 1023 and a s vitch input control 1024. The interface circuit 1020 receives a VAL FLOW valid flow signal from the input message control circuit 2021, and transmits in response the C0 IN FLY input fly signal to the parent output circuit 1011 connected thereto. :
The interface circuit 1020 also receives the C0 IN FLlT input flit signal from the sarne parent output circuit 1011 and couples, in response thereto, FLlT signals to the iDput message control circuit 1021. The VAL FLOW valid flow signal provided by the input message coDtrol circuit 1021 operates as a flow control signal regulatiDg the flow of flits represented by tbe FLIT signals from the interface circuit 1020 to the input message control circuit 1021. The iDterface circuit 1020 prov;des a VAL FLlT
signal to the input message control circuit 1021 to indicate that the FLIT signals represent valid flits of a data router rnessage packet 30, or the binary-encoded value of zero if the interface 1020 is not receiving a message packet 30.
The input message control circuit 1021 performs a number of operations. In particular, input message control circuit 1021 monitors the Fl,lT signals from the interface to detect the beginning of a data router message packet 30. In respoDse to the Fl,IT signals representing successive flits of a data router message packet 30, the input message control circuit 1021 couples IMF FLlT input message FIFO flit signals represeDtiDg successive ~its to the input message } IFO 10æ.
In providing IMF F'~lT input message E~IFO flit signals to the input message FIFO 1~22, the input message control circuit 1021 also performs some processing on the n.rr signals representing the first two flits of a data router message packet. In addition, the input message control circuil 1021 determines whether the data rou~er message packet is to be transferred to a parent or to a child data router node 22(i,j,k), and if to a child the particular child. The input message control circuit 1021 generates OUT REQ [P,C(1:0)l output request (parent, child) signals, which it transfers to the output request identification FlFO 1023, along with an OIF PUSH output identification FIFO push signal.

~... . . ~ .- : . .
:.-, , . . ,. , ,. . ,, ~ . .

; ~ . i'.: , . ! , WO 92/06436 PCl/US91/07383 2a93~
-12~
When the input message control circuit 1021 asserts the OIF PUSH signal, the output request identification FIFO 1023 stores the OUT REQ [P,C(l:O)l signals.
In processing the first two flits of the data router message packet 30, if the AFD MODE all-fall-do vn mode signal is not asserted, the input message con~rol circuit 1021 compares the biDary-encoded value of the FLIT signals represeiting the header field 40 of the data router message pacl~et 30 to the binary-encoded value of the NODE HEIGHT (2:0) sigDals representing the level of the data router node æ(iJ,k) in the data router 15. If binary-eocvded value of the FLIT sigDals representing the header field 40 is less than that of the NODE HEIGHT (2:0) signals, the data router message pacl~et 30 is to be transferred up the tree. Accordingly, the input message control circuit 1021 couples the FLIT signals representing the header field 40, as weLI as the FLIT signals represeDting the succeeding nits of the message address portion 31, as the IMF FLIT signals to the input message FIFO 1022. ID
addition, the OUT REQ [P,C(1,0)] output request signals loaded onto the output request identification FlFO 1023 indicate that the data router message packet 30 is to be transferred to the parent data router node Z(iJ,k).
On the other hand, if, while the AFD MODE all-fall-down mode signal is not asserted, the input message control circuit 1021 determines that the binary-encoded value of the Fl.lT signals representir~g the header field 40 is the same as that represe~ted by the NODE HEIGHT (2:0) signals, the data router node 22(iJ,k) is to transfer the data router message packet 30 down the data router 1~.
Accordingly, the input message control circuit generates IMF FLIT iDput message FIFO flit signals having a binary-encoded value that is one less than the birlary-encoded value represented by the FLIT
signals representillg the header field 40, which it couples to the input message FIFO 1022 as the first flit of the data router message packet 30. The inpot message control circuit 1021 also determiues whether the furst flit of the down path identification portion 41 is to be eliminated, and if so does not transfer signaLs representing that flit to the input message FIFO 1022. ID addition, the islput message control circuit 1021 generates OUT REQ lP,C(1,0)I output request signaLs for transfer to the output request identification FIFO 1023 which have a binary-encoded value that identify the child data router node 22(iJ,k) to which the message packet 30 is to be tra smitted.
Fmally, if the AFD MODE all-fall-down mode signal is asserted, the input message control circuit 1021 transfers the FLlT signals representing the header field 40 arld the down path identification portion 41 unchanged to the input message FIFO 10æ as the IMF FLIT input message FIFO flit signals. In addition, the input message control circuit 1021 generates OUT REQ lP,C(l:O~] output request signals for transfer to the output request identification FIFO 1023 which have a binary-encoded value that ideD~ifies the child data router node 22(iJ,k) ide~ltified by the CHILD MAP signals.
The input message control circu~t 1021 also generates a BOM beginning-of-message sigllal, an NEOM near end-of-message signal and an EOM end-of-message si~al, all of which it couples ~o the input message FIFO 1022 along with the IMF FLIT si~als representillg the first flit, a nit a selected number of flits from the last, and the last flit of a data router message packet 30, respectively. The input message FlPO 1022 receives and stores the BOM, NEOM and EOM sig~als along with the corresponding IMF FLIT signals.

WO 92/06436 PC'r/US91/073X:~
20~335a -121-3 ~ ~put message colltrol circuit 1021 generates an IMF PUSH input message FIFO push signal, which it asserts to enable the input message FIFO 1022 to receive IMF E~LIT signals representing a flit, along with the associated BOM, NEOM, and EOM signals. In addition, input message control circuit 1021 receives an IMF NR FULL input message FIFO nearly full signal from the input message FlFO 1022. When the IMF NR FULL signal is asserted, the input message FIFO
1û22 is nearly full. The input message control circuit 1021 uses the asserted or negated condition of the IMF NR FULL signal in controlling the assertion and negation of the VAL FLOW signal. As noted above, the interface 1020 uses the condition of the VAL FLOW signal in controDing the condition of the C0 IN FLY signal. Accordingly, the asserted or negated condition of C0 IN FLY signal wiD
represent the extent to which the input messag?e FlFO 1022 is able to receive and store additional flits.
Finally, the input message control circl~it 1021, when it receives FLrr signals representing the check field 33 of the data router message packet 30, verif~es that the data router message packet 30 was properly received. If the input message control circuit 1021 deterqlines that the data router message packet 30 was properly received, it generates a new check value which it couples as IMF FlFO input message FIFO signals to the input message FIFO 1022, representing the last flit of the data router message packet 30. If, on the other hand, the input message control circuit 1021 determines that the data router message packet 30 was not properly received, it couples an error code as the IMF FIFO
signals, and also enables the node control circuit 1û04 to assert an OUT ERROR signal. In one ~;
embodiment, the error code generated by the input message control Qrcuit 1021 corresponds to the complcment of the checlc value that the circuit 1021 would other vise provide.
The switch input control circuit 1024 performs several operations. First, the switch input control circuit 1024 retrieves from tbe output request identification FIFO 1023 the buffered OllT REQ
[P,C(1,0)~ output request signals, which it rece*es as ORIF OUT REQ IP,(~ O)] buffered output request signals. To initiate the retrieval, the switch input control circuit 1024 asserts an OIF POP
output identification F~FO pop signal. When the output request identification FIFO 1023 is empty, it asserts an OIF EMPTY outpu~ identification FIFO empq signal.
In response to retrieved ORIF OUT REQ [P,C(1:0)] buffered output request signals, the switcb input control circuit decodes the encoded child identification portion to generate OUT REQ
[P,C3:COl/SW output request to switch signals. The OU T REQ [P,C3:C0]/SW signals comprise one signal which, if asserted, ~ndicate that the flits compnsing the associated data router message packet 30 are to ~e coupled to the parent data router node æ(i,j,k), and a signal associated with each child data router node ~2(i,j,k) w'nic4 if asserted, indicate that tbe flits are to be coupled to the associated child data router node Z(iJ,k). The O11T REQ IP,C3:CûJ/SW signals are coupled to the switch 1003 When t'ne switch 1ûQ3 is in a condition to couple ~be flits comprising the associated data router messag,e packet 30 to the recipient identified by the ()~IT REQ IP,C3:Ct)]/SW signals, it ~antr. the request, represented by those signals by asserting a RE~Q GRANTED request granted signal. At tbat point, the switch input control circuit may assert the OIF POP output identification FIFO pop signal and repeat the operation. If the output request identification FIFO 1û23 is at that point asserting the WO 92/06436 P~/US91/07383 2~33~

OIF EMPrY output identification FIFO empty signal, the operatis~l~ will be delayed un~il it negates the OIF EMPl~Y signal.
After the switch 1003 has granted a request, the switch input control circuit 1024 retrieves flits representing the data router message packet 30 from the input message FIFO 1022 and couples lhem to the switch 1003. While it is able to receive the individual flits, the s vitch 1003 maintains a SW
FLOW switch flow sigDal in an asserted condition. In response, the s vitch input control circuit asserts an IMF POP input message FIFO pop signal which enables the input message FIFO 10æ to transmit FLlT/FlFO buffered flit signals which, in synchrony with successive ticlcs of the NODE CLK signal, represent successive flits of the data rouser message packet 30. l'he switch input control circuit, in turn, couples these signals to the switch 1003 as SW FLIT switch flit signals, and asserts a SW FLY
s vitch fly signal.
While the switch input control circuit 1024 is transmitting SW FLIT switch flit signals representadve of successive flits of a data router message packet 30, it asserts the OIF POP output identification FIFO pop signal to eDable the output request identification FIFO 1023 to transmit to it new ORIF Ol)T REQ lP,C(l:O)] signals for the next data router message packet 30, if any. If no additional data router message packets have been received, the output request identification FIFO
1023 maintains the OIF EM~I Y output identification FIFO signal in an asserted condition. If the OIF
EMPTY signal is negated it trarlsmits the ORIF oU'r REQ IP,C(1:0)] output request signals therefor, which the switch input control circuit 1024 recei,res at the next tick of the NODE CLK signal, decodes and couples to the switch 1003 as described aboYe.
The switch 1003 identifies and sets up the conneclioll for the new data router message pacl~et while the flits for the current data router message pacl~et 30 are being coupled therethrough. When the switch input control circuit 1024 receives the NEOM/FIFO near end-of-message from FIFO signal for the current data router message packet, it asserts a SW RELEASE switch release sigllal that enables the switch 1003 to fiDish the connection for the new data router message packet 30, and the s vitch 1003 can begin receiviag SW FLIT switch flit signals for the nits of the data router message packet 30 immediately after all of the flits for the current packet 30 have been coupled.
u. Interface Circuit Fig. 11B-1 depicts a detailed block diagram of an interface 1020 in the iDpUt child circuit 1006(0). With reference to Fig. 11B-1, the interface 1020 includes a receiver 1030 that receives the C0 IN FLIT input flit signals at a data input terrninal, and latches them at each tick of the NODE CLK
signal. The data output terminal of latch 1030 provides the FLIT signals, which are coupled to the iDput message control circuit 1021.
A latch driver 1031 transmits the C0 IN FLY input ny signal. The VAL FLOW valid flow signal from the input message con~rol circuit 1021 is received at oDe input terr~inal Ol a multiplexer 1032. The multiplexer's other input terrniDal is maintai~ed at a negated signal level. The multiplexer 1032 is controlled by a C0 NOT DIS not disabled signals, which represents the complement of the C0 DIS disable signal from node control circuit 1004 (Fig. 5). As noted above, the node control circuit 1004 asserts the c'a DIS disable signal to disable the child interface module 1001(0).

, . :, . .. ., . , ~ , . , , , , 1 : . :, .:

WO 92t06436 PCr/US91/07383 I the CO DIS signal is asserted, the multiplexer 1032 couples a negated level si8nal as a RAW
FLOW signal to the data iDput ter~inal of the latch driver 1031. The negated-level RAW FLOW
signal, wbich the latch driver 1031 latches at each tick of the NODE CLK signal, maintains the C0 IN
FLY input fly signal at a negated conditiorl.
In addition, the RAW FLOW raw flow sigDal is coupled to the irput terminal of a delay line 1033. The delay line couples the delayed R~W FI~OW signal as the VAL FLIT valid flit s~gDal to the input message control circuit 1021. While the RAW FLOW sigllal is maintaiued in a negated condition, the VAL Fl,lT signal is also negated. If, however, the condition of the RAW E~LOW signal changes from the asserted state to the negated state, the delay line 1033 delays provides a delay in a transition of the VAL FLIT signal from an asserted state to a negated state.
The amount of delay provided by delay line t033 is on the order of the propagation time for the C0 IN FLY input fly signal from the child input interface 1020 to the output interface of the child data router node æ(iJ,k) or injector port 22~ of the leaf 21 connected thereto, siDce that circmt may have transmitted signals, which will be received as C0 IN FLrr input flit signals represeDting several flits before it receives the negated C0 IN Fl,Y signal. These flits will be reoeived by the receiver 1030 and coupled to the input message control circuit 1021. The delayed r egatiol~ of the VAL Fl.IT valid ait signal enables the input message contsol circuit 1021 to receive these nitS from tbe receiver 103û
and couple them to the input message PIFO 1022. The delay provided by delay line 1033 ensures tbat the negation of the VAL ~LIT signal, as received by the input message sontrol circuit 1021, is synchronous with the provisioD thereto of Fl,IT signals by receiver 1030 represeDting a the last flit provided by the child data router node Z(iJ,k) or injector port Z3 of the leaf æ connected thereto. It will be appreciated that the input message FIFO 1022 will enable the input message control 1021 to negate ths VAL Fl,OW signal sufficiently ahead of its becoming full to enable it to receive and store these additional flits.
On the other hand, if tbe C0 DIS signal is negated, the multiplexer 1032 couples the VAL
FLOW signal from the input message control circuit as the RAW FLOW signal. The RAW ~OW
s4~al, accordingly, reflects the ability of the input message FIFO 1022 to acccpt and store additional flits. If the RAW FLOW signal is negated, indicatiDg that the input message FIFO 1022 is nearly full, the VAI, FLOW signal is will also be negated. As above, the negated VAL FLOW signaJ will be latched by driver 1031 in response to the NODE CLK signal, which udll drive the C0 lN Fl.~ input fly signal to the data router node 22tiJ,k) to a negated level to disable the child data router node æ(iJ~k) or leaf 21 connected thereto from transmitting nitS.
If, however, the mput message FIFO 1022 is able to receive additional flits, the input message control circuit 1021 asserts the VAL FLOW valid flow signal. If the CO DIS disable signal is negated, the multiplexer 1032 couples the asserted level VAL FWW valid flow signal as an asserted level RAW
FLOW signal to the input terminal of driver 1031. I:)river 1031, in turn, latches the asserted RAW
FLOW signal and drives it as an asserted C0 IN FLY input ny signal at the next tick of the NODE
CLK signal.

, ;, ~ , :: , , , , ~ .. ::::

20~3à3 The delay line 1033 also receives the asserted RAW FLOW signal and asserts the VAL FLIT
valid nit signal. In a transition of the VAL FLIT signal from a negated condition to an asserted condition, the ~ransition in the VAL FLrr signal is also delayed by the delay li~e 1033. As above, the delay is provided to permit the child data router node 22(i,j,k) or injector port 223 of the leaf 21 connected thereto to receive the asserted O0 IN FLY input ny signal and begin transmitting signals, which will be received at receiver 1030 as the C0 IN FLIT signals, representing nitS. The delay provided by delay line 1033 ensures that the assertion of the VAL FLrr signal, as received by the input message control circuit 1021, is synchronous vith the provision thereto of FLIT signals by receiver 1030 representing a ~it.
iii. Input Message Control Circuit Fig. 11B-2 depicts a general block diagram of the input message control CilCUit 1021 (Fig.
11B). With reference to Fig. 11B-2, the input message control circuit 1021 includes a nit flow control circuit 1040, an input control circuit 1041, a down path identification flit counter 1042, a data flit . . _ , . . .
counter 1043 and a checksum check/generator circuit 1044. The flit flow eontrol circlut 1040, which is depicted ;D greater detail in Figs. 11B-2B and 11E~-2C, receives the FLIT signals from the interface 1020 and generates, under control of the input control circuit 1041, IMF FLlT input message FIFO flit signals and the BOM beginning-of-message signal, NEOM near end-of-messaBe signal, and EOM end-of-message signal for storage in the input message FIFO 1022. In addition, the nit flOw control circuit 1040 generates the OUT REQ IP,C(l:O)] output request signals for storage in the output request identification FIFO 1023. The flit flow control circuit 1040 also generates the VAL FLOW signal used by the interface circuit 1020 and receives the VAL FLlT valid nit signal which corltrols its use of the FLlT sig~s.
The nit flow control circuit also provides several signals to the input contro! circui~ 1041. The input control circuit 1041 is essentially a state machine, that operates ;D a pluralit,v of states as depicted on Fig. 11B-2A. The input control circuit 1041 provides several signals to the flit flow control circuit 1040 that effectively identify various states in receipt of a data router message packet 30. In particular, the input control circuit 1041 provides a MSG INlT message il~itialization signal that enables the flit flow control circuit 1040 to identify the firs~ flit of a data router message packet. If the MSG INIT
signal is asserted, and if the VAL FLlT signal is asserted, the first nit of a data router message packet 30 is indicated by the FLIT signals received by the flit nOw control circuit 1040 having a non-zero binary-encoded value.
The asserted ~fSG INIT message iDitialization signal is also couplet to the down path identification flit counter 1042 to enable it to load the Fl.rr signals. It will be appreciated that the input message control circuit 1041 will maintain the MSG INlT sigl~al in an asserted condition until the FLlT signals represent the header field 41 of a data router message packet 30, and while the MSG
INIT signal is asserted the counter 1042 will continually load the FLIT sigllals.
As described above, if a data router node 22(i,j,k) receives a tata router message packet, traversing the data router 15 down the tree, the node may discard flits representing the dowll path ,,. . ~ . ..

WO 92/06436 PCr/US91/07383 2~93~
ideDtification portion 41 as they are used. The input control circuit 1041 generates a DROP DN FLIT
drop down Qit signal which, if asserted, enables the flit flow control circuit 1040 to discard the first flit of the down path identification portion 41. The il~put control circuit 1041 uses a HDR= header equal signal, the low-order NODE HEIGHT(0) height s4~nal and the AFD MODE all-fall-down mode sigDal to detennine whether to assert the DROP DN FLIT drop down flit sagnal. If the tlit flow control circuit 1040 is asserting the HDR = signaL the data router message packet 30 is being transmitted down the data router tree. A data router node 22(iJ,k) drops a flit iD the down path identification portion 41 orly in the odd-numbered levels (i), which is indicated by the low-order NODE HEIGHT(0) sigDal.
Finally, the data router nodes æ(ij,k) do not drip flits in the down path identification portion 41 if data router 15 is in all-fall-down mode, as indicated by the assertion of the AFD MODE all-fall-down mode signal.
Finally, the input control circuit 1041 also asserts a MSG LEN IN message length in signal, a TAG OR DATA signal and a CHECK signal contemporaneously with FLIT signals representillg the corresponding flits of the data router message packet. I~n generating these signals, the input control circuit uses si~nals from the down path identification flit counter 1042 and the data nit counter 1043.
l~e flit flow control circuit 1040 a'Lr,o provides the received FLIT signals to the do vn path itentification nit counter 1042, the data nit counter 1043 and the checksum check/generator circuit 1044. The down path identification flit counter 1042, which will be described in greater detail below in cormection with Fig. 11B-2E, seceives the ~lT signals representative of the header field 40 (Fig. 3) of a data router message packet 30. The counter 1042 iteratively decrements in response to successive ticks of the NODE CLK signal while the VAL FLIT valid flit signal is asserted. During this time, the Fl~rr signals represent the down path identification portion 41 of the data router message packet 30 being received. When the value of co~mter 1042 decrements to zero, it asserts a DNF END do vn flit end signal, which indicates that the entire down flit identification portion 41 has been received.
Similarly, ~e data flit couDter 1043, which will be described below in cormection Wit~l Fg. 11B-2D, receives the Fl,IT signals represe~ltative of the message length field 34 of a data router message packet 30. The counter 1042 iteratively decrements ;D response to successive ticlcs of the NODE CLK
s,gnal while the VAL FLlT valid flit signal is asserted. During this time, the FLlT signals represent tne message data portion 32 of the data rou~er message packet 30 being received. When tne value of counter 1043 decrements to nearly zero, it asserts a NR DATA END signal, which the nit flow control circuit uses iD generating tne MEOM near end-of -me~,e sigDal. When counter 1043 decrements to zero, it asserts a DATA END data end signal, which irldicales that the entire message data portion 32 has beien received. The flit flow control circuit uses the DATA END sigllal in generatirlg the EOM
end-of-message signal.
The checksum ehecit/generator circuit 1044 receives the FLIT signals and verifies the checksum of the FLIT signals representative of checlt field 33. Tbe checksum check/generator circuit 1044 is ïeset in response to the BOM beginDing-of-message signal provided by the nit flow control circuit 1040. In respo~se to FLIT signals representative of successive flits of a data router me.,sage WO 92/06436 2 ~ 9 3 3 ~ ~r/Usgl/07383 packet 30, the checksum check/generator circuit 1044 generates NEW CHECKSUM signals. When the input control circuit 1041 asserts the CHECK signaL the circuit 1044 generates a CHECK OK
signal whose condition indicates whether tbe checksum generated by the checksum check/geuerator circuit matches the value in the check field 33 of the data router rnessage packet. The flit flow control circuit 1040 uses these signals in generating the IMF FLIT input message FIFO nit signals representing the check field 33 of the data router message packet for storage in the input message FIFO 1022.
The Sit flow control circuit 1040 will be described in connection with Fig. 11B-2B and 11B-2C, and in connection with Fig. 11B-2A, which depicts a state transition diagram for the input control circuit 1041. Fig. 11B-2B depicts a portion of the nit flow control circuit that generates signals for the input message FIFO 1022, and Fig. 11-2C depicts a portion that generates signals for the output request identification FIFO 1023.
With reference to Fig. 11B-2B, the flit tlow control circuit 1040 receives the FLrr sigDals at an OR gate 1050. If at least one of the Qit signals is asserted, ~he OR gate 1050 is energized, which enables one input of an AND gate 1051. If the VAL FLIT valid flit signal is also asserted, AND gate 1051 is energized to assert a NOT ZERO signal. If the input control circuit 1041 is asserting the MSG
INIT message initialization signal, and if the node control circuit 1004 tFig. 1Ld~) is asserting the EN
enable signal, the assertion of the NOT ZERO signal, while the VAL FLIT signal is asserted, energizes the AND gate 1052, enabling it to assert a FIRST FLIT signal. A driver 1053 receives the asserted FIRST FLIT signal and asserts BOM begi~ing-of-message signal.
The NOT ZERO signal is also coupled to an inverter 1054, which generates a ZERO signal.
lf the ZERO signal is asserted, either the VAL FLlT sigDal is negated, which disables AND gate 1051, or the OR gate 1050 may be de-energi~d. The OR gate 1050 is de-energi~ed if all of the FLIT signals are negated.
With reference to Fig. 11B-2A, as noted above, the input control circuit 1041 is initiaDy asserting the MSG IMIT message initialization signal. At tbat point, the circuit 1041 is ill a message initialization state, indicated by the box labeDed ~MSG INIT.~ The input control circuit remains in that state if the VAL FLIT valid nit or EN eDable signals are not asserted, or of the in~erter 1054 is asserting the ZERO signal. However, while the input coDtrol circuit 1041 is in the message initialization state, in respoDse to tbe coincidence of the assertion of the VAL FLIT and EN signals, and the assertion by AN~ gate 1051 of the NOT ZERO signal resulting i~ the negation of the ZERO
signal, the input control circuit 1041 sequences to one of three other states, as described below.
Returning to Fig. 11B-2B, the FLIT signals are also coupled to the input terminal of a gated driver 1055 and to an input terminal of a comparator 1056. The gated driver 1055 con~rols the coupling of FL~T signals to the input message FIFO 10æ in response to a FLlT/FlFO EN llit to FIFO enable signal from an AND gate 1057. The comparator 1056 receives the FLIT signals and the NODE
HEIGHT (2:0) signals identifying the level ~i~ of the data router node æ(iJ,k) in the tr~e of data router 15. If the AND gate 1052 is asserting the FIRST FLIT si~al, which occurs when the FLIT signals represent the header field 40 of the data router message packet 30, the comparator 1056 is enabled to compare the binary-encoded value of the NODE HEIGHl' (2:0) signals to that of the FLIT s gnals.

"- ' " i '' .'' ' " '" : '~.~ , '',,' ,'';; ' ', ' . . ' ' ~ '.:

WO 92/06436 PCl /l~S91/0738.3 2~933~ -127-lf the comparator 105o determines that th~ binary-encoded value of the Fl,lT signals is greater than that of the NODE HEIGHT (2:0) signals, it asserts a HDR < header less than node height signal.
It will be appreciated that, with the binary-encoded value of the FLIT si~als ih~ this condition, the dala router message packet 30 is to be transmitted up the tree defined by the data router 15, ur less the AFD
MODE all-fall-down mode signal is asserted. On the other hand, if the comparator 1056 determines that the binary-encoded value of the FIIT signals is less than or equal to that of the NODE HE~IGHT
(2 0) sy~,nals, which will occur if the data router message packet 30 is to be transmitted down the tree defined by the data router 15, it asserts a HDR = header equals node height s gnal.
If the comparator 1056 is asserting the HDR= header equals node height signal, and if the AFD MODE all-fall-down mode signal is Degated and tbe input contsol circuit 1041 is asserting the MSG INlT message initialization signal, an AND gate 1060 is de-energized enabling it to assert a NEW
HDR EN new beader enable signal. It will be appreciated that this oceurs when the Fl,IT signals represent the header field 40 of a dala router message pac1~et 30 to be transmitted down the tree represented by the data router 15 while it is not in all fall-do vn mode. As noted above, the conlents of the header field 40 as transmitted by a data router node 22(i,j,k) have a binary-encoded value corresponding to that as received, decremented by one.
To provide the new contents for header field 40, the asserted NEW HDR EN new header enable signal, in turn, enables a gated driver 1061 to coupled DECR HEIGHT decrernented height signals as NEW HDR Dew header signals ODto a bus 1062. The DECR HEIGHT decremented heig~t signals are provided by a decrementation circuit 1063, which receives the NODE HEIGHT (2:0) signals alld generates DECR HEIGHT signals whose binary-encoded value is one less than that of the he NODE HEIGHT (2 0) signals node heigh~. Accordingly, the NEW HDR signals have a binary-eneoded value one less than that of the NODE HEIGHl (2:0) signal.
If the comparator 1056 is not asserting the HDR= header equals node height signaL indicating that the data router message pacl~et 30 is to be transferred up the tree detming data router 15, an AND
gate 1060 is disabled. AND gate 1060 is also disabled if the AFD MODE all-fall-down mode signal is asserted indicating that Ihe data router 15 is in all-fall-down mode, and if the MSG INIT message initialization signal is negated, indicating that the FLIT signals represent flits of a data router message packet 30 other than the header 40. In any of these cases, an inverter 1070 enables one input tenninal of AND gate 1057. If the ~nput control circuit 1041 is not asserting the CHECK signal, an inverter 1071 enables the second input terminal of AND gate 1057, eDabling it to assert the FLlT/FlFO EN flit to FIFO enable signal. The asserted FLlT/FlFO EN signal enables the AND gate 1055 to couple the FLlT signals as FLIT TO FIFO signals onto bus 1062. Thus, the gate 1055 couples th~ FLIT si~als to the bus 10~ if:
(i) the FLlT signals represent the header field 40 and ~he data router message packet 30 is to be transferred up the tree def~ing data router 15;
(iu) the Fl~lT signals repFesent the header field 40 and the APD MODE all-falJ-down mode signal is asserted; and ,- :: : , , : i, :: , - :
,,:. .; .
, , " " ,::, , :

WO 92t06436 PCl/US91/07383 ~ 20~353 (iii) the Fl,IT signals represent fields other than the header field 40 or the check field 33.
While the Fl,IT signals represent the header field 40, the asserted MSG INIT message initiali~ation and NOT ZERO signals energize an AND gate 1064 to assert a 8EG MSG begin message signal. The asserted BEG MSG signal energizes an OR gate 1065, which, in turn, enables one input tenninal of an ANI) gate 1066. Since the VAL FLlT valid flit signal is asserted, the second input terminal of AND gate 1066 is also enabled, energizing the AND gate and enabling it to assert the IMF
PUSH input message FIFO push signal. The asserted IMF PUSH signal enables tbe input message ~IFO 10æ to store the IMF FLIT input message FIFO flit signals, w~ich, as noted above, have a binary-.-ncoded value corresponding to tbe binary-enu~ded value of the NODE HEIGHl` (2:0) signals decremented by one.
Thereafter, tbe operations of the nit flow control circuit will depend on the particular state, following the message initialization state, ~o which the input cortrol circuit 1041 seque~ces (Fig. 11B-2A). As noted above, the input control circuit 1041 may sequence from the message inidalization state to one of tbree states. In two states, namely, a keep down patb identification flit state identified by tbe box labelled ~KEEP DN PATH ID FLlT,~ and a receive down path identification flits state identified by the box la~elled "RCV DN PATH ID FLITS,~ the input control circuit 1041 enables tbe flit flow control circuit 1040 to receive F~IT signals representing tbe all of the successiYe nits of the down path identification portion 41 of the data router message pacl~et 30. In tbat operation, the input control circuit 1041 negates the MSG INIT message initialization signal and maintains the DROP DN FIIT
drop down path identification flit signal in a negated condition.
In the third state, identified as a drop down path iden~ification flit state represented by the box labelled "DROP DN PATH ID FLlT,~ the input control circuit 1041 enables the flit flow coatrol circwt 1040 to drop the 'first nit of the down path identificadon portion 41. In that state, the input control circnit 1041 asserts the DROP DN FLlT drop down path identification flit signal, and also negates the MSG TNrr message initialization signal.
The input control circnit 1041 sequences from the message initiali~ation stage in resposse to the coincidence of several conditions represented by signals having selected states. If the VAL FLIT
valid flit, EN enaWe, and NOT ZERO signals are asserted, and if either the AFD MODE all-fall-down mode signal is asserted or the HDR= header equals node height signal is negated, the input control circuit 1041 sequences to the receive down pat'n identification flits state. Tbus, if the data router 15 is in all-fall-down mode, or if the flit flow control circuit 1~40 determines, while the input control circuit is in the message initia~izadon state, that the data router message packet 30 is to be transfe~Ted up the tree definiDg data router 15, the input control circuit 1041 sequences to the receive down path identification state.
On the other hand, if the VAL FLIT valid flit, EN enable, and NOT ZERO sig~als are asserted, and if the HDR- header eq~s node height signal is asserted while the NOl:)E HEIGHT (0) and AFD MODE all-fall-down mode signals are negated, the inpu~ control c rcuit 1041 sequences to the keep do vn path identification flit state. Thus, if (a) the data router is not in all-fall-do vn mode, (b) . . . :., ,. . - :

~, : ~ ''.. ' ' ,, . . :::: ' ,' :..:' ~ ' '' '' . ;' ' : : .: ~ j . , "

WO 92/06436 PCl/US91/07383 2a~3~ 29-the level ~i" of the data router node 22(iJ,k) is an even number, and ~c) the data router mess~ge pacltet is to be transferred down the tree def~4, data router 15, the input control circuit 1041 sequences to the keep down path ide~tification nit state.
Finally, if the VAL FLrr valid flit, EN enable, and NOT ZERO signals are asserted, and if the HDR- header equals Dode he;g,ht and NODE HEIGHT (0) signals arc asserted when the AFD
MODE all-fall-down mode signal is negated, the input control circuit 1041 sequences to the drop down path identification flit state. Thus, if (a) the data router is Dot in aU-fall-down mode, (b) the level ~i~ of the data router node 22(i,j,k) is an odd number, and (c) the data router message packet is to be transferred down the tree def~4~, data router 15, the input control circmt 1041 sequences to the drop down path identificatioll flit state. As noted above, in that state the input control circuit 1041 asserts the DROP DN FLIT drop down nit signal.
Returning to Fig. 11B-2B, when the input control circuit 1041 sequences from the message initialization state it negates the MSG INIT message initialization signal. The negated MS& INll`
signal is complemented by inverter 1070 to enable one input tenninal of AND gate 1057. In any of the states immediately follow~ng the message initialization state, the input control c~rcuit 1041 also negates the CHECK signal, which is complemented by inverter 1071. In that condition, the AND gate 10~7 is cnergized, enabling it to assert the FlIT~FlFO EN flit to FIFO enable signal, which enabled ga~ed driver 1055 to couple the FLlT sigllals as FLIT TO E;IFO signals onto bus 1062.
The negated MSG INIT signal also disables the AND 8ate 1064, causing it to negate the BEG
MSG begin message signal, which, in turn, disables one input tenDinal of the OR gate 10oS. The negated MSG INlT message initialization signal, on the other hand, is complcmented by an inverter 1072 to ellable one input of aD AND gate 1073. If the DROP DN FLIT signal is negated, whicb occurs if the irlput message control sequences to the keep dovn path identification nit state or the receive down path identification flits state, an invester 1074 enables the second input terrninal of AND gate 1073, thereby energizing the AND gate to assert a CONT MSG continue message signal. The asserted CONT MSG signal maintaiDs the OR gate 1065 in an energized condidon, enabling, i~ tu~n, AND gate 1066 to maintain the IMF PUSH inp~lt message F~FO push sigllal in an asserted condition while the VA'L FI~IT sigDal remains asserted. Thus, if the input control circuit 1041 sequences from the message initialization state to either the keep dowu path identification flit state or the receive down path identification nits state, it enables the flit flow control circuit 1040 to maintain the IMF PUSH input message FIFO push signal at an asserted level while FLIT TO FIPO signals representing both the header field 40 and the fust flit of the down path identification portion 41 are coupled onto bus 1062.
On the other hand, if tbe iDput control circuit 1041 sequences from the rnessage i~itialization state to the drop down path identification flit state, and thereby asserts the DROP DN FLIT drop to vn nit si~al, the inverter 1074 disables the AND gate 1073 to Degate the CONT MSG continue message signal. SiDce the AND gate 1064 is also negating the BEG MSG begin message signal, the OR gate 106~ is de-energized, disabling AND gate 1066 to thereby ne~ate ~he IM~ PUSH input message FIFO push signal. The AND Bate 1055 is at that point coupling the FLIT signals as FLlT TO

W O 92/06436 PC~r/US91/07383 2~3~
FIFO signals onto bus 1062 representing the first tlit of the down path identificalion portion 41.
HoweYer, since the IMF PUSH signal is negated, the input message FIFO 1022 is disabled from storing the signals. Accordingly, the asserted DROP DN FLIT drop down flit signal causes the first flit of the down flit identification portion 41 to be dropped from the data rouler message packet 30 to be transferred.
The negation of the MSG INIT message initialization signal by the inpu~ control circuit 1041 also inhibits the down path identification flit counter 1042 from CQntinuing to load Ihe FLIT signals.
Thereafter, while the VAL FLlT valid flit signal is asserted, the counter 1~42 is erlabled to decrement in response to each tick of the NODE CLIC signal. As noted above, while the VAL ~:LIT signal is asserted, the FLIT signals at successive ticks of the NODE CLK signal represent successive flits of the data rouler message packet 30. When the MSG INIl` message initialization signal is negated, the Fl,IT
si~nals loaded by the down path identificatioll flit counter 1042 represent the header field 40 of the data router message packet 30 being transferred, w~uch! as noted above, identifies the number of flits in the down path identitlcation portion 41.
Returning to Fig. 11B-2A, the input control circuit 1041 stays in the drop down path identification flit state w the keep do vn path identification flit state while the flit flow control circuit 1040 receives only one flit, namely, the first flit of the down path identification portion. Thereafter, the input control circuit 1041 sequences to either the receive down path identification nits state or a receive message le~gth in state, the latter being represented by the box labeOed ~RCV MSG LEN IN.~ If, with the input coDtro! circuit 1041 being in the drop down nit ideDtification flit state or ;D the Iceep down path identification nit state, the down path identification flit counter 1042 is not asserting the DNF
END town flit end signal, the input control circuit 1041 sequences to the receive down path identification flits state. In that state, the input control circuit 1041 continues to rnaintain all of its output signals, including the DROP DN FLlT signal, in a negated condition. As desclibed above, while the VAL FLIT signal is asserted, the negated DROP DN FLlT signal maintains the AND gate 1066 (Fig. 11B-2B) energized to maintain the IMF PUS}I input message Fl~O push signal in an asserted contition.
If, on the other hand, when the input control circuit 1041 is in either the dsop down path identification flit state or the keep down path identificatioD nit state aDd the down path identification flit counter 1Q42 counts out and asserts the DNF END signaL it (that is, the input control circuit 1041) vvill sequence to the receive message length in state. That will occur if the down path identification portion 41 of ~he data router message paclcet 30 be~ng received has a length of one flit. If the input control circuit 1041 is in the receive down path identification flits state, it v.ill also sequeDce to the receive message length in state when the down patb ;dentification nit counter 1042 counts out and asserts the DNF ENl~ signal. This will occur when the down path identification portion 41 of the data router message packet 30 that is being received has a lergth greater than one flit. Irl either case, when the input control circuit is in the rcceive message length in state, it asserts a MSG LEN IN message length in signal.

, . ~:~: : .: : : :: , ,, ..:,,, :, : .,,, ~ , . .
.: :,. . :: -.,: ,. :, . :
:, :, :, ::: .:.:::: -,, ": . . ..

WO 92/06436 PCr/US91/07383 2~g33~3 The input control arcl~it 1041 asserts the MSG LEN IN message leDgth in signal when the FLIT signals represeDt the message length field 34 of the data router message packet 30. The asserted MSG LEN IN signal enables the data flit counter 1043 to load the FLIT signals. The MSG LEN IN
signal also enables one input terminal of an AND gate 1075. If the Fl,IT signals have a binary-eDcoded value of zero, which u~ill be indicated if the inverter 1054 is assertirlg the ZERO signal, the message data portion 32 has no data nits 36. In that case, the second input terminal of the AND gate 1075 is also eDabled, energiziDg the AND gate, enabling it to assert a NO DATA signal. The assened NO
DATA sigDal energizes OR gate 1076, enabling it to assert the NEOM near end-of-message signal, which is conp!ed to the input message FIFO 10æ as described above.
On the other hand, if the Fl,ll` signals do not have a binary-encoded value of zero, the data flit counter 1043 will decrement~ while the VAL FLIT valid flit signal is asserted, in response to successive ticks of the NODE CLK signal, and will assert the NR DATA ~D near data end signal a predetermined number of nitS prior to the last data flit 36, and the DATA END signal contemporaneous]y with the last data flit 36 in the data router message packet 30 'oeing received.
With the input control circuit 1041 in the receive messa;ge length in state, if the YAL FLlT
signal valid flit signal is asserted the input control circuit 1041 sequences to a receive tag or data state.
In that state, the input control circuit 1041 asserts a TAC; OR DATA signal. The input control circuit maintains the TA~ OR DATA signal in the asserted condition while the Fl IT sigllals represent the tag field 35 or data flits 36 of the received data router message packet 30. The asserted TAG OR DATA
signal erLables one input terminal of an AND gate 1077. When the data flit counter 1043 asserts the NR DATA END ne~ data end signal, the s~cond input terrninal of AND gate 1077 is enabled, thereby ener~ng the AND gate 1077 to assert a NR END near end signal. l'he asserted NR END signal energizes the OR gate 1076 to assert the NEOM near end of rnessage signal.
Thereafter, the data flit counter 1043 decrements again in response to receipt of the next flit of the data router message packet 30. At that point, the counter Degates the NR DATA END near data eDd signal, causing the AND gate 1077 to be de-energized and thereby negating the NR END near end sigaal. The negation NR END signal, in turn, de-energizes the OR gate 1076 to negate the NEOM
near end-of-message signal.
W~ile the input control circuit 1041 is in the recei~le tag or data state, if the VAL FLlT valid flit signal is asserted the flit flow control circuit 1040 maintains the IMF PUSH input message FIFO
push signal in the asserted state to enable the input message FIFO 1022 to receive and store the successive flits of the data router message packet 30. While the input control c~rcuit 1041 is in the receive tag or data state, it maintains the MSG INIT message initialization and DROP DN FLIT drop down flit signals irl the negated condi~ion. Accordingly, ihlverters 1072 and 1074 maintai~ the AND
gate 1073 in the ener~zed condition, which, in turn, D~aintain OR gate 1065 energized to enabie one input terminal of AND gate 106o. While the VAL FLlT valid flit signal is asserted, the AND gate 1066 maintains the IMF PUSH signal asserted, as indicated above.
As shown on Fig. 11B-2A, when the data nit coun~er 1044 asserts the DATA END signal, the input control circuit 1041 sequences to a check state. During the check state, the input control c~rcui WO 92/06436 PCl/US9l/07383 20933~3 1041 enables the nit flow control circuit 1040 to couple IMF FLIT input message FIFO flit signals corresponding to either the true or compiement of the NEW CHECKSUM signals generated by the checksum check/generator circuit 10M as the checksum field 33 of the data router message packet 3.
If the checksum check/generator circuit 10~4, while seceiving the successive nits of the data router message packet 30 beiDg received, computes a checksum that correspoDds to the checksum in the checksum field 33, the flit flow control circuit 1040 couples the true of the NEW CHEC~SUM signals as the IMF F~IT signals. On the other hand, if the checksum check/generator circuit 10~4 computes a checksum that differs from the checksurn in the checksurn field 33, the nit flow control circuit 1040 couples that the complement of the NEW CHEC~CSUM signals as the IMF Fl.IT sigDals.
By providing the complement of the checksum signals in the checl~sum field 33 upon detection of an error by the checksum checl~/generator circuit 1044, the flit flow control circuit 1040 enhances the likeli'nood that the error indication will propagate through subsequent data router ~odes 22(iJ,k) in the path from the source leaf 21(x) to the destination leaf 21(y~. Since all of the signals in the .. . . . .
chechum field 33 are complemented, the next data router node 22(iJ,k) to receive the data router message packet 30 will likely detect an error indication, and will couple the complement of the checksum signals computed by the checksum check/generator circuit 1044 at that node to tbe next data router node æ(iJ,k), and so on. In particular, s~nce all of the signals in the checksum field 33 are complemented, single-bit errors in one or several of the signals in the transmission through subsequent data router nodes æ(i,j,k) is unlikely to result in masking of the error condition in connection with ~he data router 1nessage packet 30.
To accoinplish this, in the check state, the input control circuit 1041 negates the TAG OR
DATA signal and asserts a CHECK signal. The asserted CHECK signal enables the checksum chec~/generator circuit 1044 (Fig. 11B-2) to transmit the checksum value geDerated thereby as NEW
CHECKSIJM signals, and to transmit a CHECK OK signal indicating whether the nit flow control circuit 1040 properly received the data router message paclcet 30. If the CHE(::K OK signal is asserted, the flit flow control circuit 1040 properly received the data router rnessage packet 30, and if the signal is negated thc flit flow control circuit 1040 did not properly receive the data router message packet 30.
The CHECK, NEW CHECKSUM and CHECK OK signals are also coupled to the flit flow control circuit 1040. As shown on Fig. 11B-2B, the asserted CHECK signal is complemented by inverter 1071 to d;sable AND gate 1057, thereby disabling gated driver 1055 from coupling the FLlT
signals then being received as FI IT TO FIFO signals onto bus 1062. It will be appreciated that the FLlT signals at this point correspond to the CHECKSUM field 33 of the data router message packet 30 being received. The CHECK signal also coDtrols two gated drivers 1080 and 1081, which are also controlled by the true and complement, respectiYely, of the CHECK OK signal from checksum check/generator circuit 1044. The gated driver 1080 also receives the true of the NEW CHEC~CSUM
signals, and gates them as NEW CHECKSUM TO FIFO sigoals onto bu~s 1062 if both the CHECK
and CHECK OK sigDals are asserted.
An inverter 1082 receives the NEW CHECI~SUM signals and generates complemcnts of the respective signals, which are coupled to input termioals of gated driver 1081. If the CHECIC OK signal : ,: . . .:,. : ~ ~ .,: :

-: , : ::,. , , , . : ~ , : .
- . : ~ : : , - , ,.: :-, . . . ,~ : : ,:, . :

- : . . :: : : : :.

is2e~ga~e~ 3~ca~ing that the checksum check/generator circuit 1044 detected an error in the received data router message packet 30, an ~verter 1083 enables a respective input terminal of gated driver 1081. If ~e CHECK signal is also asserted, the gated driver 10~1 couples ~he complements of the ~W C~CKS'UM signals provided by inverter 1082 as BAD C~ECKSUM TO i~FO signals oDto bus 1062.
ln either case, since at this point the input control circuit is maiDtaining the MSG INIT
message initialization and DROP DN FLIT drop down flit signals in a negated condition, the AND
gate 1073 and OR gate 1065 are energized. If the VAL FLIT signal is asserted, AND gate 1066 remains energized to assert the IMF PUSH input message FIFO push signal, enabling the iDpUt message FIFO 1022 to load the signals on bus 1062 as the new checksum field 33 of the data router message packet 30. If the CHECK OK signal is asserted, indicating tha~ the data router message packet 30 as received was properly received, the input message FIFO 1022 loads the NEW
CHECKSUM TO FIFO signa!s as Ihe checksum field 33. However, if the CHECK OK signal is Degated, indicatirlg that the data router message packet 30 as received was not properly received, the input message FIFO 1022 loads the BAD CHECKSUM TO FIFO signals as the ehecKsum fileld 33.
The CHECK signal is also coupled to a d~iver 1084 whicsh provides the EOM end-of message signal. When the CHECK signal is asserted, the driver 1~4 asserts tne EOM signal, which is loaded alor~, with the IMF FLIT input message '~IFO flit signals on bus 1062.
ReturDi~g to Fig. 11B-2A, when the input control cirrlit 1041 is in tne CHECK state, if the VAL FLIT s,gnal is asserted it sequences to the message initJalizatiorl state at the ne~ tick of the NODE CLK sigDal. Thus, the input control circuit 1041 remains iD the check state while the flit flow control circuit 1040 receives Fl~IT signals representing one nit, namely the flit representing checksum field 33. When the input control circuit 1041 leaves the checls state, it negates the CHECK signal, which disables gated drivers 1080 and 1081 (Flg. 11B-2B). In addition, the negated CHECK signal is complemented by inverter 1071 to enable one input terminal of AND gate 1057, to allow the AND gate 1057 to be thereafter controlled by the complemented NEW HDR l:N new header enable signal as provided by inverter 1070.
As described above, when the input control circuit 1041 is in the message initialization state, it asserts the MSG INIT message initialization sigraL whicb enables respective input terminals of AND
gates 1052, 1060 and 1064, and through inverter 1a72 disables an input terrninal of AND gate 1073.
Thus, the flit fiow coDtrol circuit 1040 is in condition to begin receiving flits for a new data router message packet 30.
The flit flow control circuit 1040 also provides the VAL FLOW valid flow signal to the interface 1020, which the interface uses to control the condition of the C0 IN FLY input fly signal.
With reference to Fig. 11B-2B, the flit flow control circuit 1040 includes an inverter 1085, which receives the Ih~lF NR E~ULL input message FIFO nearly full signal from the input message FIFO 1022 and transmits the complement as the VAL FLOW signal. Thus, the VAL FLOW signal reflects the extent to which the input message FlFO 1022 has been filled.

WO 92t06436 PFI'/US91/07383 2~933~

As noted above, the nit llow control circuit 1040 also provides the OUT REQ [P,C(1:0)]
output request signal and OIF PUSH output identification FIFO push signal to output request identification FIFO 1023. The circuitry for this is depicted on Fig. 11B-2C. With reference to Fig. IIB-2C, the circuitry includes a parent request generator portion 1090, a child request genesator portion 1091 aDd a push signal generator portion 1092. The parent request generator portion 1~0 includes an AND gate 1093 thal receives a CHILD PORT signal which is asserted if the circuit is in an input child interface 1006(i), and the complement of the AFD MODE all-fall-down mode signal as complemented by an inverter 1094. If the C~ILD PORT signal is asserted and the AFD MODE all-fall-dow~ mode signal is negated, if HDR c header value less than n~de height signal from eomparator 1056 is asserted, indicating that the data router message packet 31~ is to be transmit~ed up the tree def~ing data router 15, the AND gate 1093 is energized to assert a GO UP sig~al.
The GO UP signal is coupled to one data input terminal of a rnultiplexer 1095. If ~he VAL
FLIT signal is asserted, multiplexer 1095 couples the GO UP signal, now asserted, to the data i~put terminal of a flip-9Op 1096, which is set in response to the next tick of the NODE CLOCK signal. The set flipflop 1096 asserts the OUT REQ [P] output request (parent) signal, which forms one of the OUT REQ [P,C(1:0)] output request signals. If the VAL FLIT signal is negated with the flip-flop 1096 in that condition, multiplexer 1095 is enabled to couple the OUT REQ [Pl signal to the data input terminal of the flip-flop 10~6, so that the fli~flop 1096 wal remain set in response to subsequent ticks of the NODE CLK signal.
If (a) the HDR < header value less than node height signal from comparator 1056 is negated, or (b) the CHILD PORT s;gnal is negated indicatillg that the circuit is in an input parent interface lOlO(i), or (c) the AFD MODE all-fall-down mode signal is asserted, indicating that the data router 15 is in all-fall-do vn mode, the data router node æ(iJ,k) is to transfer the data router ~essage packet 30 being received down the tree definiDg Ihe data router 15. In that case, the AND gate 1093 is de-energized to negate the GO UP signal. If the VAL FLlT valid flit sig~lal is asserted, multiplexer 1095 couples the negated GO UP signal to the data input terminal of fli~flop, which is clear in response to lhe next tick of the NODE CLK sigQaL to, in turn, negate the OUT REQ lPl output request (parent) signal. If the VALFl,IT si~nal is Degated witb the nip-nOp 1096 in that condition, multiplexer 1095 is enabled to ~ouple the OUT REQ IP] signal to ~he data input terminal of the flip-aop 1096, so that the flip-flop 1096 ill remain clear in response to subsequent dclcs of the NODECLK signal.
The child request generator portion 1091 includes a gated driver 1100 that receives the FLlT
signals &om the interface 1020 and the VALFLIT valid flit signal. If the VAL FLlT signal is asserted, the gated driver 1100 coupJes the FLIT signals as GATED FLIT signals to input terminals of a multiple~cer 1101. As noted above, the FLIT signals comprise four signals in parallel. The high-order GATED FLIT signals, identified as GATED FLIT (3:2) sigDals, are coupled to one set of input terrninals of muldplexer 1101, and the low-order GATEDFLIT signals, identified as GATEE) FLIT
(1:0) signals, are coupled to a second sct of input terminals of multiplexer 1101.
As noted above, if the FLlT signals represent flits in the down path identification portion 41, the high-order bits ;D each flit, which are represented by the high-order GATED FLIT(~:2) signals, -: ,. , , , ,: . : . ,, :: .,., . , ., :.

:, , : . . : : . , , . . . , , ::

WO 92/06436 P~r/llS9l/07383 , 2 ~ 9 3 ~ 35-are used in a data router node æ(iJ,k) at an even-Dumbered level ~i~ to deterrnine the child to receive the data router message packet 30. On the other ha~ld, the low-order bits in each flit, which are represented by the low-order GATED FLI~ 0) signals, are used in a data souter node 22(i~,k) at an odd-numbered level ~i~ to determine the child. In addidon, the low-order NODE HEIGHT(0) signaL
which if asserted indicates that the data router node 22(i,~,k) is at an odd-numbered level and if negated indicates that it is at an even-numbered Jevel.
Accordingly, the NODE HEIGHT (0) signal controls the multiplexer 1101. If the NODE
HEIGHT (0) signal is asserted, multiplexer 1101 couples the &ATED FLrr (1:0) signals as SEL DN
ID (1:0) selected down path identification signals to an input terminal of a second multiplexer 1102.
On the other hand, if the NODE HEIGHl (0) signal is negated, îhe ~ultiplexer 1101 couples the GATED Fl IT (3:2) signals as the SEL DN ID (1:0) signals.
Multiplexer 1102 receives the SEL DN ID (1:0) selected down path identification signals at one set of input terminals. At a second set of input terminals, the multiplexer 1102 receiYes CHILD
MAP (1:0) child map signals. The CHILD MAP (1:0) signals are provided by the Dode control circuit 1004 (Flg. 1LA) to identify, for each of the input child and parent interface circuits 1006(i) and 1010(i), one output child circuit 1007(i) to wbich data router message packets 30 are to be coupled while the data router 15 is in all-fall-down rnode. The multiplexer 1102 is controlled by an AFD DN PA SEL all-fall-down dowll path select signal from an all-fall-down latch circmt 1103.
The all-fall-down latch circuit 1104 includes a multiplexer 1104 which receives the A~D
MODE all-fall-down mode signal at one input terminal. If the input control circuit 1041 is asserting the MSG INIT message initializatiorl signal, the multiplexer 1104 couples the AFD MODE all-fall-down mode to the input ter~inal of a flip-flop 1105. The flip-nop 1105 is set or clear in response to the next tick of the NODE CLK signal to generate an asserted or negated AFD DN PA SEL AFD MODE
all-fall-down down path select signal. As noted above, the input control circuit 1041 maintains ~he MSG INIT signal asserted for only one tick of the NODE CLK signal followiDg receipt of FLlT signals representing the first flit of a data router message packet 30. Thereafter, the MSG INIT signal is negated, which enables the multiplexer 1104 to couple the AFD DN PA SEL signal to the data input terminal of the nip-nOp 11Q5.
Accordingly, the flip-flop 1105 mahtains the AFD DN PA SEL all-fall-down down path select signal in a constant coDdition a~er the flit flow control ~ircuit 1040 receives the first nit of the data router message packet 30. If the AFD MODE all-fall-down mode signal is at that point negated, indicatiDg that the data router 15 is not in all-fall-down mode, the flip-flop 1105 is clear, thereby negating the AFD DN PA SEL signal. On the other hand, if the AFD MODE all-fall-down mode signal is asserted, indicating that the data router 15 is in all-fall-do~vn mode, the ni~nOp 1105 is set, thereby ~sserti~g Lhe AFD DN PA SEL SigDal.
If the AFD DN PA SEL signal is Degated, the multiple~er 1102 is enabled to couple tl~e SEL
DN ID (1:0) selected down path ideDtificadon signal as the OUT REQ lC(1:0~] output request signals In this condition, the OUT Rl~Q IC(l:O)I signals are derived from the FLIT sigllals representing tlits of , : , ,:
" , , WO 92t06436 PCI/U!~9l/07383 20933~
.~
the down path identification portion 41 of the data rouler message packet 30. If, on the other hand, the AFD MODE all-fall-down rnode signal is asserted, the multiplexer 1102 is enabled to couple the CHILD MAP (1:0) sigllals as the OUT REQ ~C(1:0)] signals.
It will be appreciated that the pareDt request generator portion 10g0 and child request generator portioD 1091 will resyond to FLI-r signals representiDg all of the successive flits received by the nit flow control portion 1040. The push signal generator portion 1092, which generates the OIF
PUSH output identification FIFO push signal, enables the output request identification FIFO 1023 to load the OUT REQ [P,C(1:0)] signals wheD they are base~d on FLIT signals represeDting the header field 40 and first flit of the down path identification portioD 41. The push signal generator portion includes a multiplexer 1110 tha~ receives at one iDput terminal the FIRST FLIT sig~al from AND gate 1052 (Fig. 11B-2B). As ~oted above, the FIRST FLIT signal is asserted when the FLIT signals represent the header field 40 of the data router message packet 30 being received. It vill be appreciated that at that point the HDR< header value less than node height signal received by the .
AND gate 1093 (Fig. 11B-2C) corresponds to the result of the comparison between the node height and the value of the header field 40 as perfor~ned by comparator 105o.
The outpu~ of multiplexer 1110 is coupled to the data input terminal of a flip-flop 1111, which is set in response to the next tick of the NODE CLK signal to assert an OIF PUSH EN output identification P~FO push enable signal. It will be appreciated that at that point, the flip-~lop 1096 in the parent request generator portion 1090 also latches the signal fro~n the multiplexer 1095 that represents the state of the GO UP signal. AccordiDgly, the OIF PUSH EN signal is asserted at the same point that the OUT REQ lPl signal indicates whether the data router message packet 30 is to be transmitted up the tree defining the data router L5.
Contemporaneously, if the VAL FLIT signal is asserted, the GATED FLIT signals will represent the first flit of the down path identification portion 41 of the data router message packet 30 being receive. Thus, the OUT REQ [C(1:0)] signals will identify a down path identifier. Since the VAL FLIT and OIF PUSH EN output identification FIFO push enable signa]s are asserted, an AND
gate 111 is e3~ergized to assert the OIF PUSH output identification FIFO push signal, wbich is coupled to the output request identification FIFO 1022 (Fig. 11B). The output request identification ~FO 1022 loads the OUT REQ IP,C(l:O)l signals at the next tick of the NODE CLK signal.
As noted above, the FIRST FLIT signal is asserted by AND gate 1052 only while the Fl,lT
signals represent the header field 1041. Auordingly, at ~hat point, the FIRST FLIT signal will be negated. Since the VAL FLIT signal is asserted, multipla~er 1110 couples a negated signal to the data input terminal of fli~flop 1111, which is reset at the next tick of the NODE CLK signal to negate the OIF PUSH EN output identification FIFO push enable signal. Since the OIF PUSH ~N signal is negated, the OIF PUSH output identification FIFO push signal ~ill also be negated.
It ~ill be appreciated that the flip-flop 1096 in parent request generator portion 1090 and flip-flop 1111 in pusb signal generator portion 1092 effedively correspond to delay lines. The respective flip-flops delay the respective GO UP and OIF PUSH EN output identification FIFO push enable : ~ , , ., : ,, :~, -, ::, . . .

,, ,. ,, :. , :: ::, ,;. . , ~

WO 92/0~436 PCI/US91/07383 2~33~ -137-signals so that they will be coupled to the output request identification FIFO 1023 coniemporaneously with the generatioD by the child request generator portion 1091 of the OUT REQ IC(1:0)] signals in response to FLIT signals representing the first flit of the down path identification portion 41 of the data router message packet 30 being received. It will be apprecia~ed, however that the FLIT signals representing the first flit of the dowD path identification portion 41 may be stalled. In that case, the VAL FLlT valid flit signal be negated. The negated VAL FLIT signal enables the multiplexers 1095 and 1110 to, in turn, eDable respective flip-fiops 1096 and 1111 to maintaiD their respective condition at subsequeDt ticks of the NODE CLK signal. The negated VAL FLIT signal also &ables AND gate 1112, which also negates Ihe OIF PUSH output identification FIFO push signaL inhibit~g the output request identification FIFO 1022 from loading the OUT REQ [P,C(1:0)l signals.
Figs. 11B-2D and 11B-2E depict detailed diagrams of, respectively, the data flit counter 1043 and down patb identification flit counter 1042. With reference to Fig. 11B-2D, the data nit counter 1042 comprises two counters, namely, a binary counter 1114 and a ring counter 1115. As noted above, the valuo in the message ieDgth field 34 of a data router message packet 30 identifies the number of thirty-two bit words contained in the data flits 36 in the data portion 32, aDd each thirty-two bit word is contained in eight successive four-bit flits. In that case, the ring counter 1115 decrements when each flit is received and counts out after rece;pt of the number of flits containing each word. When the r~ng counter 1115 counts out, it eDables ~he binary counter 1114 to decremen~. At that point, the nng counter is re-initialized and resumes decrementing while flits for the next word are being received These operations are repeated until the binary counter 1114 has counted out and the ring counter 1115 has almost counted out, AND gate 1116 is energ zed to assert the NR DATA END Dear data end signal. When the ring counter 1115 later counts out, an AND gate 1117 is energized to assert the DATA END signal.
More particularly, the FLIT signals are coupled to the dlata input terminals of binary couDter 1114. When the input cor,trol circuit 1041 asserts the MSG LEN IN message, at which point the FLIT
signals represent the flit corresponding to the message length field 34 of the data router message packet 30, the binary counter 1114 loads the FLrr signals. Contemporaneously, the ring counter 1115 is initialized to load a value correspondiDg to the number of flits in the data portion 32 of a data router message packet 30 are reqmred to hold a thirty-two bit data word. SiDce in one embodiment eight flits are required, the ring couDter 1115 has eight bits. It will be appreciated that, in embodiments having a different number of flits for each data word, the riDg couDter 1115 may have a correspooding differe~t number of bits. To accommodate the addidonal flit for the tag field 35, the low-order bit of the nng counter 1115 is eDergized to load a value of ~one,~ and the other bits are de-energized to load values of ~zero.~
While the VAL FLIT valid signal is asserted, indicating that FLIT signals represeD~afive of flils of the data router message packet 30 are being received, the ring counter 115 is enabled to decremen~.
While enabled, since the flit flow control circuit 1040 receives successive flits in synchronism with successive dcks of the NODE CLK signaL the riDg counter decrements at each tick of the NODE CLK
s~

WO 92/06436 PCr/US91/07383 2Q933a5 -138' ' ' ' At the first tick of the NODE CLK signal after being loaded, which occurs when the nit contair~ing the tag field 35 is being received, the ring counter 11~ sequences to ener~ze its high-order bit. It will be appreciated that that high-order bit is ener~zed contemporaneously with the receipt by the flit flow control circuit 1040 of FLIT sigllals corresponding to the first data Dit 36. While the V~L
FLIT signal is asserted, in synchrony with successive ticks of the NODE CLK signa-, the flit flow control circuit 1040 receives successive data flits 36 of the data router message packet 30 being received, and wheh it counts out, ~he number of data flits 36 have been rcceived corresponding to the Dumber of flits in a thirq-two bit word.
At that point, the ring counter 1115 asserts a FLIT/WORD D0 flits-per-word D0 s~nal, which energizes one input term nal of an A~D gate 1120. If the VAL FLIT signal is asserted, the other input terminal of AND gate 1120 is also energi7ed to assert a WORD CNT DN wort count down signal, which enables the binary counter 1114 to decrement. In response to the next tick of the NODE CLK
signal, the binary counter 1114 decremeDts. The binary counter 1114 transmits RCVD WORD
received word signals that identify, in binary-encoded form, the n~unber of thirty-two bit worts to be received.
While the VAL FLlT signal is asserted, since ~he counter 1115 is a ring counter, at the next tick of the NODE CLK signal after its low-order bit is energized, the high-order bit will be energized, and with successive NODE CLK signals the bit that is energ,zed Will correspond to the number of data flits 36 remaining to be received for the tl~irty-two bit data word. If the VAL FLll- signal is negated, indicadng tbat the recepdon of flits has been stalled, the ring counter 1115 stops decrementiDg. It resumes decrementing when the VAL FLlT signal is again asserted, indicadng that reeeptiol of nitS
has resumed.
These operations continue until the RCVD WORD received word signals are all negated. At that point, the RCVD WORD signals have a binary-encoded value of zero, indicatinp that the llit flow control circuit 1040 is receiving the data flits 36 representing the last thirty-two bit data word in .he data router message packet 30. The s~egated RCVD WORD signals are complemented to ener~2e an AND gate 1121, which asserts a LAST WORD signal. The asserted lAST WORD signal, in t~n, enables one input terrninal of AND gates 1116 aud 1117. Contemporaneously vitn tne reception by tne flit flow control circuit of the third to last data flit 36 in the data portion 32 of the d~ta router ~essage packet 30 being received, the ring counter 1115 asserts a Fl IT/WORD D2 flits per word D2 signal, which enables the second input terrninal of AND gate 1116. This energizes the AND gate to assert the NR DATA END near data end signal. As descri'oed above, the flit flow control circnit 1040 uses tnis signal in generating the NEOM near end-of-message signal.
Thereafter, contemporaneously with the reception by the flit flow contJol circuit of the last data flit 36 in the data portion 32, the ring counter asserts the FLIT/WORD D0 ~its per word DO
signal, which enables the second input terminal of AND gate 1117. This energizes the AND gate to assert the DATA END signal. As described above, the flit flow control circuit 1040 uses this signal in generating the EOM end-of message signal.

. - ,: ,. ~, , , : :., ~: ..: : .: ,. ,:,", ,. . : ~ ; : : . . :; :

WO 92/06436 PCr/US9l/07383 21~933~ ~J39~
It will be appreciated that the ring COuDter 1115 also generates the FLlTtWORD D2 and FLll /WORD D0 signals contemporaneously with reception by the flit f~ow control circuit 1040 of the third to last and last flits 36 of the data portion 32. However, except during reception of the last thir~y-two bit data word, the AND gate 1121 is de-energized, which maintains the LAST WORD sigDal negated, which consequently maintains the AND gates de-enerRized and the NR DATA END and DAT~ END signals negated.
F~. 11B-2E depicts a detailed logic diagram of the down path identification flit counter 1042.
With reference to Fig. 11B-2E, the counter 1W2 includes a decoder portion 1122 and a counter portion 1123. The decoder portion generates a plurality of CNT (i) count sigDals ("i~ is an integer from zero to eight) that identifies the number of flits in the down path idenlification portion 41 of the data router message packet 30 being received. The cour~ter portion 1123 decrements contemporaneously with the receipt by the flit flow control circuit 1040 of FLlT sig~lals representing the successive flits of the down path identificatioD portion, and generates the DNF END down flit end signal when the last ni~ in portion 41 is being received.
As described above, the header field 40 in a data router message packet 30 contains a value that essentially identifies the number of down path identificatio~ fields 42 in the down path identification portion 41. In addition, each flit in the down path identification portion 41 includes two down path iden~ification fields 42. Thus, if the value in header field 40 is an even number, the nurnber of nits in the down path ideDtification por~ion is one-half the value in Ihe header field 40. On the other hand, if the value in the header field 40 is an odd number, the Dumber of flits in the down path identificatiou portion is the one plus the greatest integer in one-half the value in the header field 40.
The decoder portion 1122 energizes the CNT (i) count signal whose index ";" corresponds to this number.
In particular, the decoder portion 1122 includes a decoder 1128 that receives the high-order FLIT (3:1) signals and generates in response thereto HALF FLIT (i) signals (~i~ is an iDteger from zero to eight). The high-order FLlr (3:1) signals represent a binary-encoded value correspondirlg to the greatest integer in one half of the binary-encoded value of the four-bit FLlT signals. The decoder 1128 asserts one of the HALF FLIr (i) sigoals whose index ~i~ corresponds to this value.
Tbe decoder portion 11æ also includes a set of multiplexers 1124(i) (~i~ is an integer from zero to eight). Each multiplexer 1124(i) generates oDe of the CNT (i3 count signals of corresponding index ~i~. In additioQ, eacb multiplexer 1124(i) receives, at one input tenninal the HALF FLIT (i) sig~al aud at the other input terrninal the E~ALF FLIT (i-1) signal from the decoder 1128.
l"he multiplexers 1124(i) are controlled i~ parallel by the low-order FLIT (0) signal, which, if negated, indicates that the down path identification portion 41 includes an even number of down path identification fields 42, and if asserted indicates tha~ it includes an odd number of down path identification fields 42. If the FLIT (0) signal h negated, indicating that the down path identification portion 41 contains an even number of flits, it enables the multiplexers 1124(i) to couple the HALF
FLIT (i) signals as the CNT (i) signal, so that the one of the CNT(i) signals tbat is asserted ., .,, ,, : ; . .

WO 92/06436 PCTtUS91/07383 2 0 9 3 3 ~ ~

corresponds to one-half the Yalue of ths header field 40. Oll the otber hand, if tbe FLIT (0) signal is asserted, indicating that the down path identification portion 41 contains an odd number of flits, it enables the multiplexers 1124(i) to couple the HALF Fl,rr (i-1) si~als as the CNT (i) signal. In that case, the one of the CNT (I) signals that is asserted corresponds ~o one plus one-half the value of the header field 40.
The counter portion 1123 includes a plurality of count stages 1~.25(i) (~i~ is an index from zero to eight) each of generates one DNF (i) down nit sigral (~i" is an index from zero to eight) wbcsc index ~i~ identifies the number of flits in the down path identificatioD portion 41 currently being received.
Eacb stage 1125(i) includes a multiplexer 1126(i) and a flip-flop 112~(i). Each multiplexer 1126(i) recei~res at one input terminal one of the CNT (i) signals of corrwponding index ~ ch multiplexer 1126(i) provides a SEL CN T (i) selected count si~al that is coupled to the data output terminal of the flip-flop 1127(i) of corresponding index ~ Each multiplexer 1126(i) also has an input terrninal connected to the data output tenninal of the flip-flop 11271i) and another input terminal cormected to the data output terminal of the fli~flop 1127(i+ 1). The multiplexers 1126(i) are controlled in parallel by the MSG INIT message initialization signal from the input control circuit 1141 and by a DNF CNT
GO down nit count go signal from an AND gate 1130. The flip-nops 1127(i) are clocked irl paraUel by the NODE CLK signal.
Prior to and during receipt by the nit flow con~rol circuit 1040 of the FLrr signals representing the header field 40 of a data router message packet 30, the MSG INIT message init;alization signal enables the multiplexers 1126(i) to couple the CNT (i) count signals as SEL CNT (i) selected count signals to the data input terrninals of flip-flDps 1127(i). When the flit flow control circuit 1040 is receiving FLll signals representing the header field 40, the decoder portion 11æ asserts one CNT (i') count signal whose index ~i"' identifies the number of flits iD the dowD flit identification portion 41 of tbe data router message packet 30. Eacb flip-flop 1127(i) latches the SEL CNT (i) signaL indudillg the SEL CNT (i') signal that is asserted, at the next ticl~ of the NODE CLK signal. The one nip-nOp 11~7(i') is set to assert the DNF (i') down nit si~a~ while the s)ther fli~flops are cleared to negate the DNF (i) down flit signals of other indices ~ At that point, the input control circuit 1041 negates the MSG INrr signal.
The AND gate 1~30 is controlled by the VAL FLIT valid nit signal and the complement of the MSG ~ message initialization signal as generated by an in~/erter 1L30. If the MSG INII signal is asserted, inverter 1131 maintairls the AND gate 1130 in a de-energized condition, so that the DNF
CNT GO down nit count go signal will remain negated. However when the MSG INIT 5ignal is negated, os~e input terminal of AND gate 1130 is enabled and the other input terminal, which is coDtrolled by the VAL FLIT valid nit signal, controls the eDerg~7ation of the AND gate 1~0 and thus the conditiou of the DNF CNT GO signal. Thus, while the VAL FLll signal is asserted, indicating that successive flits are being received, the DNF CNT GO sigDal is asserted, and otherwise it is Degated.
While the DNF CNT GO signal is asserted, the multiplexcrs 1126(i) are enabled to couple the DNF (i+1) down ~it signal as the SEL FLIl- (i) signal. As noted above, the correspondine flip-flops .: : - ,' . '' ' ' ; . : ' : . :. ' "' "' . -WO 92/06436 PCl`tUS91/07383 20~33~ 141-1127(i) latch ,he SEL FLIT (i) signal at each tick of the NODE CLK signal. Thus, wnile the DNF
CNT GO signal is asser,~ed, the index ~ of the one SEL FLIT (i') signal that is asserted is decremented at successive dcks of tne NODE CLK s~al. According',y, successive ticks of the NODE
CLK signal, the corresponding index ~ of DNF (i') down nit that is asserted is aL~,o decremented. It will be appreciated that at some point the flip-llop 1127(1) will be set to assert the DNF (1) signal, which corresponds to the DNF END down flit end sigr,al.
If dtuing t~is process ,~he VAL FLlT si~ial is ne~,ated, indicating a stall condition, tne AND
gate 1130 Degates the DNF CNT GO down flit count go s,gnal. Tne negated DNF CNT GO signal enables the multiplexers 1126(i) to couple the DNF (i) signals from their respect;ve flip-aops 1127(i) a~, tbe SEL CNT (i) selected couDt signals, instead of the DNF (i+ 1) signal from the flip-flop in the next higher indexed stage 1125(i+1). Thus, each flip-nop 1127(i), including the one flip-flop 1127(i') that is set, maintains it condition. When the VAL FLIT is again asserted~ indiuting termination of the staL
condition, the AND gate 1130 again asserts the DNF CNT GO signal, to enable the coul~ter porlion 1123 to operate as described above.
iv. Switch Input Control Ciscuit Fig. 11B-3 depicts a logic diagrarn of the switch input control circuit 1024 (Fig. 11B). With reference to Fig. 11B-3, Ihe s vitch input control circuit 1024 includes three primary sections. An output request section 1140 controls the obtaining of s vitch control information &om the output request identification FlFO 1023, decoding it, and providing the decoded infonnation to the switch 1003 (Flg. 1~A). A message flit control section 1141 controls retrieval of flits of data router message pacl;ets 30 stored in the input message FIFO loæ and transmittal to the switch 1003. In addition, the message flit control section 1141 receives an generates control signals for controlling tra~er of flits from the input message FIFO 1022 ~o the message flit control section 1141 and from the section 1141 to the switch 1003. Finally, a control section 1142 synchronizes the operadons of both the output request section and the message flit control section 1141.
The cortrol section 1142 includes a control circuit 1143 that is essentially a state machine. ~g.
11B-3A compr,ses a state transition diagram depicting the conditions of input signals under which the control circuit sequences from state to state. In each state transition, the control circuit 1143 changes state at a tick of the NODE CLK signal. Initially, the control circuit 1143 is io an idle state, as ideDtified by the box la'oelled "IDLE,~ and it remains there as long as the output request identification FIFO 1023. (Fig. 11B) is asserting tbe OIF EMPI'Y output identification FIFO empty signal. As noted above, i~ the output request identification FIFO 1023 is asserting the OIF EMPI`Y signal, it is empty.
If the output request identification FIFO 1123 becomes not empty, it negates the OIF EMPl Y signal, and the control circuit 1143 at tbe next tick of the NODE CLK signal sequerlces to the request pending state, as identified by the box labelled REQUEST PENDING. ln that state, the control circuit 1143 asserts a REQ PENDING request pending signal.
Contemporaneously, the output request identifica~ion FIFO 1023 transmits the new request as ORIF OUT REQ [P,C(1:0)] buffered output request signals. Returning to Fig. 11B-3, the ORIF OUT

. . ~ . .

' ' ,''', ~.' ' '~ ." '', ,' ',' ,:,,, ~'''- ' ;` ' ' WO 92~06436 PCl /US91/07383 20933~3 REQ IP,C(l:O)] signals are coupled to data iDput terrnir~s of a latch 1150, which latches them in response to the next tick of the NODE CLK signal. In response to the latched signals, the latch 1150 transmits a P REQ parent requested signal and binary-encoded C REQ (1:0) child reques~ signals.
The P REQ parent requested signal is enables one input ter~ninal of an AND gate 1152.
The C REQ(1:0) child reques~ signals are coupled to a decoder 1151, which decodes the C
REQ (1:0) signals and transrnits in response four signals identified as CO REQ through C3 REQ
(generally identified as "Ci REQ~ child ~i~ requested signals). The decoder 1151 asserts the one of the Ci REQ signals (Ri~ having a value from zero to three) that has the index ~i~ having the value identified by the binary encodin~ of the C REQ(1:0) signals.
The Ci REQ child ~i~ requested signals from decoder 1151 are coupled to a gated driver 1153.
If the P REQ parent requested signal is nol asserted, an inverter 1154 enables the gated driver 1153 ~o couple the Ci REQ signals to input terrninals of a second gated driver 1155. The REQ PENDING
request pending si~nal from the control circuit 1143 enables the AND gate 1152 to generate the OUl REQ P/SW output requested parent to switch signal and the gated driver 11~5 to ge~erate the OUT
REQ IC3:CO~/SW output request children to switch signals, which together fonn the OUT REQ
[P,C3:COJ output request signals (Fig. 11B) that are transmitted to the switch 1003. Thus, if the P REQ
parent requested signal is asserted, the asserted REQ PENDING signal will enable AND gate 1152 to assert the OUT REQ P/SW signal, and otherwise the OUT REQ P/SW signal will be negated.
Si~nilarly, if the P REQ signal is negated and one of the Ci REQ child ~i~ reguested signals is asserted, the asserted REQ PENDING signal will eDable the assertion of the one ~i-th~ OUT REQ [Cil/SW
output request to switch signal.
It will be appreciated that the inverter lW ensures that the gated driver 1153 will be disabled if the P REQ parent requested signal is asserted, ensuring that ~he request that the data router message packet 30 be transmitted to a parent data router node ~(iJ,k~ u,ill take precedence over a request that it be transmitted to a child data router node 22(iJ,k). This, in turn, ensures that the data router message packet 30 will be transrnitted up the tree defming data router 15 umtil it reaches a da~a router node 22(i,3,k) at the level ~i~ identi~led in the data router message packet 30 as origirally irar~nitted.
The REQ PENDING signal is also coupled to the data input termiDal of a flip-flop 1144 in the coDtrol sectior. 1142. The assertion of the REQ PENDING signal enables the fli~nop 1144 to be set at the next tick of the NODE CLK signal, enabliDg it to assert a DEL REQ PENDING delayed request pendirlg signal. The asserted DEL REQ PENDING signal eDables one input term~ral of an AND gate 1146 in the output request section. The asserted DEL REQ PENDING signal also enables a second flip-flop 1048 to be set at ~he next ticlc of the NODE CLK signaL enabling it to assert a DDEL
REQ PENI)ING delayed (tw~ce) request pending. The DEL RES~ PENDINC; aDd DDEL REQPENDING signals are thus asserted one and two ticks, respectively, of the NODE CL~C signal after assertion of the RE~Q PENDING signal. It will also be appreciated that the DEL REQ PENDING
and DDEL REQ PENDING signals will be negated one and two ticks, respec~ively, after negation of the REQ PENDING signal.

WO 92/06436 PCr/US91/07383 20S33~3 143-As will be described below in connection with Figs. 11C-1 through 11C-6, wben the switch 1003 receiYes one of Ihe OUT REQ IP,C3:CO] signals that is asserted, it perforrns an arbitra~ion operation in conneaion with requests from other input child and parent circuits 1006(i) and 1010(i) for a parent or the cbild identified by lhe asserted oU'r REQ [P,C3~ signal. At some point, the request will be granted, at which point the switch 1û03 asserts the REQ GRANTED request granted signal.
Tbe control circuit 1143 receives the REQ GRANTED signal, and when asserted sequences to the request granted state at the next tick of the NODE CLK signal (see Flg. 11B-3A). In tbat state, the control circuit 1143 negates the REQ PENDING request pending signal and asserts a l?EQ GR
request granted signal.
The asserted RFQ GR request grarlted signal e~ables the second input termirial of AND gate 1146. Since, as Doted above, the DEL REQ PENDING signal remains asserted for one tick of tbe NODE CLK sigual after negation of the REQ PENDING signal, both input terminals of AND gate 1146 will be enabled, tbereby energizing the AND gate to assert the OIF POP output identification FIFO pop signal. This enables the output request identification FIFO 1023 to couple new ORIF OllT
REQ [P,C(1:0)l signals to the output request section 1140, which caD be latched and decoded as described above and the resulting GATED C3:CO REQ and P REQ signals coupled to the respective input terminals of gated driver 1155 and AND gate 1152 to be available for gating as the OUT REQ
P,C3:CO] signals when the REQ PENDING signal is next asserted.
The asserted REQ GR request granted signal also enables an input terrninal of an AND gate 1160 ;D the message nit control section 1141. SiDce the l~DEL REQ PENDING delayed (twice) request pending signal is asserted, AND gate 1160 is energi~ed, which, in turn, energizes an OR gate 1161 to Csert a FLY signal. The Pl~Y signal is coupled to the data input termulal of a flip-flop 1162, which is set in response to the next tick of tbe NODE CLK signal to assert the SW FLY fly to switch signal, which is coupled to the switch 1003.
The }:LY signal is also coupled to the input message FIFO lOæ as the IMF POP input message FIFO pop signal. When the IMF POP signal is asserted, the input message PIFO 1022 is enabled to transmit FLIT/FIFO flit fsom FIFO signals representing, at successive ticks of the NODE
CLK signal, successive flits of the data router message packet 30.
At the point at which the OR gate 1161 first asserts the FLY signal, the input message ~IFQ
1022 (Fig. 11B) is transmitting FLIT/FIFO flit from FIFO signals representing the first fiit of the data router message packet 30. The FLIT/FIFO signals are buffered in a latch 1163, and are latched thereby in response to the ticks of the NODE CLK signal. The output signals transmilted by the latch 1163 comprise the SW FLIT flit to switch s;g~als that are transmitted to the switch 1003. While FLY
signal is negated, the input message FIFO 1022 maiDtains the FLIT/FIFO signals unchanged, representing the first flit of tbe data router message packet 3(), so that the SW FLIT signals continualJy represent the first flit at successive ticks of the NODE CLK signal. However, while the FLY signal is asserted, the asserted IMF POP signal enables the input message FIFO 1022 to transmit the successive nitS to the latch 1163, which latches them and ~ransmits them as the SW FLI~ signals to switch 1003.

~, .: - -:. -:., ' , , ; ~ .
:. : . , :

WO 92/06436 PCr/7.J5~1/07383 2 ~ ~ 3 3 ~ ~
-14~
As noted above, the DDEL REQ PENDING delayed (twice) request pending signal remains asserted for two ticks of the NODE CLK signal after the control circuit 1143 Degates the REQ
PENDING signal. When the DDEL REQ PENDING signal is negated, the AND ~ate 1160 is de-energized. At that point, the latch 1163 will have transmitted SW FLIT flit to swtch signals represeDting the first hvo flits of the data router message packet 30. To enable transmission of SW
FLIT signals representing additional flits, the switch 1003 asserts the SW FLOW flow from switch signal. The coincidence of the asserted REQ GR request granted and SW FLOW flow from switch signals energize an AND gate 1164, which maintains the OR gate 1161 energized.
The switch 1003 may, after receiving SW FLIT flit to switch signals representing the first two flits of the data router message packet 30, stop the flow of successive flits thereto by negating the SW
FLOW signal. If it negates the SW FLOW signal, the AND gate 1164 is de-energ~zed, which de-energizes the OR gate 1161 in turn negating the FLY signal. At the next tick of the NODE CLK
signal, the negated FLY signal clears flip-fi'op 1162, which negates the SW FLY signal. Negation of the FLY sigr~al also causes the IMF POP input message FIFO pop signal to be negated, which stops sequencirJg of the input message FIFO 1022. The s vitcb 1003 may thereafter enable the flow of the successive flits to resume by re-asserting the SW Fl.OW switch flow sigrial, which re-energizes the AND gate 1164 and OR gate 1161 to assert the Fl,Y and IMF POP signals, and enables the flip-flop to again be set to assert the SW FLY sig~
At some point in transmission of FLIT/FIFO flit frosn FIFO signals representing the successive flits of a data router message packet 30, the input message FIFO will assert the NEOM/FlFO Dear end-of-message from FIFO signal. As indicated above, the NEOM/FTFO signal is asserted contemporaneously with the FLIT/E~FO riit from FIFO signals representmg the third-from-last flit of the data router message packet 30. The NEOM/FIFO sigDal enables one iriput terminal of an AND gate 1165. Since the FLY signal is asserted, the AND gate 1165 is energized, which enables the direct-set input terminals of two tli~flops 1166 arid 1167, in trirn setting both nip-tlops. The set flip-flop 1166 asserts a PRERELEASE signal.
Returning to Fig. llB-3A, the assertion of the PRERELEASE signal enables ~ie control arcuit 11U to sequence to another state. If Ihe OIF EM~Y signal is contemporaDeously asser~ed, iridicating that the output request identification FIFO 10~3 is empty, the cootrol arcu~t 1143 seguences to the idle state at the next tick of the NODE CLK si~aL and negates the REQ GR request granted signal. On the other hiand, if the OIF EMPIY signal is negated, the control circuit returns to the request pending state. In the request pending state, the coDtrol circuit 1143 also negates the REQ GR
signal and also asserts the REQ PENDING request pending signal.
Since in either case the REQ GR signal is negated, the AND gate 1164 is de-energized.
However, the set fli~flop 1167 asserts an ALMOST DONE signal, which maintains the OR gate 1161 in an energi~ed conditioD, in turn maintaining the FLY signal asserted. At this point, since the switch 1003 cannot, through the SW FLOW flow from switch signal, control the FLY and SW FLY signals, it accepts the SW FLIT signals representing the last few nitS of the data router message packet 30.

"

. .: :: , : : , : . : ::. , ,:. , .:,. :: : : ,:: , . . : -, : : :: : :: :
-. - -: : . , : ,:: : ~ : ::: : : . . , : . .
: : : :: : .: : ::: : : ., j :
.. :. : :,: :: . ::,.:.:: , ~ : ,: . : , .:

WO 92/06436 PCl /US91/~7383 2~9~353 -145- !
As noted above, the flip flop 1166 asserts the PRERELEASE sigoal at the same point that the SW FIIT signals ~om latch 1163 represent the third-from-last flit of the data router message packet 30. The asserted PRERELEASE signal also enables the data input terminal of a flip-flop 1168, which ~, is set in response to the next Iick of the NODE CLK signal. The set flip-flop 1168 asserts the SW
RELEASE releasç to switch signal. It will be appreciated that the SW RELEASE signal is asserted contemporaneous with the second-to-last flit of the data rouler message packet 30. The SW
RELEASE signal also controls the direct-reset input terminal of flip-flop 1167, and resets the flip-flop when the signal is asserted. The reset aip-nOp 1167 nçgates the PRERFLEASE signal, which enables the flip-flop 1168 to be reset in response to the next tick of the NODE CLK signal. It will be appreciated that at that poin~, the SW FLlT flit to switch signals will represent the last nit of the data router message packet 30.
At the same time, the input message FIFO 1022 will be asserthg the EOM/FlFO end-of-message from FIFO signal. Sir ce the ALMOST DONE sigDal is stiU asserted, energiang OR gate 1163, the FLY signal is also asserted. The coincider ce of assertion of the EOM/FIFO and FI,Y sig~als energizes an AND gate 1169, which energizes the direct-reset input terminal of flip-tlop 11~7. This enables the flip-aop 1167 to be reset, in turn negating the FLY and IMF POP input message FIFO pop sigQals. At the next tick of the NODE CLK sigDal, the flip-flop 1163 is reset, negating the SW FLY
signal.
As noted above, the control circui~ 1143 sequences from the request granted state ~o either the idle state or the request pending state at the s~une point that the SW FLrr signals represent the third-to-last flit of the data router message packet 30, the particular state depending on the state of the OIF
~M~I-Y signal. If the control circuit 1143 sequences to the idle state, the switch input control circuit 1024 can repeat op~ratioDs, as desc~ibed above, when the OIF EMPl-Y signal is negated. If, OD the other hand, the cs)ntrol circuit 1143 sequences to Ihe request pending state, it will be appreciated that the REQ PENDING signal will be contemporaneously asserted, enabling the AND gate 1152 and gated dri~/er 1155 to couple OUT REQ [P,C3:C0]/SW output request to switch sigDals to ~he switch 1003. As w;ll be described below in conDectiorl with Fgs. 11C-1 through 11C~, the switch 1003 may assert the REQ GRANTED signal, enabling the control circuit 1143 to sequence to the request granted s~ate, immediately after the message flit control circuit 1141 transmits SW FLIT nit to switch signals representing the last flit of a data router message pacl~et 30, to enable the message flit control circuit 1141 to immediately begin transmitting SW FLlT signals representing the first flit of the next data router message packet 30.
As described above, the input parent circuits 1010(i) are similar to the iDput ehild circuits 1006(i) described above in connection with Fgs. 11B through 11B-3A, with one exception noted here.
In particular, as described above, when data router nodes 22(iJ,k) begin passing a data router message packet 30 down ~he tree defining data router 15, the nodes 22(iJ,k) to not thereafter pass the packet 30 back up the tree. If an input parent circuit 1010(i) receives a data router message packet 30 the packet 30 is being passed down the tree. Thus, ~e input parent circuit 1010(i) will not enable the switch 1003 . ,. ., ,,. , - :
;: : ;; , .. . .
,, . .,.- ,. :. ~

. - . . - ~ , ~, , , WO92/06436 PCl/US~1/07383 2 ~ 9 3 3 ~ ~

to couple the packet 30 to an output paren~ circuit 1011(i), since that would pass the packet 30 back up the tree. Accordingly, the input parent circuits 1010(i) need not iDclude circuitry, in their respective input message control circuits 1021 and s vitch ioput control circuits 1024 for generating signals corresponding to the OUT REQ P output request parent s gnals in the oV'r REQ IP,C(l:O) sigrals and the our REQ lP,C3:CO]/SW signals as described above in connection with Figs. 11B through 11B-3A.
3. Switch 1003 i. General Figs. 11C-1 and 11C-2 together depict, in general block diagram form, s vitch 1003 (Fig. 1L~) in a data router node 22(iJ,k). The swiIch 1003 includes a control section 1200 shown in Flg. 11C-1 and a switching section 1201 sbown in Fig. 11C-2. With referellce initially to Fig. 11C-2, the switchiDg section 1201 includes a plurality of switch cells, each generally identified herein by reference numeral 1202(x,y). The details of a switch cell 1202(x,y) will be described below in connection with Flg. 11C~.
.. . . . . ..
As depicted in Fig. 11C-2, the switch cells are arraDged in a matrix haviDg a plurality of rows and columns. The switching cells in each row are associated wi~h a particular input child or parent circuit 1006(i) or 1010(i) (Fig. 11A) identified by the mnemonic identifier ~ICi~ (input child "Ci~) or ~
(input parent ~Pi~)~ where ~i~ is a~ index having values from zero to three. The switc~ing cells in each colurn~ depicted in Fig. 11C-2 are associated with a particular output c~ild or parent circuit 1007(i) or lOll(i), which are identified by the mnemonic ideutifier ~OCi~ (output child "Cin) or IPi (output parent ~Pi~), where ~i~ is an index haviDg values from æro to three. In the reference numeral 1202(x,y) for a switching cell, the index ~ refers to the input child or parent circuit associated with the ceD's row, and the index ~ refers to the output child or pa~ent circuit associated with tbe row's colur~
Each switch cell 1202(x,y) selectively couples the SW FLlT flit to switch signals from the input child or parent circuits 1006(i) or 1010(i) that is associated with the ce31's row to the output child or parent circuit 1007(i) or 1011(i) that is associated with the cell's COh~llD, under coD~rol of enabling signals from the control section 1200. The switch cell 1202(x,y) couples the SW FLlT signals received thereby as SW F~lT/Ci switched nit to child ~Ci~ signais or SW FLIT/Pi switched flit to parent ~Pi~
signals, depending on the output child or parent circuit 1007(i) or 1011(i) colmected theretD. For exa~nple, switch cell 1202(C0,C3) selectively couples the SW FLIT signals from input child circuit 1006(0) (which are identified on Fig. 11C-2 as C0/SW Fl l'r sigDals, where the ~Con ahead of the slash identifies the source of the SW FLIT signals) as SW FLIT/C3 s vitched fli~ to child C3 signals to output child circuit 1007(3). Sirnilarly, s vitch cell 1~02(C0,P0) selectively couples the SW FLlT signals from input child circuit 1006(0) to as SW FLlT/ output pareDt circuit 1011(0) .
Each switch cell 1202(x,y) also receives the SW FLY fly to switch signal from the corresponding input child or parent circuits 1006(i) and 1010(i) and selectively couples it to the output parent or child circuit 1007(i) or 1011(i) comlected thereto as the SW FLY/Ci switched ny to child ~Ci~
signal or SW FLY/Pi switched fly to parent nPin signal. The switch cell 1202(x,y) also receives a SW
FLOW/Pi flow to switch from parent ~Pi~ signal or SW FLOW/Ci flow to switch &om child ~Ci~ sig~laL

~. : .,.. . - . : :

20~3~5~ -147-as appropriate, from the i~put child or parent c~rcu~t 1011(i) or 1007(i) and selectively couples it to the input child or parent circuit 1006(i) or 1010(i) connectet thereto as the SW FLOW/Ci flow from switch to child ~Ci~ signal or SW FLOW/Yi flow from switch to parent "Pi~ signal. With reference to Fig. 11B, the input child circuit 1006(i) of child ~Ci~ interface circuit 1001(i) (Fig. 11A), for exa~nple, receives the SW FLOW/Ci signal as the SW FLOW flow from switch sigDal. Each input parent circuit lOlO(i) receives the SW FLOW/Pi sigrlal sirnilarly.
Each switch cell 1202(x,y) also receives a ci/sw RELEASE switch release from child "Ci"
signal or the Pi/SW RELEASE switcn release from parent ~Pi" signal from the input child or parent circuit 1006(i) or 1010(i) connected tnereto. Each Ci/SW RELEASE signal is directed to all of the switch cells 1~02(Ci,y) arld, when assened disables the one that is coupling tne Ci/SW FLIT swltch to flit from child ~Ci~ signals to an output child or parent circuit 1007(i) or 1011(i) connected thereto. At thal point, the switching section 1200 generates a ~Y~ SEL EN select enable signal indicating that the cells 1202(x,y) connected to output child or parent circuit 1007(y) or 1011(y~ are available for seleaion.
It should be noted that the columns associated with the output parent circuits 1011( i) in the embodiment depicted in Flg. 11C-2 do not illclude includes swilching cells in rows associaled with the input parent circuits 1010(i). In thal embodimenl, as noled above, when the dala router nodes æ(i,j,k) begin directing a data router message packet 30 down the tree definiDg data router 15, they do no~
thereafter direct the packet 30 back up the tree. Thus, if the switching circuit 1201 recei~res a data router message packet 30 frDm an input parent circuit 1010(i), i~ will not direct the packet 30 to an output parent circuit 1011(i), and so the s~nlching circuil 1201 does nol need switchi~g cells 1202(x,y) therefor.
The control section 1200, which is shown generally in Fig. 11C-1, generates selection control signals that selectively enable the cells 1202(x,y). ID this operation, the control section 1200 uses the Ci OUT REQ lP,C3:CO]/SW output request from child ~Ci" to switch signals and Pi OUT REQ
[P,C3:C0]/SW ou~put request from parent ~Pi~ to switch s~gnals from the input child and parent circuits 1006(i) and 1010(i), and P3:P0 SEL EN output parent selection enable signals and Ci SEL EN output child selection enable signals from the switching section 1201. ~The "Ci~ or ~lPi" prefix in the mnemonic signal identifiers Ci OUT REQ IP,C3:COl/SW and Pi OUT REQ lP,C3:COl/SW identifies the source input child or parent circui~ 100o(i) or 1010(i). Thus, for example, the Ci oUr REQ
[P,C3:C0]/SW signals, for Ci corresponding to C0, comprise the oUr REQ IP,C3:C0]/SW depicted on F~ 11B.}. In response to all of these signals, the control section 1200 generates P3:P0 SEL lC3:C0]
selection signals and OCy SEL IP3:P0,C3:C01 selection signals l~y~ is an index identifying the particular output child circuits 1006(y)1.
The switch control section 1200 ircludes five circuits, including one parent arbitration circuit 1210 and four child arbitration circuits 1211(y) (~ being an index having integer values from ~ero to three). The parent arbitration c~rcuit 1210 recei~es C3:C0 REQ P child requests parent signals aDd P3:P0 SEL EN parent select enable signals and geDerates the P3:P0 SEL [C3:C0] output parent selection signals irl response. The C3:C0 REQ P signals comprise the parent request portions of the Ci - , , , . , . :

, ~
, . .. . . :
. . ; , ~ , , .
.

WO 92/06436 2 ~ ~ 3 ~D~/US91/07383 OUT REQ [P,C3:C0]/SW output request from child ~Ci~ to switch signals from the input child circuits ~(i).
The P3:P0 SEL [C3:COl selection signals generated by the parent arbitralion circuit 1210 comprise sixteen signals that control the switching cells in the columns associated with the output parent c rcui~r. 1011(i). Each signal, which has a mnemonic identifier of the forrm ~Py SEL Cx," whcn asserted enables the s vitching cell 1202(x,y). It will be appreciated that one Py SEL Cx signal may be asserted at arly given time for each value of ~. This ensures that SW P~T signals f~om only one input child circuit 1006(i) are coupled to an output parent circuit 1011(i) at any given time.
The parent arbitration circuit 1210 also transmits the P3:P0 SEL IC3:C0] selection signals to the input c~ild circuits 1006(i) as P GRANTS [C3:COl parent grants child signals. The P GR~NTS
[C3:C0] signals comprise four signals, each identified by the mDemonic ~P GRANTS Ci,~ one associated with each input child circuil 1006(i). When the parent arbitration circuit 1210 asserts a P3:PO SEL IC3:C0~ selection signal to enable a swit hjng cell 1202(x,y) iD the row of associated with the input child circuit 1006(i), it also asserts the P GRANTS Ci signal.
Each child arbitration circuit 1211(y) is associated with oDe column of switching cells 1202(x,y), identified by index ~y.~ Each child arbitration circuit 1211(y) receives P3:PO,C3:C0 REQ Cy parent and child request child signals and the OCy SEL EN output select enable signal from the associated column of switching cells 1202(x,y). In response, the child arbitra~ion circuit 1211(y) geDerates the OCy SEL IP3:PO,C3:C0~ output child selection signals for that column.
The OCy SEL IP3:POtC3:CO~ signals actually comprises eigbt sigDals, having the general mnemonic identifiers ~OCy SEL ~ and ~OCy SEL Cx", where "x~ identifies a input child or parent circuit and, thus, a pasticular row of switching cells 1202(x,y) in the switching section 1201. Each signal OCy SEL Px and OCy SEL Cx, when asserted, enables ~e switching cell 1202(x,y). It will be appreciated that orlly one of the OCy SEL Px and OCy SEL Cx sigDals can be assested at any given time. This ensures that SW Fl,IT signals from only one input child or pareDt circuit 1006(i) and 1010(i) are coupled to an output child circuit 1007(i) at aDy gh~en time.
Each child arbitration circuit 1211(y) also trans~nits the OCy SEL lP3:P0,C3:C0l selection signals to the input child and parent circuits 1006(i) and 1010(i) as Cy GRANTS p'3:P0,C3:C0j child grants parent and child signals. The Cy GRANTS IP3:PO,C3:COl signals comprise eight signals, each identified by lhe mnemonic ~Cy GRANTS pin one associated with each input parent circl~it 1010(i), or ~Cy GRANTS cin, one associated with each input child circuit 1006(i). WheD the child arbitration c ircuit 1211(y) asserts an OCy SEL IP3:P0,C3:C0] selection signal to enable a switchiDg cell 12()2(x,y) in the row associated with the input child or parent circuit 1006(i) or 1010(i), it also asserts the Cy GRANTS Pi or Cy GRANTS Ci signal.
The switch controi section 1200 also ORs together the P GRANTS Ci and Cy GRANTS Ci signals associated with each input child circuit 1006(i) to fonn the Ci REQ GRAN~D signal, which is coupled to the input child circuit 1006(i) as the REI) GRANTED signal (Figs. 11B and llB-3). The OR operation for input child circuit 1006(0) is represe~ted in Fig. 11C-1 by an OR gate 1212(0). It will 2 0 9 3 ~ 149-be appreciated that the switch control section 1200 will include aD OR gate to generate the corsesponding Ci REQ GRANTED sigDal for each of the other input child circuits 1006(i) in response to the P GRAN'I'S Ci and Cy GRANTS Ci signals. In addilion, the switch control section with include an OR gate to generate the corresponding Pi REQ GRANTED signals for each of the input parent circuits 1010(i) in response to the Cy GRAN~ Pi signals.
The parent arbitration circuit 1210, which will be described in more detail below, essentially comprises a two-dimensional priority chain. The circuit '''' Each child arbitration circuit 1211(i), which will be described in more detail below in connection with Flgs. 11C-3 and 11C 1, esseDtially comprises a one-dimensionaJ priority chain. In this connectioD, each child arbitration circwt 1211(y) assigns up to ei8ht requests, each represented by one of the eight P3:PO,C3:C0 REQ Ci signals, to one resource, which is represented by the column of switching cells 1202(x,y) associated with the output child or parent circuit 1007(y) or 1011(y). The child arbitration ~ircuits 1211(y) grants access to the resource on a priority basis, but where the priority rotates among the requesters so that no requester can be inhibited from accessing the resource for an undue length of time by large numbers of requests from other requesters.
ii. Switch Control Section a. Child Arbitration Circuit 1211(i) The details of a child arbitration circuit 1211(i) will be described in connection with Figs. 11C-3 and 11C-4. With reference to Fig. 11C-3, the child asbitration circuit 1~11(i) controlling access by the input child and parent interface circuits includes a binary arbitration tree 1213, an output circuit 1214 and a round-robin counter circuit 1215. The round-robiD counter circuit 1215 generates binary-encr>ded D(2:0) signals representing values from zero to seveD, each associated with olle of the eight request signals, that is, the four Cx REQ Ci input chiltl ~Cx4 requests output child "C;n signals and four Px RE(2 Ci input parent ~Px~ requests output child "Ci~ sigDals, that are received by the child arbitration circuit 1211(i).
The arbitration tree 1213 receives the eight Cx REQ Ci and Px REQ Ci request sigDals and selects oDe identified by the D(2:0) signals from the round-robiD counter ciscuit 1215, and asserts a C
SEL input child ~Cx" select signal or a Px SEL input parent ~Px" select signal in response. T~e output circuit 1214 asserts a corresponding Ci SEL Cx output child ~Ci~ selects input child ~Cx~ signal or (::i SEL Px output child ~Cx" selects input parent ~Px" signal when the switching section 1201 asserts the Ci SEL EN output child "Ci~ select enable signal. Contemporaneously, the l)utput circuit 1214 asserts the correspondiDg Ci C;RANTS Cx or Ci G}~ANl S C~ signal, which, as des~ibed above in colmection vith Fg. 11C-1, is used by the switch control sectioD 1200 in generating the Cx REQ GRAN~D signal for transmission to the selected input child circuit 1006(x), or a correspondiDg signal for transmission to the selected input parent circuit 1010(x). A~s described above in connection with Figs. 11B through 11B-3A, the input child or parent circuit 1006(x) or 1010(~) then Degates the Cx REQ Ci or Px REQ Ci request signal.
If the selected input pareDt or child circuit 1006(i) or 1010(i) is not assert~ng the one of the Cx REQ Ci or Px REQ Ci signals associated with the current value of the D(2:0) signals, the round-robin . ,;:
,: . . . .

:. :: . : . . .. . .
. . .. .
.. ... . . .
.. . ,: . :: . , , - : . .
, ., , . ,: .. , .: . . . . ,, ,, .:, ,,, .. -: . .

WO 92/06436 PCI'/US91/07383 ~ ~3 9 3 ~ r r counter increments until the value of the D(2:0) signals is associated witb an asserted Cx REQ Ci or I'%
REQ Ci signal. Wben tbe counter 1215 reacbes sucb a value, it stops incrementing. In addition, tbe arbitration tree 12L3 at that point asserts the Cx SEL input cbild sele~ signal or P:~ SEL input parent select signal that is associated vith that biDary-encoded value of tbe D(2:0) s~gnals. Thus, the round-robin counter 121S ensures that priority among the Cx REQ Ci and Px REQ Ci signals rotates, so that the input cbild and parent circuits 1006(i) and 1010(i) aU have a reasonably equal lil~elihood of being selected, and that none are irlhibited from coupling data router message packets 30 for an unduly long time.
Generally, the arbitration tree 1213 comprises arbitration cells generally identified by reference numeral ~216(iJ) organized a plurality of ~evels where index ~i~ identifies the level of the arbitration cell in the arbitration tree 1213 and index ~j~ uniquely identifies the arbitration cell among others in the same level. Each arbitration cell ~16(i,j) performs two geDeral operations. First, each arbitration cell 1216(i,j) receives request signals from two input child or parent circuits 1006(i) or lOlO(i), or from two arbitration cells 1216(i-1) in the next lower level, and generates a consolidated request signal that is the OlR thereof.
Second, each arbitration cell 1216(i,j) performs a pair-wise arbitration determination in response to (a) the request or consolidated request signals from the input child or parent circuits 1006(i) and 1010(i), or the consolidated request signals, (b) UDary preference signals ~hat generated by the next lower level 1216(i-1j), and (c) the D(i) signals for the particular level (i). In that operatio4 the arbitration cell 1216(i,j) generates unary preference signals to iderltify one of the input child or parent circuits 1006(x) or 1010(x) that is asserting a request signal, for use by the arbitration cell 1216(i+1,j) in the arbitration tree 1213. Thus, each arbitration cell L~16(i,j) produces a preferenee signal for each of the input child or parent circuits 1006(i) or 1010(i) that is coDnected to those arbitration cells 1216(0,j) ;D the first level which are in the sub-tree depending from the arbitration cell 12~6(i~)).
For e~Tnple, the arbitration cell 1216(0,0) in the first level receives C0 REQ Ci and C1 REQ
Ci input child "C0" and ~C1" request output child ~Ci~ signals and asserts a C0/Cl REQ Ci input child ~CO~ or "C1~ requests output child ~Ci~ sigr,al if either of the C0 REQ Ci or C1 REQ Ci signals are asserted. In addition, arbitration cell 1216(0,0) asserts the one of unary-encoded C0/C1 PREF input child "C0~ or ~C1~ preferred signals to identify child ~ or child ~C1~ as being preferred. The C0/C1 PREF signals actually comprises two signals, one associated with the input child circlut "C0~ 1006(0) and the other associated ~vith input child ~Cl~ 1006(1), with at most one signal being asserted. The arbitration cell 1216(0,0) selects at most one of the C0/C1 PREF signals to be asserted in respoIIse to the conditions of the C0 REQ Ci and C1 REQ Ci request sigl~aLs9 CO PREF and Cl PREF child aco~
or ~Cl" preferred signals and a low-order D(0) round-robin count signal from round-robin cou~lter circuit 1215.
The otber arbitration cells ;D the first Ievel, namely celLs 1216(1) through 1216(3) operate similarly. It will be appreciated that D(0) round-robin select signal cnables the arbitration celLs - - , .. . . . ........... .. . .. i .. ... . ... .... . . . . . .

:. :- :: . . :. : : . . : :

WO 92~06436 PCI/US91/07383 ;,~
209335~
1216(0J) in the furst level, if both Cx REQ Ci or Px REQ Ci signals received thereby are asserted, to select one of the correspond~, input child or parent circuit whose preference signal is to be asserted.
If the D(0) signal is a~serted, the arbitration cell 1216(0J) will assert the preference signal that is associated with the input child circuit 1006(x) whose inde~ ~ is odd. On the other hand, if the D(0) signal is negated, the arbitration cell 1216(0J) will assert the pre&rence signal that is associated with the input child circuit 1006~x) whose index ~x~ is zero or even.
The arbitradon cell 1216(1,0) in the second level receives (a) the C0/C1 REQ Ci input cnild ~CO~ or ~C1~ rèquests output cbild ~Ci~ signal from arbitradon cell 1216(0,0) and (b) the C2/C3 REQ
Ci input child ~C0~ or ~C1~ requests output child ~Ci~ signal from arbitration cell 1216(0,1) and generates in response a C3:C0 REQ Ci input child requests output cnild ~Ci~ signal. The C3:C0 REQ
Ci signal is asserted if any of the Cx REQ Ci signals is asserted.
In addition, the arbitration cell 1216(1,0) generates C3:C0 PREF preference signals, which cornprises four unary-encoded signals each associated with one of the input child circuits 1006(3) through 1006(0). If the arbitration cell is asserting the C3:C0 REQ Ci signal, i~ also asserts one of the C3:C0 PREF signals. The arbitration cell 1216(1,0) uses the C0/C1 PREF and C2/C3 PREF
preference signals which it receives from the arbitration cells 1216(0,0) and 1216(0,1) in its sub-tree, along with the D(1) signal from the round-robin counter 1215. As noted above, at most one of ~e C0/C1 PREF signals asserted, and similarly at most one of the C2/C3 PREF signals will be asserted.
If one of the signals in each pair of C0/C1 PREF siguals al~d C2/C3 PREF signals is asserted, the D1 signal is used to select one of the C3:C0 PREF signals to be asserted. The arbitration cell 1216(1,1) operates similarly to generate the P3:P0 REQ input parent request signal and the P3:P0 PREF input parent preferred signals.
FlDally, the arbitration cell 1216(2,0) at the root of the arbitration tree 1213 Dperates similarly to generate an C3:C0/P3:P0 REQ Ci input child/input parent requests output child ~Ci~ signal in response to the C3:C0 REQ Ci input c~ild requests output child ~Ci~ signal and P3:P0 REQ Ci input parent requests output child ~Ci~ signal. In addition, the arbitration cell 1216(2,0~ operates similarly, in response to the C3:C0 REQ Ci and the P3:P0 REQ Ci request sigDals, the C3:C0 PREF and the P3:P0 PREF preference signals,-and the D(V signals, to generate una~-encoded C~ SEL iDpUt child ~Cx"
select sign01s and Px SEL input parent ~Px" select signals. If one of the C3:C0 PREF preference signals and one of the P3:P0 PREF preference signals is asserted, the D(2) signal will determine which of the Cx SEL signals or Px SEL signals ~rill be asserted. If the D(2) signal h asser~ed, the arbitration cell ~
assert one of the Px SEL sig~als, corresponding to the one of the P3:P0 PREF preference signals that h asserted. In addition, if the D(2) signal h negated, the arbitration cell 1216(2) ~ill assert one of the Cx SEL signaLs, corresponding to the one of the C3:C0 PR~P preference signals that h asserted.
A~ noted above, the round-robin counter circuit 1215 generates binary-encoded D(2:0) s;gnals having values between zero and seven, each of which h associated with one of the Cx REQ Ci or Px REQ Ci request signals from the input child and parent circuits 1006(i) or ml0(i). If the Cx REQ Ci or Px REQ Ci request signal associated with the culTent binary-encoded value of the D(2:0) signals is ~ : ' :': , . .. , ~ . , WO 92/06436 PClr/US91/07383 2~33'~7 not asserted, the round-robin counter increments until the value of the D(2:0) signals is associated with an asserted Cx REQ Ci or Px REQ Ci signal. As shown in Fig. 11C-3, the round-sobin counter circuit 1215 includes a birlary counter 1217, a multiplexer 1220 and an iDverter 1221. The binary counter 1217 generates the D(2:0) signaLs, which are identified in Flg. 11C-3 as signals D(2), D(1) and D(0).
In addition to beiug directed to the arbitration tree 1213, the D(2:0) signals are also directed to control input terminals of multiplexer 1220. The data input terminals of multiplexer 1220 receive the Cx REQ Ci and Px REQ Ci request signals, and the multiplexer 1220 couples as a SEL REQ selected request signal the one associated vith the binarv-encoded value of the D(2:0) signals. The inverter 1221, which controls an increment enable terminal of the counter 1217, complernents the SEL REQ
selected request signaL to enable the counter 1217 if the SEL REQ, and thus the selected Cx REQ Ci or Px REQ Ci signal, is negated. If the counter 1217 is enabled, it increments in response to successive ticks of the NODE CLK signal. Thus, the SEL REQ signal is asserted, enabling inverler 1221 to disable cosnter 1217, when the D(2:0) signals identify oDe of the Cx REQ Ci or Px REQ Ci signals thal is asserted.
The output circuit 1214 includes a pluraliq of AND gates 12æ(0) through 12~2(7), each of which generates a Ci SEL C:x or Ci SEL Px signal and an associated Ci GRANTS Cx or Ci GRANTS
Px signal. Each AND gate 1222(i) asserts its respective output signals in response to the coincidence of the corresponding Cx SEL or Px SEL signal and the C3:C0,P3:P0 2EQ Cl signal from the arbitration ceU 1216(2,0) and the Ci SEL EN select enable signal from the switching seaion 1201 (Fig. 11C-2). In addition, tbe output circmt 1214 includes an AND gate 1223 which generates a Ci TA~CEN signal in response to the coincidence of the C3:C0,P3:P0 RE~Q CI signal and the Ci SEL EN select enable signal. The Ci TAKEN signal is cs~upled to the node control circuit 1004 (Fig. l~A) to indicate when the arbitration circuit 1211(i) for a particular output child circuit 1007(i) has seleaed one of the input child or parent circuits 100o(i) or 1010(i).
Fig. 11C-4 depicts a !ogic diagram of an arbitration cell 1216(0,0) in the first level of the arbitration tree 1213. The other arbitration cells 1216(iJ) are similar. With reference to Flg. 11C-4, the arbitration cell mcludes an OR gate 1224 that seceives the C0 REQ Ci and C1 REQ Ci request signals from the input child circuits 1006(0) and 1006(1) aDd generates the C0/C1 REQ Ci consolidated request signal if either input signal is asserted.
The C0 REQ Ci and C1 REQ Ci signals are also coupled, along with the D(0) sigual from round-robin counter 1217, to a selector circuit læs. The selector arcuit includes an AND gate 1226 that generates a SEL S ~0 PREF EN select child ~C0~ preferred enable in response to the coincidence of the C0 REQ Ci request signal and the complement of a SEL C1 select child ~C1~ signal from an inverter 1226. The SEL C1 signal, iu turn, is generated by an AND gate 1230. If either or both of the D0 or C1 REQ Ci siguals is negated, the AND gate 1230 is disabled to negate the SEL C1 signal. The negated SEL C1 signal is, in turn, complemented by the in~erter 1227 to enable one iuput terr~inal of AND gate læ6. Thus, if the C0 REQ Ci sig~al is asserted, the AND gate læ6 asserts the SEL C0 PREF EN signal.

:

..

WO 92/06436 PCl/US91/07383 20~33S~
The asserted SEL CO PREF EN signal, in turn, energizes one input tenninal of an AND gate 1231, whose other input terminal is maintained in an enabled condidon by an asserted C0 PREF child ~CO~ preferred signal. The AND gate 1231 is thus energized to assert the one of the C0/C1 PREF
preferred signals associa~ed with input child circuit 1006(0). The asserted SEL C0 PREF EN signal, in turn, is colnplemented by an inverter 1232 to negate a SEL Cl PREF EN select child ~C1~ preferred enable signal. The negated SEL Cl PREF EN signal disables an AND gate 1233, causing it to negate the one of the COtCl PREF preferred signals associated with input child circuit 1006(1).
On the other hand, if both of the C1 REQ Ci and the D(0) signals are assened, the AND ga~e 1230 is energized to assert the SEL C1 signal. The asserted SEL C1 signal, in turn, enables the inverter 1227 to disable AND gate 1226, regardless of the condition of the C0 REQ Ci signal, to mainta~ the SEL C0 PREF EN signal in a negated condition. The negated SEL CO PREF EN signal disables AND
gate 1231, causmg it to negate the one of the C0/C1 PREF preferred signals associated with input child circuit 1006(0). On the other hand, inverter 1232 asserts the SEL C1 PREF EN signal, which enables one UlpUt terrninal of AND gate 1233. The other input terminal of AND gate 1233 is maintained in an enabled condition by an asserted C1 PREF chi~d ~C1" preferred signal. The AND gate 1233 is thus ener~zed to assert the one of the C0/C1 PREF signals associated with the input cDild circuit 1006(1).
As noted above, the olher arbitration cells 1216(iJ) are similar. In arbitration cells 1216(iJ) in the second and third levels (for which ~i~ in the respective reference numeral is "1~ or ~), it will be appreciated that there ~ill be multiple AND gates for each of AND gates 1231 and 1233, one for each of the uDary preferred signals received by the respective cell, controlled in parallel by the sigDals correspondiDg to the SEL C0 PREF EN and SEL C1 PREF EN signals.
b. Parent Arbitration Circuit 1210 F~ 11C-5 depicts the detailed circuit of the parent arbitradon circuit 1210. With refesence to F~g. 11C-5, the parent arbitration circuit 1210 includes a child request priority circuit 1240, a parent availability prioriq circuit 12~1 and a child request/parent availability match circuit 1242. Generally, the child request priority circuit 12~0 receives the C3:C0 REQ P child ~Ci~ requests parent signals ~rom the input child circuits 1006(i) and establishes priorities thereamong. The priorities periodically change, on a round-robin basis. The paîent availability priority circuit 1241 receives the P3:P0 SEL EN
parent select enable signals from the switching section 1201 and establishes priorities thereamong. The priorities established by the parent availability priori~y circuit also change periodically, but the priorities are established on a generally random basis. Fmally, the child request/parent availability match circuit matches child requests with parents available in the respective priorities, and geDerates ~e P3:P0 SEL
sigDals in response.
More specifically, the child request priority circuit 1240 receives the C3:C0 REQ P child ~Ci~
request parent signals from ihe iDput child circuits 1006(i) and generates, for each, a set of Ci REQ P
PRI (3:0) child ~Ci~ requests parent priority sigDals. Tbe Ci REQ P PRI (3:0) signals for each input child circuit 1006(i) comprise a pluralit~ of priority signals identif~ed by the mnemonic Ci REQ P PRI
(x), each representative of a priority level. The Ci REQ P PRI (x) signals for each input child port . . ..

- : , .
.~ :-, ~ .. . .

~IU9~3~5 1006(i), and for descending values of ~x" represen~s descendillg priority levels for the input child port ~(i).
The child request priority circuit 1240, in response to assertion by at least one input child circuit 1006(i) of a Ci REQ P signal, determines a priority ranking among the input child circuits. In that operation, the child request priority circuit 1040 makes use of a round-robin number geoerator 1243 and a child request enwnerator 1244. The rouDd-robin nwnber generator 1243 ~enerates RND-RBN PRI round-robin priority signals represeDting a value ~ which identifies o~e of the input child circuits 1006ti') as having the highest priority. The child request enumerator 124~ receives the RND-RBN PRI signals and asserts the Ci' REQ P PRI (3) signal for that input child circuit 1006(i'). The child request enurnerator 1244 also negates the Ci REQ P PRI (3) signals for the other input child circuits 10Qo(i) (where ~;" does not equal ~
If any of the other input child circuits 1006(i) are asserting their Ci REQ P child "Ci" requests parent sigDals, the child request enumerator 1244 asserts the Ci REQ P PRI (x) signals, for ~x" and ~i~
both in descending order (with the value of ~in returning to and descending from the ma~amwn value if the value represented by the value of the RND-RBN PRI signal is less than the ma~mum value of ~in).
Thus, i~, for example, tbe input child circuits 1006(0),1006(2) and 1006(3) are asserting their Ci REQ P
signals, and if the RND-RBN PRI signal has the value ni~ equalling ntwo,~ indicating that input chi!d circuit 1006(2) has the highest priority, the child request enurnerator 1244 ~ill assert the C2 REQ P
PRI (3) signaL representative of the highest priority. The child request enurnerator will negate the Ci REQ P PRI (3) signals for the other input child circuits.
Continuing with the example, since the irput child circuit 1006(1) is not asserting its C1 REQ
P signaL the child request enumerator 1244 wiD negate all of the C1 REQ P PRI (3:0) signals. The asscr'tion by the input child circuit 1006(1) of the C1 REQ P sigDal will enable the child request enur~lerator 1244 to assert the C0 REQ P PRI (2) signal, and to negate the Ci lREQ P PRI (2) signals for other i~put child circuits. Further, sillce the illput child circuit 1006(3) is the lowest priority as identified by the value represented by the RND-RBN PRI round-robin priority signal, the asserted C3 REQ P signal enables tbe child request eDumerator to assert the C3 REQ P PRI (1) signal, and to negate the C; REQ P PRI (1) signals for the other input child circuits. Final5y, since only three input child circuits 1006(i) are asser~.ing their Ci REQ P signals, the child request enumera~or 1244 does not assert any of the low priority Ci REQ P PRI (0) signals.
The round-robin number generator 1243 is essentially a counter that s enabled to increment by a round-robiD coDtrol circuit 1245. The round-robin control circuit 1245 receives the Ci REQ P
child ~Ci~ re~quests parent signals from all input child circuits 1006(i) and the P3:P0 SkL [C3:C0] signals and enables the round-robirl number generator 1243 to inemeDt under two circumstances. First, if the rouod robin control arcuit 1245 determines the coDditioD o~ the Ci REQ P sigl~al from the input c~ild circuit 1006(i) whose index ~i" corresponds to the value represe~ted by the RND-RBN PRI round robiD priority sigDals from the round-robiD number generator 1243, and, if it is negated, it eDables the rouDd-robiD nulDber generator 1243 to increment. Second, the round-robin control circuit 1245 also WO 92/06436 PCl/US91/0738~.

enables thQ roun~-r~ob~ number generator 1243 to increment if it determines that one of Ihe P3:P0 SEL lCi] signals, whose index ~ia corresponds to the value represented by the RND-RBN PRI round robin priority signals, is asserted.
The parent availability priority circuit 1241 receives the P3:P0 SEL EN pareDt select enable signals from the switching section 1201 and geDerates, for each, a set of Pi AVAIL PRI (3:0) parent ~Pi~ availability priority signals. The Pi AVAIL PRI (3:0) signals for each ~pjR comprise a plurality of priority signals ideDtified by the ;nnemonic Pi AVAIL PRI (x), each representative of a priority level.
The Pi AVAIL PRI (x) signals for each ~Pi~ and for descending values of ~xa represents descending priority levels for the ~Pi.^
The parent availability priority circuit 1242 operates in a ~nanner generally si~nilar to that of the child request priority circuit, except that the parent ~Pi~ having the highest priority is selected at random. The parent availabilily priority circuit 1242 includes a parent availability enurnerator 12'16 that operates in response to the P3:P0 SEL EN parent np;n select enable signal and RNDM NUM PRI
random-nwnber priority signals representing a random nwnber. The pareDt availability enurnerator 1246 operales in response to these signals in the same way that the child request enumerator 1244 operates in response to the Ci REQ P and RND-RBN PRI signals, respectively. The parent availability enumerator 1246 generates Pi AVAIL PRI l3:0) parent ~p;a availabiLity priority signals which identify the pr;ority level for each ~Pi,~ in a manner similar to the prioritics for each ~Ci~ as identified by Ci REQ P PRI (3:0) signals produced by child request enumerator 1244.
As noted above, the parent ~Pi~ having the highest priGrity, as established by the parent availability priority circuit 1241, is selected at random. This selection is enabled by the random values represented by the RNDM NUM PRI random number priority signals. The RNDM NUM PRI sigllals are gerlerated by a ra~dom number generator 1247, under control of a control circuit 1250. The control circuit 1250 receives the Ci REQ P request signals and the Ri SEL EN select cnable signals and enables the random number generator 1247 to generate a new r mdom Dumber in response to the coincidence of the conditions that (a) at least one of the Ci REQ P signals is asse~ed and (b) at least t vo of the Pi SEL EN select enable signals is asserted.
The child request/parent availability match circui~ 1242 includes a ConceDtrator 1251 that receives all of the Ci REQ P PRI (3:0)) input child ~Ci~ requests parent priority signals for all input child circuits 1006(i), and all of the Pi AYAIL PRI (3:0) parent ~Pi" availability priority signals for all ~pia arld generates in response thereto the Pi SEL IC3:C0] select sigDals that control the switching section 1201. In generating the Pi SEL [C3:COl select signals, the concentrator 1251 e~fectively deterrnines the one of the Cx3 REQ P PRI (3:0) signals ha~ing the highest priority ana the one of the Py3 AVAIL PRI (3:0) signals having the highest priority, and asserts the Py3 SEL Cx3 signal. This signal enables s vitching cell 1~02(x3,y3), so that the data router message packet 30 from input child circuit 1006(x3) to be coupled through the s vi~chiDg section 1201 and transmitted through output parerlt circuit lOll(y3). The concentrator 1251 does the same in connection vith the remaining Ci REQ P PRI (3:0) signals.

' ' ,: ! ' . ' , . . . .
.' , : ,: ' . ., '.' .: '... '., '. ,. ' :' ' ' '': '. : ' . ' ' .' . '. .. .:

WO 92/06436 PCI/US91/~7383 20933~

The concentrator 1251 also generates the P GRANTS Ci parent grants child ~Ci~ signals, which are used by the switch control section 1200 (Fig. 11C-1) in generating the Ci REQ GRAN I ED
signal. In particular, the concentrator 1251 generates each P GRANTS Ci signal as essentiaDy the OR
of the Pj SEL Ci parent ~P;" selects child ~Ci~ signals for all parents ~Pj.~
As noted above, in one embodiment of data router 15 the fan-out, going up the fat-tree def~g the data router, may .rary from level to level. At son3e le~/els, the fan-out is four, so that the data router node 22(iJ,k) will have four input child circuits 1006(i) and four output parent circuits lOll(i). At ot~er levels, the fan-out is two, so that the data router node 22(i,j,k) will have four input child circuits 1006(i) but only two active output parent circuits 1011(i). In that embodiment, the data router nodes æ(i,j,k) are all implemented in the circuitry, and so the circuitry for the other two output parent circuits 1011(i) will be present, but rendered inactive by the node coDtrol circuit 1003 (Fig. 11A).
In that case, the Pi SEL EN select enable signals for the inactive output parent circuits 1011(i) will be condnuaDy negated, and the parent availability enumerator 1246 will maintaihl the Pi AVAIL PRI (3:0) signals in a neBated condition. Since, in those levels, there will be fewer output parent circuits 1011~i) that input child circuits 1006(i), the round-robin priority maintained for the input child circuits 1006(i) by the child request priority circuit 1240 wiD ensure that the input child circuits 1006(i) will have be able to transmit their data router message packets 30 on a reasonably equal basis.
In addidon, it ~ill be appreciated tha~ the random-number priority provided for the Pi AVAIL
PRI (3:0) signals, under control of the random number generator 1247, will ensure that the data router message packets 30 going up the tree defming data router 15 are distributed randomly among ~he data router nodes æ(iJ,k). This minimr~es the likelihood of bottlenecks developing as data router message packets 30 are transmitted up the tree.
iii. Switch Cell 1202(i) F~ 11C~ depicts the detailed circuit of a switch cell 1202(0,0) in the switching seaion 1201 (Fig. 11C-1). All of the switch cells 1202 are generaDy sunilar. With reference to F~g. 11C-o, the switch cell 1202(0,0) receives the C0/SW FLl~r input child ~CO" flit to s vit~ signals in parallel OD a blls 1260, the C0/SW FLY input child ~C0~ ny to s~vitch signal on a line 1261, and C0/SW FLOW flow from switch to input child "C0~ signal on a line 1262. It will be appreciated that bus 12O0 and lines 1261 and 1262 extend through all switching cells 1202(0J) in the row of switching section 1201 associated vith input child circuit 1006(03. The switcb cell also irlcludes a bus 1263 ~hat cames SW F~lT/C0 flit from s~vitch to output child ~C0~ signals, a line 1264 that carries a SW FLY/C0 Sy from switch to output child ~C0~ signal, aud a line 1265 that carries a SW FLOW/C0 flow to switch from output child ~C0~
signal. Sirnilarly, bus 1263 and lineis 1264 and 1265 extend through all switching cells 12~ O) in the column of switchiDg cells assoaated with output child ci~cuit 1007(0).
The switch cell 1202(0,0) also includes three gated driver circuits 1270,1271 and 1272, tbat are controDed in parallel by a CELL EN cell enable signal. Gated driver circuit 1270 has an input terminal connected to bus 1260, and, when enabled, couples the C0/SW FLIT signals through its output terrninal onto the bus 1263 as the SW FLlT/C0 sigDals. S~milarly, gated driver circuit 1271 has an ' " '.. . , " : , ~ , ', " ' " ' ' : ' ,', ' ', ' ' ' ',';, ' : : . , , :
'' :::'' . '. ,.. .. '' :.. :: '' ,.,; :
. . : . ,': . . . . ,,, '',, .' ' . , .' . ".. '",. . ' .: ..

WO 92/06436 P~r/US91/07383 2093~ 157-input terr~inal cor~nected to line 12O1 and, when enabled, couples the SW FLY/C0 signal tbrough its output terminal onto to line 1264 as the SW FLY/C0 signal. Fmally, gated driver 1272 has an ~put terminal connected to line 1262 and, when enabled, it couples the SW FL,OW/C0 signal tbrough its an output tenninal onto line 1265 as the C0/SW FLOW signal.
As noted above, the gated drivers 1270, 1271 and 1272 are controlled in parallel by tbe CELL
EN cell enable signal. The CELL EN signal is controlled by a cell control circuit 1273. The celJ
control circuit 1273 asserts tbe CELL EN signal upon receipt of an asserted C0 SEL [C01 signal from arbitration circuit 1211(0), and negates the CELL EN signal upon receipt of an asserted C0/SW
RELEASE release to switcb from input child nCo~ signal on a line 1274. As witb bus 1260 and lines 1261 and 1262, the line 1274 e~dend through all switcning cells 1202(0,;) in the row of switching section 1201 associated with input child circuit 1006(0).
The control circuit 1273 includes a flip-flop 1275 whose data output terminal provides the CE13, EN cell enable sipnal. If the switch cell 1202(Q0) is not enabled, the tlip-flop 1275 is reset to negate the CELL EN signal. In that coDdition, a multiplexer 1276 is in condition to couple the C0 SEL
[C0~ signal as a LAT SEL latch select signal to the fJip-flop's direct set terrninal. The CELL EN cell enable signal is also complemented by an inverter 1280 to provide an asserted CELL(0,0) SEL ~N cell (0,0) select enable signa4 which is coupled by a multiplexer 1277 as a LAT REL latch release signal to the direct reset ter~ninal of 9ip-flop 1275 to maintain the flip-flop ~n a reset condition.
When the C0 SEL ~C0] signal is asserted to enable switch cell 12G2(0,0) the multiplexer 1276 asserts the LAT SEL signal, wnich sets tne flip-nop 1276 lo assert the CELL EN s pnal and enable gated dri~ers 1270, 1271 and 1272. I~l addition, the asserted CELL EN signal enables the multiplexer 1276 to couple the CELL EN sigllal as the LAT SEL latch select signal to maintain the nip-nOp 1275 in the set condition. In addition, inverter 1280 negates the CELL ~O,O)SEL EN signal.
The ne8ated CELL (O,O)SEL EN signal cnables multiplexer 1277 to couple a REL EN release enable sigDal from an AND gate 1281 signal as the LAT REL latch release signal. The REL EN si~al is genera~ed by an AND gate 1281. AND gate 1281 is energized, to assert the REL EN signal in response to the coincidence of the negation of the C0 SEL IC0] output child "CO" select i~put child "C0"
signal and the assertion of the C0/SW RELEASE release from input child ~C0~ to sw~ich signal. Thus, if either the CO SEL IC0~ signal is asserted, indicating that the switch cell 1202(0,0) is being selected by s~vitch control section 1200 (Fig. 11C-1) or the CO/SW RELEASE signal is negated, indicating that the switch cell is being released, the AND gate 1281 maintains the REL EN signal in a negated condidon.
Whe~, after selection of the switch cell 1202(0,0), the input child circuit 1006(0) 2sserts the CO/SW RELEASE signal at the end of a data router message packet 30, if the CO SEL [CO] signal is nogated the AND gate 1281 is energized to assert the REL EN signal. When that occurs, the muldplexer 1277 asserts the LAT REL signal to reset the flip-nop 1275 to negate the CELL EN cell enable signal, which, in turu, disables gated drivers 1270,1271 and 1272. ln addition, the negated CELL EN signal enables the inverter 1280 to assert the CELL (0,0) SEL EN signal. In that condition, the multiplexer 1276 couples the negated CO SEL ICO] signal as the LAT SEL to the direct set input 20~33~
.~
terminal of flip-flop 1275, and the multiplexer 1277 couples the asserted CELL (0,0) SEL EN signa] as the LAT REL signal to the fli~nop~s d rect reset tenninal, to mainta~ the flip-flop 1~75 in the reset condition.
It will 'oe appreciated that, wben the input child circuit 1006(0) asserts the C0/SW RELEASE
signal at the end of a data router message packet 30, if the C0 SEL ~C0] signal is asserted the AND
gate 1281 remains de-energized, to maintain the REL EN signal in a negated coldilion. When that occurs, the arbitration circuit 1211(0) u maintaining selection of s vitch cell 1202(0,0) for the next data router message packet 30 from input child circuit 1006(0), and so the switch control circuit 1273 maintairls the gated drivers 1270, 1271 and 1272 er abled.
As noted above, when the fli,~flop 1275 negates the CELL EN cell enable sig~aal at the end of a data router message packet 30, inverter 1280 asserts the CELL (0,0) SEL EN select enable signal indicatirlg that the switch cell 1202(0,0) is in a released condition. As shown on Fig. 11C-6, the CELL
(0,0) SEL EN signal is coupled to an AND gate 1282, which receives corresponding CELL (Oj) SEL
EN signals from the other switching cells 1202(0J) in the sarne cohunn of switching sedion 1201 and geDerates the C0 SEL EN select enable sigDal when all of these signals are asserted. It will be appreciated that all of the switching cells 1202(0j) in that column will be cor~ected to the same output child circuit 1007(0), and so if the C0 SEL EN sigllal is asserted, all of the s vitch cells 1202(0J) in the column will be i n the released conditiom As noted a'vove, the assertion of the CO SEL EN sig~al enables the arbitration circuit 1211(0) to perform select an input child or parent circuit 1006(i) or lOlO(i) to begin transmitting a data router message packet 30 through a switching cell 1202(0J) in the column to the output child circuit 1007(0).
4. Output Cnild Circuit 1007(0) F~. 11D depicts a detailed diagram of output cnild circuit 1007(0). With rcference to Fig.
11D, the output child circwt 1007(0) includes a switch interface section 1300, a buffer section 1301 and an output interface section 1302. Generally, the switch interface section 1300 receives flits of data router message packets 30 from the ~wi~cnmg section, provided frorn a~. input child or parent circuit 1006(i) or 1010(i), and couples them to the buffer section ~01. If the C0 OUT FLY signal from child data router node Z!(i,i,k) connected to the output child circuit 1007(0) is asserted, indicating that it can receive the ~it, the output interface section 1302 receives the flit from the buffer section and transmits it as CO OUT FLIT sig~als. In addidon, the buffer section 1302 enables the switch interface sectio~
1300 to maintain the SW FLOW/C0 signal asserted, which indicates that ~he output child circuit 1007(0) is able to receive additional flits.
On the other hand, if the C0 OUT F~Y signal beeomes Degated, indicating that the child data router node 22(iJ,k) is unable to receive additional flits, the output intetface section enables the buffer section 1301 to begi~ buffer~ng flits from the switch interface section. If the C0 OUl' FLY signal is later asserted, the output interface section L302 enables the buffer section 1301 to resume providing flits for it to transmit, which the buffer section W1 provides from the flits it has buffered. If at somc point the buffer section 1301 has buffered a number of flits, such that it becomes nearly full, it negates :.. ... . .. . . . .. . . . ..

WO 92/06436 PCl /US9l /07383 2~9335~ -159-the SW FLOW/C0 signal, which is coupled through the switch 1003 to the input child or parent circuit 1006(i) or lOlO(i) supplyir~, the nieS. The regated SW FLOW/C0 signal disables the input child or parent circuit 1006(i) or 1010(i). If the number of fli~s buffered in the buffer section 1301 thereafter is reduced, the buffer section 1301 may thereafter enable the switch interface section to ~ert the SW
FLOW/C0 signal to, in turn, enable the input child or parent circuit 1006(i) or 1010(i) to resume supply flits for transmission to the c~ild data router node 22(i,j,k).
More specifically, the switch interface section 1300 includes a latch 1303 that latches the SYV
FLlT/C0 flit from switch to output c~ild ~C0~ signals and a latch 1304 that latches the SW FLY/C0 ny from switch to output child ~C0~ signal from tbe switch 1003 in respoDse to the successive ticks of the NODE CLIC signal. In addition, the switch inter~ace section 1300 provides the SW FLOW/C0 si~al to the switch 1003. As desaibed above in connection with Fig. 11C-6, the enabled switch eell in s vitch 1003 couples the SW FLOW/C0 signal to the appropriate input child or parent circuit 1007(i) or lOll(i) as the Ci/SW FLOW or the Pi/SW FLOW signal. In response, the input child or parent circuit couples signals which the enabled s vitch cell will coup~e to the output child circuit 1007(0) as the SW
Fl IT/C0 and SW FLY/C0 signal. Accordingly, it will be appreciated that, if the SW Fl.Y/C0 signaJ is asserted at a tick of the NODE CLK signal, the SW FLIT/C0 signals represent a flit of a data router messa~e packet 30 being transmitted by the source input child or parent circuit.The SW FLIT/C0 signals are latched by latch ~03 at every tick of the NODE CLK signal.
Latch 1303 provides, at its output terminaLs, LAT OUT FLIT latched output flit signals. The LAT
OUT ~LIT signals are coupled to one input te~.~ninal of a multiplexer ~6. If a first-in first-out buffer 130S is em~pty, it asserts a FIFO EMPI'Y signal, enab~ing the FIFO 1306 to couple the LAT OUT FLIT
signals as BUF OUT FLIT buffered output flit signals to a gated driver 1313.
A latch 1312 latches' the CO OUT I:LY sig~al from the child data router node 22(iJ,k) at each tick of the NODE CLK signal. If the CO OUT FLY sipnal is asserted whetl the latch 1312 is enabled by the ticl~s of the NODE CL~C signal, the latch 1312 maintains the OUT FLOW signal asserted. If the OUT FLOW signal, and if an EN enable sigDal is asserted by node colltrol circuit 10~4 (Fig. llA) an AND gate asserts an EM OUT enable out signal. The asserted EN OI~T signaL ~n turn, eDables the gated driver 1313 to couple the BUF OIJT F LIT signals as GATED OUT FlIT signals to da~a input terminals of a latch 1315. The latch 13L5, hl turn, latches the GATED OUT FLIT signals at each tick of the NODE CLK sig~lal, and transmits the latched signals as C0 OUT FLIT signals to tlbe child data router node æ(i,j,k) cormected theretG.
Contemporaneously, since the SW FLIT/C0 signals at that point represent a flit of a data router message packet 30, the SW FLY/C0 signal is also asserted. The asserted SW E:LY/C0 signal enables a latch 1304 to l~e set in response to the NODE CLK signal to assert a LAT OUT FLY latched output fly signal. The asserted LAT OUT FLY signal is coupled to the buffer section 1301, in particular to a push enable terminal of a first-in first-out buffer (FIFO) 1305, to enable the lFlFO 1305 to buffer the flit represerted by the LAT OUT FLIT signals.
The asserted LAT OUT FLY signal also energizes an OR gate 1307 to assert and OUT E~LIT
PRESET signal. The asserted OUT FLIT PRESENT signal enables one input terminal of an AND

: - . . . ": ~, . :,;: ' : . : ',, ~:, " . . ~ : , :~ : :- . : ;:, , . : : - ~,. .

WO 92/06436 PCl /US9 1/07383 2~3~

gate 1311. The second input terminal of AND gate 1311 is controlled by the OUT FLOW signal from the output interface section L302, which at that point is asserted, energizing the AND gate to assert a POP signal. The POP signal eDables the FIPO 1305 to transmit the buffered flit as BUF FLIT signals.
ID addition, the FIFO 1305 negates the FIFO EMPTY SigDa], enabling the multiplexer 1306 to thereafter couple the LAT OUT FLIT sigl~als as the BUF OUT FLIT signals.
It will be appreciated that connecting multiplexer 1306 to selectiYely couple the IAT FLlT
OUT su~nals or the BI~F FLlT signals as the BUF OUT FLIT signals, in respODSe to a FIFO EM~I-Y
sigDal indicati~g whether the FIFO 1305 is empty, enables the signals from the switch 1003 to be coupled to the BUF OUT FLIT signals direc~ly, through multiplexer 1306, without requir~ng them Io be first buffered in F~FO 1305. This will eliminate any propagation delay through the FIFO 1305 if the FIFO 1305 is empty.
On the other harld, if the CO OUT FLY signal is negated, the latch ~312 will negate the OUT
FLOW. The disabled OUT FLOW signal disables AND gate 1314 to negate the EN OIJT signal, .
which, in turn, disables gated driver 1313 from coupling the BUF oUr FLlT signals as the GATED
OUT FLIT signals. Accordingly, the latch 1315 thereafter transmits negated CO OUT Fl.IT signals to the child data router node æ(i,i,k) connected thereto.
The rlegated OUT FLOW signal also disables AND gate 1311, thereby negating the POP
signal. ~ecordingly, while the latcb 1304 is asserting the LAT OUT F~Y signal, F~FO W5 is enabled to buîfer the LAT OUT FLIT signals, which represent successive flits of olle or more data router message packets 30, at successive ticks of the NODE CL}C signal. If the F3FO W5 becomes nearly full, it asserts a NR FULL signal, which is complemented by as~ inverter 1316 to negate the SW
FLOW/C0 signal. The negated SW Fl,OW/C0 signal, in turn, disables the input cnild or parent ~rcuit 1006(i) or 1010(i~ from CODtinUing to couple tlits thereto. In addition, the SW FLY/C0 signal will be negated, eDab'~ing the latch 1304 to Degate the LAT OUT FLY signal, wMch disables ~he FIFO 1305 from la~c~ the LAT OUT FLIl- signals.
WheD the child data router node 22(i,i,k) again asserts the CO OUT ~Y signal, tbe latch 13L2 asserts thc OUT FLOW signal at the ne~t tick of the NODE CLK signal. The asserted OUT FLOW
signal energizes AND gate 1314 to assert the EN OUT eDable out signal, which, in turn, enables gated driver 1313. The asserted OUT FLOW signal also enables one input terminal of AND gate 1311.
Since at this point the FIFO 1305 is not empty, the FIFO negates the FIFO EMm si~al, which is complemented by iDverter 1310 to energ~ze OR gate 1307 to assert the OUT FLIT PRESENT signal.
The asserted OUT FLIT PRESENT signal also energizes the second input terminal of AND gate 1311, enab~ing it to assert the POP signal. While the POP signal is asserted, the FIFO 1305, at successive ticks of the NODE CLK signal, transmits BUF FLlT sigDals representi~g the successive flits buffered therein as the BUF FLIT buffered nit signals. In addition, the negated nFO EM~I-Y signals enables the multiplexer L306 to couple the BUF PLIT sigoals as the BUF OUT FLIT signals to gated driver 1313. Since the gated driver is enabled, it couples the BUF PLIT OUT signals as the GATED OUT
FLIT signals to latch 1315.

- : :- ,, - : , : . :, ., :, .

., , . . , " . .. ;.;, .. :,.,. . . .: :, WO 92/06436 PC~/US91/07383 2~33S~ -161-When tbe conteDts of the F'IFO 1305 has been rcduced below a predetermiDed munber of buffered flits, it will negate the NR FULL signal, enabling, in turn, the in~erier 1316 to assert the SW
FLOWtCO signal. The asserted SW FLOW/C0 signal is coupled througb the switch 1003 to enable the appropriate input child or parent circuit 1006(i) or 1010(i) to resume transmitting nits thereto.
When the FIFO 1305 thereafter becomes empty, it re-asserts the FIFO EMPI Y signal, which conditions muldplexer ~6 to resume coupling the LAT OUT F'LIT si~als as the BUF OUT FLIT
.

. . .. . . ................................. .

- . ; . .. . :. - . . : :; : : :: .:: : :

WO 92/06436 PC~/US91/0738~s 20~3~5~

D. CQntrol Network 1. General Fip. 12A is a general block diagram of a control network node S1 osed in the coDtrol networl;
14 descnbed above, and Figs. 12B through 12D-1 comprise detailed bloc~ and logic diagrarns of the con~rol network node 51. With reference to Fig. 12A, the control network node 51 includes a flick up control portion 1401, a flick down control portion 1402 and an up/down comrnon portion 1403. The control network node 51 also includes a diagnostic network interfac~ 1404, which pJovides an interface to the diagnostic network 16. In additior., the control network node 51 includes a clock buffer 1405 that receives the SYS CLK system clock signal from the clock ciralit 17 (F~g. 1) and generates a NODE
CLIC node cloclc signal in response. In one particular ombodiment, the clock buffer 1405 comprises a buffer as described in aforementioned Hillis, et al., Patent Appn. Ser. No. 07/489,079, filed March 5, 1990, entitled Digital Clock Buffer Circuit Providing Controllable Delay. It will be appreciated tha~ if all control network nodes 51 in a co~ltrol network node group 50 (Fig. 4B) are packaged together on, for example, a single integrated circuit chip, as is the rase in one embodiment, the chip may be provided with one clock buffer 1405 which can provide the NODE CLK sigDal to all control network nodes 51 in the control network node group 50.
Gencrally, the flick up control portion 1401 receives control Det vork rnessage packets 60 from its child nodes and generates a control network messa~e pacl~et 60 in respos~se. If ~he colltrol net~vork node 51 is not a root node, the flick up control portion 1401 transmits the generated control nehvosk message packet 60 to the parent control nehvork node, thereby transmitting the packet up the tree defining the control net vork 14. On the other hand, if the control network node 51 is a root node, the flick up control portion 1401 transmits the generated control network message packet o0 to the flick down control portion 1402 of the same control network node 51 for transmission down~ the tree comprising the partition of the control network 14 of which the coDtrol network node 51 is the root node. In addition, if ~e control network message packets 60 enable the node 51 to perforrn a scan operation, the flick up control portion 1401 generaaes scan data and pro~ides it to the flic~ down control portion 1402.
The flick down control portion 1402 receives control network message packets 60 and generates control ne~rork message packets 60 representative thereof for tra~smissio~ to the child control network nodes, thereby transmitting message packets down the tree defining control network 14. If the control network node 51 is not a root node, the nick down control portio~ 2 uses control network message packets 60 from the parent node. On the other hand, if the node 51 is a root node, it uses the control network message packets 60 from the flick up control portion 1401 of the node 51. In addition, if the node 51 is not a root node, if the control network message pac~ets 60 enable a SCaD
operation, the flick down control portion 1402 uses scan tata provided by the flick up control portion 1402.
The common portion 1403 provides commuDications between the flick up control portion 1401 and the flick down control portion 1402. The commoD portion 1403 transmits control network message " ..

2~933~

packets 60 and scan data from the flick up control portion 1401 to the flick down control portion 1402.
In additiorJ, if the flick up control portion 1401 determines that the node 51 is to oe a root node, the common portion 1403 notifies the flick down control portion 1402 and enables it to begin receiving control neh,vork message packets 60 provided by the flick up control portion 1401.
More specifically, the flick up control portion 1401 receiYes C(L) FLICK UP (4:0) signals representing successive flicks of control network message paclcets 60 (Fig. S) from a left child control network node 51 and C(R) FLICK UP (4:0) signals representmg successive flicks of control network message packets 60 from a right child control network node 51 and, in response thereto, generates P
FLICK UP (4:0) signals. If the control network node 51 is not a root node, it transmits the P FLICK
UP (4:0) sigllals to the parent coDtrol network node. On the other hand, if the control network node Sl is a root node, it buffers the P FLICK UP (4:0) signals in a packet buffer 1406 in the up/dov~
common portion 1403. In either sase, if the received message packets 60 are of the multiple-souru type ("multiple-source messages~) initiating scan operations, the control network node 51 also loads data generated in response to the message packe~s in a scan buffer 1410 in the common portion 1403.
In addition, the flick up control portion 1401 also controls establishrnent or elimination of the control network node 51 as a root node. If it receives a control network message packets 60 that it determined to be of the single source message type (~single-source message~) and configuration packet type, which establishes or eli;minates the control network node 51 as a root node, tbe flick up control portion 1401 conditiolls a root flag 1407 in the common portion 1403, and enables assertion or negation of a ROOT UP signal. The root flag 1407 controls selection by the flick dewn control portion 1402 of the source of control network message packets 60 received thereby.
The ROOT UP signal provided by the control network node 51 is received by the parent node.
If the ROOT UP signal is asserted, indicating that the node 51 is a root node, the parent node iDterrlally couples to its nick up control portion control network message packets 60 of the absta~n type.
In addition, the parent node is thereafter enabled to transmit control network message packets of the nil packet type to the Qick down control portion 1402. It will be appreciated that, if the parent's other child node is not a root node, the parent node will contiDue transmitting to that other child node control network message packets 60 of message types representative of the packets it receives from its parent, or representative of the packets its flick up control portion receives if it is also a root node. If the control network node 51 thereafter negates ~he ROOT UP signal, the parent node continues transmittiDg, to the child node comprising a root node, control network message packets of the nil packet type, until it is ready to begin transmission of a control network message packet 60 of another type. Thus, the parent node does not begin transmission of a control Detwork message packet 60 to a child node 51, whose status as a root node has been eliminated, in the middle of the packet; instead, the parent node waits until the be~inning of the next packet after the end of the packet it is then transrnittirJg.
The flick up control portion 1401 also generates, in response to receip~ of a control network message packet 60 in which the all-fall-down bit 81 (Fig. S) is set, an AFD all-fall down signal which may be coupled to the data router 15.

, : .. : . . . .
; - : . . : . , ... : . , .
: . ;. . :.: . : . ,; :
.: .; : ; : .,: ; : . .; ,,.:. . ,: ;. :: .
;,. :; :: . . .,~ :. :: . : , : - .:. : :.: :. - ., . : :: ,.,.; ,: : . . . .. ..

, . : :: . - ; .. ~

WO 92/06436 2 0 9 3 3 3 ~

The flick down control portion 1402 receives P FLICK DN (4:0) signals or, if the control network node 51 is a root node, signals buffered in the packet bu~er 1406, and data b~ffered in the scan buffer 1410. In response, the flick down control portion 1402 transmits C~L) FLICK DN (4:0) signals representing flicks of a control net vork message packet 60 (Fig. S) to a left child control nehvork node 51 and C(R) FLICK DN (4:0) signals representing flicks of a control network message packet o0 to a right child control net vork node 51.
The flick down control portion 1402 also receives a C(L) ROOT UP signal from the left cnild node and a C(R) ROOT UP signal from the right cbild node, whicb are used to control the types of message packets transmitted to the respective cbild nodes. If the C(x) ROOT UP signal (~x" referring to ~R~ or ~L~) is asserted, the respective cbild node is a root sode, and in that case the flick do vn control portion 1402 Iransmits message packets of the NPAC nil packet type to the child node.
Otberwise, if the control network node 51 itself is a root node, the flick down control portion 1402 receives control network message packets from the packet buffer 1406 and uses them tc geDerate message packets for transmission to the child nodes. If the messag,e packet received by the flick down control portion 1402 is of the single-source or idle types, it transmits it to the child nodes that are not asserting their C(x) ROOT UP signals. On the other hand, if the received message packet is a mu tiple-source message, the flick down control portion 1402 uses the message packet from the packet buffer 14~. in generating mmtiple source message packets for transmission to the child nodes that are not asserting their C~x) ROOT UP signals.
Both the fliclc up control portion 1401 and the fliclc down control portion 1~02 also excnange flOw coDtrol information over a link 1411, ~hat they provide in the scan flow bits 72(i) (Fig. S) in the control network message packets 60 that they transmit. In addition, each portion 1401 and 1402 provides the flow control inforrnation from the control network message packets 60 that they receive to the other portion, which uses the information in regulatiDg the transmission of packets 60. The respective portions remain disabled, until the other portion receives flicks in which the scan flow bit ;'2(i) is clear, after which they resume transmission.
In the control network nodes 51 comprising the control network 14 the flow cootrol is on a control network message packet basis, so that if a control network node 51 begins transm-ssion of a control network message packet 60 to another node, the receiving node receives Ihe entire packet.
Thus, if the receiving node transmits a control network message packet 60 to the transmitting node in which the scan flow bit 72(i) is set, to disable transmission by the transmittins node, the transmitting node will continue transmission of the packet it is currently transmitting and will thereafter become disabled.
In addition, the flow control only coDtrols transmission of message packets 60 of the multiple source typ~. If a disabled node is to transmit a message packet of a ~ype other than multiple sollrce, it transmits it in the satne maDner as if it were not disabled. On the other haDd, if the is the ne~ message packet to be transmitted by a disabled nude is of the multiple source type, it delays transmission and instead ~ransmits message packets of the idle type. When the receiving node at some point later ., : ,:

WV 92/06436 PCT/US91/~7383 ., 20~33~ -165~
indicates that it can resume reception, after it finishes transmission of the current idle message packet, the transmitting node transmits the delayed multiple sowce message packet 60.
The details of seYeral elements depicted on Fig. 12A will be described in connection with Figs.
12B through 12D-1. In particular, the flick up control portion 1401 will be described in conneaion with Figs. 12B through 12B-4D. The details of the root flag 1407 and control circuitry therefor wiD be described in cormection wilh Fig. 12C. Finally, ~he details of the fLick down control portion will be desc~ibed in connection with Figs. 12D through 12D-1.
2. flick up control portion 1401 i. General Figs. 12B depicts a geDeral block diagram of the flick up control pordon 1401, and Figs. 12B-1 through 12B-4D depict detailed block and logic diagrams of the flick up control portion 1401. Wilh reference to Fig. 12B, the flick up control portion 1401 includes a child (leh) receiver/buffer 1420(L) which receives the C(L) FLICK UP (4:0) signals from the left child contro! network node 51 and a child (right) receiver/buffer 1420(R) which receives the C(R) FLICK UP (4:0) signals from the right child control network node 51. The receiver/buffers, which are generally identified by reference numeral 1420(x) are generally similar, and will be described below in connection with Figs. 12B-1 through 12B-lG.
Generally, each receiver/buffer 1420(x) receives the C(x) FLICK UP signals from the respective left or right child control network node 51 representing successive flicks of control network message packets 60. In response, the bufferJreceivers 1420(x) provides SEL INP DATA (x) selected input data signals to a flick up data processs)r 1421 arld to an up output packet assembler 1422, and rNP
TAG (x~ input tag signals to a tag processor 1423. In addidon, the receiver/buffers 1420(x) provide (X~ lNP STA/CTRL lefL and right status/control signals to an up control circuit 1424 identifying the timing of receipt of the respective control network message packets 60, and the respective types of message packets being received.
In resposse to the (X) INP STA/CrRL status/cs)ntrol sigDals idendfying the types of message packets being received by the respective buffer/receivers 1420(x), the up control circl~it 1424 provides OUT SEL output select signals that identifies a message type to be generated by the up output packet assembler 142~. The up output packet assembler 1422 ~egins generati~g P FLICK UP signals ~or a control network message packet 60 of the t,vpe identified by the OI~T SEL s~gnals. A flick latch 1430 latches P FLICK UP signals at each tick of the NODE CLK signal, and transmits the latched signals to its parent node as FLICK OUT (P) flick out to parent signals.
In addition, as the up output packet asse~bler 1422 generates P FLICK UP signalsrepresenting the sequential flicks of the control network message packet 60, the OUT SEL sig~
enable it to use the SEL INP DATA (x) signals from the lef~ or right child buffer/receiver 1420(x) or PROC FLICK (UP) DATA processed Dick data signals form the flick (up) data processor 1421. The PROC FLICK (UP) DATA signals represent the sum, logical OR, logical XOR and maxunum of the SEL INP DATA (x) signals, as generated by an adder 1425, a-~ OR circl~t 1426, an XOR circui~ 1427 : . ; ,: .. . ., : :. :. :. ,.: :., . , , ;.
, : . ~ , , ., , :, , .~ .. ... , ~ . ............ . . . .

. . . . . . . . :

:

WO 92/~6436 2 o 9 3 ,,~US91/0738~s and a comparator 1428, respectively. The OUT SEL output select signals may enable the up output packet assembler 14æ to use the PROC FLICK (UP) DATA signals from one of these circuits 1425 through 1428 in a control net~,vork messa~e packet 60 it is currently genera~ing.
It will be appreciated that the PROC FLICK (UP) DATA signals from the 9ick (up) data processor 1421 are used in connection vith control net vork message packets 60 of the multiple-source message type and scan and reduce packet types. The flick (up) data processor 1421 provides the combination, as called for by the scan or reduce operation initiated by the received Dlessage packets 60, of the data in the data nibbles 70(i) (Fig. 5). Thus, tJe up control circuit 1424 enables the up output packet assembler 14æ to use the PROC FLICK (UP) DATA signals in a control network message packet 60 if the packets o0 received frcm the children are of the appropriate message and packet types.
The particular type of control network message packet 60 iderltified by the OUT SEL output select signals from the up control circuit 1424 depend on the types of packets 60 received by the receiver/buffers 1420(x) and by a selected priority arrangement. In one particular embodiment, if both receiver/buffers 1420(x) receive message packets 60 of the same message type, the OUT SEL signals will generally enable the up output packet assembler 1422 to generate P FLICK UP signals for a control network messagc packet 60 of that type. If, for example, the receiver/buffers 1420(x) both receive single-source messages, the up control circuit 1424 will enable the up output packet assembler 1422 to transmit a single-source message.
In that case, if the message packet is of the configuration packet t~pe, the up control circuit 1424 will also enable the up output packet assembler to use a signal, provided by the comparator 1428 identifying which of the SEL INP DATA (L) and SEL INP DATA (R) selected input data signals represer~ts the larger value, and uses the identitled selected input data signals in the OutgOillg control net vork message packet 60. Thus, the root height value pro~ided in data nibbles 70(0) and 70(1) is the maximum of the qalues in the control network message packets 60 received by the receiver/buffers 1420(x). Howcver, if the root he;ght values in the receiqed message packets differ, an error has occurred in the system 10. In that case the up control circuit 1424 geDerates an ERR error signal, which enables the flick down control portion 1422 to set the S ERR soft vare error bit 76 in the control network message packets 60 it is transmitting.
Similarly, if the control nehvork message packets 60 being received by the receiver/buffers 1420(x) are of the multiple-source message type, and of the same packet type, the up control circuit 1424 generates OUT SEL signals that enable the up output packet assembler 1422 to geDerate P
- FLICK UP signals representing a control network message packet 60 of the same message and packet type. The up control circuit 1424 eDables the up output packet assembler 14æ to include, in the packet data portion 62 (Fig. 5), PROC FLICK (UP) DATA processed flick up data signals fro~ the flick (up) data psocessor I421. The parlicular one of circuits 1425 through 1428 whose PROC FLIC~C (UP) DATA signals are used depends upon the particular type of scan or reduce operation is enabled by the rece;ved control Dehvork message packets 60. In addition, the up control circuit 1424 generates SCAN
BUF WE ssan buffer write enable signals that enable the scan buffer 1410 to load the SEL INP DATA
(L) signals from the receiver/buffcr 1420(1) representiDg the successive data nibbles 70(i). ^~

. , : , , ; .: ;, , " ,., WO 92/06436 PCr/US91/()7383 20~33~ -167-On the other hand, if one receiver/buffer 14~0(x) receives a single-source message, and the other receiver/buffer 1420(x') receives either a multiple-source message or a control network message packet o0 of the abstain message type (an ~abstain message~) or the idle message type (an ~idle message~), the up control circuit 1420 generates OUT SEI, signals that enable the up output packet assembler 14æ to transmit a single-source message. In that operation, the up outpot packet assembler uses the SEL INP DATA (x) signals from the receiver/buffer 1420(x) that received the single-source message. Thus, a single-source message takes priority over multiple-source, abstain or idle messages.
If the receiver/buffer 1420(X') receives a multiple-source message, the up control circuit 1424 alSO
enables it to buffer the packet. As will be described below, the buffered packet o0 will be used ;D
connection with a multiple-source message when received by the first receiver/buffer 1420(X), that is, the receiver/buffer that received the single-source message.
In addition, if a receiver/buffer 1420(X) ;S buffering a multiple-source rmessage, the up control circuit 1424 generates Fl OW CIRL (DN) aOw control signals that are coupled through the line 1411 .
(Fig. 12A) to the aick down control portion 1402. In response, the flick down control portion sets he scan aOw bits 72(i) (Fig. S) of the control net vork message packets 60 of the particular left or right child node connected to the receiver/buffer 1420(x), to disable the child node from transmitting addidonal message packets o0. When the buffered packe~ 60 is thereafter used, the up control circ~ut 1424 generates FLOW CIRL (DN) signals that thereafter enable the flick down control portion to clear the scan flow bits 72(i) of the packets 60 transmitted to that child node, enabling the child node to resume trausmittiDg message packets thereto.
Alternadvely, if oDe receiver/buffer 1420(x) receives or is buffering a multiple-source message and the other receiws a~ abstain message, the up contsol circuit 1424 geDerates OUT SEL outpu~
select signals that enable the up output pacxet assembler to generate P FLI~K UF~ signals from tne SEL INP DATA (x~ signals representiDg the received multiple source message. The abstain message indicates that the leaf 21 (Figs. 4A and 4B) comprising the source of tne message is abstaining from the operation initiated by the multiple-source message received or buffered by .~he other receiver/buffer 1420.
In addition, if a receiver/buffer 1420(x) receives a multiple-source message and the other receiver/buffer 1420(x') receives an idle message, the up control circuit 1424 enables the receiver/buffer 1420(x) to buffer the multiple-source message. When the other receiver/buffer 1420(x') thereafter reCeiYes an abstain or multiple-sonrce message, tne up control circuit 1424 enables the receiver/bu. ffer 1420(x) to provide the bufferet message as SEL INP DATA (x) signals to 'oe used by the flick up tata processor 1421 and up output paclcet assembler 1422 in generating a control network message packet 60 for transmissioD to the parent control net vork node 51. In that case, the 'ouffered multiple-source message is used in the same manner as if it werc being received contemporaneously with receipt by the receiver/buffer 1420(x') of the abstain or multiple-source message being received tbereby. Similarly, if botb receiver/buffers 1420(x) are buffering multiple-source messages, and if both receive idle messages, the up control circui~ enables them to provide the WO 92/06436 2 ~ ~ 3 3P~CTIUS91/07383 buffered rnessages as SEL INP DATA signals to be used by thc nick up data processor 1421 and up output pac~et assembler 1422 in generating a colltrol network message packet 60 for transmission to the parent control network Dode 51.
Firlally, if both receiver/buffers 1420(x) receive control network message packets 60 of the idle, abstain or NPAC (nil packet) message types, and if they are not bufferiug multiple-source message packets, the up control circuit 1424 ellables the up output packet assembler 14 7 7 to transmit a control network message packet 60 to the parent control network node 51 of the same message type.
In generating P FLICK UP signals representing the sequential flicks of a control network message packet 60, the up output packet assembler uses PROC TAG (UP) processed tag signals from the tag processor 1423. the tag processor 1423 receives the INP TAG (x) signals from the receiver/buffers 1420(x) and generates PROC TAG (UP) processed tag signals that the up output packet assembler 1422 uses in generating the tag bit in each flick. The PROC TAG (UP) signals represents the lo~cal AND and logical OR of the INP TAG (x) input tag signals. Dependirlg on the ~ ~ ~ ~- particular flick and tag bit being transmitted, tbe up control~circuit 1424 may enable the up output packet assembler 1422 to transmit the signal representing the logical AND or the signal representing the lo~pcal OR.
For some flicks, particularly those containih~g the scan tlow bits 72(i) and the overnow bit, the up control circuit 1424 enables the up output packet assembler 1422 to usg signals from other circuits in generating the tag bits. In particular, when the up output packet asscmbler li22 is generating P
FLICK UP signals representing flicks in which the ta!ç bits comprise the scan flow bits 72ti), the up control circuit 1424 enables the up output packet assembler 14Z to use FLOW CONTROL (UP) signals in generating the bits. The up output packet assembler 1422 receives the FLOW CONTRQL
(UP) signals from the flick down control circuit 14~2 through the flow control linl; 1411 in the common control portion 1403.
In addition, when the up output packet assembler 1422 is generating the fli~k containing ~he scan overflow bit 80 (Flg. 5), the up control circuit 1424, if it enabled !he up output packet assembler 1422 to transmit PROC FLICK (UP) DATA signals from ~he adder circuit 1425 iD the data nibbles 70(i), also ellables it to use a CRY IN carry in signal and an OVFL overflow signal from the flick up data processor 1421. The CRY lN and OVFL signals represent carry and overflow signals generated by the adder circuit 1425, and if asserted indicates a carrry and overflow, respecti~/ely, in the surn provided by the adder cucuit 142S.
The up control circuit 1424 provides several additional sig~als for controlling the operation of the Qick up control portion 1401. In particular, the up control circuit 1424 generates timing control signals, identified as UP RCV ST (12:0) signals, which are tirn~g signals similar to the XMlT (12:0) transmit flick and RCV (12:0~ receive fiick signals generated by the control network inter~ace 204 in the networlc interface 202 (Fig. 8) as described above. In addition, the up control circl~it 14~ generates an UP RCV RESET receive reset ti~ning signal. In particular, the up control circuit 1424 receives the SFL W DATA (x) selected input data signals from the receiver/buffer circuits 1420(x) and~ if they ,, ~ , : . ;

:, . .. .. . ..

WO 92/06436 PCr/l 1Sg1/07383 20~3~ -169-indicate that flicks being received represent NPAC nil packet message packets 60, the up control circuit 1424 asserts the UP RCV RESET timing signal.
On the other hand, if the SEL INP DATA (x) signals indicate that at least one receiver/buffer is receiving the first flick of a control nenvork message packet 60, up control circuit 1424 begins asserting successiw ones of UP RCV ST (~ 0) up receive state s~gnals that identify successive receive states. The UP RCV ST (12:0) signals represent thirteen signals, identified by the mnemonic UP RCV
ST 0 througb UP RCY ST 12, which the up control circuit 1424 successively asserts, al successive ticks of the NODE CLK timing signal. Generally, the successive UP RCV ST "i" sigDals (ni~ is an integer between zero and twelve~ are asserted in synchronism with the receipt by the receiver/buffers 1420(x) of corresponding oDes of the thirteen flicks of a control network message paclcet 60. The up RCV ST 0 through UP RCV ST 12 signals are used to other circuitry on the flick up control portion 1401.
In addition, the up control circuit 1424 depictet on Fig. 12B generates a ROOT UP LAT root up latch signal that enables conditioning of the root flag 1407 (Fig. 12A) which, in turn, controls the condition of the ROOT UP sigslal. If the SEL INP DATA (x) signals represent a single-source messages of the conf~guration type, and if the root height value corresponds to the level and sub-levei of the control network node 51, the up control circu~t 1424 asserts the ROOT UP IAT signal to set the root flag 1407. On the other hand, if the SEL INP DATA (x) signals represent a single-source messages of the configuration type, and if the root height value is greater thall the level and su~level of the control network node 51, the up control circuit negates the ROOT UP LAT signal to enable cleanng of the root flag 1407. When the root ~ag 1407 is set, the ROOT UP sigllal is asserted, which energizes a write enable terrninal of the packet buffer 1406, enabl~ng the buffer 1406 to buffer the P
Fl,ICK UP signals from the up output packet assembler 1~22. The paclcet buffer 14l06 provides BUF P
FLICK UP buffered parent flick up sig~lals to the flick down control portion 1402.
As described above, the flick latch 1430 latches the P FLICK UP signals at each tick of the NODE CLE~ si~al. Ln addition, while the ROOT UP signal is asserted, tbe packet buffer 1406 latches the P FLICK UP signals at each tick of the NODE CLEC signal. In one pa~ticular embodimer t, the right and left c~ild receiver/buffers 1420 also latch the respective C(x) F~ICK UP signals from their respective child riodes at each tick of the NODE CLK signal. Irl that emb~nent, the delay of tbe signals from the receiver/buffers 1420 to tbe flick latch 1430 and packet buffer 1406 is one tick of the NODE CLK signal. That is, unless a message packet 60 is being bu~fered by a receiver/buffer 1420(x), the flick latched by the receiver/buffers 1420 is processed by the flick (up) data processor 1421, tag processor 1423, and up output packet assembler 14æ during the tirne behveen successive ticks of the NODE CLK signal. I~ that embodhnent, the up output packet assembler 1422 maintaiDs ir~ormation as to the type message and packet it is transmitting for use by otber circuits shown on Fg. ~.
Flgs. 12B-1 through 12B-3 depict details of some of the circuits in the flick up control portion 1401. In particular, Fig. 12B-1 depicts a detailed block diagram of the child (left) receiver/buf~er 1420(L), and Figs. 12B-~A through 12B-lG depicts det~ils of circuits in the child (left) receiver buf~er 1420(L). The child (right) receiver/buffer 1420(R) is substantially similar to the child (left) receiver ,, , -: . . :. .: , , : . . ::, ,. :, .

WO 92/06436 PCl/US91/07383 2~33~
-17~
buffer 1420(L) and vill not be described in detail. Fig. 12B-2 depicts dctails of the flick (up) data processor 1421. Finally, Fig. 12B-3 depicts a detailed bloclc diagram of the output packet assembler.
ii. Child (Left~ Receiver/Buffer 1420(L) Fig. 12B-1 depicts a detailed block diagram of the child (left) receiver/buffer 1420(L), and Figs. 12B-LA through 12B-lG depicts details of circuits in the child (left) receiver bu~fer 1420(L). With reference to Fig. 12B-1, the child (left) receiver/buffer 1420(L) includes a source data selector 1440 that, under control of SRC DATA SEL sowce tata select signals from an mput source identifie~ circ ut 1441, selects from among several signal sources to provide the SEL INP DATA (L) selected input data (left) signals.
The source data selector 1440 c~n select among number of sigDals input thereto. In particular, the source data selector 1440 may select C(L) FLICK UP (3 0) LAT latched child (left) Qick up si~als, as latched by an input latch 1443. The input latcb latches the C(L) FLICK UP (4:0) signals received from the child control network node 51 at each tick of the NODE CLK signal. In addition, the C(L) FIICK UP (4:0) sigDais are coupled to a check circuit 1444 that, in response to the NODE CLK and RCV ST (12:0) receive state signals, perfonns a check operation iD comlection with the checl~sum in field 63 (Fig. 5) of the control network message packet 60 being received to verify proper receipt of the packet 60.
The source data selector 14~0 may also select PMCL parked multi-source !e~t child signals from a left park buffer 1442. As noted above, the receiver/buffer 1420(L) may buffer data from a multiple source message if the other receiver/buffer is Dot then receiving a Multiple-source message whose data is to be used in the required mathematical operation. The left park buffer 1442 provides this facility in the receiver/buffer 1420(L).
~ As also l~oted above, in cormection with a scan backward operation enabled by a multiple-source message, the left and right inputs to each control net vork Dode 51 are effe~tively reversed. To accommodate that, the source data selector 1440 of the left receiver/buffer 1420(L) can also receive C(R) FLICK llP (3:0) LAT latched flick up signals from the ri~t child node, and also PMCR parked multi-source right child signal from a right park buffer (not shown). Fmally, if both child nodes are root nodes, the source data selector 1440 may couple ABSN abstain signals whicb have the encoding corresponding to that in the message qpe field M of a control network message packet 60.
The receiver/buffer 1420(L) also includes a tag input seleaor 1445 that, under control of the input source identifier, selects among a number of signal sources as the SEL INP TAG (L) selected input tag signal. The tag input selector 1445 also may couple the C(L) Fl,ICK UP (4) LAT latched left child flick up signal from the inpu~ latch 1443, or the corresponding C(R) Fl ICK UP (R) LAT signal received by the receiver buffer 1420(R) from the right child. It will be appreciated that, if the input source i~len~ er conditions the SRC DATA SEL source data select signals to enable the source data selector 1440 to couple the C(x) FLICK UP t3:0) IAT signals (nx~ is ~L~ or ~R~) as the SEL INP DATA
(L) signa~ it will also condition the SRC TAG SEL si~als to enable the tag in selector 1445 to couple the C(x) FLICK UP (4) LAl' signals as the SEL INP TAG (L) signai.

: ~ . . . .: : ; , .: .
, . .: :: ,................... : . : . ....
,:, .,: : :: ~

.. . . ..

, " :, WO 92/06436 ~'Cr/US9l/07383 20~3~5~ -171-ln addition, the tag input selector can select a POVL parked overslow left sigssal ~rom a park overtlow buffer 1446 in the left child receiver/buffer 1420(L), or a POVR parked overflow right signal from a correspondi~sg bslffer in the right child receiver/buffer 1420(R). The park overslow buffer 144o buffers the scan overflow bit 80 of a received control network message packet oO, whose data is parked in the park blsffer 1442. Thus, if the input so~srce identifier 1441 ellables the sossrce data seleaor 1440 to couple PMCx (~x~ is ~L~ or ~RR) parked multi-source signals from a park buffer 1442, or the corresporsding buffer in the right child receiver/buffer 1420(R), it will also enable the tag hs selector 1445 to Isse tlse POVx parked overflow signal as the SEL INP TAG (L) selected input tag signal for the scan overflow bit. It will be appreciated that, if the input source identifser conditions the SRC DATA
SEL source data select signals to enable the source dsata selector 1440 to couple the PMCx signals (~
is ~L~ or ~R~) as the SEL INP DATA (L) signals, it will also condition the SRC TAG SEL signals to enable the tag in selector 1445 to couple the POVx signals as the SEL IN~' TAG (L) signal for the scan overflow bit.
The input source identifier 1441 uses a nurnber of signals in generating the SRC DATA SEL
source data select signals. In particular, an irput packet type decoder 1447 receives the C(L) E~LICK
UP (3:0) LAT from the input latch 1443, the C(R) FLICIC UP (3:0~ L~T signals from the ri6ht child receiver/buf~er 1420(R) arld the UP RCV ST 0 signal. It will be appreciated that the C(x) FLICK UP
(3:0) LAT signals, when the UP RCV ST 0 signal is asserted, identifies the particular message type of the incoming control network message packet 60. The input packet type decoder 1447 generates a series of signals, including signals Cx/lDLE idle, Cx/SS single-source, Cx/MS multiple-source, Cx/ABS abstain, and Cx/NPAC nil packet [~x~ identifies ~L" (left) znd ~R~ (right)]. These signals identify the particular type of control network message packet 60 being received from each of the left and right child nodes. The input source identifier 14qO uses these signals, along with ~he UP RCV ST
(12:0) up receive state signals, to determine whether the source data selector 1440 should couple signals from an input latc~, such as latch 1443, or from a park buffer 1442, or the ABSN signals, as the SEL rNP DATA (L) selected input data signals.
The iDpUt source identifier 1441 uxes an OU r P~Cr MS output packe~ multiple-source signal and an OUT P~ SCF/RED output packet scan for vardjreduce signal, both of which are geDerated by the up output pzcket assember 1422, to determine whether the source data selector 1440 is to couple signals from the right or left child, or right or left parlc buffer, zs the SEL INP DATA (L) signals. As described above, the up output packet zssembler 14æ generatcs signals providing information as to the packet 60 being transmitted thereby, of which the Ol lT P~ MS and ou'r P~ SCF/RED signals are two. The OU T PE~r MS and OU r PE~r SCF/RED reduce signals, when asserted, indicate that the P
FLICK IJP signals generated by the up output packet assember 1422 comprise a multiple source coDtrol network message packet 60, and that the packet type enables a scan forward or reduce operation. In that case, the input source identifier 1441 enables the source data selector 1~40 to couple either the C(L) FLICK UP (3:0) LAT signals or the PMCL signals as the SEL INP DATA (L) signals.
CoDtemporaDeously, the input source ideDtifier 14~1 gerlerates SRC TAG SEL source tag select signals WO 92/06436 PCr/US9l/07383 2~1933~3 that enable the tat mput selector 1445 to couple either the C(L) FLICK UP (4) LAT signal or the POVL signal as the SEL INP TAG (L).
On the other hand, if the OUT PKT MS signal is asserted, but the OUT PKT SCF/REDsignal is negated, the control net vork message packet 60 is of the multiple-source message type, and indicating a scan backward operation. In that case, the input source identifier generates SRC DATA
SEL signals that enable the source data selector 1440 to couple either the CR FLICK UP (3:0) LAT
signals latched by the right child receiver/buffer 1420(R), or the PMCR signals provided by the park buffer of the right child receiver/buffer 1420(R) as the SEL INP DATA (L) left selected input data.
Thus, if the control net vork message packet 60 is of the multiple-source message type, and enabling a scan bachvard operation, the right and left child receiver/buffers 1420 interchange the C(x) FLIC:~;
UP signals received thereby, as described above.
Furthermore, if the C(L) ROOT UP signal from the child node connected thereto is asserted, indicating that the child Dode is a root node, the input source identifier 1441 generates SRC DATA
SEL signals to enable the soùrce data selector to couple ABSN abstaiD signals as the SEL INP DATA
(L) signals. Thus, if the child node is root node, the child receiver/buffer 1420(1) irlternally pro lides abstain packets, regardless of the types of control net~vork message packets 60 directed thereto by the child node.
In any case, the input source identifier 1441, while ~he RCV ST 11 and RCV ST 12 signals are asserted, indicating that the global information field 71 and checkswn field 63 of the control nenvork message packet 60 are being received, generates S~C DATA SEL source data select signals to enable the source data selector 1440 to couple the C(L) FLICK UP (3:0) I~T signals as the SEL INP DATA
(L) signals. With respect to the global h~formation field 71, regardless of the message types of the rnessages including,the field 71, the signals representative of the field are ORed together with corresponding signals from the right child receiver/buffer 1420(r). The resnltiDg signals are transmitted ;D control net vork message packets 60 up the tree comprising the control net vork 14 to the root node, which, in turn, broadcasts them ;D control network message packets 60 transmitted dow~ r the tree to the leaves 21 in the pardtion. Thus, the global bits may be used to provide infonnation concerniDg, for example, syncllror~ization of the operations of the processing elements 11, regardless of the types of corltrol net vork message packets 60 being transmitted up and down tbe tree definillg the control net~vork 14.
The left c~ild receiver/buffer 1420(L) also includes control aDd status circuits 1450 and 1451 for controlIing the park buffer 1442 alld park overflow buffer 1446, respectively. The park buffer control and status circuit also receives a number of signals and generates, in response, PK BUF (L) ClRL left park buffer control signals, a PK I~UF SRC SEL (L) left park buffer source select signal, and P~; BUF tLj ST left park buffer status signals. The PK PUB SRC SEL (L) sigl~al enables a multiplexer 1452 to couple either the SEL INP DATA (L~ selected input data signals representing a control network message packet 60 from the source data selector 1440 to data input terminals of the park buffer 1442. The PK BUF (L) CIRL signals enable the signals from the multiplexer 1452 to be WO 92/~6436 PCl /US9l /07383 20 933 .~ -173-buffered in the park buffer 1442, to enable it to buffer a control network message packet 60 In particular, the contents of park buffer 1442 comprise the low-order four bits of each flick of the control network message packct 60. The PK sUF (L) ST left park buffer status signals indicate whether the park buffer 1442 is bufferirlg a control net vork message packet 60, and, if so, whether the seg nent bit 77 is set or cleared The only tag information buffered by the receiver/buffer 1420tL) indicates the condition of the segment bit 77 and the scan overflow bit ~0 It will be appreciatet that these tag bits 77 and 80 are associated with, or provide i~nformation as to the processing cf the data in the data nibbles 70(i) of the control network message packet S0 On the other hand, the other tag bits of a control netvork message packet 60 provide control irlformadon to control the flow of control network rnessage packets 60 through the control network 14, or to control the all-fall-down operations of the tata router 15 In any event, if the input packet type decoder 1447 is asserting the CL/MS signal and either the CR/SS or CR/IDLE signal, the control net vork message packet 60 represented by the SEL INP
DATA (L) signals is a candidate to be buffered in the park buffer 1442 In that case, the packet 60 received by the left child receiver/buffer mO(L) is a multiple-source message, aDd the rigbt child receiverlbuffer 1420(R,~ is not receiving a multiple-source message to be used in connection there vith If the park buffer in the right child receiver/buffer 1420(R) is not buffering a packet 60 which can be used with the multiple-source message being received by the left child seceiver/buffer 1420(L), the parlc buffer control/status circuit 1450 will condition the PK BUF SRC SEL (L) park buffer source select signals to enable multiplexer 1452 to couple tbe SEL INP DATA (L) signals representing the packet 60 to the park buffer 1442, and the PK BUF Cl RL (L) park buffer control signals to enable the park buffer to buffer the packet 60.
On the other hand, if the right child receiver/buffer 1420(R) is receiving a multiple-source message packet while (a) the park buffer 1442 of the left child receiver/buffer 1420(L) is buffering a multiple-source message packet, and (b), the left child receiver/buffer 1420(L) is not receiving a control net vork message pac~et 60 of a type, such as a su gle-source message packet having higher prioriq of transfer through the control network 14, the park buffer control/status 1450 enables the input source identifier 1441 to, in turn, enable the park buffer control/status 1450 to transmit the PMCL signals representative of the successive flicks of the multiple-source control net vork message packet 60 The input source identifier 1~41 of the left child receiver 1420(L), or tbe corresponding circuit of the right child receiver 1420(R), enables the respeaive source data selector 1440 to couple the PMCL signals as the SEL INP DATA (L) signals, if the operation enabled is a scan forward or reduce cperadon on the one hand, or a scan back vard operation on the other harld As noted above, the PK BUF (L) ST left park buffer status signals from the park buffer control/status circuit 1450 provide information as to whether the segment bit T7 was set in ~e message packet 60 buffered in park buffer 1442 The left park buffer control/status circuit 1450 receives the SEL INP TAG (L) selected left input tag signal from the tag input seleetor 1445 for this The park overnow buffer status/control circuit 1451 uses the PK BUF (L) ST left park buffer status signals, the UP RCV ST 10 receive timing signal, and the OUT PKT MS output packet multiple-.. . .. .. . . ..... . . .................. .. .. . . , : ................. ' ''." ' ',: ~, ' ' '. ' '~: : .

- , . ,, - -. , ; ~ . ,., :
,: : :

WO 92/1)6436 PCl/US9l/07383 20~33~

source signal, and generates a PK OVFL SRC SEL (L) park overflow source select and PK OVFL
BUF CTRL park overflow buffer contJol signals to control the park overflow buffer 1446 and an input multiplexer 1453. If the PK BUF (L) ST left park buffer status siE~als iudicate that the park buffer control/status circuit 1450 is enabling buffering of a control networlt message packet 60, the park overflow buffer status/control circuit 1451 enables the multiplexer 14S3 to couple the SEL INP TAG
(L) signal representing the scan overflow bit 80 to be buffered in the park overflow buffer 1446.
With this background, the details of the left child receiver/buffer 1420(L) will be briefly descsibed in connection with ~igs. 12B-lA through 12B-lG. Figs. 12B-lA and 12B-lB depict details of the source data selector 1240 and tag input selector 1445, respectfully, along with respective portions of the input source identifier 1441. Figs. 12B-1C through 12B-lE depict details of the park buffer 1442, multiplexer 1452 and the park buffer control/status circuit 1450. Fig. 12B-lF depicts details of the park overflow buffer 1446, multiplexer 1453 a~d the park overflow buffer status/control circuit 1451.
FmaUy, Fig. 12B-lG depicts details of a segment bit latc~, in the park buffer control/status circuit 1450, which provides a SEG L left segment signal indicating the condition of the received or buffered segment bit of the control network message packet oO defined by the successive SEL rNP DATA (L) selected input data signals. The SEG L signal is buffered il~ the scan buffer 1410.
Since the operation of the circuits depicted on Figs. 12B-LA through 12B-lG will, in view of the description in connection with Flgs. 12A and 12B-1, be apparent to those skilled in the ~rt, they will not be described in detail. In any event, with reference to Fig. 12B-LA, the source data selector 1440 includes to multiplexers 1460 and 1461. The multiplexer 1460 select among the C(x) FLICK UP (3:03 LAT latched left and right child flick up signals, the PMCR sig~lals from the park buffes of the right child receiver/buffer 1420(R), arld SEL PMCL/ABSN signals from the multiplexer 1461. The multiplexer 1461 selects among the'PMCL signals from the park buffer 1442 of the left child receiver/buffer 1420(L) and ABSN abstain signals encoded to conform to the encoding which, in message type field 64 of a control network message packet 60, identifies the abstain message type.
The multiplexer 1461 is controlled by an abstainlpark buffer select control circuit 1462, and the multiplexer 1460 is controlled by a control circuit 1463. The abstain/park buffer select control circuit 1462 enables the multiplexer 1461 to couple the ABSN signals to the multiplexer 1460 if an OR
gaîe 1464 is energized, which occurs when either the UP RCV ST RST or the UP RCV ST 12 receive timing signal is asserted, or when an AND gate 1465 ~s energized. AND gate 1465 is energ~zed in responce to the coinciderlce of the UP RCV ST O signal is asserted and both the C~/NPAC sigGals are asserted. The last condidon occurs if both the child nodes are root nodes.
The control circuit 1463 includes several sections, inchlding a multiplexer control arcuit 1466, a left/right select enable circuit 1467, and a message packet priority circuit 1470. The multiplexer control circuit 1466, in turn, includes three paJt5. An output enable part 1471 generates a RCV DATA
OUT EN (L) left received data output enable signal, which enables or disables output by the muldplexer 1460. A lefttright select circuit 1472 enables the ~nultipJexer 1460 to generally selectively couple either signals received from the left child or the right child as the SEL INP DATA (L) selected left irlput data signals.

, ~ . :: . : . : .
,, .- ~

, . ~ ~, :
- . , ,: :

WO 92/06436 PCr/llS91tO7383 More specifically, the left/right select circuit 1472 provides a high-order SEL LEFT LAT
select left latched signal, tha~, when asserted, erlables the multiplexer 1460 to couple either the C(L) FLICK UP (3:0) LAT or SEL PMCL/ABSN signals as the SEL INF~ DATA (L) sigDals, aDd when negated enables it to couple the C(R) FLICK UP (3:0) LAT or PMCR s,gnals as she SEL INP DATA
(L) signals. Finally, a latch/buffer select urcuit 1473 provides a low-order SEL PARK BUF LAT
(LFT) select left park buffer latched sigr~al that e~ables the multiplexer 1460 to select between the PMCR and the C(R) Fl ICK UP (3:0) LAT s4~nals if the left/right select circuit 14n is negating the SEL LEFT LAT signal, and between the C(L) FLICK UP (3:0) LAT and SEL PMCL/ABSN signals the left right circuit 1472 is asserting the SEL LEFT LAT signal.
The left/righl select enable circuit 1467 provides two signals for contro'lling the Yarious circuits 1471 through 1473 of the mnltiplexer control section 1466 in nnison, namely, a low-order select enable portion 1474 and a nign-order select eDable portion 1475. The low-order select enable portion 1474 provides an asserted RCV ST O (NOT NF~AC)/10/11/MS SCB signal generally d~lring three timing periods as identified by t'ne asserted UP RCV ST (12:0) signals. In particular, the RCv ST O (NOT
NPAC)/10/11/MS SCB signal is asserted if (a) the RCV ST O t~ning sigrlal is asserted, and if the CL/NPAC and CR/NPAC signals are not both asserted, (b) the RCV ST 10 ~iYning signal is asserted, and (c) the RCV ST 11 timing signal is asserted.
In additioD, the signal is asserted if the control network message packet 60 being transmitted by the up output pacl~et assembler 1422 is a multiple-source message enablillg a scan backward operadon, as indicated by an assertcd OU T MS SCB sigl~al from the rnessage packet priority circuit 1470. It will be appreciated that, if the RCV ST O dming signal is asserted, the first flick of the control network message packet 60 is being received, which contains the mcssage type field 64. If the CL/NPAC or CR/NPAC sgnals are theD asserted, the coDtrol network message packet 60 received from the respective left or right child is of the NPAC nil packet type.
Similarly, if the RCV ST 10 or RCV ST 11 signal is asserted, flicks ten and eleven of the control network message packet 60 are curreDtly being reccived. Howeverj the multiplexer control section 1466 provides a delay of orle tick of the NODE CL~C signal ~ controlling the multiplexer 1460, and so the control of multiplexer 1460 enabled by the RCV ST O(NOT NPAC)/10/11/MS SCB signal ~11 be effective with flicks eleven and twelve, that is, the last two flicks of the control network 7nessage packet 60. It will be appreciated that these flicks contain the global information field 71 and the checksum field 63.
The bigh-order select enable portion proYides a RCV ST 1-9 sigDal which is asserted when the UP RCV ST 1 through UP RCV ST 9 signals are asserted, and is otherYvise negated. As noted above, the multiplexer control section 1466 provides a delay on one ticl~ of the NODE CLK signal in controlling the multiplexer 1460, and so the control of the ~ultiplexer 1460 eDabled by the RCV ST 1-9 signal ~ill be effective with flicks two through ~en, which contain the combine function fi~ld 66 and the data nibbles 70(i) (F~g. 5).
It will be appreciated that the RCV ST O(NOT NPAC)/10/11/MS SCB and RCV sr 1-9 signals are not asserted contemporaneously, bu~ instead contemporaneous with receipt of different . : , . , , :, ;, . .,........... , , ..... : ,,: , . ~ . :. , , , : . : :,:, .: ., :,: , ... . .

W O 92/06436 PC~r/US91/07383 -17~ 20933aa portions of a control network message packet 60. In one embodiment, the RCV ST 0(NOT
NPAC)/10t11/MS SCB signal will be asserted during receipt of the message type field 64, unless the message type is NPAC nil packet, and the Each of the enable and selea circ~its 1471 through 1473 includes a multiplexer 1476 throllgh 1478 and a flip-flop 1480 through 1482 clocked by the NODE CLK sigl~al. The output s~gnals provided by the flip-flops 1480 and 1~82 comprise the enable and select signals for the multiplexer 1460, and so the flip-flops 1480 through 1482 effectively provide the one-tick delay noted above. The multiplexers 1477 through 1478 are controlled in unison by the RCV ST 0(NOT NPAC)/10/11/MS SCB and RCV
ST 1-9 signals. Thus, while the RCV ST 0(NOT NPAC)/10/11/MS SCB signal is asserted, the multiplexer 1476 in erlable circuit 1472 couples a CL ACllVE left child active signal as a SEL RCV
DATA OUT EN (L) select receive data out enable (left) signal to the data input terminal of flip-flop 1480, which is latched at the next tick of the NODE CLK signal. The CL ACTIVE signal is coDtrolled by a control segister (not sho vn) on control net vork node 51, which is set by the diagnostic Detwork 16 . .
if the node is cormected to a left child. In that case, the multiplexer 1460 is enabled to provide SEL
INP DATA (L) selected left input data signals. r Contemporaneously, the left/right select circuit 1472 couples an asserted sig~al (as identified by "+~ on the Fig.) as a SEL LEFT select left signal to the data input terrninal of flip-hop 1481. The flipflopl481 is set at the next tick of the NODE CLK signal, to provide an asserted SEL LEFT LAT
seled left latch signal. The asserted SEL LEFT LAT sigoal enables the multiplexer 1460 to couple either the C(L) FLICK UP (3:0) LAT signals or the SEL PMCL/ABSN signals as the SEL INP DATA
(L) signals.
In one embodime~t, isl the right child receiver/buffer 1420(R), the multiplexer cosresponding to multiplexcr 1477 receives a negated signal instead of the asserted signal. Is that case, the corresponding SEL RT selected right output signal is negated, resulting in a negated SEL RT LAT
selected right latched sigDal.
Finally, the latch/buffer select circuit 1473 couples a P~r PRIORrrY (L) packet priority signal as a SEL PARK BUF (LFT) select park buffer left signal to the data input terminal of ai~flop 1482. The PKT PRIORll~ (L) sigDal is controlled by the message packet priority circl~it 1470, and is geslerally asserted if the left chad receiver/buffer 1420(L) is receitqng an idle message packet, a~d the right child receives/buffer 1420(R) is receiti~g either a idle, abstain, NPAC nil packet or a multiple-source message packet. If the right child receiver/buffer 1420(R)is receiving such a control network message packet 60, the PICI PRlORlTY (L) signal is asserted, to enable the latch/buffer select circuit 1473 to, ill turn, eDable the multiplexer 1460 to couple signals from the park buffer 1442 as the SEL
INP DATA (L) signals. Otherwise, the multiplexer 1460 is coDditioDed to couple the C(x) FLICK UP
(3:0) LAT sigl~als as the SEL INP DATA (L) sigllial. As noted above, while the RCV ST 0(NOT
NPAC)/10/11/MS SCB is aisserted, the multiplexer 1478 couples the PKT PRIORI~Y (L) as the SEL
PARK BUF (LFI') signal, which is latched by flip-nop 1482 at the ~ext tick of the NODE CLK si~nal.
The fJip-flop 1482 provides the SEL PARK lBUF LAT (LFT) sigDal.

:. : , . ......................................... . .
, .
, ~

WO 92/06436 PCr/US9l/07383 2~33~ -177-While the RCV ST 1-9 signal is asserted, if the packet priority circuit 1470 is negating the OUT MS SCB output multiple-source sean backward signal, the RCV ST 1-9 signal e~ables the multiplexers 1476 through 1478 to couple the output signals of the respective flip-flops 1480 through 1482 to their respective data input terri~inals. Thus, the ~ flops maintain their state at the successive ticks of the NODE CLK signal, and the multiplexer 1460 continues to couple the selected SEL
PMCL/ABSN or C(L) FLICK UP (3:0) LAT signals as the SEL INP DATA (L) signals.
However, if the OUT MS SCB signal is asserted, which may occur while the UP RCV ST 2 timing sigDal is asserted to i~dicate that a scan backward operation is taking place, the RCV ST 0(NOT
NPAC)/10/11/MS SCB is also asserted. In that case, the multiplexers 1476 through 1478 couple the correspondirJg signals the flip-flops in correspondiDg enable and select circuits in the right child receiver/buffer 1420(R) to the ni~nops 14~0 through 1482, where they are latched at the next tick of the NODE CLK signal.
In that case, if the RCV DATA OUT EN (R) receive data out e~able (right) signal is asserted, the right child is active, and so the fli~flop 1480 wiD be set to assert the RCV DATA OUT EN (L) signal to enable the output of multiplexer 1460. In addition, the SEL RT LAT si~al from the right child receiver/buffer 1420(R) will be negated, enabling the flip-flop 1481 to be cleared to negate the SEL LEFT LAT signal. When that occurs, the ~ultipl~xer 1460 is eDabled to couple the PMCR or C(R) FLICK UP (3:0) LAT s~als as the SEL INP DAl'A (L) sig~ls. The SEL PARK BUF LAT
(RT) select park buffer latched (right) signal from right child receiver/buffer 1420(R), latched by flip^
flop 1482, will control which of the PMCR or C(R) FUCK UP (3:0) LAT signals w ill be coupled by the multiplexer 1460.
When the UP RCV ST 2 sigllal is again negatet, the OUT MS SCB signal is also negated, which Degates the RCV ST 0(NOT NPAC)/10/11/MS SCB. Thus, for the rest of the tirne the RCY
ST 1-9 is asserted, the multiplexers 1476 through 1478 will couple the output signals from the flip-flops 1480 through 1482 to ~heir data input ~errninals, enabliDg, in turn, them to maintain the condition they were in while the IJP RCV ST 2 signal was asserted. Thus, if the Olrr MS SCB signal was negated during the assertion of the UP RCV ST 2 signal, the multiplexer 1460 ~AII couple either the SEL
PMCL/ABSN or the C(L) FLICK UP (3:0) I~T signals, originating frorn the left cbild Dode, as the SEL INP DATA (L) signals. ;
On the otber hand, if the OUT MS SCB signal was asserted at that point, indicating a scan backward operation, the multiplexer 1~0 will couple either the PMCR or C(R) UP (3:0) LAT signals, origiDating îrom the right child node, as the SEL INP DATA (L) signals. Thus, the enable and select circuits 1471 through 1473, and in particular tbe left/right select circuil 1472, enables the le~t child buffer/receiver 1420(L) to couple signals from the right child node as ~he left inpu~ sigllals if a sean backward operation is occurring. It ~11 be appreciated that, while the re~rersal is initiated by the OUT
MS SCB signal iD synchronism with the UP RCV ST 2 signaL because of the one-tick delay prGvided by the eDable and select circl~ts 1471 througb 1473, tbe SEL INP DATA (L) signals reflect the reversal in flick three, which is the begiDDing of the data nibbles 70(i). In addition, the reversal will end iD

,, ;' '~ '' : ' . ; , ,' . ! ' : ' ' ,: :' ' ' ' ~ ,,'':., ' ''.:`. ~ .:.

WO 92/06436 P~/I~S9l/07383 2~933~

response to the negatioo of the RCV ST 1-9 signal, arid, because of the one-tick delay, will be after flick 10, which contains the last of the data nibbles 70(i).
The tag input selector 1445 also includes a multiplexer 1490 and enable and select circuits 1491 through 1493. The multiplexer 1490 selects P OV ~x~ parked overflow signals (~x~ reference lcft or right) and C(x) TAG signals, which corresponds to the C(x) E~LICK UP (4) LAT signals, as the SEL
INP TAG (L) selected input tag signals. The multiplexer 1490 is controlled by enable and select signals in a manner sirnilar that of multiplexer 1400. In addition, the eDable and select circuits 1491 througb 1493 are controlled by the RCV ST O(NOT NPAC)/10/11/MS SCB and XCV ST 1-9 signals iD a ;~
manner similar to that of the enable and selea clrcuits 1471 througb 1473.
Figs. 12B-lC through 12B-lE depict details of the park buffer 1442, mnltiplexcr 1452 and the park buffer control/status circuit 1450. Generally, Figs. 12B-1C and 12B-lD include circuitry for controlling storage of data from a control network message packet 60 in the park buffer 1442. Fig. 12B-lE includes circuitry for indicating the status of the park bufs^er, that is, whether it is storing data from a control network message packet 60 and, if so, whether the packet~s segment bit 7'7 (Fig. 5) was set.
With reference to Figs. 12B-1C and 12B-lD, the multiplexer 1252 is controlled by a multiplexer coDttol circuit 1500, which generates the PK BUF SRC SEL (L) packet buffer source select (left) signal. If the PK 8UF SRC SEL tL) signal is asserted, multiplexer 1452 couples the S~L INP
DATA (L) signals to data input tenninals of the patk buffer 1442 for storage. The park buffer 1442 buffers the signals at its data input terrninals at successive ticks of the NODE CLK signal. On the other hand, ~ the PK BUF SRC SEL (L) signal is negated, the multiplexer 1452 couples the PMCL
signals at the park buffer's output terrninals to its input serrninals.
The multiplexer control circuit 1500 asserts the PK BUF SRC SEL (L) signal under several circumstances. In particular, if the up output packet assembler 1422 is asser~g the OUT PK MS
signal, indicating that it is transmitting a multiple-source message pzcket, several of gates ~501 through 1504 will be energized if the multiplexer control section 146o (fig. 12B-lA) is asserting both the SEL
LEFT LAT arld SEL PARK BllI: LAT (1~1) signals, or the corresponding arcuit in she right c~ild receiver/buffer 1420(R~ is asserting both the SEL RT LAT and SEL PAR~ BUF IAT (RT) signals, and if she up control circuit 1424 is asserting the UP RCV ST 1 through UP RCV ST 10 tirlling signals.
This occurs if lhe source data selecsor 1440 ir. either child receiver buffer 1420(x) is coupling signals from its respective parlt buffer as its SEL INE> DATA (x) selected input data signals.
In addition, the muitiplexer control circuit 1500 asserts the PK BUF SRC SEL (L) park buffer source select signal if the up output packet assembler 1422 is riegating the OUT PK~ MS output packet multiple source signal, indicating that it is transmitting a message packet of other than the multiple source message type. In that case, if a PARK PKT (L) park packet (left) signal is asserted, gates 1505 and ~06 enable the PK BUF SRC SEL ~L) to be asserted while the up control circuit 1424 is asserting the UP RCV ST 1 through UP RCV ST 10 timiDg signals. The PARK PlCr (L) signal is Benerated by circuitry depicted on Fig. 12B-lD.
With reference to Fig. 12B-lD, a park enable circuit 1510 ge~erates a PARK EN (L,) park enable (left) signaL which, when asserted, enables a control network message packet 60 being received -, - - : . , ,,., ~. -: ,- , :

W O 92/06436 PC~r/US91~07383 20333~ -179-by the left child receiver buffer 1420(L) to be buffered in park buffer 1442. The park enable circui~
1510 asserts the PARK EN (L) signal in response to conditions, each represented by an AND gate 1511 through 1515. If any one of the conditions is satisfied, wheD the UP RCV 0 receive state signal is asserted, the park enabie circuit asserts the PARK EN (L~ signal.
In particular, AND gate lS11 asserts a CLMS/SRSS signal if the leh child node is providing a multiple-source message, and the right child node is providing a single-source message packet. As noted above, the single-source message has priority, and so the left child buffer~receiver 1420(L) u~ill buffer the multiple-source message packet. If the left child node provides a multiple-source message, the CL/MS left child multiple-source signal is asserted when the UP RCY ST 0 signal i~ asserted, and when the right child node is providing a single-source message the CR/SS right child sirlgle-source signal is asserted, when the UP RCV ST 0 sig~al is asserted.
Similarly, AND gate 1512 asserts a CLMS/CR IDLE/l`~O (R) PARKED signal if both ~a) the left child node is providing a multiple-source message packet, and (b) the right child node is providing an idle message, as indicated by the asserted CR/IDLE signal and the park buffer in the right child receiver/buffer 1420(R) is not buffering a message packet, as indicated by a negated PAR~CED (R) signal. In that case, the right child receiver/buffer 1420(R) has ~;o message packet to be used in connection with the multiple-source message packet being received by the left child receiver/buffer, and so the PARK EN (L) signal is asserted to enable the multiple-source message packet to be buffered.
The remaining gates 1513 through 1515 of the pa~k enable circuit 1500 enable assertion of the PARK EN under several conditions while the parent Dode is disabling flow, as indicated by the negation of an UP FLOW OK sigDal. The UP FLOW C)K signal is derived from the FLOW CrRL
(tJP) flOw control sigDals received frorn the nick down control portion 1400. While the UP FLOW OK
signal is asserted, the flicl~ up co~trol portion 1400 ca~ transmit control network message packets o0 to ils parent, but when negated the flick up control portion is disabled from transmitting multiple-source message packets thereto. In particular, if the UP FLOW OK sigllal is negated, and if the right child receiver/buffer's park buffer is buffering a message packet 60, while the left child receiver/buffer 1420(L) is receiving a multiple source message, a CLMS/(R) PARKED/FLOW OFF sigQal is asserted, enabling asserdon of the PARK EN (L) signal. Similarly, if the left child rece*er/buffer 1420(L) is recei~ing a multiple-source message, while the right child node is pro~iding all absolute or a NPAC nil packet message, AND gate 1514 asserts the CLMS/(R) ABSNPAC/FLOW OFF signal, svhich enables assertion of the PARP~ EN (L) signal. Finally, if both child nodes are providing multiple source messages, AND gate 1515 asserts a (L)(R) MS/FLOW OFF signal, which 21s0 enables assertioD of the PARK EN (L) signal.
A park control circuit 1516, also shown on Fig. 12B-lD, generates the PARK P~CI (L) park packet signal which controls gates 1505 and 1506 (Fig. 12B-1C). The park control circuit 1516 also generates signals which indicate whether the seg~nent bit M (Fig. S) of the control network message packet 60 being buffered in the park buffer 1442 is set or cleared. If the se~nent bit is cleared, a - . . ........ , ,. ....... , ... . :, , ... , ., .: .

: ~ : . . , , .: . ,: .. . .. .

WO 92/06436 ~2 ~ 9 ~b~91/07383 multiplexer 1517, under CODtrol if the PARK EN(L)signalandtheDegatedSELIN~ TAG (L) signal, which at that point corresponds to the segment bit 77, enables a flip-flop 1518 to be set to assert a PARK/NO SEG park/segment clear sigrlal. The asserted PARK/NO SEG signal indicates that a control network message packet 60 whose segment bit is cleared is being buffered in park buffer 1442.
On the other hand, if the segrnent 'oit is set, a multiplexer 1520, urlder control of the asserted PARK
EN (L) signal and asserted SEL]NP TAG (L) signal, enables a flip-flop 1521 to be set to assert a PARK/SEG park/segment set signal. The PARK/SE~G and PARKlNO SET signals are both coupled to an OR gate 1522, which generates the PARK P~Cr (L) signal which is coupled to AND gates 1505 and L506(Fig.12B-lC).
Fig.12B-lE depicts park buffer status circuitry L530 which indicates th~ status of the park buffer 1442 (Fig. 12B-L~). Normally, the park buffer status circuitry 1530 will indicate, after a control network message packet 60 has been buffered in the park buffer 1442, that a packet has been buffered, and whether the packet's segment bit 77 was se~. However, under some circuIDstafices, if, for example, a set flush bit 75 is received in the either control network message packet 60 being buffered i~ a control network message packet 60 beiDg received by ~he right child receiver/buffer 1420(R), the eontrol network message packet 60 buffered is flushed, in which case park buffer status circuitry 1530 is conditioned to indicate that the packet buffer 1442 is empty. Further, if the source data selector 1440 couples the contents of the park buffer 1442 as the SEL INP DATA (L) signals, the park buffer status circuitry 1530 is conditioned to indicate that the packet buffer is empty.
More specifically, the park buffer status circuitry 1530 includes a parked/segment clear status section 1531 and a parked/segmeDt set status section 1532, and a clear control section 1533. The parked/segrnent clear status section 1531 generates a PARKED/NO SEG parked/seglDent clear signal that, when asserted, indicates that a coDtrol network message packet 60 whose segmeDt bit 77 was clear is parked park buffer 1442. The parked/se~ent set status section 1532, on the other hand, geDerates a PARKED/SEG parked/segment clear signal that, when asserted, indicates tbat a control network message packet 60 whose segment bit 77 was set is parked park buffer 1442. The PARKED/NO SEG aDd PARICED/SEG signals, when asserted, energize an OR gate 1533 which gene}ates a PARKED (L) parked signal which controls a gate corresponding to AND gate 1512 in the right child receiver/buffer 1420(R) to control the operations of the park enable circuitry therein.
The clear control section 1533 controls a CLR PK BUF STATUS clear park buffer status signal. Section 1533 includes a flush control portion 1537 that asserts a FLUSH/ST 10 signal in response to the coincidence of the FLUSH LAT sigDal and the UP RCV ST 10 signal. The SEL INP
TAG (L) selected input tag signal from tag input selector 1445 and SEL INP TAG (R) from the right child receiver/buffer 1420(R) are coupled to an OR gate 1540, which provides a COMP TAG
composite tag signal to one data input termiDal of a multiplexer 1541. If the UP RC~ ST 9 signal is then asserted, the COMP TAG signal represents the flush bit 75 of the control net vork message packet 60 being received, and so at this point the multiplexer 1541 couples COMP TAG sigrlal is eoupled as a FLUSH signal to a flip-flop 1542. The flip-flop 1542 latches the FLUSH signaJ at the next tick of the WO 92/06436 PCr/US91/0738~s 20~33~ -181-NODE CLK signal to provide the FLUSH LAT signal. If the FLUSH LAT signal is negated, neither control net vork message packet 60 being received by a child buffer/receiver 1420(x) has a set flush bit 75. On the other hand, if the FLUSH LAT signal is asserted, at least one such control net vork message packet 60 does ha~e a set flush bit 75.
Clear control section 1533 also has a circuit 1543 that generates a USE PK BUF signal if the input source identifier enables the source data selector 1440 (Fig. 12B-1) to use the control Detwork message packet o0 buffered in the park buffer 1442. This is indicated if both the SEL LEFT LAT and SEL PARK BUF LAT (LFT) signals are asserted by left/right select circuit 1472 and latch/buffer select circuit 1473 in synchronism with the RCV ST 0 signal. In that case, the USE PK BUF signal is asserted.
The parked/segment clear status section 1531 includes a multiplexer 1534 and ni~nOp 1535.
Multiplexer 1S34 is controlled by an AND gate 1530, which is energized if the UP RCV ST 10 and PARK/NO SEG signals are asserted, and if the FLUSH l AT signa! is negated. As ~oted above, the FLUSH LAT signal is controlled by the clear control section 1533, and is negated if no control network message packet 60 is recei~ed in which the flush bit 75 is asserted. If the clear coDtrol section is negatiDg a CLR PK BU~ STATUS clear park buffer status signal, multiplexer 1534 couples an asserted signal to the flip-flop 1535, which is set in response to the next tick of the NODE CLiC signal to assert the PAR~D/NO SEG signal. On the other hand, if the FLUSH LAT signal is assened, the AND
gate 1536 is de-energized, but the clear control section 1533 will be asserting the CLR PK BUF
STATUS signal. In that case, multiplexer 1534 will couple a negated signal to flip-nop 1535, to reset the flip-flop 1535 and Degate the PARKED/N(:) SEG signal.
After parked/segment clear status section 1531 has established the condition of the PARKED/NO SEG signal while a coutrol ne~work message packet 60 is loaded into the park buffer 1442, the multiplexer 1534 nonnally couples the PARECED/NO SEG signal to the data input termihal of the nip-flop 1535 to enable it to maintain its condition. HoweYer, the dear control section may enable a change in the PARKED/NO SEG signal under two circumstances, namely, if a control network message packet 60 is received in whidh the flush bit 75 is set, or if the buffered packet is used by the source data selector 1440. If, while the pacl;et is being buffered, a control network message packet 60 is received in which the flush bit 75 is set, the FLUSH/ST 10 signal will be asserted, enabling assertion of the CLR PK BUF STATUS clear park buffer status sigaal. Silnilarly, if the source data selector 1440 selects the buffered packet as the SEL INP DATA (L) signal, the USE PK BUF signal is asserted, also enabling assertion of the CLR P~C BUF STATUS signal. ID either case, the assested CLR PK BUF STATUS signal enables the multiplexer 1534 to coupled a negated sig~lal to the data input termir~al of the flip-flop 1534, enabling the flip-flop 1534 to be cleared to negate the PARI~ED/NO SEG signal.
The parked/segment set status section 1532 operates in a manDer similar to the parked/segment clear status section 1531 if the PARK/SE~G signal is asserted instead of the PARK/NO SEG signal.

, - , , ,;

: : . ~

WO 92/06436 P~/US91/07383 3 ~
-182- r Fig. 12B-lF depicts details of the park overflow buffer 1446, multiplexer 1453 and park overflow buf~er (left) status~control circuit 1451. With reference to Fig.12B-lF, ~he park overflow buffer 1446 cornprises a fli~Qop, identified by the same reference numeral. The status/control circuit 1451 includes a condition circuit 1544 and a clear enable circuit ~45. The condidon circuit 1544 enables the flip-flop 1446 to be conditioned in response to the scarl overflow bit 80 of the control network message packet 60. In particular, if a control network message packet 60 is beiltg stored in the park buffer 1442 an AND gate 1546 is energized in coincidenee with the assertion of the UP RCV ST
signal to assert the PARK LEFT OVFL BUF park left overflow buffer signal. The asserted PAE~K
LEFI OVFL BUF signal enables the multiplexer 1453 to couple the SELINP TAG (L~ selected input tag signal from tag input selector 1445 (Fig. 12B-1) to the data input terminal of flip-flop 144o. The signal conditions flip-flop 1446 at the next tick of the NODE CLK signal.
Thereafter, the PARK LEFT OVFL BUF sigualis negated. If a CRL LEFT OVFL BUF
clear left o~rerflow buffer signal from clear enable circuit 1545 is also negated, multiplexer 1453 couples the POVL park overflow left signai from flip-flop 1446 back to its input terminal, to enable nip-flop 1440 to maintain its condition at successive ticks of the NODE CLK signal. If the clear enable circuit 1545 thereafter asserts the CLR LEFT OVFL BUF sigDal, the multiplexer 1453 couples a negated signal to the data input tennmal of the flip-flop 1446, enabling the flip-flop to be cleared at the next tick of the NODE Cl~C signal, to negate the POVLsigDal.
The clear enable circuit 1545 asserts the CLR LEFT OVFL BUF signal under four urcumstances. Fi~st, the clear enable circuit 1545 asserts the CLR LEFT OVFL BUF signal if the up control circuit 1424 asserts the UP RCV ST RST sigllal, which occurs if both child rodes are root nodes. Second, the clear enable circuit 1545 asserts the signal if an AND gate 1547 is energized, which occurs if flush control portion 1537 (Flg. 12B-IE) asserts the FLUSH LAT signal indicating reception of a control network message packet ~0 in which the flush bit 75 is set. In addition, if AND gates 1550 or 1551 are energized, indicating respectively, that a pacl~et buffered in packet buffer in the left or right child receiver/buffer 1420(x), is coupled by the respective source data seleaor 1440 as the SEL INP
DATA (x) signals.
Finally, Fig. 12B-lG depicts a segment bit latch 1560 maintained by the park buffer control/status circuit 1450. the segment bit latcb 15oO provides a SEG L segment left signal that indicates the coDdition of the segment bit 77 in a control network message paclcet 60 received from the left child if it i9 a multiple-source message enabling a scan forward or reduce operation, or of a packet received from the right child it is a multiple-source message enabling a scan backward operation. The SEG L signal indicates the state of the segment bit of the control network message packet 60 regardless of whether the multiple-source message data is processed when received, or if it is buffered in ~ne pac~et buffer 1442.
The segment bil latch 1560 includes a nip-nOp 1561, which geDerates the SEG L signal, and a multiplexer 1562 which controls the source of signals coupled to the data input terminal of the flip-nop 1561. A seg nent latch source control circuit 1563 generates a SE~G BlT SRCE SE~l, segment bit source W O 92/06436 PC~r/US91/07383 20~3~.~5 -183-select signal to control the sourcc of the signal coupled by thc multiplexer 1562 to the flip-flop 1561, effecdvely selecting bet veen the C(L) FLICK UP (4) LAT signal, representing the received lag signa~
or the PAR~CED/SEG parked/segrnent signal from parked/segment set status section 1532 (Fig. 12B-lE). A segment latch hold select circuit 1564 generates a SEG BIT HOLD segment bit hold signal to enable the muldplexer 1562 to maintain its condition, or to intercharlge condidons with the segment bit latch in the right child receiver/buffer 1420(R). The segment latch hold select circuit 1564 asserts the SEG BIT HOLD signal contemporaneous with the assertion by the up control circuit 1424 of the UP
RCV ST 1 through UP RCV ST 9 signals.
If neither the SEG BIT SRCE SEL sigrlal nor the SEG BIT HOLD signal is asser~ed, the multiplexer 1563 couples the C(L) FLICK UP (4) LAT signal as a SEL SEG (L) selected segmeDt (left) signal to the data input ter~ninal of flip-flop ~61. The nip-nOp ~61 latches the SEL SE& (L) signal at each tick of the NODE CLK signal. The circuit comprising segment latch source coDtrol L563 is similar to the circuit of the message packet priority circuit 1470, and provides an asserted SEG BIT
SRCE SEL signal, coDtemporaneous with the assertion of the UP RCV ST 0 signal, if the controi network message packet 60 being received is to be buffered in ~he park buffer 1442. ID that case, the asserted SEG BIT SRCE SEL signal enables the multiplexer 1562 to couple the PARKED/SEG
parked/segrnent signal from parked/segment set status section 1532 (Fig. 12B-lE), which indicates whether the segment bit 77 of the buffered control network message packet ~ was set.
When the UP RCV ST 0 signal is negated, the SEG RIT SRCE SEL signal is negated, and the segment latch hold select circuit LS64 asserts the SEG BIT HOLD signal. l'his enables the multiplexer 1562 to couple the SEG L signal from flip-aop 1561 as the SEL SEG (L) signal to the data input terminal of flip-flop 1561, enabL;ng it to maintain its state while the SEG BlT HOLD signal is asserted.
If, while the SEG BIT ~IOLD sigr al is asserted, an AND gate 1565 in the seg~nent latch source control circuit 1563 is also eDergized, contemporaneous with the assertion of the UP RCV ST 2 signal, to assert the SEG BIT LAT SRCE SEL sigDal. This will occur if the control network node 51 is transmitting a multiple-source message eDabling a scan backward operation. In that case, the multiplexer ~i62 couples a SEG R seE~neut right signal from a segment bit latch 1560 in the right child receiver/buffer 1420(R) as the SEL SEG (L) selected segment (left) signal, which the nip-flop 1561 latches. In that case, the conditions of the SEG ~x~ (~x~ indicates ~L~ or ~R~) are effectively interchanged, thereby completing the interchange of data from control network message packets 60 from the left and right child nodes. When the UP RCV ST 2 signal is then negated, the SEG BIT
SRCE SEL signal is again negated, to enable only the SEG BIT HOLD signal to control the multiplexer 1562.
When the SEG BIT HOLD si~al is later negated, contemporaneous with ~he assertion of the UP RCV ST 10 signal, the SEG BIT HOLD signal is negated, eDabling the multiplexer 1562 to couple the C(L) FLICK UP (4) LAT signal to the flip-flop 1561 as the S~L SEG (L) signal.
iu. Flick (Up) Data Processor 1421 Fig. 12B-2 depicts details of the tlick (up) data processor 1421. With reference to Fig. 12B-2, the flick (up) data processor 1421 includes the adder 1425, OR circuit 1426, ~COR (exclusive OR) 'VO 92/06~36 PCr/US9l/07383 ~3~

circuit 1427 and comparator 1428, as represented OD Fig. 12B. The adder 1425 provides, in respoDse to the SEL INP DATA (x) selected input data siBnals from the left and right child receiver buffers 1420(x), SUM signals an UP CRY oU'r carry signal and an OVFL overflow ill signaL which together represent the sum of the values represented by the SEL ~P DATA (x) sigDals. Associa~ed with the adder 1425 is a carry latch 1570 and a carr,v select multiplexer ~571 that, under coDtrol of a control circuit 1577, selectively latches the UP CRY OUT carry signal. The carry latch 1570, at each tick of the NODE CLK signal provides a CRY IN carry in signal which is also used by the adder 1425 ;D
generating the SUM signals. Similar circuitry (Dot shown) is pro~ided for the OVFL overflow signaL to provide the OVFL IN signal. In addition, if the SEL INP DATA (~1:) signals represeDt the last data nibbles 70(i) of their respective coDtrol ne~work message pacl~ets 60, and if the controJ network message packets 60 enable a scan or reduce operation, the up output packct assember 1422 uses the CRY IN and OVFL IN signal in conditioning the scan overflow bit 80 of the control network message packet 60 it is trar smitting.
The OR circuit 1426 and XOR circuit 1427 generate OR and XOR signals representing the bit- vise OR and XOR, respectively, of the SEL rNP DA.TA (x) signals. In addition, the complements of the XOR signals are directed to an AND gate 1574. If all of the XOR signals are negated, which occurs if the corresponding bits of the SEL INP DATA (L) and SEL rNP DATA (R) sigDals have the same value, the AND gate 1574 is energi~ed to assert a L EQ RT left equals right sigl~al. Associated vith the XOR gate 1427 is a latch 157S and a multiplexer lS76 that selectively latches the L EQ RT
siE~nal. The latch 1575, at each ticl~ of the NODE CLK signal, generates a DATA EQ LAT data equals latched signal.
The comparator 1428 generates L GT RT left greater than right sigDal whose co~dition indicates whether the binary-encoded value of the SEL INP DATA (L) signals is greater th~ the binary-encoded value of the SEL INP DATA (R) signals. Asso~ated with the comparator 1428 is a htch 1572 and multiplexer 1573 which selectively latches the L GT RT sig~al. The multiplexer 1573, along with multiplexer 1576, are controlled by thc control circuit 1577. Generally, while the DATA EQ
LAT signal is asserted, indicating that the bih~ encoded value of the successive data nibbles 70(i), represented by the SEL INP DATA (x) sigllals at successive ticks of the NODE CLK signal, are equaL
the contzol circuit 1577 enables the multiplexers L576 and 1573 to direct the L EQ RT and L GT RT
signals, respec~ively, to their respective latches 1575 and 1572.
However, if the L EQ RT signal is llegated, indicating that the binary-encoded values of the SEL INP DATA (x) signals are not equaL the latch 1575 negates the DATA EQ IAT sigllal. At that point the L GT RT signal, which is latched by the latch lS72, indicates which of the SEL INP DATA
(x) signals has the greater binary-encoded llalue. The control ci~cuit 1577 tbereafter enables the multiple;cers 1576 and lS73 to couple the output signals from their latches ~sn to their i~put terminals, so that they maintain their co~ditions at succeeding ticl~s of the NODE CLK 5igllal5. Thus, the control circuit 1577 ensurcs that the COMP L/R LAT signal ~ill identify the control network message pacl~et 60, represented by the SEL INP DATA (x3 signals, having a packet data portion 62 with the greater binary-encoded value.

., , ,:

, ,: ~, :. , - .~,:: , ": :- : :, -: , . .. . . . . . .

WO 92/06436 PCr/US~1/073~'"
2Q~335.~ -185-The control circuit 1577 gelerates t vo control signals for controlling the multiplexers 1573 and 1576, n~mely7 a LAT EQ/COMP LAT latch equal/eomparator latch signal and a HOLD EQ/COMP
LAT hold equal/compafator latch sig~al. While the l..AT EQ/COMP LAT sigDal is asserted7 the multiplexers 1573 and 1576 couple the L GT RT and L EQ RT signal to the data input terminals of their respective latehes 1572 and 1575. Initially, contemporaneous with t7ne assertions of the UP RCV
ST 0 through UP RCV ST 2 signals by the up con~rol circuit 1424, 'aoth the HOLD EQ/COMP LAT
and LAT EQ/COMP lAT signals will be negated. In that eonditiorl, tne multiplexers 1573 and lS73 couple asserted signaLs to the data input latches of their respective latches 1572 and 1575, to set the latches and assert the COMP L/R IAT and DATA EQ LAT signals.
However, while the UP RCV ST 3 througb UP I?CV ST 9 signals are asserted, which occurs while the SEL INP DATA (x) signals represent the successive data nibbles 70(i) of ~he control networl~
message pacl~ets 60, an AND gate 1580 asserts the LAT EQ/COMP LAT s~gnal, if the latch 1575 is asserting the DATA EQ LAT signal, indicating that t.he binary-encoded values of the data nibbles 70(i) are equal, and if ~he up output pac~et assember 14æ is asserting either the UP OUT PICI' MS or UP
OUT P~ SS signals. The last coDdition indicates that the data represented by at least one of the SEL
INP DATA (x) signals is from a single-source or multiple-source message. If the DATA EQ LAT
sigual is negated, indicating ehat the binary-encoded values of the SEL INP DATA (x) sigl~als are not the same, the IAT EQ/COMP LAT signal is negated.
The control circuit 1577 asserts the HOLD EQ/COMP LAT signal under several arcumstances. If, while the UP RCV ST 3 through UP RCV ST 9 signals are assereed, an UP BOTH
SS sig~al is asserted, and if the DATA EQ LAT signal is rlegated, an AND gate ~81 is energized eo enable an OR gate 1582 to assert the HOLD EQ/COMP IAT signal. The UP BOTH SS signal is asserted if boeh child nodes are trallsmitting single-source messages. In this G~ both single-sDurce mes~age are of the configuration type, and if AND gate 1581 is energized, the data nibbles 70(i) single-source messages will repJesent different height values. As noted a~ove, if control network node 51 rcceives sin~le-source messages, of the configuration type, having diffeseDt height Yalues, it tsansli~its a si~gle-source message i~ whlch the height value is the ma~amum of the ~wo receiYed height values.
AND gate 1581 onables this to occur.
In addition, if, while the UP RCV ST 3 through UP RCV ST 9 si{~als are asserted, if the up output packet assembler 1422 is asserting the UP OUT P~Cr MS up output packet multiple-sourcc signal, indicatiDg that it is transmitting a mul~iple-source message, an AND gate 1583 is energ~ed to enable the OR gate ~82 to assert t~e HOLD EQ/COMP 1~1- signal. This enables the latch ~572 to maintaiII its condition at that point, and, if the multiple-source message enables a ma~um operadon, ~he up output message assembler may use the COMP L/R LAT signal in determiDillg which of the SEL lNP DATA (x) signals to use ;D the control network message packet ~ it is transmitting.
In addition, if the control network message packet 60 being transmitted by the up output packet asse~ber 14æ is multiple-source message enabliDg a multiple-word operation, which effectively uses subsequent control network message packets 60 of the multiple-source message type, an AND

WO 92/06436 ~ ~ ~ 3 ~ US91/07383 gate 1~84 enables the OR gate 1582 to maintain the HOLD EQ/COMP LAT signal asserted. That a multiple-source message enables a multiple word operation is indicated by the assertion, by the up output packet assembler 142~ of a UP M SCAN multiple-word scan signal, along with the assertion of the UP OUT PKT MS signal. When the up control circuit 1421 then asserts the UP RCV ST 17, AND
8ate 1~4 is energiæd to enable the OR gate 1582 to assert the HOLD EQ/COMP LAT signal.
It will be appreciated that the adder 1425, OR circuit 1426, XOR circuit 1427 and comparator 1428 operate in response to each set of the SEL INP DATA (x) signals provided thereto, without regard to the types of messages aDt message packets as represented by the SEL INP DATA (x) signals, or the operations enabled thereby. The up output packet assembler deterrnines which, if any, of the output signals provided by each of the circuits 1425 through 1428 to use iD assembling control nehvor3~ message packet 60 for transmission thereby. Thus, the circuits 1425 through 1428 to not require or use enabling signals based on the type of control network message paclcet 60 being assembled in order to operate.
iY. Up Output Packet Assembler 14æ
Fig. 12B-3 depicts a detailed block diagrarG of the up output packet assembler. With reference to Fig. 12B-3, the up output packet assembler l4æ comprises an output data selector 1590 tha~ receives signals from a number of sources and, under coutrol of OUT D~TA SEL output data source seloct signal!s from an output source identifier circuit lS92, couples signals from olle source as the low-order P FLICK 1 lP (3:0) sigllals. In addition, an output tag selector circuit 1592 that, under control of OUT
TAG SEL output t~, select signals from the output source identifier circuit 1591, selectively couples signals from a number of sources as the P FUCK UP (4) signal. It ~ill be appreciated that the low order P FLICK UE' (3:0) signals, at succossi~e ticlcs of the NODE CLK sigtlal, reprcsents the packet inforrnadon portion of successive flicks of a control network ~nessage paclcet 60. Similarly, the high-order P ~LICK UP (4) signal, also at succossive ticks of the NODE CLK sigual, represents the succossive tag sigl~ah~ the control network message packet 60.
The up output pac'set assembler 1422 also includes an output packet status store 159~. that recei~es the P FJIC}C UP (3:0) sign ls, as well as the UP RCV ST (2:0) signak. a~d the NODE CL~C
signal aud generates up output packet status signals identif~, the messaF,e type, packet type and, if the control network message packet 60 represented by the P FLICK UP (4:0) signals is of the multiple-source type, the particular operatioDs enabled thereby. In addition, the up output packet assembler 14æ includes a checksum generator 1594 that, after being reset by the UP RCV ST O sigDal, at successive ticks of the NODE CLK signal psoYides CHECK (4:0) checksom signals in response to the P FLICK UP (4:0~ signals. Dur ng transm5sion of the 'ast fliclc o~ the control neework message packet 60, contai~ing the checksum field 63 (Fig. 5), the output source identifier circuit ~591 enables the output data and tag selectors 1590 and 1592 to couple the CHECK ~3:0) and CHECK (4) checksum signals, respectively, as the P E:~ICK 'UP (3:0) ~d P FLICK UP (4) si~als.
The structures of the various c~cuits depicted on F;g. 12B~ ill not be described in detail.
The output data and tag selectors 1590 and 1592 each comprise one or more multiplexers, which, under WO 9Z/06436 PCT/USg1/07383 2~93~ 187-coDtrol of the OI~T DATA SEL and OUT TAG SEL sigDal, selectively couple signals input thereto to their respective output terrninals. The particular signals coupled by tbe respective selectors 1~90, as determined by the output source identifier circuit 1591, depends OD seYeral factors, including which types of control network message pacltets 60 are being received by tbe left and right child receiver/buffers 1420(x), on whether tbe park buffers 1442 iD the left and right child receiYer/buffers 14~0(x) are storing message packets, and on which UP RCV ST ~ 0) state signal is being asserted.
The output source identifier lS91 determines which type of control network message packet 60 to generate, which information is latched in the output packet status store for use by the output source identifier 1591 as it enables the output data selector 1590 to pro~/ide P FLICK UP (3:0) signals represent~ng the successive flicks of the control network mes~age pacl~et 60.
ln addition, the output source identi~er 1591 in response to the INP PKT l YPE signals and the UP RCV ST (12:0) si~als, enables the output tag selector 1592 to selectively couple signals input thereto as the P FLICK UP (4) signal. The particular input sigDal seleGted to be coupled will depend OD the particular UP RCV ST (12:0) signal asserted, so that the control network message packet 60 generated has the correct sequence of tag bits represerlted by the successive P FLICK UP (4) signal.
Generally, the output data selector 1590 receives the SEL INP DATA (x) signals from the left and right ch;ld receiver/buffers 1420(x), as well as the SUM, OR, and XOR signals from the flic~ (up) data proces~or 1421. In addition, the output data selector 1590 reGeives lDLE, MS: and ABSN signals representing the encoding of the message type field 64 for a control network message pacl~et 60 of, respectively, the abstain, multiple-source and abstain message type. Finally, the output data selector lS90 receives the CHEC~C (3:0) chec~sum sigDals from the checksum generator 1594.
The output source iden~ifier 1591 enables the output data selector 1590, at each tick of the NODE CLEC signal, and in synchronism with the UP RCV ST (12:0) sienals, to couple one of these sets of signals to its output terminals as the P Fl.ICK UP (3:0) s;gnals. In selectiDg which signals to be coupled, the output source identifier uses rNP PKT TYPE input packet type signals from the input packet type decoders 1447 in the child receiver/buffers 1420(x), representing the types of messages received from the left arld nght children. Thus, if control network n~essage packets 60 of different message types are being received from the child nodes, the output source identifier 1591 detennilles which, if either, type v.ill be traDsmitted.
If, for example, the INP PKT TYPE signals ~ndicate that a single-source message is being received from one child node and no message is being received from the other child node, the output source identifier 1591 will generate OUT DATA SEL signals to enable the output data selector to couplc the SEL INP DATA (x) signals from the child receiver/buffer 1420(x) whose c~ild ~ode is providing the single~source message. The output source identifier 1591 enables Ihe output data sdector 1;90 to ~ouple the SEL lNP DATA (x) si~nals while the UP RCV ST 0 through UP RCV ST
10 sigoals are being asserted. In that case, the output packet status sts)re 1593 asserts the UP Ol lT PK
SS up output pac~et single-source si~al while the UP RCV ST 0 is asserted in synchroDism with the next tick of the NODE~ CLK signal. In addition, if the single-sousce message is of the configuration - : : : . : .: . : ~ : .
, . . ..
:, , : . ,', , . ,: . ' ., '' ~

WO 92/~6436 2 ~ ~ 3 ~/US91/073~3 type, the output packet status store 1593 asserts the UP OUT PKT CONFIG up output pacl~et configuration signal while the UP RCV ST 1 is asserted in sync~ronssm with the next tick of the NODE
CLK signal. The output source identifier uses the UP OUT PlCr SS and UP OUT P~r CONFIG
signals to identify the tylse of packet it is generating as it generates OUT DATA SEL sigllals enabling generatio~ of successive fliclcs oî the message packet.
However, if INP PKT TYPE signals indicate that single-source messages of the configllration type are beirlg received from both child nodes, the output source identifier 1S91 will generate OUT
DATA SEL signals to erlable the s~ulput data selector to couple the SEL INP DATA (x) signals from either child receiver/buffer 1420(x). If the COMP L/R IAT compare left/right latched signal from comparator 1428 (Flg. 12A) illdicates that the bi~ary-el~coded value of the pac~et data portion 62 from one child is greater than the binary-encoded value of the packet data portion 62 received from the other, it generates OUT DATA SEL signals to enable the output data selector at that point to begin coupling the SEL INP DATA (x) signals from that child's receiver/buffer 1420(x) as the P FLICK UP
(3:0) siguals. The output source identifier 1591 ena'oles the output data selector 1590 to couple the SEL INP DATA (x) signals while the UP RCV ST Q through UP RCV ST 10 signals are being asserted. In addition, the output packet status store 1593 asserts the UP OUT PK SS up output pacl~et siDgle-source signal and UP OUT PKT C:ONFIG up output packet configuration signal while, respe4tive1y, the llP RCV ST 0 and UP RCV ST 1 sigllals are asserted iD syrchronism with the next tick of the NODE CLK signal. The output source identifier uses the UP OUT P~ SS and VP O~T
P}CI` CONFIG signals to identify ~he type of packet it is generating as it generates OUT DATA SEL
sigDals enabling generation of successive flicks of the IDessage pacl~et.
However, if INP P~ TYPE signals iDdicate that multiple-source mcssages are being receiYed from both child nodes, the output source identifier 1~91 wilL in synchror~ism with the assertion oî the UP RCV ST 0 sigDaL generate OUT DATA SEL signals to enable the output data sele tor 1590 to couple the MS signals, which correspo~d to the MS message typ& code, as the P Fl,ICK UP (3:0) signals. The output packet shtus s~ore 1593, in response, asserts the UP OUT PK~ MS up output packet multiple-source signal, which controls subsequent operads)ns by the output source identifier circuit 1591. The output source identifier circuit, at the asserfion of the UP RCV ST 1 and UP RCV
ST 2 sigl~als, generates OUT DATA SEL signals that eDable the output data selector to couple the SEL INP DATA (x) sigl~als ~om either of the child receiqer/buffers 1420(x) as ~he P Pl~ICK UP (3:0) signals. The outpu~ data selestor 1590 may selectively couple either the SEL INP DATA (x) sigDals, since the op~ratioD eDabled by the multiple source messages will be the same.
The output packet status store 1593" enabled by the assertion of the UP RCV ST 1 siglaals, in syDchro~ism with the tick of the NODE CLK sigllal, may also assert the OUT P~ SCF/RED output pacl~et ssa~ fonvard/reduce signal, an UP REDUCE sigrlal and the UP M SCAN up multi-word scan signal. Thes sigllals, which are deri~ed from Lhe encoding of the packet type fields 65 and patte~ bits 67 of the message packets 6C 'oeing received, provide inforr~Jation as to the operation enabled by the multiple-ss)urce messages. If the OllT P~ SCF/RED signal is asserted, the o~"eration e nabled by the .~ -: . . . . . .. .. . ..
", ., ,, , : . , .. .. .: , :, : ,......... .

WO 92/0~436 PCI /US91/07383 2~933~ -189-multiple-source message is either a scan forward operatiorl or a reduce operation; if the signal is negatcd the operation is a scan baclcward operation. If the OperatiOD is a retuce operatiorl, the output packet status store 1593 also asserts the I~P REDUCE signals. In addition, if the operation is a multi-word scan operation, the output pacl~et store ~593 asserts the UP M SCAN signal. Thc output source identificr 1591 uses these signals in subsequeut operations.
When the UP RCV ST 2 sigllal is asserted, in synchronism with the tick of the NODE CLK
signaL the output pacl~et status store 1593 asserts one of ~e UP COMB ADD, UP COMB OR, UP
COMB XOR and UP COMB MAX up combine add, OR, XOR, and ma~nu~n signals. l'hese signals, which are deri~ed from the encoding of the combine function field 66 of the message packets 60 beiug recei~ed, pro~ide additional infor~nation as to the operaion enabled thereby. The output source identifier 1591 uses these signals in subsequent operalions. ~ particuJar, if the UP COMB ADD, UP
COMB OR, UP COMB XOR signals are asserted at the assertior s of the UP RCV ST 3 through UP
RCV ST 10 signals, the output source idendfier circuit 1591 generates OUT DATA SEL signals to enable the output data selector to couple the SUM, OR or XOR signals, respectively, as the P FLICK
UP (3:0) signals On the other hand, if the UP COMB MAX sigllal is asserted, the output source ideutifer 1591 uses the COMP L/R LAT compare left/right latched signal to select one of the SEL INP DATA (L) or SEL INP DATA (R) signals to be coupled as the P lFLICK UP (3:0) signal, and generates appropriate OUT DATA SEL signals. In particular, if the COMP L/R LAT cor~pare left/right latched signal from comparator 1428 (Fig. 12A) iDdicates that the binary-encoded value of the packet data portion 62 from one chilt is greater thaD the binary-encoded valuc of the packet data portioD 62 received from the other, it generates OUT DATA SEL signals to enable the output data selector at that point to begin CoupliDg the SEL ~P DATA (x) signals from that child's rceeiver/bu~er 1420(x) as the P FIICK UP (3:0) sigDals.
lf INP PKT TYPE signals indicate that a muhiple-source message is beiDg received ~om one child node, and an idle message is being received from the other child node, the output source identifier nses the PK BUF (x) ST park buffer status signals to determine whether a message pacl~et from tho other child node is beiog buffered in a packet buffer. If so, the operations described above are pcrformed, in the same manDer as if multiple~source messages were being received from both child nodes. Similarly, if the input packet type sigoals indicate that idle or NPAC nil packet messages are beiDg recoived from both cbild nodes, and the PK BUF (x) ST status signalls indicate that park buffers 1442 in both c~ild receiver/buffers 1420(x) are buffering message packets, the oper~tio~s described above are perfonned, in the sarne manner if multiple-source messages were being received from both child nodcs.
ln any case, while the UP RCV ST 11 signal is asserted, which occurs during transmission of the global information portion 71, the output source identifier generates OUT DATA SEL signals that enable the output data selector to couple the OR signals as the P FLICK UP (3:0) signals. This occurs regardless of the message types of the message packets 60 received from the respective child nodes, or , : .

". .. . ... . . . . ..

WO 92/06436 PCr/US91/07383 ~3`3~~3 the message type of the message packet o0 transmitted to the parent node. As notcd above, the global information portion 71 in the packets 60 as transmitted by the leaves 21 (Figs. 1, 4A and 4B) contairls status information that is ORed by the control network 14 and the result broadcasted to all leaves in the partition, including the ssalar processors 12. Thus, by monitoring the global information portion 71 of packets 60 that they receive~ the scalar processors 12 can determine sta~us of, for example, the processing elements 11.
ln addition, while the UP RCV ST 12 signal is asserted, it enables the output data selector 1590 to couple the CHECK (3:0) checksum signals as the P FLICK UP (3:0) signals to provide the low-order portion of the checksu n field 63 of the control network message packet 60 bemg generated.
3. Root Flag 1407 And Associated Control Circuitry Fig. 12C depicts details of the root ~lag 1407 (Fig. 12A) and circuitry for controlling its condition. With reference to Fig. 12C, the control circuitry incJudes a left height comparison circuit 1600 and a correspoding right height comparison circuit (not sho~vn). Generally, each of the height comparison circuits, if the respective left or right child nodes are providing single-source message packets of the configuration type, compares the height values in data nibbles 70(0) aDd 70(1) (Fig. 5) to NODE HT Dode height signals whose binary-encoded value represents the level of the control networlc node 51 ;D the control net vork 14. If the values are the sarne, the height comparison circuit generates a DATA EQ HT (x) data equals height signal l~ corresponds to ~L~ (left) or ~R~ (right)l. On the other hand, if the values differ, a DATA NE~ HT (x) data not equal to height signal is asserted. The DATA EQ HT (x) and DATA NE HT (x) signals are used to control the condition of the root nag 1407.
More specifically, if (a) the CL/SS left child single-source signal is asserted co~temporaneous with the assertion of the UP RCV ST 0 signal, indicating that the left child node is providing a single-source message, and (b) the up output packet assembler 1422 is asserting the oU'r PKT SS output message packet siDgle-source and OUT P~r TYPE CONFIG output packet type configuration signals, in response to the assertion of the UP RCV ST 3 and UP RCV ST 4 siE~als the left height comparison circuit 1600 compares the SEL INP DATA (L) selected input data signaLs to the NODE HT signals.
As shown in Flg. 5, while the UP RCV ST 3 and UP RCV ST 4 signals are asserted, the SEL INP
DATA (L) signals represent the data nibbles 70(0) and 70t1~ containing the root height data. If tbe }oot height data in the co~trol network message packet 60 corresponds to the binary-e~coded value of the NODE HT signals, the left height comparison circuit 1600 asserts the DATA EQ LT (L) signal.
Alternadvely, the left heigbt comparison circuit 1~00 asserts the DATA LT HT (L) signal or DATA
GT HT (L) si~al if ~e value of the root height data is less than or greater than the b~nary-encoded ~alue of the NODE HT signals. If the left height comparison sirsuit 1600 asserts either the DATA LT
HT (L) or the DATA &T HT (L) signals, an OR gate 1601 asserLs lhe DATA NE HT (L) left data not equal to height signal.
The DATA EQ HT (x) and DATA ME HT (x) signals (~xn references ~L~ and ~R~) from the left and right hei8ht somparison circuits are soupled to a height compar~son resolver circuit 1602. The . . . :: : :; :

. :. , . .: : . : : : . : :

WO ~2/06436 PCI/US91/07383 2093~5~ -191-height companson resolver circuit 1602 includes circuitry for performing two functions. First, a root establishment enable circuit 1603 asserts a ROOT EN root enable signal if at least one of the left or right height comparison circuits asserts the DATA EQ HT (x) signaL and the other heigh compar~on circuit is Dot asserting its DATA NE HT (x') signal. This ensures t~at the ROOT EN signal is not asserted if both child nodes are providing single-source messages of the coofiguratioo packet type, but with different hcight values. The height comparison resolver circuil also includes a configuration error detector circuit 1604, which asserts a CONFIG S ERR conf~uration software error signal if at least one of the left or right height comparison circuits asserts the DATA EQ HT (x) signal, and the other heigh comparison circuit is asserting its DATA NE HT (x~) signal.
More specifically, the root establishlnent enable circuit 1603 includes two AND gates loO5 and 1606, and an OR gate 1607. If the left heigh comparison circuit 1600 is asserting the DATA EQ HT
(L) signal, and the right height comparison circuit is not asserting its DATA NE HT (R) signal, the AND gate 1605 is energized to asscrt a (L) ROOT EN left root enable signal. On the other hand~ if the right height comparison circuit is asserting the DATA EQ HT (R) signal, and ~he Icf~ height comparison circuit 1600 is not asserting its DATA NE HT (L) signal, the AND gate 1606 is energized to assert a (R) ROOT EN right root enable signal. If either the (L) ROOT EN signal or the (R) ROOT EN signal is asserted, the OR gate 1607 is energzied to assert the ROOT EN root enable signal.
It will be appreciated that the (x) ROOT EN signal will be asserted if (a) the ~x" (left or right) height comparison circuit determines that a control network message packet ~ has bee~i received from the lx" child indicating that this control network node 51 should be a root node, and ~b) no control nenvork message packet 60 has been received from the other child node indicating that another node should be root node. If b~th child nodes provide control llet vork rnessage paclsets 60 indicating that the control nenvork node 51 should be a ros~t node, both the AND gates 1605 and 1606 are ener~zed to assert the respective (x) ROOT EN root eDable signals.
Similarly, the cor~lguration error detector circuit 1604 includes two AND gates 1610 and 1611 and an OR gate 1612. If the left heigh comparison circuit 1600 is asserting the DATA EQ HT (L) signaL and the right height comparison circuit is a1so assertirg its DATA NE HT (R) signal, the AND
gate 1610 is energized to assert a LFT EQ/RT NE S ERR left equal/right not equal software error signal. On the other hand, if the right he;gh~ comparison circl~it is asserting the DATA EQ Hl (R) signaL and the left height comparisoo circui~ 1600 is also assertiog its DATA NE HT (L) s~gnaL the AND gate 1606 is eDergized to assert a RT EQ/LFT NE S ERR right equal/left not equal soflware error signal. It will be appreciated t'nat either signal will 'c,e asserted if (a) the a neight comparison circuit determines that a control network message packet 6Q has been received from the its cbild indica~iing that this cootroi network node 51 sLould be a root uode, aod (b) aDother control De~work message pac~et 6C, has been received from the other child node indicating that another node should be root node. If either the L~:T EQ/RT NE S ERR signal or the RT EQ/LFT NE S ERR signal is asserted"~he OR gate 1612 is energi~ed to assert the CONFIG S ERR co~lguration sofhvare error ,"" ,. ...

WO 92/06436 Pcr/us91/n7383 2 ~ ~ 3 3 ~ ~

signal, which may be used by the up output message assembler 1422 in establishing the condition of the software error bit 76 in a control network message packet 6û.
The ROOT EN and CONFIG S ERR signals are used by a root flag conditioning circuit 1620 to establish the condition of the root nag 1407. If the CON~IG S ERR configuration software signal is negated, and if the up output packet assembler 1422 is asserting the OUT Pl~r SS and OUT PKT
TYPE CONFIG signals, an AND gate 1621 is energized in synchronism with the assertion the UP
RCV ST S signal to assert a COND ROOT signal to enable the root flag to be conditioned. As noted above, up output pac~ct asscmbler 14æ asserts th~ oU'r PKT SS and OUT PKT TYPE CONFIG
signals, if if it is assembling a rnessage of the single-source message type and configuration pac~et type.
whic4 in turn, occurs if at leas~ one child node is providing such a ~ontrol networ~ message packet o0.
The assertion of the COND ROOT condition root signal enables input tenninals of AND
gates 1622 and 1623. The A~ID gates 1622 and 1623 are also controlled by the true and complemeDt, respecti~ely, of the ROOT EN signal. If the COND ROOT signal is asserted, and if the ROOT EN
signal is also asserted, AND gate 1622 asserts a SEL SET select set signal, which- controls a multiplexer 1624. When the SEL SET signal is asserted, multiplexer 1624 couples an asserted signal as a SET
ROOT signal to the data input terminal of a flip-flop 1625. The asserted SET ROOT signal enables the flip-flop 1625 to be set at the next tick of the NODE CLK signal to assert a ROOT SET EN root set enable signal.
Thereafter the AND gate 1621 is de-ener~sized, to negate the COND ROOT signal. The negated COND ROOT signal, in turn, de-energizes AND gate 16æ which negates the SEL SET select set signal. Since the AND gate 1623 is also de-energized, a SEL CLR select clear signal is also negated, enabling the muldplexer 1622 to couple the ROOT SET EN signal as the SET ROOT signai to the data input tenninal of the fli~flop 1625. Thus, while the SEL SET and SEL CLR signals are negated, the flip-flop 1625 mainlains its set condition at successive ticlcs of the NODE CLK signal.
The now-asserted ROOT SET EN signal is also coupled to an input terminal of a multiplexer 1626. When an UP RCV ST 12 signal is asserted, multiplexer 1625 couples the ROOT SET EN sig~
to its output ter~ninal as a ROOT FLAG EN root flag enable signaL whicb, in tum, is coupled to the data input ter~ninal of ~oot flag 1407. Since the ROOT SET EN sig~al is asserte~L the ROOT FLAG
EN sigDal will be asserted as welL enabling the root flag to be set at the next tick of the NODE CLK
sigllal. The setting of the root flag 1407 enables it to assert the ROOT UP IAT root up latch sig~al, which controlls the packet buffer 1406 (Flg. 12B) and is coupled to the parent node as ~he ROOT UP
signal. After t~e UP RCV ST 12 signal is negated, the multiplexer 1625 couples the ROOT UP LAT
signal as the ROOT FLAG EN signal, enabling the root flag 1407 to maintain i~s condition at successive ticks of the NODE CLK signal.
On the other ha~d, if, while the AND gate 1621 is assertmg tbe CONl;) RC)OT signal, the root establishlnent enable circuit 1603 is neBating the ROOT EN signal, AND Bate 1623 is energized to assert the SEL CLR select clear signal. ID that case, the AND gate 1622 will be de-energ~;~ed, negating the SEL SET select set signal. In that condition, the multiplexer 1624 couples a negated signal as the .. .. . .. . .. . . . .. ..
:: ~
: : . :;:: : : . . ,. :.
.. ; :: : :; ::
,,' ' ' ' .; .. . ~ ' . ' ' ', ' ' ` ; '.'. ' ' ' ,. ' '' ' '., ~', " . .. ' .,' ~.
. ': ':: . ' ' ' ' . : :~: : . ,: , ' , WO 92t06436 PCr/US91/07383 20~3~ 193.
SET ROOT slgna~?which clears flip-tlop 1625 at the Dext tick of the k~ODE CLK signal, and negating the ROOT SET EN root set enable signal. As above, when the AND gzle 1621 negates the COND
ROOT signaL both AND gates 1622 and 1623 will be de-energized, negating the respective SEL SET
and SEL CLR signals, so that multiplexer 1624 wilJ maintain the flip-flop 1625 in the same condition at successive ticks of the NODE CLK signal.
The now-negated ROOT SET EN signal is also coupled to an input terminal of multiplexer 1626. When an UP RCV ST 12 signal is asserted, multiplexer 1625 couples the negated ROOT ~ET
EN signal to its output terminal as a ROOT FLAG EN root flag enable signal, which, in turn, is coupled to the data input terminal of root nag 1407. Sirlce the ROOT SET EN signal is negated, the ROOT FLAG EN signal will be negated as welL enabling the root nag to be cleared at the next tick of the NODE CLK signal. The clearing of the root flag 1407 enables it to negate the ROOT UP LAT
root up latch signal. After the UP RCV ST 12 signal is negated, the multiplexer 1625 couples the ROOT UP l~T signal as the ROOT FLAG EN signal, enabling the root flag 1407 to maintain its condition at successive Licks of the NODE CLK signaL
As noted above, the multiplexer 1625 couples the ROOT SET EN root set enable signal to tbe data input terminal of root flag 1407 as the ROOT FLAG EN signal in synchronism with assertion of the UP RCV ST 12 signal. Thus, the root nag 1407 is conditioned contemporaneous w~th transmission by the up output packet assembler 1422 of the end of the control network message packel 60 it is currently transmitting. Since the ROOT UP LAT signal provided by the root flag 1407 controls transmission of control networlc message packets 60 to the parent node and also controls storage of packets 60 in the packet buffer 1406, it will be appreciated that conditioning of the root nag in synchronism with assertion of the UP RCV ST 12 signal ensures that partial packets not be transmitted to the parent node or stored in the packetlbuffer.
4. flick down coDtrol portion 1402 Fig. 12D depicts a detailed block diagram of the flick do vn control portion 1402. With reference to Flg. 12D, the flick down control poriton 1402 includes a down source select circuit 1650 that receives P FLICK DN (4:0~ signals from a parent node in the control nen~ork 14 and BUF P
FLICK UP (4:0) signals from the packet buffer 1406 (fig. 12B), along with the ROOT IJP LAT signal.
In response to the ROOT UP LAT signal, at successive ticks of the NODE CLK signal the down source select circuit 1650 selectively couples either the P FLICK DN (4:0) signals or the BUF P FLICK
IJP (4:0) signals as SEL DN INP DATA (3:0) selected down input data signals and a SEL DN INP
TAG signal. In addition, the down source select circuit 1650 provides DN INP STA/CTRL down iDpUt status/control signals and the SEL DN INP T~G signal to a down control circuit 1651.
In turn, the down control circuit 1651, like up control circuit 1424, generates a number of control and timing signals for controlling the operations of the flick down control portion. ln particular, the down control circuit generates DN RCV ST (12:0) down receive state timiDg signals, wbich comprise thirtec~ signals DN RCV ST 0 through DN RCV ST 12, which are successively asserted, in response to successive ticks of the NODE CLK signal, in synchrol~sm with the receipt, by the down source select circuit 1650, of the tbirteen flicks of a control network message packe~ 60.

., :: : ,, :: . ; : ,, .. ,:: ,: ,: ::

WO 92/06436 ~ ~) 3 ~i91/0738?.

-19~ . , In addition, if the root flag 1407 is asserting the ROOT UP LAl root up latch signal, the down control circuit provides the PACKET BUF RE packet buffer read enable signal to enable the packet buffer 1406 (Figs. 12A, 12B) to generate BUF P PLICK UP buffered flick up signals representing a control network message packet 60. S;milarly, if the control network message packet 60 receiYed by the down source select circuit, either from the parent node in the control nehYork 14 or from the packet buffer 1406, is a multiple-source message, the down control circuit 1651 pro~ides a SC~' BUF
RE scan buffer read enable signaL which enables the scan buffer (Flgs. 12A, 12B) to collple message packet data stored therein to the flick down coDtrol pOrtiOD 14~. The data from the scan buffer 1410 is combined with message packet data from the the message packet 60 being received by the down source select arcuit 1650 by a aick down data processor 1652.
The down control circuit 1651 also receives the SEL DN INP TAG selected down input tag signal from the down source select circuit 1650 and, in response to the signals representing the condition of the scan flow bits 72(i) ~Fig. 5), generates FLOW CrRL (UP) flow control up sigriaL~. that are coupled to the up control circuit 1424. As described above, the rLOW CrRL (UP) signals are used by the flick up control portion to control the transfer of control network message pac'cets 60 of the multiple-source message type to the parent node. Similarly, the down control circuit 1681 receiYes a Fl,OW CONTROL (DN) flow control down signal from the up control circuit, and uses it to control the transfer of control network message packets 60 of the multiple-source message type ts:~ the child nodes.
Furthermore, the donw. control circuit 1651 receives an ElRR error signal from the up control circuit 1424. The down control circuit may use the ERR signal to enable the conditioning of the softwarc error bit 76 in control network message packets 60 transmitted by the flick down control portion 1402.
The flick down control portion 1402 includes leh and right down output packet assembler circuits 1653 and 1654 which receive signals from the down source select circuit 1650, from the flick down data processor 1652, from a down tag processor 1655 and, under control of C(x) DN OUT SE
child dow~ output sclect signals l~x" references ~L" (left) or ~R~ (right)~ from the down control circuit 1651, generates control nenvork message packeOE 6~ for transfer to the respectiv~ left and right child nodes in the control net vork 14. In particular, at successive ticks of the NODE CLK s~gnal the down output packet assembler circuits 1653 and 1654 generate C(x) FLICK DN signals representing successive flicks of control network message packets 60 generated thereby.
The types of control network message packets 60 generated by the down output packet assemblers 1653 and 16~4 depends in part on the condition of a (x) ROOT UP signal as controlled by the child node connected thereto. In particular, in response to the (x) ROOT UP signal, from a child node, shifting from the negated condition to an asserted coDdition, the corrcsponding down output packet assembler circuit, at the end of the cnntrol network message packet 60 it is then transmittirlg begins t~ansmitting C(x) FLICK DN signals representing message packets of the NPAC nil packet message type. The down p~cket assembler circuit waits until the end of the packet it is transmitting ... .. , , .:: :,.. ~ , WO 92/06436 PCr/US91/07383 20~3~ 195-wherl the condition of the (x) ROOT UP signal changes, so that the flick down coDtrol portion 1402 of the child node receives the entire control network message packet 60. Similarly, if the condition of the (x) ROOT UP signal changes from aD asserted condition to a negated condition, the down packet assembler circuit continues transmitting NPAC messages ~o the child node until the begiIming of the rlext pacl~et enabled by the down control circuit 1651. Accordingly, the do vn output packet assernblers geDerate arld transfer complete message packets to their respective child nodes.Otherwise the types of control networl~ message paclcets 60 generated by the respective down output packet assembler circuits 1653 and 16~4 at any particnlar time depends upon the type of control network message packe~ 60 being received by the down source select circuit 1650. In p~icular, if the dow~ source selea circuit 16~0 receives a packet 60 of the single-source message type or idle message type, the down control circuit 1651 enables the down output packet assembers ~o use the low-order four bits of each flick, comprising packet information portion (Fig. 5) thereof irl generating single-source message packets for trarlsfer to the child nodes. In generating the high-order tag bits for the successive flicks of the message paclcet beirJg generated, the down output packet assembler circuits use signals from various sourus, including the dov~n tag processor 1655 and the do vn control circuit 1651, to determine the condition of ~he respective bits.
Similarly, if the down source select circuit 1650 receives a message packet of the multiple-source type, the down control circuit 1651 will enable down output packet assembler circuits to use either the SEL DN INP DATA selected down input data, the SCAN BUF DATA scan buffer data signals or DN PROC DATA .down processed data signals ~rom the flick down data processor 1652 in generatiDg the data portion 62 of a control network message packet 60. The PROC F~ICK (DN) DATA processed flick down data signals from down data processor 1652 represent the sum, logical OR, logical XOR arld ma~omum of th~e SEL DN INP DATA sigDals, as generated by an adder 1660, all OR circuit 1661, an XOR circuit 1662 and a comparator 1663, respeceively. If, for example, the multiple-source message enables a scan forw~rd operatioD, the down co~trol circuit 1651 enables the down output packet assembler circuit 1653 to use the S~AN BUF DATA in the data portion 62 in the packet 60 being transmitted to the left child. In addition, the down con~rol circuit 1651 enables the do vn output packet assembler circuit 1654 to use the DN PROC DAl'A down processed data signals from the nick down data proressor 1652 in the packet 60 being transmitted to the right child.
On the other hand, if the multiple sowce message enables a scan backward operation~ the down control circuit 1651 eDables the down output packet assernbler circuit 1653 to use the DN PROC
DATA down processed data signals from the flick down data processor 1652 in the pac~et o0 being transmitted to the left child. ID additio~, the down control circuit 1651 enables the down output packel assembler circuit 1654 to use the SCAN BUF DATA in the data portion 62 in the packet 60 being tral smit~ed to the right child. E~fectively, if a multiple-source message enables a scan bachirard operation, the down control circuit 1651 enables the down output packet assembler circuits 1653 and 1654 to r~verse the message packets transmitted thereby to their respective child nodes, to achieve the reverse scan operatioD as described above.

:,,: , ,, , .,, . . , : .~

WO 92/06436 , ~CI/US91/07383 -196 . ., Furthesmore, if the m~lltiple source message enables a reduce operation, the down control circuit enables the down output packet assembles circuits 1653 and 1654 to use the SEL DN INP
DATA signals representing the flicks of the packet data portion 62 i~l the message packets 60 assembled thereby. As described above, in a reduce operation, the combination gf the data from ~he respective leaves 21 (Figs. 1, 4A and 413) is performed by the flick up control portions 1401 as the control network message packets 60 are bei~ transmitted up the control network 14 to the root node, a~d the packets transmitted down carry the results of the reduce operation as determined by the root node.
To enable generation of portions of a multiple source message other thaD the packet tata portion 62, the down control circuit 1651 may enable the down output packet assembler ciscuils to use SEL DN INP DATA signals received by the down source select circuit 1650 representing those portions of the message. For exarnple, the down control circult 1651 may enable the down output packet assembler cirucits to use SEL DN INP DATA signals representing the first three flicks, comprising the packet header 61, in the message packets genesated thereby for transmission to the child nodes. In addition, each down output packet assembler circuit 1653 and 1654 has a checksmn generator that generates a checksum value for use in the flick representing the checks un field 63 of the cootrol network message packet 60 'oeing geDerated.
Funally, if the FLOW CTRL (UP) sigDa'l from the up colltrol circui~ 16~ dicates that a c'nild node is unable to receive message packets 60, if a multiple soolce message is tnereafter received from the parent node, the down control circuit 1681 enables the left and right down output packet assembler circuits 1653 alld 1654 to ~,enerate idle ~essages for tra~sfer to their respective child nodes. As will be described below in cormection Y.~ith Fig. 12D-1, the dowu source select circuit 1550 will buffer tbe m~,Jltiple-source message received from the parent node, and the down control circuit 1651 will enable the up control circwt 1424 to, in turn, enable the up output packet as~sembler to set the scan flow bits 72 of control network message paclcets 60 it is trar~smitting to the parent node. The par~nt node is thereafter in`hibited from transmitting multiple-source messages to the flick dow~ control portion 1402.
Many of the circuits of the flick down control portion 1402 are similar to correspoGding ~ircuits of the flick down control portion 1401 and will not be described. Fig. 12D-1 depicts details of the down source select circuit 1650, showing the selection of either the P FLICK DN (4:0) signals frum the parent node or ~he BUF P FLIC~ UP (4:0) signals from the flick up control portion 1401 as the SEL DN INP DATA (3:0) and SEL DN INP TAG signals. As noted above, the down source select cirtuit 1650 makes the selection in resposne to the ROOT UP LAT root up latch signal from the root nag 1407 (F~gs. 12A arld 12C).
With reference to Fig. 12D-1, the down source select circuit 1650 includes a do vn source seleclor circuil 1670 that selec~ively couples either the P FLiCK DN parerlt flick down signals or the BUF P FLICK UP buffered pareDt flick up signals, or alternatively PAREOED P FLICK parked pareDt flick signals from a park buffer 1671 as tbe SEL DN INP DATA (3:0) signals and the SEL DN INP
TAG signals. The park buffer 1671 caD buffer a control net vork message packet 60 of the multiple-source message type if a child node is disabling transfer of such messages therelo.

. :: :. : , :: :, : . : , , . . :.

WO 92/06436 . PCI/US91/07383 20~3~3 -197-The own source selector circuit 1670 is controUed by two circuits, na~nely, a parent/packe~
buffer select enable circuit 1672 and a park buffer select eDable circuit 1673. Both circuits 1672 and 1673 operate in connection with message type identification signals from a down packet type decoder 1674, which, in synchronism with ~he DN RCV ST RST and DN RCV ST 0 down receivc state reset arld zero timing signals from the down control circuit 1651, generates signals identifying the message type of the message packet 60 being coupled by the do vn source selector circuit 1670. In particular, the down packet type deccder generates P/NPAC, P/MS, P/SS, PlIDLE and P/ABS signals whic4 wheD asserted, in&cate that the message packet 60 is of the NPAC nil packet, mulitple-source, single-source, idle or abst~un types, respectively.
T~e parent/packet buffer select enable circuit 1672 generates a ~igh-order SEL PAR/ROOT
select parent or root signal for controlling the down source selector 1670. The circuit 1672 operates in resposne to ROOT UP LAT root up latch and ROOT SET EN root set enable signals to condition a flip-nop 1675 to, in turn, control tbe SEL PAR/ROOT signal. If the parent node is transmitting P
FLICK DN signals representinF NPAC riil packet messages, the down packet type decode circuit i674 will assert the P/NPAC signal in synchronism with the DN RCV ST 0 signal. In that coJ~dition, an AND gate 1676 is de-energ~zed, which disables one input terminal of an OR gate 1677. Since at this point the DN RCV ST 10-12 signals, comprising tbe DN RCV ST 10 through DN RCV ST 12 rceive state timing signals, are also Degated, the OR gate is de-energized, negating a SEL ROOT SRCE select root source signal. Since the DN RCV ST 1-9 signals, comprising the DN RCV ST 1 through DN
RCV ST 9 signals, are also negated, a SEL ROOT SRCE HOLD select root hold sign~l is also negatd, enabling a multiplexer 1680 to couple the ROOT SET EN root set enable signal from fli~nop 1625 of the root flag conditioning circn~t 1620 (Fig. 12C) to the data input tenninal of fli~flop 1675.
If the ROOT SET EN signal is negated, fli~flop 1675 is clear at the ne~a tick of the NODE
CLK signal, thereby negateing the SEL PAR/ROOT signal. In that case, the down source selector 1670 couples the P FLICK DOWN signals ~rom the parent node, or PAR~OED P FLICK sig~als from the park buffer 1671, as the SEL DN INP DATA (3:0) selected down mput data signals and the SEL
DN INP TAG signal. When the DN RCV ST 1-9 si~als are asserted, the multiplexer 1680 couples the SEL PAR/ROOT signal &om the output terminal of flip-flop 1675 back to its data input terminal.
thereby maintaining the ni~nop 1675 irl the clear condition, and the SEL PAR/ROOT sigllal negated, at successive dcks of the NODE CLK signal.
On the other hand, if the ROOT SET EN signal is asserted, flip-nop 167~ is set at the next tick of the NODE CLK signal, thereby assertirlg the SEL PAR/ROOT select parent/root sig~al. It will be appreciated that this will be contemporaneous with the settir~ of root nag 1~07 in response to the asserted ROOT SET EN signaL The assertion of the SEL PAR/ROOT signal enables the down source seleclor 167u to couple the BUF P FLICK UP signab from the packet bu~fer 1410, or the PARKED P FLICK signals as the SEL DN INP DATA (3:0) and SEL DN INP TAG signals. Thu.s, if the parent node is transmitting NPAC nil packet messages to the control network node 51, the parent/packet buffer select enable circuit 1672 enables the down source selec~or 1670 to couple BUF P

WO 92/06436 2 ~ ~ ~P~91/0738~

-198- ' FLICK UP signals from the packet buffer 1406, or the PARE~ED P FLICK signais frbrn the park buffer, iInmediately upon the node becoming root node.
The assertion of the DN RCV ST 1-9 signals enables the multiplexer 1680 to couple the asserted SEL PAR/ROOT signal to the input terminal of flip-flop 1675, enabling it to maintain its set coDdition.
Similarly, if the parent node is transmitting NPAC nil packet messages to the control network node 51, when the DN RCV ST O signal is asserted an AND gate 1681 is energized, enabling an OR
gate 1682 to assert a TST ROOT UP ~est root up sigDal. The asserted TST ROOT UP signal enables a multiplexer 1683 to couple the ROOT UP LAT root up latch signal from root flag 1407 to the data irpu termir al of a flip-flop 1684. If the root flag 1407 is clear and the ROOT UP LAT signal is negated, indicating that the control network node ~1 is not a root node, the nip-nOp 1684 is reset at the next tick of the NODE CLK signal. On the other haDd, if the root nag is set and the ROOT UP LAT
signal is asserted, the nip-nOp 1684 will be set, asserting a ROOT DNLAT root down latched signal.
- On the other handj if the parent node is not traDsmitting NPAC nil packet messages to the control network node 51 when the DN RCV ST O signal is asserted, the OR gate 1682 will assert the TST ROOT UP signal in rseponse to the assertion of the DN RCV ST 12 signal, which is contemporaneous with P FLICK DN signals representing the last flick of the control network message packet 60 currently being received. At that point, the multiplexer 1683 will couple the ROOT LAT UP
signal to the data input terminal of fli~flop lo84. If the ROOT LAT UP signal is asseited, the flip-Qop 1684 uill be set at the next tick of the NODE CL~C sigllal to assert the ROOT DN LAT root town latched signal. Alternatively, if the ROOT UP LATsignalis negated, the flip-flop 1684 will be clea~ed to negate the ROOT DN LAT signal.
It will be appreciated that, after the DN RCV ST 12 signal is later negated, enabling the OR
gate 1683 to negate the TST ROOT UP signal, !he multiplexer 1683 couples the ROOT DN LAT to the data input tenninal of nip-nOp 1684 to ma ntain the flip-nop 1675 in its condition at successive ticks of the NODE CLEC sigllal. Effectively, if the parent is not transmit~ing N~AC nil packet messa8es, the multiplexer 1683 enables the flip-flop 1684 to bc conditioned in response to the ROOT UP LAT signal from root flag 1407 when the DN RCV ST 12 signaL which is contemporaneous SEL DN INP DATA
and SEL DN INP TAG signals representing the last flick of the control ne~ vosk message packet 60 being received by the down source selector 1670.
Regardless of the condition of ~he ROOT DN LAT signal, the OR gate 1677 is contemporaneously energized to assert the SEL ROOT SRC signal. Thus, the multiplexer 1683 couples the ROOT DN l~T si~al to the data input tenDinal of flip-ilop 1675. If the ROOT IJP LAT
signal is negated, indicating that the control net vork ~ode 51 is not a root node, the flip-Bop 1675 will be cleared to negate the SEL PAR/ROOr signal. In that condition, the down source selector 1670 couples the P FLICK DOWN (4:0) sigDals, or the PARKED P F~ICK signals from the park buffer 1671, as the SEL DN INP DATA (3:0) and SEL DN INP TAG signals. Alterrlatively, if the ROOT UP
LAT signal is asserted, indicating that the control net vork Dode 51 is a root node, the nip-flop 1675 will - ., : : ,: . ., . .. . : ~ : , -~, .. -. , . , , , ." ,, . ~ . ~'1 !

WO 92t06436 PCI/US91/07383 20933r' -199' oe set to assert~SEL PAR/ROOT signal. In that condition, the down source selec~or 1670 couples the BUF P FLICK DOWN (4:0) signals, or the PARKED P FLICK sigllals from the park buffer 1671, as the SEL DN INP DATA (3:0) and SEL DN INP TAG signals.
The park buffer select enable circuit 1673 generates a SEL PKD P~P select parked packet signal that, when asserted ellables the dowD source selector circuit 1670 to couple thc PARKED P
FLICK signals from the park buffer 1671 as the SEL DN INP DATA (3:0) and SEL DN INP TAG
signals. If, when the DN RCV ST 0 signal is asserted, the down packet type decoder is ass rting either the P/IDLE or the P/ABS signals, an OR gate 1690 is energizet to enable aD AND gate 1691. If the DN RCV ST 0 is concurrently asserted, the message packet message packet 60 being coupled by the down source selector 1670 is of the idle or abstain message type. If, in addition, a PKD DN PI~
parked down packet signal is being asserted by the park buffer 1671, indicating that the park buffer 1671 contains a parked message packet, the AND gate 1691 is energized to assert a SEL PK BUF
signal.
The asserted SEL-PK BUF signal enables a multiplexer 1692 to couple an asserted signal to the data input terminal of a fli~tlop 1693. The aip-flop 1693 is set at the next tick of the NODE CLK
signaL thereby asserting the SEL PICD PI~ select parked packet signal.
When the DN RCV ST 0 sigDal is negated, the AND gate 1691 is de-energized, to negate the SEL PK BUF signal. At that point, however, the DN RCV ST 1-9 signals, comprising tbe DN RCV ST
1 through DN RCV ST 9 signals are asserted, which comprise a HOLD PK BUF hold park buffer signal. The asserted HOlD PK BUF signal enables the multiplexer 1692 to couple the asserted SEL
PKD PKT select parked packet signal to the data input terminal of flip aOp 1693, to eDable the nip-nOp to maintain its state turing the successive ticks of the NODE CLK signal.
When the DN RCV ST 10 signal is later asserted, both the SEL P~C BUF and the HOLD PK
BUF signals will be negateed, enabling the multiplexer 1692 to couple a negated signal to the data input terminal of the flip-aop ~693. The flip-flop 1693 is reset at the next tick of the NODE CLK
signal, thereby negatiDg the SEL PKD PKT select parked packet signal. At that point, the down ssurce selector 1670 couples the P FLICK DN or BUF FLICK DN sigllals, as determined by the cs~ndition of the SEL PAR/ROOT select parent or root signaL as the SEL DN INP DATA (3:0) and SEL DN INP
TAG signals.
The down control circuit 1651 can also enable a control net~vork message packet 60 to be parked in the park buffer 1671, in a manner similar to the parking of a packet 60 in the park buffer 1442 in the left and right child receiver/buffers 1420(x). If, while the FLOW Cl'RL (DN) sigDals from the flick up control portion 1401 indicates that a child node is unable to receive additional multiple-source messages, the down source select arcuit 1650 receives a message packet 60 of the multiple-source message type, the down control circl~it 1651 asserts a PARK EN park enable signal that enables a multiplexer 1694 to couple SEL DN INP DATA (3:0) signals to data input termh~als of the park r buffer 1671. The park buffer 1671 latches the sigl~als at the succes~sive ticks of the NODE CLK signa to buffer the packet. After the packet 60 is buffered, the down control circuit 1651 negates the PARK

WO 92/06436 ~ ~ ~ 3 3 ~glu~g1/~73~3 200 ~
EN signal, which couples the output of the park buffer 1671 to its input terminals. Aftcr a messa~e packet 60 is buffered in the park buffer 1671, the down control circuit 1651 conditions the Fl OW
CrRL (U~') signals to, in turn, enable the ~lick up control portion 1401 to provide scan flow bits 72(i) (Flg. 5) to disable the parent noc',e from ,ransmitting packets ~ of the multiple source type thereto.
i ... . . .. . . . .

WO 92/06436 PCI/US9l/07383 .

2 t)~ s~ic Netwo~k -201-1. General Fig. 13A is a general block diagram of a diagnostic network node 100(h,p,t-l) used in the diaguostic network 16 described above, and Figs. 13B~1 through 13C comprise detailed block and logic diagrams of the diagnoslic network node 100(h,p,r-l). With reference to Fig. 13A, the diagnostic network node, which will be geDerally identified by refereDce numeral 100, includes an address token/data control portion 2000 and a test data control portion 2001. The address token/data control portion 2000 generally corresponds to the address control circuit 102 (Flg. 6A) and the test data control portion 2001 generally corresponds to the data control portion 103 (Fig. 6A).
The diagnosdc network node 100 receives PAR ADRS CrRL parent addsess control signals from a parent node, or from the diagnostic processor 101 (Fig. 6A) at one set of data input terminals of a multiplexer/demultiplexcr 2002. The multiplexer/demultiplexer 2002 irJcludes another set of data input terininals, which receive a corresponding set of DP ADRS CrRL diagnostic processor signals over a bus 2C03 from a local diagnostic processor (Dot shown). - The local diagnostic pri>cessor also generates a P SEL parent select signal, which controls the transfer of signals oetween buses 2003 or 104(P) and a bus 2004 connected to the address token/data control portion 200~. The local diagnostic processor may negate the P SEL signal to enable the multiplexer/demultiplexer 2002 to couple the address control signals between bus 104(P) and the bus 2004, to thereby enable the signals to be transferred between the parent node 100 or the diagnostic processor 101 and the address token/data control circuit 2000. Alternatively, the local diagnostic processor may assert the P SEL signal to enable the multiplexer/demultiplexer 2002 to couple the address control signals bet veen the bus 2003 and bus 2004 to thereby enable the signals to be transferred betweerl the local diagnostic processor and the address tol~en/data control circuit 2000.
The diagnostic net vork node 100 also receives PAR DATA parent test data signals over bus 110(P) from a pare~t node or from the diagnostic processor 101 ~Fig. 6A) at one set of data i~put termir,als of a multiplexer/demultiplexer 2005. The multiplexer/demultiplexer 2005 includes another set of data input terminals, which receive a corresponding set of DP DATA diagnostic processor data signals over a bus 2006 from a local diagnostic processor (not shown). The P SEL parent select signal also controls the transfer of signals bet veen buses 110(P) or 2006 and a bus 2007 colmected to the test .;.r data control portion 2001. The local diagnostic processor may negate the P SEL signal to enable the multiplexer/demultiplexer 200S to couple the test data signals bet veen bus 110(P) and the bus 2007, to thereby ellable the signals to be transferred between the parent node 100 or tbe diagnostic processor 101 a~d the test data coDtrol circuit 2001. Alternatively, the local diagnostic processor may assert the P
SEL signal to enable the multiplexer/demultiplexes 2005 to couple tbe test data signals between the bu~ 2006 and bus 2007 to thereby enable the signals to be transferred between the local diagnostic processor arld the test data control circuit 2U01.
It ~11 be appreciated that, if the node 100 comprises the root node 100(M,0,0...0) in the height decodillg tree (Figs. 6A through 6C), the diagnostic processor 101 may be connected to either buses ,~

WO 92t06436 PCl/US91/07383 2~933~
-202.
104(P) and 110(P), respectively, or to buses 2003 and 2006. If the diagnostic processor 101 i connected to buses 1Q4(P) and 110(P), it will maintain the P SEL parent select sigDal negated, and if it is connected to buses 2003 and 2006 it will maintaio the P SEL signal asserted. Alternatively, diagnostic processors may be connected both to buses 104(P) and 110(P), on the one hand, and to buses 2003 and 2Wo, on lhe other hand, and the P SEL signal may be controlled to enable coupling of signals between there and buses 2004 and 2007.
The address token/data control portion 2000 transmits signals to, and receives signals from, the various child diagnostic network nodes 100 connected thereto over buses 104(Cj). In the ennbodiment depicted in Figs. 13A through 13C, the diagnostic network node 100 may be connected to ~m~ child nodes, each over a separate bus 104(Cj). Similarly, the test data control portion 2001 transmits signals to, and receives signals from, ~m~ child diagnostic network nodes 100 connected thereto over buses 110(Cl), each over a separate bus 110(Cj).
As also sho vn in Fig. 13A, the address token/data control portion 2000 includes the flags 106(Cj). Each flag 106(Cj) controls an EN(i) enabling signal that controls transfer by the diagnostic ~ ~ ~
network node 100 over the buses 104(Ci) and llO(Cj) in tandem. In particular, the nag 106(Cj), when set, enables the address token/data control circuit 2000 to transmit and receive signals over the bus corresponding 104(Ci). ln addition, the nag 106(Cj), when set, enables asserdon of the corresponding EN(i) enabling signal that, in turn, enables the test data control portion 2001 to transmit and receive signals over the associated bus 110(Cj).
Before proceeding further, it would be helpful to describe the various signals transmi~ted over the buses 2002 and 2004. Bus 2002 comprises lines for carrying six signals, five of wbich, naisely, Gnes 2010 through 2014, are received by the address loken/data control portion 2000. The bus 2002 Dlcludes a sixth line 201~ for carrying a,signal generated by the address tokeD/data control portion 2000 up the tree defining diagnostic network lo to the pareDt node or to the dia~ostic processor connected to multiplexer/demultiplexer 2003, depending on the condition of the P SEL signal. Buses 1~4(Cj) connected between the address token/data control portion 2000 and child diaE~nostic network nodes have lines for carrying similar signals therebetween.
In particular, bus 2002 includes a line 2010 that carries an ACL~C (P) address clock from parent signal, which the diagnostic network node 100 uses as a clock signal to synchronize operations in the address token/data control portion 2000 in conDectlon with other signals comprising bus 2004. In addition, a line 2011 carries an AMS (P) address mode select from parent SigDaL which controls a control circuit in the address token/data coDtrol circuit 2000. The node transrnits the ACLK (P) signal and AMS (P) to all of its children.
A line 2013 carries an All (P) address token in from parent signal, and a line 2014 carries an ADI (P) address data in from parent signal, both of which cooperate to sequentialiy condition the 9ags 106(C;) in the diagnostic network node 100. The conditioDing of flags 106(Cj) in the address token/daîa control 2000 is controlled by a token, which is shifted 2hrough a shift register (described below in cormection with Fig. 13B-1) in the address token/data control portion 2000. The shift register ', '- .."' ' , :"- :, , ' '':; '' '' ' " " " ' , ' . ' , :
' ', .' ', . ' ' . ' '. '.,.. . ,, , ", "', "' ", ,' ' , ,: , '' . ' ~. . ' ' ' '-'' ' ' ', ' " ,;' " ~, ' .
' ''' , : ' '' " . "' ' ' ' ' ' "' "i .:'' ' ,' ' ' '''`: ' ' "~ ' :'. ' " ', ' ",, ' , ' ' ' ' ', . .

WO 92/0~436 PCI/US91/073~3 2 ~ g ~ ~ !
has a Du nber of stages, each corresponding to one of tbe flags 106(C;) in the diagnostic network node 100. When the token is in a stage in the shift regi ter associated with a particular flag 106(CK), if the ADI (P) signal is asserted, the flag is set at the next tick of the ACL~C (P) address clock signal. On the other hand, if the ADI (P) signal is negated, the flag 106(CK) is cleared.
The condition AMS (P) address mode select signal, along with the ticks of the ACLK (P) signal, controLs shifting of the token through the shift register. After the token h2s shifted through the shift register on the diagDostic ne~work node 100"t shifts out and is transmitted over the buses 104(Cj) associated with those of the flags 106(C;) that are set to the nodes conneaed thereto. The node also transmits the ADI (P) signal to all of its children. Accordingly, it will be appreciated that the conditioning of the flags 106(Cj) in each of the child nodes which recei~e the token will be accomplished in parallel, with the condi~ions of the flags lOo(Cj) in the respective child nodes being controlled in parallel by the condition of the Al)l (P) signal in response to the nex~ tick of the ACLK
addret.s clock signal.
Irl addition, after a. nag 106(Cj) is conditioned, the flag's state may be retrieved. RetAeval is enabled under control of the AMS (P) signal, and the state is represented by the condition of an ADO
(P) address data out to parent signal over a line 2015. If address token/data coDtrol circJit 2000 of node 100 receives ADO (Cj) address data out signals from one or more of its child nodes, associated w`ith set flags 106(C;), the address token/data control circuit 2000 may combine them under control of an EADb ~) expected address data out from parent signal on a line 2012 in bus 2004. If the ADO
(Cj) s gnals are expected to be asserted, the EADO (P) signal enables the address tokel/data control circuit 2000 to logically AND them together. In that case, if the ADO (C:;) signals from child nodes associated with the set flags 106(C;) are aD~ asserted, Ihe ADO (P) address data out to parent signal will be asserted, but if one of the ADO (C;) signals is negated the ADO (P) signal will be negated. On the other hand, if the ADO (C;) tignals from the child nodes are expected to be negated, the EADO (P) signal enables the address token/data control circoit 2000 to logically OR tbem together. In that case, if the ADO SC;) sigrlals from child nodes associated with tlle set flags 106(C;) are a~l negated, the ADO
(P) address data out to parent signal will be negated, bDt if one of the ADO (C;) signals is asserted the ADO (P) signal v.ill also be asserted.
Bus 2007 comprises lines for carrying five sigDa~, four of which, namely, lines 2020 through 2023, are received by the test data control porfion 2001. The bus 2007 includes a fifth line 20~4 for carrying a signal generated by the test data control portion 2007 up the tree defining diagnostic network 16 to the parent node or to the diagnostic processor conneaed to multiplexer/demultiplexer 2005, depending on the condition of the P SEL sigDaL Buses 104(C;) coDnected between the address token/data control portion 2000 and child diagnostic network nodes have lines for carrying similar signals therebe~ween.
As noted above, the interface between leaf nodes in the diaBnostic network 16 and each pod in one embodiment corresponds to the JTAG (nJoiDt Test Action Group~) interface, as described in IEEE Std. 1149.1 (hereinafter ~JTAG specification~) The JTAG interface comprises four signals .: . , . ~ , , :: . ..: . ~

.

WO 92/06436 PCT/l lS91 /07383 2a~3~
including a TCK test clock signal, a TMS test mode signal and a ll~I test data in signal, all of whicb are provided to the pod by the leaf node 100, and a TDO test data out signal provided by the pod to the leaf node 100. The use of the signals is defined in the aforementioned JTAG documentation.
Generally, the TCK sig~al operates as a clock signal, the TMS test mode select signal operates as a test control signaL and the TDI signal defmes test data. The TDO signal defines test results.
Wi~h this background, the test data controi portion 2001 in each node 100 reeeives a TCLK (P) test clock from parent si~al, a lMS (P) test mode select firom parent signal, and a TDI (P) test data i~
from parent signal on lines 2020, 2021 and 20~, respectively of bus 2007. The test data control portion 2001 couples these signals onto respective lines in those of buses 110(Cj) associated with the asserted EN (i) enable signals. As noted above, those of EN(i) eDable signals that are asserted corresponds to those of flags 106(C;) that are set. The signals are thus passed from respeetive parent node to respective child nodes down the paths defined by the set nagS 106(Cj) to the selected o~es of the pods.
ln addition, bus 2007 includes a line 2023 for carryirlg an ETDO (P) expected test data out from parent signal. The test data control portion 2001 couples this signal to child nodes along with the TDI (P) and other signals on lines 2020 through 2022. The bus 2007 also includes a line 2024 for carrying a TDO (P) test data out to parent signal, whose udlily Will be made clear in the following.
The leaf nodes 100 in the diagnostic network 16 provide the TCLK (Cj), TMS (Cj) and TDI
(Cj~ signals to the respective pods as the TCK, TMS and TDI signals, as called for by the aforementioned JTAG specification. In respo~lse, the pods provide a TDO test data out signal, which is also called for by the aforemendoned JTAG spccification. The ll)O signal is received by the test data control portion 2001 as a TDO (Cj) signal in thc pod's bus 110(Cj). The test data control portion 2001 of a leaf node 100 receives the TDO (Cj) signals from all of the pods associated with asserted EN(i) enable signals, and combines them as called for by the ETDO (P) expected test data out fro~
parent signal.
The test data control portion 2001 uses the ETDO (P) signal in comlection with the TDO (Cj) signals that are associated with asserted EN(i) signals ihl the same way the atdress token/data control portion 2000 uses the EADO (P) expected address data out signal in connection with the ADO (C;) signals asso~ated with set flags 106(Cj). If the test data control Qrcl~it 2001 recei~es ADO (C;) address data out sigDals from one or more of the pods or child nodes connected thereto that are associated with asserted EN(i) signals, the test data control circuit 2001 may combine them under control of the EI'DO (P) signal. If the TDO (Cj) signals are expected to be asserted, the EIDO (P) sigllal enables the test data control circuit 2001 to logically AND them together. ID that case, if the TDO (C;) signais from child nodes associated with the asserted EN(i) signals are aLi asserted, the Tr~O (P) test data out to parent signal willi be asserted, but if one of the TDO (C;) signals is negated the l'DO (P) signal will be negated. ()n the other hand, if the TDO (C;) signals from the chiid nocies are expected to be negated, the EI'DO (P) signal enables the test data control circuit 2001 to logically OR them together.
In that case, if the TDO (Ci) signals from chilid nodes associated with the asserted EN(i) signais are all negated, the TDO (P) test data out to parent siE~al will be negated, but if one of the l'DO (Cj) signaLs is asserted the TDO (P) si~al wiLI also be asserted.

- ~ . .:,: . . ............ ., . . :
.. ,~ , ~ , , ;:
. : .

Wl) 92/06436 PCr/US91/07383 20~33~ -205-2. Address TokeD/Data Control Portion 2000 Tbe address token/data control portion 2000 will be described in connection with Flgs. 13B-1 and 13B-2. With reference to Fig. 13B-1, tbe address token/data con~ol portion 2000 includes a flag register 106 and a token register ~030. Tbe flag register 106 comprises a set of nip-nOps each corresponding to one of flags 106(C;). The address tokerl/data control portion 2000 depicted in Ftg.
13B-1 includes ~m~ flags 106(C;), identified by reference numerals 1û6(Co) througlt 106(Cm l). Eacb flag 106(Cj) generates a corresponding EN(i) cnable signal.
The token register 2030 includes a like nurnber of stages ~030(1) througb 2030(m-1) [generally identified by reference numeral 2030(i)] connected to a lil~e uumber of multiplexers 2032(i) that together forrn a shift register. Each tol~en register stage 20~0(i), in tuM, cootrols a multiplexer 2031(i) tbat controls the source of signals provided to ~te ioput terminal of the corresponding flag 106(C;) in flag register 106. In particular, the multiplexers 2031(i) have one data input terminal that receives the ADI (P) address data in signal from line 2014, and a second data input terminal that receives ihe EN(i) erlable signal-output by the respective flag 106(C;). If the corresponding token register~stage 2030(i) is asserting a TR (i) token register signal, the multiplexer 2031(i) couples the ADI (P) signal to the data input terrninal of the flag 106(C;). The flag 10o(C;) latches the A~I (P) signal at the next tick of the ACLK (P) address clock from parent signal. On the other hand, if the corresponding stage of the tol~en register stage 2030(i) is not asserting tke TR (i) tol~en register signal, the multiplexer 2031(i) couples the EN (i) enable signal output by the flag 106(Cj) to the data input terminal of the flag, whicb latches it at the next tic~ of the ACLK (P) sig~al. Accordingly, the condition of the TR (i) signal from the respective token register stage 2030(i) determines whether state of the correspollding nag 106(C;) remains the same or whether it is controlled by the ADI (P) signal at the next tick of the ACLK ~P) 1-signal.
The multiplexers 2032(i) are coDtrolled by a SHII:T TOKEN signal ~om a control circuit 2033. The SHI~;T TOKEN signal enables a token, represented by a set token register stage 2030(i), resulting in an asserted TR (i) signal, to be shifted from the first token register stage 2030(0) to the last stage 2030(m-1), in response to the successive ticks of the ACLK signal. The rec ip~ of the token by the first tol~en rGgister stage 2030(0) is represented by the assertioD of the ATI (P) address tokeD in from parent signal when the control circuit 2033 asserts the SHIFT TOKEN signal. The ATI (P) s~l is coupled to one data input terminal of the ~uldplexer 2032(0) connected to the first stage 2030(0) of token register 2030. The second data input terminal of the multiplexer 2032(0) is connected to rec~ive the TR(0) token register signal output by the stage 2030(0).
If the control circuit 2033 is asserting Ihe SHIFT TOKEN signal, the multiplexer 2032(0) couples the ATI (P) signal to the input terminal of the token register stage 2030(0), which la~ches the signal at the next tick of the ACLK (0) signal. If the ATI (I~) signal is negated, the stage 2U3U(0) is cleared, which, in turn, enables the stage to negate the TR (0) signal. On the otber hand~ if ~he ATI
(P) signal is asserted, which indicates that the pareut diagnostic nelwork node or the diagnostic processor is transmitting the token to this node 100, the slage 2030(0) is set, which, in turn, enables the stage to assert the TR (0) signal.

WO 92/06436 PCr/US91/07383 2~9~3~

The series of multiplexers 2032(i) are controlled ;D ~mison by the SHIFT TOKEN signal.
Thus, if a token register stage 2030(i) is set, indicating that that stage 2030(i) has the token, if the control circuit 2033 is asserting the SHIFT TOKEN signal the multiplexer 2030(i+1) is enabled IO
couple the TR (i) signal to the input of its respective to~en register stage 2030(i + 1), where it is latched at the next tick of the ACLK (P) signal. If the token register stage 2030(i) is se~, asserting its TR (i) signal, the stage 2030(i + 1) u,ill be set to assert its TR (i + 1) signal. Similarly, if the token register stage 2030(i) is clear, negating its TR (i) sigrlal, the stage 2030(i+1) ~ill be clear. Thus, while the control circuit 2033 asserts the SHIFT TOKEN signal, the token register stages 2030(i) and multiplexers 2032(i) effectively shift the token at successive ticks of the ACLK (P) signal.
On the other hand, if the control circuit 2033 is negatiDg the SHIFT TO~CEN signal, each multiplexer 2032(i) is enabled to couple the signal at its other data input terminal, namely, the TR(i) signal output by its respective token register stage 2030(i), to the stage's input terminal. The stage 2030(i) latches the signal at the next tick of the ACLK (P) signal. Thus, the negated SHIFT TOKEN
signal eDables the token register stages 2030(i) to maintain their respective states. - - -The TR (m-l) output signal from the last token register stage 2030(m-1j in the token register 2030 is coupled to one data input terminal of a muldplexer 2037. The multiplexer 2037 co~trols the coupling of the TR (m-1) signal through those of gated drivers 2040(m-1) through 2040(0) lgenerally identified by reference numeral 2040(i)] to those of the child diagnostic network nodes whose EN (i) sigoals are asserted. Each gated driver 2040(i) provides the ATI (Cj) address token in to child signaL
which the respective child node receives as the ATI (P) address token in from parent signal on its line 2014. T~us, after the token has passed through the series of token register stages 2030(i) in diagl~ostic network node 100, it can be passed to the child nodes whose EN (i) sigDals are asserted.
The control circuit 2033 also controls several other operations in the address tokentdata control circuit 20û0. In particular, the control circuit generates a READ FR ST read flag register state signal which cor,trols a multiplexer 2034. The output terminal of multiplexer 2034 is connected to Ih~e ~015 and provides the ADO (P) address data out to parent signal. O~e data input terminal of multiplexer 2034 is provided by a series of multiplexers generally identified by reference numeral 2035(i). Specifically, each multiplexer 2035(i~ receives at one data input terminal the EN (i) signal from an associated flag 106(C;) in nag register 106, and at another data input terrninal the signal from the next multiplexer 2035(i+1). The second data input telminal of the last multiplexer 2û35(m-1) is pro~ided by 2n address data combining circuit 2036, which, as described below, receives ADO (C;) signals from child nodes for which corresponding EN (i) signals are asserted, and combines them according to logical operations as selected by the EADO (P) expected address data out signal from line 2012.
Iî a TR (ij signal from token register 2030 is asserted, the correspondiDg multiplexer 203S(ij is enabled to couple the EN (i) signal represendng the condidon of the associated nag 106(C;) to a data input termiDal of the next multiplexer 2035(i~ the series. If the TR (0) signal is asserted, the muldplexer 2035(0) couples the EN (0) signal to a data iDput terminal of the multiplexer 2034. On the ;, , , ,~, ,: . :. ; :: - .. :
: , : . . .. . :

WO 92/06436 PCI/U~9l/07383 ~ . ~ . . .
28~3~f -207 other han the TR (i) signal is negated, the corresponding multiplexer 2035(i) is cnabled to couple the signal from the next multiplexer 2035(i+1) to the second data input terminal of the next multiplexer 2035(i-1) in the series.
Thus, if the to~en register 2030 is asserting a TR (i) signal, the series of multiplexers 2035(i) couples the EN (i) enable signal to a data input termillal of the multiplexer 2034. If, however, noDe of the TR (i) signals is asserted on diagnostic register node 100, the series of multiplexers 2035(i) couples the signal from address data combining circuit 2036 to the same data input terminal of multiplexer 2034. In either case, if the READ FR ST read flag register state signal is asserted, the multiplexer 2034 couples that signal onto line 2015 as the ADO (P) signal. The second data input terminal of multiplexer 2034 is conneaed directly to the output terminal of the address data combining circuit 20~6. Accordingly, if the READ FR ST signal is negated, the multiplexer 2034 ~ ill couple the output signal from the address data combining circuit 2036 onto line 2015 as the ADO (P) signal.
The address data combining circuit 2036 recei es ADO (Cj) signals from ~he respective child diagnostic network nodes ~or which the EN (i) enable signals are asserted, combines tbem according to ~ ~ ~
a logical function identified by the EADO (P) expected address data out signal from line 2012, and .
provides the result to one data input ter~ninal of multiplexer 2035(m-1). The address data combining circuit includes two general sections, incJuding an ~N~ seaion 2W1 and an OR seaion 2042, along with a multiple~ser 2043 which is controlled by the EADO (P) signal. If the EADO (P) signal is negated, the multiplexer 2043 couples an EXP AD NEG expect negated output signal from the OR
section 2042 to the multiplexer 2035(m-1~, which Will be transmitted through the multiplexers 2035(i) and 2034 as the ADO (P) signal. If the OR section 2042 is negating the E~CP AD NEG signal, all of the ADO (Cj) signals, from the nodes for which the EN (i) signals are asserted, are negated.
Accordingly, the conditions of the ADO (C;) signals will correspond so the negated condition of the EADO (P) signal.
However, if one of the ADO (Cj) signals, from the nodes for which the EN (i) signals are asserted, is asserted, the OR section 2042 ulill assert the EXP AD NEG expect negated signal. The asserted signal will be coupled to the multiplexer 2035(m-1) and through the multiplexers 2035(i) and 2034 as the ADO (P) signal. In that case, the coodition of at least one of the ADO (C;) signals will differ f~om the negated condition of the EADO (P), indicating an error.
On the other hand, if the EADO (P) signal is asserted, the multiplexer 2043 couples an EXP
AD AST expect asserted output signal from the AND section 2041 to the multiplexer 2035(m-1), which will be transrnitted through the multiple~ers 2035(i) and 2034 as the ADO (P) signal. 1f the AND
section 2042 is asserting the EXP AIS AST signal, all of the ADO (Cj) signals. from the nodes for which the EN (i) sigDals are asserted, are asserted. Accordingly, the conditions of those ADO (Cj) signais will correspond to the asserted condition of Ihe EADO (P~ signal.
However, if one of the ADO (C~) signals, from the rlodes for which the EN (i) signals are asser~ed, is Degated, the AND section 2041 will Degate the EXP AD AST expect asserted signal. The negated sigr~al ill be coupled to the multiplexer 2035(m-1) and through the multiplexers 2035(i) and - .. : : ~ ., : .................................. . . .
- - , ~ . .,. ; . . .
.
. ; .' ;, ~ :

WO 92/06436 PCl'/US91/07383 2~9~3~
.208-2034 as the ADO (P) signal. ID thal case, the condition of at least one of the ADO (Cj) signals will differ from the asserted condition of the EADO (P)"ndicatirlg an error.
The AND section 2041 of address data combine circuit 2036 includes an AND gate 2043 which receives input sigDals from a series of OR gates, generally identified by reference nwneral 2045(i).
Each OR gate 2045(i) receives at one i~put terminal an ADO (Ci) address data out signal from a child diagnostic networlc node. At its other input terminal, the OR gatc 2045(i) rcceives the complemenl of the EN (i) signal, as generated by an inverter, generally identified by reference numeral 2046(i).
Accordingly, if an EN (i) sigr al is not asserted, the inverter 2046(i) energizes the OR gate 2045(i) to enable the correspondi~g input terrninal of AND gate 2044.
On the other hand, if the EN (i) signal is asserted, the illverter 2046(i) disables that input terminal of the respective OR gate 2045(i). Thus, the condition of the OR gate 2045(i) is controlled by the condition of the ADO (Cj) signal. If the ADO (C;) signal is asserted, the associated OR gate 2045(i) will be energized to energize the respective input terminal of the AND gate 2044. However, if -an ADO (C;) signal is negated, the associated OR gate 2045(i~ will be te-energized to, in turn,`disable the AND gate 2044. Thus, if all of the ADO (C;) signals from the child diagnostic ne~vork nodes, for which EN (i) signals are asserted, are asserted, the EXP AD AST expect asserted signal will be asserted. However, if one of the ADO (Cj) signals from the child diagnostic network nodes, for which EN (i) signals are asserted, is negated, the AND gate 2044 will be disabled and the EXP AD AST
expect asserted signal will be negated.
The OR section 2042 of address data combine circuit 2036 Lncludes an OR gate 2050 which receives input signals from a series of AND gates, generally identified by reference numeral 2051(i).
Each AND gate 2051(i) receives at one input terminal an ADO (C;) address data out signal from a child diagnostic network node. At its other input terminal, the AND gate 2051(i) receives,the EN (i) signal from the nags 106(Cj) of flag register 106. Accordingly, if an EN (i) signal is asserted, the corresponding input terminal of AND gate 2051(i) is enabled. On the other hand, if the EN ti) signal is negated, the AND gate 2051(i) is disabled.
Thus, the condition of the AND gates 2051(i) enabled by the asser~ed EN (i) SiBDalS ;S
controlled by the condition of the ADO (Cj) signal. If the ADO (Cj) signal is negated, the associated AND gate 2051(i) ~ill be de-energized to disable the respective input terminal of the OR gate 2050. If all ADO (Cj) signals, for which EN (i) sig~als are asserted, are negated, the OR gate 2050 will be de-energized to negate the EXP AD NEG expect negated signal. However, if an ADO (Cj) signal, for which an EN (i) signal is asser~ed, is negated, the associated AND gate ~051(i) will be energized to, in turn, energi~e the OR gate 2050 and a~ssert the EXP AD NEG signal. Thus, if all of the ADO (C;) signals from the child diagnostic network nodes, for which EN (i) signals are asserted, are negated, the EXP AD NEG expect ~egated signal will be negated. However, if one of the AI~O (Cj) signals from the chlld diagnos~ic network nodes, for which EN (i) signals are asser~ed, is asserted, the OR gate 2050 will be energized and the EXP AD NEG expect NEGATED signal will be asserted.
As noted abovc, multiplexer 2037 controls the coupling of the TR (m-l) signal from token register stage 2030(m-1) to an input terminai of each of gated drivers 2040(i), and those of the EN (i) " , :,, , . . ., :: :: .: : ~.
- : ;. . ,.:.:, ,: : .: , :.

: , . ~............. .. ...

2t06436 PCr/US91/07383 2 0 9 3 3 ~ 209-signals that are asserted enables their respective gated drivers 2040(i) to, in turn, couple the TR (m-1) signal to their respective the ch;ld diagnostic network nodes. The TR (m-1) signal is coupled to one data input terminal of the multiplexer 2037. The multiplexer's other data input terminal is connected to line 2014 to receive the ATI (P) signal. The muJtiple%er 2037 is controUed by the READ FR ST
read flag register state signal from the control circuit 2033. If the READ FR ST sy~nal is asserted, the multiplexer 2037 couples the TR (m-1) sigDal to tbe input terminals of gated drivers 2040(i) and if the READ FR ST signal is negated the multiplexer 2037 couples the ATI (P) signal thereto.
The diagnostic network node 100 also includes several drivers 2052 through 2055 for transmitting several to all of its child nodes. In particular, drivers 2052 through 2055 transmit, respectiYely, the ACLK (P) signal from line 2010, the AMS (P) signal from line 2011, the EADO (P) from line 2012 and the ADI (P) from line 2014, to all of its child Dodes as the ACLK (Cj), AMS (Cj), EADO (C;) and ADI (C;) signals. The diagDoslic network node 100 effectively broadcasts the signals, without being gated or coDtrolled by the EN (i) enable signal, to all of its child nodes. r The control circuit 2033 also provides several'addition'al signals for coDIrolling the operations of the circuitry depicted on Fig. 13B-1. A RESET TO~CEN REG signal enables all of the token register stages 2030(i) of the token register 2030 to be cleared, or reset, to a predetermined state.
When the stages 2030(i) are reset, all of the TR (i~ token register signals are negated. In addition, a RESET I;l.AG REG signal enables all of the flags 106tCi) to be conditioned to a known state. In one particular erabodiment, the flag 106(Co) is conditioned to a set state, and the other tlags 106(CI) through 106(C", I) are cleared.
In that case, the dia8nostic processor can determine the coDfiguration of the diagnostic network Dodes 100(h,p,r-l) in the diagnostic network by, after enabling the control circlaits 2033 to assert the RESET FLAG REG signal, to iteratively retrieve the states of the respective llags 106(C;) in the various nodes. In that operation, tbe diagnostic processor 101 caD control sequeneing of a token down the diagDostic network, and if the ADO (P) signal is asserted the diagnostic processor can determine from that that the location of the token in a token register 2030 identifes the first flag 106(Co) in a node. As the token is sequenced througb the token register 2030, the ADO (P) s~al will be negated. The token will then be transmi~ted to the child node connected to the bus 104(C1) and when it is received in the first token stage 2030(0) in the token register 2030 therein, the ADI (P) will again be asserted. Thus, the diagnostic processor can determine the number of slages in the flag register 106 by determir~ing the number of steps required in tbe sequence between assertions of tbe ADO (P) signal.
As noted above, the control circuit 2033 generates the SHIFT TOKEN, READ FR ST, RESET TOKEN REG and R~ET FLAG REG signals to control the other circuit elements depicted OD Fg. 13B-1. In one embodiment, tne control circuit 2033 is a state machine controlled by the AMS
(P) address mode select from parent signal and the ACLK (P) address clock signal. For each state, the condition of the AMS (P) signal detennines a target state for the control circuit 2033, and the ticks of the ACLK (P) signal determine the timing of the state transition. It will be appreciated that, since the .: ~, , , :
- : . .. ~ , :, .,., . . ' , , . ;;
,: . :~

WO 92/06436 PCr/US91/07383 2 ~ ,3 rj AMS (P) and ACLK (P) signals are transmitted to, and received by, all diagnostic nehvork nodes100(h,p,r-l) in parallel, the control circuits 2033 in all of the nodes will be controlled in parallel and v.~ill be in the sarne state at the same time. The various states and state transitions, and the conditions of the signals Benerated by a control circuit 2033 in each state, a e depicted in F~g. 13B-2.
With reference to Fig. 13B-2, the control circuit is initially in a reset state, as represented by the box of the sarne label. In that state, as shown in the Signal Condition/State Table on Fig. L3B-2, the control circuit 2033 asserts the RESET TOKEN REG and RESET FLAG REG signals to reset the token register 2030 and aag register 106 as described above. While the diagnostic processor 101 maintains the AMS (P) signal asserted, the control circuut ~033 remains in the reset state.
!f the diagnostic processor 101 negates the AMS (P) signal while the control circuit 2033 is in the reset state, the control circuit sequences to a ~clear token~ state, as represented by the box of the same label. As noted above, the state transition occurs at the next tick of the ACLK (P) signal after negation of the AMS (P) signal. In the clear token state, the control circuit 2033 asserts the RESET
TO~N REG signal, to reset the stages 2030(i) of the token register 2030, and maintains the other signals negated. If the diagnostic processor re-asserts the AMS (P) signal, at the next tick of the ACLK
(P) sigrlal the control circuit 2033 returns to the reset s~ate. Other vise, if the diagnostic processor maintains the AMS (P) signal negated at the next tick of the ACLK (P) signal, the control circuit 2033 sequences to a s~ift token s~ate. The control circuit's shift token state is represented by a box on Fig.
13B-2 of the same name. If the diagnostic processor 101 thereafter maintains the AMS (P) signal negated at successive ticks of the ACLK (P) signaL the control circuit 2033 remains in the shift token state.
In the shift token state, the control circuit 2033 asserts the SHIFT TOKEN signal and the READ FR ST read nag register state signal. As noted above, while the SHIFT TOKEN signal is asserted, each multiplexer 2032(i+ 1) couples the TR (i) signal from the preceding token reg~ster stage 2030(i) to be latched in its stage 2030(i~ 1) at successive ticks of the ACLK (P) signal. Thus, if a token register stage 2030(i) is in a condition indicating that it has a token, or if stage 2030(0) rcceives the tol;en from the parent node or the diagnostic processor 101, while the SHIFT TOKEN signal is asserted at successive ticks of the ACLK (P) signal the t.oken shifts through the succeeding stages 2030(i) and out the gated drivers 2040(i) associated with asserted EN (0) enable signals. In addition, the READ FR ST signal enables the multiplexer 2034 to couple the signal from the multiplexes series 2035(i) as the ADO (P) address data out to parent signal to the parent diagnostic neh~ork node.
OQ the other hand, if, while the control circuit 2033 is in the shift token sta~e, the diagnostic processor 101 asse;ts the AMS (P) signal, the control circuit 2033 sequences to a read nag register state at the nexî tick of the ACLK (P) address clock from parent signal. In the read flag register state the control circuit asserts only the EIEAD FR ST read flag register state signal, which, as noted above, enables the multiplexer 2034 to couple the signal from the multiplexer series 2035(i) as the ADO (P) address data out to parent signal to the parent diagnostic net vork Dode. If the diagnostic psocessor 101 mairltains the AMS (P) signal in an asserted condition, the control circuit 2033 returns to thc clear t ' . :' ' ~ , : . ;: ', , !

'", ' ' ': '.' :' . "' ' '', :' ':'. . ': ., '' ' ' ' .,, ,, " , ',.' ;~, '", ' :

WO 92/06436 PCr/US91/07383 2~3~5~ -211-token state at the next tick of the ACLK (P) signal. On the other hand, if the diagnostic processor 101 negates the AMS (P) signal while the control circuit 2033 is in the read flag regi$ter gtate, the control circuit 2033 returns to the shift token state.
3. Test Data Control Porlion 2001 The test data control portion 2001 will be described in connection with Fig. 13C. With reference to Fig. 13C, the test data control portion includes three general sections. One section transmits several signals received from the parent diagnostic network node, or the diagnostic proces$or 101, direc~ly to the various child diagnostic nodes connected thereto. In particular, the test data control portio~ 2001 receives the TDI (P) test data in from parent signal on line 2022 and traosmits it to all of the child diagnostic network nodes in parallel through drivers 2060(0) through 2060(m~ generally identified by reference nulneral 2060(i)] to all of the child nodes connected to the respective buses 110(C;). In addition, the test data control portion 2001 receives the ETDO (P) expected test data out signal on line 2023 and couples transmits it to ali of the child diagnostic network nodes in parallel through drivers 2061(0) through 2061(m~ generally idedified by reference numeral 2061(i)] to ail of the child nodes connected to the respective buses 110(C;).
A second section gates several other signals received from the parent diagnostic network node, or the diagnostie processor 101, to those child nodes whose EN (i) enable signals are asserted. In particular, the test data coDtrol portion 2001 receives the TCLK (P) test clock from parent signal on line 2020 and transmits it through those of gated drivers 2062(0) through 20O2~m-1) [generally identified by reference numeral 2062(i)] associated with the asserted EN (i) signals. Similarly, the test data control portion 2001 receives the TMS (P) test mode select from parent signal OD line 2020 and transmits it through those of gated drivers 2063(0) through 2063(m-1) Ipnerally identified by reference numeral 2063(i)] associated with the asserted EN (i) signals.
Fmaliy, the test data control portion includes a test data combining circuit 2064 which receives TDO (C;) signals from child nodes for which correspooding EN (i) sig~ als are asserted, and combines them according to logical ope~ations as selected by the ETDO (P) expected test data out sigoal from iine 2023. The structure and operation of the test data combining circuit 2064 is geDerally similar to the address data combining circuit 2036 described above.
The test data combining circuit 2064 receives TDO (C;) signals from the respective child diagnostic network nodes for which the EN (i) enable signals are asserted, combirles them according to a logical function identified by the ETDO (P) expected test data out signal from iine 2023, and provides the result to one data input terminal of multiplexer 2035(m-1). The tes~ data combining circui~
i~cludes t vo general sections, including an AND section 2071 and an OR section 2072, aiong with a muitiplexer 2073 which is controlled by the ETDO (P) signal. If the ETDO (P) signai is negated, the muitipiexer 2073 couples an EXP ID NEG expect negated output signal from the OR section 2072 as a COMB TD ou'r combined test data out signal ~o one input terminai of a multiplexer 2082. If the OR
section 2072 is Degating the EXP TD NEG signai, all of the TDO (Cj) siEDals, from the nodes for which the EN (i) signals are asserted, are negated. AccordiDgiy, ~he conditions of the TDO (C;) siF,Dais will corresp~nd to the negated co~dition of the Ell)O (P) siglal.

- . , .~ ~ .-:, . .
, . . .
.... . . .

WO 92/06436 PCr/US91/07383 2~933~;3 However, if one of the TDO (Cj) signals, from the nodes for whicb the EN (i) signals are asserted, is asserted, the OR section 2072 will assest the E7CP TD NEG expe~t negated si~al. The asserted signal will be coupled to the multiplexer 2035(m-1) and through the multiplexers 2035(i) and 2034 as the TDO (P) signal. In that case, the condition of at least one of the TDO (C;) signals will differ from the negated condition of the ETDO (P), indicating an error.
On the other hand, if the ETDO (P) signal is asserted, the multiplexer 2073 couples an 2XP
TD AST expect asserted output signal from the AND section 2071 to the multiplexer 2035(m-1), which will be transmitted through the multiplexers 2035(i) and 2034 as the TDO (P) signal. If the AND
section 2072 is asserting the EXP TD AST signal, all of the TDO (C;) signals, from the nodes for which the EN (i) signals are asserted, are asserted. Accordingly, the conditions of those TDO (C;) signals will correspond to the asserted condition of the FTDO (P) sigDal.
However, if oDe of the TDO (C;) signals, from the nodes for which the EN (i) signals are asserted, is negated, the AND section 2071 will negate the EXP TD AST expect asserted signal. The negated sigDal will be coupled to the multiplexer 2035(m-1) and through the multiplexers 2035(i) and 2034 as the TDO (P) signal. In that case, the condition of at least one of the TDO (C;) signals will differ from the asserted condition of the ETD(:~ (P), indica~ing an error.
The AND section 2071 of test data combine circuit 2064 includes an AND gate 2073 which receives input signals from a series of OR gates, generally identified by reference nwneral 2075(i).
Each OR gate 2075(i) receives at one irlput tenninal an TDO (C;) test data out signal from a child diagnnstic network node. At its otber input terrDinal, the OR gate 2075(i) receives the complemeat of the EN (i) signal, as generated by an inverter, generally identified by reference numeral 2076(i~.
Accordingly, if an EN (i) signal is not asserted, the inverter 2076(i) energ~es ~he ()R gate 2075(i) to enable the corresponding input terminal of AND gate 2074.
On the other hand, if the EN (i) signal is asserted, the inverter 2076(i) disables that input terminal of the respective OR gate 2075(i). Thus, the condition of the OR gate 2075(i) is controlled by the condition of the TDO (C;) sigDal. If ~he TDO (C;) sigDal is asserted, the associated OR gate 2075(i) will be ellerg~ed to energize the respective input terminal of the AND gate 2074. However, if an TDO (C;) signal is negated, the associated OR gale 2075(i) will be de-energi~ed to, in turn, tisable the AND gate 2074. Thus, if all of the TDO (C;) signals from the child diagnostic network nodes, for which EN (i) signals are asserted, are asserted, the EXP TD AST expect asserted signal will be asserted. However, if one of the TOO (C;) sig~als from the child diagnostic net vorl~ nodes, for which EN (i) signals are asserted, is negated, the AND gate 2074 will be disabled and the EXP TD AST
expect asserted signal will be negated.
The OR section 2072 of test data combine circuit 2064 includes an OR gate 2080 which receives input signals from a series of AND gates, generally identified by reference numeral 2081(i).
Each AND gate 2081(i) receives at one input lerrninal an TDO (C;) test data out signal from a child diagnostic net~,vork node. At its otber input terminal, the AI~ ) gate 2081(i) receives the EN (i) si~al from the flags 106(Cj) of flag register 106. Accordingly, if an ~N (i) sigDal is asserted, lhe . . :. :: :.: . . .::

WO 92/06436 PCr/US91/07383 2~33~3 -213 corresponding iuput terrminal of AND gate 2081(i) is enabled. On the other hand, if the EN (i) signal is negated, the AND gate 2081(i) is disabled.
Thus, the condition of the AND gates 2081(i) enabled by the asserted EN (i) signals is controlled by the condition of the TDO (Cj) signal. If the TDO (Ci) signal is negated, the associated AND gate 2081(i) wiD be de-energized to disable the respective input terminal of the OR gate 2080. If all TDO (C;) signals, for which EN (i) signals are asserted, are negated, the OR gate 2080 will be de-energi~ed to negate the EXP TD NEG expect negated signal. However, if an TDO (Cj) signal, for which an EN (i) signal is asserted, is negated, the associated AND gate 2081(i) wiD be energized to, in turn, energize the OR gate 2080 and assert the EXP TD NEG signal. Thus, if aD of the TDO (Cj) signals from the child diagnostic network nodes, for which EN (i) signals are asserted, are negated, the EXP TD NEG e~pect negated signal uiD be r~egated. However, if one of the TDO (C;) signials from the child dlagnostic nenvork nodes, for which EN (i) signals are asserted, is asserted, the OR gate 2080 wiD be energized and the EXP TD NEG expect negated signal will be asserted.
The multiplexer 2082 deterraines the source of signals coupled onto line 2024 as the TDO (F) test data out to parent si~al. An AND gate 2083, controlled by the complements of the EN (i) signals as generated by inverters 2084(0) through 2084(m-1) [generally identified by reference numeral 2084(i)~, asserts a NONEENnone enabled signal if the flags 106(C;) are not iasserting any of tne EN
(i) enable signa~Is. If the NONEENsignal ~s negated, indicating that at least one EN~i) enable sigDal is asserted, the multiplexer 2082 couples the COMB TD oUr sigaal onto line 2024 as the TDO (P) test data out signal. On the other hand, if the NONE EN signal is asserted, the multiplexer 2082 couples the TDI (P) signal received on line 2022 onto line 2024 as the TDO (P) signal.
As noted above, the test data combine portion 2064, particularly the AND sedion 2071 and the OR section 2072, along vith multiplexer 2073, is similar to the address data combine portion 2036 of the address token/data control portion 2000. In addition, it will be recognized that the address data combi~e portion 2036 and the test data combine portion 2064 wiD be used at different points in time.
That is, the address data combine portion 2036 will be used while the flags 106(C;) are being conditioned, and the test data combine portion wiD oe used thereafter. Accordingly, in one specific embodiment, the same circuitry is used for both elements.
The foregoing description has been limited to a specific embodiment of ~his invention. It will be apparent, however, that variations and modifications may be made to ~he invention, with the attainrnent of some or all of the advantages of the inveDtion. Therefore, it is the object of the appeDded claims to cover all such variations and modifications as come within the true spir~t and scope of the invendon.

..
- : , , . - , , , :
:. .:. . , ~ :.:
. : . . . . ~ ,:
' : -,: : -, .~' . '' "' , . :

Claims (360)

1. A digital computer comprising:
A. a plurality of processing elements each performing data processing and data communications operations in connection with commands, said processing elements also performing diagnostic operations in response to diagnostic operation requests and providing diagnostic results in response thereto;
B. at least one command processor for generating commands for said processing elements, said command processor also performing diagnostic operations in response to diagnostic operation requests and providing diagnostic results in respnse thereto;
C. at least one diagnostic processor for generating diagnostic requests;
D. a communicaton network comprising:
i. a data router connected to said processing elements for facilitating the transfer of data during a data communications operation;
ii. a control network connected to said processing elements and said command processor for transferring commands from said command processor to said processing elements; and iii. a diagnostic network connected to said processing elements, said command processor and said diagnostic processor fro transferring diagnostic requests from said diagnostic processor to said processing elements and said command processor and for transferring diagnostic results from said processing elements and said command processor to said diagnostic processor.

2. A digital computer comprising:
A. a plurality of processing elements each performing data processing operations in connection with commands, each processing element also generating and receiving data transfer messages, each including an address portion containing an address, for transfer to another processing elements as identified by the address and further generating and receiving control network messages;
B. at least one scalar processor for generating control network messages conrtaining commands for processing by said processin elements;
C. a control network comprising a plurality of control network nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper root level, control network nodes in the leaf level being connected to said processing elements and said scalar processor, i. said control network nodes below the root level, upon receiving a control network message containing a command from the scalar processor or a lower-level control network node, generating a control network messge for transmission to a higher-level control network node, and ii. the root control network node, upon receiving a control network message containing a command from a lower-levle control network node, and the lower-level control network nodes, upon receiving a control network message containing a command from a higher-level control network node, generating control network messages containing the command for transmission to lower-level control network nodes and the processing elements, thereby to transmit messages containing a command to all of the processing elements; and D. a communications router comprising a like plurality of router nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper root level, router nodes in the leaf level being connected to said processing elements, the nodes receiving data transfer messages and coupling them to another node or to a processing element connected thereto as determined by the address in the respective address portion.

3. A digital computer comprising:
A. a plurality of processing elements each performing data processing operations in connection with commands, each processing element also generating and receiving data transfer messages, each including an address portion containing an address, for transferto another processing elements as identified by the address and further generating and receiving control network messages;
B. a communications router comprising a plurality of router node groups interconnected in a tree pattern in a series of levels from a lower leaf level to an upper root level, each node group in the leaf level having one router node connected to a processing element, and each node group in levels above the leaf level including a plurality of router nodes, with router nodes in levels below the root level being connected to a plurality of router nodes in the next higher level thereby forming a fat-tree structure, each node receiving data transfer messages and coupling them to another node or to a processing element connected thereto as determined by the address in the respective address portion;
and C. a control network comprising a like plurality of control network node groups interconnected in a like tree pattern in a series of levels from a lower leaf level to an upper physical root level, each control network node group below the upper root level receiving control network messages from a processing element or a lower-level control network node group and generating a control network message in response thereto for transmission to a higher-level control network node group, and receiving control network messages from a higher-level control network node group and generating control network messages in response thereto for transmission to lower-level control network node groups, the control network node group at the root level generating control network messages for transmission to the lower level control network node groups in response to control network messages received therefrom.

4. A digital computer comprising:
A. a plurality of message generating elements for generating messages, each message having an address portion including a level identifier portion containing a level identifier value and a destination identifier portion containing a destination path identifier value;
B. a routing network comprising a plurality of router nodes interconnected in a tree pattern in a series of levels from a leaf level to a root level, each router node node below the root level being connected to a parent node at a higher level and each node above the leaf level being connected to a child node at a lower level and at least some of said nodes being connected to a plurality of child nodes to thereby define a tree pattern, the nodes at the leaf level being connected to receive messages from and transmit messages to the message generating elements, each node transmitting a message to its parent node if the level identifier value identifies a level higher than that of the node or alternatively transmitting said message to a particular child node as identified by said destination identifier value.

5. A computer as defined in claim 4 in which each of said message generating elements is identified by a message generating element identifier identifying the relative position of its connection to the routing network, each message generating element includes an address generator for generating said level identifier value and said destination path identifier value, said message generating element comprising:
A. an absolute address generator responsive to generation of a message for generating an absolute address value corresponding to the message generating element identifier for a message generating element to receive the message;
B. a height value generating portion for generating a level identifier value in response to the absolute address value generated by the absolute address generator and the message generating element identifier of the message generating element that is generating the message;
C. an address combiner for generating said address portion in response to the level identifier value generated by the height value generating portion and said absolute address generated by said absolute address generator.

6. A computer as defined in claim 5 in which said height value generating portion includes:
A. a relative address generating portion for generating a relative address in response to the absolute address value generted by the absolute address generator and the message generating element identifier of the message generating element that is generating the message, the relative address identifying a displacement over the routing network from the message generating element generating a message to the message generating element to receive the message;
B. a height identifier for generating a height value in response to the relative address generated by the relative address generating portion.

7. A computer as defined in claim 6 in which said absolute address value and the message generating element identifier are both represented by a word comprising a sequence of bits, said relative address generating portion performing a bit-wise exclusive-OR operation in connection with said absolute address value and the message generating element identifier to generate said relative address.

8. A computer as defined in claim 6 in which said height identifer comprises:
A. a potential height identifier circuit for generating in response to the relative address generated by said relative address generating portion a series of height signals each associated with a particular level in the routing network, said potential height identifier circuit asserting selected ones of said height signals as determined by the relative address; and B. a decoder for selectively generating, in response to the height signals from the potential height identifier circuit, one of a plurality of height identifier values, each of said plurality being associated with a particular level in the routing network, the height identifier value generated being associated with the highest level whose associated height signal is asserted.

9. A computer as defined in claim 8 in which the height identifier values generated by said decoder are defined by binary-encoded signals.

10. A computer as defined in claim 8 in which said absolute address value is defined by a binary-encoded absolute address signal having a predetermined number of bits, said down path generating portion generates said down path identifier value as a binary-encoded signal comprising a selected range of low-order ones of bits comprising said absolute address signal.

11. A computer as defined in claim 8 in which said address generator generates an address word comprising said level identifier value in a high-order portion and said destination path identifier value in a low-order portion, A. said height value generating portion further includes a height location select circuit responsive to the relative address generated by said relative address generating portion for generating a level portion identifier to identify the portion of said address word to contain said level identifier value, B. said address combiner coupling (i) said level identifier value in a portion of said address word identified by said level portion identifier and (ii) a selected low-order portion of the absolute address generated by said absolute address generator in the low-order portion of said address word below the portion occupied by said level identifier value thereby to establish said address word.

12. A computer as defined in claim 11 in which said address word generated by said address generator comprises a series of predetermined address word sections each associated with a predetermined group of at least one level of said router network, A. said height value generating portion identifying in response to said relative address one of said groups associated with the level identified by the level identifier value and generating a section identifier signal associated with the identified group, B. said address combiner comprising a plurality of coupling circuits each associated with one of said address word sections for selectively coupling in response to the section identifier signal generated by said height value generating portion either the level identifier value or a low-order portion of said absolute address as the address word.

13. A computer as defined in claim 12 in which:
A. said height value generating portion includes:
i. a level group identifier circuit for receiving said relative address generated by said relative address generating portion a series of section enabling signals each associated with a particular section in the routing network;
ii. a group decoder for receiving said section enabling signals and asserting one section identifier signal corresponding to the asserted section enabling signals associated with the highest-level group;
B. said address combiner includes a plurality of coupling circuits each associated with one of said address word sections for coupling either the level identifier value or a selection portion of said absolute address portion as the associated address word section in response to the section identifier signal associated therewith.

14. A computer as defined in claim 4 in which each node comprises:
A. a plurality of output circuits, each for transmitting messages to the parent node or a child node connected to said node;
B. a plurality of input circuits, each for receiving messages from the parent node or a child node connected thereto and for generating an output circuit identifier in response to the level identifier value and destination path identifier value, the input circuit generating an output circuit identifier identifying an output circuit for transmitting a message to the parent node if the level identifier value identifies a level higher than that of the node or alternatively an output circuit for transmitting a message to a particular child node as identified by said destination identifier value, and C. a switch for selectively coupling messages from said input circuits to said output circuits in response to the output circuit identifiers as generated by the input circuits.

15. A computer as defined in claim 14 in which each input circuit includes:
A. a message input buffer for buffering message information;
B. an output request buffer for buffering output circuit identifiers;
C. an input message control circuit for receiving messages and generating, in response to each received message, message information for buffering in said message input buffer and an associated output circuit identifier for buffering in said output request buffer; and D. a switch input control circuit for obtaining each output circuit identifier from said output request buffer and coupling them to said switch and for selectively coupling buffered message information associated therewith from said message input buffer to said switch

16. A computer as defined in claim 15 in which each message comprises a series of words, one of said words comprising said address portion and the remaining words comprising a data portion, said message input control circuit including:
A. an output request identifier processing circuit for processing said address portion to generate said output circuit identifier in response to the address portion of a received message; and B. a message processing circuit for generating said message information in response to a received message.

17. A computer as defined in claim 16 in which each node receives a router height identifier and the address portion comprises a level identifier section containing a level identifier and a series of predetermined address word sections each associated with a predetermined group of at least one level of said router network, each address word section identifying for each level of the associated group a child node, said output request identifier processing circuit including:
A. a parent request circuit for comparing the router height identifier and the level identifier from the level identifier section of an address portion and for generating a parent request signal if the value of the level identifier is larger than the router height identifier;

B. a child request circuit for generating child request signal identifying one of said child nodes connected thereto in response to the contents of a selected address word section; and C. an output circuit identifier generating circuit for generating said output circuit identifier in response to the parent request signal and the child request signal.

18. A computer as defined in claim 17 in which said parent request circuit includes:
A. a child input port identifier for generating a child input port signal having a plurality of conditions each indicating whether the input circuit receives messages from a child or a parent;
B. a level comparator for generating a signal if the router height identifier is less than and the level identifier from the level identifier section of the address portion of a received message;
C. a parent request signal generator for generating the parent request signal in response to the coincidence of the child input port signal from the child input port identifier the signal from the level comparator.

19. A computer as defined in claim 18 in which each node of said routing network operates in a first mode in response to a control signal having a first condition to couple received messages to a node connected thereto in accordance with the address portion, each node alternatively operating in a second mode in response to said control signal having a second condition to couple received messages to predetermined ones of the nodes connected thereto, the predetermined ones of said nodes being selected to facilitate transfer of a message to a nearby message generating element to facilitate the rapid emptying of the routing network of messages, said parent request signal generator further being inhibited from generating the parent request signal in response to said control signal having said second condition.

20. A computer as defined in claim 17 in which each address work section includes a series of down path identifiers each identifying a child identifier for a level of said routing network, said child request circuit including a down path identifier selector for selecting one of said down path identifiers and coupling the selected down path identifier as said child request signal.

21. A computer as defined in claim 20 in which each node of said routing network operates in a first mode in response to a control signal having a first condition to couple received messages to a node connected thereto in accordance with the address portion, each node alternatively operating in a second mode in response to said control signal having a second condition to couple received messages to predetermined ones of the child nodes connected thereto, the predetermined ones of said nodes being selected to facilitate transfer of a message to a nearby message generating element to facilitate the rapid emptying of the routing network of messages, the child request circuit of each input circuit generating further including:
A. predetermined child identifier generating circuit for generating a predetermined child identifier;
B. a selector circuit connected to said down path identifier selector for selectively coupling either the selected down path identifier or said predetermined child identifier as said child request signal in response to said control signal having said first or second condition, respectively.

22. A computer as defined in claim 17 in which said output request identifier processing circuit further includes a buffer control circuit for generating a output request buffer enabling signal to enable the output request buffer to buffer the output circuit identifier.

23. A computer as defined in claim 16 in which each message further includes, following said address portion, a message data portion, said message processing circuit includes:
A. an address word processor for processing said address word in response to a router height identifier identifying the level of said node in said routing network to generate processed address word information; and B. a message information processor for processing said message to generate message delimitation information to delimiting a received message, and for coupling the processed address word information, data portion and said delimitation information to said message input buffer as said message information.

24. A computer as defined in claim 23 in which each node has an associated router height identifier identifying a level of said routing network, said address word processor including:
A. a level comparator for generating a level comparison signal if the the router height identifier corresponds to the level identifier from the level identifier section of the address portion of a received message;
B. a router height identifier decrementation circuit for generating a decremented router height identifier;
C. a height identifier selector for selectively coupling in response to said level comparison signal one of the router height identifier or the decremented router height identifier to the message input buffer as the processed address word.

25. A computer as defined in claim 24 in which said address word comprises a series of address word sections each associated with a predetermined series of at least one level of said router network and defining a series of down path identifiers path through said series, said address word processor further including:
a. an address word section coupling circuit for selectively coupling address word sections to said message input buffer as said processed address word; and B. an address word section coupling control circuit for selectively enabling said address word section coupling circuit to couple address word sections, said address word section coupling control circuit disabling said address word section coupling circuit from coupling an address word section in response to the coincidence of (i) the level comparison signal indicating that the router height identifier indicating corresponding to the level identifier and (ii) the router height identifier identifying a level below that associated with the address word section.

26. A computer as defined in claim 23 in which said message information processor includes:
A. a message delimitation circuit for generating a begin message identification to identify the beginning of a message an end message identification circuit for generating an end message identification to identify the end of a message, and C. a message coupling control circuit for controlling the coupling of said processed address word and said data portion to said message input buffer in response to the begin and end message identifications from said message delimitation circuit.

27. A computer as defined in claim 26 in which each message includes message length information, said message delimitation including a message length counter circuit for receiving said message length information and determining therefrom when an entire message has been coupled to said message input buffer.

28. A computer as defined in claim 27 in which each message further includes, following said address portion, a message data portion including a data length section including a data length value that identifies the length of said address portion, said level identifier and said data length section comprising said address portion, said message length counter circuit comprising:
A. a begin message detector for detecting the beginning of a message;
B. an address portion counter for receiving said level identifier in response to a begin message signal and decrementing as said address portion is being received, said address portion counter generating an address portion end signal after the address portion has been received C. a data portion counter for receiving said data length value in response to a length value control signal and for decrementing as said data portion is being received, said data portion counter generating a data portion end signal after the data portion has been received; and D. a counter control circuit for controlling the operation of said address portion counter and said data portion counter, said counter control circuit enabling (i) said address portion counter to receive said level identifier when the begin message detector detects the beginning of a message; and (ii) said data portion counter to receive said data length value in response to the address portion end signal.

29. A computer as defined in claim 26 in which each message further includes an error control portion containing error control information to facilitate detection of transfer errors, said message information processor further including an error control processing portion for using the error control portion to determine whether a transfer error has occurred.

30. A computer as defined in claim 29 in which said error control processing portion includes:
A. an error control check circuit for using the error control portion to determine whether a transfer error occurred connection with a received message;
B. an updated error control information generating portion for generating updated error control information in response to the processed address word information and data portion coupled to said message input buffer by said message information processor;
C. an error code generating portion for generating an error code; and D. an error code selection portion for selectively coupling said updated error control information or said error code to said message input buffer in response to the determination by said error control circuit.

31. A computer as defined in claim 30 in which said error code generating portion generates said error code as the complement of the updated error control information generated by said updated error control information generating portion.

32. A computer as defined in claim 15 in which said message input buffer generates a condition signal indicating selected conditions, said input message control circuit generating a flow control signal in response to the parent or child node connected thereto for transmission to regulate the transmission of messages thereby.

33. A message generating element for generating messages for transmission over a routing network comprising a plurality of router nodes interconnected in a tree pattern in a series of levels from a leaf level to a root level, each node being connected to a parent node at a higher level and a child node at a lower level and at least some of said nodes being connected to a plurality of child nodes to thereby define a tree pattern, the nodes at the leaf level being connected to receive messages from and transmit messages to the message generating elements, each message having an address portion including a level identifier portion containing a level identifier value and a destination identifier portion containing a destination path identifier value, said message generating elements being identified by a message generating element identifier identifying the relative position of its connection to a routing network, said message generating element including an address generator for generating said level identifier value and said destination path identifier value comprising A. an absolute address generator responsive to generation of a message for generating an absolute address value corresponding to the message generating element identifier for a message generating element to receive the message;
B. a height value generating portion for generating a level identifier value in response to the absolute address value generated by the absolute address generator and the message generating element identifier of the message generating element that is generating the message;
C. an address combiner for generating said address portion in response to the level identifier value generated by the height value generating portion and said absolute address generated by said absolute address generator.

34. A message generating element as defined in claim 33 in which said height value generating portion includes:
A. a relative address generating portion for generating a relative address in response to the absolute address value generated by the absolute address generator and the message generating element identifier of the message generating element that is generating the message, the relative address identifying a displacement over the routing network from the message generating element generating a message to the message generating element to receive the message;
B. a height identifier for generating a height value in response to the relative address generated by the relative address generating portion.

35. A message generating element as defined in claim 34 in which said absolute address value and the message generating element identifier are both represented by a word comprising a sequence of bits, said relative address generating portion performing a bit-wise exclusive-OR operation in connection with said absolute address value and the message generating element identifier to generate said relative address.

36. A message generating element as defined in claim 34 in which said height identifier comprises:
A. a potential height identifier circuit for generating in response to the relative address generated by said relative address generating portion a series of height signals each associated with a particular level identifier value, said potential height identifier circuit asserting selected ones of said height signals as determined by the relative address; and B. a decoder for selectively generating, in response to the height signals from the potential height identifier circuit, one of a plurality of height identifier values, each of said plurality being associated with a particular level identifier value, the height identifier value generated being associated with the highest level identifier value whose associated height signal is asserted.

37. A message generating element as defined in claim 36 in which the height identifier values generated by said decoder are defined by binary-encoded signals.

38. A message generating element as defined in claim 36 in which said absolute address value is defined by a binary-encoded absolute address signal having a predetermined number of bits, said down pat generating portion generates said down path identifier value as a binary-encoded signal comprising a selected range of low-order ones of bits comprising said absolute address signal.

39. A message generating element as defined in claim 36 in which said address generator generates an address word comprising said level identifier value in a high-order portion and said destination path identifier value in a low-order portion, A. said height value generating portion further includes a height location select circuit responsive to the relative address generated by said relative address generating portion for generating a level portion identifier to identify the portion of said address word to contain said level identifier value, B. said address combiner coupling (i) said level identifier value in a portion of said address word identified by said level portion identifier and (ii) a selected low-order portion of the absolute address generated by said absolute address generator in the low-order portion of said address word below the portion occupied by said level identifier value thereby to establish said address word.

40. A message generating element as defined in claim 39 in which said address word generated by said address generator comprises a series of predetermined address word sections each associated with a predetermined group of at least one level identifier value, A. said height value generating portion identifying in response to said relative address one of said groups associated with the level identifier value and generating a section identifier signal associated with the identified group, B. said address combiner comprising a plurality of coupling circuits each associated with one of said address word sections for selectively coupling in response to the section identifier signal generated by said height value generating portion either the level identifier value or a low-order portion of said absolute address as the address word.

41. A digital computer comprising a plurality of message generating nodes interconnected by a routing network A. the routing network transferring messages among said message generating elements in accordance with address information identifying a destination message generating element to receive the message;
B. each message generating node including:
i a message data generator for generating message data items each including an address data portion, the address data portion of each message data item containing address data identifying one of the plurality of message generating nodes as a destination message generating node to receive a message generated in response to the message data item;
ii. an interface including:
a) an address translation table including a plurality of entries each identifying an original address value and a translated address value;
b) a message generator for generating, a response to the receipt of a message data item from said message data generator, a message for transmission to the routing network, said message generator including an address translator for performing an address translation operation in connection with the address data and the contents of the address translation table to generate updated address data, said message generator using the updated address data in connection with generating address information for the message, the message generator coupling the message to said routing network.

42. A digital computer as defined in claim 41 in which said address translator comprises:
A. a chunk size identifier for identifying a chunk size said chunk size identifying a number of consecutive message generating nodes;
B. a window extraction circuit for generating an address translation table entry identifier in response to said chunk size identifier and said address data and coupling address translation table entry identifier to said address translation table, the address translation table selecting an entry in response to the received address translation table entry identifier and the original address values in said entries, said address translation table providing the translated address value from the selected entry at an output;
C. a window insertion circuit for generating said updated address value in response to said chunk size identifier, the address data and the translated address value from the output of said address translation table.

43. A digital computer as defined in claim 42 in which the address data comprises a series of address digits, said window extraction circuit generating said address translation table entry identifier as a selected series of said address digits, the window extraction circuit selecting the series of address digits in response to the chunk size identifier.

44. A digital computer as defined in claim 42 in which said address value, said updated address value and said translated address value comprise respective series of address digits, said window insertion circuit generating said updated address value by substituting the series of address digits comprising said translated address value as a selected series in the address data, the window insertion circuit selecting the digits of the address data for which it substitutes the translated address value in response to the chunk size identifier.

45. A digital computer as defined of claim 41 wherein the each message data item further includes an address mode flag having a plurality of conditions identifying the address data portion as having one of a plurality of address modes, said message generator selectively using the address data from the message data item or the updated address data from said address translator in generating a message in response to the message data item in response to the condition of the address mode flag of a message data item.

46. A digital computer as defined in claim 45 in which each message generating node is identified by a network identifier, in one address mode the address data in the address data portion of the message data item containing the network identifier of the message generating node so receive the message generated by said message generator in response thereto.

47. A digital computer as defined in claim 45 in which each message generating node is identified by a network identifier, in one address mode the address data in the address data portion of the message data item containing a relative address value identifying the difference between the network identifiers of the message generating node to receive the message and a predetermined message generating node.

48. A digital computer as defined in claim 45 in which said message generator further includes:
A. a latch;
B. an address mode flag decoder for decoding the address mode flag of a received message data item and generating an address mode signal to identify the address mode;
C. a multiplexer for selectively, in response to the address mode signal from said address mode flag decoder, coupling the address data from the message data item or the updated address data from said address translator as updated address data for storage in said latch; and D. a control circuit for controlling storage of updated address data from said multiplexer in said latch.

49. A digital computer as defined in claim 48 wherein:
A. said address translation table provides an translated address value to said address translator in response to a translation enabling signal, and generates a translated address value valid signal when the translated address value provided to said address translator is valid; and B. said control circuit includes:

i. an address translation table control circuit for selectively generating said translation enabling signal in response to said address mode signal;
ii. a latch control circuit for generating, in response to said address mode signal and the translated address value valid signal from said address translation table, an enabling signal for enabling said latch to store the updated address data from said multiplexer.

50. A digital computer as defined in claim 49 in which said latch has an output terminal connected to couple the contents of said latch an updated address value processing stage, said updated address value processing stage generating an advance control signal for controlling the coupling of the contents of said latch thereto, said address translation table control circuit and said latch control circuit further operating in response to said advance control signal.

51. A digital computer as defined in claim 50 in which:
A. said message generator further includes delay indication generating means for generating, in response to a delay indication enabling signal, a delay indication to said updated address value processing stage to indicate a delay in generating an updated address value; andB. said control circuit further includes a delay indication control circuit for generating said delay indication enabling signal in response to said advance control signal, said translated address value valid signal and said address mode signal, thereby to enable said delay indication generating means to generate the delay indication enabling signal in response to a delay by said address translation table in providing a translated address value if the address mode signal identifies a selected address mode in which the updated address data is generated in response to the translated address value.

52. In a digital computer comprising a plurality of message generating nodes interconnected by a routing network, the routing network transferring messages among said message generating elements in accordance with address information identifying a destination message generating element to receive the message, a message generating node comprising:
A. a message data generator for generating message data items each including an address data portion, the address data portion of each message data item containing address data identifying one of the plurality of message generating nodes as a destination message generating node to receive a message generated in response to the message data item;
B. an interface including;
i. an address translation table including a plurality of entries each identifying an original address value and a translated address value;
ii. a message generator for generating, in response to the receipt of a message data item from said message data generator, a message for transmission to the routing network, said message generator including an address translator for performing an address translation operation in connection with the address data and the contents of the address translation table to generate updated address data, said message generator using the updated address data in connection with generating address information for the message, the message generator coupling the massage to said routing network.

53. A message generating node as defined in claim 52 in which said address translator comprises:
A. a chunk size identifier for identifying a chunk size, said chunk size identifying a number Or consecutive message generating nodes;
B. a window extraction circuit for generating an address translation table entry identifier in response to said chunk size identifier and said address data and coupling address translation table entry identifier to said address translation table, the address translation table selecting, an entry in response to the received address translation table entry identifier and the original address values in said entries, said address translation table providing the translated address value from the selected entry at an output;
C. a window insertion circuit for generating said updated address value in response to said chunk size identifier, the address data and the translated address value from the output of said address translation table.

54. A message generating node as defined in claim 53 in which the address data comprises a series of address digits, said window extraction circuit generating said address translation table entry identifier as a selected series of said address digits, the window extraction circuit selecting the series of address digits in response to the chunk size identifier.

55. A message generating node as defined in claim 53 in which said address value, said updated address value and said translated address value comprise respective series of address digits, said window insertion circuit generating said updated address value by substituting the series of address digits comprising said translated address value as a selected series in the address data, the window insertion circuit selecting the digits of the address data for which it substitutes the translated address value in response to the chunk size identifier.

56. A message generating node as defined in claim 52 wherein the each message data item further includes an address mode fag having a plurality of conditions identifying the address data portion as having one of a plurality of address modes, said message generator selectively using the address data from the massage data item or the updated address data from said address translator in generating a message in response to the message data item in response to the condition of the address mode flag of a message data item.

57. A message generating node as defined in claim 56 in which each message generating node is identified by a network identifier, in one address mode the address data in the address data portion of the message data item containing the network identifier of the message generating node to receive the message generated by said message generator in response thereto.

58. A message generating node as defined in claim 56 in which each message generating node is identified by a network identifier, in one address mode the address data in the address data portion of the message data item containing a relative address value identifying the difference between the network identifiers of the message generating node to receive the message and a predetermined message generating node.

59. A message generating node as defined in claim 56 in which said message generator further includes:
A. a latch;
B. an address mode flag decoder for decoding the address mode flag of a received message data item and generating an address mode signal to identify the address mode;
C. a multiplexer for selectively, in response to the address mode signal from said address mode flag decoder, coupling the address data from the message data item or the updated address data from said address translator as updated address data for storage in said latch; and D. a control circuit for controlling storage of updated address data from said multiplexer in said latch.

60. A message generating node as defined in claim 59 wherein:
A. said address translation table provides an translated address value to said address translator in response to a translation enabling signal, and generates a translated address value valid signal when the translated address value provided to said address translator is valid; and B. said control circuit includes:
i. an address translation table control circuit for selectively generating said translation enabling signal in response to said address mode signal;
ii. a latch control circuit for generating, in response to said address mode signal and the translated address value valid signal from said address translation table, an enabling signal for enabling said latch to store the updated address data from said multiplexer.

61. A message generating node as defined in claim 60 in which said latch has an output terminal connected to couple the contents of said latch an updated address value processing stage, said updated address value processing stage generating an advance control signal for controlling the coupling of the contents of said latch thereto, said address translation table control circuit and said latch control circuit further operating in response to said advance control signal.

62. A message generating node as defined in claim 61 in which:
A. said message generator further includes delay indication generating means for generating, in response to a delay indication enabling signal, a delay indication to said updated address value processing stage to indicate a delay in generating an updated address value; andB. said control circuit further includes a delay indication control circuit for generating said delay indication enabling signal in response to said advance control signal, said translated address value valid signal and said address mode signal, thereby to enable said delay indication generating means to generate the delay indication enabling signal in response to a delay by said address translation table in providing a translated address value if the address mode signal identifies a selected address mode in which the updated address data is generated in response to the translated address value.

63. A digital computer comprising:
A. a plurality of message processing elements, each generating messages for transfer to others of said message processing elements and for receiving messages generated by others of said message processing elements, each message including a path identifier portion identifying a path from a source message processing element to a destination message processing element;
B. a routing network for transferring messages among said message processing elements, said routing network comprising a plurality of interconnected router nodes, at least some of said router nodes being connected to receive messages from and transmit messages to the message processing elements, each router node operating in a first node in response to a mode control signal having a first condition to couple received messages to a router node connected thereto in accordance with the path identifier portion to thereby transfer each respective message along the path identified in its path identifier portion, each router node alternatively operating in a second mode in response to said mode control signal having a second condition to couple received messages to predetermined ones of the router nodes connected thereto, the predetermined ones of said router nodes being selected to facilitate transfer of a message to a nearby message processing element to facilitate the rapid emptying of the routing network of messages; and C. a control element for controlling the conditions of the mode control signals received by the router nodes to enable them to operate in the first mode or second mode generally contemporaneously.

64. A digital computer as defined in claim 63 wherein said router nodes, while operating in said second mode, tag messages to indicate that they are being coupled while said router nodes are in said second mode.

65. A digital computer as defined in claim 54 wherein said message processing elements process messages received thereby in response to their respective message tags.

66. A digital computer as defined in claim 63 wherein the router nodes, when receiving a message while in said second mode, generates a path identifier portion for the message which they transfer so that the path identifier portion of the message as received by the nearby message processing element will reflect a path through the routing network from the nearby message processing element to the destination message processing element.

67. A digital computer as defined in claim 66 in which each router node, while in the second node modifies the path identifier portion in transferring the message along the path to the nearby message processing element so that, when the nearby message processing element receives the message, it will identify a path through the routing network from the nearby message processing element to the router node which began to couple the message thereto, as well as a path from the router node which began to couple the message to the nearby router node to the destination message processing element.

68. A digital computer as defined in claim 63 wherein:
A. said message processing elements generate each message to have a path identifier portion including a path identifier value comprising a level identifier value destination path identifier value; and B. in said routing network said router nodes are interconnected in a tree pattern in a series of levels from a leaf level to a root level, each router node being connected to a parent router node at a higher level and a child router node at a lower level and at least some of said router nodes being corrected to a plurality of child router nodes to thereby define a tree pattern, the router nodes at the leaf level being connected to receive messages from and transmit messages to the message processing elements, each router node while operating, in said first mode (i) transmitting a message to its parent router node if the level identifier value of a received message identifies a level higher than that of the router node or alternatively (ii) transmitting said message to a particular child router node as identified by said destination identifier value and decrementing the level identifier value if the level identifier value of a received message does not identify a level higher than that of the router node.

69. A digital computer as defined in claim 68 wherein each router node, while operating in said second mode, transmits received messages to predetermined ones of its child router nodes.

70. A digital computer as defined in claim 69 wherein each router node, while operating in said second mode, further tags messages as being transferred when the router node is operating in said second mode, thereby to notify the message processing nodes that receive such messages that they received them while the router nodes were operating in said second mode.

71. A digital computer as defined in claim 68 wherein each router node comprises:
A. a plurality of interface modules each for receiving messages from and transferring processed messages to a selected parent node or child router node, each interface module comprising:
i a message processor for selectively processing, while said router nodes are operating in said first mode, a received message to generate an updated message having an updated path identifier portion to be substituted for the path identifier portion of the received message; and ii. an output path selector for generating an output router node identification value in response to the path identifier portion of the received message while said router node is operating in said first mode, and alternatively generating an output router node identification value identifying a selected child router node while said router node is operating in said second mode; and B. a switch connected to receive updated messages from said interface modules and to couple them to others of said interface modules in accordance with the output router node identification values generated by said output path selectors of said interface modules.

72. A digital coputer as defined in claim 71 wherein each said interface modul? includes:
A. path identifier processing means responsive to the receipt of a message by said input terminal for generating an updated pat h identifier value in response to the path identifier value of a received message;
B. are updated message generator for generating said updated message for transfer to said switch in response to receipt of a message, said updated message generator selectively coupling either the path identifier value of the received message or the updated path identifier value for the updated message in response to a selection signal; and C. updated message generator control means for generating said selection signal to control the operation of said updated message generator in response to said mode control signal.

73. A digital computer as defined in claim 72 wherein said path identifier processing means of each interface module generates an updated path identifier value by decrementing the path identifier value of a received message.

74. A digital computer as defined in claim 72 wherein, for each router node, said updated message generator control means further controls the operation of said updated message generator in response to the level identifier value and the level of the router node.

75. A digital computer as defined in claim 74 in which each said updated message generator control means includes:
A. means for generating node level identifier signals having a node level identifier value identifying, the level of the router node in the routing network;
B. a comparator for generating a level correspondence signal in response to a selected relationship between the node level identifier value of said node level identifier signals and the level identifier value of a received message; and C. a selection signal generator for generating said selection signal to enable said updated message generator to couple the path identifier value of the received message for said updated message in response to either the absence of the level correspondence signal and said mode control signal identifying the second mode, or the updated path identifier value for the updated message in response to the coincidence of the presence of the level correspondence signal and said node control signal identifying the first mode.

76. A digital computer as defined in claim 68 in which wherein said router nodes, while operating in said second mode, tag messages to indicate that they are being coupled while said router nodes are in said second mode, said message processing elements using the respective message tags to identify messages coupled thereto while the router nodes are operating in their respective operating modes.

77. A digital computer as defined in claim 76 in which, if said router nodes are operating in the first mode, the level identifiers of messages received by the message processing elements will identify the leaf level, and if said router nodes arc operating in the second mode the level identifiers of messages received by the message processing elements will identify a level other than the leaf level, a message tag corresponding to the level identifier identifying a level other than the leaf level.

78. A digital computer as defined in claim 77 in which said message processing elements, in response to a predetermined event, couple over the routing network tagged messages which they receive, to facilitate transfer of said tagged messages to their respective destination message processing elements.

79. A digital computer as defined in claim 76 in which said message processing elements includes:
A. message receiving means connected to a router node of said routing network for receiving messages from said router node;
B. tag decoder means connected to said message receiving means for determining whether received messages are tagged; and C. message processing means for processing messages received by said message receiving means, said message processing means performing processing of said messages in response to determination by said tag decoder means that said messages are tagged.

80. A digital computer as defined in claim 79 in which:
A. said message processing element further includes message generating means for generating messages for transfer over said routing network;
B. said message processing means buffers messages that are determined by said tag decoder means as being tagged and enabling said message generating means to generate a message corresponding thereto for transfer over said router network in response to a predetermined event.

81. A digital computer as defined in claim 79 in which, if said router nodes are operating m the first mode, the level identifiers of messages received by the message processing elements will identify the leaf level, and d if said router nodes are operating in the second mode the level identifies of messages received by the message processing elements will identify a level other than the leaf level a message tag corresponding to the level identifier identifying a level other than the leaf level, said tag decoder means decoding the level identifier of each message received by said message receiving means to determine whether the level identifier identifies the leaf level, said tag decoder means determining that the received message is tagged if its level identifier identifies a level other than the leaf level.

82. A digital computer comprising a plurality of message generating nodes interconnected by a routing network.
A. said routing network for transferring messages among said message generating elements;
B. each message generating nodes including:
i. a message processing element for generating messages for transmission over said routing network, and for receiving messages from said routing network; and ii. a message counter whose contents are incremented in response to transmission by said message generating element of a message over said routing network and decremented in response to reception of a message by said message processing element from said routing network;
C. a combine network for performing a combine operation in connection with the values of the message counters of said message generating nodes to determine a combined count value reflecting the number messages which have been transmitted by said message generating nodes over the routing network which have not been received by said message generating nodes.

83. A digital computer as defined in claim 82 in which the routing network comprises a plurality of router nodes interconnected in a tree pattern in a series of levels from a leaf level to a root level, the message generating nodes being connected to transmit messages to and receive messages from router nodes at the leaf level.

84. A digital computer as defined in claim 829 in which each message includes a tag field containing a tag value, said message counter selectively incrementing and decrementing in response to the tag value of the tag field of transmitted and received messages.

85. A digital computer as defined in claim 84 in which said message counter includes A. a counter for generating a digital value in response to a current value, a count increment enable signal which enables said counter to increment, and a count decrement enable signal which enables said counter to decrement;
B. an increment enable signal generator for selectively generating said count increment enable signal in response to (i) the tag value of the tag field of a message generated by the message processing element for transmission over the routing network, and (ii) a count mask; and C. a decrement enable signal generator for selectively generating said count decrement enable signal in response to (i) the tag value of the tag field of a message received by the message processing element from said routing network, and (ii) said count mask; and

86 A digital computer as defined in claim 85 in which said count mask comprises a plurality of count mask bits each having selected conditions, said increment enable signal generator comprising A increment selection means for receiving a plurality of count mask signals eachrepresentative of the condition of a count mask bit, said increment selection means being responsive to the tag value of the tag field of a message generated by the message processing element for transmission over the routing network to selectively couple one of the count mask signals as an increment enable signal;
B. increment gate means for generating said count increment enable signal in response to said increment enable signal and a signal indicating transmission of a message generated by the message processing element over the routing network.

87. A digital computer as defined in claim 86 in which message generating elements of at least some of said message generating nodes include a plurality of ports for concurrently generating a plurality of messages for transmission over said routing network, said message counter including:
A. a plurality of increment enable signal generators each associated with one of said ports for generating a port increment enable signal; and B. a count increment enable consolidation circuit for generating said count increment enable signal in response to the port increment enable signals of all of said increment enable signal generators

88. A digital computer as defined by claim 85 in which said count mask comprises a plurality of count mask bits each having selected conditions, said decrement enable signal generator comprising A. decrement selection means for receiving a plurality of count mask signals each representative of the condition of a count mask bit, said decrement selection means being responsive to the tag value of the tag field of a message received by the message processing element over the routing network to selectively couple one of the count mask signals as a decrement enable signal;
B. decrement gate means for generating said count increment enable signal in response to said decrement enable signal and a signal indicating receipt of a message by the message processing element over the routing network.

89. A digital computer as defined in claim 88 in which message generating elements of at least some of said message generating nodes include a plurality of ports for concurrently receiving a plurality of messages from said routing network, said message counter including:

A. a plurality of decrement enable signal generators each associated with one of said ports for generating a port decrement enable signal, the decrement gate means of the decrement enable signal generator associated with least one of said ports generating the port decrement enable signal in response to a selectively buffered message reception signal;
B. a selective buffer circuit for generating said selectively buffered message reception signal in response to port message reception signals from said ports, each port message reception signal indicating reception by an associated port of a message from said network;
C. a count decrement enable consolidation circuit for generating said count decrement enable signal in response to the port decrement enable signals of all of said decrement enable signal generators.

90. A digital computer as defined in claim 89 in which said selective buffer circuit includes:
A. immediate coincidence means for generating an unbuffered message reception signal in response to a port message reception signal from the port associated with the selective buffer circuit indicating receipt of a message thereby, in the absence the port message reception signals from the other ports indicating concurrent receipt of messages thereby;
B. buffer means for generating a delayed buffered message reception signal in response to a port message reception signal from the port associated with the selective buffer circuit indicating receipt of a message whereby, in the presence of the port message reception signals from the other ports indicating concurrent receipt of messages thereb; and C. means for generating said selectively buffered message reception signal in response to said unbuffered message reception signal and said delayed buffered message reception signal.

91. A digital computer as defined in claim 82 in which said message processing element further includes means for generating an initial count value, said message counter including:
A. a register for storing a current count value;
B. count modification means for receiving said current count value from said register and selectively generating an updated count value representing an incremented count value in response to transmission of a message by said message generating element or a decremented count value in response to reception of a message by said message generating element;
C. count selection means for selectively coupling said initial count value or said updated count value to said register for storage as the current count value.

92. A digital computer as defined in claim 91 in which said register stores said updated count value in response to a load enable signal generated by a load enable control circuit in response to transmission of a message by said message generating element or reception of a message by said message generating element, or in response to receipt of an initial count load enable signal received from said message generating element concurrently with an initial count value.

93. A digital computer as defined in claim 92 in which said load enable control circuit disables generation of said load enable signal in response to contemporaneous transmission of a message by said message generating element and reception of a message by said message generating element.

94. A digital computer as defined in claim 82 in which said message counter includes:
A a counter for generating a digital value in response to a current value, a count increment enable signal which enables said counter to increment, and a count decrement enable signal which enables said counter to decrement;
B. an increment enable signal generator for selectively generating said count increment enable signal in response to transmission of a message generated by the message processing element over the routing network; and C. a decrement enable signal generator for selectively generating said count decrement enable signal in response to reception of a message received by the message processing element from said routing network.

95. A digital computer as defined in claim 94 m which message generating elements of at least some of said message generating nodes include a plurality of ports for concurrently generating a plurality of messages for transmission over said routing network, said message counter including A. a plurality of increment enable signal generators each associated with one of said ports for generating a port increment enable signal; and B. a count increment enable consolidation circuit for generating said count increment enable signal in response to the port increment enable signals of all of said increment enable signal generators.

96. A digital computer as defined in claim 94 in which message generating elements of at least some of said message generating nodes include a plurality of ports for concurrently receiving a plurality of messages from said routing network, said message counter including:
A. a plurality of decrement enable signal generators each associated with one of said ports for generating a port decrement enable signal, the decrement gate means of the decrement enable signal generator associated with least one of said ports generating the port decrement enable signal in response to a selectively buffered message reception signal;
B. a selective buffer circuit for generating said selectively buffered message reception signal in response to port message reception signals from said ports, each port message reception signal indicating reception by an associated port of a message from said network;
C. a count decrement enable consolidation circuit for generating said count decrement enable signal in response to the port decrement enable signals of all of said decrement enable signal generators.

97. A digital computer as defined in claim 94 in which said selective buffer circuit includes:
A. immediate coincidence means for generating an unbuffered message reception signal in response to a port message reception signal from the port associated with the selective buffer circuit indicating receipt of a message thereby, in the absence the port message reception signals from the other ports indicating concurrent receipt of messages thereby;
B. buffer means for generating a delayed buffered message reception signal in response to a port message reception signal from the port associated with the selective buffer circuit indications receipt of a message thereby, in the presence of the port message reception signals from the other ports indicating concurrent receipt of messages thereby; and C. means for generating said selectively buffered message reception signal in response to said unbuffered message reception signal and said delayed buffered message reception signal.

98. A digital computer as defined in claim 82 further comprising means for controlling operations by said message generating nodes in response to the combined count value generated by said combine network.

99. A digital computer comprising a plurality of message generating nodes interconnected by a routing network;
A. the routing network including a plurality of routing network sections each for transferring messages among said message generating elements;
B. each message generating node including a message data receiver and an interface;
i. said message data receiver generating message retrieval requests for transfer to said interface to enable said interface to couple message data items thereto, each message retrieval request including a network section identifier identifying one of said routing network sections or a phantom routing network section; and ii. said interface including:
a) a plurality of message ejector ports each for receiving messages from an associated routing network section of said routing network and for generating a message data item in response to the receipt of each message; and b) a message retrieval control circuit for controlling the coupling of message data items from said message ejector ports to said message data receiver in response to message retrieval requests whose network section identifiers identify associated routing network sections, or from a selected message ejector port in response to message retrieval requests whose network section identifiers identify said phantom routing network section.

100. A digital computer as defined in claim 99 in which each message ejector port generates message status information in response to the generation of a message data item, said message retrieval control circuit includes:
A. a status store control circuit comprising:
i. a plurality of ejector port status information stores each associated with a message ejector port for storing message status information generated by the associated message ejector port in response to receipt of a message;
ii. a phantom status information store for storing message status information;
iii. a phantom status information store control circuit for controlling the storage of message status information from respective message ejector ports in sayd phantom status information store in response to message retrieval requests from said message data receiver identifying said phantom routing network section; and B. said message retrieval control circuit controlling the coupling of message data items from said message ejector ports to said message data receiver in response to message retrieval requests whose network section identifiers identify associated routing network sections, or from the message ejector port whose message status information was loaded into the phantom status information store in response to message retrieval requests whose network section identifiers identify said phantom routing network section.

101. A digital comptuer as defined in claim 100 in which said phantom status information store comprises:
A. a plurality of shadow status stores, each associated with an ejector port status information store, for storing status information associated with the associated ejector port;
B. a plurality of valid flags each associated with one of said ejector ports and having a valid condition, indicating the storage by the associated shadow status store of message status information, and an invalid condition;
C. a phantom storage control circuit for controlling the storage of status information from a shadow status store associated with a selected one of said ejector ports in said phantom status store in response to the valid flag associated with the selected ejector port having a valid condition.

102. A digital computer as defined in claim 101 in which said phantom status information store further comprises a plurality of valid flag conditioning circuits each associated with one of said ejector ports, each valid flag conditioning circuit enabling the conditioning of the associated valid flag to the valid condition in response to the generation by the associated ejector port of message status information for a message data item.

103. A digital,computer as defined in claim 102 wherein said message data receiver can further load message status information into any of said ejector port status information stores during a message status information loading operation identifying an ejector port, each valid flag conditioning circuit further selectively enabling the conditioning of the associated valid flag to the valid condition in response to the loading of selected message status information in the associated ejector port status information store during a message status information loading operation.

104. A digital computer as defined in claim 103 wherein said message data receiver retrieves a message data item by iteratively retrieving a series of message data words, said series of message data words comprising a message data item, message status information associated with a message including a message length field containing a message length value identifying the number of message data words in the series of message data words comprising the message data item associated with the message and a length remaining field containing a length remaining value indicative of the number of message data words to be retrieved by said message data receiver, said valid nag conditioning circuit enabling the conditioning of the valid flag during a message status information loading operationing in response to a comparison between the message length vlue and the length remaining value.

105. A digital computer as defined in claim 102 in which each said valid flag conditioning circuit further includes a valid flag reset circuit including an ejector port flag reset circuit for conditioning the associated valid flag to the invalid condition in response to the receipt by the message retrieval control circuit of a message retrieval request identifying the ejector port associated with the valid flag if the valid flag is in the valid condition.

106. A digital computer as defined in claim 105 in which:
A. said phantom status information store further includes an ejector port pointer conditioned by said phantom storage control circuit to generate an ejector port identification to identify the ejector port whose message status information is stored in said phantom status information store;
B. said valid flag reset circuit further includes a phantom port flag reset circuit for for conditioning the associated valid flag to the invalid condition in response to the receipt by the message retrieval control circuit of a message retrieval request identifying the phantom ejector port if the ejector port pointer identifies the ejector port associated with the valid flag and the valid flag is in the valid condition.

107. A digital computer as defined in claim 106 wherein said phantom storage control circuit further includes an ejector port pointer control circuit for controlling update of the condition of said ejector port pointer in response to the generation of message status information by the respective ejector ports and the current condition of said ejector port pointer.

108. A digital computer as defined in claim 107 wherein said message data receiver can further load message status information into any of said ejector port status information stores, said ejector port pointer control circuit further controls the condition of said ejector port pointer in response to the loading by said message data receiver of message status information into the ejector port status information store associated with the ejector port currently identified by said ejector port pointer.

109. A digital computer as defined in claim 107 wherein said ejector port pointer control circuit includes:
A. a plurality of respective port condition enable circuits each associated with an ejector port for enabling said ejector port pointer be conditioned to identify the respective ejector port in response to the generation by the respective ejector port of message status information if the other ejector ports are not concurrently generating message status information; and B. a received-message port arbitration circuit for enabling said ejector port pointer to be conditioned to selectively identify one of said ejector ports in response to the concurrent generation by a plurality of ejector ports of message status information and the current condition of the ejector port pointer.

110. A digital computer as defined in claim 109 wherein said message data receiver can further load message status information into any of said ejector port status information stores, s id ejector port pointer control circuit further including a message data receiver controlled port arbitration circuit for controlling the condition of said ejector port pointer in response to the loading by said message data receiver of message status information into the ejector port status information store associated with the ejector port currently identified by said ejector port pointer.

111. A digital computer as defined in claim 101 in which said phantom status information store further comprises a plurality of ejector port status information store control circuits each including A. an ejector port store control circuit for controlling storage of message status information generated by an associated ejector port in each ejector port status information store;
B. an ejector port invalidation control circuit for controlling the invalidation of message status information stored in the ejector port information store of the associated ejector port in response to a message retrieval request identifying said phantom routing network section if the phantom storage control circuit had enabled the status information from the shadow status store of the associated ejector port to be stored in said phantom status store in response to previous a message retrieval request identifying said phantom routing network section.

112. A digital computer as defined in claim 111 in which:
A. said phantom status information store further includes an ejector port pointer conditioned by said phantom storage control circuit to generate an ejector port identification to identify the ejector port whose message status information is stored in said phantom status information store;
B. said phantom storage control circuit controls the storage in said phantom status store of status information from the shadow status store associated with the ejector port identified by the ejector port identification if the associated valid flag has a valid condition; and C. the ejector port invalidation control circuit of each ejector port status information store control circuit uses the ejector port identification in controlling the invalidation of the message status information stored in the ejector port information store of the associated ejector port.

113. An interface for receiving messages from a network and for generating in response thereto respective message data items for transfer to a message data receiver in response to message retreival requests each containing a port identifier, said interface including:
A. a plurality of message ejector ports each for receiving messages from said network and for generating a message data item in response to the receipt of each message; and B. a message retrieval control circuit for controlling the coupling of message data items from said message ejector ports to said message data receiver in response to message retrieval requests whose port identifiers identify an associated message ejector port, or from a selected message ejector port in response to message retrieval requests whose network section identifiers identify a phantom port.

114. A digital comptuer as defined in claim 113 in which each message ejector port generates message status information in response to the generation of a message data item, said message retrieval control circuit includes:
A. a status store control circuit comprising:

i. a plurality of ejector port status information stores each associated with a message ejector port for storing message status information generated by the associated message ejector port in response to receipt of a message;
ii. a phantom status information store for storing message status information;
iii. a phantom status information store control circuit for controlling the storage of message status information from respective message ejector ports in sayd phantom status information store in response to message retrieval requests from said message data receiver identifying said phantom port;
and B. said message retrieval control circuit controlling the coupling of message data items from said message ejector ports to said message data receiver in response to message retrieval requests whose port identifiers identify associated message ejector ports, or from the message ejector port whose message status information was loaded into the phantom status information store in response to message retrieval requests whose port identifiers identify said phantom port.

115. An interface as defined in claim 114 in which said phantom status information store comprises A. a plurality of shadow status stores, each associated with an ejector port status information store, for storing status information associated with the associated ejector port;
B. a plurality of valid flags each associated with one of said ejector ports and having a valid condition, indicating the storage by the associated shadow status store of message status information, and an invalid condition;
C. a phantom storage control circuit for controlling the storage of status information from a shadow status store associated with a selected one of said ejector ports in said phantom status store in response to the valid flag associated with the selected ejector port having a valid condition.

116. An interface as defined in claim 115 in which said phantom status information store further comprises a plurality of valid flag conditioning circuits each associated with one of said ejector ports, each valid flag conditioning circuit enabling the conditioning of the associated valid flag to the valid condition in response to the generation by the associated ejector port of message status information for a message data item.

117. An interface as defined in claim 116 wherein said message data receiver can further load message status information into any of said ejector port status information stores during a message status information loading operation identifying an ejector port, each valid flag conditioning circuit further selectively enabling the conditioning of the associated valid flag to the valid condition in response to the loading of selected message status information in the associated ejector port status information store during a message status information loading operation.

118. An interface as defined in claim 117 wherein said message data receiver retrieves a message data item by iteratively retrieving a series of message data words, said series of message data words comprising a message data item, message status information associated with a message including a message length field containing a message length value identifying the number of message data words in the series of message data words comprising the message data item associated with the message and a length remaining field containing a length remaining value indicative of the number of message data words to be retrieved by said message data receiver, scud valid flag conditioning circuit enabling the conditioning of the valid flag during a message status information loading operationg in response to a comparison between the message length value and the length remaining value.

119. An interface as defined in claim 116 in which each said valid flag conditioning circuit further includes a valid flag reset circuit including a ejector port flag reset circuit for conditioning the associated valid flag to the invalid condition in response to the receipt by the message retrieval control circuit of a message retrieval request identifying the ejector port associated with the valid flag if the valid flag is in the valid condition.

120. An interface as defined in claim 118 in which:
A. said phantom status information store further includes an ejector port pointer conditioned by said phantom storage control circuit to generate an ejector port identification to identify the ejector port whose message status information is stored in said phantom status information store;
B. said valid nag reset circuit further includes a phantom port flag reset circuit for for conditioning the associated valid flag to the invalid condition in response to the receipt by the message retrieval control circuit of a message retrieval request identifying the phantom ejector port if the ejector port pointer identifies the ejector port associated with the valid flag and the valid flag is in the valid condition.

121. An interface as defined in claim 116 wherein said phantom storage control circuit further includes an ejector port pointer control circuit for controlling update of the condition of said ejector porn pointer in response to the generation of message status information by the respective ejector ports and the current condition of said ejector port pointer.

122. An interface as defined in claim 118 wherein said message data receiver can further load message status information into any of said ejector port status information stores, said ejector port pointer control circuit further controls the condition of said ejector port pointer in response to the loading by said message data receiver of message status information into the ejector port status information store associated with the ejector port currently identified by said ejector port pointer.

123. An interface as defined in claim 118 wherein said ejector port pointer control circuit includes:
A. a plurality of respective port condition enable circuits each associated with an ejector port for enabling said ejector port pointer be conditioned to identify the respective ejector port in response to the generation by the respective ejector port of message status information if the other ejector ports are not concurrently generating message status information; and B. a received-message port arbitration circuit for enabling said ejector port pointer to be conditioned to selectively identify one of said ejector ports in response to the concurrent generation by a plurality of ejector ports of message status information and the current condition of the ejector port pointer.

124. An interface as defined in claim 122 wherein said message data receiver can further load message status information into any of said ejector port status information stores, said ejector port pointer control circuit further including a message data receiver-controlled port arbitration circuit for controlling the condition of said ejector port pointer in response to the loading by said message data receiver of message status information into the ejector port status information store associated with the ejector port currently identified by said ejector port pointer.

125. An interface as defined in claim 115 in which said phantom status information store further comprises a plurality of ejector port status information store control circuits each including A. an ejector port store control circuit for controlling storage of message status information generated by an associated ejector port in each ejector port status information store;
B. an ejector port invalidation control circuit for controlling the invalidation of message status information stored in the ejector port information store of the associated ejector port in response to a message retrieval request identifying said phantom port if the phantom storage control circuit had enabled the status information from the shadow status store of the associated ejector port to be stored in said phantom status store in response to previous a message retrieval request identifying said phantom port.

126. An interface as defined in claim 124 in which:
A. said phantom status information store further includes an ejector port pointer conditioned by said phantom storage control circuit to generate an ejector port identification to identify the ejector port whose message status information is stored in said phantom status information store;
B. said phantom storage control circuit controls the storage in said phantom status store of status information from the shadow status store associated with the ejector port identified by the ejector port identification if the associated valid flag has a valid condition; and C. the ejector port invalidation control circuit of each ejector port status information store control circuit uses the ejector port identification in controlling the invalidation of the message status information stored in the ejector port information store of the associated ejector port.

127. A digital computer comprising a plurality of message generating nodes interconnected by a routing network;
A. the routing network including a plurality of routing network sections each for transferring messages among said message generating elements;
B. each message generating node including:
i. a message data generator for generating message data items, each message data item alternatively identifying one of said routing network sections or none of said routing network sections;
and ii. an interface including:
a) a plurality of message injector ports each for generating a message for transfer over an associated routing network section of said routing network in response to the receipt of a message data item identifying the routing network section and in response to the receipt of a message data item identifying none of the routing network sections and a condition of an enabling signal; and b) a target select circuit for selectively conditioning enabling signals for each of said message injector ports to selectively enable them to generate a message in response to message data item which identifies none of the routing network sections, said target select circuit conditioning said enabling signals in response to the current condition of each of said enabling signal and conditions of said respective message injector ports.

128. A digital computer as defined in claim 127 in which each message injector port generates a status signal indicating its current ability to receive a message data item, said target select circuit using the status signals from all of said message injector portions in conditioning the respective enabling signals.

129. A digital computer as defined in claim 127 in which said target select circuit comprises:
A. a target register for storing a target identification identifying one of said injector ports and conditioning said enabling signals in response thereto;
B. a next target identification circuit connected to receive at least some of the enabling signal from said target register for generating an update target signal to identify an updated target identification to identify another of said injector port said target register storing said updated target identification in connection with a target update enable signal to update the conditioning of said enabling signals;
C. a target register update control circuit for generating said target update enable signal in response to the enabling signals, the status signals and the receipt of a message data item from said message data generator.

130. A digital computer as defined in claim 129 in which said target register update control circuit includes:
A. a target update enable circuit for generating an update enable signal in response to the status signals and the conditions of the respective enable signals; and B. a coincidence circuit for coupling the update enable signal to enable the target register to store the updated target identification in response to receipt of a message data item from the message data generator, thereby to enable the enabling signals to be updated in response to change in the status signals resulting from the receipt of the message data item.

131. A digital computer as defined in claim 130 in which said target update enable circuit comprises:
A. a plurality of port-wise update control circuits each associated with a message injector port for conditioning a port-wise target update control signal in response to the coincidence of the associated port's status signal indicating that it is unable to receive another message data item and the associated port's enable signal conditioned to enable it to receive a message data item identifying none of the routing network sections;
B. a new target update control circuit for conditioning a new target update control signal if the status signal of the injector port identified by the updated target information indicates that the injector port is able to receive message data item;

C. an update enable signal generator for generating said update enable signal in response to the conditions of the port-wise target update control signals generated by said port-wise update control circuits and the new target update control signal from said new target update control circuit.

132. A digital computer as defined in claim 128 in which said routing network includes two routing network sections, and interface of each message generating nodes includes two injector ports, said target select circuit comprising:
A. a target flip-flop for storing a target identification identifying one of said injector ports, said target latch generating an output signal comprising the enabling signal associated with one of said injector ports in response thereto;
B. a complementation circuit connected to receive the enabling signal generated by said target flip-flop and generate an output signal comprising the enabling signal for associated with the other of said injector ports, the output signal further being coupled to an input of said target flip flop to control an updated condition said target flip-flop in response to a target update enable signal;
C. a target register update control circuit for generating said target update enable signal in response to the enabling signals, the status signals and the receipt of a message data item from said message data generator.

133. A digital computer as defined in claim 132 in which said target register update control circuit includes:
A. a target update enable circuit for generating an update enable signal in response to the status signals and the conditions of the respective enable signals; and B. a coincidence circuit for coupling the update enable signal to enable the target flip-flop to store the output signal from said complementation circuit in response to receipt of a message data item from the message data generator, thereby to enable the enabling signals to be updated in response to change in the status signals resulting from the receipt of the message data item.

134. A digital computer as defined in claim 133 in which said target update enable circuit comprises:
A. a plurality of port-wise update control circuits each associated with a message injector port for conditioning a port-wise target update control signal in response to the coincidence of the associated port's status signal indicating that it is unable to receive another message data item and the associated port's enable signal conditioned to enable it to receive a message data item identifying none of the routing network sections;
B. a new target update control circuit for conditioning a new target update control signal if the status signal of the injector port identified by the updated target information indicates that the injector port is able to receive message data item;
C. an update enable signal generator for generating said update enable signal in response to the conditions of the port-wise target update control signals generated by said port-wise update control circuits and the new target update control signal from said new target update control circuit.

135. An interface for generating messages for transfer over a routing network; in response to message data items from a message generating node, said interface including:

A. a plurality of message injector ports each for generating a message for transfer over an associated routing network section of said routing network in response to the receipt of a message data item identifying the injector port and in response to the receipt of a message data item identifying none of the injector ports and a condition of an enabling signal; and B. a target select circuit for selectively conditioning enabling signals for each of said message injector ports to selectively enable them to generate a message in response to message data item which identifies none of the injector ports, said target select circuit conditioning said enabling signals in response to the current condition of each of said enabling signal and conditions of said respective message injector ports.

136. An interface as defined in claim 135 in which each message injector port generates a status signal indicating its current ability to receive a message data item, said target select circuit using the status signals from all of said message injector portions in conditioning the respective enabling signals.

137. An interface as defined in claim 136 in which said target select circuit comprises:
A. a target register for storing a target identification identifying one of said injector ports and conditioning said enabling signals in response thereto;
B. a next target identification circuit connected to receive at least some of the enabling signals from said target register for generating all update target signal to identify an updated target identification to identify another of said injector ports, said target register storing said updated target identification in connection with a target update enable signal to update the conditioning of said enabling signals;
C. a target register update control circuit for generating said target update enable signal in response to the enabling signals, the status signals and the receipt of a message data item from said message data generator.

138. An interface as defined in claim 137 in which said target register update control circuit includes:
A. a target update enable circuit for generating an update enable signal in response to the status signals and the conditions of the respective enable signals; and B. a coincidence circuit for coupling the update enable signal to enable the target register to store the updated target identification in response to receipt of a message data item from the message data generator, thereby to enable the enabling signals to be updated in response to change in the status signals resulting from the receipt of the message data item.

139. An interface as defined in claim 138 in which said target update enable circuit comprises:
A. a plurality of port-wise update control circuits each associated with a message injector port for conditioning a port-wise target update control signal in response to the coincidence of the associated port's status signal indicating that it is unable to receive another message data item and the associated port's enable signal conditioned to enable it to receive a message data item identifying none of the injector ports;
B. a new target update control circuit for conditioning a new target update control signal if the status signal of the injector port identified by the updated target information indicates that the injector port is able to receive message data item;

C. an update enable signal generator for generating said update enable signal in response to the conditions of the port-wise target update control signals generated by said port-wise update control circuits and the new target update control signal from said new target update control circuit.

140. An interface as defined in claim 136 in which said interface includes two injector ports, said target select circuit comprising:
A. a target flip-flop for storing a target identification identifying one of said injector ports, said target latch generating an output signal comprising the enabling signal associated with one of said injector ports in response thereto;
B. a complementation circuit connected to receive the enabling signal generated by said target flip-flop and generate an output signal comprising the enabling signal for associated with the other of said injector ports, the output signal further being coupled to an input of said target flip-flop to control and updated condition said target flip-flop in response to a target update enable signal;
C. a target register update control circuit for generating said target update enable signal in response to the enabling signals, the status signals and the receipt of a message data item from said message data generator.

141. An interface as defined in claim 140 in which said target register update control circuit includes:
A. a target update enable circuit for generating an update enable signal in response to the status signals and the conditions of the respective enable signals; and B. a coincidence circuit for coupling the update enable signal so enable the target flip-flop to store the output signal from said complementation circuit in response to receipt of a message data item from the message data generator, thereby to enable the enabling signals to be updated in response to change in the status signals resulting from the receipt of the message data item.

142. An interface as defined in claim 141 in which said target update enable circuit comprises:
A. a plurality of port-wise update control circuits each associated with a message injector port for conditioning a port-wise target update control signal in response to the coincidence of the associated port's status signal indicating that it is unable to receive another message data item and the associated port's enable signal conditioned to enable it to receive a message data item identifying none of the injector ports;
B. a new target update control circuit for conditioning a new target update control signal if the status signal of the injector port identified by the updated target information indicates that the injector port is able to receive message data item;
C. an update enable signal generator for generating said update enable signal in response to the conditions of the port-wise target update control signals generated by said port-wise update control circuits and the new target update control signal from said new target update control circuit.

143. An arbitration circuit for controlling coupling of messages from a plurality of input circuits to an output circuit, said switch including a plurality of switching cells each for selectively coupling a message from an associated input circuit to the output circuit in response to a switching cell enable signal said arbitration circuit comprising:

A. an arbitration cell network for generating an input circuit select signal identifying one input circuit in response to output circuit request signals from the input circuits which identify the associated output circuit and in response to an arbitration control signal establishing a selected arbitration priority;
B. an arbitration cell network control circuit for generating the arbitration control signal to enable the arbitration cell network to vary the priorities among the respective input circuits; and C. a cell selection enable circuit for selectively generate, one of a plurality of switching cell enable signals, each associated with one switching cell of the plurality associated with said output circuit, said cell selection enable circuit generating said switching cell enable signals in response to the input circuit select signal and a select enable signal from the switching cells associated with the output circuit.

144. An arbitration circuit as defined in claim 143 in which said arbitration cell network comprises a plurality of arbitration cells connected in a plurality of arbitration stages, and said arbitration control signal generated by said arbitration cell network control circuit comprises a plurality of arbitration stage control signals each associated with an arbitration stage, each arbitration cell in the first stage generation an arbitrated output circuit request signal in response to the output circuit request signals from selected ones of said input circuits and the condition of the associated arbitration stage control signal to establish a priority among the respective input circuits connected thereto, and each arbitration cell in subsequent stages generating arbitrated output circuit request signals in response to arbitrated output circuit request signals from selected ones of the arbitration cells in the preceding stage and an associated stage arbitration control signal whose condition establishes a priority among the respective arbitration cells of preceding stages, the arbitrated output circuit request signals generated by the arbitration cell of the last stage comprising the input circuit select signal and identifying at most one input circuit.

145. An arbitration circuit as defined in claim 144 in which arbitration cell network control circuit includes:
A. an arbitration counter for generating said arbitration stage control signals comprising representing an n-ary-encoded value, where "n" corresponds to the number of input circuits whose output circuit request signals are directed to an arbitration cell, said arbitration counter operating in response to counter control signals; and B. a counter control circuit for controlling the arbitration counter to provide that the arbitration stage control signals enable the arbitration cell network to generate an input circuit select signal for an input circuit for which an output circuit request signal is asserted.

146. An arbitration circuit as defined in claim 145 in which said input circuits effectively form a series, each identified by an input circuit identification value, said arbitration cell network coupling, as the input circuit select signal the output circuit request signal received from the input circuit for which the input circuit identification value corresponds to the n-ary-encoded value from the arbitration counter.

said counter control circuit enabling the arbitration counter to increment until it generates arbitration stage control signals whose n-ary-encoded value corresponds to the input circuit identification value associated with an input circuit whose output circuit request signal is asserted.

147. An arbitration circuit as defined in claim 143 further comprising an arbitration enable circuit for generating an output request enable signal if any of the output circuit request signals associated with the output circuit are being generated by any of the input circuits, the cell selection enable circuit further conditioning generation of the arbitration control signal in response to the condition of the output request enable signal.

148. An arbitration circuit for controlling coupling of messages from a plurality of input circuits to any of a plurality of output circuits, said switch including a plurality of switching cells each for selectively coupling a message from an associated input circuit to an output circuit in response to a switching cell enable signal, said arbitration circuit comprising:
A. an input circuit request priority circuit for receiving output circuit request signals from said input circuits and generating in response prioritized output circuit request signals, the prioritized output circuit request signals identifying relative priority levels among said output circuit request signals;
B. an output circuit availability priority circuit for receiving output circuit select enable signals from said output circuits and generating in response prioritized output circuit select enable signals, the prioritized output circuit select enable signals identifying relative priority levels among said output circuit select enable signals; and C. a input request/output availability match circuit for selectively generating switching cell enable signals, each associated with one switching cell of the plurality associated with an output circuit, said cell selection enable circuit generating said switching cell enable signals in response to the prioritized output request signals and said prioritized output circuit select enable signals to match output circuit requests as indicated by the output circuit request signals from said input circuits to available output circuits as indicated by output circuit select enable signals, each in relation to respective relative priorities.

149. An arbitration circuit as defined in claim 148 in which said input circuit request priority circuit includes:
A. an input circuit priority establishment circuit for periodically establishing relative priority levels among said input circuits;
B. an output request enumerator circuit for generating, for each of said input circuits generating output circuit request signal, a selected one of a plurality of output request priority signals as determined by the priority level for the input circuit as identified by said input circuit priority establishment circuit.

150. An arbitration circuit as defined in claim 149 in which said input circuit priority establishment circuit periodically updates the relative priority levels in a round-robin manner.

151. An arbitration circuit as defined in claim 148 in which said output circuit availability priority circuit includes A. output circuit priority establishment circuit for periodically establishing relative priority levels among said output circuits;
B. output circuit available enumerator circuit for generating, for each of said output circuits generating an output circuit select enable signal, a selected one of a plurality of output select priority signals as determined by the priority level for the output circuit as identified by said output circuit priority establishment circuit.

152. An arbitration circuit as defined in claim 151 in which said output circuit priority establishment circuit periodically updates the relative priority levels in a random manner.

153. A data handling system comprising a plurality of message processing elements interconnected by a routing network, A. said routing network transferring messages among said message processing elements in accordance with a routing synchronizing signal;
B. each message processing element including:
i. message processing means for generating messages in accordance with a generating element synchronizing signal; and ii. interface means for coupling messages between said message processing means and said routing network, said interface means including:
a) a message transmitter for transmitting a buffered message to said routing network in accordance with said routing synchronizing signal, said message transmitter generating a transmitter flow control signal to control transfer of buffered messages thereto;
b) a buffer for receiving a message from said message processing means in response to a synchronized transmitter flow control signal and for supplying said received message to said message transmitter as a buffered message; and c) a synchronizer for generating said synchronized transmitter flow control signal in response to said transmitter flow control signal and said processing element synchronizing signal.

154. A data handling system as defined in claim 153 wherein said synchronizer includes:
A. a transmitter flow control signal latch for latching said transmitter flow control signal, said transmitter flow control signal latch asserting a latched transmitter flow control signal in response to assertion by said message transmitter of said transmitter flow control signal;
B. a synchronization buffer connected to said transmitter flow control signal latch for latching said latched transmitter flow control signal in response to said processing element synchronizing signal, thereby to generate said synchronized transmitter flow control signal.

155. A data handling system as defined in claim 154 in which said processing element synchronizing signal is in the form of a series of sequential clock ticks, said synchronization buffer generating said synchronized transmitter flow control signal a predetermined number of dock ticks of said processing element synchronizing signal following assertion by said transmitter flow control signal latch of said latched transmitter flow control signal.

156. A data handling system as defined in claim 154 in which said synchronization buffer if further connected to enable said transmitter flow control signal latch to negate the latched transmitter flow control signal in response to generation of said synchronized transmitter flow control signal by said synchronization buffer.

157. A data handling system as defined in claim 153 further receives messages from said routing network for transfer to said message processing means in accordance with said processing element synchronizing signal said interface means further including A. a message receiver for receiving a message from said routing network in accordance with said routing synchronizing signal, said message receiver generating a receiver transmitter flow control signal to control transfer of messages therefrom;
B. a buffer for receiving a message from said message receiver in response to a synchronized receiver flow control signal and for supplying said received message to said message processing means in accordance with said processing element synchronizing signal; and c) a synchronizer for generating said synchronized receiver flow control signal in response to said receiver flow control signal and said processing element synchronizing signal.

158. A data handling system as defined in claim 157 wherein said synchronizer includes:
A. a receiver flow control signal latch for latching said receiver flow control signal; said receiver flow control signal latch asserting a latched receiver flow control signal in response to assertion by said message receiver of said receiver flow control signal;
B. a synchronization buffer connected to said receiver flow control signal latch for latching said latched receiver flow control signal in response to said processing element synchronizing signal, thereby to generate said synchronized receiver flow control signal.

159. A data handling system as defined in claim 158 in which said processing element synchronizing signal is in the form of a series of sequential clock ticks, said synchronization buffer generating said synchronized receiver flow control signal a predetermined number of clock ticks of said processing element synchronizing signal following assertion by said receiver flow control signal latch of said latched receiver flow control signal.

160. A data handling system as defined in claim 157 in which said synchronization buffer if further connected to enable said receiver flow control signal latch to negate the latched receiver flow control signal in response to generation of said synchronized receiver flow control signal by said synchronization buffer.

161. A message processing element for use in a data handling system comprising a plurality of message processing elements interconnected by a routing network said routing network transferring messages among said message processing elements in accordance with a routing synchronizing signal, the message processing element including:

A. message processing means for generating messages in accordance with a processing element synchronizing signal; and B. interface means for coupling messages between said message processing means and said routing network, said interface means including i. a message transmitter for transmitting a buffered message to said routing network in accordance with said routing synchronizing signal, said message transmitter generating a transmitter flow control signal to control transfer of buffered messages thereto;
ii. a buffer for receiving a message from said message processing means in response to a synchronized transmitter flow control signal and for supplying said received message to said message transmitter as a buffered message; and iii. a synchronizer for generating said synchronized transmitter flow control signal in response to said transmitter flow control signal and said processing element synchronizing signal.

162. A message processing element as defined in claim 161 wherein said synchronizer includes:
A. a transmitter flow control signal latch for latching said transmitter flow control signal said transmitter flow control signal latch asserting a latched transmitter flow control signal in response to assertion by said message transmitter of said transmitter flow control signal;
B. a synchronization buffer corrected to said transmitter flow control signal latch for latching said latched transmitter flow control signal in response to said processing element synchronizing signal, thereby to generate said synchronized transmitter flow control signal.

163. A message processing element as defined in claim 162 in which said processing element synchronizing signal is in the form of a series of sequential clock ticks, said synchronization buffer generating said synchronized transmitter flow control signal a predetermined number of clock ticks of said processing element synchronizing signal following assertion by said transmitter flow control signal latch of said latched transmitter flow control signal.

164. A message processing element as defined in claim 163 in which said synchronization buffer if further connected to enable said transmitter flow control signal latch to negate the latched transmitter flow control signal in response to generation of said synchronized transmitter flow control signal b said synchronization buffer.

165. A message processing element as defined in claim 161 further receives messages from said routing network for transfer to said message processing means in accordance with said processing element synchronizing signal, said interface means further including A. a message receiver for receiving a message from said routing network in accordance with said routing synchronizing signal, said message receiver generating a receiver transmitter flow control signal to control transfer of messages therefrom;
B. a buffer for receiving a message from said message receiver in response to a synchronized receiver flow control signal and for supplying said received message to said message processing means in accordance with said processing element synchronizing signal; and c) a synchronizer for generating said synchronized receiver flow control signal in response to said receiver flow control signal and said processing element synchronizing signal.

166. A message processing elemnet as defined in claim 165 wherein said synchronizer includes:
A. a receiver flow control signal latch for latching said receiver flow control signal, said receiver flow control signal latch asserting a latched receiver flow control signal in response to assertion by said message receiver of said receiver flow control signal;
B. a synchronization buffer corrected to said receiver flow control signal latch for latching said latched receiver flow control signal in response to said processing element synchronizing signal, thereby to generate said synchronized receiver flow control signal.

167. A message processing element as defined in claim 166 in which said processing element synchronizing signal is in the form of a series of sequential clock ticks, said synchronization buffer generating said synchronized receiver flow control signal a predetermined number of clock ticks of said processing element synchronizing signal following assertion by said receiver flow control signal latch of said latched receiver flow control signal.

168. A message processing clement as defined in claim 166 in which said synchronization buffer if further connected to enable said receiver flow control signal latch to negate the latched receiver flow control signal in response to generation of said synchronized receiver flow control signal by said synchronization buffer.

169. A data handling system for transferring messages from a message source to a message sink through an interface, said message source generating messages in accordance with a message source synchronizing signal and said message sink receiving messages in accordance with a message sink synchronizing signal, said interface comprising:
A. a message transmitter for transmitting a buffered message to said message sink in accordance with said message sink synchronizing signal, said message transmitter generating a transmitter flow control signal to control transfer of buffered messages thereto;
B. a buffer for receiving a message from said message source in response to a synchronized transmitter flow control signal and for supplying said received message to said message transmitter as a buffered message; and C. a synchronizer for generating said synchronized transmitter flow control signal in response to said transmitter flow control signal and said message source synchronizing signal.

170. A data handling system as defined in claim 169 wherein said synchronizer includes:
A. a transmitter flow control signal latch for latching said transmitter flow control signal, said transmitter flow control signal latch asserting a latched transmitter flow control siganl in response to assertion by said message transmitter of said transmitter flow control signal;
B. a synchronization buffer connected to said transmitter flow control signal latch for latching said latched transmitter flow control signal in response to said message sink synchronizing signal, thereby to generate said synchronized transmitter flow control signal.

171. A data handling system as defined in claim 170 in which said message sink synchronizing signal is in the form of a series of sequential clock ticks, said synchronization buffer generating said synchronized transmitter flow control signal a predetermined number of clock ticks of said message sink synchronizing signal following assertion by said transmitter flow control signal latch of said latched transmitter flow control signal.

172. A data handling system as defined in claim 170 in which said synchronization buffer if further connected to enable said transmitter flow control signal latch to negate the latched transmitter flow control signal in response to generation of said synchronized transmitter flow control signal by said synchronization buffer.

173. A digital computer comprising:
A. a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages including messages of a configuration type;
B. a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper physical root level, nodes in the leaf level being connected to receive messages from and transfer messages to said message generating elements, each node including:
i. a root flag having a root indication condition establishing the node as a logical root and a non-root indication condition;
ii. an up-tree transfer portion for receiving messages from a plurality of message generating elements or a lower-level control network node connected thereto and for generating a message in response thereto, said up-tree transfer portion selectively coupling the message to a higher-level control network node in response to the condition of the root flag;
iii. a root flag condition establishment portion for establishing the condition of the root flag in response to reception by the up-tree transfer portion of a message of the configuration type; and iv. a down-tree transfer portion for selectively receiving messages from a higher-level node or the up-tree transfer portion in response to the condition of the root flag and generating a message in response thereto for transmission to lower-level nodes or message generating elements connected thereto.

174. A computer as defined in claim 173 in which a message of the configuration type includes a height value, and each level of the network has a level height identifier, the root flag condition establishment portion in each node further using the height value from a message of the configuration type as received by the up-tree transfer portion and the level height identifier of the node in establishing the condition of the root flag.

175. A computer as defined in claim 173 in which, in each node:
A. said up-tree transfer portion selectively couples received messages to a higher-level control network node if the root flag has a non-root indication condition; and B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the root flag has a non-root indication condition or the up-tree transfer portion if the root nag has a root indication condition.

176. A computer as defined in claim 175 in which multiple-source messages contain a data portion, each node further including a data processing portion for performing a processing operation in connection with data of contemporaneously-received multiple-source messages.

177. A computer as defined in claim 176 in which message generating elements further generates messages of an abstain message type, the data processing portion of each node, in response to the contemporaneous receipt of a multiple-source message and an abstain message generating a multiple-source message for transfer including processed data corresponding to the data in the received multiple-source message.

178. A computer as defined in claim 177 in which message generating elements further generate messages of an abstain message type, the up-tree transfer portion of each node generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

179. A computer as defined in claim 178 in which each node further includes a root identification signal generating circuit for generating a root identification signal for transfer lo a node constituting its parent in the next higher level, the root identification signal generating circuit generating the root identification signal having a condition corresponding to the condition of its root flag, each node further including an internal abstain message generating circuit connected to the up-tree transfer portion for generating an internal abstain message for transfer to the up-tree transfer portion in response to receipt of a root identification signal indicating that the root flag of the node generating the root identification signal is set.

180. A computer as defined in claim 176 in which the up-tree transfer portion of each of at least some of said nodes further includes:
A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals; and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

181. A computer as defined in claim 180 in which said multiple-source buffer control circuit includes:
A. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

182. A computer as defined in claim 181 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message received by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated therewith is of another message type.

183. A computer as defined in claim 180 in which said a multiple-source buffer control circuit further selectively enables the multiple-source message buffer circuits to couple multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message.

184. A computer as defined in claim 183 in which said multiple-source buffer control circuit includes:
A. a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits in response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

185. A computer as defined in claim 184 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message in response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

186. A computer as defined in claim 176 in which multiple-source messages further include an operation type identification, the data processing portion in each rode performing a selected processing operation as identified by said operation type identification.

187. A computer as defined in claim 186 in which the data processing portion in each node includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion in each node performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

188. A computer as defined in claim 176 in which the down-tree transfer portion in each node, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

189. A computer as defined in claim 186 in which one operation type identification identifies a scan operation, the data processing portion in each node includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

190. A computer as defined in claim 189 in which each node further includes a scan buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification, the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

191. A computer as defined in claim 190 in which the down-tree portion includes:A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit responsive to the condition of the root flag for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals.

192. A computer as defined in claim 189 in which A. said up-tree portion includes:
i an up-tree message output terminal for transmitting a message to a higher-level node;
ii a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
iii. a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion to selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up tree message output terminal;
and B. said down-tree portion includes:
i. a down-tree message input terminal for receiving a message from a higher-level node;
ii. a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal, the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

193. A computer as defined in claim 189 in which:
A. the up-scan control portion enables:
i. the data h one multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data and ii. the up-tree data processing portion to generate processed data in response to data from multiple-source messages received by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one down-tree message output terminal for inclusion in a multiple-source message transmitted thereby, and ii. the down-tree data processing portion to generate processed data in response to data from the scan buffer and a multiple-source message received by said down-tree message input terminal, the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

194. A computer as defined in claim 193 in which the down-scan control portion further operates in response to the condition of the root flag to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal, both for inclusion in a multiple-source message transmitted thereby.

195. A computer as defined in claim 193 in which each multiple-source message identifying a scan operation further identifies a scan direction, A. the up-tree portion further includes a source select circuit connected to said up-tree message input terminals, said scan buffer and said up-tree data processing portion for selectively coupling data received from one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the down-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

196. A computer as defined in claim 192 in which each multiple-source message further includes a segment identifier having predetermined values, each node further includes a segment nag having selected conditions, said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up-tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the down-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

197. A node for use in a digital computer comprising a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages including messages of a configuration type and a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper physical root level, nodes in the leaf level being connected to receive messages from and transfer messages to said message generating elements, the node including:

A. a root flag having a root indication condition establishing the node as a logical root and a non-root indication condition;
B. an up-tree transfer portion for receiving messages from a plurality of message generating elements or a lower-level control network node connected thereto and for generating a message in response thereto, said up-tree transfer portion selectively coupling the message to a higher-level control network node in response lo the condition of the root flag;
C. a root flag condition establishment portion for establishing the condition of the root nag in response to reception by the up tree transfer portion of a message of the configuration type; and D. a down-tree transfer portion for selectively receiving messages from a higher-level node or the up tree transfer portion in response to the condition of the root flag and generating a message in response thereto for transmission lo lower-level nodes or message generating elements connected thereto.

198. A node as defined in claim 197 in which a message of the configuration type includes a height value, and each level of the network has a level height identifier, the root flag condition establishment portion further using the height value from a message of the configuration type as received by the up-tree transfer portion and the level height identifier of the node in establishing the condition of the root flag.

199. A node as defined in claim 197 in which:
A. said up-tree transfer portion selectively couples received messages to a higher-level control network node if the root flag has a non-root indication condition and B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the root nag has a non-root indication condition or the up-tree transfer portion if the root flag has a root indication condition.

200. A node as defined in claim 199 in which multiple-source messages contain a data portion, said node further including a data processing portion for performing a processing operation in connection with data of contemporaneously-received multiple-source messages.

201. A node as defined in claim 200 in which the data processing portion, in response to the contemporaneous receipt of a multiple-source message and an abstain message, generates a multiple-source message for transfer including processed data corresponding to the data in the received multiple-source message.

202. A node as defined in claim 201 in which the up-tree transfer portion generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

203. A node as defined in claim 202 further including a root identification signal generating circuit for generating a root identification signal for transfer to a node constituting its parent in the next higher level, the root identification signal generating circuit generating the root indentification signal having a condition, corresponding to the condition of its root flag, the node further including an internal abstain message generating circuit connected to the up-tree transfer portion for generating an internal abstain message for transfer to the up-tree transfer portion in response to receipt of a root identification signal indicating that the root nag of the node generating the root identification signal is set.

204. A node as defined in claim 200 in which the up-tree transfer portion includes A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals; and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

205. A node as defined in claim 204 in which said multiple-source buffer control circuit includes:
A. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

206. A node as defined in claim 205 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message received by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated therewith is of another message type.

207. A node as defined in claim 206 in which said a multiple-source buffer control circuit further selectively enables the multiple-source message buffer circuits to couple multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message.

208. A node as defined in claim 207 in which said multiple-source buffer control circuit includes:
A. a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits in response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

209. A node as defined in claim 208 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message in response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

210. A node as defined in claim 200 in which multiple-source messages further include an operation type identification, the data processing portion performing a selected processing operation as identified by said operation type indentification.

211. A node as defined in claim 210 in which the data processing portion includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

212. A node as defined in claim 211 in which the down-tree transfer portion, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

213. A node as defined in claim 210 in which one operation type identification identifies a scan operation, the data processing portion includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

214. A node as defined in claim 213 further including a scan buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification, the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

215. A node as defined in claim 214 in which the down-tree portion includes:
A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit responsive to the condition of the root flag for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals.

216. A node as defined in claim 213 in which A. said up-tree portion includes:

i. an up-tree message output terminal for transmitting a message to a higher-level node;
ii. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
iii. a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion lo selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up-tree message output terminal;
and B. said down-tree portion includes:
i. a down-tree message input terminal for receiving a message from a higher-level node;
ii. a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal, the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

217. A node as defined in claim 216 in which:
A. the up-scan control portion enables:
i the data in one multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data, and ii. the up-tree data processing portion to generate processed data in response to data from multiple-source messages received by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one down-tree message output terminal for inclusion in a multiple-source message transmitted thereby, and ii. the down-tree data processing portion to generate processed data in response to data from the scan buffer and a multiple-source message received by said down-tree message input terminal the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

218. A node as defined in claim 217 in which the down-scan control portion further operates in response to the condition of the root flag to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal, both for inclusion in a multiple-source message transmitted thereby.

219. A node as defined in claim 217 in which each multiple-source message indentifying a scan operation further identifies a scan direction, A. the up-tree portion further includes a source select circuit connected to said up-tree message input terminals, said scan buffer and said up-tree data processing portion for selectively coupling data received form one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the down-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

220. A node as defined in claim 216 in which each multiple-source message further includes a segment identifier having predetermined values, said node further including a segment flag having selected conditions, said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up-tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the down-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

221. A digital computer comprising:
A. a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages of a plurality of diverse types;
B. a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper root level, nodes in the leaf level being connected to receive messages from and transfer messages to said message generating elements, each node including:
i. an up-tree transfer portion for generating a message in response to contemporanously received messages for receiving messages from lower-level nodes or message generating elements connected thereto, said up-tree transfer portion buffering at least one of the contemporaneously-received messages if they are of different types; and ii. a down-tree transfer portion for receiving messages from a higher-level node or from the up-tree transfer portion for generating messages for transmission to a lower-level node or message generating element in response thereto.

222. A computer as defined in claim 221 in which, in each node:
A. said up-tree transfer portion selectively couples received messages to a higher-level control network node if the node is not at root level;
B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the node is not at the root level or the up-tree portion if the node is at the root level.

223. A computer as defined in claim 221 in which said messages include a single-source type and a multiple-source type, the up-tree transfer portion of at least some of said nodes including:
A. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;

B. a plurality of multiple-source message buffer circuits each connected to a message input terminal for selectively buffering multiple-source messages received by one of the message input terminals;
C. an up-tree output message generator for generating an up-tree output message for transmission to a higher-level node; and D. an up-tree transfer portion controller for enabling said up-tree output message generator lo generate an up-tree output message and for controlling said multiple-source message buffer circuits to selectively buffer multiple-source messages received at one of said message input terminals if a single-source message is received at another message input terminal.

224. A computer as defined in claim 223 in which multiple-source messages contain a data portion each node further including a data processing portion for performing a processing operation in connection with data of contemporaneously-received multiple-source messages.

225. A computer as defined in claim 224 in which message generating elements further generates messages of an abstain message type, the data processing portion of each node, in response to the contemporaneous receipt of a multiple-source message and an abstain message generating a multiple-source message for transfer including processed data corresponding to the data in the received multiple-source message.

226. A computer as defined in claim 225 in which message generating elements further generate messages of an abstain message type, the up-tree transfer portion of each node generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

227. A computer as defined in claim 224 in which the up-tree transfer portion of each of at least some of said nodes further includes:
A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals; and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

228. A computer as defined in claim 227 in which said multiple-source buffer control circuit includes:
A. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

229. A computer as defined in claim 228 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message received by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated therewith is of another message type.

230. A computer as defined in claim 229 in which said a multiple-source buffer control circuit further selectively enables the multiple-source message buffer circuits to couple multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message.

231. A computer as defined n claim 230 in which said multiple-source buffer control circuit includes:
A. a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits in response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

232. A computer as defined in claim 231 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message in response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

233. A computer as defined in claim 225 in which multiple-source messages further include an operation type identification, the data processing portion in each node performing a selected processing operation as identified by said operation type identification.

234. A computer as defined in claim 233 in which the data processing portion in each node includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion in each node performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

235. A computer as defined in claim 234 in which the down-tree transfer portion in each node, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

236. A computer as defined in claim 233 in which one operation type identification identifies a scan operation, the data processing portion in each node includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

237. A computer as defined in claim 236 in which each node further includes a said buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification, the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

238. A computer as defined in Claim 237 in which the down-tree portion includes:A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals if the node is at the root level.

239. A computer as defined in claim 236 in which A. said up-tree portion includes:
i. an up-tree message output terminal for transmitting a message to a higher-level node;
ii. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
iii. a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion to selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up-tree message output terminal;
and B. said down-tree portion includes:
i a down-tree message input terminal for receiving a message from a higher-level node;
ii a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal, the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

240. A computer as defined in claim 239 in which:
A. the up scan control portion enables:
i. the data in one multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data, and ii. the up-tree data processing portion to generate processed data in response to data from multiple-source messages received by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one do up-tree message output terminal for inclusion in a multiple-source message transmitted thereby; and ii the down-tree data processing portion to generate processed data in response lo data from the scan buffer and a multiple-source message received by said down-tree message input terminal the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

241. A computer as defined in claim 240 in which the do up-scan control portion further operates to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal if the node is at the root level both for inclusion in a multiple-source message transmitted thereby.

242. A computer as defined in claim 240 m which each multiple-source message identifying a scan operation further identifies a scan direction, A. the up-tree portion further includes a source select circuit connected to said up-tree message input terminals, said scan buffer and said up-tree data processing portion for selectively coupling data received from one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the down-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

243. A computer as defined in claim 239 in which each multiple-source message further includes a segment identifier having predetermined values, each node further includes a segment flag having selected conditions, said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up-tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the down-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

244. A node for use in a digital computer comprising a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages of a plurality of diverse types, and a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper physical root level, nodes in the leaf level being connected to receive messages from and transfer messages to said message generating elements, the node including:
A. an up-tree transfer portion for generating a message in response to contemporanously received messages for receiving messages from lower-level nodes or message generating elements connected thereto, said up-tree transfer portion buffering at least one of the contemporaneously-received messages if they are of different types; and B. a down-tree transfer portion for receiving messages from a higher-level node or from the up-tree transfer portion for generating messages for transmission to a lower-level node or message generating element in response thereto.

245. A node as defined to claim 244 in which:
A. said up-tree transfer portion selectively couples received messages to a nigher-level control network node if the node is not at the root level;
B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the node is not at the root level or the up-tree transfer portion if the node is at the root level.

246. A node as defined in claim 244 in which said messages include a single-source type and a multiple-source type, the up-tree transfer portion of at least some of said nodes including A. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each connected to a message input terminal for selectively buffering multiple-source messages received by one of the message input terminals;
C. an up-tree output message generator for generating an up-tree output message for transmission to a higher-level node, and D. an up-tree transfer portion controller for enabling said up-tree output message generator to generate an up-tree output message and for controlling said multiple-source message buffer circuits to selectively buffer multiple-source messages received at or e of said message input terminals if a single-source message is received at another message input terminal.

247. A node as defined in claim 246 in which multiple-song a processing operation in connection with data of contemporaneously-received multiple-source messages.

248. A node as defined in claim 247 in which message generating elements further generates messages of an abstain message type, the data processing portion, in response to the contemporaneous receipt of a multiple-source message and an abstain message generating a multiple-source message for transfer including processed data corresponding to the data in the receiving multiple-source message.

249. A node as defined in claim 248 in which message generating elements further generate messages of an abstain message type, the up-tree transfer portion generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

250. A node as defined in claim 247 in which the up-tree transfer portion of each of at least some of said nodes further includes:

A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals; and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

251. A node as defined in claim 250 in which said multiple-source buffer control circuit includes:
A. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

252. A node as defined in claim 251 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message received by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated therewith is of another message type.

253, A node as defined in claim 252 in which said a multiple-source buffer control circuit further selectively enables the multiple source message buffer circuits to couple multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message,

254, A node as defined in claim 253 in which said multiple-source buffer control circuit includes:
A, a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits ill response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

255. A node as defined in claim 254 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message m response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

256. A node as defined in claim 247 in which multiple-source messages further include an operation type identification, the data processing portion performing a selected processing operation as identified by said operation type identification,

257. A node as defined in claim 256 in which the data processing portion includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

258. A node as defined in claim 257 in which the down-tree transfer portion, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

259. A node as defined in claim 256 in which one operation type identification identifies a scan operation, the data processing portion includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

260. A node as defined in claim 259 further including a scan buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

261. A node as defined in claim 259 in which the down-tree portion includes:
A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals if the node is at the root level.

262. A node as defined in claim 259 in which A. said up-tree portion includes:
i. an up-tree message output terminal for transmitting a message to a higher-level node;
ii. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
iii. a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion to selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up-tree message output terminal, B. said down-tree portion includes:
i. a down-tree message input terminal for receiving a message from a higher-level node;
ii. a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal, the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

263. A node as defined in claim 202 in which:
A. the up-scan control portion enables:
i the data in one multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data, and ii. the up-tree data processing portion to generate processed data in response to data from multiple-source messages reserved by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one do up-tree message output terminal for inclusion in a multiple-source message transmitted thereby, and ii the town-tree data processing portion to generate processed data in response to data from the scan buffer and a multiple-source message received by said down-tree message input terminal, the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

264. A node as defined in claim 263 in which the down-scan control portion higher operates to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal if the node is at the root level, both for inclusion in a multiple-source message transmitted thereby.

265. A node as defined in claim 263 in which each multiple-source message identifying a scan operation further identifies a scan direction, A. the up-tree portion further includes a source select circuit connected to said up-tree message input terminals, said scan buffer and said up-tree data processing portion for selectively coupling data received from one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the do up-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

266. A node as defined in claim 262 in which each multiple-source message further includes a segment identifier having predetermined values, the node further includes a segment flag having selected conditions, said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up-tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the down-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

267. A digital computer comprising A a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages of a plurality of diverse types;
B. a network for transferring messages among said message generating elements, said network comprising a plurality of nodes selected in a tree pattern in a series of levels from a lower leaf level to an upper root level, nodes h the leaf level being connected to receive messages from and transfer messages to said message generating elements, each node including:
i an up-tree transfer portion for generating a message in response to contemporanously received messages for receiving messages from lower-level nodes or message generating elements connected thereto, said up-tree transfer portion buffering a received message in response to a flow-control signal;
and ii. a down-tree transfer portion for receiving messages from a higher-level node or from the up-tree transfer portion for generating messages for transmission to a lower-level node or message generating element in response thereto, said down-tree transfer portion further generating the flow-control signal in response to flow-control information in the message received thereby.

268. A computer as defined in claim 267 in which, in each node:
A. said up-tree transfer portion selectively couples received messages to a higher-level control network node if the node is not at the root level;
B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the node is not at the root level or the up-tree transfer portion if the node is at the root level.

269. A computer as defined in claim 267 in which said messages include a plurality of message types, the up-tree transfer portion of at least some of said nodes including:
A. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of message buffer circuits each connected to a message input terminal for selectively buffering messages received by one of the message input terminals;
C. an up-tree output message generator for generating an up-tree output message for transmission to a higher-level node; and D. an up-tree transfer portion controller for enabling said up-tree output message generator to generate an up-tree output message and for controlling said message buffer circuits to selectively buffer messages of a predetermined type received at one of said message input terminals if a message of another type is received at another message input terminal and to enable said down-tree transfer portion to generate the flow-control signal in response thereto.

270. A computer as defined in claim 269 in which multiple-source messages contain a data portion, each node further including a data processing portion for performing a processing operation in connection with data of contemporaneously-received multiple-source messages.

271. A computer as defined in claim 270 in which message generating elements further generates messages of an abstain message type, the data processing portion of each node, in response to contemporaneous receipt of a multiple-source message and an abstain message generating a multiple-source message for transfer including processed data corresponding to the data in the received multiple-source message.

272. A computer as defined in claim 271 in which message generating elements further generate messages of an abstain message type, the up-tree transfer portion of each node generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

273. A computer as defined in claim 270 in which the up-tree transfer portion of each of at least some of said nodes further includes:
A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating element;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals; and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

274. A computer as defined in claim 273 in which said multiple-source buffer control circuit includes:
A a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

275. A computer as defined in claim 274 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message relieved by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated therewith is of another message type.

276. A computer as defined in claim 275 in which said a multiple-source buffer control circuit further selectively enables the multiple-source message buffer circuits to couples multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message.

277. A computer as defined in claim 276 in which said multiple-source buffer control circuit includes:
A. a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits in response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

278. A computer as defined in claim 277 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message in response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

279. A computer as defined in claim 270 in which multiple-source messages further include an operation type identification, the data processing portion in each node performing a selected processing operation as identified by said operation type identification.

280. A computer as defined in claim 279 in which the data processing portion in each node includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion in each node performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

281. A computer as defined in claim 280 in which the down-tree transfer portion in each node, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

282. A computer as defined in claim 279 in which one operation type identification identifies a scan operation, the data processing portion in each node includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

283. A computer as defined in claim 282 in which each node further includes a scan buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification, the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

284. A computer as defined to claim 283 in which the down-tree portion includes:A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals if the node is at the root level.

285. A computer as defined in claim 282 in which A. said up-tree portion includes:
i. an up-tree message output terminal for transmitting a message to a higher-level node;
u a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
11. a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion to selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up-tree message output terminal;
and B. said down-tree portion includes:
i a down-tree message input terminal for receiving a message from a higher-level node;
ii. a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal, the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

286. A computer as defined in claim 285 in which:
A. the up-scan control portion enables:
i the data in one multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data, and ii. the up-tree data processing portion to generate processed data in response to data from multiple-source messages received by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one do up-tree message output terminal for inclusion in a multiple-source message transmitted thereby, and ii. the down-tree data processing portion to generate processed data in response to data from the scan buffer and a multiple-source message received by said down-tree message input terminal the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

287. A computer as defined in claim 286 in which the down-scan control portion further operates to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal if the node is at the root level both for inclusion in a multiple-source message transmitted thereby.

288. A computer as defined in claim 286 in which each multiple-source message identifying a scan operation further identifies a scan direction, A. the up-tree portion further includes a source select circuit connected to said up-tree message input terminals, said scan buffer and said up-tree data processing portion for selectively coupling data received from one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the down-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

289. A computer as defined in claim 285 in which each multiple-source message further includes a segment identifier having predetermined values, each node further includes a segment flag having selected conditions, said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the do up-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

290. A node for use in a digital computer comprising a plurality of message generating elements each generating and receiving messages, at least one message generating element generating messages of a plurality of diverse types, and a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper physical root level nodes in the leaf level being connected to receive messages from and transfer messages to said message generating elements, the node including A. an up-tree transfer portion for generating a message in response to contemporanously received messages for receiving messages from lower-level nodes or message generating elements connected thereto, said up-tree transfer portion buffering a received message in response to a flow-control signal; and B. a down-tree transfer portion for receiving messages from a higher-level node or from the up-tree transfer portion for generating messages for transmission to a lower-level node or message generating element in response thereto, said down-tree transfer portion further generating the flow-control signal in response to flow-control information in the message received thereby.

291. A node as defined in claim 290 in which:
A. said up-tree transfer portion selectively coupled received messages to a higher-level control network node if the node is not at the root level;
B. said down-tree transfer portion selectively generates messages in response to messages from a higher-level node if the node is not at the root level or the up-tree transfer portion if the node is at the root level.

292. A node as defined in claim 290 in which said messages include a single-source type and a multiple-source type, the up-tree transfer portion of at least some of said nodes including A a plurality of up-tree message input terminals each for receiving a message from a lower level node or a message generating element;
B. a plurality of multiple-source buffer circuits each connected to a message input terminal for selectively buffering multiple-source messages received by one of the message input terminals;
C. an up-tree output message generator for generating an up-tree output message for transmission to a higher-level node; and D. an up-tree transfer portion controller for enabling said up-tree output message generator to generate an up-tree output message and for controlling said message buffer circuits to selectively buffer messages of a predetermined type received at one of said message input terminals if a message of another type is received at another message input terminal and to enable said down-tree transfer portion to generate the flow-control signal in response thereto.

293. A node as defined in claim 292 in which multiple-source messages contain a data portion, each node further including a data processing portion for performing a processing operation in connection with data of contemporaneously-received multiple-source messages.

294. A node as defined in claim 293 in which message generating elements further generates messages of an abstain message type, the data processing portion, in response to the contemporaneous receipt of a multiple-source message and an abstain message generating a multiple-source message for transfer including processed data corresponding to the data in the received multiple-source message.

295. A node as defined in claim 293 in which message generating elements further generate messages of an abstain message type, the source transfer portion generating an abstain message in response to contemporaneous receipt of abstain messages from all nodes or message generating elements from which it receives messages.

296. A node as defined in claim 293 in which the up-tree transfer portion of each of at least some of said nodes further includes:
A. a plurality of message input terminals each for receiving a message from a lower-level node or a message generating clement;
B. a plurality of multiple-source message buffer circuits each for buffering multiple-source messages received by one of the message input terminals and C. a multiple-source buffer control circuit for enabling said multiple-source message buffers to selectively buffer messages received by the message input terminals.

297. A node as defined in claim 296 in which said multiple-source buffer control circuit includes:
A. a plurality of message type determination circuits for determinating the types of messages received by respective message input terminals; and B. a storage enabling circuit for enabling respective multiple-source message buffer circuits to buffer messages in response to the message types of messages concurrently received by the message input terminals.

298. A node as defined in claim 297 in which said storage enabling circuit enables a multiple-source message buffer circuit to buffer a message received by its respective message input terminal if the associated message type determination circuit determines that the message received by the message input terminal is a multiple-source message and the other message type determination circuit determines that the message received by the message input terminal associated there with is of another message type.

299. A node as defined in claim 298 in which said a multiple-source buffer control circuit further selectively enables the multiple-source message buffer circuits to couple multiple-source messages buffered thereby to the data processing portion in response to receipt at the other message input terminal of a multiple-source message.

300. A node as defined in claim 299 in which said multiple-source buffer control circuit includes:
A a plurality of buffer status indicators each for indicating whether an associated multiple-source message buffer circuit is buffering a message;
B. a plurality of message type determination circuits for determining the types of messages received by respective message input terminals; and C. a transfer enabling circuit for enabling respective multiple-source message buffer circuits to transfer messages from said multiple-source message buffer circuits in response to the conditions of said buffer status indicators and the message types of messages received by the message input terminals as determined by the associated message type determination circuits.

301. A node as defined in claim 300 in which said transfer enabling circuit enables a buffer associated with one message input terminal to transfer a message in response to the coincidence of its buffer status indicator indicating it is buffering a message and the message type determination circuit associated with the other message input terminal indicating it is receiving a multiple-source message.

302. A node as defined in claim 293 in which multiple-source messages further include an operation type identification, the data processing portion performing a selected processing operation as identified by said operation type identification.

303. A node as defined in claim 302 in which the data processing portion includes an up-tree data processing portion and one operation type identification identifies a reduce operation, the up-tree data processing portion performing a data processing operation in connection with data from multiple-source messages received by the up-tree transfer portion for generating processed data that is included in the message generated by the up-tree transfer portion.

304. A node as defined in claim 303 in which the down-tree transfer portion, in response to receipt of a multiple-source message in which the operation type identification identifies a reduce operation, transmits messages for transmission to lower-level nodes or message generating elements connected thereto including data from the received multiple-source message.

305. A node as defined in claim 302 in which one operation type identification identifies a scan operation, the data processing portion includes an up-tree data processing portion and a down-tree data processing portion each for performing a data processing operation in connection with data of multiple-source messages received by the respective up-tree transfer portion and down-tree transfer portion.

306. A node as defined in claim 305 further including a scan buffer connected to said up-tree data processing portion and said down-tree data processing portion, said up-tree data processing portion selectively generating intermediate data for storage in said scan buffer in response to operator type identification, the down-tree data processing portion performing a data processing operation in connection with the intermediate data stored in the scan buffer and multiple source messages received by the down-tree transfer portion as determined by the operation type identification of the respective multiple-source message.

307. A node as defined in claim 306 in which the down-tree portion includes:
A. a plurality of message output terminals each for transmitting a message to a lower-level node or a message generating element;
B. an output message control circuit for enabling said down-tree data processing portion to generate data for selective inclusion in messages transmitted through selected ones of said message output terminals if the node is at the root level.

308. A node as defined in claim 305 in which A. said up-tree portion includes:
i. an up-tree message output terminal for transmitting a message to a lower-level node;
ii. a plurality of up-tree message input terminals each for receiving a message from a lower-level node or a message generating element;
iii a scan buffer connected to said up-tree data processing portion, and iv. an up-scan control portion for enabling said up-tree data processing portion to selectively generate intermediate data for storage in said scan buffer in response to operator type identification and up-tree information to be transmitted in a message through said up-tree message output terminal;
and B. said down-tree portion includes:

i a down-tree message input terminal for receiving a message from a higher-level node;
ii a plurality of down-tree message output terminals each for transmitting a message to the lower-level node or message generating element;
iii. a down-scan control portion for enabling said down-tree data processing portion to selectively generate data from either said scan buffer or from data received in a multiple-source message received by said down-tree message input terminal the generated data being coupled to respective ones of said down-tree message output terminals for transmission thereby.

309. A node as defined in claim 308 in which:
A. the up-scan control portion enables:
i. the data in one Multiple-source message that was received from one up-tree message input terminal to be stored in the scan buffer as intermediate data, and ii. the up-tree data processing portion to penerate processed data in response to data from multiple-source messages received by both up-tree message input terminals; and B. the down-scan control portion enables:
i. the data in the scan buffer to be coupled to one down-tree message output terminal for inclusion in a multiple-source message transmitted thereby; and ii. the down-tree data processing portion to generate processed data in response to data from the scan buffer and a multiple-source message received by said down-tree message input terminal, the processed data generated by the down-tree data processing portion being coupled to the other down-tree message output terminal for inclusion in a multiple-source message transmitted thereby.

310. A node as defined in claim 309 in which the down-scan control portion further operates to couple predetermined data to one down-tree message output terminal and the data in the scan buffer to the other down-tree message output terminal if the node is at the root level both for inclusion in a multiple-source message transmitted thereby.

311. A node as defined in claim 309 in which each multiple-source message identifying a scan operation further identifies a scan direction A. the up-tree portion further includes a source select circuit connected to said up tree message input terminals said scan buffer and said up-tree data processing portion for selectively coupling data received from one or the other of said up-tree message input terminals as selected by said scan direction to be stored in the scan buffer as intermediate data;
B. said down-tree portion further includes a transmit select circuit for selecting the down-tree message output circuit to receive the data in the scan buffer or the processed data generated by the down-tree data processing portion as selected by the scan direction of the multiple-source message received by the down-tree message input terminal.

312. A node as defined in claim 308 in which each multiple-source message further includes a segment identifier having predetermined values, the node further includes a segment flag having selected conditions said up-scan control portion conditioning circuit for conditioning said segment flag in response to reception by a selected up-tree message input terminal of a multiple-source message whose segment identifier has one of said predetermined values, and generating said intermediate information and said up-tree information in response to said segment flag, the down-scan control portion further controlling said down-tree data processing portion in response to said segment flag.

313. A digital computer comprising:
A. a plurality of message generating elements each generating and receiving messages, each message including a flow-control nag having selected conditions, each message generating element controlling generation of messages in response to the condition of flow-control flags of messages received thereby; and B. a network for transferring messages among said message generating elements, said network comprising a plurality of nodes interconnected in a tree pattern in a series of levels from a lower leaf level to an upper root level, the nodes generating messages for transfer to a higher-level node whose flow-control flag is conditioned in response to conditions of flow-control flags of messages contemporaneously received from lower-level nodes, the nodes also generating messages for transmission to lower-level nodes whose flow-control flags are conditioned in response to conditions of flow-control flags of messages received from the higher-level nodes, thereby to provide messages to all of said message generating elements having flow-control flags of uniform conditions to contemporaneously control generation of messages by all of said message generating elements.

314. A computer as defined in claim 313 in which each message generating element generates messages of a plurality of message types including an idle type, said message generating elements generating idle messages in response to receipt of messages whose flow-control flags have a predetermined one of said selected conditions.

315. A computer as defined in claim 314 in which each message generating element includes A. a network interface connected to a node of said network for receiving message information and for selectively generating messages for transfer to the node of said network connected thereto, said network interface generating idle messages in the absence of message information or response to receipt of messages from the node in which the flavor control flag has said predetermined condition; and B. a processor for generating said message information for transfer so said network interface to initiate generation thereby of messages.

316. A computer as defined in claim 315 in which said processor further receives message information from said network interface representing messages received thereby from the node connected thereto

317. A computer as defined in claim 314 in which said interface includes:
A. a message transmitting portion connected to a node for transmitting messages in response to message information from said processor, said message transmitting portion controlling the conditioning of the flow-control flags in the messages transmitted thereby;
B a message receiving terminal connected to a node for receiving messages and generating therefrom message information;

C. a buffer for buffering message information representing messages received thereby from the node connected thereto;
D. a buffer control circuit for controlling the storage of message information from said message receiving terminal in said buffer and for controlling, in response to the storage of message information in said buffer, the conditioning by said message transmitting portion of the flow-control flags in the messages transmitted thereby.

318. A computer as defined in claim 317 in which said processor operates in a plurality of operating modes and each message includes a mode identifier having a plurality of values each associated with an operating mode, said interface including:
A. a plurality of buffers each associated with an operating mode for buffering message information; and B. said buffer control circuit for enabling message information from each message to be buffered in one of said buffers in response to the value of the mode identifier in the associated message.

319. A digital computer comprising:
A. a plurality of processing elements each performing data processing and information communications operations, said processing elements also performing diagnostic operations in response to diagnostic operation requests and generating diagnostic results in response thereto;
B. at least one diagnostic processor for generating diagnostic requests and for receiving diagnostic results;
C. an information communications network connected to said processing elements for facilitating the transfer of information during information communications operations; and D. a diagnostic network connected to said processing elements said diagnostic processor for transferring diagnostic requests from said diagnostic processor to said processing elements and for transferring diagnostic results from said processing elements to said diagnostic processor.

320. A digital computer as defined in claim 319 in which each processing element performs data processing and information communications operations in response to commands, said computer further including:
A. a control processor for generating commands for transmission over said information communications network to said processing elements, said control processor further receiving diagnostic requests from said diagnostic network and performing diagnostic operations in response thereto to generate diagnostic results for transfer to said diagnostic processor over said diagnostic network; and B. said processing elements receiving said commands from said information communications network and performing data processing and information communications operations in response thereto.

321. A digital computer as defined in claim 319 in which:

A. said processing elements, in performing information communications operations, generate messages for transfer over said information communications network;
B. said information communications network includes a plurality of communications nodes for transferring said messages among said processing elements, said communications nodes further receiving receiving diagnostic requests from said diagnostic network and performing diagnostic operations in response thereto to generate diagnostic results for transfer to said diagnostic processor over said diagnostic network.

322. A digital computer as defined in claim 321 in which:
A. the communications nodes of said information communications network are connected in a series of stages, including an initial stage connected to said processing elements; and B. said diagnostic network includes a plurality of diagnostic network nodes connected in a series of stages, each stage being associated with a stage of the communications network or the processing elements, said diagnostic processor being connected to a diagnostic network node at a selected stage of said diagnostic network, said diagnostic requests including diagnostic request address information to identify a path through said diagnostic network nodes to selected communications nodes and processing elements, each said diagnostic network node upon receipt of a diagnostic request being responsive to the diagnostic request address information to selectively couple said diagnostic request along the path to another stage or to a communications node or processing element.

323. A digital computer as defined in claim 322 wherein at least some stages of said information communications network comprise a plurality of communications nodes, the stages of said diagnostic network each including A. a stage identifier node; and B. a plurality of stage request distribution nodes connected to establish a path between said stage identifier node and the communications nodes of a corresponding stage of the information communications network or the processing elements in response to the diagnostic request address information.

324. A digital computer as defined in claim 323 wherein said stage request distribution nodes establish paths to selected ones of the communications nodes of a corresponding stage of the information communications network or the processing elements to facilitate the transfer of diagnostic requests to the communications nodes or the processing elements to, and the transfer of diagnostic results therefrom, in parallel.

325. A digital computer as defined in claim 324 in which:
A. the stage request distribution nodes of at least one stage form a distribution tree having a root level whose stage request distribution node is connected to the stage identifier and a leaf level having at least one stage request distribution node connected to a communications node or a processing element; and B. the stage identifier node selectively couples diagnostic requests to a stage identifier node of a subsequent stage in the series or to the stage request distribution node of the root level of the distribution tree associated with the tree in accordance with the requests' diagnostic request address information.

326. A digital computer as defined in claim 325 the stage identifier node of each stage includes:
A. an input connection over which the stage identifier node receives diagnostic requests from, and couples diagnostic results to, the stage identifier node of a proceeding stage or the diagnostic processor; and B. a plurality of output connections over which the stage identifier node selectively couples diagnostic requests to, and receives diagnostic results from, the stage identifier node of a successive stage or the stage's distribution tree; and C. a coupling control circuit for controlling the coupling of diagnostic requests and diagnostic results between the input connection and selected ones of said output corrections in response to diagnostic request address information thereby to facilitate the selective parallel transfer by the stage identifier node of one stage of a diagnostic request to and diagnostic results from (i) a stage identifier node of subsequent stage, (ii) the stage's distribution tree, or (iii) both a stage identifier node of subsequent stage and the stage's distribution tree, in accordance with the diagnostic request address information of the diagnostic request.

327. A digital computer as defined in claim 326 wherein said diagnostic request address information of a diagnostic request can also enable a coupling control circuit to disable coupling of a diagnostic request to, and diagnostic results from, both the stage identifier node of a subsequent stage and the stage's distribution tree.

328. A digital computer as defined in claim 326 wherein each stage request distribution node of said distribution tree includes:
A. an input connection over which the stage identifier node receives diagnostic requests from, and couples diagnostic results to, the stage identifier node of a higher level in the distribution tree or, for a stage request distribution node at the root level, the stage's stage identifier node; and B. a plurality of output connections over which the stage identifier node selectively couples diagnostic requests to, and receives diagnostic results from, the a stage request distribution node of a lower level in the distribution tree, stage identifier node of a successive stage or the stage's distribution tree; and C. a coupling control circuit for controlling the coupling of diagnostic requests and diagnostic results between the input connection and selected ones of said output connections in response to diagnostic request address information, thereby to facilitate the selective parallel transfer by the stage request distribution node of a diagnostic request to and diagnostic results from selected ones of the stage request distribution nodes in the lower level, in accordance with the diagnostic request address information of the diagnostic request.

329. A digital computer as defined in claim 328 wherein said diagnostic request address information of a diagnostic request can also enable a coupling control circuit to disable coupling of a diagnostic request to, and diagnostic results from, all diagnostic request distribution nodes.

330. A computer comprising:
A. a plurality of processing nodes each for receiving processing requests and generating in response processed data;
B. a control node for generating processing requests for transfer to selected ones of said processing nodes as identified by associated request address information, and for receiving processed data in response, the request address information identifying selected ones of said processing nodes to receive a processing request in parallel; and C. a request distribution network including a plurality of request distribution nodes connected in a plurality of levels to form a tree-structure, including an upper root level and a lower leaf level each request distribution node being connected to receive processing requests from, and to couple processed data to, a parent, the parent of the request distribution node of the root level comprising the control node, and each request distribution node being further connected to couple processing requests to and receive processed data from, selected children, the children of the request distribution nodes of the leaf level comprising the processing nodes, each request distribution node, in response to request address information received from its parent, identifying selected ones of its children and thereafter coupling further request address information which it receives and processing requests in parallel to its children, and coupling processed data which it receives from its children to its parent.

331. A computer as defined in claim 330 in which each request distribution node comprises:
i. an input for receiving processing requests, including associated request address information from, and for coupling processed data to, a parent;
ii. a plurality of outputs each for coupling processing requests and request address information to, and for receiving processed data from, a child;
iii. an address control portion for selectively identifying ones of said children as identified by received address information; and iv. a data coupling control portion for coupling processing requests and associated request address information, in parallel, from said input to the outputs associated with the children that are identified by said address control portion, and for coupling processed data, in parallel, from the outputs associated with the children that are identified by said address control portion to said input.

332. A computer as defined in claim 331 in which said address control portion comprises:
A. a flag register including a plurality of flags each associated with an output, each flag generating an enable signal to identify the associated output if it is in a select condition, and not generating the enable signal if it is not in the select condition;
B. a nag register control circuit for controlling the condition of successive flags of said flag register in response to control signals representative of the request address information from the parent request distribution node.

333. A computer as defined in claim 332 in which said flags of said flag register are updated in parallel in response to a timing signal said flag register control circuit comprising:

A. a flag condition selection circuit including a plurality of condition selectors each associated with a flag for selectively coupling either a flag conditioning signal from the parent to enable the flag to update its condition in response to the flag conditioning signal or the flags enable signal to enable the flag to maintain its condition, in response to the timing signal;
B. an address flag token shift register including a plurality of token stages each for controlling a condition selector of the flag condition selection circuit, said address flag token shift register receiving a token from said parent and selectively shifting said token through successive token stages in response to successive ticks of the timing signal, a token stage generating said flag condition signal when it has the token; and C. an address state control circuit for controlling the shifting of said token through said address flag token shift register in response to the timing signal and a mode signal from the parent.

334. A computer as defined in claim 333 wherein said flag register control circuit further includes a token coupling circuit for coupling the token from the last stage of the address flag token shift register to the identified outputs, thereby enabling the children attached thereto to receive the token in parallel to facilitate the parallel conditioning of their flag registers.

335. A computer as defined in claim 333 wherein said address control portion couples the timing signal and said mode signal received at the input from the parent to all of the output thereby coupling the timing signal and the mode signal to all of the children in parallel.

336. A computer as defined in claim 333 wherein said flag register control circuit further comprises an address data coupling circuit for selectively coupling address data representative of the condition of a flag of said flag register under control of said address flag token shift register and said address state control circuit to the input for transfer to the parent, thereby to transfer a signal representative of the condition of the flag to the parent.

337. A computer as defined in claim 336 in which said address data coupling circuit comprises a series of multiplexer circuits each for coupling as an output signal either the enable signal from a flag of said flag register if the token stage is generating the flag condition signal, or alternatively coupling as the output signal from the preceding multiplexer circuit in the series the output signal.

338. A computer as defined in claim 337 in which said address data coupling circuit further includes a child address data receiving circuit for selectively receiving, through the outputs, address data signals from the children whose output circuits are enabled by said flag register and for coupling a composite address data signal to the first multiplexer circuit in the series, the first multiplexer circuit coupling the composite address data signal as its output signal if the token stage is not generating the flag condition signal, thereby to selectively couple address data signals from the identified children to the parent if none of the token stages of the request distribution node are generating the flag condition signal.

339. A computer as defined in claim 338 in which:
A. the control node generates an expected address data output signal which it couples to the request distribution node of the root level, each request distribution node coupling the expected address data output signal from its parent to of its children so that the request distribution nodes receive the expected address data output signal generally concurrently, and B. for each request distribution node, the address data receiving circuit includes an address data process circuit that processes the address data signals from the children whose output circuits are enabled by the flag register in response to the expected address data output signal from the parent.

340. A computer as defined in claim 339 in which said address data processing circuit includes:
A. a child address data asserted circuit for generating an asserted child address data output signal if the address data output signals from all of the children associated with identified outputs are asserted;
B. a child address data negated circuit for generating a negated child address data output signal if the address data output signals from all of the children associated with identified outputs are negated; and C. a child address data consolidation circuit for selectively coupling the output signal from the child address data asserted circuit or from the child address data negated circuit, as selected by the expected address data output signal, to the first multiplexer circuit in the series as the composite address data signal.

341. A computer as defined in claim 340 in which:
A. the child address data asserted circuit includes:
i. a plurality of OR circuits each associated with a child, each OR circuit generating an asserted child processed address data output signal in response to the assertion of either the address data output signal from the associated child and the complement of the enable signal from the flag which controls the output connected to that child; and ii. a conjunction circuit which generates the asserted child address data output signal in response to the conjunction of the asserted child processed address data output signals from all of the OR
circuits, thereby to generate the asserted child address data output signal only if the child address data asserted circuit is receiving asserted address data out signals from all of the children whose flags are in the selected condition;
B. the child address data negated circuit includes:
i. a plurality of AND circuits each associated with a child, each AND circuit generating a negated child processed address data output signal in response to the negation of bother the address data output signal from the associated child and the enable signal from the flag which controls the output connected to that child; and ii. a disjunction circuit which generates the negated child address data output signal in response to all of the negated child processed address data output signals from all of said AND circuits thereby to generate the negated child address data output signal only if the child address data negated circuit is receiving negated address data out signals from all of the children whose flags are in the selected condition.

342. A computer as defined in claim 331 in which:
A. the control node generates an expected test data output signal which it couples to the request distribution node of the root level, each request distribution node coupling the expected address data output signal from its parent to of its children so that the request distribution nodes receive the expected address data output signal generally concurrently; and B. for each request distribution node, the test data receiving circuit includes a test data processing circuit that processes the test data signals from the children identified by the address control portion in response to the expected test data output signal from the parent.

343. A computer as defined in claim 342 in which said test data processing circuit includes:
A. a child test data asserted circuit for generating an asserted child test data output signal if the test data output signals from all of the children associated with identified outputs are asserted;
B. a child test data negated circuit for generating a negated child test data output signal if the test data output signals from all of the children associated with identified outputs are negated; and C. a child test data consolidation circuit for selectively coupling a composite child test data signal generated as the output signal from the child test data asserted circuit or from the child test data negated circuit as selected by the expected test data output signal to the parent as a composite test data signal.

344. A computer as defined in claim 343 in which said address control portion identifies a child by generating an associated enable signal, each associated with A. the child test data asserted circuit includes:
i. a plurality of OR circuits each associated with a child, each OR circuit generating an asserted child processed test data output signal in response to the assertion of either the test data output signal from the associated child or the complement of the enable signal from the address control portion which controls the output connected to that child; and ii. a conjunction circuit which generates the asserted child test data output signal in response to the conjunction of the asserted child processed test data output signals from all of the OR circuits thereby to generate the asserted child test data output signal only if the child test data asserted circuit is receiving asserted test data out signals from all of the children identified by the address control portion;
B. the child test data negated circuit includes:
i. a plurality of AND circuits each associated with a child, each AND circuit generating a negated child processed test data output signal in response to the negation of bother the test data output signal from the associated child and the enable signal associated with the child; and ii. a disjunction circuit which generates the negated child test data output signal in response to all of the negated child processed test data output signals from all of said AND circuits thereby to generate the negated child test data output signal only if the child test data negated circuit is receiving negated test data out signals from all of the children identified by the address control portion.

345. A computer as defined in claim 343 in which:

A. the control node generates a test data input signal which it couples to the request distribution node of the root level, each request distribution node coupling the lest data input signal from its parent to of its children so that the request distribution nodes receive the test data input signal generally concurrently, and B. swept child test data consolidation circuit further includes a test data selection circuit for selectively coupling, to the parent as the composite test data signal, the test data input signal, if at least one of the children h identified by the address control portion, or the composite child test data signal, if none of the children are identified by the address control portion.

346. A request distribution node for use in a computer comprising a plurality of processing nodes each for receiving processing requests and generating in response processed data, a control node for generating processing requests for transfer to selected ones of said processing nodes as identified by associated request address information, and for receiving processed data in response, the request address information identifying selected ones of said processing nodes to receive a processing request in parallel, and a request distribution network including a plurality of request distribution nodes connected in a plurality of levels to form a tree-structure, including an upper root level and a lower leaf level, each request distribution node being connected to receive processing requests from and to couple processed data to, a parent, the parent of the request distribution node of the root level comprising the control node, and each request distribution node being further connected to couple processing requests to and receive processed data from, selected children, the children of the request distribution nodes of the leaf level comprising the processing nodes, each request distribution node, in response to request address information received from its parent, identifying selected ones of its children and thereafter coupling further request address information which it receives and processing requests i n parallel to its children, and coupling processed data which it receives from its children to its parent, said request distribution node comprising A. an input for receiving processing requests, including associated request address information from, and for coupling processed data to, a parent;
B. a plurality of outputs each for coupling processing requests and request address information to, and for receiving processed data from, a child;
C. an address control portion for selectively identifying ones of said children as identified by received address information and D. a data coupling control portion for coupling processing requests and associated request address information, in parallel, from said input to the outputs associated with the children that are identified by said address control portion and for coupling processed data, in parallel, from the outputs associated with the children that arc identified by said address control portion to said input.

347. A node as defined in claim 346 in which said address control portion comprises:
A. a flag register including a plurality of flags each associated with an output, each nag generating an enable signal to identify the associated output if it is in a select condition, and not generating the enable signal if it is not in the select condition;

B. a flag register control circuit for controlling the condition of successive flags of said flag register in response to control signals representative of the request address information from the parent request distribution node.

348. A node as defined in claim 347 in which said flags of said flag register are updated in parallel in response to a timing signal, said nag register control circuit comprising A. a flag condition selection circuit including a plurality of condition selectors each associated with a flag for selectively coupling either a flag conditioning signal from the parent to enable the nag to update its condition in response to the flag conditioning signal or the flag's enable signal to enable the flag to maintain its condition, in response to the timing signal;
B. an address flag token shift register including a plurality of token stages each for controlling a condition selector of the flag condition selection circuit, said address flag token shift register receiving a token from said parent and selectively shifting said token through successive token stages in response to successive ticks of the timing signal a token stage generating said flag condition signal when it has the token; and C. an address state control circuit for controlling the shifting of said token through said address flag token shift register in response to the timing signal and a mode signal from the parent.

349. A node as defined in claim 348 wherein said flag register control circuit further includes a token coupling circuit for coupling the token from the last stage of the address flag token shift register to the identified outputs, thereby enabling the children attached thereto to receive the token m parallel to facilitate the parallel conditioning of their flag registers.

350. A node as defined in claim 348 wherein said address control portion couples the timing signal and said mode signal received at the input from the parent to all of the outputs, thereby coupling the timing signal and the mode signal to all of the children in parallel.

351. A node as defined in claim 348 wherein said flag register control circuit further comprises an address data coupling circuit for selectively coupling address data representative of the condition of a flag of said flag register under control of said address nag token shift register and said address state control circuit to the input for transfer to the parent, thereby to transfer a signal representative of the condition of the flag to the parent.

352. A node as defined in claim 34, in which said address data coupling circuit comprises a series of multiplexer circuits each for coupling as an output signal either the enable signal from a flag of said flag register if the token stage is generating the nag condition signal, or alternatively coupling as the output signal from the preceding multiplexer circuit in the series the output signal.

353. A node as defined in claim 352 in which said address data coupling circuit further includes a child address data receiving circuit for selectively receiving, through the outputs, address data signals from the children whose output circuits are enabled by said flag register and for coupling a composite address data signal to the first multiplexer circuit in the series. the first multiplexer circuit coupling the composite addsess da!a si~al as its output signal if the token stage is not generating the flag condition ~;ignaL tbereby to scle~tively cotlplc adtress data gignals from the identified c~;ldrcD to the parent if nonc of the to~en stages of the request dL~,t~buLio~ node are generating the flag condition signal.

354. A node as defined in claim 353 in which:
A. the control node gcDcrates an expected address dasa output signal which it couples to tbe request distribution node of the root level, eacb requcst distribution node coupling tbe e;~pccted address data output signal from its parent to of its children so that thc r~quest distnbution nodcs receive ,~hc expccted address data output signal gcnerally concurrently, and B. for each request dLstribution node, the address data reccivtng circuit includes an addrcss data proccss~ circuit that processcs the address data signals frorn the children whosc output circuits are enabled by the ~ag register in response ~o thc expccted address data output signal from the parent.

355. A node as dcfined in claim 354 in which said address data processing circuit includes:
A. a child address data asserted circuit for gcncrating an asserted cbild address data output signal if the address data output signals from all of the children associated with ideDtified outputs are asserted;
B. a child addrcss data negated circuit for generating a negated child address data output signal if the atdress data output signals from all of the children assouated with identified outputs are negated; and C. a child atdress data consolidation circuit for selectiYely coupling the output signal f~om the child address data asserted circuit or from the child address data negated circuit, as sclectcd by the e~pected address data output signal, to the first multiplexer circuit in the series as the composite address data signaL

356. A node as defined in claim 355 in which:
A. the child address data asserted circuit includes:
i. a plurality of OR circuits each associatet with a child, each OR circ~uit generat~ng an asserted child proccsscd address data output signal in res"or,se to the assertion of either the address data output signal from thc associated child and the complcment of the enable signal from the fl~, ~hich controL, the output connccted to tnat child; and u. a coDjunction circuit which generatcs the asserted cni~d address data output signal in rcsponse to the conjunction of the asserted child processed address data output signals from all of the OR
circuits, thercby to generate the asserted child address data output signal only if the child address data asserted circuit is reCeiViDg asserted address data out signals from all of the children whose flags are in the selected condition;
B. the child address data negated circuit includes:
i. a plurality of AND circuits each assoaated with a child, each AND circ~lit generating a negated child processed address data output signal in response to the negation of bother the address data output signal from the associated child and the enable signal from the nag which controls the output conneaed to that child; and ii. a disjunction circuit which generates the negated child address data output signal in response to all of the negated child processed address data output signals from all of said AND circuits thereby to generate the negated child address data output signal only if the child address data negated circuit is receiving negated address data out signals from all of the children whose nags are in the selected condition.

357. A node as defined in claim 346 in which:
A. the control node generates an expected test data output signal which it couples to the request distribution node of the root level, each request distribution node coupling the expected address data output signal from its parent to of its children so that the request distribution nodes receive the expected address data output signal generally concurrently; and B. for each request distribution node, the test data receiving circuit includes a test data processing circuit that processes the test data signals from the children identified by the address control portion in response to the expected test data output signal from the parent.

358. A node as defined in claim 358 in which said test data processing circuit includes:
A. a child test data asserted circuit for generating an asserted child test data output signal if the test data output signals from all of the children associated with identified outputs are asserted;
B. a child test data negated circuit for generating a negated child test data output signal if the test data output signals from all of the children associated with identified outputs are negated; and C. a child test data consolidation circuit for selectively coupling a composite child test data signal generated as the output signal from the child test data asserted circuit or from the child test data negated circuit as selected by the expected test data output signal, to the parent as a composite test data signal.

359. A node as defined in claim 358 in which said address control portion identifies a child by generating an associated enable signal, each associated with A. the child test data asserted circuit includes:
i. a plurality of OR circuits each associated with a child, each OR circuit generating an asserted child processed test data output signal in response to the assertion of either the test data output signal from the associated child or the complement of the enable signal from the address control portion which controls the output connected to that child; and ii. a conjunction circuit which generates the asserted child test data output signal in response to the conjunction of the asserted child processed test data output signals from all of the OR circuits, thereby to generate the asserted child test data output signal only if the child test data asserted circuit is receiving asserted test data out signals from all of the children identified by the address control portion;
B. the child test data negated circuit includes:
i. a plurality of AND circuits each associated with a child, each AND circuit generating a negated child processed test data output signal in response to the negation of bother the test data output signal from the associated child and the enable signal associated with the child; and ii. a disjunction circuit which ge~erates the negated child test data output sigD~ in response to all of the negated c~ild processcd test data output signals from all of said AND circuits thereby to generate the negated child test data output signal only if the c~ild ~est data negated circuit is receiv~g negated test data out signals from aU of the children identified by the address co~trol portion

360. A node as defined in claim 358 in which A. the control node generates a test data inpu~ signal which it coup]es to the rcquest distribution node of the root leveL each request distribution node coupling the test data input sigual from its parent to of its childrerl so that the request distribution nodes receiYe the test data input signal generally concl~rrently~, and B. said child tcst data consolidation circuit fnrther includes a test data selection eirclut for selec~ively coupli~g, to the parent as the composite test data signal, the test data input signal, if at least one of the children is identified by the address control portion, or the composite child test data signal, if none of the c~ildren are identified by the address control portiom