CA2022259C - Method and apparatus for controlling initiation of bootstrap loading - Google Patents

Abstract

ABSTRACT

Method and apparatus for controlling initiating of bootstrap loading in a computer system having first and second discrete computing zones is disclosed. Each computing zone includes a status register for storing an operating system run (OSR) bit indicating that the zone has initiated bootstrap loading. A cable connects the computing zones to allow the first and second zones to read the status registers in the second and first zones, respectively. A CPU in each zone only enables initiation of bootstrap loading if the OSR bit in the other zone is not set.

Description

21)2225g 1 I. a~ ROUND OP TH~ INVEN~

2 This invention relates to a method for controlling the 3 bootstrap loading of the operating system in a fault tolerant ~yste~. One type of fault tolerant computer systQm~ comprises two distinct computing zones that operate in lockstep 6 ~ynchronism as a single system during normal operation. During 7 such lockstep operation, both zones ideally perform the same operat~ons, read identical data and provide identical outputs.
Each zone is al80 capable of independent operation. Independent ~ -i~ 10 operation of a zone normally occurs when one zone is removed from ser~ice for repair or i8 otherwise unable to operate.
12 While that zone i~ down, the other zone continues to run a 13 user~ 8 appllcation under control of the operating ~ystem. It is 14 critical that upon itJ return to service, the repaired zone not ¦ 15 be permitted to boot~trap load a Jeparate copy of the operating 16 system. If the repaired zone were permitted to load a separate 17 copy of the operating sy~tem, data corruption problems would 18 occur. Running two independent copies of the operating ~ystem 19 would cause the reJpective sones to read and write data not appropriate to the current operation of the fault tolerant 21 Jy-tem. Thi- ro-ult runJ counter to the basic reguirement that 22 the two computing zone~ operate in lockstep synchronism.
23 It iJ therefore important to provide a method for in~uring 24 that the two computing zone~ of a fault tolerant system operate ~AWOr~C~ ....
Fl~ HENDE~N from a ~ingle copy of the operating sy~tem.
~, ~ DUNNER : .
-- 11 TI~T, N. W. :
,. ~I~OTO-I. O. C. OOO--~0~ 0 . ' ~' ~

,, ' ' ., .

2~22259 II. SUMMARY OF THE INVENTION
Additional advantages of the invention will be set forth - in part in the description whlch follows, and in part will be obvious from the descriptlon, or may be learned by practlce of the lnvention. The advantages of the lnvention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.
The present invention overcomes the problems and disadvantages of the prior art by providing a method for initiating bootstrap loading of an operatlng ~ystem in a computer system having first and second diæcrete computing zones the first zone includlng a flrst CPU and the second zone including a ~econd CPU, said first and second CPUs for independently executing said method for initiating bootstrap loading, each of said zones having a status register acces~ible by the other zone. The method comprises the steps of determining either that a selected one of the first and second zones i8 allowed to bootstrap load or that a non-selected one of said first and second zones is incapable of ¦ running the operatlng system; determining whether the non~selected zone has initiated bootstrap loading of the operating system by accessing, from the selected zone, the status register in the non-selected zone; and initiating bootstrap loading ln the selected zone at times when said selected zone is allowed ~o bootstrap load; initiating bootstrap loading in the selected zone at times when said non-selected zone is incapable of running the operating system; and initiatinq bootstrap loading in the selected zone at times when said non-selected zone has not initiated bootstrap A

~..... - .. ... .. . .. . .. -.. ,.. ...... -. . , .. ..... ,... ....... ; .. . . . ..

2a222~

loading of the operating system; wherein the step of determining whether the selected zone is allowed to bootstrap load includes the substep of determining that bootstrap loading of the selected zone cannot be initiated at times when the non-selected zone has initiated bootstrap loading.
~ he present invention is also directed to a computer system, comprislng. a first discrete computing zone including a first CPU, and a first status register storing a flrst bit indicating whether the first zone has inltiated bootstrap loading;
a second discrete computing zone including a second CPU, and a second status register storing a first bit indicating whether the second zone has initiated bootstrap loading; a cable coupled between the first and second zones, to allow said first and second zones to read the status registers in said second and first zones, respectively; means, in said flrst CPU, for determining whether the first bit stored in said second status register indicates that said second zone has initiated bootstrap loading and for determining that bootstrap loading of said first zone cannot be initiated if æaid second zone has initiated bootstrap loading;
said first zone including a first memory for ~toring a second bit indicating that said first zone can initiate bootstrap loading;
and means for determining if said cable i8 connected to said second zone; and whereln said first CPU includes means for determining that said first zone can bootstrap load if said cable iæ not connected and said second blt indicates that said first zone can initiate bootstrap loading.

. .

2~2225~

III. BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and which constitute a par~ of this specification, illustrate one embodiment of the invention and, together with the description of . the invention, explain the principles of the invention.
. Fig. 1 is a block diagram of a preferred embodiment of ; fault tolerant computer æystem which practices the present ~ invention;
., ~` Fig. 2 is an illustration of the physical hardware ~,; 10 containing the fault tolerant computer system in Fig. 1;
;~
' Fig. 3 is a block diagram of the CPU module shown in the fault tolerant computer system shown in Fig. 1;
Fig. 4 is a block diagram of an interconnected CPU
module and I/O module for the computer system shown in Fig. l;
~ Fig. 5 is a block diagram of a meDory module for the ¦ fault tolerant computer system shown in Fig. 1;
Fig. 6 is a detailed diagram of the elements of the control loglc in the memory module shown in Fig. 5;
Fig. 7 is a block diagram of portions of the primary memory controller of the CPU module shown in Fig. 3;

' ' 3a ` ~
A

~Q~ i9 1 Fig. 8 is a block diagram of the DMA engine in the prim~ry 2 memory controller of the CPU module of Fig. 3;
Fig. 9 is a diagram of exror proce~sing cLrcuitry in the 4 primary memory controller of the CPU module of Fig. 3;
Fig. 10 i8 a drawing of 30me of the register~ of the cro~-link in the CPU module shown in Fig. 3;
7 Fig. 11 i3 a block di~gram of the elements which route 8 control signal~ in the cros~-linX~ of the CPU module ~hown in g Fig. 3;
Fig. 12 i~ a block di~gram of the element~ which route data 11 and address 8ignal8 in the pr~m~ry cross-link of tho CPU module 12 ~hown in Fig. 3;
13 Fig. 13 i8 a ~tate di~gram showing the ~tate~ for the 14 cro~s-l~nk of the CPU module shown in Flg. 3;
Fig. 14 is a blo~k di2gram of the tlming system for the 16 fAult tolerant computer system of Fig. l;
17 Pig. 15 i~ ~ timinq diAgram for the clock ~ignal~ generated 18 by the timing sy~tem in Flg. 14;
19 Fig. 16 i~ a detailed diagram of a pha~e detector for the timing ~y~te~ ~hown in Fig. 14;
21 Fig. 17 i~ a block diagram of an I/0 module for the 22 co~puter flystem of Flg. 1;
23 Flg. 18 ia a block diagram of the firew~ll eleme~t in the 24 I/0 module shown in Fig. 17;
Fig. 19 i~ a detailed diagram of the elQments of the cross-26~Wo~C~ link pathway for the computer sy~te~ of Fig. l;
FI~C~N. HENDER50N
~IIOW. C~RRETT
28~ DUNNER
.~ TIIC~, N. W.
~n~O.O~.. O. c ~000 .,..~..,0 ~; 31 -4- i ,.,,.,,'.' . I
, ~ .
. ........... .
,, ..... .

, . .. . . . . . .. ... . . . . . .

2Q22~59 Figs. 20A-20E are data flow diagrams for the computer 2 system in Fig. l;
Fig. 21 i3 a block diagram of zone 20 showing the routing 4 of resat signal~;
Fig. 22 i~ a block di~gr~m of the components involved in resets in the CPU module shown in Fig. 3;
7 Fig. 23 is a diagr~m of clock re~et circuitry;
Fig. 24 is a flowchart of A pra-boot~trap algorithm; and Fig. 25 illu3trates a circuit arrang~ment by which each of the two proce~ing systQm~ of the fault tolerant computer ystQ~ .
11 can detenmine the 8 atus of the cros~-link cable.
12 IV. DESCRIPTION QF TH~ PR~ M ED EMBODI~NT
13 Reference will now be made ~n det~$1 to a presently 14 preferred embodiment of the invention, an example of which i~
illu~trated in the accompanying drawings.
16 A. SYSTEM DI5SCRIPTION
17 Fig. 1 i4 a block diagram of R fault tolerc~nt computer 18 system 10 in accordance with the pre~ent inventlon. Fault 19 tolerant co~puter ay~tem 10 include~ dupllc~te system~, called zone~. In the normal mode, the two zones 11 and 11~ oper~te 21 3imultaneoualy. The duplication ensure~ that th~re i no ~ingle 22 polnt of failure and that ~ single error or f~ult in one of the :
23 zon~o ll or ll' ~ll not di~ble computer ~ystem 10.
24 Purth~rmore, ~ uch faults can be corrected by disabling or ignoring the devlce or elffment which cau~ed the fault. Zones ll 2~wor~1c~
FIN I~W, HENDERSON
R;~IEOW. CARRETr 9a DUNNER
?~I-- TR~
o/~. O. C. ~000-32`'''~ ~ ~

~22~
1 and 11' are shown in Fig. 1 a~ respectively includlng duplicate proce~sing syst~ms 20 and 20'. The duality, however, goes 3 beyond the proce~sing system.
Fig. 2 contains an illustration of the phy~ical hardware of f~ult tolerant co~puter sy~tem 10 ~nd graphically illustratas the duplLcation of the sy~tem~. Each zone 11 ~nd 11' iB housed 7 in e different cabinet 12 and 12', re~pectively. Cabinet 12 includeR b~ttery 13, power regulator 14, cooling fan~ 16, and AC
9 input 17. Cabinet 12' includee separate elements corre~ponding to elements 13, 14, 16 and 17 of c~binat 12.
11 As explained in gre~ter detall bHlow, processing ~y~tem8 20î and 20' include sevQral modules interconnected by backplanes.
13 If a module contains a fault or error, that module may be 14 removed and replaced without disabling ccmputing system 10.
This i8 becau~e processing sy~te~ 20 and 20' are phy~ically 16 separate, have 3eparate backplanes into which the module3 are 17 plugged, and c~n operate independently of each other. Thu~
18 modules can be removed from and plugged into the b~ckpl~ne of 19 one proces-ing sy~tem while the other proces~ing ~y~tem j 20 continues to opor~te.
21 In th~ pr ferred embodlm0nt, the duplic~te proces~ing 22 sy t _ 20 ~nd 20' are identical and cont~in identicAl module~.
23 Thu~, only proces~ing ~ystem 20 w~ll be described completely 24 with the under~tAnding that proces~ing sy~tffm 20~ operate~

26 equivalently.
27~wo,~,et.
FI~C~N, HENDEASON
e~. G~AAE~r ~ O DUNNER
30~ . W.
no~o~ o c ~ooo~
32-~--- ~ -6-. ~ 2~9`
1 Processing ~ystem 20 includes CPU module 30 which is sho~n 2 Ln greater detail in Figs. 3 and 4. CPU moduls 30 is 3 interconnected with CPU modulQ 30' in processing system 20~ by a 4 cros~-link p~thw~y 25 which i8 de~crlbed in greater detail below. Cross-link p~thw~y 25 provides dat~ tr~nsmission p~ths 6 between processing ~ystems 20 and 20' and carriQs timing signels to ensure that processing systQms 20 and 20' operate synchronously.
9 Proce~sing system 20 ~190 includes I/O module~ 100, 110, and 120. I/O modules 100, 110, 120, 100', 110' and 120' are 11 independent d~vices. I/O module 100 i8 shown in gre~ter det~il 12 in Figs. 1, 4, and 17. Although multiple IJo modules are shown, 13 duplication of such module~ iu not a requirement of the system.
14 ~ithout such duplieation, however, ~ome degre~ of fault tolorance will be lo~t.
16 Each of the I/O modules 100, 110 and 120 i~ connected to 17 CYU ~odule 30 by dual r~il ~odule interconnecta 130 and 132.
18 Module interconnects 130 and 132 serve as the I/O interconnect 19 and are routed acros~ the b~ckplano for proce~sing ~ystem 20.
For purpo-~ o~ this applic~tion, the data pathw~y including CPU
21 40~ ~emory controller 70, cros~-link 90 and ~odule interconnect 22 130 i~ con~id~red a~ one rall, ~nd the d~tA pathway including 23 CP~ SOt ~e~ory controller 75, cross-link 95, and module 24 int~rconnect 132 $8 con~ldered a~ another rail. During proper 26 operation, the dat~ on both rails i~ th.e s~me.
27~o~c~
FINY~N. HENDERSON
~BO'1V C~RRETr DUNNER
~ 11 T~T, N. W.
W~I~ OTOi~, D C. ~000--0~ 0 33 . -7-. .

21}222~9 ~ ~

1 B. FAULT TOLERANT SYSTEM PHILOSOPHY
2 Fault tolerant computer system 10 does not h~ve ~ single 3 point of failure because each element is duplicated. Processing ~- 4 system~ 20 and 20~ are each a fail stop processing ~ystem which means that those systems can detect faults or errors in the 6 subsystems and prevent uncontrolled propagation of such faults and errors to other subsystems, but they have a single point of failure because the elements in each processing system are not ~, duplicated.
;~ 10 The two fail stop processing systems 20 and 20' are ~; 11 interconnected by certain elements operatlng in a defined manner 12- ~I to form a fail safe ~y~teM. In the fa$1 safQ ~y~tem ~mbodied a~
13 fault tolerant computer sy~tem 10, the entire computer system 14 can continue processing even if one of the fail stop processing system~ 20 and 20~ i~ faulting.
16 The two fail stop proces-ing systems 20 ~nd 20~ are 17 con~idered to operate in lockstep synchronism because CPUs 40, 18 50, 40' and 50' operate in such synchroni~m. ~here are three 19 significant exception~. The fir~t i~ At initializaeion when a boot~trapping technlgue bring~ both processor~ into synchronism.
21 The ~cond exception is when the proces~ing sy~tems 20 and 20' 22 1 operat- ind-p ndently (asynchronoualy) on two different 23 ¦ worklo~d~. The third except~on occur~ when certain errors arise 24 in proce~ing syJtems 20 and 20'. In this laJt exception, the CPU and memory elsment~ in one of the proce~ing systems is Fl ~N.HE~DER~ disabled, thereby ending synchronous operation.
~0~. C.~RRETT
. 6 DUNNER . . .
2~ -t .1~.w, . ' w~ o/~. O. c. ooo-31O~ ---O

.

,~ ~ . .. . . ... . . . . .. . . . . . .. .. .... . .

2112~5g 1 When the system iQ running in lockstep I/0, only one I/0 device i8 being accessed at any one time. All four CPUs 40, 50, 3 40~ and 50~, however, would receive the ~ame data from that I/0 ~; device at substantially the same time. In the following discussion, it will be understood that lock~tep ~ynchronization of processing ~ystem~ means that only one I/O module is being 7 accessed.
8 The synchronism of duplicate processing ~ystems 20 and 20' is implemented by treating each syst~m as a deterministic machine which, ~t~rting in the same known stAte and upon receipt 11 of the sam~ inputs, will always enter tho same machino states 12- and produce the same results ln the ab~ence of error.
13 Proces~ing system~ 20 and 20' are conflgured identically, 14 receive the same input~, and therefore pa~s through the same states. Thus, as long a~ both proce~or~ operate synchronou~ly, 16 they Jhould produce th~ Jame reJultJ and enter the same state.
! 17 If the processing systems are not in the same state or produce 18 different re~ults, it iJ aJJumed that one of the processing ¦ 19 JyJtem~ 20 and 20~ ha~ faulted. The Jource of the fault must ¦ 20 then be i-ol~ted in order to take correct$ve action, such as 21 di-ablin~ th fault$ng module.
22 ~rror detectlon generally involveJ overhead in the form of 23 ad!dltional proc-~-lng time or logic. To minimize such overhead, 24 a sy~tem should check for error~ as infrequently a~ pos~ible 2S conJistent wlth fault tolerant operation. At the very least, 26 WO~Ic" error checklng mN~t occur before data i~ outputted from CPU
h~ N HENDEI~50N
~OW, C~RRETT

TI~CtT, ~. W.
W3~ ~TO O C 000--' .', :.'..

20222~9 1 moduleR 30 and 30' Otherwi~e, internal processing error~ may 2 c~use lmproper oper~tfon ln extern~l sy~tem~, llke ~ nucle~r 3 reactor, whlch 1~ the condltlon th~t f~ult toler~nt sy~tem~ ~re 4 de~lgned to prevent There are reasona for ~ddltion~l error checklng For 6 ex~mple, to l~ol~te f~ulto or errors it 1~ deolr~ble to check 7 the d~t~ received by CPU moduleo 30 ~nd 30 prior to ~torage or 8 u~e Otherwl~e, when erroneoue ~tored d~t~ ter ~cce~ed and add'tionnl erroro re~ult, lt become~ dlfficult or lmpos~lble to find the orlgin~l ~ource of error~, e~pecl~lly when the 11 erroneous d~tn h~s been otored for ~ome tlme The p~e~ge of 12 time ~a well ~ ~ub~equent proceoolng of the erroneoua d~t~ m~y 13 de~troy nny tr~ll b~ck to the ~ource o~ th- error 14 ~Error l~tency,~ whlch refer- to th- ~mount of tlme ~n error 18 stored prlor to detectlon, m~y c~u~e l~ter problema 16 well For ex~mple, ~ ~eldom-uo-d routlne m~y uncover ~ l~tent 17 error when the computer ~y-te~ l- ~lre~dy oper~tlng wlth 1 18 dlmlnl~hed c~p~clty due to ~ prevlous error When the computer 19 ~y~tem h~- d'~nl-h~d c~p~clty, the l~tent error m~y c~u~e the ¦ 20 ~y~tam to cr~-h 21 ~urthormore, it i- deolr~ble ln the du~l r~ll Ely~te~ of 22 proco--lng y-tem~ 20 ~nd 20 to check ~or error~ prlor to 23 tr~n-f-rrlng d~t~ to ~lngle r~ yotemJ~ uch ~o ~ oh~red ;
24 re~ourco lik- ~emory Thls 1~ b-cauoe there ~re no longer two lndependent sourco~ of d~t~ ~fter uch tran~fer~, ~nd lf ~ny 26~o~c~ error ln the lngle r~ll y~tem lo l~ter detected, then error FINNEC~N. HENDEIE50N . . .
~ ~ or ~A7r tr~clng beco~e- dl~flcult lf not l~po8-ible .... ,.. ~.~.. ~. ~. .
o~o~ . 0. c. ~000-~o~l~>~ o ' ~ 10 - I, ' , . 1-~22~
66822~118 The preferred method of error handling is set forth in Canadian application Serial No. ~0~2260 filed on this same date in the name of Digital Equipment Corporation and ~ rrorS
entitled "Methods of Handling Erros in Software."
.:

,,:

, . .

i: .

1. ~

'~ ' - lOa -2~2~9 1 C. ~ODULE DBSCRIPTION
2 1. CPU Module 3 The elements of CPU module 30 which appear in Fig. 1 ar~
4 shown in greater detail in Figs. 3 and 4. Flg. 3 i8 a block diagrc~m of the CPU module, and Fig. 4 shows block diagrc~m~ of 6 CPU module 30 and I/O module 100 a8 well a~ their intercon-7 nections. Only CPU module 30 will be described since the 8 operation of and the elements included in CPU module~ 30 and 30' 9 are generally the same.
CPU module 30 contain~ dual CPUs 40 and 50. CPU~ 40 ~nd 50 11 can be st~ndard central process$ng unit~ known to per-ons of 12. ordinary skill. In the preferred embodimont, CPUs 40 and 50 are 13 VAX microproce~sors m~nufacturQd by Digit~l ~quipment 14 Corporation, the assignee of this applic~tion.
As~ociated with CPUs 40 and 50 nre cache memories 42 and 16 52, reapecti~ely, which are stand~rd cache ~AN8 of ~ufficient17 memory ~ize for the CPU~. In the preferred embodlment, the 18 cach~ RAN i8 4R ~ 6~ bits. It i~ not nQce~sary for the present 19 inv~ntion to have a cache RA~, h~wQver.
2. ~amorv ~odule 21 Pr-ferAbly, CPU's 40 and 50 can ~hare up to four memory 22 ~o~ul-~ 60. Plg. 5 is a block diagram of one mamory module 6023 shown connact~d to CPU modulQ 30.
24 Durlng me ry transfer cycles, st~tu~ register tran~fercyclo~, and ~EPRQn transfer cycle~, each memory modulQ 60 26~^Wo"lC~- tran~fere dat~ to and from pr~m~ry msmory controller 70 via a FINNEC~N, HENDEI~SON
E~30~V CAARETT
~7 ~DUNNER bid$rect$on~1 dat~ bu~ 85. ~ach memory module 60 also receiYes r~ . w.
~ C Ooo-~0~ 0 . ' ' . I ~ .

:".. ..

. ~ , ~22`~55~
address, control, timing, and ECC ~ignals from memory 2 controllers 70 and 75 via bus~as 80 and 82, respectively. The 3 addreQs si~nals on buse~ 80 and 82 include board, banlc, and row 4 e~nd column addre~s ~ignals that identify the memory board, bank, and row and column address involved in the data transfer.
6 As shown in Fig. 5, each memory module 60 includes a memory 7 array 600. Each memory array 600 is a ~tandard RA~ in which the DRAMs are organLzed into eight banks of memory. In the -9 preferred embodiment, f~st page mode type DRAI~ are used.
Memory module 60 also includes control loglc 610, data 11 transceiver~/regi~ters 620, memory drivers 630, and an EEPROII
12 640. Data transceivers/receivers 620 provide a data buffer and 13 d~ta interface for tranJferring d~t~ between memory ~rr~y 600 14 and the bidirectional data lines of data bus 85. Me~ory drivers 630 distribute row and column addre~s signals and control 16 ~ignals from control logic 610 to each bank in memory array 600 17 to enable transfer of a longword of data ~nd its corresponding 18 ECC ~ignal- to or from the memory bank selected by the memory 19 board and bank addre~ signals.
2C EISPROI~ 640, whlch can be any type of 13VRAII (nonvolatile ¦ ~ ;
21 RA~ tor-- m~ory error data for off-line repair and 22 conflguratlon data, ~uch as module size. When the memory module 23 i8 r~o~r d aft-r a fault, ~tored data 1~ e~ctracted from E~PROM
24 640 to detesaino th cau-o of the fault. ~PRO~I 640 is ¦
addressed ~ row addre-~ lines from drivers 630 and by EEPROM ¦ ;
26~^wo~c~ control lgnal- frol- control logic 610. EEPRO~S 640 tran~fer3 Fl~ CAN, HENDER50 I .. .
AJO~IV, G~ARE~T ~ . :
~ DUNNER I .. .
2~ T~e..~ w w~ TOI~, O. e. 000-~u.O.~.. O I ....

::
'..." ' :'.,' ' :' ' j~ r ~ C~; ~

211`2,2~g 1 eight bits of d~ta to and from a thirty-two bit internal memory2 data bus 645 3 Control logic 610 route~ address signals to the elements of 4 memory module 60 and generate~ internal timing and control signals As shown in greater detail in iFig 6, control logic 6 610 includes a primary/mirror de~ignator circuit 612 7 Primary/mirror designator circuit 612 recei~es two sets of8 memory board address, bank .ddrQ~s, row snd column address, 9 cycle type, and cycle timing signals from memory controllers 70and 75 on buses 80 and 82, and al~o tran~fer~ two ~etJ of FCC
11 signals to or from the memory controllers on buses 80 and 82 12 Tran~ceivers/reglsters in de~ignator 612 proYide a buffer and 13 interface for transferring the~e ~ignals to and from memory 14 buses 80 and 82 A primary/mirror multiple~er blt stored in status registers 618 indicates which one of memory controllers 16 70 and 75 i8 de~ignated as the primary memory controller and 17 which i~ de~ignated a~ the mirror momory controller, and a ¦~
18 primary/mirror multiplexer signal i- provided from status 19 registers 618 to de-ignator 612 j~ 20 Primary/mirror de~ignator 612 provide~ two ~ets of ~ignal8 ¦ 21 for distrlbution in control logic 610 One set of signals 1 22 include- do~ignated primary memory board address, bank addressi, 23 row and colu~n addre~, cycle type, cycle t~ming, and 2CC
24 signals Tho other set of sign~l- include~ de~ignated mirror memory bo~rd addre~, bank ~ddre~, row ~nd column address, 26~Wor~ cycle type, cyrle timing, and ECC signals The primsry/mirrorFINNECAN. HENDER50N
F~RA~V G~RRETr 27ODUNNE~ ~ultiplex~r ~igna~ u~ed by de~ignator 612 to ~elect whether . w.
W,~ 1tO~.O.C ~000-~o~ o ! :--13~

, 21)22259 1 the signals on buses 80 and 82 will be re~pectively routed to 2 the lines for carrying de~ignated primary slgnals and to the 3 lines for carrying desLgnated mirror signals, or vice-versa A number of time division multiplexed bidirectional lines are included in buses 80 and 82 At certain timas after the beginning of memory transfer cycles, ~tatus registQr transfer 7 cycles, and EEPROM transfer cyclQs, ECC signals corresponding todata on data bus B5 are placed on these time division 9 multiplexed bidirectional lines If the transfer cycle is a write cycle, memory module 60 receive~ data and ECC signals from 11 the mem,ory controllers If the transfer cycle is a read cycle, 12 memory module 60 transmit~ data and ~CC signal~ to the memory 13 controllers At other times during transfer cycles, address, 14 control, and t~ming signals are received by memory module 60 onj 15 the tiMe divislon multiple~ed bidirectional line~ Preferably, j 16 at the beginning of memory transfer cycles, ~tatus register 1 17 transfer cycles, and EEPRQN tran~fer cycles, memory controllers 13 70 and 75 transmit memory board address, bank addres~, and cycle 19 type signals on th ~e t~me-hared line- to each memory module 60Pr-ferably, ro~ addre~ ~ignals and column address signals 21 are multiplexed on the ~am row and column addres- lines during 22 tran-f-r cycle~ Plr~t, a row addr~ provided to momory ;
23 _odul- 60 by the memory controllers, followed by a column ad-24 dres~ ~bout Jixty nano~econd~ later A 8 quencer 616 receive~ a- input~ a sy~tem clock signal 2C~WO~IC~ and a re-et signal from CPU module 30, and receives the FINNEC~N, HENDER~ON ~ . . .
~D~NNER d-~ignated, pr~m~ry cycle timing, de~ignated prim~ry cycle type, I -,~.. ,.. ,.. ~. ~ .
e~.~Ooo~ ' ."",-, ~: ~Q2~2~g de~ignated mirror cycle timing, and designated mirror cycle type 2 ~ignals from the transceiver~/registers in designator 612 3 Sequencer 616 is a ring counter with ~ssociated steering~` 4 logic that generates and distributes a number of control and sequence timing signals for the memory module that are ne~ded in ;~ 6 order to execute the various type~ of cycles The control and sequence timing signals are generated from the system clock signal~, the designated primary cycle timing ~ignals, and the 9 designated primary cycle type signals Sequencer 616 also generates a duplicate set of ~equence 11 timing signal~ from the system clock ~ignals, the de~ignated12- mirror cycle timing signal3, and the designated mirror cyele 13 type signals These duplicate seguence timing signal~ are used 14 for error checking For data transfer~ of multi-long words ofdata to and from memory module 60 in a fast page mode, each ~et 16 of column addre~es starting with the first set i~ followed by 17 the next eolumn addre~s 120 nanoseeond~ later, and eaeh long18 word o data is moved acro~ bu~ 85 120 nanoseeonds after the19 previou~ long word of data Sequeneer 616 also generates t~/rx register control 21 signnl~ Th- tx/r~ regi~ter eontrol signals are provided to 22 control the operation of data tran~eeiver~/regi~ter~ 620 and the 23 tr~n~eelver~/register~ in de~lgnator 612 The direetion of data ;3 24 flow ii~ detormined by the staering logie in sequeneer 616, which re~ponds to th- de~ignated primary eyele type signal~ by 26~o~r~c~ generating tx~rx control and ~equence timing signals to indicate FINNEC~N, HENDER50N
F~R~O~IV C~RRETr 2~DUNNER whether and when data and ECC signals should be written in~o or ~ OTO~ . C.~OOO--~0., .-~----0 .

--~ 2~22~9 1 read from the transceivers/registers in memory module 60. Thus, 2 during memory write cycle~, ~tatus register write cycles, and 3 EEPROM write cycles, data and ECC signals will be latched into 4 the transceiversJregisters from buse3 80, 82, and 85, while S during memory read cycles, status reglster read cycles, and 6 E~PRON read cycles, data and ECC signals will be latched into 7 the transceivers/registers from memory array 600, status 8 registQrs 618, or E~PROM 640 for output to CPU module 30. -Sequencer 616 also generates ~EPRQM control signals to control the operation of ~EPRQM 640.
11 The timing relationship~ that e~ist in memory module 60 ~re 12 specified with reference to the rise time of the system clocX
13 signal, which has a period of thirty nanoseconds. All ~tatus q register read and write cycle~, and All memory read and write cycles of a singlQ longword, are performed in ten sy~tem clock 16 periods, i.e., 300 nanoseconds. Nemory read and write transfer 17 cycles may consist of multi-longword tran~fers. For each 18 additional longword that is transferred, the memory transfer 19 cycle i~ extended for four additional system clock periods.
Memory refre~h cycle~ and EEPRQM write cycle~ require at least 21 twelv ~y te~ clock period~ to e~ecute, and E~PROM read cycles 22 requlr0 at lea~t twenty sy~tem clock periods. ~ -23 The de~ign~ted primary cycle t~ming signal causes sequencer 24 616 to ~tart generating the ~equence timing and control signal~
th~t en~ble the memory module selected by the memory board FIN2N6~N.H~ND~A~N addre-~ ignala to implement a requested cycle. The transition E~llo~. G~ARETT
2~DUNN~ of the d~-ignated prim~ry cycle timing ~ignal to an active state 1 ~
,,7. ,.. ~.. ~. w. I .
0-0~ . O. c. ~000- I ., ~O...... ~.. O -16-' I"'.-, 2~2~9 1 marks the start of the cycle. The return of the designated 2 prlmary cycle timing ~ignal to an inactive state marks tha end 3 of the cycle.
4 The sequence timing signals generated by sequencer 616 are associated with the different states entered by the sequencer as 6 a cycle reque~ted by CPU module 30 i8 executed. In order to 7 3pecify the timing relationship among these different states (and the timing rel~tionship among sequence timing signals 9 correspon~ing to each of these ~tate~), the discrete states th t may be entered by sequencer 616 are identified as state~ SEQ
11 IDLE and S~Q 1 to SEQ 19. ~ach sthte lasts for A single system 12 clock period (thirty nAnosecond~). Entry by nequencer 616 into 13 each different state i8 triggered by the leading edge of the 14 system clock signal. The leading edge~ of the system clock ~ignal that cause sequencer 616 to enter states SEQ IDL~ and SEQ
16 l to SEQ 19 sre referred to as tran~itions T IDL2 and T1 to Tl9 17 to relate them to the sequencer states, i.e., TN is the system 18 clock signal leading edge that cause~ sequancer 616 to enter l9 state SEQ N.
At timeJ when CPU module 30 is not directing memory module 21 60 to execute a cycle, the de~ignated primary cycle timing 22 signal i~ not a~ert d, and the ~equencer remains in ~tate SEQ
23 IDLB. Tho ~equencer i~ started (enter~ ~tate SEQ 1) in respon~e 24 to as~ertion by memory controller 70 of the cycle timing 8ignal on bu~ 80, pro~ided control logic 610 and ~equencer 616 are 26L~or~C~ loc~ted ln the _ ory madule Jelected by memory board addre3s FI~NEC~N; HENDER5~N
~DUNNER 1gna1a ~l~o tr~n~mitted from memory controller 70 on bus 80.
.w.
oTo~. O. c.~000-~0..... O 17 Q~2`2~9 The rising edge of the first ~ystem clock signal following 2 assertion of the designated primary cycle acti~e signal 3 corresponds to transition Tl.
~ Aa indicated previously, in the case of transfer~ of a ; 5 single longword to or from memory array 600, the cycle is 6 performed in ten ~ystem clock periods. The sequencer proceeds 7 from SEQ IDLE, to states SEQ 1 through SEQ 9, ~nd returns to SEQ
IDL2.
9 Memory read and write cycles may be extended, however, to transfer additional longwords. Memory array 600 preferably uses 11 "fast page mode~ DRA~s. During multi-longword reads and writes, 12 - tran~fers of data to and from the memory array after transfer of 13 the first longword are Accomplished by repeatedly updating the 14 column addres~ and regenerating a CAS (column ~ddress ~trobe) signal.
16 During multi-longword tranafer cyclea, the~e updates of the 17 column address can be implemented becauae aequencer 616 18 repeatedly loops from states SEQ 4 through SEQ 7 until all of I -19 the longworda are tranaferred. For example, ~f three longword~ ¦ ~
are being read from or written into memory ~rray 600, the ¦ -21 sequ-nc-r enter~ state- S~Q IDL~, S~Q 1, SEQ 2, SEQ 3, SEQ 4, 1 -22 S8~ 5, SBQ 6, SBQ 7, SBQ 4, SEQ 5, SBQ 6, SEQ 7, SEQ 4, SEQ 5,.
23 S~Q 6, SJQ 7, SBQ 8, SEQ 9, and SBQ IDLE.
24 During a memory tran-fer cycle, the deaignated primary ¦ 25 cycle t~ng signal ia monitored by sequencer 616 during 1 26~wo-~e-- tran~itlon T6 to determine whether to extend the ~emory read or FINNEC~N, HENDERSON
~UO~ RAETT
DUNNER writ- cycle in order to transfer at lea~t one additional ,". ,.. ,.. ~.~.
W~ TO~. O. ~. ~000~
~oO~ O
2g .
.

-1 longword. At tim~8 when the designated primary cycle timing signal i~ asserted during transition T6, the sequencer in state S~Q 7 will re~pond to the next system clock ~ignal by entering 4 ~tate SEQ 4 instead of entering state SEQ 8.
In the case of a multi-longword transfer, the designated primary cycle timing ~ignal is asserted at least fifteen 7 nanoseconda before the first T6 transition and remaina asserted `~ 8 until the final longword is transferred. In order to end amemory tran~fer cycle after the final long~ord has been transferred, the designated primary cycle tlm~ng signal is 11 deasserted at lea~t fifteen nanosecond~ before the la~t T6 12 tran~ition and remains deasserted for at least ten nanosecond~
13 after the last T6 tran~ition.
14 During memory tran~fer cycle~, the de~ignated primary row addres~ ~ignals and the de-ignated primary column addre~s 16 signals are presented at different times by designator 612 in17 control logic 610 to memory dr~ver~ 630 on a ~et of time 18 division multiplexed lines. The outputs of drivers 630 are 19 applied to the addre-~ inputs of the DRAM~ in memory array 600, and al~o are returned to control loglc 610 for comparison with 21 th- d--ignrAt-d mirror row and column addre~s signal~ to check 22 for error-. During ~tatu~ register tran~fer cycles and ~EPROMA~3 tr~n-f-r cycle~, column addre~ ~ignal~ aro not needed to ~elect ¦
24 a particular ~torage location.
2S Dur~ng a memory transfer cycle, row addres~ signals are the 26L~wo~C~ fir~t ~ignal~ presen~ed on the timeshared row and column addres~
FINNECA`I. HENDER50N
~ow. G~RRETr ~7 a DUNNER lin~ of bu~ 80 and 82. During ~tate SEQ ~DLE, row address T~ T, N. ~.
OTO~, O. C. ~000-~0~~ 0 1 11 -19~

~2~ g 1 signal~ are tran8mitted by the memory controller~ on the row and 2 column address lln~s, and the row addre~s is stable from at 3 l~st fifteen n nosecondR before the T1 tran~ition untll ten 4 nanoseconds after the Tl transition Next, column address S signals are tran~mltted by the memory controllers on the row and column addres~ lines, and the column addrQss is stable from at 7 least ten nanoseconds before the ~3 transition until fifteen 8 nanoseconds after the T4 transition In the case of multi-9 longword transfers during memory transfer cycles, subsequent column addre~s signals are then tran~mitted on the row _nd 11 column address linea, and these sub~equent column addresses are 12- stable from ten nanoseconds before the T6 trAnsition until 13 fifteen n_no~econds after the T7 tran~ition 14 Generator/checker 617 receives the two set~ of sequence timing sign~l~ generated by sequencer 616 In addition, the 16 de~ignated primary cycle type and bank addre~s sign~l~ and the 17 dQsignated mirror cycle type and bank address signals are 18 transmitted to generator/checker 617 by de~ignator 612 In the 19 gener_tor/checker, a numker of prlmary control ~ign-ls, i e , RAS (row _ddre~- ~trobe), CAS ~column _ddre~ ~trobe), _nd WE
21 (writ- enable), are generated for distribution to dri~er~ 630, 22 u~ing the primary sequence timing ~ignal~ and the de~ignated 23 primary cycle type and bank addre~ ignal- A duplicate ~et of 2~ the~e control ~ignals is generated by gener_tor~checker 617 from the duplicate (mirror) sequence timing signals and the desig-26~wo~C~ nated mirror cycle type _nd b~nk addre~ ~ignals ~hese mlrror FINNEG~N. HENDERSON : .
~DUNNER RAS~ CAS~ and write enable signal~ are used for error checking - ~
. w.
Ø0... O. c. ~000-ho~ o l ~.

.

.. . `. ' '; .

~ 22~59 1 When the primary cycle type signals indicate a memory 2 tran~fer cycle i8 being performed, the primary bank address 3 8ign~18 identify one selectad bank of DRAM8 in memory array 600.4 MQmory driver3 630 include ~ep~rate RAS drivers for e~ch bank ofS DRANs in memery array 600. In generator/checker 617, the 6 primary RAS sign~l i8 ganer~ted during the memory transfer cycle7 and demultiplexed onto one of the lines connecting the 8 generator/checker to the RAS drivers. As a re~ult, only the RAS
9 driver corresponding to the selected DRAM bank receives an as-serted RAS signal during the mamory transfer cycle. During 11 refresh cycles, tha primary RAS ~ignnl i8 not demultiplexed and12- ~n a~erted RAS signal i8 received by each RAS driver. Durlng 13 ~t~tus register tranofer cycle~ ~nd ~PROM tran~fer cycle~, the 14 bank addre~s ~ignal~ sre unneces~ry.
~emory drivQrs 630 nl80 include CAS driver~. In generator/
16 checker 617, the prim~ry CAS s$gnal is generated during memory 17 tr~nsfer cycles and refre~h cycle~. The primary CAS ~ignal i~ ¦18 not demultiplexed and an a~serted CAS signal i~ received by e~ch 19 CAS driver.
DuEing ~e~ory write cycls~, the primary WE signal is gener-21 at~d by gener~tor/checker 617. The as~erted WE ~i~nal i5 22 provid~d by driv0r~ 630 to e~ch DRA~ bank in m~mory array 600 23 Hb~ ~ r, ~ writo can only be execut~d by the Gelectad DRA~ bank, I
24 which al~o recelv~s asserted RAS and CAS sign~
In the pref~rred eQbodiment o~ the in~Qntion, during memory 2~.~WO"IC5~ tran~fer cycles the primary RAS signal i~ assorted during th~ T2FINNEC~, HENDERSON
~6DUNNER tran~ition, i8 st~ble from at lea~t ten nanos~conds before the ,,..., ..~....~,.w.
0~0~. O. c. ~000-~0.,~.. 0 . . 1~

2~2~59 1 T3 transition, and i8 dea8~erted during the last T7 transition.
2 The primary CAS cignal iB asserted fifteen nanoseconds aftsr 3 each T4 transition, and i8 dea~serted during each T7 transition.
4 D~-ring memory write cycle~ tha primary WE signal is a~serted S during the T3 transition, is ~table from at least ten 6 nanoseconds before the first T4 tran~ition, and i8 deasserted 7 during the laat T7 transition.
8 When the primary cycle type signals indicate a memory 9 refresh cycle i~ being performed, generator/checker 617 cau~es memory array 600 to perform memory refresh operation~ in 11 re~ponse to the primary ~equence timing 8ig~ provLded by 12- sequencer 616. During these refresh operation~, the RAS and CAS
13 sign~ls are generated nd distributed by the generator/checker 14 in rever~e order. Th~ mode of refresh requlre~ no external addressing for bank, row, or column.
16 During tr~nafer cycle~, ~CC ~ignals are transferred on the 17 time divioion multiplexed bidirectional lines of buse~ 80 and 82 18 at times when data i8 b~ing transferred on bu~ 85. However, 19 the~e 8a~ lines ars u~ed to tran~fer control (e.g., cycle type and addres~ (o.g., memory board addres~ and bank address) 21 ~ignsl~ at oth-r tl~ea during the tran~fer cycle. I ~ -22 Th- tran~cei~sr~/registers ~n primnry/mirror de~ignator 612 23 include r~c~l~er~ and tran~mitters th~t are re~ponsive to 24 sequance tlmlng ~ignal~ and tx/rx regi~ter control ~ignals 2S provided by sequoncer 616. The ~equence timlng ~ignals and tx/ ¦
~wOr,,c.... rx reglster control ~ign~l~ enable multiplexing of ECC signals FI~CW. HENDER50N
l~l~ow. C~RRETr 28~ DU~ R .
2~ T~--7. ~ . w.
~~ o-~. D. C. ~000~
3~o~ 0 --22-- l, '.

~22~
1 and addre~s and control ~ignals on the tLme division multiplexed ¦
2 bldirectional lines of buses ao and 82.
3 Preferably, control and addre~s ~ignals, ~uch as cycle 4 typQ, memory board address, ~nd bank address signal~, are S transmitted by memory controllers 70 and 75 and prssented on the 6 timeshared line~ of buse~ 80 and 82 at the beginning of either sLngle or multi-longword tran~far cycles. These signPls start 8 their transition (while the sequencer i8 in the SEQ IDL~ ~tate) 9 concurrent with activation of the cycle t~m1ng signal, and rema~n ~table through T2. Therefore, in the transceivers/
11 regi~ters of de~ign~tor 612, the receiver~ ~re enabled ~nd the 12 transmittars are set into their tristate mode at least until the 13 end of state SEQ 2.
14 The cycle type a~gn~ls identlfy which of the following listed functiona will be performed by memory arr~y 60 during the 16 cycle: memory read, memory write, status reglster read, statu~
A 17 regi~ter write, ~PRO~ read, ~PROM write, And refresh. The 18 designated prim~ry cycle type ~ignal~ received by designator 612 19 are provid~d to Requencer 616 and used in generating tx/rx ,, 20 control ~$gnals ~nd ~equence timing ~ign~18. For example, in j 21 data tran~c-iv~rs/regi~ters 620 and in ths tran~ceivers/
22 register~ of de~ignator 612, the receiver~ ~re enabled and the , 23 tran~itt~rs are ~et into thsir trlstate mode ffl oequencer 616 1l throughout a wrlt~ cycle. However, in data tran~ceivers/
register~ 620 and in the tran~ceivers/regi~ters of de3ignator 26.^wo~ 612 during a re~d cycle, the receivers are set into their FI~EG~N, HENDERSON
,~UOw, CARRETr U ~ DUNNER
2!r-- ~ T. N. ~.
W~O~O~. O. C. ~000 ~0~ 0 ., ' I .

- ~ 2(~22~9 1 tristate mode and the transmltters are enabled by sequencer 616 2 after the cycle type, memory board address, and bank addres~
3 ~ignals h~ve been received at the beginning of the cycle 4 In the preferred embodiment, data transferred to or from memory array 600 i8 checked in each memory module 60 using an 6 Error Detecting Code (EDC), which ia preferably the same code 7 required by memory controllers 70 and 75 The preferred code is 8 a single bit correcting, double bit detecting, error correcting 9 code (ECC) During a memory write cycle, memory controller 70 tr~nsmits 11 at laast one longword of data on data bus 85 and simultaneously 12 tran~m$ts a corresponding set of ECC signala on bus 80 13 Meanwhile, memory controller 75 trani~mits a second set of ECC
14 3ignals, which ~l~o corre~pond to the longword on data bus 85, on buJ 82 16 As embodied herein, during a memory write cycle the data ~ ¦
17 and the ECC signal- for each longword are presented to the 18 recelver- of data tran~ceiver~regiisters 620 and to the receiv-19 ers of the tran~ceiver-/reg$~ter~ of de~ignator 612 The data and the ECC ~lgnal~, whlch are ~table at least ten nanoseconds 21 befor the T4 tran~ltion and remain stable unt$1 fifteen 22 nanosecond- after the T6 transition, are latched into the~e 23 tran~ceiver~/reglster- During thiis time period, memory 24 controller~ 70 and 75 do not provide address and control ~ignals 26 on the ti e-hared l~ne- of bu-e- 80 and 82 2~,~o"lc"
FI~EC~N. HENDER50N
~o~G~RR~r ~a DUNNEI- ~ .
3~ oT~o~ a~ - ! , .
~0..... 0 I , .. .

-~ ~2`~:2~g`

1 The de~ignated primary ECC ~i~nals received by designator 2 612 and the longword of data received by transceivers/registers 3 620 during the memory write cycle are provided to the data 4 inputs of the DRAMs in each of the eight banks of memory array 600 and to ECC generator 623 The generated ECC i8 compared to 6 the designated primary ECC by comparator 625 The designated primary ECC signal~ also are provided to ~CC comparators 625, 8 together with the designated mirror ECC 3ignals 9 As ei~bodied herein, during a memory read cycle, at least one longword of data and a corresponding ~et of ECC signal~ are 11 read from mamory array 600 and re~pectively steered to data 12- transceivers/register~ 620 and to the transceivers/registers of 13 designator 612 During transition T7 of the memory read cycle, 14 tha dat~ and the ECC signals for each longword are available from memory array 600 and are latched into these transceivers/
16 rQgisters The data is also presented to the ~CC gener~tor 623 17 and its output is compared to the ECC read from memory 18 After latching, the data and the ECC aignals are presented 19 to data bu- 85 and to bu~e- 80 and 82 by the transmitter~ of data tran-ceiver-/regl~ter~ 620 and by the transmitter~ of the 21 tran~celver~/regi~ters of design~tor 612 The same ECC ~ignal3 22 are trano~itt~d from the tran-ceiver~/regi~ter~ in de~ignator 23 612 to memory controller 70 and to me~ory controller 75 The 24 data and the ECC ~ignal~ tran~mitted on data bu~ 85 and on buses 2S 80 and 82 are ~table from fLfteen nano~econd- after the T7 26~wo~C~ tran~ition until five nanosecond- beforo the following T6 FINNEC~N, HENDERSON
OW,~RRETT tran~itlon ~ln the case of a multl-longword transfer) or until ,"t ~,w.
~JI~ TOII. D. e ~0000 00~ --0 '~ .

.

2~222~9 1 five nano~econds before the following T IDLE transition (in the 2 ca~e of a single longword tran~fer or the last longword of a3 multi-longword transfer) During this time period, memory 4 controller~ 70 and 75 do not provide address and control signal~
S on the timeshared lines of buses 80 and 82 The transmitter~ of 6 data transceivers/register~ 620 and the tran~mitter~ of the 7 transceivers/registers of designator 612 are set into their 8 tri~tate mode during the followinq T ID~E transition 9 Comparator 614 is provided to compare the address, control, and timing signals originating from controller 70 with the cor-11 re~ponding address, control, and t~m~ng signals originating from 12- controller 75 The de~ignated primary cycle timing si~nal~, 13 cycle type signals, memory board addres~ signals, and bank ad-14 dres~ signals, together with the de~ignated mirror cycle timing signals, cycle type signals, memory board addre~s signal~, bank 1 16 address signals, row addre~ signals, and column address 17 signal~, are provided from de~lgnator 612 to comparator 614 18 The de~ignat~d primary row addre-- signal~ and column addres~
19 signals are provided from the outputs of driver~ 630 to ~ 20 comparator 614 Both set~ of signal~ are then compar~d ¦ 21 If ther i- a miscompare between any of the address,22 control, ~nd t~ming ignals originating from th~ mamory 23 co~troller-, comparator 614 generate~ an appropriate error24 ~ ~lgnal A~ ~hown in Figure 6, board addre~ error, bank addr~ss error, row addre-s error, column addres~ error, cycle type 26~wo"~ addre-- rror and cycle tt~1ng error signals may be output by FINNEC~N, HENDEII50N
F~WO~IV, C~RRETr ~Z7~D~NER th~ co~paratox ,, - ,. -.~..., ,.. w.
w~l~o~o~, 0. c . ,000.
~0.,.. ,.. 0 1.
-26- 1 ;

2~22~9 1 Generator/checker 617 compare~ the primary control and 2 t~ming signals generated by ~equencer 616 and generator/checker 3 617 using the designated primary bank address, cycle typst, and 4 cycle timing signal3 with the mirror control and timing signals gensrated using the designated m~rror bank address, cycle type, 6 and cycle timing signals The two sets of sequence timing 7 signals are provided by sequencer 616 to generatorJchecker 617 8 The prim~ry RAS, CAtS, and tWF ~ignals are provided from the 9 output~ of drivers 630 to generator/checker 617 As indicated previously, the mirror RAtS, CAtS, and WE signals cre generated 11 internally by the generator/checker Gener~tor/checker 617 12- compares the primary RAS, C~tS, WR, and sequence timing signals 13 to the mirror RAtS, CAtS, tWE, and sequence timing signal~
14 If there i~ ~ mi~compare between any of th~ control andtiming signals orLginating from sequencer 616 or generator/
16 checker 617, the generator/checker generates an appropriate 17 error ~ignal As shown in FLgure 6, ~equencer error, RAtS exror, 18 CAtS error, and we error signals may be output by generator/
19 checker 617 Error ~lgn~l- are provided from comparator 614 and from 21 gen rator/ch ck-r 617 to addre~/control error logic 621 In 22 re-pon-e to rec-ipt of an error signal from comparator 614 or23 fro~ gen r~tor/checker 617, addre~ct/control error logic 621 24 tran~it- ~n addre~-/control error signal to CPU module 30 to 2S $ndicate th- detection of a fault due to a miscompare between26~worrc~ ~ny addro~, control, or timing ~ignal~ The address/control FINNECAN, HEND~R50N
o~ ~RRETr error ~ignal i8 sent to error loglc in memory controllers 70 and .w.
w~ o~ol~. O. ~. 000-~0~ -.0 .

.

~ ~22~5~

1 75 for arror handling. The transmi~ion of the ~ddre~s/control 2 error signal to CPU module 30 cau~es a CPU/MEM fault, which i8 3 discus~ed in greater detail in other sections.
4 The error signals from comparator 614 ~nd from generator/
S checker 617 al80 are provided to stAtu~ registers 618. In the 6 status registers, the error signal~ and all of the address, 7 control/ timing, data, and ECC signals relevant to the fault ara 8 temporarily stored to enable error diagnosis and recovery.
9 In ~ccordance with one aspect of the invention, only a single thirty-two b~t data bus 85 i8 provided between CPU module 11 30 and me~ory module 60. Therefore, memory module 60 cannot 12~ comp~re two 8et8 of data from memory controllers 70 and 75.
13 However, data integrity i9 verified ~y mamory module 60 without 14 using A duplicate ~Qt of thirty-two d~ta line~ by checking the two separate aets of ECC 8ignal8 that are transmitted by mamory 16 controllers 70 and 75 to mamory module 60.
17 A~ shown in Piq. 6, control log1c 610 includes ECC
18 generator 623 and 8CC compar~tora 625. The designsted primary ! 13 and mirror ~CC signals are provided by design~tor 612 to the ECC ¦
3 co~p~rntor~. During a memory wr~te cycle, the de~ign~ted 21 pr$mary ~CC Jignals are compared to the de~ignated mirror ECC
22 ~lgn~l~. As ~ re~ult, memory module 60 verifies whether memory 23 con~roller~ 70 ~nd 75 ~re in agr~Qment and whether the 2~ de~ignated pri~ary ECC ~ignal~ being stored in the DRA~ of me~ory Array 600 during the memory write cyclo are correct. I
2~WO~IC~ Furth~r w r~, the data presented to the data inputs of tha DRAM~
FINNECAN. HENDER50N .
FARAIIO~ GARRETr . 27~DUNNER during the memory write cycle i8 provided to ECC generator 623.
TI-~tT, ~. w.
W~ OTO~, o. C. ~ooo-~ ~0~ 0 29 -28- ~ ~
. . 1~

~22~
1 ECC generator 623 produces a uet of generated ~CC ~ignals that I -2 correspond to the data and prevides the generated ~CC signals to 3 ECC comparators 625. The designated primary ECC signals are 4 compared to the generated ECC signals to verify whether the data S transmitted on data bus 85 by memo~y controller 70 is the same 6 as thQ data being stored in the DRANB of memory array 600.
7 During a memory read cycle, the data read from the selected 8 bank of DRAMs iB presented to the ECC generator. The generated 9 ~CC signals then are provided to the ECC comparators, which al~o recsive ~tored ECC signal0 read from the selected bank of DRAMs.
11 The generated and stored ECC ~lgnal~ re compnred by ECC
1~ comparators 625.
13 If there i8 a miscompare b~tween any of pairs of ~CC
14 signal~ monitored by ~CC comp~rator~ 625, the ~CC comparators gonerate an appropriate error ~ignal. As ~hown in Figure 6, 16 primary/mirror ~CC error, primary/generated ECC error, and 1 17 memory/generated ~CC error signal~ ~ay be output by the ECC
' 18 comparator~.
19 The~o ~CC error signals from ~CC co~parators 625 are provided to ~t~tus regLaters 618. In the ~tatu~ regi3tQrs, each I
21 of th- BCC error ~lgnal8 and all of the addre~s, control, 22 ti~lng, d~t~, and ECC signals relev~nt to an ~CC fault are 23 t-DQor~rlly stor~d to enable error diagno~i~ and recovery.
24 An ~CC error signal i8 a~erted by ~CC co~parator~ 625 on 2S an ECC error line and transmitted to CPU module 30 to indicate 26~Wo~C~ t~ detection of ~n ~CC faul~ due to a miscompare. The ~I~N, HENDERSON
~I~K~w, C~RRETr 28~ DLNNER
~ T~e-. ~ . W.
w~ .,~Ø.. O. c. ~Ooo-o~,.-. ---0 ! 31 . i ', . .

.~- , , , - ! - - ~ .
.'. . , , ' . ' . . ' ~ ~ ' ' . ' ' . , :

2~ 9 1 miQcompsre can occur during either of the t~o ECC checks 2 performed during a memory write cycle, or during the slngle ECC
3 check performed during a memory read cycle.
4 A~ hown in Figure 6, board ~elact logic 627 receives 810t S ~ignals from a memory backplane. The 910t signals ~pecify ~
6 unique slot location for each mamory module 60. 80ard selQct 7 logi~ 627 then compares ths slot signals with the de3ign~ted 8 primary bo~rd address signals tran~mitted from one of the memory 9 controllers via de~$gnator c$rcuit 612. A bo~rd selected s$gn~1 $8 generatad by bo~rd select log$c 627 if the ~lot signals are 11 the same a~ the designated primary board addres~ signal , 12 thereby enabling the other circuitry in control logic 610.
13 3. ~emorv Controller 14 ~emory controller~ 70 and 75 control the acces~ of CPUs 40 and 50, respQctlvely, to memory module 60, auxiliary memory 16 elements and, in th~ preferred embodimentt perform certain error 17 handling operations. The auxiliary memory elements coupled to 18 memory controller 70 Lnclude system RO~ 43, ~PRO~ 44, and 19 ~cratch pad RAM 45. ROM 43 holds cert~n stand~rd code, such a~
diagno~tlc~, con~ole driver~, and part of the bootstrap code.
21 E~PRON 44 i~ used to hold information such a~ error Lnformation 22 det~ct0d during the operation of CPU 40, which may need to be 23 modified, but which should not b4 lo~t when power i~ removed. I -24 Scratch pad RA~ 45 i8 U8Qd for certain operations per$ormed by 26 CPU 40 and to convert rail-unigue infor~tion (e.g., inform~tion 27L^wo~lc~
FIN~CAN . HENDER50N
F~ RRETr Z9a D~NER
30 11 IIT~CT, ~ W.
W~ O~O~ . D. C. ~OOtl--0~ 0 .

.
..

~,:. ~ , , . , , . . . . ~ ..... . .. , - , .

2~ 2~9 1 specific to conditions on one rail which is available to only 2 one CP~ 40 or 50) to zone information (e.g., information which 3 can be acce~sed by both CPUs 40 and 50).
4 Equivalent Qlements 53, 54 and 55 are coupled to memory controller 75. System ROM 53, E~PRO~ 54, and scratch pad RAM 55 6 are the same as system ROM 43, E~PROM 44, and scratch pad RAM
45, respectively, and perform the same functions.
8 The details of the preferred Qmbodiment of primary memory 9 controller 70 can be seen in Pigs. 7-9. ~irror memory controller ?5 has the same elements a~ shown in Figa. 7-9, but 11 differs slightly in operation. Therofore, only primary memory 12 controller 70'~ operation will be described, except where the 13 operation of m~mory controller 75 differs. Nemory controllers 14 70~ and 75' in processing system 20' have the same element~ and lS act the same as momory controllera 70 and 75, respectively.
16 Tho elemonts shown in Pig. 7 control the flow of data, ad-17 dreaaes and signala through primary memory controller 70. ¦ -18 Control logic 700 controls the state of the variou~ element~ in 3 19 Pig. 7 according to the signala received by memory controller 70 and tha atate engine of that memory controller which is stored 21 in control logic 700. Multiple~er 702 selects addresses from 22 on- of thr o aourcoa. The addre~sea can either come from CP7J 30 ~ -1 via recei~er 705, from the D~A engino 800 de~cribed below in 24 reference to Fig. 8, or from a refre~h re-ync address line which$' 25 is used to generate an artificial refreah during certain bulk 3 26~wo~-c~- memory tran~fers from one zone to another during re~ynchro-.~, FINNEC~N. H~NDERSON
F~O~. C~RRETr 77~DUNN~R niz~tion operationc-,~.~, ~ ~070~0C~OOO~
;0",-,~ 0 I ", "

~, .
. "" ,~

~222~9 1 The output of multiplexer 702 is an input to multiplexer2 710, a~ i8 dsta from CPU 30 received via receiver 705 and data 3 from D~A engine 800. The output of multiplexer 710 provides 4 data to memory module 60 via memory interconnect 85 and driver 715. Driver 715 i8 disabled for mirror memory control module~
6 75 and 75' because only one set of memory d~ta is sent to memory ~ modules 60 and 60~, respectively.
8 The data sent to memory interconnect 85 includes either 9 data to be stored in memory module 60 from CPU 30 or DMA engine 800. Data from CPU 30 and addresae~ from multiple~er 702 are 11 al~o sent to DMA engine 800 via th$s path and al~o via receiver 12 745 and ECC corrector 750.
13 The addresse- from multiple~er 702 al80 provide an input to 14 demultiplexer 720 wh$ch dlvides the addre~es into a row/column addres~ port$on, a boardJbank address port$on, and a single ¦ 16 board bit. The twenty-two bit~ of the row/column addre~ are ¦ 17 mult$plexed onto eleven line~. In the preferred embod$ment, the ~ 18 twenty-two row/column addre~ bit~ are sent to memory module 60 j 19 v~a dr~ver~ 721. The single board bit is prefarably ~ent to ~ 20 memory modul~ 60 via driver 722, and the other board/bank ¦ 21 addre-- blt~ are multiple~ed wlth ECC signals.
22 Nultlplexer 725 combin-~ ~ normal refre~h command for 23 m-eory controller 70 along with cycle type informat$on from CPU
! 24 30 (i.e., read, wr$te, etc.) and DMA cycle type information.
7 2S The normal refresh co~mand and the refreJh re~ync addre~s both 2~WO~ct~ cause mo~ory module 60 to $nitiate a memory refre0h operation.
, Fl ~N, HENDE~ON
7 F ~ GARRE~r :~ DUNNER
. ". w. I , ~I W~I~EOT , D C. ~OOD--32 1~ -32- ~

-~ ~, ~ . - , j . . . .

2~22~

1 The output of multiplexer 725 i8 ~n input to multiplexer 2 730 along with the board/bank addres~ from dQmultiplexer 720.
3 Another input into multiplexer 730 i8 the output of ECC
4 generator/checker 735. Nultiplexer 730 select3 one of the inputs and places it on the time-division multiplexed ECC/
6 address line~ to memory module 60. Multiplexer 730 allows those7 time-diviRion multiplexsd lines to carry board/bank address and8 additional control information as well a~ ~CC information, 9 although at different times.
ECC information i8 received from memory module~ 60 via 11 recQ~ver 734 and i~ provided as an input to ECC generator/
12- checker 735 to compare the ZCC generated by memory module 60 13 with that generated by memory controller 70.
14 Another input into ECC generator~check2r 735 is the output of multipleser 740. Depending upon whether the memory transac-16 t~on i8 a wr$te transaction or a re~d tr~ns~ction, multiplexer17 740 r~ceivas as inputs the me~ory data sQnt to memory module 60 18 from ~ultlplexar 710 or the memory data recelvad from memory ¦
l9 module 60 via receiver 745. Multiple~er 740 selects one of these ~ts of memory data to be the input to ECC generator/
21 chock-r 735. Gen~r~tor/checker 735 then generates the 22 appro~r~ate BCC code which, in addition to being ~snt to 23 ~ultlplQ~er 730, is al~o sent to ECC corrector 750. In the 24 preferr~d Hmbodim~nt, ~CC correctox 750 corrects any single bit ¦
errors in the ms~ary data received from memory module 60.
26~Wor~e~- The corrected memory data fro~ ~CC checker 750 i~ then sent FINNEG~N. HENDEII50N
, ~D~NER to the DMA engine shown in Fig. 8 a~ w~ll as to multiplexer 752. ~-:
I .
~<.Ø.. O. c. .000.
,.~"".,~ 0 -33- :

-:

'~' ' ' ' ' . :' , ' ' ' ' .

~-~`2~

1 The other input into multiple~er 752 i~ error inform~tion frsm 2 the error handling logic de~cribed below in connection with 3 Fig. 9. The output of multiplexer 752 i3 sent to CPU 30 via 4 driver 753.
Comparator 755 compares th~ d t~ sent fro~ multiplexer 710 b to momory module 60 with a copy of that data after it pa8~9~
7 through driver 715 and receiver 745. Thi8 checking determineR
8 whether driver 715 and receiver 745 are operating correctly.
9 The output of comparator 755 i9 a CMP error ~ign~l which indi-cates the pre~ence or absence of such a comparison error. The-11 CMP error feed~ the error logic in Fig. 9.
12- Two other elements in Fig. 7 provide a different kind of 13 error detection. Element 760 i~ a parity generator. ~CC dat~, 14 gen~rated either by the memory controller 70 on data to be ~tored in me~o~y module 60 or generatsd by memory module 60 on 16 data read from memory module 60 i~ sent to a parity gsnerator 17 760. The p~rity 8ign~1 from qanerator 760 i8 ~ent, via driver 18 762, to comparator 765. Comparator 765 compare~ the ECC parity 19 signal from generator 760 with ~n equivalsnt ECC parity signal generated by controller 7S'.
21 Parity gqnerator 770 porform~ the ~me type of a check on 22 the row/column and single bit board ~ddre~ ~ignal~ received .
23 fro~ demultiplex~r 720. Tho addres~ parity signal from parity 24 generator 770 i~ tran~mltt~d by a driver 772 to a comp~rator 7751 which al80 receive~ an addre~s parity ~ignal from controller 75. !
26~or~e~ Th~ output~ of compar~tor 765 and 775 are parity error ~ignals ~ FINNECAN. HE~DERSON
D~ER whic~ feed th~ error logic in Flg. 9.
17T~ Tn~T. 4. ~.
TOI-. O. C. ~000-_34_ .

,. ~0222r9 ~: ~
1 Fig. 8 shows the fundamentals of a DMA englne 800. In the 2 preferred embodiment, D~A engine 800 re~ides in memory co~trol-3 ler 70, but there is no requirQment for such placement. As 4 shown in Fig. 3, DMA engine 800 includes a data router 810, a DMA control 820, and DMA register~ 830. Driver 815 and receiver 6 816 provide an interface betw~en memory controller 70 and cross-7 link 90.
8 DMA control 820 receives internal control signal~ from 9 control logic 700 and, in response, sends control signals to placa data router 810 into the appropriats configuration.
11 Control 820 al~o cau~es data router 810 to set itd conflguration 12 to route d~ta and control signal~ from cro~-link 90 to the 13 memory control 70 circuitry shown in Fig. 7. Dat~ router 810 14 ~ends its ~tatu~ ~ignals to DMA control 820 which relay~ such signald, along with other D~A information, to error log~c in 16 Fig. 9.
17 Registers 830 includes a DMA byte counter register 832 and 18 a D~A addres~ register 836. These registers ~re ~et to Lnitial19 value~ by CPU 40 vi~ router 810. Then, during D~A cycle~, control 820 C~U8e~, via router 810, the counter register 832 to 21 increment and addre~s register 836 to decrement. Control 820 22 al80 c~u~ec the contenta of address registerA 836 to be sent to 23 memory modul~ 60 through router 810 ~nd the circuitry in Fig. 7 i 24 during DNA operatlon~.
As expla~ned ~bove, in the preferred embodimQnt of this 26uwo~ct- invention, the memory controllers 70, 75, ?0' and 75' al~o FINNEC~N. HENDERSON
'F~OW ~RRETr perform certain fund~mental error oper~tions. An e~mple of the ,77~ .
~OTO~. O. C ~OOO-- .
0~ 0 ''~ ' , .

~2~2~9` -1 preferred embodiment of the hardware to perform such error 2 operations are ~hown in Fig. 9.
3 As shown in Fig. 9, certain memory controller internal 4 ~ignals, such a~ timeout, ECC error and bus miElcompare, are inputs into diagno~t$c error logic 870, as are certain external 6 signals such as rail error, firewall miscompare, and addres~/
7 control error. In the preferred embodiment, diagnostic error 8 logic 870 receives orror signals from the other components of 9 system 10 via cross-links 90 and 95.
Diagnostic error logic 870 forms error pulses from the 11 error signals and from a control pul~e ~ignal generated from the 12 basic timing of m.~mory controller 70. The error pulses 13 generated by diagnostic error logic 870 contain certain error 14 information which i~ stored into appropriate locations in a diagnostic error register 880 in accordance with certain timing 16 sign~ls. System fault error addres~ register 865 stores the 17 addre~ in memory module 60 whlch CPU~ 40 and 50 were 18 communicating with when an error occurred.
19 The error pul~-~ from diagno-tic error logic 870 are al~o sent to error categorisation logic 850 which al~o receives 21 lnforq~tion fsom CPU 30 indicating the cycle type (e.g., read, 22 wrlt-, tc.). From that information and the error pulses, error 23 c~toQori~ation logic 850 determine~ the pre~ence of CPU/IO
24 error~, DMA error~, or CPU/MEM fault~.
2S A CPU/IO error is an error on an oporation that i8 directly 26~w-oii~c-~ attributable to ~ CPU/IO cycle on bu~ 46 and may be hardware FINNECAN, HENDE~ON
~ NNe~ recovorabl0, a~ explained below in regard to ra~ets. DMA error~
. w.
v~ OTO~, O, ~. 1000-~ ~o~ o .
.

~ 25~
1 are errors that occur during a D~A cycle and, in the preferred 2 embodimant, are handled principally by 30ftware. CPU~EM faults 3 are error~ that for which the correct operation of CPU or the 4 contents of memory cannot be guaranteed.
The output~ from error categorization logic 850 are sent to 6 encoder 855 which forms a specific error code. ~his error code 7 is then ~ent to cross-link~ 90 and 95 via AND gate 856 when the 8 error di~ble sign~ not present.
9 A~'ter receiving the error codes, cross-links 90, 95, 90 and 95' send a retry request signal b~ck to the memory 11 controller~. A~ shown ln Flg. 9, an encoder 895 in memory 12- controller 70 recoives the retry reque~t signal along with cycle ~ -13 type infor~ation and the error sign~ls (collectively shown as ~-14 cycle qualifiers). ~ncoder 895 then gener~te~ an appropriate ~ 15 error code for storage in a sy~tem f~ult error regi4ter 8g8.
¦ 16 System fault error register 898 does not store the same 17 inform~tion a4 diagno~tlc error regi4ter 880. Unlike the ~yst.~m 18 fault error regL-ter 898, the diagno~tic error register 880 only 19 contain- rail uniquo information, ~uch a- an error on one input from a cros~-link rail, and zone unique data, ~uch as an uncor-21 rectable ECC error in memory module 60.
; 22 Sy-t _ f~ult error regi-ter 898 al~o contains several bi~
23 which ~re u~ed for error handling. The-e include a NSM bit 3 24 Lndicating th~t ~ do~Lred memory location i8 m~s~ing, a NXI0 bit 226 indicating that a de-ired I/0 location is missing, a solid fault 27~wo~lc~
FIN~N, HENDE~50N
RI~ R~ETr Z9~ DUNNEI-3g ~ T, N. ~.
W~T~NOTON. O. C . ~000--~0~ 0 i ~ . ' 3332 _37_ ..
' ':
.

r ~2~9 1 bit and a transient bit The tranaient and solid bits together 2 indicate the fault level The transient bit also causes system 3 fault error address register 865 to freeze 4 Nemory controller status register 875, although technically S not part of the error logic, i~ shown in Fig 9 alao Register6 875 stores certain status informstion such as a DMA ratio code 7 in DMA ratio portion 877, an error di3able code in error disable 8 portion 878, and a mirror bus driver enablQ code in mirror bue 9 driver enable portion 876 The D~A ratio code specifies the fraction of memory bandwidth which can be allotted to DMA The 11 error disable code provides a signal for di~abling AND gate 856 12- and thu~ the error code The mirror bu~ driver enable code 13 provide~ a aignal for enabling the mirror bu~ driver~ for 14 certain data transactions 4 Cros~-link 16 Data for memory re~ync, DMA and I/O operations pas~ through -17 crosa-link~ 90 and 95 Generally, croa--links 90 and 95 provide 18 communications between CPU module 30, CPU module 30', I/O mod-¦ 19 ules 100, 110, 120, and IJO moduleJ 100', 110', 120~ (see Fig 1) 1 21 Cro---llnk~ 90 and 95 contain both parallel register~ 910 1 22 and ~-rial rogi~ter~ 920 aa ~hown in Fig 10 Both type~ of1 23 regiat-r- ~r0 uaed for interproce--or communication in the 2~ preferred embodiment of thia invention. During normAl oper~tlon, proceaaing sy~t _ 20 and 20' are ynchronizQd and 26L~wo~c~ data i~ exchanged in parallel between proceaaing ~ystems 20 and FINNEC~N, HENDER50N
,~DUNNER 20' u~ing parallel regi~terJ 910 in cro~--link~ 90/95 and17~ T~T. ~1. W.
W4~ TOI~. O. C. /000-~0~ 0 29 ~1 _33_ 1 ~

.

'.: . .

2~2~ 9 90~/95', respectilrely. When processing sy~temr, 20 and 20' are 2 not synchronized, most notably during bootstrapping, data i8 . .
3 exchanged between croæs-links by way of serial register~ 920.
4 The addre~ses of the parallel regi~ter~ are in I/O space as opposed to memory ~pace. Memory ~pace refers to locations in 6 memory module 60. I/O space refers to locations such a~ I/O and 7 internal sy~tem registers, which are not in memory module 60.
8 Within I/O space, addressQs can either be in æystem address 9 space or zone address space. The term "system address space~
!~ 10 refer~ to addresse~ that are accessible throughout the entire 11 system 10, and thus by both proce~ing sy~tem~ 20 and 20~. The 12 ter~ ~'zone address space~ refers to addre~ses which are 13 accQ~sible only by the zone containing the particular cross-14 link.
The parallel registers shown in Fig. 10 include a 16 communications register 906 and an I/O reset register 908.
17 Communications register 906 contains unique data to be exchanged 18 between zone~. Such data i~ u~ually zone-unique, ~uch as a l9 memory ~oft error (it is almo~t beyond the realm of probability ¦
that me~ory ~odule~ 60 and 60' would independently experience 21 th~ rror at the same time).
;~ 22 ~ecau~e the data to be stored into register 906 is unique, 23 the addrs~ of communication~ register 906 for purpo~es of 24 writing mu-t be in zone addre~ space. Otherwi~e, proce~ing sy~te~ 20 and 20~, because they are in lockstep synchronization i26-~wo~c~ and e~ecuting th~ l~e series of in~truction at ~ubstantially ~!FINNEC~N. HENDERSON
~EO~ G~RRE~r th-- 8~ ti~, could not store zone unique data into only th~
; ,,.... ,.. ~.. ,.. w. . . .. . ..
~...~0,.. O. c. .000.
~-Ø. ~ 0 39 _39_ 1 -. I ,,.

' , , ~' 2 ~ 2~S ~
1 commun~c~tions registQrs 906 in zone 11; they would h~ve to 2 store that same data into the communications registers 906~ (not 3 ~hown) in zone 11'.
4 Th~ ~ddres~ of communications register 906 for reading, however, is in system addrQss ~pace. Thus, during ~ynchronous 6 operation, both zones can simultaneously read the communications register from one zone ~nd then ~imultaneously read the communi-8 cations register from the other zone.
9 I/O reset regiRter 908 re~ide~ in Ryst2m addres~ space.
The I/O reoet register includes one bit per I/O module to indi-11 cate whother tho corre~ponding module i8 in a re~et ~tate. When 12- an I/O module i~ in a re~et statQ, it i8 effectively d~sabled.
13 ~arallel register~ 910 also include other registers, but an 1 14 understanding of tho~e other register~ i~ not neces~ ry to an underatanding of the pre~ent invention.
16 All of the serial cross-link registers 920 are in the zone 17 specific ~pace ~ince th~y are used elther for asynchronou~
18 communi~tion or conta$n only zone specific information. The ~ 19 purpose of the ~erihl cro~-link regiflters and the ri~l cross-3 20 link i~ to allow proco~Eors 20 and 20~ to communicat~ even ~ 21 though they ~r- not running in lockstep synchronization (i.e., 31 22 phn~-locked clocks and same mOEmory ~tate~). In the preferred 23 emhodiment, thero ars ~everal ~erial rQgistQrs, but they need .! 24 not be de~cribed to undsrstand thi~ invontion-Control and ~tatu~ regi~t~r 912 i~ a serial regiater which 2C~wo~c.... contaln~ ~t~tu~ and control flags. One of the flhg~ i8 an OSR
. FINNEC~N, HENDER50N
, ~DUNNER bit 913 which i~ u~ed for boot~tr~pplng ~nd indic~tes whether ~"~ "~ . w.
CtTO~t. 0. c . ,0.,..,....0 _40_ - ~ 20222~9 1 ths proce~sing system in the corresponding zone has already 2 begun its bootstrapping proce~ or whether the operating sy~tem 3 for that zone is currently running, either becAuse it~
4 boot~trapping process has completed, or because it undsrwent a resynchronization 6 Control and status register 912 also contain the mode bits 7 914 for identifying the current moda of cross-link 90 and thus 8 of process$ng system 20 Preferably mode bits include re~ync 9 mode bit~ 915 and cross-link mode bit~ 916 Rssync mode bits 915 identify cro~s-link 90 as being either in re~ync slave or 11 re~ync ma~ter mode The cro~-link mode blt~ 916 identify 12- cross-link 90 as being either in cro~s-link off, duplex, cross-13 link m~ter, or cross-link slave mode 14 One of the u~es for the serial registers is a status read oper~tion which allow~ the cro~--link in one zone to read the 16 status of the other zone's cros~-llnk Setting a statu~ read 17 reque~t flag 918 in serlal control and st~tu~ register 912 sends 1~ a r~que~t for statu~ $nform~tion to cro~--llnk 90' Upon 19 receipt of this ~e~-age, cro~-link 90' sends the content~ of its serlal control And statu~ register 912' back to cross-link 22 Plg 11 how- o~ of the element~ for routing control and 23 t~tu- lgnal~ (referred to a~ ~control code-~) in primary 24 cro-~-llnk 90 and mirror cro-~-link 95 Corre-ponding cro~s-26 link el _ nt- e~st in the preferred emb~diment within cross-27~AWo~r~c~ I , .
Fl ~,EGW, HENDEI~50N
s~ao~v. GARRETr DUNNER
~. ,t .. ~.. W. ¦ .
V~ lltlOTO~. O. c . .000. i -~, ~ ~o~ o ! ~. .

, ...

~ ~2~2~9 1 link8 90' and 95'. These codes are sent between the memory 2 controllers 70 and 75 and the I/0 modules coupled to module 3 inter~onnects 130, 132, 130~ and 132'.
Fig . 12 shows the elements in the preferred embodiment of primary cross-link 90 which are u~ed for routing data and 6 address 3ignals. Corresponding cros~-link elements exist in 7 cross-links 95, 90' and 95~.
8 In Fig. 11, the elements for both the primary cross-link 90 9 and mirror cros~-link 95 in processlng system 20 are shown, although the hardware i8 identical, because of an importAnt -11 ~nterconnection between the ele~ents. The circuit elements in 12^ mirror cro~s-link 95 which are equivalent to elements in primary 13 cross-link 90 are shown by the ~ame numk~r, except in the mirror 14 controller th~ letter "m~ i~ placed after the number.
With reference to Flgs. 11 and 12, the el~ments include 16 latche~, multiplexers, driver~ and receivers. Some of the 17 latches, ~uch as latches 933 and 933m, act as delay elements to 18 en~ure the proper timing through the cros~-link~ and thereby 19 m~intain ~ynchronization. AJ shown in Pig. 11, control codes from memory controller 70 are sQnt via bus 88 to latch 931 and 21 then to latch 932. The reason for such latching i8 to provide 22 approprlate delay~ to en~ur~ that data from momory controller 70 23 p~ through cro~-link 90 s$multaneou~1y with data from 24 memory controller 70'.
2S If code~ from memory controller 70 are to be sent to 2C~wo",c............. proce~-ing ~y~to~ 20' via cro~-link 90', then driver 937 i8 FINNEC~N. HENDEI-50N
~ ~RRE~r en~bl-d. The control code~ from memory controller 70 also pa~
OTO- O C. tOOO-- ¦ :
W21Lo~ O

, , . ' ' :

2~22~
1 through latch 933 and into multiplexex CSMU~A 935. If ~ontrol 2 codeE, are received into primary cros~-link 90 from cro~s-link 3 90~, then their path i~ through receiver 936 into latch 938 and 4 al~o into multiplexer 935.
Control codea to multiplexer 935 determine the source of 6 data, that i~ either from memory controller 70 or from memory 7 controller 70~, and place those ~odes on the output of 8 multiplexer 935. That output i8 ~tor~d in latch 939, again for 9 proper delay purpose~, and driver 940 is enabled if the codes are to be sent to module interconnect 130.
11 The path for data and addres~ signals, a~ shown in Fig. 12 12 is somewhat simil~r to the path of control sign,al~ ~hown in Fig.
13 11. The difference~ reflect the fact th~t during any one 14 transaction, data and addre~ses are flowing in only one direc-tion through cro~-link~ 90 and 9S, but control si~nals can be 16 flowing i~ both direetion~ during that transaction. For that 17 ~ame reason the data lines in bu~es 88 and 89 sre 1~ bidlreetional, but the control code~ are not.19 Data and addresses from the memory controllar 70, via bus 88, enter latch 961, then latch 962, and then latch 964. As in 21 Fig. 11, th~ latehe~ $n Fig. 12 provide proper timing to 22 malnt~in ~ynchronization. D~ta from memory controller 70~ is 23 buf~-r-d by reeeiver 986, stored in latch 988, and then routed 24 to the $nput o~ multiplexer MUXA 966. The ou~put of multiplexer I
2S 966 i8 ~torad in latch 968 and, if d~iver 969 i8 enabled, is C~wor~c~ sent to module interconnect 130.
Fl~ CAN, HENDERSON
~ ~IIOW. f8ARRETr ., Z ~ DUNNER
. W.
0~0~. O. c. ~000-~0...~ 0 .

2:~22~

1 The path for control codes to be sent to memory controller 2 70 is shown in Fig 11 Code~ from module interconnect 130 are 3 fir%t stored Ln latch 941 and then pre~ented to multiplexer 4 cs~nxc 942 Multlplexer 942 also receives control codes from parallel cro~s-link registers 910 and selects either the 6 parallel regi~ter codes or the code~ from latch 941 for 7 tran~mlssion to latch 943 If those control code~ are to ~e 8 transmitted to cross-link 90', then driver 946 i8 enabled 9 Control codes from cros~-link 90' tand thus from memory controller 70') are buffered by receiver 947, stored in latch 11 948, and presented a~ an input to multiplexer CSMUXD 945 12- CSMUXD 945 also receives a~ an input the output of latch 944 13 which ~tOreJ the content- of latch 943 14 Nultiplexer 945 solects either the codes from module interconnect 130 or from cros~-link 90' and presents those sig-16 nal~ a~ an input to multiplexer CSHUX~ 949 Nultiplexer 949 17 also receive~ a- input- a code from the decode logic 970 (for 18 bulk memory tr~n~fer- that occur during reaynchronization), 19 code~ from the erial cro~-link regi-ter- 920~ or a predetermined error code ERR Multiplexer 949 then select~ onss ¦
21 of tho~o input-, under the appropriate control, for storage in 22 latch 950 If tho~e codea are to be sent to m~mory controller 23 70, then driver 951 i- activated 24 The purpose of th error code ERR, which is an input into multiple~er 949, i- to ensure that an error in one of the rail3 26~wo~c~ will not cau~e the CPU- in the same zone as the rails to process FINNEC~N. HENDEI-50N
~DUNNER different inform~tion ~f this occurred, CPU module 30 would .7~ ~
; ~,~070~0c OOO.
c~0~ .0 _44~

,:

.

~ ~ 2~

1 detect a fault which would cause drastic, ~nd perhap~ unneces-ssry action. To avoid thi~, cross-link 90 contalns an EXCLUSIV~
3 OR gate 960 which compares the outputs of multiplexers 945 and 4 945m. If they differ, then gate 960 causes multiplexer 949 to select the ~RR code. EXC~USIV~ OR gate 960m similarly causes multiplexer 949m ~180 to select an ERR code. Thi~ code indic~tes to memory controllers 70 and 75 that there has been an arror, but avoids causing a CPU module error. The single rail g interface to memory module 60 accomplishes the same result for data and addre~ses.
11 The dat~ and address flow ~hown in Fig. 12 i~ Jlmilar to 1 12 the flow of control signals in Fig. 11. Data and addresses from !~ 13 module interconneet 130 are stored in lateh 972 and then ~ 14 provided as an input to multiplexer MUX~ 974. Data from the i 15 parallel registers 910 provide another input to multiplexer 974.
16 The output of multiplexer 974 is an input to multiple~er NU2C
17 976 which also reee~ves data and addre~se~ stored in lateh 961 18 that were originally sent from memory eontroller 70.
19 Nultiplexer 976 then seleet~ one of the input~ for storage in lateh 978. If the data and addre~e~, either from the module 21 intereonn-et 130 or from the memory eontroller 70, are to be 22 sent to eroa~-link 90', then driver 984 i8 enabled.
3 D~t~ fro~ ero~-link 90' iJ bufferod by reeeiver 986 and 24 stored in lateh 988, whieh al~o provide~ an input to multiplaxer 2S ~ m D 982. The other input of multiplaxer NUXD 982 is the output 26~0~1Ct~ of lateh 980 whleh eontain~ data and addre~e~ from latch 978.
FlNNec~N~ HENDERsoN
F~ G~RRETr ~ . -~ 27~D~NeR ~ultlple~or 982 then seleets one of it~ input~ whieh is then ~"~ " ~T~ , N. ~It ~N~-OI-. O. C. 000--': ~0~ 0 ~ .
2~ 45-.

~ 202~2~9 1 stored into latch 990 If the dat~ or addresse~ are to be sent 2 to memory controller 70, then dr$ver 992 is activated Data 3 from serial regi~ter~ 920 are 3ent to memory controller 70 vLa 4 driver 994 The data routing in cross-link 90, and more particularly 6 the xonreol elements in both Flgs 11 and 12, i8 controlled by 7 ~everal signals generated by decode logic 970, decode logic 971, 8 decode logic 996, and decode logie 998 This logic provide~ the 9 sign~ls which control multiplexers 935, 942, 945, 949, 966, 974, 976, and 982 to select the appropriate input ~ouree In 11 addition, the deeode logie also control~ drivers 940, 946, 951, 12- 969, 984, 992, and 994 13 Mo-t of the control ~lgnal~ are generated by decode logic 14 998, but ~ome are generated by deeode logie 970, 971, 970m, 971m, and 996 Deeode logie 998, 970 and 970m are conneeted at 16 position~ that will ensure that the logie wlll reeeive the data 17 and eode~ neees~ary for control whether the data and codes are 18 reeeived fro~ it~ own zone or from other zone9 T~8 purpoae oi deeode logie 971, 971m and 996 is to ensure that the dr~v r- 937, 937m and 984 are ~et into the proper 21 ~tate Thl- ~early deeode~ make~ sure that data addresses and 22 code- will bo forwarded to the proper eross-links in all case~
23 Without ueh early deeode logie, the cros--links could all be in 24 a ~tat- with their driv-r~ d$sabled If one at the memory eontrollQrs were al~o dl~abled, then its cro~s-link~ Twould never ¦
26~wo~C~ r-eelve ~ddre~ses, data and control eode~, effeetively di~abling ¦
FINNEG~N, HENDER5aN
~ DUNNER hll the I/O module8 eonneeted to that eros8-link I ~
,"., .."..-....w.
W~ OTO~. O. e. ~ooo~
;o~J~ O I .' -46- 1 ~

- ~ 20222~g 1 Prior to describing the driver control signals gener~ted by decode logic 970, 971, 970m, 971m, and 998, it i8 nece~sary to 3 understand the different modes that these zone~, and therefore 4 the cross-links 90 and 95, can be in. Fig. 13 containe a diagram of the different state~ A-F, and a table explaining the 6 states which correspond to each mode.
! 7 At start-up and in other instancea, both zones are in state 8 A which is known as the OFF mode for both zone~. In that mode, the computGr sy~tems in both zones are operating independently.
After one of the zones' operating system requests the ability to 11 communicate with the I/O of the other zone, and that request i8 12 honored, then the zones enter the master/slave mode, ~hown as 13 ~t~tes B and C. In such modes, the zone which is the ma~ter, 14 has an operatlng CPU and ha~ control of the I/O modules of its lS zone and of the other zone.
16 Upon initiation of resynchronization, the computer system 17 leaves the master/slave mode~, e~ther ~tates ~ or C, and ent~r~a resync slave/re~ync ma~ter mode, which is shown a~ states E
19 and F. In tho~e mode~, the zone that was the master zone is in charge of bringing the CPU of the other zone on line. If the 21 re~ynchronization fail~, the zone~ revert to the ~ame ma~ter/
22 slave modo that they were in prior to the resynchronization at-23 tempt.
24 If the re-ynchronization i~ ~ucce~ful, however, then the ¦ 25 zones enter st~te D, which is the full duplex mode. In this, 26~o~C~ mode, both zones are operating together in lockstep ~ynchroniza-~ FINNEC~N. HENDER50N
~ . ~RRETT tion. Operation continue~ in thi~ mod until there i~ a CPU/MEM
,~ .77~ ,~ .. ,.. , ~. ~.
~ 0~. O. c. ~000-~ i30 ~ 47-.-, ' - .

' .

21~222~9 1 fault, in which ca~e the system enters one of the two mastar/
2 ~lave modes The slave is ths ~one whose processor experienced 3 the CPU/MEM fault 4 When operating in ~tate D, the ful duple~ mode, certain errors, most notably clock phase errors, necessitate ~plitting 6 the system into two independent processing ~ystems This csuse3 7 system 10 ~o go bacX into 3tate A
8 Decode logic 970, 970m, 971, 971m, and 998 (collectively 9 referred to as the cross-link control logic~, which are shown inFigs 11 and 12, have acces~ to the re~ync mode bits 915 and the 11 cross-link mode bits 916, which are shown in Fig 10, in order 12 to determine how to set the cross-link drivers and multiplexers13 into the proper ~tate~ In addition, the cro~s-link decode 14 logic also receive~ and analyze- a portion of an address sent from memory controllers 70 and 75 during data transactions to 16 extract addressing information that further indicates to the 17 cross-link decode log$c how to set the state of the cross-link 18 multiplexers and drivers The information needed to set the states of the multiplex-ers i~ fairly ~traightforward once the different modes and 21 tran~action- aro understood The only determination to be made ¦ 22 i~ the ~ource of the data Thus when cross-links 90 and 95 are 23 in th lav mode, multiplexer~ 935, 935m, and 966 will ~elect 24 data addre--e~ and codo~ from zon~ 11' Tho~e multiplexers will al~o select data, addre~-e~ and code- from the other zone if i 26~wo"~c~ cro~s-link~ 90 and 95 ara in full duple~ mode, the addresq of an FINNECAN. HEN~ER50N
EA~ RRETr 77~DUNNER I/0 in-truction i- for a device connected to an I/0 module in I -~7T--11 T~tCT. ~. W.
~-~OTO O C OOO--~~ 30 -48-. I ~

- ~ 2:0:~22~9 1 zone 11, and the cro~-link with the affected multiplexer i8 in 2 a cross-over mode In a cross-over mode, the data to be ~ent on 3 the module interconnect i~ to be received from the other zone 4 for checking In the preferred embodiment, module interconnect 130 would receive data, addre~se~ and codss from the primary 6 rail ~n zone 11 and module interconnect would receive data, 7 addres~es and codea from the mirror rail in zone 11' 8 Alternatively, module interconnect 132 could receive data, 9 addresses and codes from the primary rail in zone 11' which would allow the primary rail of one zone to be compared with the 11 mirror rail of the other zone 12- Multiplexer~ 945, 945m, and 982 will be set to accept data, 13 addre~s and codes from wh~chever zone is the ~ource of the data 14 This i~ true both when all the cross-links are in full duplex mode and the data, address and codes are received from I/0 16 modules and when the cros~-link is in a re~ync slave mode and 17 the data, addre~ and codeJ are recei~ed from the memoryla controller- of the other zone 19 If the addre~ing information from memory controllers 70 and 75 indicate- that th ~ource of re~pon~e data and codes is 21 tho cro---link'- own parallol register~ 910, then multiplexers 22 942, 942m, and 974 are set to select data and codes from tho~e 23 regl-tor~ Slmilarly, lf the addressing information from memoxy 24 controllers 70 and 75 indicate~ that the source of response data is the cross-link'~ own serial register 920, then multiplexers 26~wo~c 949 and 949~ are set to select data and codes from tho~e FINNEC~N, HENDER50N
F,~30W CARRETT
~7 a DUNNER reg~ter~ j 1~ li TTllt~T, N. ~11.
~II-OTO- . D. C. ~000--q-o~ ---O
_49_ '.

~ ~2259 1 Nultiplexers 949 and 949m are al80 set to select data from 2 decode loglc 970 and 970m, respectively, if the information i8 a 3 control code during memory re~ync operations, and to select the ERR code if the EXCLUSIVE OR gates 960 and 960m identify a ml~compare between the data transmitted vla cross-links 90 and 6 95 In this latter case, the control of the multiplexers 949 7 and 949m i8 generated from the ~CLUSIVE OR g~tes 960 and 960m 8 rather than from the cross-link control logic Multiplexers 949 9 and 949m also select codes from serial cros~-link registers 910 when tho~e regi~ters are requested or the output of multiple~ers 11 945 and 945m when those codes are requested Nultiple~ers 945 12- and 945m ~elect either the output~ from multiplexer~ 942 and 13 942m, re~pectively, or I/O code~ from cro~-linka 90' and 95~, 14 respectively Multiplexer 976 selects either data and addresses from 16 module interconnect 130 in the ca~e of a tranaaction with an I/O
17 module, or data and addresses from memory controller 90 when the 1~ data and addresseA are to be sent to cro~-link 90' either for 19 ItO or during memory re~ynchronization Drivera 937 and 937m are activated wh n cros--linka 90 and 21 95 ar- in duple~, master or re~ync master mode~ Driver~ 940 22 and 9~0m ~re ~ctivated for I/O tranaaction~ in zone 11 Driver~
23 946 and 946~ ~re activated when cro---llnk~ 90 and 95 are in the duplex or slave mode~ Driver~ 951 and 951~ are always activated 26~wo,,c........... Driver 969 i- activated during I~O writ~ to zone 11 1 -,-FI~NECAN. HENDER50N
~ eowcARRETr Driver 964 i~ ~ctlv~ted when cro~-link 90 i8 sending data and ., ,", " .,"",, ". ~. , . .
. O. c. 000-~, .Ø~ O
2~
;~;

.~ .
, 2Q2~259 1 addresse~ to I/O in zone 11~, or when cro~-link 90 i~ in the 2 resync master mode. Receiver g86 receives data from cros~-link 3 90'. Drivers 992 and 994 are activated when data is being ~ent 4 to memory controller 20; driver 994 is activated when the contents of the serial cross-link register 910 are read and 6 driver 992 i8 activated during all other read~.
7 5. O~cillator 8 When both processing sy~tems 20 and 20' are each performing ¦
9 the s~me functions in the full duplex mode, it i~ imperative that CPU modules 30 and 30' perform oporation~ at the same rate.
11 Otherwise, mas~ive amounts of processlng tlme wlll be con~med 12 ~ in resynchronizing processing systemJ 20 and 20' for 1/0 ~nd 13 interproce~sor error checking. In the preferred embodiment of 14 proce~slng ~ystems 20 and 20', their basic clock ~ignAl are ~ynchronized and phase-locked to each other. The fault tolerant 16 computing ~ystem 10 includQs a timing ~ystem to control the 17 frequency of the clock signals to proce~ing sy~tems 20 and 20 18 and to mini~lze the pha~e differenco between the clock signals 19 for each proc~s-lng ~y~tem.
Fig. 14 ~how~ a block diagram of the timing system of thi 21 in~nt~on embedded in proce~sing systems 20 and 20~. Tha timing 22 sy-t-- comprl~e- o~cillator ~y~tem 200 in CPU module 30 of pro-23 ce~-ing sy~t~m 20, and oscillator sy~tem 200' in CPU module 30 24 of proca~lng systQm 20'. The elements of oscillator 200' are ¦ equivalent to tho~e for oscillator 200 and both oscillator 2C~w or~c~ j systemJ' operation i~ the ~ame. Thus, only the element~ and ~C~N. HENDER50N

5~30~. C~RRETr ~6 DUNNER
29~T~T, ~. W. l 0-0~-. D. C. 000-- ¦
~-0~ 0 1 !

` 2:~2~3 1 oper~tion of e6cillator system 200 will be de~cribed, except if 2 the oporations of oscillator 8y8tem~ 200 and 200' differ.3 A~ Fig. 14 show~, much of oscill tor system 200, 4 specifically the digital logic, lies inside of cross-link 95, but that placement i~ not required for the present invention.

6 Oscillator syatem 200 includes a voltage-controlled crystal 7 oscillator (vcxo) 205 which generates a ba~ic oscill~tor ~ignal 8 preferably at 66.66 Mhz. The frequency of VCXO 20S can be9 ad~usted by the voltage level at the input.
Clock distribution chip 210 divide~ down the ba~c 11 oscillator signal and preferably produce~ four primary clock~
12 all having the sam~ frequency. For primary CPU 40 the clock~
13 are PCL~ L and PCL~ H, which are logical inverses of each other.
14 For mirror CPU 50, clock distribution chip 210 produces clock signals ~CLR L and MCLR H, which are alJo logical inverses of 16 aach other. The timing and pha~e relatlonship of the~e clock I ~
17 signal~ are shown in Fig. 15. Preferably, requency of clock ~ -18 signals PCLR L, PCLR H, MCL~ L, and MC~ H is about 33.33 Mhz.
19 Clock chip 210 al~o produce~ a phase-locked loop sign~l CL~C H
at 16.66 Mhz, al~o ~hown in Fig. lS. Thi~ phaso locked loop 21 signal i- ~ent to clock logic 220 which buffer~ that signal.
22 Clock logic buff-r 220 ~end~ the CLXC H signal to 23 o~cillator 200' for u~e in ~ynchronization. Clock logic buffer I -24 220' in o~cillator 200' sends it~ own buffered phase-locked loop signal CLRC' H to pha~e detector 230 in oscillator 200. Phase 26~wo~c~- detector 230 al~o receives the buffered pha~e locked loop ~ignal C,~N. HENDERSON .
e~ ~BO~ C~RRETr ~ a DUNNEA . .
2~ ., . W.
.v~ o~, O. c. ,000.
~ ~0., .. , .. 0 ':
31 -52- ~

.~,..

.' ~

-- 2~222~

1 CLRC H from clock logic 220 through del~y elQment 225. Delay 2 element 225 approxLmates the delay due to the cable run from 3 clock logic buffer 220~.
Phase detector 230 compares its input ph se locked loop ~ignals snd generates two output~. One is a phase differences 6 signal 235 which i8 sent through loop amplifier 240 to the 7 voltage input of VCXO 205. Phs~e differences signal 235 will 8 c.use amplifier 240 to generate a signal to alter the frequency 9 of VCXO 205 to compensate for phase difference~.
The other output of phase detector 230 is a phase error 11 signal 236 which indicate4 possible ~ynchronism fault~. .
12- Fig. 16 i8 a detailed diagram of pha~e detector 230. Pha~e 13 detector 230 include- a pha~e comparator 232 and a voltage 14 comparator 234. Pha~e comparator 232 receive~ the clock signal from dolay element 225 (CI.KC H) and the pha~e lock loop clock 16 signal from oscillator 200~ (CL~C' H) and generates phase dif-17 ferences signal 235 a~ a voltage level representing the phase 18 difference of tho-e signal~. ¦
19 If proce~ing ~ystem 20 were the Hslave" for purposes of clock 4ynchron~zation, ~witch 245 would be in the 'S~AVE."
21 po4itlon (l.e.~ clo~ed) and the voltage level 235, after being 22 ~mpllf~ d by loop a~plifier 240, would control the frequency of 23 VC~O 205. If both switche~ 245 and 245' are in the "master~
24 position, proce~sing systems 20 and 20' would not be pha~e-26 locked and would be running a~ynch onou~ly (independently).
27~o.~.~,c..
C~N. HENDER50N I
12 ~UIOW. G~RRE~r . . .
Z ~ DUNNER :
31pt~ w. , yllltto~ott~D ' '- ! . .
3i '-;'--' ! -33 1 -53- :

, ~ ~2~2~9 1 The ~oltage level of phaee differenceo o,ignal 235 ie al80 2 an input to voltage comparator 234 a,~ are twc reference 3 voltages, Vrefl and Vref2, r9pre~enting acceptable range~ of 4 phase lead and lag. If the phase difference i~ within tolerance, tha PHA~,~ E M OR ~ignal will not be activated. If the 6 phase difference is out of tolerance, then the PHA8~ ~RROR
7 signal 236 will be activated and sent to croe,s-link 95 via clock 8 decoder 220.
9 6. I/O Module Fig. L7 showe a preferred embodimQnt of an I/O module 100.
11 The principle~ of op~,ration I/O modul~ 100 are applicablo to the 12- other I/O modules as well.
13 Fig. 18 show- the elemento in the preferred em,bodiment of 14 firewall 1000. Firewall 1000 include~ a 16 bit bue interface 1810 to module interconnect 130 and a 32 bit bus interface 1820 16 for connection to bu- 1020 ~,hown in Fig. 17. Interfaces 1810 17 and 1820 are connected by an internal firewall bus 1815 which 18 aleo interconnect~ with th- other elements of firewall 1000.
19 Preferably bu- 1815 i~ a parallel bu~ either 16 or 32 bit~ wide.
I/O module 100 i8 connected to CPU module 30 by mean~ of 21 du~l rail module interconnectJ 130 and 132. Each of the module 22 int-rconnecto i8 receLved by firewallJ 1000 and 1010, reepec-, 23 tiv ly. Ono of tho flrewall~, which i8 UQiUally, but not alway~ ¦
2~1, firewall l000, write- the data from, module interconnect 130 onto 2S bu- 1020. The other firewall, in thi~ ca~e firewall 1010, 26~WO~IC~ check- that data again~,t its own copy received from module ;~. HENDER5C'N :
F~i ~EiOW, C~RRETr , ~ a DUNNER ¦
, 0.0-.. O. c ~000~ 1 - . .
~-0- ~9~ -0 1 31 1 _54_ ~`~

. ':
: . .
,' ~

interconnect 132 using firewall comparison circu$t 1840 ~hown in 2 Fig. 18. That checking is effective due to the lockstep 3 synchronization of CPU modules 30 and 30' which causes data 4 written to I/O module 100 from CPU modules 30 and 30' to be available at firewalls 1000 and 1010 substantially 6 ~~imultaneously.
7 Firewall comparison circuit 1840 only checks data received 8 from CPU modules 30 and 30'. Data sent to CPU module~ 30 and 9 30' from an I/O device have a common origin and thus do not requirQ checking. Instead, data rece$ved from an I/O de~ic~ to 11 be ~ent to CPU modules 30 and 30' i- checked by ~n error 12- detection code (~DC), ~uch a~ a cycl$cal r~dundancy check (CRC), 13 which i~ performed by ~DC~CRC generator 1850. EDC/CRC generator 14 1850 i8 also coupled to internal firewall bu~ 1815.
EDC/CRC generator 1850 generate~ and check~ the ~ame 16 EDC/C~C code that is uaed by the I/O device. Preferably, I/O
17 module 100 generate~ two ~DC. One, which can also be a EDC/CRC, 18 is used for an interface to a network, such a~ the Ethernet 19 packet network to which module 100 ia coupled (see element 1082 in Fig. 17). Th- other i~ u-ed for a dlsk interface ~uch a~
21 di~k int-rface 1072 in Fig. 17.
22 EDC/CRC cover~go i8 not required between CPU module 30 apd 23 I~ module 100 becau~e the module interconnect~ are dupllcated.
24 For example $n CPU module 30, cross-link 90 communicates with 2S firewall 1000 through module $nterconnect 130, and cross-link 95 ¦
26~wo~r~C~ communicates with firewall 1019 through module interconnect 132. !
~N. HENDER50N
F ~BOW, G~RAnr 11~ DUNNE~ . .... . - . :.
w.
TO~.O c ~000- ~
-O I ,:

. I . ',.

, .. ., . . .. .. .. ... . ... , ... . .... . . . . ... .. . . ..... , . .. . . ~. ...

~2~2~9 1 A me~age received from Ethernet network 1082 is checked 2 for a valid EDC/CRC by network Gontrol 1080 ~hown in Fig. 17.
3 The data, complete with EDC~CRC, i8 written to a loc 1 RAM 1060 4 al80 ~hown in Fig. 17. All data in local RAM 1060 i8 S tr~nsferred to memory module 60 using DMA. A DMA control 1890 6 coordinate~ the tran~fer And directs BDC/CRC generAtor 1850 to 7 check the validity of the EDC/CRC encodad data being 8 tran~ferred.
9 Most data transfers with an I/O device are done with DMA.
Data i~ moved between main memory and I/O buffer memory. When 11 d~ta i8 moved from the main memory to an I/O buffer memory, an 12 EDC/CRC may be appendad. When the data is moved from I/O buffar13 memory to ~ain m~ory, an EDC/CRC may bo check~d and moved to 14 main mamory or may ~e strippQd. When data i~ moved from the I~Obuffer memory through an external device, ~uch a~ a disk or 16 Ethernet ~daptor the EDC/CRC may be checked locally or at a 17 di3tant receivin~ node, or both. The memory data packets may 18 have thQir ~DC/C~C generated at the distant node or by the lo~al 19 interface on th~ I/O ~odule.
Thi- op ation ensure~ that data re~iding in or being 21 trnn-forr~d through a single rail sy~tem like I/O module 100 is22 cov r d by ~n error detection code, which i8 preferably at lea~t 23 ~- r li~blo ~8 tho communications media the data will evQntually 24 pass through. Different I/O modules, for example those which handle ~ynchronou~ protocols, preferably h~ve an EDC/CRC
26~wo~c~- qenorator which generatea and checks the EDC/CRC code3 of the INNEG~. HENDER50N
~DUNNER appropriate protocols.
177~ T~ . N. 1~.
OTO- . O. ~ OOO-- ¦
~ ~0~ 0 1 3r~

~ 9 1 In generAl, DMA control 1890 handles the portion of a D~A
2 operation spQcific to the shared memory controller 1050 and 3 local RAM 1060 being addressed. The 32 bit bus 1020 is driven 4 in two different modes. During DNA setup, DMA control 1890 uses bu~ 1020 as a standard asynchronous microprocessor bus. The 6 address in local RAN 1060 where the DMA operation will occur i~ f 7 supplied by shared memory controller 1050 and DNA control 1890.
8 During the actual DMA transfer, DMA control 1890 directs DMA
9 control lines 1895 to drive bus 1020 in a synchronous fashion.
Shared memory controller 1050 will transfer a 32 bit data word 11 with bus 1020 every bus cycle, and DMA control 1890 keep~ track 12- of how many words are left to be transferred. Shared memory 13 control 1050 also controls local RAM 1060 and create~ the ne~t 14 DMA addre~
The I/O modules (100, 110, 120) are responsible for 16 controlling the re~d/write operations to their own local RAM
17 1060. The CPU module 30 i~ responsible for controlling the 18 tran~fer operat$ons with mRmory array 60. The D~A engine 800 of 19 memory controller~ 70 and 75 ~shown in Flg. 8) directs the DMA
operation- on the CPU module 30. Thi~ divi~ion of labor 21 pre~ nt- a fault in the DMA logic on any modulo from degrading 22 the d~t~ integrlty on any other module in zonQs ll or 11'.
23 The function~ of trace RAM 1872 and trace RAM controller 24 1870 are do~cribed in greater detail below. Briefly, when a fault is detected and the CPUs 40, 40', 50 and 50' and CPU
26.~wor~lc~- module~ 30 and 30' ~re notified, various trace RAM8 throughout NNEC~N, HENDER50N
F~A~OW G~RRETr Z~ ~ DUNNER l computer ~y~tam 10 are caused to perform certain functions ,", " .",~,.. ,. w.
0-0~ .O. c ., .~ 0 ' 30 -57-.' .

2~2~9 1 descrlbed below. The commun~cation~ with the trace RA~ take~
2 pl~ce over tr~ce bus 1095~ Trace ~AM control 1870, ~n re~ponse 3 to 8ign~18 from trace bus 1095, cause~ trace RAM 1872 either to 4 stop ~toring, or to dump its contents over tr~c3 bus 1095.
I/O module bus 1020, which i~ preferably a 32 bit p~r~llel 6 bus, couples to firewalls 1000 and 1010 a~ well as to other 7 elomonts of the I/O module 100. A shared memory controller 1050 8 i~ al90 coupled to I~O bus 1020 in I~O module 100. Shared 9 momory controller 1050 i~ coupled to a local memory 1060 by a ~hared memory bus 1065, which preferAbly carries 32 b~t data.
11 Preferably, local memory 1060 i~ a RAN with 256 Rbyte~ of 12- me~ory, but the size of RA~ 1060 i8 di~cretion~ry. Th~ sh~red 13 memory controller 1050 and local RAM 1060 provide mQmory 14 cnpability for I~O ~odule 100.
Di~k controller 1070 provid~s a ~tandard interface to a 16 disk, such as disks 1075 and 1075' in ~ig. 1. DisX eontroller 17 1070 i~ also coupled to shared memory controller 1050 either for 1~ U80 of local RA~ 1060 or for communicatlon with I~O module bus 19 1020. I
A n~twor~ controller 1080 provides an interface to a ¦ -21 st~nd~rd n~twork, such as the ~THERN~T network, by way of 22 network interf~ce 1082. Network controller 1080 i~ o coupled 23 to Jh~red ~mory controller 1050 which act~ as an interface both 2~ to local RAM 1060 ~nd I/O modula bus 1020. Thare is no I
r~uirement, however, for any ono sp~cific org~nization or 26~o~c~ ~truCturR of I/O module bus 1020.
~N, HENDERSON
.. ~ow, G~RRETr ' :j DUNNER , .s p~ ,. --.. ~., ,. ~.
.~GTOI-. D. C ~000--'., .~0~ 0 32 l -58-.~ ... . .,.. . . .. ... . ... ... .. .. - ... ... .. - .... ~. . .. .. .

~ 2 ~ 9 1 PCIM (power and coolin~ interface module) ~upport element 2 1030 i8 conn~cted to I/0 module bus 1020 and to sn ASCII3 interface 1032. PCIM support el~ment 1030 allow~ proce~sing 4 system 20 to monitor the s~atu~. of the power ystem (i.e., batteries, regulator~, etc.) and the cooling F.y~.tem (i.e., fan~) 6 to en~ure thair proper operation. Preferably, PCIN ~upport 7 element 1030 only receives messsges when there is some fault or 8 potential fault indication, such a~ an unaccept~bly low battery 9 voltage. It i8 al80 possible to u8e PCIM support element 1030 to monitor all the power and cooling subsystem~ periodically.
11 Alternatively PCIK support element 1030 may be connected 12- directly to firewall S 1000 and 1010.
13 Diagno~tics mlcroprocessor 1100 i8 also connected to the 14 I/O module bus 1020. In general, dlagnostic~ ~lcroprocessor 1100 is used to qath0r error chscking infor~ation fro~ trace 16 RAMS, ~uch as trace RA~ 1872, when fault~ are detected. That 17 data is gathered into trace buse~ 1095 and 1096, through fire- ¦
18 walls 1000 and 1010, re~pectively, through module bu~ 1020, and 19 into microproco~or 1100.
D. INTERPROC~SSO~ AND IN~ERMODU~E COMMUNICA$ION
21 1. Da~a Paths 22 Th~ elem~nts of co~putor sy~tem 10 do not by themselves 23 con~tit~te a fault tolerant sy~te~. Ther~ needs to be a com-2~ municatlon~ pathwny and protocol whlch allow~ communLcation 2S during nor~al oper~tion- and operation during fault detection 2~Wo~lC~ and correction. gey to such communicatlon i~ cros~-link pathway NNEC~N HENDERSON
~ ~ DUN~ER 25. Cros~-l$nk pathway 25 compri~e~ the par~llel link~, serial 17711 1~ ~TR~t7. N. W. I
~1~070t . D. C ~000-- , ~0~ 0 1 _59_ I

~ 2~2~9 1 link~, and clock signals already describ~d. These are shown in 2 Fig. 19. The parallol lLik include~ two identical 8et8 of data 3 and address lines, control lines, interrupt lines, coded error 4 line3, and ~ soft re~et request line. The dat~ and ~ddra~line~ and the control lines contain information to be e~changed 6 between the CPU modules, such as from the module LnterconnectR
7 130 and 132 (or 130' and 132') or from memory module 60 (60').
8 The interrupt linee preferably contain one line for each of 9 the interrupt levels available to I/O subsystQm (modules 100, 110, 120, 100~, 110~ and 120'). The~e lines are sh~red by -11 croo~-link~ 90, 95, 90' and 95'.
12 - ~he coded error linea preferably include codes for 13 synchronizing a console ~HAL~ reque~t for both zones, one for 14 synchronizing a CPU error for both zone~, one for indicat~ng the occurrence of a CPU/memory failuro to the other zone, one for ¦
16 synchronizlng DMA error for both zones, snd one for indicatLng 17 clock phaT3e error. The error llne~ from each zone 11 or 11~ are ¦
18 inputs to an OR gate, ~7uch a~ OR gate 1990 for zone 11 or OR
~ 19 gate 1990' for zone 11~. ~he output at each OR gat- provide~ an i 20 input to the cross-links of the other zone.
21 Th- fault tolerant processing sy~te~ 10 i~ designsd to 22 contlnuQ oper~tlng as a dual rail sy~te~ despite transient 23 f~ult~. T~e IJo 8ub8y8tom (module~ 100, 110, 120, 100', 110', 24 120') can nl80 e~porience transient error~ or f~ult~ and contl nue to operate. I~ the preferred embodiment, an error N2N6~NHENDER5oN detectQd by ~irewall comp~rison circuit 1840 will cause a F~RAT~OW. CARRETr 27~ DUNNTR 3YnChrOniZ~d error report to be made through pnthway 25 for CPU
0~0~ c 2000-,'~Z ~o~ 0 . ' j 29 3~` -60-.'. .
..

2~222~

1 directed operation8. H~rdware in CPU 30 and 30' will cau~e a 2 ~ynchronized soft reB~t through pathway 25 and will retry the 3 faulted operation. Por DMA directed operation~, tha same error 4 detection result~ in cynchronous interrrApts through pathway 25, and ~oftw~re in CPUs 40, 50, 40' and 50' will restart the DMA
6 operation.
7 Certain transient errors are not immediately recoverable to 8 allow continued operation in a full-duplex, synchronized 9 fashion. For example, a control error in me~ory module 60 can result in unknown data in memory module 60. In this situation, 11 the CPU~ and memory elements can no longer function reli~Ably a~
12 part of ~ f il safe ~y~te~ ~o they ~re removed. Memory array 60 13 must then undergo a memory re~ync before the CPUB and memory 14 element~ can re~oin the syste~. The CPU/ma~ory fault code of tha coded error line~ in pathw~y 25 indlc~tes to CPU 30' that 16 the CPU~ and memory ele~ent~ of CPU 30 have been faulted.
17 The control lines, which repre~ent a combination of cycle 18 type, error type, and ready conditions, provids the handshaking 19 betwoen CPU module~ (30 and 30~) and the I/0 module~. Cycle typa, a~ e~pla~ned above, defines the type of bus operation 21 being perfor~edAs CPU I/0 re~d, DMA transfer, D~A setup, or 22 int-rrupt vector reque~t. Error typo define~ either a firewall 23 ml~cnmpar~ or a CRC 0rror. "Roady~ me~Ages are sent between 24 the CPU and I/0 module~ to indicate the completion of reque~ted operation~.
2 7~AW .0 ~ ~ I C C ~

~N, HENDERSON
~ ow, C~RRETr 29~ DUNNER
3~ ,~ .-,-..-.,, ~.
,"0-0~ .0 c ~Ooo-Jl~ot)~ o . .

~ 2~222~9 1 ~he serial cross-link include~ two sQts of two lines to 2 provide a serial data transfer for a status read, loopback, and 3 data transfer.4 The clock 3ign~1s exchanged sre the phase locked clock signals CLRC H and CLRC' H (delayed).
6 Figs. 20A-D sho~ block dlagr~ms of the element~ of CPU
7 modules 30 and 30' and I/O module~ 100 and 100' through which 8 data passes during the different operation~. Each of those 9 elements has each been described previously.
Fig. 20A ~how~ the data pathw~y~ for a typical CPU I/O read 11 operation of data from an I/O module 100, such a~ a CPU I/O
12- register read operation of regi~ter data from shaved memory 13 controller 1050 (1050'). Such an operation will bo referred to 14 as a read of locnl data, to distingui~h it from a DMA read of data from local memory 1060, which u~ually contains d~ta from an 16 internal device controller. Tho looal data are pre~umed to be 17 stosed in locnl RAN 1060 (1060') for transfer through ~hared 18 memory controller 1050 (1050'). For one path, tha data pass 19 through firewall 1000, module inter~onnect 130, to cross-link 90. As seen in Flg. 12, cross-link 90 delays the data from 21 fire~all 1000 to memory controller 70 ao that the data to cross-22 link 90' may be pre~ented to memory controller 70 at the same 23 tim~ th- dat~ are pre-ented to memory controller 70, thus 24 allowing proce~ing systems 20 and 20' to remain synchronized. I
The data then proceed out of msmory controllers 70 and 70~ into 26~^wo~c-- ¦ CPUs 40 and 40' by way of internal busses 46 and 46'.
~N. HENDERSON
FX BOW. GARRETT . . .
2 ~ DU~;NER
2 ~ -.,.W. :. :
090C~oOOO- !
32 -62- ~ ~

~ ', " .
:' ' -;:, .

~ 20222~9 1 A similAr path i8 taken for reading data into CPU~ 50 and 50'. Data from the 8hared memory controller 1050 proceeds 3 through firewall 1010 and ~nto cross-link 95. At that time, the 4 data are routed both to cross-link 95' and through a delay unit inside cross-link 95.
6 CPU I/O read operation~ may also be performed for data 7 received from the I/O devices of proce~ing ~y~tem 20~ via a shared memory controller 1050' and local ~AM in I/O device 100~.
Although I/O modules 100, 110, and 120 are similar and cor-respond to I/0 module~ 100', 110', ~nd 120', reapectively, the 11 corresponding I/O modules are not in lock~tep ~ynchronization.
12 Us$ng memory controller 1050' and local RAM 1060' for CPU I~0 13 re~d, the data would first go to cro~s-link~ 90' and 95'. The 14 remaining dat~ p~th iR equivalent to the path from memory con-troller 1050. The data tr~vel from the cross-l$nks 90' and 95' 16 up through memory controllers 70' and 75' and finally to CPUs 17 40' ~nd 50', re~pectively. Simultaneou~ly, the data travel 18 across to cro~s-link~ 90 ~nd 95, re~pectively, and then, without 1~ pa~sing through ~A del~y element, the data continue up to CPUs 40 and 50, respectively. !
21 Fig. 20B ~how- a CPU I/O writo operation of loc~l data.
22 Such loc~l d~ta ar- tran~ferred from the CPUs 40, 50, 40' and 23 50' to ~n I/O module, ~uch a~ I/O module 100. An example of ¦~
~1 24 such an op~r~tion i~ a write to a register in ~hared memory con- I - -i 25 troller~ 1050. The d~t~ tr~n~ferred by CPU 40 proceed along the ~ 2~WOr~lC" same p~th but in a direction oppo~ite to that of the d~ta during NNECAN HENDERSON ..
, i~ DuNNER the CPU I/O re~d. Specifically, 8uch d~ta pa~s through bu~ 46, ,". ~ ..,.~... ,.. ~.
0~. O. c. ~000- . .
c0,,,.,.. 0 .-~ 30 -63-., ~ I, ,, .
:.:
, ~ .

2~ .259 1 memory controller 70, various latches (to permit 2 synchronization), firewall 1000, and memory controller 1050.
3 Data from CPU 50' al80 follow the path of the CPU I/0 reads in a 4 rever~e direction. Specifically, such data pass through bus 56', memory controller 75', cross-link 95', cross-link 95, and 6 into firewall 1010. As indicated above; firewall~i 1000 and 1010 7 check the data during I/O write operations to check for errors 8 prior to storage.
9 When writes are performed to an I/0 module in the other zone, a similar operation i~ performed. However, the data from 11 CPU~ 50 and 40' are used in~tead of CPU~ 50~ And 40. ~
12 The data from CPUs 50 and 40' are transmitted through sym-13 metrical path~ to shared memory controllor 1050'. The data from 14 CPU~ 50 and 40' are compared by firewall- 1000' and 1010'. The reason different CPU pair~ are used to service I/O write data is i16 to allow checking of all data path~ during normal use in a full 117 duplex system. Interrail checks for each zone were previously 18 performed at memory controller- 70, 75, 70' and 75'.
19 Fig. 20C ~ihow~ the datA paths for DMA read op ations. The ~20 data from me~ory arr~y 600 pa~ simultaneou~ly into memory i21 controller~ 70 and 75 and then to cro~-link~ 90 and 95. Cross-¦22 link 90 delay~ the data transmitted to firffwall 1000 80 that the ~23 d~ta from cro~is-links 90 and 95 reach firewall~ 1000 and 1010 at j24 ~ubstantially the same time.
Similar to tho CPU I/O write operation, there are four 26~Wo~lcs~ copies of data of di~ta to the various cros~-link~. At the !NNEC~N. HENDER50N -:
'~ DUNNER firewall, only two copie~ are received. A different pair of r ~ tr, ~. w.
ol~.o.C ~000-Z~ o . - .

-~, .

~ 5~ 9 1 data are u~ed when pQrforming reada to zone 11. The data path~
2 for tha DMA write operation are shown in Fig. 20D and are 3 ~imilar to tho~e for a CPU I/0 rsad. Specifically, data from 4 shared m~mory controller 1050' proceed through firewall 1000~, cross-link 90' (with a delay), memory controller 70~, and into 6 memory array 600'. Simultaneously, the data pass through 7 firewall 1010', cros~-link 95' (with a delay), and memory 8 controller 75', at which time $t i~ compar~d with the data from 9 memory controller 70' during an interrail error check. A3 with the CPU IJO read, the data in a DMA write operation may 11 alternatively be brought up through ~hared memory controller 12- 1050 in an equ~valent operation.
13 The data out of cross-link 90' al~o pa~ through cro~s-link 14 90 and memory controller 70 ~nd into memory array 600. The data from cros~-link 95~ pa~s through cro~s-link 95 and memory con-16 troller 75, at which time they are compared with the data from 17 memory controller 70' during a ~imultaneous interrail check.
18 The data path for a memory r~ynchronization (resync) 19 operation i8 shown in ~lg. 20~. In thi~ operation the contentq of both memory array~ 60 and 60' must be ~et equal to each 21 other. In memory re~ync, data from memory array 600~ pass 22 th o~gh memory controllers 70' and 75' undar DMA control, then 23 through cro~-llnk~ 90~ and 95', re~pectively. The data then 24 enters cross-links 90 and 95 and memory controllers 70 and 75, respectively, before being stored in mamory array 600.
2~L~w or~lce~
~ ~,EC~N. HENDERSON
;~.~aow. C~RRETr Y li DuNNER
3~ T~T, ~ w.

~I ItCTO/t. D. C 000--1~0~ 3 ---0 ..
33 -65- ~

.

~2~2i259 ~ ~
1 2. Reset8 2 The preceding di~cussions of system 10 have made reference 3 to many different needs for re~ets. In certain inotance~ not discussed, resets are usad for standard functions, such as when power L~ initially applied to sy~tem 10. Nost systems have a 6 single reset which alway~ sets the processor back to ~ome pre-7 determined or initi~l ~tate, and thus disrupts the proces~ors~
8 instruction flow. Unlike mo~t other systems, however, resets in 9 ~yst~m 10 do not affect the flow of instruction execution by CPUs 40, 40', 50 and 50' unless absolutely necessary. In i 11 addition, resets in system 10 affect only those portlons that i~ 12 need to be re~et to re~tore normal operation.
13 Another a~pect of the re~ets in system 10 i9 thoir 14 containment. One of the prime considerations in a fault tolerant system is that no function should be allowed to stop 16 the ~y~tem from operating should that function fail. For this 17 reason, no single re-et in system 10 controls elements of both 18 zone~ 11 and 11' without direct cooperation between zones 11 and 19 11~. Thu-, in full duple~ mode of operation, all re~ets in zone 11 will be independent of reset~ in zone 11'. When ~ystem 10 i3 i: . .
21 in m~-ter/~lave mode, however, the slave zone uses the re~et~ of 22 tho r--ter zono. In ~ddition, no reset in system 10 affec~ the 23 cont-~t- of meoory chip~. Thu~ neither cache memory 42 and 52, 24 scratch pad me~ory 45 and 55 nor memory module 60 lose any data 26 due to a re~et.
2~wo,~,ct.
I ~N. HENDER50N
l g~llO~lV. C~RRETr Y~ DUNNER
J ~ w ,Ø.. ~. c . .000.
~0...~ .0 i ~a2~5~
1 Ther~ are preferably three classQr of resQts in system 10;
2 ~clock res~t,~ h~rd resQt,~ and "soft reset.~ A clock reset 3 realigns all the clock phase yenerators in a zone. A clock 4 reset in zone 11 will also initialize CPUs 40 and 50 and m.~mory module 60. A clock reset doe~ not affect the module 6 interconnacts 130 ~nd 132 except to re~llgn the clock ph~F3e 7 generators on tho~e module~. Even when sy~te~ 10 Ls in master/
8 slave mode, a clock reset in the slave zone will not diqturb 9 data tr~nsfer~ from the ma ter zone to the ~lave zone module interconnect. A clock reset in zone 11~, how~7ver, will 11 initialize the corresponding element~ in zone 11'.
12 In general, a hard re~7et returns all state device~ and 13 registers to some predetermined or initial ~3tate. A soft re~et 14 only returns state engines and temporary 3torage r2gisterR totheir predetermined or initial state. The atate engine in a 16 module iR the circuitry that defines the state of that module.
17 Regi3ters containing error information and configur~tion data18 will not bQ affected by a soft reset. Additionally, system 109 will ~electively apply both h~rd re~et~ and soft re~etn at the ame ti~e to reset only those elementa that need to be 21 reinitialized in order to continue proce~sing.
22 The hard re~et~ clear ~y~tem 10 and, a~ in conventional 23 systems, return sy~tem 10 to a known configur~ion. Hard reqet~3 24 are used after power is applied, when zones are to be synchro-nized, or to initialize or diQable an I/0 module. In sy~tem 10 26~wo~c~ there are preferably four hard resats: ~powQr up reset,~ CPU
I:~;NEC~N. HE~DERSON
~ ~ Du~NER hard re~et," ~module reset,~' and ~device re~et.~ Xard resets ~"~ c~
.0OO.
L ~0,,,~3...0 -67_ I ~2~2~

1 can be further bxoken down into local and system hard resetQ. A
2 local hard reset only aff eCt8 logic that responds when tha CPU
3 i~ in the slave mode. A sy~tem hard resQt is limited to the 4 logic thst is connected to cross-link cables 2S and module interconnects 130 and 132.
6 The power up reaet i8 u~ed to initialize zone~ 11 and 11~
7 immediately after power i8 ~upplied. The power up reset forces 8 an automatic reset to all parts of the zone. A power up re~et 9 is never connected between the zone~ of system 11 because each zone has its own power supply and will thus e~perience different 11 length ~power-on~ events. The power up reset Ls ~mplemented by 12- applying ~11 hard resets and ~ clock re~et to zone 11 or 11'.
13 The CPU hard reset i~ u8ed for dlagno~tic purpo~e~ in order 14 to raturn a CPU module to a known ~tate. The CPU hard ras~t clear~ all information in the CPUa, me~ory controllar~, a~d 16 memory module status registers in the ~ffected zone. Although 17 the cache memorie~ and ~emory module~ are dis~bled, the contents 18 of the ~cratch pad RAN8 45 and 55 and of the memory module 60 19 are not ch~nged. In additlon, unlike th~ power up r~eset, the CPU hard reset doe- not modify the zone identification of the 21 cro~-link~ nor the clock ma~terahlp. The CPU hard reset is the 22 sum of all loc~l hard resets thht c~n be applied to a CPU module 23 and a clock r~ot.
24 ThQ module hard reset i8 used to sQt the I/0 modules to a known tate, such as during bootstrapplng, and is also used to 26~orr~c~ remove a faulting I/O module from the ~ystem. The I/0 module ~ ~GW. HENDER50N
~ ~;, ,~I~OW, G~RRETr .: ~ DUNNER
, ", 0~04. O. C ~000-~0~ 0 31 ~ -68-, ., .

~2`2~59 1 hard reset clears everything on the I/O module, leaves the 2 firewalls in a diagnostic mode, and disables the drivers.3 A device reset L3 used to re3st I/O device9 connected to 4 the IJO modules. The reRets are device dependent and are provided by the I/O module to which the device is connect~d.
6 ~ The other class of reset~ is soft resets. As explained 7 above, soft resets clear the ~tate ~ngines and temporary 8 regi~ters in sy~item 10 but they do not change configuration 9 information, such as the mode bits in the cros~-links. Inaddition, soft reset3 also clear the error handling mechanisms 11 in the module~, but they do not change error rogi3ter~ such ~8 12- syste~ error register 898 and system fault address register 865.
13 Soft resets are targeted 80 that only the neces~ary14 portion~ of the ~ystem are re~et. For ex~mple, if moduleinterconnect 130 needs to be reset, CPU 40 is not reset nor are 16 the deviceis connected to I/O module 100.
17 There ~re three unique ai~pects of soft resets. One is that 18 each zone i~ responsible for generating lts own re~et. Faulty 19 error or reset logic in one zone i~ thus prevented from causing reset~ in the non-faulted zone.
21 The second a~pect is that the soft reset does not disrupt 22 the ~e~uence of instruction execution. CPU~ 40, 40', 50, SO~
23 are re~et on a combined clock and hard reset only. Addition~lly 24 memory controller~ 70, 75, 70~ and 75' hAve those state engine~ I
and regi~ters necessary to ~ervice CPU instruct~ons attached to 2~Wo~7lc~ hard reset. Thus the soft reset is transparent to software iNNEGW. HENDERSO~J .
F~ ac~ C~RRETr i 27~ D~;NNER ! ax-cution.
77~ T~-e~ I
~~. c. ooo-~-0-- - 9~ 0 , 1 I ~ ~

' ' ~2~g 1 The third aspect is that the range of a soft reset, that i~
2 the n~mber of elements in ~y~tam 10 that is affectsd by a ~oft3 reset, i8 dependent upon the modQ of system 10 and the original 4 reset request. In full duplex mode, the soft re~et request originating in CPU module 30 will i~,~ue a soft re~et to all 6 elamQnts of CPU module 30 a~ well as all firewalls 1000 and 1010 7 attached to module interconnect 130 and 132. Thus all modules 8 serviced by module interconnect 130 and 132 will hav~ their 9 state engines and temporary registers reset. This will clear the ~ystem pipeline of any problem caused by a transient error.
11 Since system 10 is in duple~ mode, zone 11~ will be doing ~
12 everything that zone 11 i8. Thu- CPU module 30~ will, at the 13 same time as CPU module 30, is~ue a soft reset request. Ihe 14 soft reset in zone ll' will have the same effect as the soft reset in zone 11.
16 When ~ystem 10 is in a master/slave mode, however, with CPU
17 module 30' in the slave mode, a soft reset request originating18 in CPU module 30 will, a~ e~pected, is~ue a soft re~et to all 19 elements of CPU module 30 a~ woll a- all firewalls 1000 and 1010 attached to module interconnectJ 130 and 132. Additionally, the 21 soft ro~ot r~que-t will be forwardod to CPU module 30~ via 22 cro~--llnk~ 90 and 90', cro~s-link c~bles 25, and cross-links 23 90' and g5'. Parts of module interconnect~ 130' and 132' will 24 receive th~ soft reset. In thi3 same configuration, a Yoft res~t request originating from CPU module 30~ w~ll only re3et ~wo~c~ memory controllars 70' and 75~ and portions of cross-links 90~ j eC~N. HE~DERSON
F.~R,~80W C;ARRETr E
~7a DUNNER and 95'.
,", ~ .. ,.. ,.. w.
-O~.O c ooo-~ uO~ O
29 i 1 _70-, .

~ 9 1 Soft resets include "CPU soft re8ets~ and " ~yst2m soft 2 re~ets.- A CPU soft reset i8 a soft reset that affects the 3 ~tate engines on the CPU module thst originated the request. A
system ~oft reset i8 a ~oft rQset over the module interconnect and those element~ dlrectly attached to it. A CPU module csn 6 always request a CPU soft reset. A system soft reset can only 7 be reque~ted if the cross-lLnk of the requesting CPU i3 in 8 duplex mods, master/slave mode, or off mode. A cross-link in the slave mode will take a system soft reset from the other zone and generate a system soft reset to its own module 11 interconnects.
12 CPU soft resets clear the CPU pipeline following an error 13 condition. The CPU pipeline include~ memory interconnects 80 14 and 82, latches (not shown) in memory controllers 70 and 75, DM~
engine 800, and cro~s-links 90 and 95. The CPU soft reset can 16 also occur following a DMA or I/O time-out. A DNA or I/O time-17 out occurs when the I/O device does not respond within a ~8 specified time period to a DMA or an I/O request.
19 Fig. 21 show~ the reset line~ from the CPU modules 30 and 30' to the I/O module~ 100, 110, 100', and 110' and to the 21 memory module~ 60 and 60'. ~he CPU module 30 receives a DC 0 22 signal indicating when the power ~upply ha~ settled. It i3 thiY,:
23 signal which initializes the power-up reset. CPU module 30 24 receive~ a similar signal from its power supply.
One system hard reset line is sent to each I/O module, and i26~wO~c~ one system ~oft reset is sent to every three I/O modules. The 1~ ;~N, HENDER50N
U~ v, CARRETr DUNNER
I It ~ t--. N. W. ..
O~ON. O. C ~OOO--0 .:

.

-` 2D22259 1 reason that ~ingle hard re~et i8 needed for each module i3 be-2 cau~e the sy~tem hard reset line are used to remove individual I/O modules from eystem 10 The limitation of three I/O modules 4 for each ~y~tem soft reset is mer~ly a loading considerAtion In addition one clock re~et line i8 ~ent for every I/O module 6 and memory module The reason for using a single line per 7 module i8 to control the skew by controlling the load ¦ -8 Fig 22 shows the elsments of CPU module 30 which relate to 9 reset~ CPUs 40 and 50 contain clock generator~ 2210 and 2211, re~pectively Nemory controllers 70 and 75 contaLn clock gen-11 erators 2220 and 2221, renpectively, ~nd cros~-links 90 and 95 12 - contaln clock generators 2260 and 2261, re~pectively The clock 13 generators divide down the sy~tem clock signals for use by the 14 individual modules Memory controller 70 contain~ re~et control circuitry 2230 16 and a soft reset request register 2235 Memory controller 75 17 contains reset control circuitry 2231 and a ~oft re~et reque~t 18 regi~ter 2236 19 Cro~-link 90 cont~in~ both a local re-et gene~ator 2240 1 -and a syste~ re~et generator 2250 Cro~-link 95 contains a ¦ -~
¦ 21 local re~et generator 2241 and a sy~tem reset generator 2251 1 22 Tho ~local~ portion of a cro~s-link i~ that portion of the 23 cross-link wh$ch remalns with the CPU module when that cross- j24 link is in th~ slave mode and therefore include~ the serial regi~ter~ and ~o~a of the parallel regi~ters Tho n sy3tem~
3 25~wo lc....... portion of ~ cros--link is that portion of the cros~-link that ; ;~CAN, HENDERSON : ~
;3 ~ 30W, C~RRETr ~, ~ ~ DUNNER
T~CT 1~ W l OTO~ O C 1000- ~
~ 332 1l _72-, .. . . .

210222~9 1 is needed for acceB8 to module interconnects 130 and 132 (or 2 130~ and 132~) and cross-link cables 25.3 The local re~et generators 2240 and 2241 generate reset~
4 for CPU module 30 by sending hard and ~ioft reset signals to the local reset control circuits 2245 and 2246 of cross-links 90 and 6 95, respectively, and to the reset control circuits 2230 and 7 2231 of memory controller 70 and 75, respectively. Local cross-8 link re~et control circuits 2245 and 2246 respond to the ~oft 9 re~et signals by resetting theLr state engines, the latches storing dat~ to be transferred, and their error regi~ters.
11 Tho~e cir~uits respond to the hard reset signal~ by taking.the 12- samQ action~ as are taken for the soft re~et~, and by also 13 re3etting the error registers and the configuration register~.
14 Rese~ control circuits 2230 and 2231 re~pond to hard and soft reset signals in a similar manner.
16 In addition, the local reset generator 2240 send~ clock 17 resQt signals to the I/O modules 100, 110 and 120 via module 18 interconnect~ 130 and 132. The I/O module~ 100, 110, and 120 19 use the clock reset ~ignals to reset their clocks in the manner described below. Soft reset request registers 2235 and 2236 21 send ~oft r~que~t 8ignal~ to local r0~et generator~ 2240 and 22 2241, re-pecti~ely. .
23 sy~tQm reset generator~ 2250 and 2251 of cross-links 90 and ~
24 95, respectively, ~end system hard re~ot signal~ ~nd ~yst.~m soft I
reset ~ignals to I/O module~ 100, 110, and 120 via module 26~Or~lC~, interconnects 130 and 132, respectively. I/O modules 100, 110, INNECAN. HENDER50N
~DUN~ER and 120 respond to the 30ft reset ~ignal~ by re~etting all ~; ,". " ",~",, ,.. w.
~NOTOI~. a. c ~ooo-,j ~ ~o~ o 1 _73_ .
., , ' .

~-22~
1 registers that are dependent on CPU data or commands ~hose 2 modules respond to the hard reset signals by resetting the same3 register ~8 soft re~ets do, and by al80 resetting any configura-4 tion registers In addition, the system reset generAtors 2250 and 2251 also 6 ~send the ~y~tem soft and syst~m hard reset signals to the system 7 reset control circuit 2255 and 2256 of each cross-link System 3 reset control circuit 2255 and 2256 re~pond to the sy~tem soft g reset signal~ and to the system hard reset signals in a manner similar to the re~pon~e of the local re--t control circuits to the local soft and local hard re~et ~ignals 12- MamoA~y controllers 70 and 75 cause cro~s-llnk~ 90 and 95,13 respectively, to generate the floft re-ot~ when CPU~ 40 and 50,14 respectively, write the appropriate code- into soft rsset reqAest registers 2235 and 2236, re~pectively ~oft reset 16 request registers 2235 and 2236 send ~oft reset request signals17 to local reset generators 2240 and 2241, respectively The 18 code~A error signal is sent from memory controller 70 to local 19 reset generator~ 2240 and 2241 Sy~t~m oft re~-t~ are sent between zone- along the ~Ame 21 data path~ data and control ~ignal~ are sent Thus, the same 22 phllo-ophy of equalizing delayr is used for re~ets a~ for data 23 and Addr---es, and re--t- reach all of th element~ in both 24 zone~ at approxim~tely the same time Hard re~ets are generated by CPUs 40 and 50 writing the ap-26Awo~c~ propriate codo into the local hard re~et regi~ter~ 2243 or by INNECAN~ HENDER50N
F~A~OW GARREJr th~ reque8t for a pow~r up reset caused Aby the DC 0~ signal w. ~ ., TO~. D C. ~000-- ¦
.' ~ ~0~ 0 I . ',,.

_ ~1222~ 1 1 Synchronization circuit 2270 in cross-link 90 include~
2 approprinte del~y elements to ensur~ that the DC OR 9ign~1 goes 3 to all of the local and reset generator~3 2240, 2250, 2241 and 4 2251 at the same time.
S In fact, ~ynchronization of resets is very important in 6 sy~tam 10. That is why the rQset signA 18 originate in th~
7 cross-links. In that way, the resets can be sent to arrive at 8 different modules and elements in the modules appro~im~tely 9 synchronously.
With the understanding of the 3tructure in Fig~. 21 snd 22, 11 the e~ecution of the different hard reset~ can be better 12 understood. The power up re~3et generate~ both a ~y~tem hard 13 reset, a loral hard reset and a clock reset. Generally, cross-14 link8 90, 95, 90 ' ~nd 95' are initially in both the cro~s-link off ~nd re~ync off mode~, ~nd with both zone~ asserting clock 16 master~h~p.
17 The CPU/MEN fault re~et is automatically activated whenever 18 memory controller~ 70, 75, 70~ and 75~ detsct ~ CPU/MEM fault.
19 The coded error logic is oent from error logic 2237 and 2238 to both cro~-links 90 and 95. The CP~ module which generated the ¦ 21 fault ia then removed from system 10 by setting its cross-link 22 to the slave state and by satting the cro~s-link in the other 1 23 CPU modul~ to t~e master state. Tho non-~aulting CPU module 24 will not experience a re~et, however. Instead, it will be 26 noti~ied of the fault in the other module through a code in a 27.~w or~le~J
CAN. HENDERSON
~ ~BOw. ~RRETr ~ ~ DUNNER ¦
3~0~ 1 0~.0 c.~000- j .,~.3...0 ~

l ,:' ~22~
1 serial cross-link error register (not shown). The CPU/MEM fault I
2 reset consi~t~ of a clock reset to tha zone with the failing CPU ¦
3 module and a local soft resat to that module.
4 A re~ync reset i8 essentially a system soft re~et with a S local hard re~et and a clock reset. The re~ync reset is used to 6 bring two zon0s into lockstQp synchronization. If, after a 7 period in which zones 11 and 11' were not synchronized, the 8 contQnts of the memory modules 60 and 60', including the ~tored 9 st~tes of the CPU registers, are ~et equal to each other, the re~ync reset is u~.~d to bring tha zones into a compatible 11 configur~tion 80 they can restart in a duple~ ~ode.
12- The rasync reset i~ e~sentially a CPU hard reset and ~
13 clock reset. The re~ync reset is activated by software writing 14 the re~ync reset addres~ into one of the parallel cros~-link regi~ters. At that time, one zone should be in the cross-link 16 master/resync ma~ter mode and the other in the cxoss-link 31ave~ l -17 resync slave mode. A simultaneou~ reset will then be performed 18 on both the zonec which, among other thinqs, will set all four 19 cros~-link~ into the duple~ mode. Since the re~ync reset is not a system ~oft re~et, the I~0 modules do not receive reset.
21 5h~ pref~rred embodiment of sy~tem 10 ~180 en~ures that 22 clock r~--t ~lgnal- do not re-et conforming clocks, only non-23 confor~ing clock~. The rea~on for thiR i8 th~t whenever a clock 2~ i~ re~et, it alter~ the timing of the clock~ which in turn 26 affect~ the operation of the module~ with such clock~. If the 27~wo~lc~
N. HENDERSON
~I~OW. C~RRETr ~ 6 DUNNER
30'~ W I
, ~"Ø0.. O. c. .000~ 1 ~o~ o i 32 r 20222~9 ~
1 module was performing correctly and it8 clock was in the proper 2 phase, then altering its operation would be both unnece~sary and 3 wssteful.
4 Fig. 23 shows a preferred embodiment of circuitry which S will en~ure that only nonconforming clocks are reset. The 6 circuitry ~hown in Fig. 23 preferably re~Ldes in the clock 7 generator0 2210, 2211, 2220, 2221, 2260, and 2261 of the 8 corresponding module~ shown in Fig. 22.
9 In the preferred embodLment, the different clock generators 2210, 2211, 2220, 2221, 2260, and 2261 include a rising edge 11 detector 2300 and a phace generator 2310. The rising edgo 12 detector 2300 receivec the clock reset signals from the cross-13 links 90 and 95 and generate~ a pulse of known duration 14 concurrent with the risinq edge of the clock re~et signal. That pul~e i8 in an input to the pha~e generator 2310 as are the 16 internal clock signals for the particular module. The internal 17 clock signals for that module are clock signals which are de-18 rived from the 3y~tem clock ~ignal~ that have been distributed 19 from oscillator system~ 200 and 200'.
Pha~e generator 2310 is preferably a divide-down circuit 21 which form~ different phases for the clock signals. Other I -22 de~ignJ for pha~e generator 2310, such a~ recirculating shift 23 regi~ter~, can al-o be u~ed. I
24 Preferably, the rising edge pulse from rising edge detector ¦23G0 causes phase generator 2310 to output a preselected phase. I .
26~WO~IC~ Thus, for example, if phase generator 2310 were a divlde-down !NNECAN, HENDEASON :. :
~A90W. CARRETT circuit with several stages, the clock reset ri~ing edge pulse 17~ YTI C~T, li. W, I
~",0.0,..0 ~ .000. 1 ~0.. ~.. 0 1 ~:, ' 30 -77- `- ~
,', ~, , - ~ 20222!~9 1 could be a ~et input to the stage which generates the 2 presele,cted phase and a reset input to all othe,r ~tages If 3 phase gener~tor 2310 were already ganerating that phase, then 4 the presence of the synchronizQd clock reset signsl would be es~entially transparent 6 The reset~ thus organized are designed to provide the 7 minimal disruption to the normal execution of ~y~tem 10, and 8 only cau~e the dr~stic a~tion of interruptlng the no-mal 9 sequences of instruction execution when ~uch drastlc action i3 required ~his is particularly important in a du~l or multiple 11 zone e,nvironment because of the problem~ of resynchronizatlon 12- which conventional resets cause Thus, it i8 preferable to 13 minimize the number of hard re-et-, a~ is done in syote~ 10 14 E BOOTSTRAPPING TO INSURE A SINGLE OPERATING SYST~M
AB indicated above, proce~sing systems 20 and 20' generally I
3 16 operate in lockstep synchronism There are ~ituAtiOns in which 17 one or both processing sy~tem~ may cease operating For il 18 example, one proces~ing system may be intentionally di~abled by 19 a ~ervice technician Another example i~ a predetermined one of the proce-~ing ~y t-mJ m~y di~able itself upon the di~connection 21 of th cro---llnk pathway When the processlng ~ystem that 22 ce~ o~eratlon 1- to be returned to service, it i8 e88enti~1 1 ~ 23 th t lt not boot-trap load it~ own copy of the o~eratlng system 3 24 if the other proce~-$ng system is already operating Thi~ would , -f 2S re-ult in the twc, proce-sing systems re~adLng and writing ~ 2C~wo"~c~. dlfferent data AJ a reJult, thQ two systems would be unable to ~CAN. HENDER50N
I~IOW C~RRETr A 2~.T~T, ~ ~. I
J~TO~ . . C Z
,1 U o ~ ------o I 32 -78- ;

.

~2259 . ~
1 oper~te in lockstep 8ynchronism~ ~owever, it is noted thAt Lf 2 neither operating system is operating, it i8 de~irable that one of the t7dO zones be able to bootstrap load.
4 With respect to the servicing of the processing sy~t~m of one zone, a proce~sing system would preferably be taken out of 6 ~service by the technician inputting an appropriate instruction 7 via the sy~t~m console to remove the processing system of one i 8 zone. When the service instruction is inputted to remove the processing ~ystem of one zone, for example system 20', the other processing system, sy~tem 20 in the e~ample, 7~rites a fail stop 11 bit in two æEpRoM~ 44 and 54 (Fig. 3) re-pectl~ely ~ociated 12 with corresponding CPUs 40 and 50 in CPU module 30. Thu~, the 13 fail stop bit i~ written in the proce~ing ~y~tem that i~ not 14 being taken down for ~ervice.
~he purpose of the fail s~op bit i8 to allow the remaining 16 processing system to determine immed$ately that it can reboot if 17 nece~sary while the other proces~ing system 20' is disabled.
18 The fail ~top bit thereby enhanco- the overall availability of 19 system 10. The fail stop bit is also u~ed in a pre-bootstrap loading algorithm, de-cribed below, for a~uring that the --21 re~p ctlve procs~ing ~y-tem~ of the two operating zone~ do not 22 soparat-ly boot~trap load copie- of the operating sy-tem.
23 The fail stop bit i~ cleared during the memory 24 resyn~hronizing process when the previou~ly di~abled processing ~ystem i~ returned to service.
26~orr~C~ OSR bit 913 in serial control and ~tatu~ regi~ter 912INNEC~N. HENDEl~5oN
F2~0W ~RRETr (Fig. 10) i8 also used in the pre-boot~trap lo~ding algorithm.
~.~ t.7~
A~1~070~0c,OOO.
.,.. 0 ~, .

~-~ 2~2~25g 1 Since the bit i8 stored in thi~ register, its status in each 2 proces~ing ~ystQm can be read by both processing ~y8tem8.
3 OSR bit 913 L~ set a8 part of the bootstrap loading 4 ~ operation and remains ~et 80 long as the processing ~y8tem~
continues to successfully load and operate The OSR bit is also 6 set during resynchronizing of either proce~sing system and 7 remains set as long a~ the sy~t~m continue~ to successfully 8 resynchronize and operate Thus, during normal lock~tep 9 operation, the OSR bit will be set in both processing ~ystems 20 and 20 ll The OSR bit i~ re-et in a procec-ing system a~ a re~ult of 12- proces~ing sy~tem 20 or 20 cea~ing to operate, either through 13 operator instruction or when a procea~ing sy~tem is disabled due 14 to certain faults The pre-bootstrap algorithm is set forth in a flow chart 2400 illustrated in Fig 24 The program~ for 16 carrying out the algorithm contained in flow chart 2400 are 17 preferably stored in ROMa 43, 53, 43 and 53 (Fig 3) in proce6sing systQma 20 and 20 Thu~, the algorithm is stored l9 for separate execution by each proce~alng ~y~tem doth procoaaora in eAch zone are preferably exocuting th- algorithm 21 indep ndently of the proces~ors in the other zone The 22 algorlthm en-ure- that only one zone will boot, however For 23 rea~on- of ~implicity, it will be aaaumed in the following 24 diRcus~ion that proce-~ing ~y~tffm 20 i8 executing the algorithm E~ecution of the algorithm begin~ with receipt of a command 26~wore to initiate bootstrap loading the operating ~y-tem for '~;NEG'~N, HENDER50N
~ 30w ~RRE7r procecsing sy~tQm 20 (8tep 2402) Ne~t, proce8-ing system ~O
1?7~ 11 llrl-ttT. ~ W
o~o~. O. c. ~000.
~0.,.-~ 0 ' 2g 1 , -80- - -'. ' '''': "
.~ . .

-` ~ 20222~9 1 determines whether cro88-link psthw5y 25, i.e., the cro~a-link 2 c~ble, is connected between zone~ nd 11~. In the preferred 3 Lmplementation of this ~y8tem, each processing sy~tem performs a 4 ~tatus check to determine whether cross-link cable 25 i8 connected and whether the proce~sing syst~m in the other zone 6 has power.
7 Fig. 25 illustrate~ a preferred circuit arrangement by 8 which each proces~ing sy~tsm 20 and 20' can m~e a status 9 determination as to whether the cross-link cabls is present and if the other zone ha~ power. Two conductors 2502 and 2504 of 11 the cro~-link cable are allocated for perform1ng this tat~us 12- determin~tion. Both conductor~ are term$nated, at opposite 13 end~, at cros~-lInk~ 90 and 90' of procesuing systems 20 and 14 20~, r~spectively.
Within crose-link 90, tri-state line driver 2506 and 16 receiver 2508 are connectad to the snd o~ conductor 2502 for 17 tran~mitting and receiving signals, respectively. The logic 18 state of conductor 2502 i~ read at a node A via receiver 2508.
The logic ~tate of conductor 2504 i~ read ~t a node B via a line receiver 2510. Conductor 2502 i8 further connec~ed, within 21 cros--link 90, to a voltage ~upply Vcc through a diode 2512 and 22 to a ground point 2514 through a resistor 2516. Also within , 23 cro~-link 90, conductor 2504 i8 connected to the voltage ~upply 24 Vcc through a re~istor 2518.
The other end of conductor 2502 i~ terminated in cross-link ' 26~or~1c~ 90~ in a manner identical to the termination of conductor 2504 1~1 C~l. HE?`IDERS~N
1~ A~O~ CARRETr Z 6 DU'`J~IER
~ 2~
0~0~. O. c ~u.O.,.~ .O
: 32 -81-..

:

2~222~9 1 in cross-link 90. Thus, conductor 2502 i8 termlnated in cros~-2 link 90~ at a node B' through a line receiver 2510'. Conductor 3 2502 is al80 connected to a voltage ~upply Vcc' through a 4 resi3tor 2518'. The logic state of conductor 2502 i~ read at node B~. Conductor 2504 is terminated in cross-link 90~ in a 6 manner identic41 to the termination of conductor 2502 in cross-7 link 90. Thus, conductor 2504 i~ tsrminated at driver 2506~ and 8 receiver 2508'. The logic state of conductor 2504 is read at 9 node A' connected to receiver 2508'. Also within cro~s-link 90', conductor 2504 i8 connected to voltage supply Vcc~ through 11 a diode 2512~ and to a ground point 2514~ through a resl~tor 12 2516'.
1 13 Drivers 2506 and 2506' are provided for re~pectlvely 14 driving conductors 2502 and 2504 at time~ other than when the 3tatus determination i~ being made. The~e drivers are therefore 16 in a high impedance statQ during the stAtu~ determination.
17 Either ~oltage ~upply Vcc or Vcc' is only present when its 18 corresponding zone ha~ power. When a zone does not have power, 19 each connection to the power supply act- as a connection to ground. For example, if zone 11' ha~ no power, voltage Vcc~ i~T
21 at th~ ~0'~' logic level, thereby connecting diode 2512' and 1 22 re-i-tor 2518' to ground.
23 The follow~ng truth table de~cribe- the interpretation of 24 the conductor 2502 and 2504 statu~,e~ a~ read at node,s A and B

27,~worrlc,, ~IILEG~`J, HE~JDeR5ol~l eow. G~RRE~r ~ ~ & DW?`IER .
30--rl ~TII~t. r~
? ~r~rrlr otor~. o. c. ~ooo~ , ~o~ o j 33 ~ -82-, . . .
. . .

, . . . . .

-~ 2~ 2~

1 within ~ro~s-link 90. The equivalent interpretations would be 2 made for node~ A~ and B' a~ read in cro~s-l~nk 90~.
3 CABL~ OTH~R ZONE

9 O 1 NO '~NgNO'~N
1 1 Y~S YES
:
12 If the cro~s-link cable i8 not pre~ent, then conductors 13 2502 and 2504 are not present. A~ a re~ult, d$ode 2512 i8 14 nonconducting and the connection to the ground point 2514 cau~e~
node A to read a logic n o~ voltage level. A loglc ~1~ voltage 16- level i8 read ~t nede ~ due to tho pull-up effect of voltage 17 supply Vcc through resistor 2518. A~ indicat~d in the truth 18 table, the A, B st~te of 0,1 corre~ponds to the ~b~ence of the 19 cros~-link cable 25 and, of cour~e, an unkn~wn condition with respect to whether power i~ present in the other zone. I
21 If the cross-link cable i~ present, then conductor~ 2502 22 and 2504 ~re pra~ent. ~he volt~ge level~ read at nodQs A and B
23 then depond on whether there ~8 power in zone 11~ and hence 2~ provision of voltag~ ~upply Vcc'- If there i8 no powar in zon~ 11', conductor 2S02 connects together diode 2512 and 26 re~l~tor~ 2516 and 2518'. With voltage Eupply Ycc present, 27 diodo 2512 i8 nonconducting. Voltage Vcc' i8 at the ll0" logic 28 level aince zone 11' has no power. AJ a r2sult, both re~i~tors 29 2516 ~nd 2518' are connscted to the l~0~ log~ level, i.e., to 30-^wo~c............ ~round. A ~0~ loglc level i~ therefore read at node A. The I~I~C~N~ HENDER50~
F~RA30W ~.RRETr . . : .
316 DUNNER presence of conductor 2504 will cause reJistors 2516~ and 2518 1~7~ 71~tT. ~ W.
OTO~.~t C.~OOO- I
S~o~ O I
~ 34 1 -83- ~

'' '' .' ~2~5~
1 and diode 2512' to be connected together. Since Vcc' i8 at the 2 l-0~ logic level, diode 2512' i connected to ground and i~ in a 3 conducting state. A~ a result, a '0' logic level is read at 4 node B.
If there is power in zone 11~, then voltage supply Vcc~ i8 6 applied to conductor 2502 through resistor 2518'. As a re~ult, 7 diode 2512 is not conducting. ResiRtors 2516 and 2518~ are 8 selected such that the voltage drop from Vcc' to ground produces 9 a ~1" logic level on conductor 2502. A~ a re~ult, a logic ~1"
voltage level is read at node A. Still with respect to the case 11 in which thare i8 power in zone ll', diodo 2512' is not 12 ~ conducting. Re-istors 2516' ~nd 2518 ~re selected such th~t the 13 voltage drop from Vcc to ground cause~ a ~1~ logic level on 14 conductor 2504. A~ a result, a logic ~1~ voltage level wlll be read at node B.
16 As the truth table indicates, when a logic ~ voltage 17 le~el is re~d at node A, cro~s-link cable 2S i8 present and the 18 other zone i~ powered on. This detenmination is made 19 irrespectlve of the logic level at nodo B. The node 8 logic level need only be referred to when thare is a logic ~'0~ level 21 at nod~ A.
22 R forring agaln to flowchart 2400 in Fig. 24, if cro~s-link 23 cablo 25 1B not connected, each of CPUs 40 and S0 of procassing 24 ~ystem 20 check theix fail stop bit in their respective E~PRO~ -~
44 or 54 (~tep 2406). Processing sy~tem 20 detQrmines that the 26-^wo"~ce~ fail stop bit is set when both CPUs 40 and S0 read thi~ ~tatui~.
'~C~N. HENDERSON
~ABOW G~RRETr DUNNER
2~ T~ W.
OTO~, O C ~- !
o~ --o 1 1 32 ~ -84- ~

.:

. - .
., 2~222~9 1 Upon determining that the fail eltop bit i8 ~et, proces~ing 2 sy~tem 20 sets OSR bit 913 in control and status register 912 3 (stQp 2408). Then proce~sing sy~tem 20 commtences boot~trap 4 loading of the operating system (step 2410). This i8 a8 . ~ . .
intended becau~e, when the fail ~top bit is ~et it means that 6 processing sy~tem 20' i8 not operating and therefore it ~hould 7 be permi~ible for processing system 20 to proceed with a 8 bootstrap load.
9 If proces~ing sy3tem 20 determine~ that the fail stop bit i~ not set, processing system 20 conclude~ it cannot bootstrap ll load and enters the console mode (step 2412) where it awaits 12 further instructions.
~ ~f,,~ss~n~ S15~e~
13 If proce~eor 20 deterimine~ that cable 25 is connected 14 (step 2404), processing system 20 determine~, via the cro~s-link cable 25, whether processing sy~tem 20' i8 po~ered on 16 (~tep 2414). If not, procRssing ~ystem 20 ~et~ its OSR bit 17 (step 2408) and initiates bootstrap loading (~tep 2410).
18 800tstrap loading is permissible under the~e condition~ ~ince 1l:
l9 the other proce~-ing system is not only not operating but is not even powered on. Thu~, there i8 no risk of running two copies 21 of the operating ~y~tem.
22 If proce~ing ~ystem 20' ha~ power (step 2414), proce~sing 23 syste~ 20 attempts, by the ~tatu- read operation, to read the 24 contents of control and status register 912 of processing ~ystem 20'. If the read is unsuccessful, processing ~y~tem 20 2$L~O~Ie~ concludes that the operating ~tatus of proce~ing system 20~ is ~7 !CW HENDERSON
El LA80W C~RRETT ..
;~ 6 DUNN~R
T~ T. 1 . ~. ~
OTOI~. O. C. ~000- 1 11 ~
1 .
i l `

2~2~259 1 unknown, and proceed8 to checX the fail 8top bit ~tatu~ (8tep 2 2406) and repeat the procedure described above with reg~rd to 3 that stsp. .
4 If the control and status r0gi~ter in processing system 20' is accessible (step 2416), procQs~ing sy~tem 20 check~ whether 6 the OSR bit in proce~sing systam 20' is ~et (step 2418). Becau~e 7 the OSR bit iB normally set as part of the bootstrap loading 8 oper~tion, it cannot have been set in proce~sing syst2m 20 while 9 executing step 2418. If procassing sy~tem 20 determines that tha OSR bit is 8et in processing syst~m 20', proce~sing system 11 20 concludes that it cnnnot boots~rap load the operating 8y8tem 12 and the proce~ing ~y~tam goe~ to the console mode where it 13 aw~its fuxther inatruction~ (step 2412). ThQ proce~ing systam 14 would therefore hnve to w~it for the opportunity to perform a memory resynchronizing operat$on in order to enter synchronized 16 operation with the proce~sing systam of tho other zone.
17 If processinq systam 20 determine~ that the OSR bit i8 not 18 ~et in proce~ing syst~m 20~, it ~QtJ it~ OSR bit using a 19 sem~phore con~truct known to per~ons of ordinary skill (step 2420). The OSR bit ~an be ~et be~use At thi~ ~tage, it ha~
21 beon d termin~d that both procQ~sing sy~te~s 20 and 20~ have 22 pow-r, tho cro~-link cable 25 i8 connected botween zones 11 ànd 23 11' ~nd pro~o~lng sy~tem~ 20 and 20~ are able to communicate 24 with one another over the crc~s-link cable. It Ls thereforequite likely that both procQ~sing systemJ Are attempting to 26~o~rlct> d0toxmin~ whother it i~ pormLs~ible for th~m to boot~trap load.
INNEC,W, HENDERSON
~a DUNNER ~hU~, proce~ing 8y~tam 20 attempt~ to 8et th~ OSR b~t uniquely ,~. " "",~,,". ~. I
w~lt~OTOI~. 0. c. ~000-~0.,.-.---,0 ....
-86- .

.

~ 2~2~9 1 using a semaphore con8truct. The processing Bystem in which the 2 OSR blt i~ ~et will sub8equently proceed to bootstrap load the 3 operating system (step 2422).
This algorithm accommodate~ the different situations that S may result. For sxample, one situation occur3 when both 6 processing systems 20 and 20' have bean operating in lockstep 7 synchronism when one of the proce~sing systems i8 disabled. For 8 example, a technician may simply have turned off the power to 9 one ~ystem, 9.g. qystem 20. As a result, the fall stop bit will not ba ~et in the remaining operating proce~6ing system, L.e., 11 sy~tem 20'. Further, the OSR bit in proce~ing ~y~tQm 20 12- becomss re~et while the OSR bit rsm~n~ set in proce~slng system 13 20'.
Upon completing the ~ervice of processing ~ystam 20, the techn$cian in~truct~ the processing By~tem to inltiate ~ootstrap 16 loading of the operatlng sy~tem. ~pon execution of the 17 algorithm illustrated in flQw chart 2400, proces~ing qystem 20 18 of zone 11 finds the cro~s-link c~ble 25 connected (2404) and 19 proceeds to step 2414 in whlch it determines that proces~ing ~yste~ 20~ of zone 11~ i~ powered on. Processing sy~tsm 20 21 therefore proceed~ to step 2416 where it should find control and 22 ~tatu~ rogi~ter 912 of proce~sing system 20~ accessible.
23 As~umlng the 8y~tem 20~ regi~ter is accee~ible, proce~ing ¦
24 ~y~tem 20 proceeds to Qtep 2418 where it find~ th~t the OSR bit 2S is ~et in sy~tem 20~. It therefore concludes it cannot ~-2~wo~r~ct~ boot~trap load the operating ~yst~m a~ comM~nded and proceeds to `;NEC,~I. HENDER50N
F~ow.~RRErr th~ con~ole m~de (8tep 2412). If register 912 of ~y~tQm 20~ is 7~ ~ ~T~ttT. ~. w.
V~oTor . ~. C ~ooo-~ ~o~ o ~

~ -87-.' ~

~02~2~9 1 not accessible, processing system 20 proceeds to step 2406 in2 which it determinee that its fail stop bit i8 not set 3 Processing ~ystem 20 therefore proceeds to the console mode 4 (~t~p 2412) In another situation, the cross-link cable 25 is 6 disconnected while both processing systems 20 and 20' are 7 operating in lock step ~ynchronism As a result, a 8 predetermined one of the proces~ing sy~tem~, e g ~ystem 20~j 9 takes itself out of service Su~sequently, the cross-link cable 0 i8 reconnected and syst~m 20' is comm~nded to bootstrap lo~d 11 Referring to flow chart 2400 in Fig 24, proces~ing ~ystem 20' 12 of zone 11' determine- that the cro-~-link cable 25 is connscted 13 (step 2404) ~nd determines that zone 11 i- pcwered on (step 14 2414) Processing sy~tem 20' then attempt~ to ~cces~ the control and statu~ regl-ter of proce-sing sy~tem 20 (step 2416) 16 If processing sy~tem 20 returns the register 912 information, 17 processlng y~tem 20' determine~ whether the OSR bit is set in 18 processing system 20 (step 2418) That bit will be set since 19 system 20 is oper~ting Proce-sing system 20' therefore ! ~-conclude~ th~t lt cannot bootstrap lo~d ~ copy of the operating 21 ~y~te~ ~nd goe~ to con~ole mode (step 2412) If proces~ing 22 sy-tem 20 doe~ not return the inform~tion contained in its 23 r-gl-t-r 912, then proce~ing 20' determines th~t lt~ fail ~top 24 blt 1~ not set (~tep 2406) Sy~tem 20' therefore proceeds to 2S tho con~ole mode (-tep 2412) The Algorithm thu~ assures the 2C^wo~c~ correct result ~ince bootstrap loading by processing system 20' INNEC~N . HENDER50N
EA~90Yr G~RRETr 27 ~ D~INER l would re~ult in two ep~r~te copie~ of the oper~ting ~yatem ,". " .. ,.. w.
~0~0~-.0 c ~000-~0...~ 0 .

. .,' . ': .
'': ' ' ' ~'. :' 2,)222~g l, 1 baing run by the two proce~sing ~ystem~ 20 and 20', 2 respectively. This i8 the situation that operation of the 3 algorith~ of flow chart 2400 2xpressly avoids.
4 V. CONCLUSION
The present invention, a~ shown by it~ preferred 6 ~ embodiment, thus achieves significant advantage~, ~uch as 7 insuring that the t~7O computing zo~e~ of a f ault tolerant ~y~tem 8 operate from a Bingle copy of the op~rating ~y~tem. It will be 9 apparent to those skilled in the art that modificat~ons and variat~on~ can be made in the me~hod~ and apparatu~ of thi~
11 invention. The invention in its broader a~poct~ i~, there~ore, 12- not l~m~ted to th~ spe¢ific details, r~pre~entative method~ and 13 apparatus, and illustrative exa~ples shown and dQ~cribed.
14 ~ccordingly, departure m~y be made from such detall~ without departing from the spirit or ~cope of the g~neral inventiv~
16 concept.
,17 ~;
. ~.,., ~O~C--., ~NE~N HENDERsoN
F.~R~30W. ~RRET7 uNNER I
';~'i 177~ TI~t~
t~ 70~0C.OOO.
Ø,.., ...0 ,4~,, ~, . I, ',?~ ' ,~;, .

i~: . . . .. . .
:. . . - . : . - .

Claims (12)

1. In a computer system having first and second discrete computing zones, a method for initiating bootstrap loading of an operating system, the first zone including a first CPU and the second zone including a second CPU, said first and second CPUs for independently executing said method for initiating bootstrap loading, each of said zones having a status register accessible by the other zone, said method comprising the steps of: determining either that a selected one of the first and second zones is allowed to bootstrap load or that a non-selected one of said first and second zones is incapable of running the operating system;
determining whether the non-selected zone has initiated bootstrap loading of the operating system by accessing, from the selected zone, the status register in the non-selected zone; and initiating bootstrap loading in the selected zone at times when said selected zone is allowed to bootstrap load; initiating bootstrap loading in the selected zone at times when said non-selected zone is incapable of running the operating system; and initiating bootstrap loading in the selected zone at times when said non-selected zone has not initiated bootstrap loading of the operating system; wherein the step of determining whether the selected zone is allowed to bootstrap load includes the substep of determining that bootstrap loading of the selected zone cannot be initiated at times when the non-selected zone has initiated bootstrap loading.

2. The method of claim 1 wherein the step of determining whether the selected zone is allowed to bootstrap load includes the step of reading at the selected zone an availability signal in said selected zone.

3. The method of claim 1 wherein an availability signal is intentionally set in said selected zone when said non-selected zone is removed from service; and wherein said step of determining whether the selected zone is allowed to bootstrap load comprises the substep of reading said availability signal.

4. The method of claim 1 wherein the zones are connected by a cable and wherein the step of determining either that a selected one of the first and second zones is allowed to bootstrap load or that a non-selected one of said first and second zones is incapable of running the operating system includes the substep of:
reading at the selected zone a zone signal indicating whether said cable is connected and whether the non-selected zone is not receiving power.

5. The method of claim 1 including the additional step, upon initiating bootstrap loading, of setting a predetermined bit in the status register of the selected zone indicating that bootstrap loading has been initiated.

6. In a computer system having first and second discrete computing zones, a method for bootstrap loading the first zone, the first and second zones each having a memory and being interconnected by a cable by which the first and second zones can read status bits in a status register in the second and first zones, respectively, the first zone including a CPU for executing said method for initiating bootstrap loading, said method comprising the steps of: determining if the cable is connected to the second zone; reading by the CPU, at times when the cable is not connected to the second zone, a first status bit stored in the memory of the first zone; setting, at times when the first status bit is set in the first zone, a second status bit in the status register of the first zone and causing the first zone to bootstrap load; determining, at times when the cable is connected to the second zone, if the second zone is powered on; setting, at times when the second zone is not powered on, the second status bit in the status register of the first zone and causing the first zone to bootstrap load; reading by the CPU, at times when the second zone is powered on, a state of the second status bit in the status register of the second zone; attempting, at times when the second status bit is not set in the second zone, to set the second status bit in the status register of said first zone using a semaphore construct and causing the first zone to bootstrap load if the second status bit is set; and reading by the CPU, at times when the state of the second status bit in the second zone status register cannot be read, the first status bit in the first zone and setting, at times when the first status bit is set in the first zone, the second status bit in the status register of the first zone and causing the first zone to bootstrap load.

7. The method of claim 6 including the additional step of:
determining that the first zone cannot bootstrap load if the second status bit is set in the second zone.

8. The method of claim 6 including the additional steps of resynchronizing the first and second zones; and clearing the first status bit in said first and second zones when said first and second zones successfully resynchronize with each other.

9. The method of claim 6 including the step of: causing the first zone to go to an inactive state if, upon reading the first status bit, the first status bit is not set.

10. A computer system, comprising: a first discrete computing zone including a first CPU, and a first status register storing a first bit indicating whether the first zone has initiated bootstrap loading; a second discrete computing zone including a second CPU, and a second status register storing a first bit indicating whether the second zone has initiated bootstrap loading; a cable coupled between the first and second zones, to allow said first and second zones to read the status registers in said second and first zones, respectively; means, in said first CPU, for determining whether the first bit stored in said second status register indicates that said second zone has initiated bootstrap loading and for determining that bootstrap loading of said first zone cannot be initiated if said second zone has initiated bootstrap loading; said first zone including a first memory for storing a second bit indicating that said first zone can initiate bootstrap loading; and means for determining if said cable is connected to said second zone; and wherein said first CPU
includes means for determining that said first zone can bootstrap load if said cable is not connected and said second bit indicates that said first zone can initiate bootstrap loading.

11. A computer system, comprising: a first discrete computing zone including a first CPU, and a first status register storing a first bit indicating whether the first zone has initiated bootstrap loading; a second discrete computing zone including a second CPU, and a second status register storing the first bit indicating whether the second zone has initiated bootstrap loading; a cable coupled between the first and second zones, to allow said first and second zones to read the status registers in said second and first zones, respectively, said cable including a first conductor; means, in said first CPU, for determining whether the first bit stored in said second status register indicates that said second zone has initiated bootstrap loading; and means for determining at said first zone when said cable is connected between said first and second zones and said second zone is receiving power; said cable determining means including a first resistor coupled between the first zone end of said first conductor and ground, a second resistor coupled between a second zone end of said first conductor and a first point that is at a predetermined logic level when said second zone is receiving power, and means for reading a logic voltage at the first zone end of said first conductor.

12. The computer system of claim 11 wherein said cable comprises a second cable; said cable connected determining means including: a first diode coupled between the first zone end of said first conductor and a second point that is at a logic "1"
voltage level when said first zone is powered on, said second point being connected to ground when said first zone does not have power; a second diode coupled between the second zone end of said second conductor and the first point; a third resistor coupled between a first zone end of said second conductor and the second point; and a fourth resistor coupled between the second zone end of said second conductor and ground; whereby logic "0" and "1"
voltage levels are respectively read on the first zone ends of said first and second conductors when said cable is not connected between said first and second zones and further whereby logic "0"
voltage levels are respectively read on the first zone ends of said first and second conductors when said cable is connected between said first and second zones and said second zone does not have power. 95