CA2132756C - High efficiency learning network - Google Patents
High efficiency learning networkInfo
- Publication number
- CA2132756C CA2132756C CA002132756A CA2132756A CA2132756C CA 2132756 C CA2132756 C CA 2132756C CA 002132756 A CA002132756 A CA 002132756A CA 2132756 A CA2132756 A CA 2132756A CA 2132756 C CA2132756 C CA 2132756C
- Authority
- CA
- Canada
- Prior art keywords
- layer
- network
- computational
- output
- inputs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
Abstract
Nodal outputs are discretized to values of S2n where n is an integer and S is equal to +1 or -1. During forward propagation, this offers the advantage offorming a product of a nodal output and a weight using a simple shift operation rather than a multiply operation. Replacing multiply operations with shift operations through out a neural network improves response times and permits building largernetworks that have broader applicability. Training is also improved by increasing the efficiency of backward propagation. The multiplications involved in backwardpropagation are reduced to shift operations by discretizing the errors associated with each node so that they are represented as S2n where n is an integer and S is equal to +1 or -1.
Description
,., 1 --High Efficiency Learning Network Field of the Invention The invention relates to learning networks; more specifically, learning nelwolk~ with high efficiency forward propagation and backward propagation.
S Description of the Prior Art Learning networks or neural networks are available in a variety of architectures that are implemented in haldw~e, software or a combination of hardware and software. U. S. patent 5,067,164 entitled "Hierarchical ConstrainedAutomatic I ~rning Neural Network for Character Recognition" and U. S. patent 10 5,058,179 entitled "Hierarchical Constrained Automatic T ~rning Network for Character Recognition" disclose two of many architectures available to learning ne~w~lk~,. Learning networks comprise co~ u~a~ional nodes or neurons that pelru an activation function using a sum of its inputs as an operand to provide an output.
These nodes are typically arranged in layers with the output of a node from one layer lS being connected to the input of several nodes of the next layer. Each input to a node is typically multiplied by a weight before being summed by the node.
I e~rning ne~wolh~, or neural networks typically include a large number of nodes or neurons that receive inputs from many other neurons within the learning nelwc lk. This results in an architecture where there are many interconnections 20 between the outputs and inputs of the nodes. As mentioned earlier, these interconnections normally include a multiplication function. As a result, large neural networks require a large number of multiplications to produce an output. In addition, these multiplications typically involve multiplying a 16-bit word by another 16-bit word and thereby require a great deal of colnpulaLional power. As a 25 result of these conlpu~aLional requirements, large neural networks often have slow response times and slow training times.
With the ever-expanding demand for products that involve functions such as speech recognition, handwriting recognition and pattern recognition, there is an increasing need for large neural networks with fast response times and short 30 training times.
Summary of the Invention The present invention provides a learning or nodal network that replaces multiplications with a shift operation. A shift operation greatly reduces the computational overhead of the learning network. As a result, ne~wolk~ with large35 numbers of interconnections can be provided while m~int:~ining fast response times and fast training times.
._ -2- 2 ~ 3 2 7 ~ 6 The present invention decreases computation overhead by discretizing the output of each node or neuron of the network so that its output is represented by S2n where n is an integer and S is +1 or -1. This offers the advantage of implementing the multiplication associated with each connection by shifting the weight associated with that 5 connection n times. As a result, each connection of the network uses a simple shift operation as opposed to a multiply operation.
In the training or backward propagation mode, the gradient used to adjust the weights is also represented as S2n where n is an integer and S is +1 or -1. This enables the value of new weights to be calculated by using a simple shift operation as opposed to 10 a multiplication operation. As with the forward propagation case, this reduces the computation overhead and thereby permits faster training of the network.
Providing networks with less computational overhead, and therefore faster response times and faster training times, permits implement~ting more complex networks.
By implementing networks with many more connections and less computational 15 overhead, the recognition capability of a network is greatly increased without increasing response times or training times to unacceptable levels.
In accordance with one aspect of the present invention there is provided a computational network that produces a plurality of network outputs in response to a plurality of network inputs, characterized by: a plurality of first layer computational 20 nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero power of 2; and a plurality of second layer computational nodes forming a second layer of the computational network, each of said 25 plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer 30 output being used to produce at least one of a plurality of network outputs.
. .
-2a- ~ ~ 3 ~ 7 ~ ~
Brief Description of the Dr~wi~ s FIG. 1 illustrates a learning network architecture;
FIG. 2 illustrates the relationship between the inputs of one node and the outputs of several nodes;
S FIG. 3 illustrates a SIGMOID activation function;
FIG. 4 illustrates a discretizing activation function;
FIG. 5 illustrates a simplified multiplier;
FIG. 6 illustrates an error discretizing function;
FIG. 7 illustrates an approximation for the activation function of FIG. 4;
FIG. 8 illustrates a derivative of an activation function; and FIG. 9 illustrates the relationship between nodes of adjacent layers during backward propagation.
Description of the Preferred Embodiment FIG. 1 illustrates a generic neural or learning network architecture that may beimplemented in hardware or software. In this example, there are 3 layers to the network;
however, it is possible to configure networks with any number of layers. It is also possible to use other well-known connecting patterns such as those used by convolution neural networks and locally connected neural networks. Nodes 10, 12, 14 and 16 comprise the first layer, nodes 18, 20, 22 and 24 comprise the - -;A ' ..~, second layer and nodes 26, 28, 30 and 32 comprise the third layer. The third layer is the output layer. In this example, layers one, two and three are shown to contain L, M and P nodes, respectively, where L, M and P are any positive integer and whereeach layer may have a dirrerell~ number of nodes. The outputs of nodes 10, 12, 14 5 and 16 are represented by Xk where k indexes from 1 through L. Middle layer nodes 18, 20, 22 and 24 have outputs represented by X i where i indexes from 1 through M. Likewise, the network outputs, which are the outputs of nodes 26, 28, 30 and 32, are represented by Xj where j indexes from 1 through P. In this example,nodes of adjacent layers are fully connected, that is, each node in one layer is10 connected to every node of an adjacent layer. For example, the output of node 20 is connected to nodes 26, 28, 30 and 32, and the input to node 28 is connected to the output of nodes 18, 20, 22 and 24. Each connection between the output of a node in a precefling layer and the input of a node in a subsequent layer includes a multiplication which forms the product of the precefling node's output and a weight.
15 The resulting product is received by an input of the node in the subsequent layer.
FIG. 2 illustrates the operation of one node. Node 30 sums all of its inputs and then executes an activation function using that ~u~ a~ion as an operand to produce output Xj. The inputs to node 30 are received from the outputs of multipliers 52, 54, 56 and 58. Each multiplier receives an input from the output of a 20 node in a previous layer and multiplies that input by a weight. In this example, outputs X i (i = 1 to M), which are received from the nodes of a previous layer, are multiplied by weights Wji (i = 1 to M) in multipliers 52, 54, 56 and 58. If node 30 is not part of the output layer, output Xj is used as an input to nodes in the next layer, and if node 30 is part of the output layer, output Xj is one of the outputs from the 25 network.
The network of FIG. 1 operates in two modes, a fol~val(l propagation mode and a backward propagation mode. In the forward propagation mode, an input or inputs are provided to the first layer of the network and outputs are provided at the output of the network. This involves the multiplication associated with each 30 connection within the network, a sllmm~tion of the products at the inputs to each node, and an activation function performed on the summation at each node. In thesecond or backward prop~g~tion mode, the network is trained. In this mode, inputs are provided to the network and the resulting outputs are observed for accuracy. An error is associated with each output and then an error gradient is calculated so that 35 each of the weights in the intel~;onnecting network can be adjusted. Each weight is adjusted by forming a product using the gradient, the output of the node from the previous layer and a learning rate.
Equation 1 illustrates the relationship between the outputs X i from the nodes in a previous layer and output Xj of a node in a following layer.
Xj=f ~ XiWji (1) ~ i , 5 The equation shows that each output Xi received from a previous layer is multiplied by a weight Wji to produce i products. The resulting products are sllmmt-cl, and the sum is used as an operand by activation function f ( ). In the past, the activation function was a function such as a SIGMOID illustrated in FIG. 3. It is also possible to use many other well-known functions such as linear, spline and hyperbolic 10 tangent. It is also possible to use different activation functions for nodes in dirrelc;llt layers, or for nodes in the same layer. The present invention uses an activationfunction that discretizes its output so that it can be represented by a power of 2 or more speçific~lly by S2n where n is an integer and S is +1 or -1. FIG. 4 illustrates such an activation function. This function can be implemented by using a look-up15 table in software embo liment~ or by using a PROM in hardware embodiments. The figure shows that for each value of the operand or summation on axis 70, a givenvalue of X is produced as shown on axis 72. In this example, the values of X can be represented by S2n where n can have a value of 0, -1, -2 or -3. It is also possible to create functions where n can be any integer (. . . +3, +2, +1, 0, -1, -2, -3 . . .).
In reference to FIG. 2 the outputs X i from nodes of a previous layer are multiplied by weights Wji to form products using multiplier 52, 54, 56 and 58. By using the discretized activation function of FIG. 4, the outputs Xi from each node can be represented by a sign bit and the value of n. As a result, the multiplications carried out by multipliers 52,54, 56 and 58 may be executed by simply making the25 appl~liate change to the sign bit of weight Wj i and then shifting the weight by the number of locations that are specified by n. For example, if the summation produced at the input to prior node 18 is equal to -0.4, the output of node 18 that results from using -0.4 as an operand of the activation function of FIG. 4 is -0.5. As a result, the output of node 18 is r~lesented as - 2- 1. When this output is used as an input to 30 multiplier 52, weight Wj i= 1 is simply shifted one location to the right and the sign bit is changed to produce the result X i= l Wj,i= 1. This enables multiplier 52 and the other multipliers used in the connections within the network to be replaced by a simple shift register that shifts weight Wj i n times to the left or to the right. When n ,., .~
is negative, the weight is shifted to the right and when n is positive, the weight is shifted to the left. The sign bits of Wj i and Xi can be passed through an EXCLUSIVE-OR gate to produce the sign bit of the product.
Inputs to the first layer of the network may also be discretized in 5 accordance with FIG. 4. In netwolk configurations where the network input is multiplied by a weight, representing the input as a power of 2 permits implementing these initial multiplications with shift operations.
FIG. 5 illustrates a simplified multiplier. A simple shift register and EXCLUSIVE-OR gate produce the product Xi Wj i without the use of a 10 conventional multiplier. The value of n controls the number of shifts and thedirection of the shifts. Using a shift register and simple logic gates to replace multipliers throughout the network results in a haldwa.~ and/or software reduction which reduces the colllpu~a~ional overload associated with the network. When a network is implemented in software, replacing a multiplication with a shift operation 15 decreases computation time and thereby decreases the network's response time.In an effort to accommodate a wide variation in the values of the weights, it is possible to format the weights in the form of a mantissa and exponent;
however, the weights may be represented in other forms such as a fixed point representation. When operating using a m~nti~S~ and exponent, the shift register of 20 FIG. 5 can be replaced with an adder that is used to sum the exponent of the weight and n to obtain the product's exponent. In our prior example, where X i is equal to -0.5 and is expressed as _ 2- l where n = -1 and s = -1, the exponent of the weight is summed with -1 and sign bit of the weight is EXCLUSIVE-OR with the sign bit -1.
It should be noted that the node receiving the resulting products in the form of a 25 m~nti~S~ and exponent should perform the applupliate shifts to align the mantissas before a ~".,~"~tion is formed.
The second mode of operation for learning or neural networks is the backward propagation mode. This mode is used to train or teach the network by adjusting the weights used in the network. The process involves providing a set of 30 training inputs to the network and determining the error at the network's outputs.
The errors are used to adjust the values of the weights. This process is continued using many dirrelc;nt training inputs until the network produces outputs with acceptable errors. After allowing the network to produce a set of outputs in response to a set of training inputs, backward propagation begins by detelmilling the error at 35 each output of the network. The following equation provides an expression for output error E.
E = ~, ej ej = 2 [Xj - Tj~ (2) Equation 2 shows that the error ej at output "j" is represented by the difference between an actual output Xj and desired output Tj. Substituting equation 1 into 5 equation 2 results in, equation 3 which provides another expression for error ej.
~ 2 ej = 2 f ~ Xi Wji -Tj = 2 [f(Sj) - T,] (3) whereSj = ~, Xi Wj,i-As a result, the output error E may be represented by equation 4.
E = ~ 1/2 [f(Sj) - T~] (4) 10 The unit gradient aaS for unit j of the last layer is given by equation 5.
Gj = aS = (f(Sj) --Tj)f'(Sj) = Hj f'(Sj) (5) where the error term Hj = f(S j - Tj ) Once the unit gradients of one layer are known, the unit gradients of previous layers may be calculated using the chain rule. For example, equation 6 15 illustrates calculating unit gradients Gi for unit i of the layer preceding layer j.
Note that Xi = f ~ Xk Wik = f(Si) and that Sj = ~ Xi Wji = ~, f(Si) Wji As a result, Gi = aaSi = ~ aaS aSj = ~, Gj Wji f'(Si) = Hi f'~Si~ (6) In reference to equations 5 and 6, it can be seen that a gradient G for a 5 layer may be com~u~ed using an error term H and a derivative term f' ( ). For the output layer or last layer, the error term H is expressed by Hj = f(Sj) - Tj and for previous layers, the error term H is expressed in the form Hi = ~ Gj Wji ( ) j 10 where layer i precedes layer j, and layer j is closer to the output layer than layer i.
The weights W, are adjusted using the gradient of E with respect to the weight of interest. For example, the gradient of E with respect to weight Wji isgiven by equation 8.
aE = aE aaSj = Gj Xi (8) The weight adjustment is carried out by descending the gradient of E
with respect to Wji as shown in equation 9.
WjnieW = WJ~ild -- LR aaE = Wj~ild -- [LR Gj Xl] (9) Similarly, the weights in the layer i, which precedes layer j, are adjusting using aE = aE aaSi =GiXk (10) and WinkeW = Wi~kd _ LR a W = Wikd -- [ LR G i X k] ( 1 1 ) 21327~6 LR is the learning rate and typically has a small value such as 0.001.
Many implementations vary the learning rate from unit to unit, and can vary the learning rate over time. It is also possible to limit LR to powers of 2, that is, LR = 2n where n is an integer. It is preferable to limit n to negative integers.When beginning backward propagation, the gradient G j of the output layer is first computed according to equation 5. The term Hj, which is equal to (F (S j ) - Tj ), is discretized to the nearest power of two in accordance with the discretization function of FIG. 6. This function can be implemented using a look-up table in software embodiments and a PROM in haldw~ut; embodiments. In FIG. 6, 10 every value on the horizontal axis is mapped to a power of two on the vertical axis.
After discretization, Hj can be stored in the form of S 2n where n is an integer and S
is -1 or +1.
Equation 5 also requires forming the product Hj f' ( S j ). For the purpose of calculating f' ( ) (the derivative of the activation), f( ), (the activation 15 function of FM. 4) is appr~ ated by the function of FIG. 7. As can be seen inFIG. 8, the value of f' ( ) is either 0 or 1. (The value 0 could also be a small power of 2 such as 0.125 to f~cilit~te learning). When other activation functions are used, it is also possible to represent f' ( ) as S 2n when n is an integer and S is _ 1.
MultiplyingHj byf'(Sj)iscarriedoutbydeLel~ gifS; isbetween-l and 1. If 20 S j is between -1 and 1, the product is equal to Hj, and if S j is not between -1 and 1, the product is 0 (or a small power of 2 times Hj). If a small power of 2 such as 2-n is used instead of 0, Hj is shifted to the right by n bits. This completes the colllpu~alion of the gradient Gj . It should be noted that Gj is in the form of S2n, where n is an integer, because Hj is discretized in accordance with FIG. 6 and f' (S j ) 25 is either 0 or 1.
The weight update is made according to equation 9. It should be noted that Xi is in the form S2n where n is an integer and S is +l or -1. (Recall the activation function of FIG. 4). As a result, when beginning the calculation of equation 9 the product of gradient Gj and node output Xi is a simple addition of the 30 values of n associated with Gj and Xi. The sign bits of the product Gj Xi is formed by EXCLUSIVE-ORing the sign bits of Gj and Xi. It should be noted that since both Gj and Xi are in the form of S2n, the product Gj Xi is also in the form S2n.
As a result, multiplying learning rate LR by Gj Xi involves simply shifting LR
based on the value of n associated with the product Gj Xi and EXCLUSIVE-35 ORING the sign bits. As a result, the new value of Wji is obtained using simpleshift and add operations.
~.
g It should be noted that, by representing gradient Gj and nodal output X
as S2n, multiplication is no longer necessary to establish a new weight using equation 9. As mentioned earlier, elimin~ting multiplications permits faster operation and more complex networks.
Generally speaking, and in reference to FIG. 9, weights at preceding layers of the network are adjusted using error values that are propagated backward through the network. For illustrative purposes we will assume that layer n precedes n + 1 and layer n + 1 is closer to the network output than layer n. An error term H i for each node of layer n is calculated by ~",.~ -g the products of the unit gradients 10 Gj from layer n + 1 and their associated unadjusted weight values Wji from layer n + 1 as illustrated in equation 7. The error term Hi associated with each node of layer n is discretized to a power of 2, and is used to produce a unit gradient G i associated with each node of layer n in accordance with equation 6. The weights Wik of layer n are adjusted using the unit gradients Gi of layer n and the outputs Xk 15 of layer n- 1 as illustrated in equation 11. This procedure is repeated for preceding layers until all of the weights have been adjusted.
The nodes and multipliers may be implemented in software on a com~ule~, or in haldwal~e using analog or digital circuitry. In a hardware configuration, each node may be implem--nted using a microprocessor or a digital20 signal processing (DSP) chip. It is also possible to implement each node with an acc-lmlll~tor and a PROM where the acc~lm~ tor sums the inputs and the PROM
uses the sum to produce an output in the form of S 2n.
The weights can be stored in a memory such as a RAM or register. The weights can be updated by implementing the network on colll~u~er and then using a 25 set of training inputs to develop output errors that are back propagated to change the weights. It is also possible to program the microprocessors or DSP chips to use error information from subsequent layers and outputs from previous layers to adjust the weights.
A nodal or learning network may be used to recognize speech, 30 handwriting, pattern or objects. For example, the inputs to the first layer may be signals or values representative of an input such as an utterance or handwrittensymbol. Once trained using backward propagation, the outputs produced in response to these inputs may be used to identify the utterance or handwritten symbol.
S Description of the Prior Art Learning networks or neural networks are available in a variety of architectures that are implemented in haldw~e, software or a combination of hardware and software. U. S. patent 5,067,164 entitled "Hierarchical ConstrainedAutomatic I ~rning Neural Network for Character Recognition" and U. S. patent 10 5,058,179 entitled "Hierarchical Constrained Automatic T ~rning Network for Character Recognition" disclose two of many architectures available to learning ne~w~lk~,. Learning networks comprise co~ u~a~ional nodes or neurons that pelru an activation function using a sum of its inputs as an operand to provide an output.
These nodes are typically arranged in layers with the output of a node from one layer lS being connected to the input of several nodes of the next layer. Each input to a node is typically multiplied by a weight before being summed by the node.
I e~rning ne~wolh~, or neural networks typically include a large number of nodes or neurons that receive inputs from many other neurons within the learning nelwc lk. This results in an architecture where there are many interconnections 20 between the outputs and inputs of the nodes. As mentioned earlier, these interconnections normally include a multiplication function. As a result, large neural networks require a large number of multiplications to produce an output. In addition, these multiplications typically involve multiplying a 16-bit word by another 16-bit word and thereby require a great deal of colnpulaLional power. As a 25 result of these conlpu~aLional requirements, large neural networks often have slow response times and slow training times.
With the ever-expanding demand for products that involve functions such as speech recognition, handwriting recognition and pattern recognition, there is an increasing need for large neural networks with fast response times and short 30 training times.
Summary of the Invention The present invention provides a learning or nodal network that replaces multiplications with a shift operation. A shift operation greatly reduces the computational overhead of the learning network. As a result, ne~wolk~ with large35 numbers of interconnections can be provided while m~int:~ining fast response times and fast training times.
._ -2- 2 ~ 3 2 7 ~ 6 The present invention decreases computation overhead by discretizing the output of each node or neuron of the network so that its output is represented by S2n where n is an integer and S is +1 or -1. This offers the advantage of implementing the multiplication associated with each connection by shifting the weight associated with that 5 connection n times. As a result, each connection of the network uses a simple shift operation as opposed to a multiply operation.
In the training or backward propagation mode, the gradient used to adjust the weights is also represented as S2n where n is an integer and S is +1 or -1. This enables the value of new weights to be calculated by using a simple shift operation as opposed to 10 a multiplication operation. As with the forward propagation case, this reduces the computation overhead and thereby permits faster training of the network.
Providing networks with less computational overhead, and therefore faster response times and faster training times, permits implement~ting more complex networks.
By implementing networks with many more connections and less computational 15 overhead, the recognition capability of a network is greatly increased without increasing response times or training times to unacceptable levels.
In accordance with one aspect of the present invention there is provided a computational network that produces a plurality of network outputs in response to a plurality of network inputs, characterized by: a plurality of first layer computational 20 nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero power of 2; and a plurality of second layer computational nodes forming a second layer of the computational network, each of said 25 plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer 30 output being used to produce at least one of a plurality of network outputs.
. .
-2a- ~ ~ 3 ~ 7 ~ ~
Brief Description of the Dr~wi~ s FIG. 1 illustrates a learning network architecture;
FIG. 2 illustrates the relationship between the inputs of one node and the outputs of several nodes;
S FIG. 3 illustrates a SIGMOID activation function;
FIG. 4 illustrates a discretizing activation function;
FIG. 5 illustrates a simplified multiplier;
FIG. 6 illustrates an error discretizing function;
FIG. 7 illustrates an approximation for the activation function of FIG. 4;
FIG. 8 illustrates a derivative of an activation function; and FIG. 9 illustrates the relationship between nodes of adjacent layers during backward propagation.
Description of the Preferred Embodiment FIG. 1 illustrates a generic neural or learning network architecture that may beimplemented in hardware or software. In this example, there are 3 layers to the network;
however, it is possible to configure networks with any number of layers. It is also possible to use other well-known connecting patterns such as those used by convolution neural networks and locally connected neural networks. Nodes 10, 12, 14 and 16 comprise the first layer, nodes 18, 20, 22 and 24 comprise the - -;A ' ..~, second layer and nodes 26, 28, 30 and 32 comprise the third layer. The third layer is the output layer. In this example, layers one, two and three are shown to contain L, M and P nodes, respectively, where L, M and P are any positive integer and whereeach layer may have a dirrerell~ number of nodes. The outputs of nodes 10, 12, 14 5 and 16 are represented by Xk where k indexes from 1 through L. Middle layer nodes 18, 20, 22 and 24 have outputs represented by X i where i indexes from 1 through M. Likewise, the network outputs, which are the outputs of nodes 26, 28, 30 and 32, are represented by Xj where j indexes from 1 through P. In this example,nodes of adjacent layers are fully connected, that is, each node in one layer is10 connected to every node of an adjacent layer. For example, the output of node 20 is connected to nodes 26, 28, 30 and 32, and the input to node 28 is connected to the output of nodes 18, 20, 22 and 24. Each connection between the output of a node in a precefling layer and the input of a node in a subsequent layer includes a multiplication which forms the product of the precefling node's output and a weight.
15 The resulting product is received by an input of the node in the subsequent layer.
FIG. 2 illustrates the operation of one node. Node 30 sums all of its inputs and then executes an activation function using that ~u~ a~ion as an operand to produce output Xj. The inputs to node 30 are received from the outputs of multipliers 52, 54, 56 and 58. Each multiplier receives an input from the output of a 20 node in a previous layer and multiplies that input by a weight. In this example, outputs X i (i = 1 to M), which are received from the nodes of a previous layer, are multiplied by weights Wji (i = 1 to M) in multipliers 52, 54, 56 and 58. If node 30 is not part of the output layer, output Xj is used as an input to nodes in the next layer, and if node 30 is part of the output layer, output Xj is one of the outputs from the 25 network.
The network of FIG. 1 operates in two modes, a fol~val(l propagation mode and a backward propagation mode. In the forward propagation mode, an input or inputs are provided to the first layer of the network and outputs are provided at the output of the network. This involves the multiplication associated with each 30 connection within the network, a sllmm~tion of the products at the inputs to each node, and an activation function performed on the summation at each node. In thesecond or backward prop~g~tion mode, the network is trained. In this mode, inputs are provided to the network and the resulting outputs are observed for accuracy. An error is associated with each output and then an error gradient is calculated so that 35 each of the weights in the intel~;onnecting network can be adjusted. Each weight is adjusted by forming a product using the gradient, the output of the node from the previous layer and a learning rate.
Equation 1 illustrates the relationship between the outputs X i from the nodes in a previous layer and output Xj of a node in a following layer.
Xj=f ~ XiWji (1) ~ i , 5 The equation shows that each output Xi received from a previous layer is multiplied by a weight Wji to produce i products. The resulting products are sllmmt-cl, and the sum is used as an operand by activation function f ( ). In the past, the activation function was a function such as a SIGMOID illustrated in FIG. 3. It is also possible to use many other well-known functions such as linear, spline and hyperbolic 10 tangent. It is also possible to use different activation functions for nodes in dirrelc;llt layers, or for nodes in the same layer. The present invention uses an activationfunction that discretizes its output so that it can be represented by a power of 2 or more speçific~lly by S2n where n is an integer and S is +1 or -1. FIG. 4 illustrates such an activation function. This function can be implemented by using a look-up15 table in software embo liment~ or by using a PROM in hardware embodiments. The figure shows that for each value of the operand or summation on axis 70, a givenvalue of X is produced as shown on axis 72. In this example, the values of X can be represented by S2n where n can have a value of 0, -1, -2 or -3. It is also possible to create functions where n can be any integer (. . . +3, +2, +1, 0, -1, -2, -3 . . .).
In reference to FIG. 2 the outputs X i from nodes of a previous layer are multiplied by weights Wji to form products using multiplier 52, 54, 56 and 58. By using the discretized activation function of FIG. 4, the outputs Xi from each node can be represented by a sign bit and the value of n. As a result, the multiplications carried out by multipliers 52,54, 56 and 58 may be executed by simply making the25 appl~liate change to the sign bit of weight Wj i and then shifting the weight by the number of locations that are specified by n. For example, if the summation produced at the input to prior node 18 is equal to -0.4, the output of node 18 that results from using -0.4 as an operand of the activation function of FIG. 4 is -0.5. As a result, the output of node 18 is r~lesented as - 2- 1. When this output is used as an input to 30 multiplier 52, weight Wj i= 1 is simply shifted one location to the right and the sign bit is changed to produce the result X i= l Wj,i= 1. This enables multiplier 52 and the other multipliers used in the connections within the network to be replaced by a simple shift register that shifts weight Wj i n times to the left or to the right. When n ,., .~
is negative, the weight is shifted to the right and when n is positive, the weight is shifted to the left. The sign bits of Wj i and Xi can be passed through an EXCLUSIVE-OR gate to produce the sign bit of the product.
Inputs to the first layer of the network may also be discretized in 5 accordance with FIG. 4. In netwolk configurations where the network input is multiplied by a weight, representing the input as a power of 2 permits implementing these initial multiplications with shift operations.
FIG. 5 illustrates a simplified multiplier. A simple shift register and EXCLUSIVE-OR gate produce the product Xi Wj i without the use of a 10 conventional multiplier. The value of n controls the number of shifts and thedirection of the shifts. Using a shift register and simple logic gates to replace multipliers throughout the network results in a haldwa.~ and/or software reduction which reduces the colllpu~a~ional overload associated with the network. When a network is implemented in software, replacing a multiplication with a shift operation 15 decreases computation time and thereby decreases the network's response time.In an effort to accommodate a wide variation in the values of the weights, it is possible to format the weights in the form of a mantissa and exponent;
however, the weights may be represented in other forms such as a fixed point representation. When operating using a m~nti~S~ and exponent, the shift register of 20 FIG. 5 can be replaced with an adder that is used to sum the exponent of the weight and n to obtain the product's exponent. In our prior example, where X i is equal to -0.5 and is expressed as _ 2- l where n = -1 and s = -1, the exponent of the weight is summed with -1 and sign bit of the weight is EXCLUSIVE-OR with the sign bit -1.
It should be noted that the node receiving the resulting products in the form of a 25 m~nti~S~ and exponent should perform the applupliate shifts to align the mantissas before a ~".,~"~tion is formed.
The second mode of operation for learning or neural networks is the backward propagation mode. This mode is used to train or teach the network by adjusting the weights used in the network. The process involves providing a set of 30 training inputs to the network and determining the error at the network's outputs.
The errors are used to adjust the values of the weights. This process is continued using many dirrelc;nt training inputs until the network produces outputs with acceptable errors. After allowing the network to produce a set of outputs in response to a set of training inputs, backward propagation begins by detelmilling the error at 35 each output of the network. The following equation provides an expression for output error E.
E = ~, ej ej = 2 [Xj - Tj~ (2) Equation 2 shows that the error ej at output "j" is represented by the difference between an actual output Xj and desired output Tj. Substituting equation 1 into 5 equation 2 results in, equation 3 which provides another expression for error ej.
~ 2 ej = 2 f ~ Xi Wji -Tj = 2 [f(Sj) - T,] (3) whereSj = ~, Xi Wj,i-As a result, the output error E may be represented by equation 4.
E = ~ 1/2 [f(Sj) - T~] (4) 10 The unit gradient aaS for unit j of the last layer is given by equation 5.
Gj = aS = (f(Sj) --Tj)f'(Sj) = Hj f'(Sj) (5) where the error term Hj = f(S j - Tj ) Once the unit gradients of one layer are known, the unit gradients of previous layers may be calculated using the chain rule. For example, equation 6 15 illustrates calculating unit gradients Gi for unit i of the layer preceding layer j.
Note that Xi = f ~ Xk Wik = f(Si) and that Sj = ~ Xi Wji = ~, f(Si) Wji As a result, Gi = aaSi = ~ aaS aSj = ~, Gj Wji f'(Si) = Hi f'~Si~ (6) In reference to equations 5 and 6, it can be seen that a gradient G for a 5 layer may be com~u~ed using an error term H and a derivative term f' ( ). For the output layer or last layer, the error term H is expressed by Hj = f(Sj) - Tj and for previous layers, the error term H is expressed in the form Hi = ~ Gj Wji ( ) j 10 where layer i precedes layer j, and layer j is closer to the output layer than layer i.
The weights W, are adjusted using the gradient of E with respect to the weight of interest. For example, the gradient of E with respect to weight Wji isgiven by equation 8.
aE = aE aaSj = Gj Xi (8) The weight adjustment is carried out by descending the gradient of E
with respect to Wji as shown in equation 9.
WjnieW = WJ~ild -- LR aaE = Wj~ild -- [LR Gj Xl] (9) Similarly, the weights in the layer i, which precedes layer j, are adjusting using aE = aE aaSi =GiXk (10) and WinkeW = Wi~kd _ LR a W = Wikd -- [ LR G i X k] ( 1 1 ) 21327~6 LR is the learning rate and typically has a small value such as 0.001.
Many implementations vary the learning rate from unit to unit, and can vary the learning rate over time. It is also possible to limit LR to powers of 2, that is, LR = 2n where n is an integer. It is preferable to limit n to negative integers.When beginning backward propagation, the gradient G j of the output layer is first computed according to equation 5. The term Hj, which is equal to (F (S j ) - Tj ), is discretized to the nearest power of two in accordance with the discretization function of FIG. 6. This function can be implemented using a look-up table in software embodiments and a PROM in haldw~ut; embodiments. In FIG. 6, 10 every value on the horizontal axis is mapped to a power of two on the vertical axis.
After discretization, Hj can be stored in the form of S 2n where n is an integer and S
is -1 or +1.
Equation 5 also requires forming the product Hj f' ( S j ). For the purpose of calculating f' ( ) (the derivative of the activation), f( ), (the activation 15 function of FM. 4) is appr~ ated by the function of FIG. 7. As can be seen inFIG. 8, the value of f' ( ) is either 0 or 1. (The value 0 could also be a small power of 2 such as 0.125 to f~cilit~te learning). When other activation functions are used, it is also possible to represent f' ( ) as S 2n when n is an integer and S is _ 1.
MultiplyingHj byf'(Sj)iscarriedoutbydeLel~ gifS; isbetween-l and 1. If 20 S j is between -1 and 1, the product is equal to Hj, and if S j is not between -1 and 1, the product is 0 (or a small power of 2 times Hj). If a small power of 2 such as 2-n is used instead of 0, Hj is shifted to the right by n bits. This completes the colllpu~alion of the gradient Gj . It should be noted that Gj is in the form of S2n, where n is an integer, because Hj is discretized in accordance with FIG. 6 and f' (S j ) 25 is either 0 or 1.
The weight update is made according to equation 9. It should be noted that Xi is in the form S2n where n is an integer and S is +l or -1. (Recall the activation function of FIG. 4). As a result, when beginning the calculation of equation 9 the product of gradient Gj and node output Xi is a simple addition of the 30 values of n associated with Gj and Xi. The sign bits of the product Gj Xi is formed by EXCLUSIVE-ORing the sign bits of Gj and Xi. It should be noted that since both Gj and Xi are in the form of S2n, the product Gj Xi is also in the form S2n.
As a result, multiplying learning rate LR by Gj Xi involves simply shifting LR
based on the value of n associated with the product Gj Xi and EXCLUSIVE-35 ORING the sign bits. As a result, the new value of Wji is obtained using simpleshift and add operations.
~.
g It should be noted that, by representing gradient Gj and nodal output X
as S2n, multiplication is no longer necessary to establish a new weight using equation 9. As mentioned earlier, elimin~ting multiplications permits faster operation and more complex networks.
Generally speaking, and in reference to FIG. 9, weights at preceding layers of the network are adjusted using error values that are propagated backward through the network. For illustrative purposes we will assume that layer n precedes n + 1 and layer n + 1 is closer to the network output than layer n. An error term H i for each node of layer n is calculated by ~",.~ -g the products of the unit gradients 10 Gj from layer n + 1 and their associated unadjusted weight values Wji from layer n + 1 as illustrated in equation 7. The error term Hi associated with each node of layer n is discretized to a power of 2, and is used to produce a unit gradient G i associated with each node of layer n in accordance with equation 6. The weights Wik of layer n are adjusted using the unit gradients Gi of layer n and the outputs Xk 15 of layer n- 1 as illustrated in equation 11. This procedure is repeated for preceding layers until all of the weights have been adjusted.
The nodes and multipliers may be implemented in software on a com~ule~, or in haldwal~e using analog or digital circuitry. In a hardware configuration, each node may be implem--nted using a microprocessor or a digital20 signal processing (DSP) chip. It is also possible to implement each node with an acc-lmlll~tor and a PROM where the acc~lm~ tor sums the inputs and the PROM
uses the sum to produce an output in the form of S 2n.
The weights can be stored in a memory such as a RAM or register. The weights can be updated by implementing the network on colll~u~er and then using a 25 set of training inputs to develop output errors that are back propagated to change the weights. It is also possible to program the microprocessors or DSP chips to use error information from subsequent layers and outputs from previous layers to adjust the weights.
A nodal or learning network may be used to recognize speech, 30 handwriting, pattern or objects. For example, the inputs to the first layer may be signals or values representative of an input such as an utterance or handwrittensymbol. Once trained using backward propagation, the outputs produced in response to these inputs may be used to identify the utterance or handwritten symbol.
Claims (10)
1. A computational network that produces a plurality of network outputs in response to a plurality of network inputs, characterized by:
a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero power of 2; and a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer output being used to produce at least one of a plurality of network outputs.
a plurality of first layer computational nodes forming a first layer of a computational network, each of said first layer computational nodes receiving at least one first layer input resulting from at least one of a plurality of computational network inputs to produce a first layer output, said first layer output being discretized to a nonzero power of 2; and a plurality of second layer computational nodes forming a second layer of the computational network, each of said plurality of second layer computational nodes receiving a plurality of second layer inputs to produce a second layer output, each of said plurality of second layer inputs being a product of a weight value and said first layer output produced by one of said plurality of first layer computational nodes, said second layer output being produced by using a sum of said plurality of inputs as an operand of a first activation function, said second layer output being used to produce at least one of a plurality of network outputs.
2. The computational network of claim 1, characterized in that said first layer output is in the form of S2n where n is a nonzero integer and S is ~1.
3. The computational network of claim 2, further characterized by product means for producing said product by shifting said weight value n times.
4. The computational network of claim 2, further characterized by product means for producing said product by summing n and an exponent portion of said weight value.
5. The computational network of claim 1, characterized in that said plurality ofnetwork inputs are discretized to a power of 2.
6. The computational network of claim 1, characterized in that each of said first layer computational nodes produces said first layer output by using a sum of a plurality of first layer inputs as an operand of a second activation function.
7. The computational network of claim 6, characterized in that said first layer output is in the form of S2n where n is a nonzero integer and S is ~1.
8. The computational network of claim 7, further characterized by product means for producing said product by shifting said weight value n times.
9. The computational network of claim 7, further characterized by product means for producing said product by summing n and an exponent portion ofsaid weight value.
10. The computational network of claim 6, characterized in that said plurality of network inputs are discretized to a power of 2.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US150,677 | 1993-11-09 | ||
| US08/150,677 US5473730A (en) | 1993-11-09 | 1993-11-09 | High efficiency learning network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2132756A1 CA2132756A1 (en) | 1995-05-10 |
| CA2132756C true CA2132756C (en) | 1999-05-18 |
Family
ID=22535542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002132756A Expired - Fee Related CA2132756C (en) | 1993-11-09 | 1994-09-23 | High efficiency learning network |
Country Status (6)
| Country | Link |
|---|---|
| US (1) | US5473730A (en) |
| EP (1) | EP0652525B1 (en) |
| JP (1) | JPH07191950A (en) |
| CA (1) | CA2132756C (en) |
| DE (1) | DE69424372T2 (en) |
| TW (1) | TW270186B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5630024A (en) * | 1994-01-19 | 1997-05-13 | Nippon Telegraph And Telephone Corporation | Method and apparatus for processing using neural network with reduced calculation amount |
| US6141044A (en) * | 1996-09-26 | 2000-10-31 | Apple Computer, Inc. | Method and system for coherent image group maintenance in memory |
| US5883716A (en) * | 1997-07-15 | 1999-03-16 | Litton Systems, Inc. | Rate control loop for fiber optic gyroscope |
| US7362892B2 (en) | 2003-07-02 | 2008-04-22 | Lockheed Martin Corporation | Self-optimizing classifier |
| US7814038B1 (en) | 2007-12-06 | 2010-10-12 | Dominic John Repici | Feedback-tolerant method and device producing weight-adjustment factors for pre-synaptic neurons in artificial neural networks |
| US8583577B2 (en) * | 2011-05-25 | 2013-11-12 | Qualcomm Incorporated | Method and apparatus for unsupervised training of input synapses of primary visual cortex simple cells and other neural circuits |
| JP6567381B2 (en) * | 2015-09-30 | 2019-08-28 | 株式会社東芝 | Arithmetic apparatus, method and program |
| EP3557484B1 (en) * | 2016-12-14 | 2021-11-17 | Shanghai Cambricon Information Technology Co., Ltd | Neural network convolution operation device and method |
| JP7029321B2 (en) | 2017-04-20 | 2022-03-03 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | Information processing methods, information processing equipment and programs |
| US11256978B2 (en) * | 2017-07-14 | 2022-02-22 | Intel Corporation | Hyperbolic functions for machine learning acceleration |
| CN110036384B (en) * | 2017-09-29 | 2021-01-05 | 索尼公司 | Information processing apparatus, information processing method, and computer program |
| US11960565B2 (en) | 2018-03-02 | 2024-04-16 | Nec Corporation | Add-mulitply-add convolution computation for a convolutional neural network |
| CN113554163B (en) * | 2021-07-27 | 2024-03-29 | 深圳思谋信息科技有限公司 | Convolutional neural network accelerator |
Family Cites Families (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2625347B1 (en) * | 1987-12-23 | 1990-05-04 | Labo Electronique Physique | NEURON NETWORK STRUCTURE AND CIRCUIT AND ARRANGEMENT OF NEURON NETWORKS |
| EP0400147B1 (en) * | 1988-08-31 | 1997-10-15 | Fujitsu Limited | Neurocomputer |
| US4972363A (en) * | 1989-02-01 | 1990-11-20 | The Boeing Company | Neural network using stochastic processing |
| US5222195A (en) * | 1989-05-17 | 1993-06-22 | United States Of America | Dynamically stable associative learning neural system with one fixed weight |
| US5138924A (en) * | 1989-08-10 | 1992-08-18 | Yamaha Corporation | Electronic musical instrument utilizing a neural network |
| US5109351A (en) * | 1989-08-21 | 1992-04-28 | Texas Instruments Incorporated | Learning device and method |
| GB2236608B (en) * | 1989-10-06 | 1993-08-18 | British Telecomm | Digital neural networks |
| US5067164A (en) * | 1989-11-30 | 1991-11-19 | At&T Bell Laboratories | Hierarchical constrained automatic learning neural network for character recognition |
| JPH0782481B2 (en) * | 1989-12-26 | 1995-09-06 | 三菱電機株式会社 | Semiconductor neural network |
| US5255346A (en) * | 1989-12-28 | 1993-10-19 | U S West Advanced Technologies, Inc. | Method and apparatus for design of a vector quantizer |
| US5167006A (en) * | 1989-12-29 | 1992-11-24 | Ricoh Company, Ltd. | Neuron unit, neural network and signal processing method |
| US5058179A (en) * | 1990-01-31 | 1991-10-15 | At&T Bell Laboratories | Hierarchical constrained automatic learning network for character recognition |
| JP3110434B2 (en) * | 1990-03-17 | 2000-11-20 | 科学技術振興事業団 | Analog backpropagation learning circuit |
| US5337395A (en) * | 1991-04-08 | 1994-08-09 | International Business Machines Corporation | SPIN: a sequential pipeline neurocomputer |
| DE69130656T2 (en) * | 1990-06-14 | 1999-06-17 | Canon Kk | Neural networks |
| US5097141A (en) * | 1990-12-12 | 1992-03-17 | Motorola, Inc. | Simple distance neuron |
| US5228113A (en) * | 1991-06-17 | 1993-07-13 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Accelerated training apparatus for back propagation networks |
-
1993
- 1993-11-09 US US08/150,677 patent/US5473730A/en not_active Expired - Fee Related
-
1994
- 1994-09-23 CA CA002132756A patent/CA2132756C/en not_active Expired - Fee Related
- 1994-11-02 EP EP94308066A patent/EP0652525B1/en not_active Expired - Lifetime
- 1994-11-02 DE DE69424372T patent/DE69424372T2/en not_active Expired - Fee Related
- 1994-11-07 TW TW083110291A patent/TW270186B/zh active
- 1994-11-09 JP JP6299020A patent/JPH07191950A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| JPH07191950A (en) | 1995-07-28 |
| TW270186B (en) | 1996-02-11 |
| DE69424372D1 (en) | 2000-06-15 |
| DE69424372T2 (en) | 2000-09-21 |
| EP0652525A2 (en) | 1995-05-10 |
| CA2132756A1 (en) | 1995-05-10 |
| US5473730A (en) | 1995-12-05 |
| EP0652525B1 (en) | 2000-05-10 |
| EP0652525A3 (en) | 1995-12-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2132756C (en) | High efficiency learning network | |
| Robinson et al. | Static and dynamic error propagation networks with application to speech coding | |
| US5262976A (en) | Plural-bit recoding multiplier | |
| US5280439A (en) | Apparatus for determining booth recoder input control signals | |
| WO1994022074A1 (en) | Fuzzy logic design generator using a neural network to generate fuzzy logic rules and membership functions for use in intelligent systems | |
| US5257218A (en) | Parallel carry and carry propagation generator apparatus for use with carry-look-ahead adders | |
| Herault et al. | Boundary and interface conditions meshless methods [for EM field analysis] | |
| US5535309A (en) | Single layer neural network circuit for performing linearly separable and non-linearly separable logical operations | |
| US5845051A (en) | Learning method for multilayer perceptron neural network with N-bit data representation | |
| US4495593A (en) | Multiple bit encoding technique for combinational multipliers | |
| US4912653A (en) | Trainable neural network | |
| EP0385637A2 (en) | Learning machine with multi-input single output circuits connected in hierarchical structure | |
| Danysh et al. | A recursive fast multiplier | |
| Hikawa | Implementation of simplified multilayer neural networks with on-chip learning | |
| EP0733982B1 (en) | Method of speeding the execution of neural networks for correlated signal processing | |
| EP0428942A2 (en) | Plural-bit recoding multiplier | |
| JP2760170B2 (en) | Learning machine | |
| US4891782A (en) | Parallel neural network for a full binary adder | |
| US6061673A (en) | Learning methods in binary systems | |
| US5227678A (en) | Fast digital comparison circuit for fuzzy logic operations | |
| EP0533193A2 (en) | Neural network circuit | |
| GB2236608A (en) | Digital neural networks | |
| JPH10111791A (en) | Division device | |
| EP0280216B1 (en) | A pattern recognition apparatus using a composite similarity method | |
| CN113065648B (en) | Hardware implementation method of piecewise linear function with low hardware cost |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed |