WO1996014616A1 - Concurrent learning and performance information processing system - Google Patents
Concurrent learning and performance information processing system Download PDFInfo
- Publication number
- WO1996014616A1 WO1996014616A1 PCT/US1995/014160 US9514160W WO9614616A1 WO 1996014616 A1 WO1996014616 A1 WO 1996014616A1 US 9514160 W US9514160 W US 9514160W WO 9614616 A1 WO9614616 A1 WO 9614616A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- processor
- values
- input
- vector
- elements
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
- G06F18/2113—Selection of the most significant subset of features by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
Definitions
- the present invention relates to the field of parallel processing neurocomputing systems and more particularly to real-time parallel processing in which learning and performance occur during a sequence of measurement trials.
- Conventional statistics software and conventional neural network software identify input-output relationships during a training phase, and each apply the learned input-output relationships during a performance phase. For example, during the training phase a neural network adjusts connection weights until known target output values are produced from known input values. During the performance phase, the neural network uses connection weights identified during the training phase to impute unknown output values from known input values.
- a conventional neural network consists of simple interconnected processing elements. The basic operation of each processing element is the transformation of its input signals to a useful output signal. Each interconnection transmits signals from one element to another element, with a relative effect on the output signal that depends on the weight for the particular interconnection.
- a conventional neural network may be trained by providing known input values and output values to the network, which causes the interconnection weights to be changed.
- Multi-layer perceptrons have two or more processing element layers, most commonly an input layer, a single hidden layer and an output layer.
- the hidden layer contains processing elements that enable conventional neural networks to identify nonlinear input-output relationships.
- Conventional neural network learning and performing operations can be performed quickly during each respective stage, because neural network processing elements can perform in parallel.
- Conventional neural network accuracy depends on data predictability and network structure that are prespecified by the user, including the number of layers and the number of processing elements in each layer.
- Conventional neural network learning occurs when a set of training records is imposed on the network, with each such record containing fixed input and output values. The network uses each record to update the network's learning by first computing network outputs as a function of the record inputs along with connection weights and other parameters that have been learned up to that point.
- Conventional neural network training and performance phases differ in two basic ways. While weight values change during training to decrease errors between training and computed outputs, weight values are fixed during the performance phase. Additionally, output values are known during the training phase, but output values can only be predicted during the performance phase. The predicted output values are a function of performance phase input values and connection weight values that were learned during the training phase.
- the present invention provides a data analysis system that receives measured input values for variables during a time trial and (learns) relationships among the variables gradually by improving learned relationships from trial to trial. Additionally, if any input values are missing, the present invention provides, during the time trial, an expected (imputed) output values for the missing value that are based on the prior learned relationships among the analyzed variables.
- the present invention provides the imputed values by implementing a mathematical regression analysis feature values that are predetermined functions of the input values.
- the regression analysis is performed by utilizing a matrix of connection weights to predict each feature value as a weighted sum of other feature values.
- Connection weight elements are updated during each trial to reflect new connection weight information from trial input measurements.
- a component learning weight is also utilized during each trial that determines the amount of impact that 1 the input measurement vector has on learning relative to prior vectors received.
- the present invention may process the input values in parallel or process the values sequentially.
- the different input values may be provided in the form of vectors. Each of the values of the input feature vector is operated on individually with respect to prior learned parameters.
- a plurality of processors process the input values, with each processor dedicated to receive a specific input value from the vector. That is, if the system is set up to receive sixteen input feature values (i.e., corresponding to a vector of length sixteen), sixteen processing units are used to process each of the input feature values. In the sequential embodiment, one processor is provided to successively process each of the input feature values.
- each of the processing units is operative to receive, during a time trial, individual input values from an input vector.
- a plurality of conductors connect each of the processing units to every other processing unit of the system.
- the conductors transfer weighted values among each of the processor unit according to processes of the present invention.
- Each of the processing units provide, during said time trial, an imputed output value based upon the weighted values.
- each of the processing units is operative to update connection weights for computing the weighted values based on the input values received.
- the present invention provides a plurality of switching junctions located along the conductor interconnecting to alleviate the problem associated with a single processor communicating with many others.
- the switching junctions are operable for uniquely pairing each of the processors to every other processor of the system.
- the present invention further provides memory elements that are coupled to the switching junctions.
- Each of the memory elements are individually coupled to a separate switching junction and each of the memory elements contains a connection weight value.
- the connection weight memory elements, located at the switching junctions are the connection weight elements of the matrix used in computing an output value.
- the switching junctions may be operative to selectively connect each of the processors to only one other processor at a time, thereby forming multiple paired sets of the processors for communicating the weight values during the time interval.
- the switching junctions successively connect different sets of said multiple paired sets of the processors during multiple time intervals.
- the switching junctions are preferably operative to connect the different sets of multiple paired processors in all possible combinations in the minimum number of steps.
- a control unit is operative to provide switching signals to the switching junctions in order to control the transfer of weighted values among the processors.
- the conductors though which processor communication occurs preferably are provided in a first conductor layer and a second conductor layer, with the first and second conductor layers operable for a connection at the switching junctions.
- the present invention is also be implemented in a sequential manner in which a conventional computer processing unit may be used along with a conventional computer memory unit to process input values.
- the sequential embodiment of the present invention similarly computes output values from input values received during a time trial.
- the processing unit is operative to sequentially receive input values from an input vector.
- the elements of the connections weight matrix are stored in sequential order as a data string in a memory unit.
- the processing unit of the sequential system is operative to provide, during the time trial, an imputed output value based on the elements of the connection weight matrix and is operative to update the element of the connection weight matrix during the time trial.
- the sequential system quickly operates on each element of the connection weight matrix in ic a specially designed sequence. In conventional systems because matrix multiplication operations are generally nested access loops (one for the rows and one for the columns) concurrent operations are slower than the sequential embodiment method of the present invention.
- the present invention also provides a system for updating a connection weight matrix during early trial.
- a processing unit operative to receive values from an input feature vector during a time trial and a memory unit that contains connection weight elements that identify a relationship among feature variables.
- the processing unit is operative to update the connection weight elements based on non-missing values of the input vector received.
- the processing unit of the present invention is operative to update the connection weight elements based on a component learning weight that is a distinct learning weight for each input vector received. By using the component learning weight, accurate relationships among feature variables may be determined.
- output values and learned values may be evaluated and controlled by controller units within the information processing system.
- a learning weight controller may be provided that automatically adjusts the learning weight from trial to trial in a manner that generally regulates the relative effect that each input vector has on prior learning.
- a user may interface with the system to provide desired learning weights different than the learning weights that may be automatically provided by the system.
- the present invention may provide a feature function controller that is operative to convert measurement values initially received by the system to input feature vectors for imputing and learning use by the system.
- the feature function controller is also operative to either provide default initial connection weights or receive connection weight elements externally so that a user of the system may supply initial weights as desired.
- the learning weight controller may disable the learning function of the computer system if an abnormal deviation of input values occur.
- the feature function controller is operative to create a variety of statistics such as the first-order difference between a current measurement value and the corresponding measurement values stored from previous trials to identify a sudden change in measurement values. A sudden change in input values may indicate that an instrument from which the input values are received is faulty.
- the processes of the present invention include: receiving, at a processing unit, an input vector [IN](f) during a time trial; computing, during the time trial, an output value from a missing input value of the input vector based on connection weight elements; and updating, during the time trial, the connection weight elements based on input values of the input vector.
- the processes of the present invention may further include the step of updating the connection weight elements based on the component learning weight element discussed above.
- the learning weight element may be calculated by: receiving a global learning weight /; receiving a learning history parameter ⁇ that is an indicator of the prior learning weight of each input vector; and receiving a viability vector v(f), that indicates the extent to which an input feature vector is missing; and multiplying those values together to obtain the learning component weight (i.e., /(C) (f) - I v (j) 1(f)).
- the present invention is enabled to quickly update the connection weight matrix during the same trial in which the system imputes a value by utilizing as part of the connection weight updating process a mean vector ⁇ [OUT], of all feature vectors received.
- the prior mean vector ⁇ [IN] equals ⁇ [OUT] from the previous measurement trial. If the process is in the first trial, then ⁇ [IN] may equal a system default value, preferably the value 1.0 for a user-supplied value.
- ⁇ [OUT] calculated by the following process equation:
- ⁇ [OUT]( ) ( l[C)(f) [IN](f) + ⁇ [IN]( ) ) / ( 1 + l[C](f) ).
- Processes of the present invention also include updating the connection weight elements utilizing an intermediate imputed vector, e[TN].
- Elements of the e[JN] vector may be calculated by the following equation:
- e[TN](f) v(f) ( m[lN](f) - ⁇ [IN]( ) ) / ( 1 + l[C](f) ).
- connection weight matrix may be updated utilizing the following process equations:
- ⁇ [IN] ⁇ [OUT] from the previous trial. If the current trial is the first trial, then ⁇ [IN] may equal a system default value, preferably equals the identity matrix ⁇ or a user- supplied value.
- m[O T](f) ⁇ [W] (f) + e[IN] (f)( 2 - v(f) ) + x(f)( v(f) - 1 ) / ⁇ (f ).
- the present invention provides a method of accessing multiple pairs of processors for computing the x vector.
- the process includes accessing multiple sets of uniquely paired processors during a time interval; retrieving each of the connection weight elements, located at the switching junctions that connect the paired processor units; and transferring e[IN](/) located in each processor to the other processor connected at the switching junction; then computing a running sum of e[ ⁇ N] ⁇ [IN] until all processor pairs of the system have computed their corresponding values for x.
- the processes of the present invention also provide a method of accessing each set of processors for updating connection weight elements.
- the process includes accessing multiple sets of uniquely paired processors during a time interval; retrieving, by one of the processors located at the switching junction, the connection weight element located at the switching junction; updating the connection weight element by the processor that retrieved the connection weight element; and transferring the updated connection weight element back to the memory element of the switching junction.
- Figure 2 is a block diagram that illustrates a parallel processor embodiment of the preferred embodiment of the present invention.
- Figure 3 is a block diagram that illustrates a sequential computer embodiment of the preferred embodiment of the present invention.
- Figure 4 shows an array of pixel values that may be operated on by the preferred embodiment of the present invention.
- Figure 5 shows a circuit layout for the joint access memory and processors used in the parallel embodiment of the present invention.
- Figure 6a shows switching detail for a node in the joint access memory of the preferred embodiment of the present invention. / /
- Figure 6b shows a side view of switching detail for a node in the joint access memory of the preferred embodiment of the present invention.
- Figure 7 shows timing diagrams for joint access memory control during intermediate matrix/vector operations of the parallel embodiment .
- Figure 8 shows timing diagrams for joint access memory control timing for updating operations associated with a switching junction of the joint access memory of the preferred embodiment of the present invention.
- Figure 9 shows processing time interval coordination for parallel embodiment of the preferred embodiment of the present invention.
- Figure 10 shows a block diagram of the overall system implemented in the parallel embodiment of the preferred embodiment of the present invention.
- Figure 11 shows a block diagram of the overall system implemented in the sequential embodiment of the preferred embodiment of the present invention.
- Figure 12 shows communication connections for a controller used in the parallel embodiment of the preferred embodiment of the present invention.
- Figure 13 shows communication connections for another controller used in the parallel embodiment of the preferred embodiment of the present invention.
- FIGS 14 through 22 are flow diagrams showing preferred steps for the processes implemented by the preferred embodiment of the present invention.
- a concurrent learning and performance information processing (CIP) neurocomputing system made according to the preferred embodiment of the present invention is shown.
- a CIP system 10 is implemented with a computer 12 connected to a display monitor 14.
- the computer 12 of the CIP system 10 receives data for evaluation from a data acquisition device (DAD) 15, which may provide multiple measurement values at time points via a connection line 16.
- Data acquisition devices such as data acquisition computer boards and related software are commercially available from companies such as National Instruments Corporation.
- the computer 12 may also receive input data and/or operation specifications from a conventional keypad 17 via an input line 18.
- Receiving and responding to a set of input measurement values at a time point is referred to herein as a trial, and the set of input values is referred to herein as a measurement record.
- the system determines (learns) the relationships that exist among the measurements received during the trials. If some measurement valves are missing during the trial, the CIP system 10 provides imputed values that would be expected based on the prior learned relationships among prior measurements along with the non- missing current measurement values.
- the CIP system 10 receives a measurement record from the data acquisition device 15 and converts the measurement values to feature values.
- the conversation of measurement values to feature values operates to reduce the number of learned parameters that are needed for learning or imputing.
- the feature values and other values calculated from the feature values provide useful data for predicting or imputing values when certain measurement values are missing or for determining that a monitored measurement value of a system has abnormally deviated from prior measurement values.
- the CIP system 10 Upon receiving an input measurement record at the beginning of a trial, the CIP system 10 performs the following operations as quickly as each input record arrives (i.e. , system 10 performs concurrently): deriving input feature values from incoming measurement values (concurrent data reduction); identifying unusual input feature values or trends (concurrent monitoring); estimating (i.e. , imputing) missing feature values (concurrent decision-making); and updating learned feature and means, learned feature variances and learned interconnection weights between features (concurrent learning).
- the CIP system is useful in many applications, such as continuous and adaptive: (a) instrument monitoring in a chemically or radioactively hostile environment; (b) on-board satellite measurement monitoring; (c) missile tracking during unexpected excursions; (d) in- patient treatment monitoring; and (e) monitoring as well as forecasting competitor pricing tactics.
- high speed is less critical than in others.
- the CIP system has provision for either embodiment on conventional (i.e. , sequential) computers or embodiment on faster parallel hardware. Although speed is not a major concern in some applications, CIP high speed is an advantage for broad utility.
- Sequential CIP embodiment is faster than conventional statistics counterparts for two reasons: first, the CIP system uses concurrent updating instead of off-line training; second, the CIP system updates the inverse of a certain covariance matrix directly, instead of the conventional statistics practice of computing the covariance matrix first and then inverting the covariance matrix. Concurrent matrix inverse updating allows for fast CIP implementation. When implemented using a sequential process, CIP response time increases as the square of the number of data features utilized increases.
- CIP response time increases only as the number of features utilized increases.
- a processor is provided for each feature.
- parallel CIP response time is faster than sequential CIP response time by a factor of the number of features utilized.
- the CIP subsystems include a system bus 19, a transducer 20, a kernel 21 and a manager 22.
- the transducer 20 and the kernel 21 operate successively in order to accomplish the various concurrent operations described above.
- Input measurement values are first converted to input feature values by the transducer 20.
- the input features are then processed by the kernel 21 to produce imputed (i.e., output) features, updated learned parameters and monitoring statistics.
- Output feature values are then converted to imputed (i.e. / Y output) measurement values by the transducer 20.
- the manager 22 coordinates transducer 20 and kernel 21 concurrent operations and occasionally refines system operation.
- the basic components of the transducer 20 are an input processor 24 that has a recent feature memory (RFM) 25 and an output processor 26.
- the input processor 24 and output processor 26 are each controlled by input and output control units 27 and 28, respectively.
- the recent feature memory 25 stores a preselected number of input feature values [IN], obtained from prior trials (all vectors in this document are row vectors).
- the stored recent features may be utilized in conjunction with input measurements ./[IN] to calculate, as discussed below, concurrent input feature values [IN] for the current trial.
- the input processor 24 receives an input measurement vector j[TN] and a corresponding plausibility vector p. Plausibility vector elements identify input measurement vector elements as non-missing or missing.
- the input processor 24 then (a) converts the input vector /[IN] to some input features and combines those converted input features with other converted features in the recent feature memory
- the output processor 26 receives an output feature vector m[OUT]. The output processor 26 converts the output feature vector [OUT] to a corresponding output measurement vectoryTOUT].
- the input measurement vector ./[IN] received by the transducer input processor 24 contains the values of the input measurement vectors from the DAD 15 (shown in Figure 1). Plausibility values, provided either externally by the DAD 15 or internally by the manager 22, indicate if a measurement plausibility value is 0 (missing), 1 (non-missing) or some intermediate value (a combination of missing and non-missing quantum values as discussed below). If an element of ./[IN] is missing, as determined by the corresponding element of p being 0, then the corresponding element of ./[OUT] is imputed, based on non-missing elements of ./[IN] and/or previously learned information.
- the imputing process utilizes measurement to feature conversion within the transducer input processor 24, followed by missing feature value imputing within the kernel, followed by imputed feature to imputed measurement conversion within the transducer output processor 26.
- the transducer input processor 24 computes feature values and viability values according to functions that are determined by the manager 22.
- Each feature element in m[IN] is a function of the measurement elements in ./[IN], and each viability element of v is a corresponding function of the plausibility elements in p.
- Each feature viability value is the product among the plausibility vaiues for the measurements that are independent variables in the feature function.
- Feature viability elements are computed as products of corresponding measurement plausibility elements. Every CIP system input measurement value is treated as an average of non-missing quantum measurement values from a larger set, some of which may be missing. The corresponding plausibility value of each input measurement is further treated as the proportion of component quanta that are non-missing within the larger set.
- the kemel 21 includes: a processor 311 for feature 1 through processor 31 p for feature p ; a kemel control module 32; and a joint access memory (JAM) 23 connected by buses 45 through 45 f , to the processors 31 through 31 r.
- JAM joint access memory
- Outputs from the kernel 21 include imputed feature values in mfOUT], feature function monitoring statistics that are sent to the manager via connections 411 , through 41/r and 41JAM, an ⁇ ⁇ feature value monitoring statistics that are sent to the manager via connections 40] through 40 p.
- the kernel processors 311 through 31 r use preferably arithmetic logic units (ALUs) that implement basic arithmetic functions, in order to reduce the cost and size of the processors 31 j through 31 r.
- ALUs arithmetic logic units
- basic processors as such may be designed using commercially available chip design software packages, such as Mentor Graphics®, a product of the Mentor Graphics Co ⁇ oration.
- Kernel processors 31 ⁇ through 31/r operate to: impute missing feature values based on non-missing elements of m[IN] and/or previously learned kemel 21 parameters; update leamed parameters that reside in each processor and in the joint access memory 23; and produce monitoring statistics for use by the manager 22.
- the kernel processors 311 through 31/r utilize two steps of inter-processor communication to transfer relevant values from each processor to every other processor.
- Kemel processor operations also compute a distance measure d in the kemel distance ALU 34. Communication between the distance ALU 34 and each kernel processor occur through connections 351 through 35p.
- the kemel input learning weight / is a non-negative number that — like input plausibility values and viability values — is a quantum/probabilistic measure.
- the learning weight / for each trial is treated by the CIP system as a ratio of quantum counts, the numerator of which is the number of quantum measurement vectors for the concurrent trial, and the denominator of which is the total of all quantum measurements that have been used in prior learning.
- the concurrent input feature vector [IN] has a high learning weight / value
- the input feature vector will have a larger impact on learned parameter updating than if the input feature vector has a lower learning weight / value, because the input feature vector [IN] will contain a higher proportion of the resulting plausible quantum measurement total.
- the learning weight / is supplied as an input variable during each trial, but the learning weight can also be generated optionally by the CIP system manager 32 as discussed below.
- Kernel imputing, memory updating and monitoring operations are based on a statistical regression framework for predicting missing features as additive functions of non-missing feature values.
- the weights for imputing each missing feature value from all others are well-known.
- Formulation for the weights used for imputing are functions of sample covariance matrix inverses.
- the F by F covariance matrix is computed first v based on a training sample, followed by inverting the covariance matrix and then computing regression weights as functions of the inverse.
- the conventional approach involves storing and operating with a training set that includes all measurements received up to the current input trial. Storing all prior measurements is typical for conventional systems, because all prior measurements are needed in order to first calculate present covariances from which the inverse matrix may be obtained.
- CIP kemel 21 operation does updates the inverse of v directly, based only on: (a) the inverse of v and other parameters that have been leamed up to that trial; (b) incoming feature values m [IN]; and (c) the input learning weight /. Consequently, CIP operations can keep up with rapidly arriving information, without the need for either storing and operating with a training data set or inverting a covariance matrix.
- the process of updating the inverse elements of v is the
- CIP counterpart to conventional learning.
- CIP fast updating capability from trial to trial provides a statistically sound and fast improvement to conventional learning from off-line training data.
- the CIP System 10 provides an enhancement over the prior art.
- the joint access memory 23 contains the feature interconnection weights, one for each of the possible F x (F -l)/2 pairs of features.
- the feature connection It weights correspond to the lower triangular elements of v inverse.
- the main diagonal elements of v inverse are also used during kemel imputing and feature function monitoring, and are modified during concurrent learning. Individual elements of the v inverse main diagonal reside in their corresponding kemel processor.
- the kemel sends the imputed feature vector m[OUT] back to the transducer output processor 24 via line 33, where the imputed feature values in m[OUT] are converted to imputed measurement values in ./[OUT] for system output by the transducer output processor 26.
- the output processor 26 converts imputed features to imputed measurements by excluding all but the imputed measurement set from the imputed feature set.
- more elaborate conversion may be utilized.
- one CIP system feature alternative may convert a set of measurements to the average of the set of measurements during transducer input processing, in which case the transducer output processor 26 sets all imputed output measurement values to their common imputed average value.
- the outputs can be useful in several ways, including: (a) replacing direct measurement values, such as during periods when instruments break down; (b) predicting measurement values before the measurements occur, such as during econometric forecasting operations; and (c) predicting measurement values that may never occur, such as during potentially faulty product classification operations.
- the manager 22 monitors and controls CIP system operation.
- the subsystems of the manager 22 include: the coordinator 38, which provides the CIP system-user interface; the executive 39, which dictates overall system control; the learning weight controller 40, which provides / to the kemel 21 in place of externally supplied / values from the data acquisition device 15 ( Figure 1 ); and the feature function controller 41, which establishes and modifies measurement-feature function structure.
- the kemel and transducer 20 modules are active in a concurrent mode.
- the kernel 21 and transducer 20 operate continuously based on input measurement values, plausibility values and leaming weights, according to system control parameters that are set by the executive 48. These parameters include input measurements specification, feature computing specifications, inter ⁇ module buffering specifications and output measurement specifications.
- the CIP system produces imputed feature values, feature value monitoring statistics, updated learned kemel parameter values and imputed output measurement values.
- the CIP system may also perform feature value monitoring operations, which are performed by the kemel 21 , the leaming weight controller 40 and the coordinator 49.
- each kemel processor 311 through 31/r sends monitoring statistics via connections 401 through 40/r to the manager 22.
- Deviance monitoring statistics are used during each trial by the leaming weight controller 40 within the manager 22 to assess the extent that each feature is unexpected, relative to: (a) the mean value that has been computed from prior learning for that feature, and (b) the value of that feature that would be imputed if the feature was missing.
- Feature value statistics that are sent from each kemel processor include the observed value, a leamed mean, a regressed value and a leamed variance value for the feature of the processor.
- the learning weight controller 40 uses the feature value monitoring statistics to compute concurrent feature deviance measures. These deviance measures are then sent from the leaming weight controller 40 to the coordinator 38 to produce monitoring graphics, which are then sent through the system bus 19 to the monitor 14.
- the CIP manager 22 specifies feature function assessment and assignment occasionally, and the CIP manager 22 controls leaming weight assignment alternatively.
- Feature function assessment and assignment are performed by the feature function controller 41 within the manager 22, by simultaneously accessing the interconnection weights in the joint access memory 23 through a parallel port 41 JAM, along with other weights in processor 1 to processor F through connections 41 ] to 41/ .
- the feature function controller 41 first examines the interconnection weights to identify features that are either redundant or unnecessary, that is, features that do not provide information useful for leaming and imputing. Feature function controller 41 then commands the transducer input processor 24 to combine redundant features, remove unnecessary features or add new features accordingly, through control lines 43.
- CIP feature function monitoring and control operations are based on a statistical regression framework. For example, all of the necessary partial correlation coefficients and multiple correlation coefficients for identifying redundant or unnecessary features can be computed from the elements of v inverse that reside in the joint access memory 23, and architectures closely resembling the kemel 21 architecture can be used to perform such refinement operations. Although the refining operations are not performed as fast as concurrent kemel 21 operations, the refining operations can be performed almost as quickly and in concert with ongoing kemel 21 operations by using parallel refinement processors as discussed below.
- leaming weight schedules to be computed that will produce: (a) equal impact leaming, through which each input feature vector will have the same overall impact on parameter leaming; (b) conservative leaming, through which less recent input feature vectors will have higher overall impact on parameter leaming than more recent input feature vectors; and (c) liberal leaming, through which more recent input feature vectors will have lower overall impact.
- the leaming weight controller 40 is used to supply leaming weights to the CIP system in a basic form, the system is programmed to only supply equal impact leaming weights. In another form, the leaming weight controller 40 may use the CIP system monitoring statistics to identify unusual trends in imputing accuracy.
- the leaming weight controller 40 changes the leaming weight computing schedule to produce more liberal leaming, based on the assumption that imputing accuracy degradation is caused by a new set of circumstances that & / require previously leamed parameters to be given less impact.
- the leaming weight may also modify elements of the plausibility vector p if feature value monitoring indicates erratic measurement behavior.
- FIG. 3 a block diagram illustrating the CIP system 10 embodiment on a conventional computer with one central processing unit is shown.
- the basic components of the conventional sequential computer embodiment 12a of the sequential CIP system 11 include: a transducer input 24a process; a kemel process 21a; a transducer output process 26a; a coordinator 49a; an executive 39a; a leaming weight controller 40a; and a feature function controller 41a.
- Each of the components of the sequential CIP system 11 perform the same basic functions as the parallel CIP system 10.
- only one central processor is utilized.
- the time for processing input data takes more time to implement than in the parallel CIP computing system 12.
- the sequential system receives an input vector y * [IN] and a plausibility value p.
- the input vectors IN] are also converted to input feature values m[IN].
- Plausibility values p are converted to viability values v as discussed above.
- the kemel process 31a receives the feature input value [IN], the viability value, and a leaming weight / from the system.
- the kemel 21a process produces an output feature vector [OUT] based upon connection weights stored in conventional memory 301 that is allocated by the executive 39a.
- the output feature vector m[OUT] is transferred to the output transducer 26a for conversion to an output measurement value j[0 ⁇ )T] for external use, as discussed above in connection with the CIP system 10.
- the executive block 39a represents the sequential computer main function and other blocks represent CIP subroutines.
- Memory 301 for the kemel subroutine embodiment has conventional data array form as known to those skilled in the art, and all shared memory storage is allocated and maintained by the main executive function.
- the executive 39a first initializes the CIP system by calling the coordinator 39a subroutine, which in turn obtains user-supplied system specifications, such as the length of the measurement vector /[IN] and the number of feature functions, through the keyboard 18. The executive 39a then allocates leamed parameter memory and other storage accordingly.
- each trial for the sequential embodiment includes reading an input measurement vector ./[IN] and a plausibility vector p, followed by writing an imputed measurement vector ./[OUT].
- input-output operations utilize input files 15 and output files 17.
- the input files are data files read from a storage medium that receives input values from the DAD 15.
- the output files may be utilized outside the CIP system in any manner the user chooses.
- the sequential embodiment may utilize occasional refinement operations, as discussed above in connection with the parallel system.
- the executive 38a will interrupt concurrent operation occasionally, as specified by the user during initialization.
- the executive 39a will call the feature function controller 41a, which will receive the connection weight matrix as one of its inputs.
- the feature function controller 41a will then use the connection weight matrix to identify redundant and unnecessary features, after which it will return new feature specifications to the executive 38a accordingly.
- the executive 38a will then convey the new specifications to the transducer input subroutine 24a and the transducer output subroutine 26a, during future concurrent operations that follow.
- Each array has nine measurement variables, labeled as JC(1 , 1 ) through (3,3).
- the black squares may be represented by input binary values of 1
- the white squares may be represented by binary input values of
- the three arrays can thus be represented as CIP binary measurement vectors j[ ] having values of (1, 0, 0, 0, 1, 0, 0, 0, 1,),
- the CIP system uses each plausibility value in p to establish missing or non-missing roles of its corresponding measurement value in ./[IN].
- all nine p values corresponding to the Figure 4 measurements are 1 then all nine ./[IN] values will be used for leaming.
- two p elements are 0, indicating that the two corresponding ./[IN] values are missing, the two corresponding ./[OUT] values will be imputed from the other seven j[JN] values that have corresponding p values of 1.
- the output pattern will be either 70a or 70c, depending on previously leamed parameter values.
- the pattern that occurred most between the possible 70a and 70c patterns in previous trials number 1 through 100 would be imputed.
- all nine values of p were 1 for each such trial, and ./[IN] values corresponded to types 70a, 70b and 70c for 40, 19 and 41 such trials, respectively.
- the unknown upper right and lower right pixels during trial 101 will be imputed as white and black, respectively in If keeping with 70c, because the CIP system will have been taught to expect 70c slightly more often than the only other possibility 70a.
- Transducer Input Operation Input measurements can be termed arithmetic, binary and categorical. Arithmetic measurements such ⁇ s altitude, temperature and time of day have assigned values that can be used in an ordered way, through the arithmetic operations of addition, subtraction, multiplication and division. When a measurement has only two possible states, the measurement may be termed binary and may be generally represented by a value of either 1 or 0. Non-arithmetic measurements having more than two possible states may be termed categorical. CIP measurement vectors j may thus be grouped into arithmetic, binary and categorical sub-vectors ( [A], j[ ], j[C]), where each sub-vector can be of any length.
- the CIP system either (a) converts arithmetic and binary measurement values to feature values in the transducer input processor 24, or (b) sends the arithmetic or binary measurement values directly to the CIP kemel without transforming the measurement values.
- the CIP system converts categorical measurement values to equivalent binary feature values.
- the CIP system converts each categorical measurement value j[C] having one of c possible values to a binary feature vector m [C] having c - 1 elements, which also have C possible values. For example, if a categorical variable has possible values 1 , 2, 3 and 4, the resulting categorical feature vector has corresponding values (1, 0, 0), (0, 1, 0), (0, 0, 1) and (0, 0, 0).
- the transducer contains only features that are either arithmetic measurements in their original input form, binary measurements in their original input form or binary equivalents to categorical measurements. All of these can be treated as arithmetic features and sent to the kemel directly, or they can be optionally converted to other arithmetic features by the input processor 24.
- Such optional features include, but are not limited to: arithmetic measurements raised to powers; second-order and higher- order cross-products among arithmetic measurements, binary measurements or arithmetic and binary measurements; averages among such features; principal component features; orthogonal polynomial features; and composites made up of any such features, combined with other such features in the RFM 25 that have been computed from recent measurements.
- Recent features can be used to both monitor and impute (or forecast) concurrent feature values as a function of previously observed feature values stored in the recent feature memory 25. For example, suppose that measurements from a chemical process are monitored for unusual values to identify sudden measurement changes that may indicate a system failure.
- the CIP system identifies sudden changes by creating CIP first-order difference features, each being the difference between a measurement for the concurrent trial and the same measurement for the immediately preceding trial that has been stored in the recent feature memory 25.
- first order difference features the CIP system can quickly leam means and variances for such features, which in turn enables the CIP system to identify unusual values of the first-order difference features as indications of sudden change.
- forecasts for concurrent values of a continuous process can be utilized to predict expected concurrent values before the values are actually observed.
- the CIP system may use the recent feature memory 25 to create each concurrent feature as the concurrent measurement values as well as the last 5 measurement values.
- the CIP system may then use the first 6 observed values to leam how to impute the sixth value from the last 5 during trial 6; the CIP system may then impute the seventh value from the second value through the sixth value at the beginning of trial 7 while the seventh value is missing; may then update its leamed parameters at the end of trial 7 after the non-missing seventh value has been received; the CIP system may then impute the eighth value from the third value through the seventh value at the beginning of trial 8 while the eighth value is missing; and so on.
- 0 and 1 Values between 0 and 1 can occur naturally in several settings, such as pre-processing operations outside the CIP system.
- the CIP system will operate to: (a) set a plausibility value to 1 for the average if all 3 of its component pixels are non- missing, (b) set the plausibility value to 0 if all 3 of the pixels are missing; and (c) set the plausibility to some intermediate value if 1 or 2 of the 3 are missing. If one of the values is missing, the appropriate input measurement value is the average among non-missing pixel values, and the appropriate plausibility value is the proportion of pixel values that are non-missing among the total number of possible pixel values.
- Plausibility values between 0 and 1 also may be used in settings where CIP system users wish to make subjective ratings of measurement reliability instead of calculating plausibility values objectively.
- the CIP system treats a plausibility value between 0 and
- the CIP system identifies relevant parameters and implements an accurate process for leaming the relationships among the multiple input measurements.
- the mean updating formula takes on the following simplified form if all prior and concurrent viability values are 1 :
- ⁇ [OUT] represents the mean of all prior measurements up to and including the current measurement value.
- Equation (1) changes ⁇ values toward m[IN] values more for higher values of / than for lower values of /, in accordance with the above leaming weight discussion. Equation (1) is preferably modified, however, because Equation (1) may not accurately reflect different plausibility histories for different elements of ⁇ [IN].
- Equation (1) is modified to combine // [IN] with /n[IN] according to elements of a leaming history parameter ⁇ , which keeps track of previous leaming history at the feature element level.
- Equation (1) can be justified and derived within the quantum conceptual framework for the CIP system, as follows: suppose that the leaming weight / is the ratio of concurrent quantum counts for m [IN] to the concurrent quantum counts ⁇ y[PRIOR] associated with ⁇ [IN], as the CIP system does; suppose further that ⁇ [IN] is the mean of ⁇ 7[IN] prior quantum counts and that [IN] is the mean of ⁇ 7[PRIOR] concurrent quantum counts; algebra can show that Equation (1) will be the overall mean that is based on all q[TN] prior counts along with all [PRIOR] concurrent counts.
- Equation (1) only applies when all viability values are 1.
- the CIP system preferably uses a more precise function than Equation (1), in order to properly weight individual elements of ⁇ [OUT] differentially, according to the different viability histories of ⁇ [OUT] elements.
- the mean updating formula is, for any element ⁇ if) of ⁇ ,
- Equation (2) resembles Equation (1), except a single learning weight / as in Equation (1), which would be used for all feature vectors elements during a trial, is replaced by a distinct leaming weight l[ ](f) for each component feature vector element.
- each feature vector element may be individually rather than each feature vector element have the same weight as in Equation (1).
- the learning history parameter ⁇ is also updated, after being used to update feature means, to keep a running record of prior leaming for each feature, as follows:
- Equation (6) is similar in general form to Equation (3), because Equation (3) is based on the same CIP quantum count framework. Just as Equation (3) produces an overall average ⁇ [OUT] of assumed quantum count values, Equation (6) produces an overall average v [OUT] of squared deviance and cross-product values from the mean vector ⁇ [OUT].
- ⁇ [OUT] ( 1 + / ) ( ⁇ [IN] - / (e ⁇ [IN] ) ⁇ e ⁇ [IN] ( 1 + d ) ), (7)
- Equation (7) is based on a standard formula for updating the inverse of a matrix having form (6), when the inverse of the second term in (6) is known. Equation (7) is also based on the same quantum count rationale as Equations (3) and (6).
- Equations (5) through (8) are only approximate versions of the preferred CIP error vector and updating formulas. However, different and preferred alternatives are used for four reasons: first, the CIP system counterpart to the error vector formula Equation (5) is based on ⁇ [IN] instead of ⁇ [OUT], since the CIP kernel can update learned parameters more quickly utilizing ⁇ [IN], thus furthering fast operation. Second, the preferred CIP embodiment equation to Equation (5) reduces each element of the error vector e 3* toward 0 if the corresponding concurrent viability v(f) is less than 1. This reduction gives each element of the e vector an appropriately smaller role in updating elements of v [D] and ⁇ if the e vector's corresponding element of ⁇ n[IN] has a low viability value.
- Equation (6) and Equation (7) are only accurate if previous ⁇ values that have been used to compute previous ⁇ [OUT] and ⁇ [OUT] values are the same as ⁇ [OUT] for the current trial. Since all such ⁇ values change during Equation (7) the CIP system uses an appropriate modification to Equation (7) in the preferred CIP embodiment. T e preferred alternatives to Equation (5) through (8) are,
- v[D,OUT]() le[lN](f) 2 + v [ D , I N ](/) / ( 1 + / ) (10)
- ⁇ [IN], ⁇ [IN] and v [D,IN] values represent values that were the output values from the previous trial that were stored in the leamed parameter memory.
- CIP feature imputing formulas are based on linear regression theory and formulations, that fit CIP storage, speed, input- output flexibility and parallel embodiment. Efficient kemel operation is enabled because regression weights for imputing any feature from all other features can be easily computed from elements of v inverse, which are available in ⁇ .
- the CDP kemel imputes each missing w ⁇ [IN] element as a function of all non-missing m[IN] elements, where missing and non- missing w ⁇ [IN] elements are indicated by corresponding viability v element values of 0 and 1, respectively. If a CIP application utilizes F features and the viability vector for a trial contains all 1 values except the first element, which is a 0 value, the CEP kemel imputes only the first feature value as a function of all others.
- the regression formula for imputing that first element is,
- Formulas for imputing other m[IN] elements are similar to Equation (15), provided that only the OT[EN] element being imputed is missing.
- the CIP kemel uses improved alternatives to Equation (15), so that the CIP system can operate when any combination of m[EN] elements may be missing.
- the kemel imputes each missing m[IN] element by using only other m[IN] elements that are non-missing.
- the kemel also replaces each m[OUT] element by the corresponding [IN] element whenever m[JN] is non- missing.
- the regression formulas used by the CIP system are also designed for parallel as well as efficient operation that makes maximum use of other kemel computations.
- the kemel saves time and storage by using the elements of e[IN] and x from Equation (9) and Equation (14) for imputing, because t?[IN] and x are also used for leaming.
- the kemel imputing formula is,
- the kemel produces several statistics for feature value monitoring and graphical display. These include the leamed feature mean vector ⁇ [OUT], feature variances v [D] and d from Equation
- the kemel also produces another set of regressed feature values, which are the imputed values that each feature would have if the feature was missing. These regressed values have the form,
- the CIP system can use the statistics in several ways, as specified by user options.
- One use is to plot deviance measures as a function of trial number, including the Mahalanobis distance measure d, standardized squared deviance values from leamed means,
- the Mohalonobis distance measure d and the three deviance measures of equations (17), (18), and (19) are useful indices of unusual input behavior.
- the Mahalanobis distance measure d is a useful global measure for the entire feature vector, because d is an increasing 3 function of the squared difference between each observed feature vector and the feature leamed mean vector.
- the standardized deviance measures d[ l](f) are component feature counterparts to the global measure d, which can help pinpoint unusual feature values.
- the standardized residual measures d[2](f) indicate how input features deviate from their regressed values, based not only on their previously leamed means but also on other non-missing concurrent feature values.
- the CIP system can also use special features in conjunction with their monitoring statistics to produce useful information about unusual feature trends. For example, for any feature of interest, a new feature can be computed that is the difference between the concurrent feature value and the feature value from the immediately preceding trial. The resulting deviance measure from Equation (17) provides a useful measure of unusual feature value change.
- the CIP system can also use a similar approach based on second-order differences instead of first-order differences to identify unusual deviations from ordinary feature changes. The CIP system can thus provide a variety of graphical deviance plots for manual user analysis outside the CIP system.
- the CIP system can also use deviance information internally to control learning weights and schedule feature modification operations. For example, the system can establish a preselected cutoff value for the global distance measure d. The system can then treat a d value exceeding the preselected cutoff value as evidence of a data input device problem, and the system can set future leaming weight values to 0 accordingly until the problem is fixed. Likewise, the component deviance measures d[l](f] and d[2](f) can be used to set measurement plausibility or feature viability values to 0 after the component deviance measures have exceeded prespecified cutoff values. Setting the leaming weight to zero prevents input problems from adversely affecting the accuracy of future CIP operations.
- the CIP system uses the leaming weight / as part of the system leaming.
- the leaming weight / is the ratio of a quantum count associated with the concurrent feature vector to the quantum count associated with prior parameter leaming. From that basis, the CIP system produces equal impact learning weight sequences, that is, sequences based on an equal number of quantum counts for each trial. If a learning weight sequence is labeled by 1(1), 1(2) and so on, equal impact schedules have the form,
- the constant R is the ratio of the common quantum count for all such trials to an initial quantum count. The role of this ratio and the initial quantum count is discussed below.
- CIP users or the CIP system can generate sequences that are either liberal or conservative.
- Liberal sequences give more impact to more recent trial feature values, while conservative sequences give more impact to less recent trial feature values.
- a leaming sequence with all leaming weights set to 1 is liberal, while one that sets all but the first leaming weight to 0 is conservative.
- Conservative sequences are appropriate when the input CEP data are being generated according to continuously changing parameter values, and conservative sequences are appropriate when more recent information is not as reliable as less recent information.
- the CIP system treats initial values for learned regression parameters ⁇ , v [D] and ⁇ as if they were generated by observed feature values.
- the CIP kemel combines initial values with information from the first feature vector to produce updated parameter values, according to Equations (2), (10) and (11); and during the second trial the CIP kemel combine initial values and first trial values with information from the second vector to produce new updated parameter values. This process is repeated for subsequent trials.
- the impact of initial parameter values on overall leaming will be less than the initial parameter impact before the trial.
- effects due to particular initial regression parameter values will be small after a small number of leaming trials, unless very conservative leaming weight sequences are provided to the kemel.
- initial values for leamed regression parameters can be important.
- the CIP system may accept user- supplied initial regression parameter values from a keypad 17 as illustrated in Figure 1.
- the CIP system provides default initial values for leamed regression parameters as follows: the default value for each element of the mean vector ⁇ is 0; the default value for the connection weight matrix ⁇ is the identity matrix; and the default value for each element of the variance vectorv [D] is 1.
- the identity matrix as the initial default value for ⁇ produces initial imputed feature values that do not depend initially on other feature values.
- the initial identity matrix also enables the CIP system to impute feature values from the first trial onward.
- conventional statistical approaches require that at least F leaming trials (where F is the number of features) before any imputing can occur.
- the CIP system initializes elements of the leaming history parameter vector ⁇ .
- the leaming history parameter vector dictates how much an input feature vector element will affect leaming, relative to previous leaming.
- the default initial value for each element of the learning history vector ⁇ is 1, which gives each input feature vector element the same impact on leaming during the first leaming trial.
- the CIP system alternatively may implement three kinds of feature function monitoring statistics for graphical display: a vector of squared feature multiple correlations ⁇ [M], a vector of tolerance band ratio values r and an array of partial correlations ⁇ [P].
- Each element ⁇ [M] (f) of ⁇ [M] is the squared multiple correlation for imputing the corresponding feature vector element m(f) from the other elements in m.
- squared multiple correlations can be interpreted according to well-known statistical properties. Such statistical properties imply that each feature can be predicted by other features if the feature's squared correlation is near the maximum possible value of 1 instead of the minimum possible value of 0.
- Each squared multiple correlation ⁇ [M](f) also may be optionally used to compute and supply the corresponding tolerance band ratio element r(f).
- Each element of r can be expressed as a ratio of two standard deviations.
- the numerator standard deviation is the square root of v [D]( )
- the denominator standard deviation is the standard deviation of m (f). Since error tolerance band widths are routinely made proportional to standard deviations, it follows that each r(f) value is the tolerance band width for imputing m(f) if all other m[IN] elements are not missing, relative to the tolerance band width for imputing m(f) if all other [IN] values are missing.
- Each partial correlation is an index of how highly two features are correlated, once they have been adjusted for correlations with all other features.
- users can examine the partial correlations to decide if any given feature is unnecessary for imputing any other given feature.
- Users can also examine rows of the partial correlation matrix to identify if a pair of features can be combined to produce an average, instead of being used separately.
- each partial correlation for the first feature is the same as the corresponding partial correlation for the second feature. Both such feature values can then be replaced by their average value for imputing the third feature value, without loss of imputing accuracy.
- the CIP system may use the following formula to obtain squared multiple correlation values,
- the system may use the following formula for tolerance band values,
- the system may use the following formula for partial correlation values,
- the CIP system can also supply the connection weight matrix ⁇ to users for user modification and interpretation.
- users can compute and assess principal component coefficients, orthogonal polynomial coefficients and the like from ⁇ to identify essential features to fit user needs. Once a user has identified essential features, the user can either reformulate the input transducer functions or supply features outside the CIP system, accordingly.
- the CIP system can use the statistics internally and automatically, through its feature function controller ( Figures 2 and 3).
- the feature function controller can identify unnecessary features for removal by checking partial correlation and squared multiple correlation values against predetermined cutoff values.
- the feature function controller can identify redundant feature pairs by checking squared differences among their partial correlation values against predetermined cutoff values. Once such unnecessary or redundant features have been identified, the feature function controller can send feature function modification commands to the transducer input processor and transducer output processor accordingly.
- the CIP system can also modify elements of the connection weight matrix ⁇ .
- the elements of ⁇ can be adjusted for the removal of an unnecessary feature, for example, feature /, to produce a new, adjusted connection weight matrix with one less row and one less column, say ⁇ ⁇ / ⁇ , as follows. If the submatrix of ⁇ excluding row / and column / is labeled by ⁇ ⁇ f > and the deleted row / is labeled by ⁇ ⁇ /> , then an appropriate adjustment formula based on a standard matrix algebra function is,
- the parallel CIP kemel 21 utilizes connection weights between features processors during concurrent imputing, monitoring and learned parameter memory updating operations. As also noted above, the parallel CIP kemel 21 processes F features per trial and uses F parallel feature processors 31 1 through 3 I , along with a Mahalanobis distance processor 34.
- the parallel CIP system solves the output problem by providing a joint access memory 23 that has a switching junction for connecting pairs of feature processors that are accessed according to a coordinated timing scheme.
- the switching junctions serve to enable each processor to exchange pertinent information so that a large number of processors may operate in parallel.
- a processor output only drives one input at any given coordination time interval, as explained below.
- the illustrated circuitry enables each feature processor to be connected to elements of the joint access memory 23, identified by circles and labeled M, and the circuitry enables every processor to be paired with every other processor through an organized scheme that is described below.
- Each processor 311 through 3116 is also connected to a register Dj through Di 6 in the distance processor 34.
- Figure 5 shows the layout of the conductor buses between the processors 311 through 31 J6 and the distance processor 34.
- Figure 5 also illustrates the interconnections of the processors 311 through 31 16 with the joint access memory 23 and the kemel control unit 32.
- the circuitry illustrated in Figure 5 may be implemented in a silicon chip layout, containing a lower bus layer, an upper bus layer, a set of semiconductor layers between and connecting the lower and upper bus layers and a control bus layer above all of the other layers.
- Lines 45 Lj through 45Li6 in Figure 5 represent buses in the lower bus layer
- lines 45U2 through 45U] 5 represent buses in the upper bus layer
- lines E i through Ei 6 represent connection extensions between the lower layer buses and upper layer buses along the diagonal edge formed by the joint access memory elements
- transverse lines 321 through 3229 represent joint access memory 23 control buses that are operated by the kemel control unit 32 to control joint access memory 23 switching.
- each of the circles M(2,l ), M(3,l ) and M(3,2) through M(16,15) represents a JAM memory and switching node containing a switching junction, switching logic and a memory registei, all of which may lie within semiconductor layers between the lower bus layer and the upper bus layer.
- the circles, M(16,l) through M(16,15) within the feature processors 31 x through 311 represent connection weight registers within the processors that contain the main diagonal of the connection weight matrix; and the circles Dj through Di within the distance processor 34 represent registers for communicating between the distance processor 34 and the corresponding feature processors 311 through 3116-
- Each of the sixteen lower buses 45L ⁇ through 45L ⁇ connects the corresponding feature processor 31 j through 3116 to the corresponding register D ⁇ through D in the distance processor 34.
- the connection extensions Ej through E ⁇ connect each lower buse 45Lj through 45Li6 to its corresponding upper buse 45Uj through 45Ui6-
- Each of the lower buses 45L] through 45Lj 6 also connects the corresponding feature processor 311 through 31 ⁇ to the dedicated joint access memory nodes M(2,l), M(3,l) and M(3,2) through M(16,16), as follows: lower bus 45Lj is connected at the bottom of JAM nodes M(2,l) and M(3,l) through the bottom of
- lower bus 45L2 is connected at the bottom of nodes M(3,2) through M(16,2) and at the top of node M(2,l) through the connection extension E2 to the upper bus 45U.
- lower buses 45L3 through 45Li 5 are connected to the bottom of corresponding joint access memory 23 nodes along each respective lower bus, and to corresponding upper buses 45U3 through 45Uj5 via their corresponding connection extensions E3 through E15, as discussed in connection with lower buses 45Lj and 45L2 and upper buses 45U2 and E2.
- the upper bus 45U3 connects at the top of nodes M(3,2) and M(3,l).
- Each of the upper buses 45U2 through 45Ui6 connect at the top of each node along the respective upper bus in the same manner as upper bus 45U3. (Because many connection extensions and nodes are illustrated, reference labels to all extensions E3 through E j 5 are not included in the figure in order to provide a more readable figure. It should be appreciated by one skilled in the art that the connection extensions and nodes may be identified using the convention utilized above.) Lower bus 45Lj6 is connected at the top of all of the corresponding nodes M(16,l) through M(16,15) and to the corresponding upper bus 45Ui6 via its corresponding connection extension Ej ⁇ .
- the lower and upper buses and the interconnections include one line of conductor for each storage bit that is implemented by the CIP kemel 21.
- each of the bus lines represent 32 conductors, one for each bit of precision.
- the parallel kemel 21 can communicate between feature processors and access JAM elements in parallel and quickly.
- each of the control buses 32j through 3230 m Figure 7 represents three conductors, the uses of which are described below.
- FIG. 6a is a top detail view of the node M(16,15) along with the node's related switches and buses
- Figure 6b is a side view of M(16,15) along with the node's related switches and buses.
- Figure 6a shows the lower bus 45Li5 and processor 15 offset a small amount to the right for clarity of presentation, as indicated by the dashed off-set line 805.
- Figure 6a shows joint access memory node M(16,15) details, including the following: a memory cell containing ⁇ (16,15); a memory input switch SI for updating ⁇ (16,15); a memory output switch S2 for accessing ⁇ (16,15); and a dual switch S3 for joining the processor 3116 upper bus 45Ui6 and the processor 3115 lower bus 45Li5 at the output of S2.
- Figure 8b shows a side view of the memory cell and same three switches illustrated in Figure 8a.
- Figure 6a shows the processor 3115 upper bus 45U ] 6 connected to the input for the memory cell containing ⁇ (16,15) through S2.
- the memory cell containing ⁇ (16,15) is updated to contain the contents of the processor 31 j 6 upper bus 45Ui6-
- the processor 3116 upper bus 45Ui6 and the processor 3115 lower bus 45Lj5 are interconnected when S3 is closed and S2 is opened.
- both the processor 3116 upper bus 45Ui6 nd me processor 3115 lower bus 45Lj5 are connected to the output of the memory cell containing ⁇ (16,15) when both S2 and S3 are closed.
- both S2 and S3 are closed, the contents of the memory cell containing ⁇ (16,15) reside on both buses.
- Switches SI, S2, and S3 in Figure 8a are controlled by signals on control lines Cl, C2, and C3, respectively. These three control lines comprise the control bus lines 3229 in Figure 5. When a signal is positive on any of these control lines, the corresponding switch will be closed. Switches S4 through S7 in Figure 6a are similarly controlled by signals on the corresponding control lines C4 through C7. Switches S4 and S5 are connected to an input bus 801 and an output bus 802 for processor 3115, respectively, while switches S6 and S7 are connected to an input bus 803 and an output bus 804 for processor 16, respectively.
- Parallel kemel 21 operation is coordinated by the control unit 32 so that (a) each of the feature processors 31 j through 31 ⁇ is continuously busy during a trial; (b) each joint access memory bus contains no more than one variable value at any given time; (c) each memory cell sends an output value to no more than two buses at any given time; (d) and each feature processor sends the output value of the processor to no more than one input at any given time.
- the coordination steps are controlled by control buses 32j through 3229, along with other control signals that are sent to individual feature processors.
- Each feature processor F computes its feature processor x(F) value by first initializing the x value at 0 and then accessing joint access memory nodes along the feature processor's lower and upper bus, one at a time.
- each feature processor F performs the following sequence of operations: first, fetching the stored ⁇ element along that node; second, fetching the e[IN] element that is available at that node; third, multiplying the two elements together to obtain a cross-product; and fourth, adding the cross- product for the processor to the running sum for x(F) implemented in the processor.
- the feature processor of focus may be processor 31 ⁇
- the processor 3116 may access M(16,15) as discussed with reference to Figure 5.
- processor 31 j 6 would thus update the ⁇ (16) value that processor 31i6 is computing, by multiplying ⁇ (16,15) with e[IN](15) and adding the product to the running J (16) value.
- processor 3115 is updating the computation of JC(15) value in processor 3115, by accessing ⁇ (16,15), accessing e[IN](16), multiplying the values together and adding the product to the processor 31 ⁇ running *(15) value.
- Figure 6 illustrates the control unit timing for the x updating step described above.
- the top signal illustrates a CIP system clock pulse as a function of time and the next 7 plots below the graph show the switch control values along lines Cl through C7 as a function of time.
- the switches are set in accordance with switch operation (a) above, sending ⁇ (16,15) to feature processors 3115 and 31 ⁇ .
- the switches are set in accordance with switch operation (b) above, sending e[IN](15) to processor 31 ⁇ after which processor 3116 adds the product between ⁇ (16,15) and e[IN](15) to the miming computed value of J (16).
- the switches are set in accordance with switch operation (c) above, sending e[IN](16) to processor 3115, after which processor 3115 adds the product between ⁇ (16,15) and e[EN](16) to the running computed value of -r(15).
- switches S2 and S3 will be open as indicated by their corresponding C2 and C3 control values being low, thus allowing other updating operations to occur without interference along the processor 3115 bus and the processor 31 ⁇ bus.
- Computing x proceeds such that each processor is computing a cross-product and adding the cross product to the running x sum of the processor, while each of the other processors is computing another cross-products and adding the product to the processor's x term.
- ⁇ [EN](16,15), (15) and (16) are all first available in a single processor.
- the single processor then computes ⁇ [OUT](16,15) according to equation (11), after which the processor sends the updated value to the storage cell for ⁇ (16,15).
- control timing for the updating sequence of operations is illustrated.
- the system clock pulses are shown as a function of time and the four plots below the clock pulse show control values along lines C2, C3, C6, and C7 as a function of time.
- the switches are set as discussed above in connection with switching operation (d), sending JC(15) to feature processor 16.
- switches S2 and S3 are opened as indicated by their corresponding C2 and C3 control values being low, thus allowing other updating operations to occur without interference along the processor 31 15 bus and the processor 31 1 6 bus.
- Each entry in Figure 9 shows the time interval during which a processor is performing by itself or with one other feature processor at every interval in the overall x or ⁇ updating process.
- the triangular table labeled as 911a has rows labeled U ⁇ through Ui 6 and columns labeled Li through o, which correspond to the rows and columns illustrated in the Figure 5 joint access memory.
- the entries represent control timings as described below.
- the row table labeled as 91 lb has columns that correspond to the Figure 5 feature processors 311 through 3116.
- the entries represent control timings as described below..
- Entries in the table having the same numeric value represent the sets of processors that are uniquely paired during the time interval indicated by the table entry.
- the entries 911a indicate which processors are paired during every interval in the x updating process.
- Feature processor pairing at any time interval is determined by locating in Figure 9 the time interval along the processor's bus or within its processor.
- the bus for processor 31 ⁇ corresponds to the bottom row in port 91 la of Figure
- the entries 911b in Figure 9 indicate which processors are performing operations but are not paired with another processor for that particular time interval. For example, at time interval 1 processor 1 is updating J (1 ) by adding e[IN](l) x ⁇ [N](l) to the processor 311 running sum for x(l), while at the same time interval processor 9 is updating x(9) by adding e[IN](9) ; ⁇ [IN](9) to the processor 9 miming sum for x(9). By accessing the processors and updating in the proceeding manner, the parallel CIP kemel is able to keep all of the feature processors busy throughout the x updating process.
- the number patterns in Figure 9 also indicate a systematic pattern of control lines that can be used to implement processor operations during x computations. For example, when using sixteen-features, each given time interval number in the sequence falls along a line from the lower left boundary of the 911a values in Figure 9 to the upper right boundary. As a result, all corresponding joint access memory nodes along that line can share the same set of 3 control lines, because these control lines will share the same timing. Thus, the pattern of coordinating time intervals illustrated in Figure 9 is representative of the pattern of control lines that is illustrated in Figure 5.
- Equation (27) The same coordination that is formulated in Equation (27) for computing x is used by the CIP system for updating the elements of ⁇ , with one exception. While x updating implements the computing the operations indicted in Figure 7 at each Equation (27) interval . ⁇ updating implements computing operations indicated in Figure 8 at each Equation (27) interval i.
- Equation (14) the distance processors Dj through DF, as shown in Figure 7, simultaneously and in parallel; each such product will be sent to distance processor registers Dj through DF, as shown in Figure 7, simultaneously and in parallel; third, the distance processor will sum the contents of registers Di through DF to obtain d, according to Equation (14); finally, the distance processor will return d to each of the feature processors through registers Dl through DF, for use by the processors in calculating the updated variance and updated connection weight matrix, in keeping with Equation (10) through Equation (12).
- the design of the joint access memory and connected processors is advantageous for compact embodiment in highly integrated circuitry.
- a compact embodiment is advantageous for optimal speed during each trial.
- the kemel 21 is preferably implemented with as many feature processors and JAM elements as possible on a single chip. Otherwise, the parallel processing speed advantage may be negated by serial communication between chips. Also, shorter inter-component distances result in faster electrical signal transmissions between components.
- the CIP parallel kemel also satisfies other design concerns: (a) a signal degradation concern (commonly known as fan- out)-minimizing the maximum number of inputs that a single feature processor of JAM elements supplies at any given time; and (b) a space utilization concern-minimizing the number of required conductors for communicating between feature processors and JAM elements.
- the CIP parallel kemel satisfies these various design concerns through the JAM bus and switching structure, along with parallel kemel feature processing coordination discussed above.
- each analog JAM bus is a single conductor instead of a collection of digital bit wires;
- each digital JAM switch has only one contact instead of several contacts;
- each digital JAM memory element is a small resistance- capacitive network instead of a much larger digital memory element; and
- simple (non-sequential) analog circuits are used to perform the arithmetic operations.
- some of the above JAM analog operations can be combined with digital ALU operations to produce an analog-digital hybrid that is more compact and faster as well as acceptably accurate.
- the sequential kemel utilizes the basic kemel operations that are discussed above in connection with the parallel kemel. Thus, sequential kemel operations produce the same outputs as parallel kemel operations whenever both kemel receive the same inputs.
- sequential operations will generally be slower, because they are obtained using only one processor instead of using F processors.
- Some of the sequential kemel operations are implemented in a different manner for efficiency, rather than identically simulating parallel kemel operations.
- the x computing step and the ⁇ [OUT] computing step of the sequential kemel operation are implemented for optional storage and speed. Both steps are based on storing the elements of ⁇ as a consecutive string containing ⁇ (l ,l), followed by ⁇ (2,l), followed by ⁇ (2,2), followed by ⁇ (3,1), followed by (3,2), followed by (3,3) and so on to ⁇ (F,F).
- the x computing step and ⁇ [OUT] computing steps access the consecutively stored elements of ⁇ from the first element to the last element. The overall effect is to make both such steps far faster than if they were to be performed conventionally, using a nested loop.
- the sequence of operations for computing x and ⁇ updating steps are discussed in connection with Figure 19 and Figure 20 below, respectively.
- concurrent operations include transducer input operations 504 performed by the transducer input processor 24, kemel operations 505 performed by the kemel procesor 24 and transducer output operations 506 performed by the transducer output processor 26.
- Buffer storage can make the output values from each device available as input values to the next device so that both devices can simultaneously operate on data for different trials at the same time.
- kemel operations 505 may produce kemel output functions and leamed parameter updating functions for the concurrent trial, while transducer input operations
- transducer output operations 506 are producing imputed measurement values from the preceding trial.
- Management operations which can be performed occasionally over a period of several trials, include leaming weight control operations 508 performed by the leaming weight controller 40; feature function control operations 511 performed by the feature function controller 41; and graphical display operations 517. Buffering enables parallel management operations, and buffering enables parallel concurrent operations performed by the coordinator
- output information from the kemel 21 is made available as input information to the leaming weight controller 40 via lines 40 ⁇ through 40F, 0 ⁇ , and output information from the kemel 21 is made available to the feature function controller via 41 ⁇ through 4 IF- Buffers are used so that management operations based on previous trial statistics in the buffer, can proceed while concurrent operations are continuing.
- the feature values monitored during operation 510 are transmitted for leaming weight control operation 509.
- the outputs from the kemel are processed during feature function operation 513 and are transmitted to control feature functions during operation 512.
- the feature function controller 41 can perform the Equation 23 through Equation (25) computations and the leaming weight controller 41 can performing the Equation (17) through Equation (19) computations.
- the kemel 21 continues its concurrent operations.
- the three operations 504, 505, and 506 thus proceed simultaneously.
- the CIP system occasionally monitors feature functions.
- the monitoring operations include receiving leamed connection weight values and leamed variance values from the kemel 21 ; computing feature multiple correlation values according to Equation (23); computing feature tolerance band ratio values according to Equation (24); and computing partial correlation values in keeping with Equation (25).
- the statistics can either be: (a) provided to CIP users for manual interpretation and refinement via graphical display operations 517; or (b) used for CIP for automatic refinement
- Changing feature specification is implemented by controlling the modification switching operation 507 as a result of monitor feature function operation 513. Triggering the switch operation 507 causes the CIP system 10 to reinitialize feature specifications through operation 502 as indicated.
- the initial mean vector is an average of q(0) quantum initial vectors
- input feature vectors [IN] for trials 1 through / are averages of (1) quantum vectors for trial 1 through q(t) quantum vectors for trial t, respectively.
- all concurrently leamed regression parameter values and all concurrently available refinement parameters can be interpreted as average statistics based on equally weighted quantum counts from an overall sample size of (0) + q( ⁇ ) + . . . + q(t ).
- the leamed feature mean vector at the end of trial 10 has the interpretation of an average among q(0) + q( ⁇ ) + . . . + q(10) quantum values, and the leamed feature mean vector after any other number of trials has the same interpretation.
- Equation (20) through Equation (22) can be generated by setting q( ⁇ )
- Equation (28) provides a quantum- observation interpretation for equal-import leaming by the CIP
- the parallel CIP system can operate more quickly through the use of buffered communication. Also, discussed above, implementing as many CEP operations as possible on one chip can avoid considerable inter-chip communication time loss. As a result, implementing several CIP subsystems on different layers of a single chip and communicating between the layers through parallel buffering can maximize overall CIP operation speed.
- Figure 11 shows memory locations 311 within parallel kemel features processors 31 ] through 31p that are accessed by the leaming weight controller 40; the leaming weight controller buffer 1101 ; and corresponding buses 41 ⁇ a through 41pd to the leaming weight controller buffer 1101.
- the buffer elements 111 are geometrically configured in the same manner as kemel 21 memory locations, correspond to respective parallel kemel memory locations and may be aligned in parallel to the kemel memory locations as shown.
- the parallel structure of the buffer with respect to kemel 21 memory locations enables the buffer to reside on a layer above or below the parallel kemel with minimal wiring.
- Figure 12 shows: memory locations 411 within parallel kemel feature processors that are accessed by the feature function controller 41 ; the feature function controller buffer 1201; and corresponding buses 41JAM(2,1 ) through 41JAM(F,F) and 41 ⁇ through 41 p to the feature function controller buffer 1201.
- the buffer elements in Figure 12 are configured in the same manner as kemel 21 memory locations, correspond to respective parallel kemel memory locations and aligned in parallel to the kemel 21 memory locations.
- the buffer may reside on a layer above or below the parallel kemel with minimal wiring requirements.
- CIP subsystems to reside on a single chip or a single array of aligned chips. As a result, time-consuming communication time within the system is minimized.
- FIG. 13 the subsystems of the sequential kemel are shown.
- Figure 13 shows more specifically than Figure 3, the vectors and parameters that are transferred among the subsystems.
- the various inputs and outputs of the leaming weight controller 40a, feature function controller 41a and coordinator 38a are shown.
- the various monitored parameters and control functions implemented by the sequential system are the same as with the parallel system except that the transfer of relevant data occurs sequentially.
- the sequential CIP system can perform all operations associated with the parallel system discussed above, although not as fast, because only one CIP operation is performed at time using a single available processor. Also, at the subsystem level simultaneous operations are not implemented as in the parallel kemel embodiment, because only one processor is available for kemel operations. Beyond speed concerns, however, the CIP system is no less powerful when implemented sequentially than it is when implemented using parallel processors. Also, the sequential CIP embodiment has at least two advantages over parallel embodiment : the sequential embodiment is generally less expensive because it may be embodied in a conventional computer rather than specially designed parallel circuits; and sequential embodiment can accommodate many more features per trial on conventional computers than the parallel embodiment can accommodate on specialized circuits. As a result, the sequential CIP embodiment is useful, in applications where trial occurrence rates are low relative to the number of features per trial.
- Alternative operations of the kernel include: (1) updating a coefficient matrix that is used by the David-Fletcher-
- the DFP method is one of several iterative methods for finding the maximum (or minimum) independent variable values for a function of several variables.
- Numerical optimization methods are generally useful but are also generally slow.
- numerical optimization methods are used to find optimum values associated with five-day weather forecasts, but generally take many hours to converge, even on supercomputers.
- the DFP is especially useful in a variety of applications, because the DFP method learns derivative information during the iterative search process that may not be readily available.
- the parallel kemel process is used to implement a fast concurrent information processing system, a modified version of can be used for a fast new numerical optimization system.
- the DFP method continuously updates the inverse of a matrix as part of normal operation. Instead of updating the inverse of a covariance matrix as in the CIP system, the DFP algorithm updates the inverse of an estimated matrix of second-order derivatives, which is called the information matrix.
- the formula for updating the DFP inverse is distinct from the formula for updating the CIP inverse, an extension to the parallel CIP kemel algorithm can be used for DFP updating.
- the DFP information matrix inverse updating formula is,
- ⁇ [DFP,OUT] ⁇ [DFP,IN] - c [DFP] x[DFP] ⁇ x[DFP]
- the DFP updating formulas (29) through (33) may be tailored to suit DFP updating.
- the DFP counterpart to the kemel process utilizes the same number of steps, as the parallel CIP kemel 21; the DFP counterpart computes Equation (29) just as the parallel CEP kemel computes Equation (11); the DFP counterpart computes its distance function satisfying Equation (32) and its inner product function satisfying (33) just as the parallel CIP kemel computes its distance function satisfying Equation (14) ; and the DFP counterpart computes its matrix-vector product satisfying Equation (31), just as the parallel CIP kemel 21 computes its matrix-vector product satisfying Equation (13).
- Equation (29) is simpler than its corresponding parallel CIP kemel Equation (12); the DFP counterpart solves two inner products to compute Equations (32) and (33) instead of a single parallel CIP kemel inner product to compute Equation (14); and the DFP counterpart computes terms for two outer products to compute the second and third terms in Equation (29) instead of a single corresponding parallel CIP kemel outer product operation to compute the second term in Equation (11).
- a less computationally involved tailored kernel embodiment may be implemented where multiplication of a symmetric matrix by a vector is performed repeatedly and quickly.
- the kemel embodiment may be simplified to compute such products, of which Equation (13) is one example, by preserving only operations that are needed to compute such products and removing all others.
- the feature removal adjustment formula (26) is a simplified version of the kemel updating formula (11), in that: (a) the Equation (26) second term constant coefficient does not utilize a distance function, and (b) only an outer product among 2 vectors is needed to compute the Equation (26) second term, without first requiring a matrix-vector product as in Equation (13).
- the parallel CEP kemel can be simplified to solve Equation (26).
- "student input transducers” can first be “taught” to use only useful features, after which the operations can be used to produce features. For example, suppose that a CIP system is needed to forecast one dependent variable value feature 1 as a function of several independent variable values for feature 2 through feature 100. During a series of conventional leaming trials a modification of the kemel process can leam to identify the 99 optimal connection weights for imputing feature 1 from feature 2 through feature 100. After the leaming has occurred, the trained module can be used in place of an input transducer having 99 inputs corresponding to features 2 through 100 and only one output corresponding to feature 1.
- the module When used as an input transducer, the module would differ from the kemel in that the module's leaming and updating operations would be bypassed.
- the only modifications of the kemel needed to implement such a module are an input binary indicator for leaming versus feature imputing operation, along with a small amount of intemal logic to bypass leaming during feature imputing operation.
- the kemel receives a leaming weight /, a feature vector m [IN] and a viability vector v when initial processing begins by the kemel 21.
- the kemel accesses leamed parameter memory elements that include a mean vector ⁇ [IN], ⁇ [IN], v [D,IN], ⁇ [IN], and ⁇ [IN] values.
- each of the ⁇ [IN], ⁇ lN], ⁇ /lN], and /[IN] values were calculated as outputs stored in leamed parameter memory during the previous trial. If, however, this is the initial CIP system iteration, then the ⁇ [IN] values are zero, the ⁇ [IN] values correspond to the identity matrix, and the ⁇ [EN] values as well as the v [D,EN] values are one.
- the kemel calculates a component feature leaming weight from the viability vector ⁇ , the global leaming weight / and the leaming history parameter /[EN] values according to Equation (3). The process then proceeds to step 1404.
- the feature mean vector m[IN] is updated according to Equation (2).
- the intermediate imputed feature vector e[IN] is calculated according to Equation (9).
- the leaming history parameter /[OUT] is updated according to Equation (4).
- the process then proceeds to B of Figure 15.
- FIG. 15 illustrates the preferred process by which the intermediate matrix/vector product is calculated as discussed above.
- each element x(f) of intermediate matrix/vector product is initialized to zero and the process proceeds to step 1502.
- the kemel begins to access the processor pairs according to the coordination time scheme discussed above in conjunction with Figure 9.
- each paired processor / retrieves the appropriate connection weight ⁇ [IN]( , g) at the joint access memory switching node.
- the appropriate intermediate imputed feature value e[IN](g) is retrieved as discussed in connection with Figure 7.
- step 1508 the element x(f) is incremented by the cross product as discussed above.
- the process proceeds to step 1510 where a determination is made as to whether the final coordination time interval for the coordination time scheme has been reached. If the final coordination time interval has not been reached, then the process proceeds to the next time interval at step 1512 after which follows a reiteration of steps 1502 through 1508. The reiteration of st ⁇ ps 1502 through 1508 produces the running sum for the calculation of x(f) If at step 1510, the final connection time interval has been reached, then at step 1522 calculated values are stored to the distance processor as discussed above. The process then proceeds to C of Figure 16.
- step 1602 the output regressed feature vector m[IN] is computed according to Equation (17).
- the imputed feature vector m[OUT] is then computed at step 1604 according to Equation (16).
- the process then proceeds to step 1606.
- the distance processor calculates the Mahalanobis distance value at the distance ALU 34 according to Equation (8).
- each processor stores to the distance processor the distance value cross-product calculated by the particular processor. After receiving each of the cross-product values from all of the processors, the distance processor sums all of the distance values provided by the feature processors to obtain the distance measure d. The process then proceeds to D of Figure 17.
- connection weight matrix element ( ⁇ (f,g) in the preferred embodiment of the present invention.
- the variance v [D](f) is calculated according to Equation (10).
- the process then proceeds to step 1706, where processor g is accessed by processor F according to the connection time scheme discussed above in connection with Figure 9.
- a determination is made as to whether the processor g is being accessed through a lower bus lines. (See Figure 8.) If the processor is being accessed through the lower bus for processor g, then the process of processor / proceeds to step 1712.
- the intermediate matrix/vector product x(g ) is retrieved for the processor /.
- the appropriate connection weight element which corresponds to the memory element located at the node of the currently paired processors, is updated according to Equation (11) by processor/
- the kemel determines whether the final connection time interval for the coordination time scheme has been reached. If the final interval for the coordination time scheme has not been reached, then at step 1722 the kemel proceeds to the next coordination time interval. Following step 1722, steps 1706 through 1720 are repeated.
- step 1710 the processor g is being accessed through an upper bus for processor g then the processor / produces outputs for the concurrent trial and reads inputs for the next trial.
- step 1720 the final connection time interval of the connection time scheme has been reached, then the kemel functions end for the current trial.
- the sequential CIP kemel 21a receives a leaming weight /, a feature vector in [EN] and a viability value v when initial processing begins by the kemel 21a.
- the kemel accesses leamed parameter memory elements that include a ⁇ [EN] values, ⁇ [EN] values, v [D,EN] values and ⁇ [IN].
- each of the ⁇ [IN], ⁇ [IN], and /[IN] values were calculated as outputs stored in leamed parameter memory during the previous trial. If, however, this is the initial CIP system iteration, then each ⁇ [IN]( ) value equals zero, the ⁇ [EN] values correspond to the identity matrix, and the ⁇ [IN] values as well as the v [D,EN] values are one.
- the kemel calculates a component feature leaming weight from the viability vector ⁇ , the global leaming weight /, and the ⁇ [IN] value according to Equation (3). The process then proceeds to step 1804.
- the feature mean vector is updated according to Equation (2).
- the intermediate imputed feature vector [EN] is calculated according to Equation (9).
- the leaming history parameter ⁇ [OUT] is updated according to Equation (4). The process then proceeds to E of Figure 19.
- the preferred steps of the sequential CEP kemel processes implemented in the present invention for computing the intermediate matrix/vector x are shown.
- the process discussed in connection with Figure 19 provides a method of calculating the intermediate matrix/vector x without performing a conventional double loop (i.e., one loop for all possible row values of a matrix and one loop for all possible column values of a matrix).
- the matrix ⁇ elements are stored in a single string in consecutive order corresponding from ft)(l,l) to ⁇ (2,l) to ⁇ (2,2) to ⁇ (3,l) to ⁇ (3,2) to ⁇ (3,3) and so on to ⁇ F, F).
- the location h of the first ⁇ element is initialized to one.
- the intermediate/matrix vector x is set to zero.
- the process then proceeds to step 1906 where the row value /, that corresponds to ⁇ elements stored in matrix form is set to zero.
- the process then proceeds to step 1912 where the row value is incremented by one.
- the column value g corresponding to the ⁇ element as stored in matrix form is next set to zero.
- the process then proceeds to step 1914 where the column value g is incremented by one.
- the process then proceeds to step 1916 where the running sum for computing the intermediate matrix/vector product ⁇ (f,g) times the corresponding x(f) is incremented by the current intermediate imputed feature vector e[EN](g) for column g.
- step 1920 a determination is made as to whether the column value g is less than the row value /. If the corresponding column value g is not less than the row value /, indicating that the ( ⁇ (f,g) element is on the main diagonal of the connection weight matrix, the process then proceeds to step 1924 where the location h of the element is incremented by one. The process then proceeds to step
- step 1930 a determination is made as to whether the row value / equals the column value g which indicates that the ⁇ element resides on the main diagonal of the connection weight matrix ⁇ [EN].
- the process proceeds to step 1914 where the column value a g is incremented by one and the process proceeds to step 1916 as discussed above.
- the running sum for computing the intermediate matrix/vector product x(g) is incremented by the current ⁇ (f,g) element times the corresponding intermediate imputed feature vector e(f) for column/.
- step 1924 the location h of the ⁇ element is incremented by one in order to access the next ⁇ element. If at step 1930 the row value equals the column value then, the process proceeds to step 1940. At step 1940, determination is made as to whether the row value equals the total number of features for the system. If the row value does not equal the total number of features, which indicates that all ⁇ values have not been evaluated, the process then proceeds to step 1912 and the process follows as discussed above. If at step 1940, the row value does equal the number of features, indicating that all ⁇ values have been evaluated then the process proceeds to F of Figure 20.
- the steps of the preferred embodiment of the present invention which compute the output values of the sequential kemel subsystem 21a are shown.
- the output regressed feature vector is computed according to Equation (17).
- the imputed feature evaluation m [OUT] is then computed at step 2004.
- the process then proceeds to step 2006.
- the distance ALU calculates the Mahalanobis distance value at the distance processor according to Equation (8).
- each processor stores to the distance processor the distance value calculated by the particular processor.
- the distance processor sums all of the distance values provided by the feature processors to obtain the distance measure d.
- the process then proceeds to G of Figure 21.
- step 2102 the location of the ⁇ element in the sequence of ⁇ elements is initialized to one.
- step 1204 the row value corresponding to the ⁇ element in the string is initialized to zero.
- step 2106 the row value /is incremented by one.
- step 2108 the variance element v [D,OUT] is updated with respect to the current row / value.
- step 2112 the column value g is initialized to zero.
- step 2114 the column value g is incremented by one.
- step 2116 the connection weight element of ⁇ is updated according to Equation (11).
- step 2118 the ⁇ location value h is incremented by one in order to access the next ⁇ element in the string.
- step 2120 If at step 2120 the column value g equals the column value / indicating that an ⁇ element that corresponds to the main diagonal has been reached and the process proceeds to step 2130.
- step 2130 a determination is made as to whether the row value / equals the number of features. If the row value does not equal the number of features then the process proceeds to step 2106 where the row value is incremented and the process then proceeds as discussed above in connection with the previous steps outlined. If at step 2130, the row value / does equal the number of features then the updating process has ended, then kemel functions end for the current trial.
- step 2202 processes of the preferred embodiment of the present invention for system monitoring are shown.
- the leamed feature connection weights ⁇ [OUT] and leamed feature variances v [D,OUT] are received from the kemel.
- the process then proceeds to step 2203 to compute ggfeature multiple correlations c[M] according to Equation (23).
- Tolerance band ratios r are computed at step 2204 according to Equation (24).
- step 2206 partial correlations are computed according to Equation (25).
- leaming may disabled if abnormal deviations are detected in system inputs, as discussed above.
- the Mahalanobis distance may be plotted to the output display monitor 14 and also the standard deviance measures may be calculated according to Equations (18) and (19) then displayed. According to user specifications, any desired output of the CIP system may be display at step 2212 for user evaluation. 2.
Abstract
Description
Claims
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA002203776A CA2203776C (en) | 1994-11-02 | 1995-11-01 | Concurrent learning and performance information processing system |
AU41413/96A AU706727C (en) | 1994-11-02 | 1995-11-01 | Concurrent learning and performance information processing system |
EP95939691A EP0789869A4 (en) | 1994-11-02 | 1995-11-01 | Concurrent learning and performance information processing system |
JP08515395A JP2000511307A (en) | 1994-11-02 | 1995-11-01 | Simultaneous learning and performance information processing system |
NZ296732A NZ296732A (en) | 1994-11-02 | 1995-11-01 | Information processing system involving a time trial of learning and performance |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/333,204 US5835902A (en) | 1994-11-02 | 1994-11-02 | Concurrent learning and performance information processing system |
US08/333,204 | 1994-11-02 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO1996014616A1 true WO1996014616A1 (en) | 1996-05-17 |
Family
ID=23301789
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US1995/014160 WO1996014616A1 (en) | 1994-11-02 | 1995-11-01 | Concurrent learning and performance information processing system |
Country Status (7)
Country | Link |
---|---|
US (2) | US5835902A (en) |
EP (1) | EP0789869A4 (en) |
JP (1) | JP2000511307A (en) |
CN (1) | CN1229482A (en) |
CA (1) | CA2203776C (en) |
NZ (1) | NZ296732A (en) |
WO (1) | WO1996014616A1 (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1998022885A1 (en) * | 1996-11-20 | 1998-05-28 | Jannarone Robert J | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
US6216119B1 (en) | 1997-11-19 | 2001-04-10 | Netuitive, Inc. | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
US7624174B2 (en) | 2003-05-22 | 2009-11-24 | Microsoft Corporation | Self-learning method and system for detecting abnormalities |
CN107249434A (en) * | 2015-02-12 | 2017-10-13 | 皇家飞利浦有限公司 | Robust classification device |
WO2020084287A1 (en) * | 2018-10-24 | 2020-04-30 | Arm Limited | System, method and apparatus for decreasing power consumption |
CN111401221A (en) * | 2020-03-12 | 2020-07-10 | 重庆农村商业银行股份有限公司 | Card ticket identification method, device, equipment and storage medium |
US10747845B2 (en) | 2018-08-31 | 2020-08-18 | Arm Limited | System, method and apparatus for computationally efficient data manipulation |
US11640533B2 (en) | 2018-08-03 | 2023-05-02 | Arm Limited | System, method and apparatus for training neural networks using multiple datasets |
Families Citing this family (78)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5835902A (en) * | 1994-11-02 | 1998-11-10 | Jannarone; Robert J. | Concurrent learning and performance information processing system |
US6008817A (en) * | 1997-12-31 | 1999-12-28 | Comparative Visual Assessments, Inc. | Comparative visual assessment system and method |
EP0935202A1 (en) * | 1998-01-19 | 1999-08-11 | Sony France S.A. | Hardware or software architecture implementing self-biased conditioning |
JP3349455B2 (en) * | 1998-09-30 | 2002-11-25 | 宮崎沖電気株式会社 | Management method and management system for semiconductor manufacturing equipment |
US6269479B1 (en) * | 1998-11-30 | 2001-07-31 | Unisys Corporation | Method and computer program product for evaluating the performance of an object-oriented application program |
US6591255B1 (en) | 1999-04-05 | 2003-07-08 | Netuitive, Inc. | Automatic data extraction, error correction and forecasting system |
US6850891B1 (en) | 1999-07-23 | 2005-02-01 | Ernest H. Forman | Method and system of converting data and judgements to values or priorities |
US6957172B2 (en) | 2000-03-09 | 2005-10-18 | Smartsignal Corporation | Complex signal decomposition and modeling |
US6876988B2 (en) | 2000-10-23 | 2005-04-05 | Netuitive, Inc. | Enhanced computer performance forecasting system |
US6928398B1 (en) * | 2000-11-09 | 2005-08-09 | Spss, Inc. | System and method for building a time series model |
US7660734B1 (en) * | 2000-12-20 | 2010-02-09 | Demandtec, Inc. | System for creating optimized promotion event calendar |
US7877286B1 (en) | 2000-12-20 | 2011-01-25 | Demandtec, Inc. | Subset optimization system |
JP4878085B2 (en) | 2001-04-20 | 2012-02-15 | ラピスセミコンダクタ株式会社 | Management method for manufacturing process |
EP1410325A2 (en) * | 2001-06-08 | 2004-04-21 | Netuitive, Inc. | Automated analyzers for estimation systems |
US6644974B2 (en) | 2001-07-24 | 2003-11-11 | International Business Machines Corporation | Educational process that produces deliverable document |
US20030028353A1 (en) * | 2001-08-06 | 2003-02-06 | Brian Gventer | Production pattern-recognition artificial neural net (ANN) with event-response expert system (ES)--yieldshieldTM |
AU2002317618A1 (en) * | 2001-08-06 | 2003-02-24 | Mercury Interactive Corporation | System and method for automated analysis of load testing results |
US7577631B2 (en) * | 2001-09-10 | 2009-08-18 | Feldhake Michael J | Cognitive image filtering |
JP3891807B2 (en) | 2001-09-14 | 2007-03-14 | ジーイー・メディカル・システムズ・グローバル・テクノロジー・カンパニー・エルエルシー | Superconducting magnet failure prediction apparatus and method, and magnetic resonance imaging system |
AU2002360691A1 (en) * | 2001-12-19 | 2003-07-09 | Netuitive Inc. | Method and system for analyzing and predicting the behavior of systems |
US7031884B1 (en) * | 2002-07-12 | 2006-04-18 | Netscout Systems, Inc. | System and method for analyzing data streams |
US20040054515A1 (en) * | 2002-09-18 | 2004-03-18 | Todi Rajat Kumar | Methods and systems for modeling the performance of a processor |
US7475390B2 (en) * | 2004-01-12 | 2009-01-06 | International Business Machines Corporation | System and method for automatic natural language translation during information transfer |
US8689125B2 (en) * | 2003-10-23 | 2014-04-01 | Google Inc. | System and method for automatic information compatibility detection and pasting intervention |
US7496230B2 (en) * | 2003-06-05 | 2009-02-24 | International Business Machines Corporation | System and method for automatic natural language translation of embedded text regions in images during information transfer |
WO2005008420A2 (en) * | 2003-07-09 | 2005-01-27 | Smartsignal Corporation | Information processing apparatus and method |
WO2005045678A2 (en) * | 2003-10-27 | 2005-05-19 | Netuitive, Inc. | Computer performance estimation system configured to take expected events into consideration |
US20050102630A1 (en) * | 2003-11-06 | 2005-05-12 | International Busainess Machines Corporation | Meta window for merging and consolidating multiple sources of information |
US7478336B2 (en) * | 2003-11-06 | 2009-01-13 | International Business Machines Corporation | Intermediate viewer for transferring information elements via a transfer buffer to a plurality of sets of destinations |
US7340685B2 (en) * | 2004-01-12 | 2008-03-04 | International Business Machines Corporation | Automatic reference note generator |
US7346853B2 (en) * | 2004-01-12 | 2008-03-18 | International Business Machines Corporation | Online learning monitor |
US7996814B1 (en) | 2004-12-21 | 2011-08-09 | Zenprise, Inc. | Application model for automated management of software application deployments |
US7565333B2 (en) * | 2005-04-08 | 2009-07-21 | Caterpillar Inc. | Control system and method |
US7877239B2 (en) | 2005-04-08 | 2011-01-25 | Caterpillar Inc | Symmetric random scatter process for probabilistic modeling system for product design |
US8364610B2 (en) | 2005-04-08 | 2013-01-29 | Caterpillar Inc. | Process modeling and optimization method and system |
US8209156B2 (en) | 2005-04-08 | 2012-06-26 | Caterpillar Inc. | Asymmetric random scatter process for probabilistic modeling system for product design |
US7818131B2 (en) * | 2005-06-17 | 2010-10-19 | Venture Gain, L.L.C. | Non-parametric modeling apparatus and method for classification, especially of activity state |
CA2656832A1 (en) * | 2005-07-08 | 2007-01-18 | Brainlike, Inc. | Efficient processing in an auto-adaptive network |
US7487134B2 (en) | 2005-10-25 | 2009-02-03 | Caterpillar Inc. | Medical risk stratifying method and system |
US7499842B2 (en) | 2005-11-18 | 2009-03-03 | Caterpillar Inc. | Process model based virtual sensor and method |
EP1954184B1 (en) | 2005-11-29 | 2017-10-11 | PhysIQ Inc. | Residual-based monitoring of human health |
US8280805B1 (en) | 2006-01-10 | 2012-10-02 | Sas Institute Inc. | Computer-implemented risk evaluation systems and methods |
US7505949B2 (en) | 2006-01-31 | 2009-03-17 | Caterpillar Inc. | Process model error correction method and system |
US9785951B1 (en) | 2006-02-28 | 2017-10-10 | International Business Machines Corporation | Scalable tuning engine |
US9858579B1 (en) | 2006-02-28 | 2018-01-02 | International Business Machines Corporation | Plan tuning engine |
US7702631B1 (en) | 2006-03-14 | 2010-04-20 | Google Inc. | Method and system to produce and train composite similarity functions for product normalization |
US7912773B1 (en) * | 2006-03-24 | 2011-03-22 | Sas Institute Inc. | Computer-implemented data storage systems and methods for use with predictive model systems |
US8275577B2 (en) | 2006-09-19 | 2012-09-25 | Smartsignal Corporation | Kernel-based method for detecting boiler tube leaks |
US7467067B2 (en) * | 2006-09-27 | 2008-12-16 | Integrien Corporation | Self-learning integrity management system and related methods |
US7707285B2 (en) * | 2006-09-27 | 2010-04-27 | Integrien Corporation | System and method for generating and using fingerprints for integrity management |
US8478506B2 (en) | 2006-09-29 | 2013-07-02 | Caterpillar Inc. | Virtual sensor based engine control system and method |
US20080080389A1 (en) * | 2006-10-03 | 2008-04-03 | Hart Richard D | Methods and apparatus to develop management rules for qualifying broadband services |
US8311774B2 (en) | 2006-12-15 | 2012-11-13 | Smartsignal Corporation | Robust distance measures for on-line monitoring |
US7483774B2 (en) | 2006-12-21 | 2009-01-27 | Caterpillar Inc. | Method and system for intelligent maintenance |
US7787969B2 (en) | 2007-06-15 | 2010-08-31 | Caterpillar Inc | Virtual sensor system and method |
US7831416B2 (en) | 2007-07-17 | 2010-11-09 | Caterpillar Inc | Probabilistic modeling system for product design |
US7788070B2 (en) | 2007-07-30 | 2010-08-31 | Caterpillar Inc. | Product design optimization method and system |
US7542879B2 (en) | 2007-08-31 | 2009-06-02 | Caterpillar Inc. | Virtual sensor based control system and method |
US7593804B2 (en) | 2007-10-31 | 2009-09-22 | Caterpillar Inc. | Fixed-point virtual sensor control system and method |
US8224468B2 (en) | 2007-11-02 | 2012-07-17 | Caterpillar Inc. | Calibration certificate for virtual sensor network (VSN) |
US8036764B2 (en) | 2007-11-02 | 2011-10-11 | Caterpillar Inc. | Virtual sensor network (VSN) system and method |
US8521631B2 (en) | 2008-05-29 | 2013-08-27 | Sas Institute Inc. | Computer-implemented systems and methods for loan evaluation using a credit assessment framework |
US8086640B2 (en) | 2008-05-30 | 2011-12-27 | Caterpillar Inc. | System and method for improving data coverage in modeling systems |
US8631117B2 (en) * | 2008-08-19 | 2014-01-14 | Vmware, Inc. | System and method for correlating fingerprints for automated intelligence |
US7917333B2 (en) | 2008-08-20 | 2011-03-29 | Caterpillar Inc. | Virtual sensor network (VSN) based control system and method |
US20100332423A1 (en) * | 2009-06-24 | 2010-12-30 | Microsoft Corporation | Generalized active learning |
US8793004B2 (en) | 2011-06-15 | 2014-07-29 | Caterpillar Inc. | Virtual sensor system and method for generating output parameters |
US8620853B2 (en) | 2011-07-19 | 2013-12-31 | Smartsignal Corporation | Monitoring method using kernel regression modeling with pattern sequences |
US9256224B2 (en) | 2011-07-19 | 2016-02-09 | GE Intelligent Platforms, Inc | Method of sequential kernel regression modeling for forecasting and prognostics |
US8660980B2 (en) | 2011-07-19 | 2014-02-25 | Smartsignal Corporation | Monitoring system using kernel regression modeling with pattern sequences |
US9250625B2 (en) | 2011-07-19 | 2016-02-02 | Ge Intelligent Platforms, Inc. | System of sequential kernel regression modeling for forecasting and prognostics |
US9195436B2 (en) * | 2013-10-14 | 2015-11-24 | Microsoft Technology Licensing, Llc | Parallel dynamic programming through rank convergence |
US10068175B2 (en) | 2014-02-20 | 2018-09-04 | International Business Machines Corporation | Question resolution processing in deep question answering systems |
JP5816771B1 (en) * | 2015-06-08 | 2015-11-18 | 株式会社Preferred Networks | Learning device unit |
US10282614B2 (en) * | 2016-02-18 | 2019-05-07 | Microsoft Technology Licensing, Llc | Real-time detection of object scanability |
DE102016223079B4 (en) * | 2016-11-23 | 2024-03-28 | Robert Bosch Gmbh | Method and device for determining an association between a matrix element of a matrix and a comparison matrix element of a comparison matrix using a correspondence table |
CN107818380B (en) * | 2017-09-29 | 2021-01-15 | 深圳和而泰智能控制股份有限公司 | Information processing method and server |
CN112305970A (en) * | 2020-03-12 | 2021-02-02 | 沈寿娟 | Internet of things intelligent linkage control method and Internet of things intelligent linkage control center |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5148514A (en) * | 1989-05-15 | 1992-09-15 | Mitsubishi Denki Kabushiki Kaisha | Neural network integrated circuit device having self-organizing function |
US5155802A (en) * | 1987-12-03 | 1992-10-13 | Trustees Of The Univ. Of Penna. | General purpose neural computer |
US5170463A (en) * | 1988-12-29 | 1992-12-08 | Sharp Kabushiki Kaisha | Neuro-computer |
US5241509A (en) * | 1988-11-18 | 1993-08-31 | U.S. Philips Corporation | Arrangement of data cells and neural network system utilizing such an arrangement |
US5253328A (en) * | 1989-11-17 | 1993-10-12 | Microelectronics & Computer Technology Corp. | Neural-network content-addressable memory |
US5274746A (en) * | 1989-12-26 | 1993-12-28 | Mitsubishi Denki Kabushiki Kaisha | Coupling element for semiconductor neural network device |
US5475793A (en) * | 1989-02-10 | 1995-12-12 | The Secretary Of State For Defence In Her Britannic Majesty's Government Of The United Kingdom Of Great Britain And Northern Ireland | Heuristic digital processor using non-linear transformation |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4903226A (en) * | 1987-08-27 | 1990-02-20 | Yannis Tsividis | Switched networks |
US4875183A (en) * | 1987-11-19 | 1989-10-17 | American Telephone And Telegraph Company, At&T Bell Laboratories | Neural networks |
DE69031842T2 (en) * | 1989-02-20 | 1998-04-16 | Fujitsu Ltd | Learning system and learning method for a data processing device |
US5151969A (en) * | 1989-03-29 | 1992-09-29 | Siemens Corporate Research Inc. | Self-repairing trellis networks |
US5222195A (en) * | 1989-05-17 | 1993-06-22 | United States Of America | Dynamically stable associative learning neural system with one fixed weight |
US5214743A (en) * | 1989-10-25 | 1993-05-25 | Hitachi, Ltd. | Information processing apparatus |
JPH03236275A (en) * | 1990-02-14 | 1991-10-22 | Olympus Optical Co Ltd | Memory element and memory device |
JPH03250243A (en) * | 1990-02-27 | 1991-11-08 | Toshiba Corp | Neural circuit network arithmetic unit |
JP3210319B2 (en) * | 1990-03-01 | 2001-09-17 | 株式会社東芝 | Neurochip and neurocomputer using the chip |
US5243688A (en) * | 1990-05-22 | 1993-09-07 | International Business Machines Corporation | Virtual neurocomputer architectures for neural networks |
JPH04195246A (en) * | 1990-09-28 | 1992-07-15 | Olympus Optical Co Ltd | Optimizing device |
US5157738A (en) * | 1990-12-18 | 1992-10-20 | Trustees Of Boston University | Rapid category learning and recognition system |
US5087826A (en) * | 1990-12-28 | 1992-02-11 | Intel Corporation | Multi-layer neural network employing multiplexed output neurons |
EP0567557A1 (en) * | 1991-01-15 | 1993-11-03 | Trustees Of Boston University | Vector associative map system |
JP3088171B2 (en) * | 1991-02-12 | 2000-09-18 | 三菱電機株式会社 | Self-organizing pattern classification system and classification method |
US5408588A (en) * | 1991-06-06 | 1995-04-18 | Ulug; Mehmet E. | Artificial neural network method and architecture |
US5195170A (en) * | 1991-08-12 | 1993-03-16 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Neural-network dedicated processor for solving assignment problems |
DE4230419A1 (en) * | 1992-09-11 | 1994-03-17 | Siemens Ag | Neural network with rule-base network configuration |
US5461696A (en) * | 1992-10-28 | 1995-10-24 | Motorola, Inc. | Decision directed adaptive neural network |
US5448684A (en) * | 1993-11-12 | 1995-09-05 | Motorola, Inc. | Neural network, neuron, and method for recognizing a missing input valve |
US5701398A (en) * | 1994-07-01 | 1997-12-23 | Nestor, Inc. | Adaptive classifier having multiple subnetworks |
US5796920A (en) * | 1994-08-19 | 1998-08-18 | Harris Corporation | Multiprocessor system and method for identification and adaptive control of dynamic systems |
US5430830A (en) * | 1994-08-22 | 1995-07-04 | Motorola, Inc. | Adaptive weight adjusting circuit for an neural network |
US5835902A (en) * | 1994-11-02 | 1998-11-10 | Jannarone; Robert J. | Concurrent learning and performance information processing system |
-
1994
- 1994-11-02 US US08/333,204 patent/US5835902A/en not_active Expired - Lifetime
-
1995
- 1995-11-01 WO PCT/US1995/014160 patent/WO1996014616A1/en active Application Filing
- 1995-11-01 CN CN95196552A patent/CN1229482A/en active Pending
- 1995-11-01 NZ NZ296732A patent/NZ296732A/en not_active IP Right Cessation
- 1995-11-01 CA CA002203776A patent/CA2203776C/en not_active Expired - Fee Related
- 1995-11-01 EP EP95939691A patent/EP0789869A4/en not_active Withdrawn
- 1995-11-01 JP JP08515395A patent/JP2000511307A/en active Pending
-
1998
- 1998-08-20 US US09/137,265 patent/US6289330B1/en not_active Expired - Lifetime
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5155802A (en) * | 1987-12-03 | 1992-10-13 | Trustees Of The Univ. Of Penna. | General purpose neural computer |
US5241509A (en) * | 1988-11-18 | 1993-08-31 | U.S. Philips Corporation | Arrangement of data cells and neural network system utilizing such an arrangement |
US5170463A (en) * | 1988-12-29 | 1992-12-08 | Sharp Kabushiki Kaisha | Neuro-computer |
US5475793A (en) * | 1989-02-10 | 1995-12-12 | The Secretary Of State For Defence In Her Britannic Majesty's Government Of The United Kingdom Of Great Britain And Northern Ireland | Heuristic digital processor using non-linear transformation |
US5148514A (en) * | 1989-05-15 | 1992-09-15 | Mitsubishi Denki Kabushiki Kaisha | Neural network integrated circuit device having self-organizing function |
US5253328A (en) * | 1989-11-17 | 1993-10-12 | Microelectronics & Computer Technology Corp. | Neural-network content-addressable memory |
US5274746A (en) * | 1989-12-26 | 1993-12-28 | Mitsubishi Denki Kabushiki Kaisha | Coupling element for semiconductor neural network device |
Non-Patent Citations (1)
Title |
---|
See also references of EP0789869A4 * |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1998022885A1 (en) * | 1996-11-20 | 1998-05-28 | Jannarone Robert J | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
AU721842B2 (en) * | 1996-11-20 | 2000-07-13 | Robert J. Jannarone | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
US6216119B1 (en) | 1997-11-19 | 2001-04-10 | Netuitive, Inc. | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
US6647377B2 (en) | 1997-11-19 | 2003-11-11 | Netuitive, Inc. | Multi-kernel neural network concurrent learning, monitoring, and forecasting system |
US7624174B2 (en) | 2003-05-22 | 2009-11-24 | Microsoft Corporation | Self-learning method and system for detecting abnormalities |
CN107249434A (en) * | 2015-02-12 | 2017-10-13 | 皇家飞利浦有限公司 | Robust classification device |
CN107249434B (en) * | 2015-02-12 | 2020-12-18 | 皇家飞利浦有限公司 | Robust classifier |
US11640533B2 (en) | 2018-08-03 | 2023-05-02 | Arm Limited | System, method and apparatus for training neural networks using multiple datasets |
US10747845B2 (en) | 2018-08-31 | 2020-08-18 | Arm Limited | System, method and apparatus for computationally efficient data manipulation |
WO2020084287A1 (en) * | 2018-10-24 | 2020-04-30 | Arm Limited | System, method and apparatus for decreasing power consumption |
US10970201B2 (en) | 2018-10-24 | 2021-04-06 | Arm Limited | System, method and apparatus for data manipulation |
CN111401221A (en) * | 2020-03-12 | 2020-07-10 | 重庆农村商业银行股份有限公司 | Card ticket identification method, device, equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
JP2000511307A (en) | 2000-08-29 |
AU4141396A (en) | 1996-05-31 |
EP0789869A4 (en) | 2006-09-13 |
US6289330B1 (en) | 2001-09-11 |
US5835902A (en) | 1998-11-10 |
CA2203776A1 (en) | 1996-05-17 |
EP0789869A1 (en) | 1997-08-20 |
CN1229482A (en) | 1999-09-22 |
AU706727B2 (en) | 1999-06-24 |
NZ296732A (en) | 1999-11-29 |
CA2203776C (en) | 2002-10-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US5835902A (en) | Concurrent learning and performance information processing system | |
US7082419B1 (en) | Neural processing element for use in a neural network | |
Singh et al. | Learning without state-estimation in partially observable Markovian decision processes | |
US4744026A (en) | Methods and apparatus for efficient resource allocation | |
Pottmann et al. | A nonlinear predictive control strategy based on radial basis function models | |
US5479571A (en) | Neural node network and model, and method of teaching same | |
US20040093315A1 (en) | Neural network training | |
Peng et al. | A new Jacobian matrix for optimal learning of single-layer neural networks | |
Bontempi et al. | Local learning for iterated time series prediction | |
EP0384709B1 (en) | Learning Machine | |
Khan | Particle swarm optimisation based feature selection for software effort prediction using supervised machine learning and ensemble methods: A comparative study | |
Lemke et al. | Self-organizing data mining for a portfolio trading system | |
AU706727C (en) | Concurrent learning and performance information processing system | |
Rogers et al. | Parallel Processing in a control systems environment | |
JPH076146A (en) | Parallel data processing system | |
Chen et al. | An action strategy generation framework for an on-line scheduling and control system in batch processes with neural networks | |
Khoshgoftaar et al. | Predictive modeling of software quality for very large telecommunications systems | |
Bontempi et al. | Lazy learning: a logical method for supervised learning | |
Panigrahi et al. | Hierarchical decision making in semiconductor fabs using multi-time scale Markov decision processes | |
KR20220155650A (en) | Method for calculating important variable of financial data using machine learning and reinforcement learning based on statistics | |
Wang et al. | Feature-Embedding Triplet Networks with a Separately Constrained Loss Function | |
Hachem et al. | Neural network implementation for reliability evaluation | |
Shi et al. | FPGA based run time reconfigurable gas discrimination system | |
Breitner et al. | Training of Large Three-layer Perceptrons with the Neuro-computer SYNAPSE | |
JPH04112362A (en) | Accumulation circuit |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 95196552.2 Country of ref document: CN |
|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AL AM AT AU BB BG BR BY CA CH CN CZ DE DK EE ES FI GB GE HU IS JP KE KG KP KR KZ LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK TJ TM TT UA UG US UZ VN |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): KE LS MW SD SZ UG AT BE CH DE DK ES FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN ML MR NE SN TD TG |
|
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) | ||
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 296732 Country of ref document: NZ |
|
ENP | Entry into the national phase |
Ref document number: 2203776 Country of ref document: CA Ref document number: 2203776 Country of ref document: CA Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 1996 515395 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1995939691 Country of ref document: EP |
|
WWP | Wipo information: published in national office |
Ref document number: 1995939691 Country of ref document: EP |
|
REG | Reference to national code |
Ref country code: DE Ref legal event code: 8642 |