DE19610849C1 - Iterative determination of optimised network architecture of neural network by computer - Google Patents

Abstract

The method involves performing iterations by computer. In each iteration step, neural nets of different architectures are generated using a probability vector and an evolutionary method in which each vector component probability is described to form neurons and/or neuron links. For each neural net, a weighting vector is optimised using a training data set and a fitness function determined for the optimised weighting vector. The optimised network architecture of a neural net results from the neural net with a minimal fitness function value if the value lies below a defined level. A further probability vector is determined for generating neural nets in a further iteration step taking into account the value of the fitness function, if the value is not below the defined level.

Description

The invention relates to a method for determining a Neural network, which has the best possible complexi with the best possible generalization of the neural Network with regard to new input data records.

For a neural network with a given architecture exist ren various methods to optimize the weights of the Neural network based on a training data set. It is however, no automatic approach for determining an opti architecture of a neural network for a speci fishing use case, i.e. for the respective training ten known. Under architecture of the neural network is in following the combination of neurons and connections between understand the individual neurons of the neural network the.

To optimize the network architecture, it is necessary to have one to consider high-dimensional discrete space of variables term. In the optimization, the optimal possible com promiss sought between the following extreme cases. Is this Neural network too small, it is not possible to create a complex one To reproduce the structure of the training data records. Is that neuro nale network too large, a tendency can be observed that through the "oversized" neural network "noise" of the Training data records is simulated. This situation will referred to as over-learning. This will result in a reduction in the Generalization property of the neural network.

There are methods for so-called pruning weights of a Neural network known, for example from the document [1].  

A major disadvantage of the pruning process is there too see that the pruning itself is not fully automated leads, but always only interactively with one User. The user needs very special expertise about pruning. Thus, procedures based on the Pruning based, not feasible for laypeople.

Furthermore, a method for iteratively determining a optimized network architecture of a neural network, where the network architecture uses an evolutionary process is optimized and an optimization of the for each individual Weights are weighted using a gradient descent method leads [2].

This procedure uses special mutation operators used, resulting in a complicated process and therefore too an increased computing time when performing Ver driving through a computer.

Another disadvantage of the method is that this procedure only for the classification of data sets is used and suitable after the training phase. For a fine approximation of one through the training data sets Given the mapping function, this method is not ge is suitable.

There is also a continuity in the development of new individuals and therefore a constant improvement of the individuals is not ge ensures.

There is also a so-called population-based increment The learning process (population-based incremental learning, PBIL) is known [3].  

There are several methods for optimizing weights known to a given network architecture, for example from [4] and [5].

From Zhang, Byoung-Tak; Mühlenbein, Heinz, Evolving Optimal Neural Networks Using Genetic Algorithms with Occam’s Razor, In: Complex Systems 7, pp. 199-220, 1993 it is known neural networks for the classification of input patterns under Optimize the use of genetic algorithms, whereby in the fitness function the training errors of the training phase of the neural networks with the training data record is done.

Other methods for generating neural networks under Ver genetic algorithms are from Shamir, Nachum; Saad, David; Marom, Emanuel, Using the Functional Behavior of Neu rons for Genetic Recombination in Neural Nets Training, In: Complex Systems 7, pp. 445-467, 1993 and US 51 40 530 knows.

These methods have significant disadvantages, in particular regarding the complexity of the procedures that these procedures only suitable for relatively small examples. For These processes lead to more complex application examples nem very considerable computing need, which the implementation of the Processes can be considerably cost-intensive. Furthermore are the procedures only for the classification of input signals len and not to approximate nonlinear functions is suitable because the generalization ability of the mitit there are no neural networks.

The invention is therefore based on the problem of a method to determine an optimized network architecture of a neuro nal network to specify which is quick, easy to implement and suitable for the approximation of nonlinear functions is.  

This problem is solved by the method according to claim 1 solved.

In the method according to the invention, iteratively using egg an optimized network architecture of a neural network. Every Ite ration step the individual individuals of the neural networks generated, the individual neurons and / or connections of the neural networks with a probability will or may not be generated in a true probability vector can be specified. The network architecture is encoded in an architecture vector which is only binary Has values. For each neural network that is created by the specific architecture vector is characterized the weights of the neural network using known methods optimized. Then a value of a fitness function determined by the suitability of the network architecture of the new ronal network for the application is described. In the fitness function will at least one determined with a validation data set Validation error be considered. As the starting architecture for the generation of the next generation with individuals, their neurons and / or Connections are generated with a probability each in a further probability vector is given, the information about the most predecessors of neural networks.

A significant advantage of the method according to the invention is in its simplicity and thus in the low computing time allowed to perform the procedure with a computer see. The simplicity is mainly due to the binary code justification of the respective network architecture.

Another advantage is that the process is fully auto matically without interaction with a user with special Expertise can be carried out.  

Furthermore, the method according to the invention makes it possible a mapping function given by the training data sets approximate very finely.

Advantageous further developments of the method according to the invention result from the dependent claims.

It is advantageous to consider the further probability vector by a moving averaging over a be any number of previous probability vectors form, with which a fine continuity in the development of the Individuals of each generation is guaranteed.

It is also advantageous for the probability vector ren to carry out a so-called mutation, whereby a fi Fixing the values of the components of the probability vector tors with the value 0 or the value 1 is avoided.

It is also advantageous that the values of the weight vector increase Start of optimization of a new generation of neurons Networks result from a moving averaging over any number of weight vectors Generations of Neural Networks. It is advantageous as respective starting point the values of the weight vector of the best neural network of the previous genera tion to use. This speeds up the process because the optimization of the weight vectors is shortened.

It is also advantageous, at least in the fitness function the complexity of the network architecture and / or the rest of the trail errors due to this optimization parameter enable a very fine and very good approximation.

Reliability is further improved when the trai nings data set and the validation data set disjoint sets of data are.  

By using a quasi-Newton method (QNM) the Optimization of the respective weight vector significantly verbes accelerates and accelerates.

Become the training record and the validation record in each iteration step using cross-validation averaged, an adaptation to the "noise" of the fig function, which is supported by the data records for a finite number Data given is reduced.

An embodiment of the method according to the invention is shown in the figures and will be explained in more detail below tert.

Show it

Fig. 1 is a flowchart in which the individual procedural steps of the inventive method are shown;

Fig. 2 is a sketch showing a computer with which the method is necessarily carried out.

Fig. 3 is a sketch in which a neural network is shown as an example.

In Fig. 1, the individual process steps of the inventive method are shown.

The process becomes an evolutionary process of education different generations of neural networks NE each with different network architecture used. The procedure is carried out iteratively, each for a new generation ronal networks NE.  

In each iteration step t are in a first step 101 the different neural networks NE of a generation, which correspond to the iteration step t, from a computer R, with which the inventive method necessarily is carried out according to the respective evolutionary procedure ren formed.

The network architecture of each neural network NE is described by an architecture vector x , the components of which only have the value 0 or the value 1. A component of the architecture vector x uniquely represents each of the neurons NEU of the neural network NE and / or each connection V _ÿ between the neurons NEU, which are characterized by a first index i and a second index j. The first index i and the second index j are natural numbers (cf. FIG. 3).

The value 0 indicates that either the corresponding neuron NEU does not exist or that the corresponding connection V _ÿ does not exist. The value 1, on the other hand, indicates that either the corresponding neuron NEU exists or that the corresponding connection V _ÿ exists.

To generate the individual neural networks NE, a probability vector p _{t is} used which has the same power (number of components) as the architecture vector x . In the probability vector p _t , a component is given in each component, with which in each iteration step t a neuron NEW and / or a connection V _{ÿ is} generated between the neurons NEW when generating the respective neural network NE.

For the simple example shown in FIG. 3 of a neuronal network NE with 5 neurons NEU1, NEU2, NEU3, NEU4 and NEU5, distributed over three layers, an input layer IL, a hidden layer HL and an output layer OL there is for the simplified Case that only the connections V _{ÿ are described} in the architecture vector x :

x = (V₁₁, V₁₂, V₁₃, V₁₄, V₁₅, V₂₃, V₂₄, V₂₅, V₃₄, V₃₅, V₄₅) = = (0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1) (1)

Furthermore, a weight w _{ÿ is} assigned to the individual connections V _{ÿ of} the neural network NE. Each input signal at the beginning of a connection V _ÿ , which is identified by the first index i, is multiplied by the corresponding weight w _ÿ and then supplied to the connection V _ÿ by the neuron NEj identified by the second index j. The individual weights w _ÿ are combined to form a weight vector w .

The following weight vector w generally results for the simple example from FIG. 3:

w = (w₁₁, w₁₂, w₁₃, w₁₄, w₁₅, w₂₃, w₂₄, w₂₅, w₃₄, w₃₅, w₄₅) (2)

A mapping vector w _sp characterizing the mapping function of the respective neural network NE results from the architecture vector x and the weight vector w :

w _sp = w ^T * x (3)

being with
w ^{T is} the transposed weight vector w .

The mapping vector w _sp results for the example case:
w _sp = (0, 0, w₁₃, w₁₄, 0, w₂₃, w₂₄, 0, 0, w₃₅, w₄₅).

After the different neural networks NE of a generation corresponding to the iteration step t have been formed by the computer R in accordance with the respective evolutionary method (101) in each iteration step t, the following steps are carried out for each neuronal network NE of the iteration step t ( 102):
The individual weights w _{ÿ of} the respective neural network NE are optimized on the basis of a predetermined training data set TDS (103). This training phase can be carried out using a wide variety of training methods known to the person skilled in the art. So-called gradient descent methods are usually used to optimize an error function E, which indicates the deviation of a respective output value y _{l of} the neural network NE from a predetermined target value t _l when a training date z _{l is created} . A training date is clearly identified with an index l.

In the event that the training data set TDS has a number s of training data z _l with target values t _l assigned to the training data z _l , the following results, for example, as an error function E:

in which
d is an arbitrary natural number.

A non-exhaustive overview of various gradient descent methods can be found in [4] and in [5]. The Quasi-Newton method (QNM) described in document [5] has the advantage that the method can be carried out very quickly and with good results. Further methods for optimizing the weights w _{ÿ of} the respective neural network NE are known to the person skilled in the art and can be used without restrictions in the method according to the invention.

After the end of the training phase (103), a value of a fitness function F is determined from a residual error that results from the value of the error function E at the end of the training phase and from a validation error E _V (104).

The validation error E _V is determined using a validation data record VDS. In the event that the validation data set VDS has a number of r validation data, it admits the validation error E _V to:

being with
k each validation date is clearly identified,
y _{k is} an output value of the neural network NE when the validation date is created,
t _{k is} a target value of the neural network NE when the validation date is created.

The fitness function F results for different variants of the method according to the invention in different ways. The residual error and the validation error E _{V can} flow into the fitness function F. For the special case that only the validation error E _{V flows} into the error function F, the result for the fitness function F is:

F = E _V (6).

However, it is in a variant of the method according to the invention rens also provided the complexity of each neurona len network NE to be taken into account in the fitness function F. In this case there is, for example, fitness function F:

being with
δ an influencing factor is described, which is an arbitrary natural number with which the extent to which the complexity of the respective neural network NE is to be taken into account in the fitness function F is specified.

Furthermore, it is also provided in a variant of the method according to the invention to take into account both the residual error and the validation error E _V and the complexity of the respective neural network NE in the fitness function F. The fitness function F then results from equation (6) and from equation (7):

X _ÿ denotes a component of the architecture vector x which is clearly identified by the indices.

Is the value of the fitness function F of a neural network NE smaller than a predeterminable barrier SCH (106), this is ent speaking neural network NE that for the application New ronal NE network with optimal network architecture, and the process ren has ended (107).  

Are the values of the fitness function F of all neural networks NE of a generation t larger than the predefinable barrier SCH, the optimal network architecture has not yet been determined, and at least one further iteration t + 1 must be carried out will.

A further probability vector p _{t + 1 is} determined for the further iteration step t + 1 (105).

Different variants are provided for forming the further probability vector P _{t + 1} from the probability vector p _t . For example, the individual components p _{i, t + 1 of} the further probability vector p _{t + 1 can} result from a moving averaging over the corresponding components p _{i, t + 1 of} at least one previous probability vector p _t . This can advantageously be done by using any function of a digital filter. When determining the further probability vector p _{t + 1} , the probability vector p _t that was assigned to the neural network NE that has the best value of the fitness function F is advantageously used in each case.

It has proven to be very simple and therefore advantageous that the individual components p _{i, t + 1 of} the further probability vector p _{t + 1 are} formed according to the rule:

p _{i, t + 1} = (1 - α) p _{i, t} + αx _{best, t} (9)

being with
p _{i, t + 1} each denotes the component of the further probability vector p _{t + 1} for the iteration step _{t + 1} , t denotes the iteration index,
i denotes a component index with which each component of the probability vector p _t or the further probability vector p _{t + 1 is} uniquely identified,
α is a decay constant,
p _{i, t} each denotes a component of the probability vector,
x _{i, best, t} a component of the architecture vector of the best neural network NE of an iteration step t is identified.

The decay constant α is an arbitrary number.

The individual connections V _ÿ and / or the individual neurons NEW are each generated in the next iteration step with the probability that is specified for the respective connection V _ÿ and / or the respective neuron NEW in the further probability vector p _{t + 1} .

In a development of the method according to the invention, it is provided to carry out a so-called mutation step to change the components p _{i, t + 1} when forming the components p _{i, t + 1 of} the further probability vector p _{t + 1} .

The mutation serves, for example, to avoid fixing the values of the components p _{i, t + 1} to the value 0 or the value 1. The following rule for mutating components p _{i, t + 1} has proven to be advantageous:

pm + 1 _{i, t} = (1 - β) pm _{i, t} + βb (10)

being with
pm + 1 _{i, t + 1} a component p _{i, t + 1 of} the further probability vector p _{t + 1} for the iteration step _{t + 1} , in which a mutation of the component p _{i, t + 1 was} carried out,
t the iteration index is designated,
i denotes the component index with which each component p _{i, t + 1 of} the probability vector p _t or the further probability vector p _{t + 1 is} uniquely identified,
β is a mutation constant,
pm _{i, t} each denotes a component p _{i, t of} the probability vector p _t for the iteration step _t , in which a mutation of the component p _{i, t is} carried out,
b a specifiable number in the interval [0, 1] is designated.

The mutation constant β is an arbitrary number.

Furthermore, it is provided in a further development of the method that the components w _{ÿ, t + 1 of} a further weight vector w _{t + 1 of} each neural network NE at the beginning of the optimization of the individual weights w _{ÿ, t + 1} by means of a moving averaging over the to determine corresponding components w _{ÿ, t of} the weight vector w of the preceding neural network NE with respect to the fitness function F. This can advantageously be done by using any function of a digital filter.

It has proven to be very simple and therefore advantageous that the individual components w _{ÿ, t + 1 of} the further probability vector w t + 1 are formed according to the rule:

w _{ÿ, t + 1} = (1 - η) w _{ÿ, t} + ηw _{best, t}

being with
w _{i, t + 1} each denotes a component of the weight vector w at the start of the optimization of the weight vector w for the iteration step t + 1,
t an iteration index is designated,
i, j a first index and a second are designated, with which each component of the weight vector w is uniquely identified,
η is a weight decay constant,
w _{i, t} each denotes a component of the weight vector w at the beginning of the optimization of the weight vector w for the iteration step t,
w _{best, t} is a component of the weight vector w of each be most neural network NE one iteration t marked in draws.

To further simplify the method according to the invention it is provided in a further development of the method for Generation of the neural networks NE in the individual iterations the so-called population-based increment telle learning processes (population-based incremental learning, PBIL) to use, which is described in the document [3].

It is also advantageous in a further development of the method liable if the training data set TDS and the validation da VDS are disjoint. This will be the result of the trai ning phase and validation further improved.

A further improvement in results is achieved if in each iteration step t the training data record TDS and the VDS validation data record is newly formed. For the In case all available data is already in the training data set TDS or in the validation data set VDS use find it is advantageous to cross-validate each two records to perform in each iteration step t a "new" training data set TDS and one  To get "new" validation data set VDS. With that the Adaptation to the "noise" of the data significantly reduced.

Cross validation is known from document [6].

In Fig. 2, the computer R is shown, with which the procedural ren is necessarily carried out. Furthermore, the training data set TDS and the validation data set VDS are shown symbolically. The optimized neural network NE is presented to the user, for example on a screen BS.

The following publications have been cited in this document:
[1] F. Hergert et al, A Comparison of Weight Elimination Methods for Reducing Complexity in Neural Networks, International Joint Conference on Neural Networks, Baltimore, pp. 1-8, 1992
[2] H. Braun and P. Zagorski, ENZO-M - A Hybrid Approach for Optimizing Neural Networks by Evolution and Learning, Parallel Problem Solving from Nature, Y. Davidor et al (eds.), International Conference on Evolutionary Computation, Proceedings of the Third Conference on Parallel Problem Solving from Nature, Jerusalem, Israel, October 9-14, ISBN 3-540-58484-6, Springer Verlag, pp. 440-451, 1994
[3] S. Baluja and R. Caruana, Removing the Genetics from the Standard Genetic Algorithm, Proceedings of the Twelfth International Conference on Machine Learning, Lake Tahoe, CA, pp. 1-11, July 1995
[4] A. Zell, Neural Network Simulation, Addison-Wesley, 1st edition, ISBN 3-89319-554-8, pp. 105-178, 1994
[5] R. Fletcher, Practical Methods of Optimization, John Wiley, 2nd edition, ISBN 0-471-91547-5, pp. 49-57, 1987
[6] L. Breiman, Classification and Regression Trees, The Wadsworth statistics / probability series, J. Kimmel (ed.), ISBN 0-534-98053-8, p. 11, 1984

Claims (16)

1. Method for iteratively determining an optimized network architecture of a neural network (NE) using a computer (R) with the following steps:

• a) in each iteration step (t), neural networks (NE) of different network architectures are generated (101) using a probability vector ( p _t ), the generation being carried out by means of an evolutionary method, and each component in the probability vector ( p _t ) ( p _{i, t} ) of the probability vector ( p _t ) each describes the probability of forming neurons (NEW) and / or of forming a connection (V _ÿ ) of neurons (NEW) of a neural network (NE),
• b) for each neural network (NE) the following steps are carried out (102):
• a weight vector ( w ) of the neural network (NE) is optimized on the basis of a training data set (TDS) (103),
• a value of a fitness function (F) is determined (104) for the optimized weight vector, at least one validation error (E _V ) determined with a validation data record (VDS) being taken into account in the fitness function (F),
• c) the optimized network architecture of a neural network (NE) results from the neural network (NE) with a minimum value of the fitness function (F) if the value is below a predefinable barrier (SCH) (106, 107), and
• d) a further probability vector ( p _{t + 1} ) for generating neural networks in a further iteration step (t + 1) is determined taking into account the values of the fitness function (F) (105) if the value of the fitness function (F) is not is below the predefinable barrier (SCH) (106).

2. The method according to claim 1, in which components (p _{i, t + 1} ) of the further probability vector ( p _{t + 1} ) are obtained by moving averaging over the corresponding components (p _{i, t} ) of at least one previous probability vector ( p _t ) can be determined. 3. The method according to claim 2, in which the moving averaging by any digital filter function is formed. 4. The method of claim 2, wherein the components (p _{i, t + 1} ) are each formed according to the rule: p _{i, t + 1} = (1 - α) p _{i, t} + αx _{best, t} being with
p _{i, t + 1} each denotes a component of the further probability vector for the iteration step t + 1,
t an iteration index is designated,
i is a component index with which each component of the probability vector is uniquely identified,
α is a decay constant,
p _{i, t} each denotes a component of the probability vector,
x _{best, t} a component of the architecture vector of the best neural network of an iteration step is identified. 5. The method according to any one of claims 1 to 4, in which a mutation of the components (p _{i, t + 1} ) is carried out in the formation of the further probability vector ( p _{t + 1} ). 6. The method according to claim 5, wherein the mutation is carried out according to the following rule: pm + 1 _{i, t} = (1 - β) pm _{i, t} + βbw whereby with
pm + 1 _{i, t + 1} each denotes a component of the probability vector for the iteration step t + 1, in which the component was mutated,
t an iteration index is designated,
i is a component index with which each component of the probability vector is uniquely identified,
β is a mutation constant,
pm _{i, t} each denotes a component of the probability vector for iteration step t, in which the component is mutated,
b a specifiable number from the interval [0, 1] is designated. 7. The method according to any one of claims 1 to 6, in which components (w _{ÿ, t + 1} ) of the weight vector ( w _{t + 1} ) at the beginning of the optimization of the weight vector ( w _{t + 1} ) by a moving average over the corresponding components ten (w _{ÿ, t} ) of at least one weight vector (w _t ) of a previous neural network (NE) are determined. 8. The method according to claim 7, in which the moving averaging by any digital filter function is formed. 9. The method according to claim 7, in which the components (w _{ÿ, t + 1} ) of the weight vector ( w _{t + 1} ) at the beginning of the optimization of the weight vector ( w _{t + 1} ) are each formed according to the rule: w _{ÿ, t +1} = (1 - η) w _{ÿ, t} + ηw _{best, t} being with
w _{ÿ, t + 1} each denotes a component of the weight vector at the beginning of the optimization of the weight vector for the iteration step t + 1,
t an iteration index is designated,
i, j are a first index and a second, with which each component of the probability vector is uniquely identified,
η is a weight decay constant,
w _{i, t} each denotes a component of the weight vector at the beginning of the optimization of the weight vector for the iteration step t,
w _{best, t} a component of the weight vector of the _best neural network of an iteration step is identified. 10. The method according to any one of claims 1 to 9, where in the fitness function (F) there is also a residual exercise errors in the respective neural network (NE) is done. 11. The method according to any one of claims 1 to 10, where in the fitness function (F) the complexity of the respective current neural network (NE) is taken into account. 12. The method according to any one of claims 1 to 11, in which the so-called for generating the neural networks (NE) called population-based incremental learning methods (Population-Based Incremental Learning, PBIL) is used. 13. The method according to any one of claims 1 to 12, where the validation data set (VDS) and the training da tset (TDS) are disjoint. 14. The method according to any one of claims 1 to 13, wherein the optimization of the weight vectors ( w ) is carried out with a gradient descent method. 15. The method according to any one of claims 1 to 14, wherein the optimization of the weight vectors (w) with the Qua si Newton method (QNM) is performed. 16. The method according to any one of claims 1 to 15, for the for the validation data set (VDS) and the trai nings data set (TDS) one in each iteration step (t) Cross validation is performed.