CN102855630A - Method for judging image memorability based on saliency entropy and object bank feature - Google Patents

Abstract

The invention relates to a method for judging image memorability based on saliency entropy and object bank feature. A research proves that the image memorability can be influenced by an object contained in an image and the dispersing degree of the image to human visual attraction, so that the object bank feature is used for expressing the object contained in the image and the visual saliency entropy of the image is used for expressing the dispersing degree of the image to the human visual attraction. The two models have excellent expression for the image memorability, so that the two features are combined for expressing a piece of image, and a memorability value of the image with unknown memorability value is forecast through a training supporting regression vector machine model. The method provided by the invention belongs to the field of computer image processing. According to the technical scheme, the image memorability can be judged; the method can be applied to the industries such as advertising industry and news editing; a suitable image can be selected by a practitioner; and the method has a high commercial value.

Description

A kind of iconic memory sex determination method based on saliency entropy and object bank feature

Technical field

The present invention relates to a kind of iconic memory sex determination method based on saliency entropy and object bank feature, can be applied to all kinds of visible images, the Memorability numerical value of process decision chart picture.

Background technology

The Memorability of image is the new research direction of digital image processing field, and it has a lot of application.Can select easily the image remembered by people as the front cover of magazine such as editor, the image that the advertising man can select easily to be remembered is as propagating poster etc.Therefore, when given piece image, if general-purpose computers determine automatically that can it be remembered by people will be highly significant.

Because the Memorability research of image is a newer research direction, so in the research of this problem, method is not a lot.The certain methods that exists is at present at first extracted the global characteristics (such as SIFT, GIST, HOG etc.) of image, by making up sorter and training pattern, then differentiates the Memorability of a given image.Yet because the Memorability of image is a very complicated problem, be difficult to represent this characteristic with the global characteristics of image, so the judgement effect of the method for existence is relatively poor at present.Therefore need to carry out deep research to the Memorability of image, propose the method for new iconic memory sex determination.

Summary of the invention

The technical matters that solves

In order to solve the deficiencies in the prior art part, the present invention proposes a kind of iconic memory sex determination method based on saliency entropy and object bank feature.

Thought of the present invention is: by studies show that the object and the image that comprise in the image degree of scatter that human vision attracts all can be exerted an influence to the Memorability of image, so object to comprising in the image, we represent with object bank feature, to the degree of scatter that image attracts human vision, we represent with the vision saliency entropy of image.Because these two models all have good expression to the Memorability of image, therefore with two kinds of feature combinations, can obtain better iconic memory sex determination effect.

Technical scheme

A kind of decision method of the iconic memory based on saliency entropy and object bank feature is characterized in that:

Step 1: the saliency entropy that extracts object bank feature and the computed image of each width of cloth input picture;

Extract the object bank feature of each width of cloth input picture, concrete step is as follows:

Step a: utilize Li-Jia Li at the object bank routine package of issue in 2010, to each width of cloth input picture, utilize the down-sampling technology to obtain 12 scalogram pictures of input picture, and 208 object templates in these 12 scalogram pictures and the object bank routine package are carried out convolutional calculation, make every width of cloth input picture obtain 208 * 12 width of cloth response images;

Step b: utilize two interpolation methods, with the response image interpolation of input picture corresponding to 12 yardsticks of each template, obtain the image of same size; Then to each pixel of the image of same size, calculate its maximal value on the scalogram picture of 12 same scale that obtain after the interpolation, consist of a peak response image; Then ask the pixel average of peak response image, obtain the character representation of one 208 dimension of every width of cloth input picture, be the object bank feature of input picture;

The saliency entropy of computed image: at first extract the saliency image of input picture, then the saliency image binaryzation that obtains is obtained bianry image; Calculate bianry image not connected region number with and corresponding area, the saliency entropy of computed image then, computation model is:

$\left\{\begin{array}{c}{P}_j=\frac{S_j}{S}\\ H=-\underset{j=1}{\overset{m}{\mathrm{\Σ}}}{P}_j\ln \left(P_j\right)\end{array}\right.$

Wherein, m is the number of connected region not in the bianry image; S _jIt is each not area of connected region; S is the area of bianry image; P _jIt is the ratio of connected region area and the area of whole image-region not; H represents the saliency entropy of bianry image;

Step 2 model training: with the Memorability numerical value of given training image as training sample, utilize the method for step 1, extract the object bank feature of image in the training sample and the feature of saliency entropy, then consist of the proper vector of one 209 dimension, utilize the Memorability numerical value of image in the training sample as label, training obtains a support vector regression model;

Step 3: for the image an of width of cloth Memorability numerical value the unknown, by the feature of step 1 its object bank feature of extraction and saliency entropy, the support vector regression model that then utilizes step 2 to obtain is judged the Memorability numerical value that obtains this width of cloth image.

The threshold value of described binaryzation

Value is 0.70 ~ 0.80.

Beneficial effect

The iconic memory sex determination method based on vision saliency entropy and object bank feature that the present invention proposes, object bank feature by extracting image and to the calculating of the saliency entropy of image can better be studied processing to this problem of iconic memory.Because this model of saliency entropy is relevant with people's vision noticing mechanism and memory mechanism, the object that comprises in the image simultaneously also can exert an influence to the Memorability of image, object bank feature then is a kind of reliable expression to the object that comprises in the image, so this method has obtained and judges preferably effect.Because two kinds of methods that the present invention adopts can be good at the lower-order questions of presentation video, therefore can obtain better result than the method for the global characteristics of existing extraction image.

Description of drawings

Fig. 1: the basic flow sheet of the inventive method

Fig. 2: the feature of saliency entropy is obtained figure in the inventive method

Fig. 3: the feature of object bank is obtained figure in the inventive method

Fig. 4: use this method process decision chart as the example of Memorability numerical value

(a) training image for example with and corresponding Memorability numerical value (need the great amount of images training, just select one here as an example).

(b) test pattern with and the Memorability numerical value judged with the inventive method.

Embodiment

Now in conjunction with the embodiments, the invention will be further described for accompanying drawing:

The hardware environment that is used for implementing is: Pentium-43G computing machine, 1GB internal memory, 128M video card.The software environment of operation is: Matlab7.0 and Windows XP, Ubuntu12.04.We have realized the method that the present invention proposes with Matlab software.

Implementation of the present invention is as follows:

1, experimental data: we select 2222 width of cloth images that Phillip announced in 2011 and the corresponding Memorability numerical value of every width of cloth image as experimental data.Get at random 1111 width of cloth images as training data, other 1111 width of cloth images are as test data.The data of Phillip are seen paper: Phillip I, Jianxiong X, Antonio T, et al.What makes an image memorable[C] .CVPR, 2011,145-152.

2, feature extraction: to 2222 width of cloth experimental image extract its object bank feature with and saliency entropy feature

(1) obtaining of object bank feature: utilize Li-Jia Li at the object bank routine package of issue in 2010, to each width of cloth input picture, utilize the down-sampling technology to obtain 12 scalogram pictures of input picture, and 208 object templates in these 12 scalogram pictures and the object bank program are carried out convolutional calculation, make every width of cloth input picture obtain 208 * 12 width of cloth response images; Utilize two interpolation methods, with the response image interpolation of input picture corresponding to 12 yardsticks of each template, obtain the image of same size; Then to each pixel of the image of same size, calculate its maximal value on the scalogram picture of 12 same scale that obtain after the interpolation, consist of a peak response image; Then ask the pixel average of peak response image, obtain the character representation of one 208 dimension of every width of cloth input picture, be the object bank feature of input picture;

The object bank program that described Li-Jia Li announced in 2010 is seen paper: Li-Jia L, Hao S, Eric X, et al.Object bank:A high-level image representation for scene classification and semantic feature sparsification[C] .NIPS, 2010.

(2) obtaining of saliency entropy: the method that we utilize Tilke to propose in this example is obtained the saliency(saliency of input picture) image.Then the saliency image binaryzation to obtaining is got threshold value and is

Obtain the bianry image of saliency image corresponding to input picture, to bianry image, the saliency entropy of formula calculating input image below utilizing

Wherein, m is the number of connected region not in the bianry image, S _jBe each not area of connected region, S is the area of bianry image, P _jThat the connected region area does not account for the ratio of the area in whole zone, i.e. probability.H represents the entropy of bianry image.

The method that described Tilke proposes is seen paper: Tilke J, Krista E, Fredo D, et al.Learning to Predict Where Humans Look[C] .ICCV, 2009,2106-2113.

3, model training: take in 1111 width of cloth training images of training with and corresponding Memorability numerical value, utilize the method in step 1 and the step 2, extract the object bank feature of training sample image, and the feature of saliency entropy, then consist of the proper vector of one 209 dimension, the Memorability numerical value of image is as label, train a support vector regression model, here realize with the libSVM software package, parameter is set to '-s3-p0.01-t0-c100 ', the Memorability numerical value that is used for image is judged.

4, the Memorability numerical value of image is judged: for the image an of width of cloth Memorability numerical value the unknown, extract the feature of its object bank feature and saliency entropy, then the support vector regression model that utilizes model training to obtain can be judged the Memorability numerical value that obtains this width of cloth image.

5, the calculating of relative coefficient: in order to verify the effect of the inventive method, the relative coefficient of computational discrimination value and actual value.To 2222 width of cloth experimental image, get at random 1111 width of cloth as training data, 1111 width of cloth extract the feature based on saliency entropy and object bank of each width of cloth image as test data.With training data training support vector regression model, then judge the Memorability numerical value of test data, then calculate related coefficient, the computing formula of related coefficient is as follows:

$\mathrm{\ρ}=\frac{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}(X_i-\stackrel{\‾}{X})(Y_i-\stackrel{\‾}{Y})}{\sqrt{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{(X_i-\stackrel{\‾}{X})}^2}\sqrt{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{(Y_i-\stackrel{\‾}{Y})}^2}}$

X wherein _iThe Memorability true value of expression test pattern, Y _iThe Memorability decision content of expression test pattern,

The mean value that represents the Memorability true value of 1111 width of cloth test patterns,

The mean value that represents the Memorability decision content of 1111 width of cloth test patterns, N=1111 represent to have 1111 width of cloth test patterns, and ρ represents related coefficient.Robustness in order to ensure experiment repeats above-mentioned experimentation 25 times, calculates the mean value of related coefficient, can obtain the decision content of the inventive method experiment and the average correlation coefficient of actual value.Table 1 has shown the average correlation coefficient of distinct methods, and average correlation coefficient is larger, shows decision content more near actual value, and that namely judges is more accurate.

The comparison of table 1 relative coefficient

Claims (2)

1. decision method based on the iconic memory of saliency entropy and object bank feature is characterized in that:

Step 1: the saliency entropy that extracts object bank feature and the computed image of each width of cloth input picture;

Extract the object bank feature of each width of cloth input picture, concrete step is as follows:

Step a: utilize Li-Jia Li at the object bank routine package of issue in 2010, to each width of cloth input picture, utilize the down-sampling technology to obtain 12 scalogram pictures of input picture, and 208 object templates in these 12 scalogram pictures and the object bank routine package are carried out convolutional calculation, make every width of cloth input picture obtain 208 * 12 width of cloth response images;

Step b: utilize two interpolation methods, with the response image interpolation of input picture corresponding to 12 yardsticks of each template, obtain the image of same size; Then to each pixel of the image of same size, calculate its maximal value on the scalogram picture of 12 same scale that obtain after the interpolation, consist of a peak response image; Then ask the pixel average of peak response image, obtain the character representation of one 208 dimension of every width of cloth input picture, be the object bank feature of input picture;

The saliency entropy of computed image: at first extract the saliency image of input picture, then the saliency image binaryzation that obtains is obtained bianry image; Calculate bianry image not connected region number with and corresponding area, the saliency entropy of computed image then, computation model is:

$\left\{\begin{array}{c}{P}_j=\frac{S_j}{S}\\ H=-\underset{j=1}{\overset{m}{\mathrm{\Σ}}}{P}_j\ln \left(P_j\right)\end{array}\right.$

Wherein, m is the number of connected region not in the bianry image; S _jIt is each not area of connected region; S is the area of bianry image; P _jIt is the ratio of connected region area and the area of whole image-region not; H represents the saliency entropy of bianry image;

Step 2 model training: with the Memorability numerical value of given training image as training sample, utilize the method for step 1, extract the object bank feature of image in the training sample and the feature of saliency entropy, then consist of the proper vector of one 209 dimension, utilize the Memorability numerical value of image in the training sample as label, training obtains a support vector regression model;

Step 3: for the image an of width of cloth Memorability numerical value the unknown, by the feature of step 1 its object bank feature of extraction and saliency entropy, the support vector regression model that then utilizes step 2 to obtain is judged the Memorability numerical value that obtains this width of cloth image.

2. the decision method of a kind of iconic memory based on saliency entropy and object bank feature according to claim 1 is characterized in that: the threshold value of described binaryzation

Value is 0.70 ~ 0.80.

Images