CN103824273A - Super-resolution reconstruction method based on compound motion and self-adaptive nonlocal prior - Google Patents

Abstract

The invention provides a super-resolution reconstruction method based on compound motion and self-adaptive nonlocal prior. The method comprises the following steps: selecting a reference frame image and a non-reference frame image from p low-resolution images; performing image registration by means of global motion parameters and local light stream to obtain a motion field mk(x) of the non-reference frame image relative to the reference frame image, and constructing a motion transformation matrix Mk by using mk(x); calculating an interpolation image X, nonlocal prior parameter hi,j and European threshold of the reference frame image; calculating the similarity weight wNLM[i, j; s, t] of each pixel to other pixels, and constructing a nonlocal weight matrix S related to a high-resolution image X by using wNLM; solving a target function shown in the specification by using the motion transformation matrix Mk and the nonlocal weight matrix S to obtain a reconstructed high-resolution estimation image. Compared with the prior art, the super-resolution reconstruction method has the advantages that the defects of high calculated amount, poor scalability and low accuracy in the conventional motion estimation are overcome effectively by adopting a compound motion model, and distortion of a reconstructed image is reduced by adopting self-adaptive nonlocal prior.

Description

Based on the super resolution ratio reconstruction method of compound motion and the non local priori of self-adaptation

Technical field

The present invention relates to the technical field of image processing, computer vision, particularly relate to the super resolution image reconstruction method based on compound motion and the non local priori of self-adaptation of a kind of spatial resolution that improves image in image sharpening field.

Background technology

In imaging field, the image of high spatial resolution is one of target of pursuing always.The image of high spatial resolution has recorded the detailed information of object fully, abundanter information is provided can to reasoning, judgement, the decision-making of people and computing machine.Therefore, in many imaging applications, high-resolution image is very important conventionally, such as: the application such as video monitoring, medical diagnosis, military surveillance, remote sensing.Improve the spatial resolution of image and can pass through two kinds of approach, " hardware approach " and " software approach ".Under regular situation, people mainly obtain high-resolution image by improving the hardware devices such as high-precision CCD and cmos sensor.But, merely improve resolution and can be subject to many limitations by improving hardware facility, such as the electric charge rate of transform, the thermonoise of sensor, the Rayleigh entropy of optical lens, and the restriction such as hardware costs.Therefore,, for hardware device and consideration economically, " software approach " becomes a kind of more feasible scheme.First Tsai in 1984 and Huang propose Super-resolution Reconstruction problem is from " software approach ", improves the spatial resolution of image by theory of development, algorithm, has become one of research direction that image processing field enlivens the most.In recent years, super-resolution rebuilding is divided into the super-resolution rebuilding based on multiframe and the super-resolution rebuilding based on study by some scholars.The present invention is a kind of Super-resolution Reconstruction method based on multiframe.

Can be divided into again based on frequency domain and the method for reconstructing based on spatial domain the super-resolution rebuilding based on multiframe.The up-to-date method for reconstructing based on frequency domain be Rhee in 1999 and Kang propose based on discrete cosine transform method for reconstructing, and the method for reconstructing based on wavelet transformation that proposed in 2003 of Chan.Method for reconstructing advantage based on frequency domain be theoretical succinct, calculate simply, but inferior position is that the relative motion between multiframe can only be overall similar movement, and to be only applicable to the fuzzy of image be the situation of linear time invariant.Therefore for these inferior positions, the method for reconstructing based on spatial domain has produced the Super-resolution Reconstruction method of many classics, as non-homogeneous interpolation method, iterative backprojection method, convex set projection method, maximum likelihood method, maximum a posteriori, mixing maximum a posteriori/convex set projection method.The present invention is a kind of method for reconstructing of estimating based on maximum a posteriori.

In the method for estimation based on maximum a posteriori, estimation and the design of priori item are two very important tasks.In most of the cases, the required motion vector of image super-resolution is unknown, in order to solve these motion vectors, can there be two kinds of thinkings: a kind of thinking is first to solve these motion vectors and then carry out super-resolution rebuilding, although the mode separately solving this is calculated very simple, all exists separately significant limitation; Another kind of thinking is that motion vector and super resolution image are combined and solved, and these class methods are than separately the estimation of method for solving is accurate separately, rebuilds effectively, but shortcoming is that to solve speed slow, is difficult to practical application.For the estimation of motion vector, can be divided into again based on estimation parameter model and based on light stream.Estimation based on parameter model is fairly simple, but retractility is low; And estimation retractility based on light stream is high, but estimated accuracy is low, rebuilds effect undesirable.For this reason, the present invention is based on the thinking that separately solves motion vector and super resolution image, proposed a kind of have very high scalability and compound motion model estimated accuracy, low operand.

For priori item, be also regular terms design, many priori items have been proposed in domestic and foreign literature, comprise Tikhonov priori, Huber priori, TV priori, BTV priori etc.These priori are all described based on Image neighborhood difference, fail to describe out exactly the prior imformation of natural image, cause rebuilding image fault.Recently, based on the fact of picture structure that exists bulk redundancy in natural image and repeat, proposed in the world a kind of non local priori, and be successfully applied to image deconvolution.But, do the artificial parameter regulating of many needs owing to existing in non local priori, can not accomplish the self-adaptation of parameter.

Summary of the invention

The problem existing for prior art, fundamental purpose of the present invention is to provide a kind of super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation that reduces calculated amount, improves the precision of images.

For achieving the above object, the invention provides a kind of embodiment of the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation, high-definition picture X is through degrading Procedure Acquisition p width low resolution observed image Y _k(k=..., p), the size of every width observed image is m × n, this utilizes p width low resolution observed image Y _k(k=1 ..., the super resolution ratio reconstruction method of p) rebuilding high resolving power estimated image comprises the steps (1) to step (5):

(1) in p width low resolution observed image, choose reference frame image Y _ref(1≤ref≤p) and non-reference frame image Y _k(k=1 ..., ref-1, ref+1 ..., p), the compound motion model that adopts global parameter motion and local light stream for the sub-pix motion between reference frame image and non-reference frame image, the relation table between reference frame image and non-reference frame image is shown: $Y_{\mathrm{ref}}\left(x\right)=Y_k\left(m_k\left(x\right)\right)=Y_k(m_k^g(x;{\mathrm{\θ}}_k)+m_k^l\left(x\right))=Y_k\left(m_k^g(x;{\mathrm{\θ}}_k)\right)+{\mathrm{\ϵ}}_k\left(x\right),$ Wherein m _krepresent two dimensional motion field,

represent global parameter motion,

for local light stream campaign, θ _kfor globe motion parameter,

represent the reference frame image with non-reference frame image prediction, ε _k(x) represent residual image;

(2) solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) and local light stream

, adopt globe motion parameter θ _k=(a ₀, a ₁, ₂, a ₃, a ₄, a ₅) and local light stream

method carry out image registration, obtain the sports ground m of non-reference frame image with respect to reference frame image _k(x), utilize m _k(x) construct motion converter matrix M _k;

(3) computing reference two field picture Y _refr times of interpolation image , non local priori parameter _{i, j}the European threshold value of (0≤i<rm, 0≤j<rn) and similar image;

(4) utilize non local priori parameter, European threshold value, and with interpolation image

as the initial pictures of full resolution pricture X, calculate interpolation image

in each pixel (i, j) (wherein 0≤i<rm, 0≤j<rn) with the similarity weight w of other pixels (s, t) (wherein 0≤s<rm, 0≤t<rn) _nLM[i, j; S, t], utilize similarity weight w _nLMbuild non local weight matrix S;

(5) utilize motion converter matrix M _ksolve cost functional with non local weight matrix S $X=\arg \min [\underset{k=1}{\overset{p}{\mathrm{\&Sigma;}}}{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2+\mathrm{\&lambda;}{\left|\right|(I-S)\left|\right|}_{\mathrm{\&rho;}}^{\mathrm{\&rho;}}],$ Wherein B _kfor observed image Y _kcorresponding clear function, the M of falling _kfor observed image Y _kwith respect to the sub-pix motion of reference frame image, non local weight matrix S is the non-local mean wave filter of a self-adaptation high-definition picture X, and ρ > 0, adopts conjugate gradient iterative procedure to minimize cost functional, obtains the high resolving power estimated image of rebuilding.

Further, this step (1) adopts the compound motion model of global parameter motion and local light stream for the sub-pix motion between reference frame image and non-reference frame image, and wherein motion vector is m _k(x)=[m _{k, u}(x) m _{k, v}(x)], x=[x _ux _v], two dimensional motion field m _kbe expressed as m _k(x)=m _k ^g(x)+m _k ^l(x)=m _k ^g(x; θ _k)+m _k ^l(x), wherein

represent global parameter motion,

for local light stream campaign, θ _kfor globe motion parameter.

Further, this step (2) solves globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) comprise the steps (21A) to step (22A):

(21A) adopt a ₀, a ₁, a ₂, a ₃, a ₄, a ₅the affined transformation of six parameters is as global parameter motion model: $m_{k,u}^g(x;{\mathrm{\&theta;}}_k)=a_0+a_1{x}_u+a_2{x}_v,m_{k,v}^g(x;{\mathrm{\&theta;}}_k)=a_3+a_4{x}_u+a_5{x}_v;$

(22A) set up least square standard

solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅).This step (22A) is set up least square standard

solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) comprise the steps (221A) to step (224A): (221A) by θ _kwrite as θ _k+ Δ θ; (222A) least square cost functional is carried out to Taylor expansion and obtain the linear function about Δ θ; (223A) function after launching being carried out to a series of arithmetic operations obtains $\mathrm{\&Delta;\&theta;}=H^{-1}\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T[Y_{\mathrm{ref}}\left(x\right)-Y_k\left(m_k^g(x;{\mathrm{\&theta;}}_k)\right)],$ Wherein $H=\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{\&PartialD;m_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T\underset{x}{\mathrm{\&Sigma;}}[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}],$ $\&dtri;Y_k=\frac{{\&PartialD;Y}_k}{{\&PartialD;m}_k^g(x;{\mathrm{\&theta;}}_k)},$ $\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}=\left(\begin{array}{cccccc}{x}_u& 0& x_v& 0& 1& 0\\ 0& x_u& 0& x_v& 0& 1\end{array}\right);$ (224A) judge whether to meet || Δ θ ||≤∈, threshold value 0≤∈≤0.01, if do not meet, θ _k← θ _k+ Δ θ, and return to step (223A); If meet, show parameter θ _kconvergence, stops iteration, obtains globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅), finally can obtain global parameter motion

Further, this step (2) solves local light stream comprise the steps (21B) to step (24B):

(21B) utilize gradation of image shape constancy hypothesis and image gradient shape constancy hypothesis to obtain about local light stream

data degree of confidence energy function $E_{\mathrm{Data}}(u,v)=\underset{\mathrm{\&Omega;}}{\&Integral;}\mathrm{\&Psi;}({|Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-Y_k\left(\stackrel{\&OverBar;}{x}\right)|}^2+|\&dtri;Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-\&dtri;Y_k{|}^2)d\stackrel{\&OverBar;}{x},$ Wherein, $\stackrel{\&OverBar;}{x}={(x,t)}^T={(x_u,x_v,1)}^T,v={(m_k^g{\left(x\right)}^T,0)}^T,\mathrm{\&omega;}={(m_k^l{\left(x\right)}^T,1)}^T={(u,v,1)}^T,$ Function $\mathrm{\&Psi;}\left(s^2\right)=\sqrt{s^2+{\mathrm{\&tau;}}^2},$ Wherein 0< τ≤0.01;

(22B) according to image segmentation, smoothness assumption obtains level and smooth penalty $E_{\mathrm{Smooth}}(u,v)=\underset{\mathrm{\&Omega;}}{\&Integral;}\mathrm{\&Psi;}\left({|\&dtri;u|}^2{+|\&dtri;v|}^2\right)d\stackrel{\&OverBar;}{x};$

(23B) obtaining whole energy function is E (u, v)=E _data+ α E _smooth, wherein α >0 is regularization parameter;

(24B) adopt Nonlinear Numerical method for solving to whole energy function E (u, v)=E _data+ α E _smoothask optimal value just to obtain local light stream

solution.

Further, this step (2) is utilized m _k(x) construct motion converter matrix M _kcomprise following steps (21C) to step (23C):

(21C) calculate the sports ground m of non-reference frame image with respect to reference frame image _k(x)=m _k ^g(x)+m _k ^l(x), and by sports ground m _k(x) carry out r times of linear interpolation, obtain the sports ground m after interpolation _k(x), this r is the enlargement factor of the final high-definition image of rebuilding;

(22C) calculate relative displacement Δ x _k=m _k(x)-x=(Δ x _{k, u}-Δ x _{k, v}), and d _k=Δ x _{k, u}-floor (Δ x _{k, u}), e _k=Δ x _{k, v}-floor (Δ x _{k, v}), wherein operational character floor (.) represents to get the maximum integer that is less than or equal to designated value;

(23C) calculate motion converter matrix M _kin the value of each element: M _k(j*m+i, floor (Jx _{k, u})+x _u+ ceil (Jx _{k, v}+ x _v+ 1) * m)=d _k* (1-e _k), M _k(j*m+i, ceil (Δ x _{k, u})+xu+ceil (Δ x _{k, v}+ x _v+ 1) * m)=d _k* e _k, M _k(j*m+ ⁱ, floor (Δ x _{k, u})+x _u+ floor (Δ x _{k, v}+ x _v+ 1) * m)=(1-d _k) * (1-e _k), M _k(j*m+i, ceil (Jx _{k, u})+x _u+ floor (Jx _{k, v}+ x _v+ 1) * m+x)=(1-d _k) * e _k, wherein operational character ceil (.) represents to get the smallest positive integral that is greater than or equal to designated value, for matrix M _kother element values except these four row in capable of j*m+i be zero.

Further, this step (3) is calculated non local priori parameter h _{i, j},

wherein std (N _{i, j}) be centered by pixel (i, j), region of search is N _{i, j}standard deviation, β is greater than zero constant (1< β <5), σ ²for reference frame image Y _refnoise variance, r is image enlargement factor, this reference frame image Y _refnoise variance σ ²be estimated as: ${\widehat{\mathrm{\&sigma;}}}^2=\frac{1}{n-2}\underset{i=2}{\overset{n-1}{\mathrm{\&Sigma;}}}{\widehat{\mathrm{\&epsiv;}}}_i^2,$ The pixel sum that wherein n is image, ${\widehat{\mathrm{\&epsiv;}}}_i=\sqrt{\frac{4}{5}}(Y_{\mathrm{ref}}\left(x_i\right)-\frac{1}{4}\underset{x_j\&Element;N\left(x_i\right)}{\mathrm{\&Sigma;}}{Y}_{\mathrm{ref}}\left(x_j\right)),$ N (x _i) be pixel x _ineighbours territory.

Further, the maximum weighted Euclidean distance that in this step (3), the European threshold value of similar image adopts is 4 σ ², also: ${\left|\right|R_{k,l}\stackrel{.}{X}-R_{i,j}\stackrel{.}{X}\left|\right|}_{2,a}^2\&le;4{\mathrm{\&sigma;}}^2\&ap;4{\widehat{\mathrm{\&sigma;}}}^2.$

Further, this step (4) is utilized non local priori parameter, European threshold value, and with interpolation image

as the initial pictures of full resolution pricture X, calculate interpolation image

in the similarity weight w of each pixel (i, j) and other pixels (s, t) _nLm[i, j; S, t], utilize similarity weight w _nLMbuild non local weight matrix S and comprise following steps (41) to step (43):

(41) calculate the similarity of the block of pixels centered by coordinate (s, t) in each block of pixels centered by coordinate (i, j) and its neighborhood N (i, j)

wherein R _{i, j}the image block that is pixel centered by (i, j) extracts operator, the weighted euclidean distance that represents two image blocks, wherein a>0 is the standard deviation of gaussian kernel function, h _{i, j}for filter smoothing parameter, depend on noise size and the image itself of image, f is the normal function that depends on the geometric distance of two central pixel point;

(42) all pixel similarity values in normalization pixel (i, j) and neighborhood N (i, j), $\mathrm{\&omega;}[i.j;s,t]=\frac{{\mathrm{\&omega;}}_{\mathrm{NLM}}[i,j;s,t]}{{\mathrm{\&Sigma;}}_{(s,t)\&Element;N(i,j)}{\mathrm{\&omega;}}_{\mathrm{NLM}}[i,j;s,t]};$

(43) build non local weight matrix $S_{j*m+i,s*m+t}=\left\{\begin{array}{cc}\mathrm{\&omega;}[i,j;s,t]& (s,t)\&Element;N(i,j)\\ 0& \mathrm{otherwise}\end{array}\right..$

Further, in this step (5), work as

time, iteration stops, and wherein n is iterations.

With respect to prior art, first, the present invention is applied to non local priori the multiframe super-resolution rebuilding of image first, has proposed a kind of parameter adaptive method for solving of non local priori; Secondly, by adopting compound motion model, the calculated amount that has effectively solved current estimation is large, the shortcoming that scalability is not strong, precision is not high; Again, adopt adaptive non local priori more can be exactly the prior imformation of Description Image automatically, reduced the distortion of rebuilding image.For this reason, the method has been accomplished telescopic high-precision estimation and adaptive priori description accurately, can more effectively be applied to actual characteristics of image sharpening engineering.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the embodiment of the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation of the present invention

Fig. 2 is the globe motion parameter θ that solves of the present invention _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) the process flow diagram of embodiment

Fig. 3 is the local light stream that solves of the present invention

the process flow diagram of embodiment

Embodiment

Below in conjunction with accompanying drawing, describe the specific embodiment of the present invention in detail.

The super-resolution rebuilding of image, reverts to ideal image by observed image.Observed image is a series of low-resolution image, and ideal image is required high-definition picture.The high-definition picture X of given certain scene, through the process that degrades of a series of how much motions, optical dimming, sub-sampling and additional noises, produces P width low resolution observed image Y _k, by the relation between conventional image observation model description ideal image and an observed image, this observation model is: Y _k=DB _km _kx+n _k, k=1 ..., p, wherein M _kfor motion change matrix, B _kfor fuzzy matrix, D is down-sampling matrix, n _kfor additional noise.

Based on above-mentioned observation model, the present invention adopts maximum a posteriori method to estimate ideal image X.P width low-resolution image is expressed as Y=[Y ₁ ^t, Y ₂ ^t..., Y _p ^t] ^t, the Super-resolution Reconstruction problem representation based on maximum a posteriori estimation theory is: X=argmaxP (X|Y), through simple calculations operation, can obtain: $X=\arg \max P\left(Y\right|X)P\left(X\right)=\arg \max \underset{k=1}{\overset{P}{\mathrm{\&Sigma;}}}P\left(Y_k\right|X)P\left(X\right).$ P (Y herein _k| X)=P (n _k) representing the type of observation model noise, common hypothesis noise is that average is 0, variance is σ _k ²gaussian noise, $P\left(Y_k\right|X)=\frac{1}{C_1}\exp \{-\frac{{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2}{{{2\mathrm{\&sigma;}}_k}^2}\},$ Wherein C ₁for constant.Image priori probability density general type is:

wherein C ₂for constant, η is for controlling parameter, and U (X) is the priori energy function about image X.Through simple abbreviation, finally can obtain following cost function: $X=\arg \min [\underset{k=1}{\overset{p}{\mathrm{\&Sigma;}}}{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2+\mathrm{\&lambda;U}\left(X\right)],$ Wherein

at this, the present invention supposes that the fuzzy core of image is known, is also fuzzy matrix B _kknown.

According to $X=\arg \min [\underset{k=1}{\overset{p}{\mathrm{\&Sigma;}}}{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2+\mathrm{\&lambda;U}\left(X\right)]$ Known, obtain ideal image X, need to solve motion converter matrix M _kwith priori energy U (X), i.e. the non local priori design of image registration and self-adaptation.Finally adopt conventional method of conjugate gradient to cost functional $X=\arg \min [\underset{k=1}{\overset{p}{\mathrm{\&Sigma;}}}{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2+\mathrm{\&lambda;U}\left(X\right)]$ Solve and obtain ideal image X.

Therefore, the present invention mainly comprises: the method that (1) adopts globe motion parameter to add local light stream in the time of multiframe low-resolution image registration is carried out image registration, and builds motion converter matrix M _k; (2) the non local priori of design self-adaptation, builds non local weight matrix S; (3) set up cost functional, and adopt method of conjugate gradient to solve.

As shown in Figure 1, be the process flow diagram of the embodiment of the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation of the present invention.The embodiment of super resolution ratio reconstruction method comprises the steps that S1 is to step S8:

S1, obtain p width by the high resolution graphics X low resolution observed image Y that process forms through degrading _k, the size of every width observed image is m × n.Obtain image viewing model Y _k=DB _km _kX+ n _k, k=1 ..., p, wherein M _kfor motion change matrix, B _kfor fuzzy matrix, D is down-sampling matrix, n _kfor additional noise, X is high-definition picture, and k is the numbering of the p width low-resolution image that obtains.

S2, in p width low-resolution image, choose reference frame image Y _ref(1≤ref≤p) and non-reference frame image Y _k(k=1 ..., ref-1, ref+1 ..., p), adopt global parameter motion to add the compound motion model of local light stream for the sub-pix motion between reference frame image and non-reference frame image, the relation table between reference frame image and non-reference frame image is shown: $Y_{\mathrm{ref}}\left(x\right)=Y_k\left(m_k\left(x\right)\right)=Y_k(m_k^g(x;{\mathrm{\&theta;}}_k)+m_k^l\left(x\right))=Y_k\left(m_k^g(x;{\mathrm{\&theta;}}_k)\right)+{\mathrm{\&epsiv;}}_k\left(x\right),$ Wherein m _krepresent two dimensional motion field,

represent global parameter motion,

for local light stream campaign, the motion of Ye Ji nonparametric local equalize, θ _kfor globe motion parameter,

represent the reference frame image with non-reference frame image prediction, ε _k(x) represent residual image.In the compound motion model of global parameter motion and local light stream, motion vector is m _k(x)=[m _{k, u}(x) m _{k, v}(x) object of], estimating this motion vector is for building motion converter matrix M _k, wherein x=[x _ux _v], two dimensional motion field m _kbe expressed as $m_k\left(x\right)=m_k^g\left(x\right)+m_k^l\left(x\right)=m_k^g(x;{\mathrm{\&theta;}}_k)+m_k^l\left(x\right),$ Wherein

represent global parameter motion,

for local light stream campaign, θ _kfor globe motion parameter.

S3, solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) and local light stream

adopt globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) and local light stream

method carry out image registration, obtain the sports ground m of non-reference frame image with respect to reference frame image _k(x), utilize m _k(x) construct motion converter matrix M _k:

Solve globe motion parameter θ _k.(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) time, model least square standard $\underset{{\mathrm{\&theta;}}_k}{\min }\underset{x}{\mathrm{\&Sigma;}}{[Y_k\left(m_k^g(x;{\mathrm{\&theta;}}_k)\right)-Y_{\mathrm{ref}}\left(x\right)]}^2,$ Wherein $m_k^g(x;{\mathrm{\&theta;}}_k)=\left[m_{k,u}^g(x;{\mathrm{\&theta;}}_k)m_{k,v}^g(x;{\mathrm{\&theta;}}_k)\right],$ $m_{k,u}^g(x;{\mathrm{\&theta;}}_k)=a_0+a_1{x}_u+a_2{x}_v,m_{k,v}^g(x;{\mathrm{\&theta;}}_k)=a_3+a_4{x}_u+a_5{x}_v;$ Then by θ _kwrite as θ _k+ Δ θ, solves θ in the mode of increment _k, be also θ _k← θ _k+ Δ θ.Least square cost functional is carried out to Taylor expansion and obtain the linear function about Δ θ, the function after launching is carried out to a series of arithmetic operations and obtain $\mathrm{\&Delta;\&theta;}=H^{-1}\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T[Y_{\mathrm{ref}}\left(x\right)-Y_k\left(m_k^g(x;{\mathrm{\&theta;}}_k)\right)],$ Wherein $H=\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{\&PartialD;m_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T\underset{x}{\mathrm{\&Sigma;}}[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}],$ $\&dtri;Y_k=\frac{{\&PartialD;Y}_k}{{\&PartialD;m}_k^g(x;{\mathrm{\&theta;}}_k)},$ $\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}=\left(\begin{array}{cccccc}{x}_u& 0& x_v& 0& 1& 0\\ 0& x_u& 0& x_v& 0& 1\end{array}\right).$ In the time that kinematic parameter increment Delta θ is less than certain threshold value ∈ (0≤E≤0.01), be also || Δ θ ||>=∈, parameter θ _kconvergence, stops iteration, obtains globe motion parameter θ _k.(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅), finally can obtain global parameter motion otherwise upgrade θ _k, be also θ _k← θ _k+ Δ θ, and recalculate Δ θ.

Solve local light stream

time, first utilize gradation of image shape constancy hypothesis and image gradient shape constancy hypothesis to obtain about local light stream

data degree of confidence energy function $E_{\mathrm{Data}}(u,v)=\underset{\mathrm{\&Omega;}}{\&Integral;}\mathrm{\&Psi;}({|Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-Y_k\left(\stackrel{\&OverBar;}{x}\right)|}^2+|\&dtri;Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-\&dtri;Y_k{|}^2)d\stackrel{\&OverBar;}{x},$ Wherein, $\stackrel{\&OverBar;}{x}={(x,t)}^T={(x_u,x_v,1)}^T,v={(m_k^g{\left(x\right)}^T,0)}^T,\mathrm{\&omega;}={(m_k^l{\left(x\right)}^T,1)}^T={(u,v,1)}^T,$ Function

0< τ≤0.01; Then according to image segmentation, smoothness assumption obtains level and smooth penalty

finally obtaining whole energy function is E (u, v)=E _data+ α E _smooth, wherein α >0 is regularization parameter.Adopt Nonlinear Numerical method for solving to ask optimal value just to obtain local light stream to whole energy function

solution.

Finally obtain the sports ground m of non-reference frame image with respect to reference frame image _k(x)=m _k ^g(x)+m _k ^l(x), and with this build motion converter matrix M _k.First by sports ground m _k(x) carry out doubly (enlargement factor of the final high-definition image of rebuilding) linear interpolation of r, obtain interpolation _aftersports ground m _k(x).Then calculate relative displacement Δ x _k=m _k(x)-x=(Δ x _{k, u}-Δ x _{k, v}), and d _k=Δ x _{k, u}-floor (Δ x _{k, u}), e _k=Δ x _{k, v}-floor (Δ x _{k, v}), wherein operational character foor (.) represents to get the maximum integer that is less than or equal to designated value.Finally calculate motion converter matrix M _kin the value of each element: M _k(j*m+i, floor (Δ x _{k, u})+x _u+ ceil (Δ x _{k, v}+ x _v+ 1) * m)=d _k* 1-e _k), M _k(j*m+i, ceil (Δ x _k,u)+z _u+ ceil (Δ x _{k, v}+ x _v+ 1) * m)=d _k* e _k, M _k(j*m+i, floor (Δ x _{k, u})+x _u+ floor (Δ x _{k, v}+ x _v+ 1) * m)=(1-d _k) * (1-e _k), M _k(j*m+i, ceil (Δ x _{k, u})+x _u+ floor (Δ x _{k, v}+ x _v+ 1) * m+x)=(1-d _k) * e _k, wherein operational character ceil (.) represents to get the smallest positive integral that is greater than or equal to designated value, for matrix M _kj*m ⁺other element values except these four row during i is capable are zero.

Step S2, S3 have completed image registration and kinematic matrix M _kstructure.

S4, computing reference two field picture Y _refr times of bicubic interpolation image

, non local priori smoothing parameter h _{i, j}european threshold value with similar image.Because ideal image X is unknown in advance, non local weight matrix is just according to the reference frame image Y of low resolution _refinterpolation image

estimation obtains.Therefore, smoothing parameter h _{i, j}be one about reference frame image Y _refnoise, view data itself, the function of down-sampling multiple.

wherein std (N _{i, j}) be centered by pixel (i, j), region of search is N _{i, j}standard deviation, for being greater than zero constant (1< β <5), σ ²for the noise variance of reference frame image, r is image enlargement factor.This reference frame image Y _refnoise variance σ ²be estimated as:

the pixel sum that wherein n is image,

n (x _i) be pixel x _ineighbours territory.Finally adopt European threshold value to carry out dissimilar pixel removal, the maximum weighted Euclidean distance that European threshold value (being similarity threshold) adopts is 4 σ ², also: ${\left|\right|R_{k,l}\stackrel{.}{X}-R_{i,j}\stackrel{.}{X}\left|\right|}_{2,a}^2\&le;4{\mathrm{\&sigma;}}^2\&ap;4{\widehat{\mathrm{\&sigma;}}}^2.$

S5, calculate each pixel (i, j) (wherein 0≤i<rm, 0≤j<rn) with other pixels (s, t) the similarity weight w of (wherein 0≤s<rm, 0≤t<rn) _nLM[i, j; S, t], build about with interpolation image

non local weight matrix S as initial full resolution pricture X comprises following steps: the similarity of calculating the pixel (s, t) in each pixel (i, j) and its neighborhood N (i, j)

wherein R _{i, j}the image block that is pixel centered by (i, j) extracts operator, is generally square (q=5,7,9 of q × q ...),

the weighted euclidean distance that represents two image blocks, wherein a>0 is the standard deviation of gaussian kernel function, h _{i, j}for filter smoothing parameter, depend on noise size and the image itself of image, f is the normal function (monotone non-increasing function) that depends on the geometric distance of two central pixel point; All pixel similarity values in normalization pixel (i, j) and neighborhood N (i, j), $\mathrm{\&omega;}[i.j;s,t]=\frac{{\mathrm{\&omega;}}_{\mathrm{NLM}}[i,j;s,t]}{{\mathrm{\&Sigma;}}_{(s,t)\&Element;N(i,j)}{\mathrm{\&omega;}}_{\mathrm{NLM}}[i,j;s,t]};$ The non local weight matrix of final structure $S_{j*m+i,s*m+t}=\left\{\begin{array}{cc}\mathrm{\&omega;}[i,j;s,t]& (s,t)\&Element;N(i,j)\\ 0& \mathrm{otherwise}\end{array}\right..$

The self-adaptation that step S4, S5 have completed non local priori parameter solves with the S of non local weight matrix and builds.

S6, utilize motion converter matrix M _ksolve cost functional with non local weight matrix S $X=\arg \min [\underset{k=1}{\overset{p}{\mathrm{\&Sigma;}}}{\left|\right|Y_k-{\mathrm{DB}}_k{M}_{k}X\left|\right|}^2+\mathrm{\&lambda;}{\left|\right|(I-S)\left|\right|}_{\mathrm{\&rho;}}^{\mathrm{\&rho;}}],$ Wherein ρ > 0.Adopt conjugate gradient iterative procedure to minimize cost functional;

S7, judge whether to meet stopping criterion for iteration:

wherein n is iterations, if do not meet, returns to step S6, if meet, shows to separate convergence, and iteration stops, and enters step S8;

S8, obtain rebuild high resolving power estimated image.

Step S6 has completed and has solved cost functional to step S8.

As shown in Figure 2, be the globe motion parameter θ that solves of the present invention _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) the process flow diagram of embodiment, comprise the steps that S31A is to step S32A:

S31A, employing a ₀, a ₁, a ₂, a ₃, a ₄, a ₅the affined transformation of six parameters is as global parameter motion model: $m_{k,u}^g(x;{\mathrm{\&theta;}}_k)=a_0+a_1{x}_u+a_2{x}_v,m_{k,v}^g(x;{\mathrm{\&theta;}}_k)=a_3+a_4{x}_u+a_5{x}_v;$

S32A, set up least square standard

solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅), comprise the steps that S321A is to step S324A:

S321A, because least square cost functional is about θ _kbe nonlinear need be by θ _kwrite as θ _kthe form of+Δ θ, adopts increment iterative mode to solve θ _k, be also θ _k← θ _k+ Δ θ.And least square cost functional is carried out to Taylor expansion obtain the linear function about Δ θ; S322A, the function after launching is carried out arithmetic operation and obtained $\mathrm{\&Delta;\&theta;}=H^{-1}\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T[Y_{\mathrm{ref}}\left(x\right)-Y_k\left(m_k^g(x;{\mathrm{\&theta;}}_k)\right)],$ Wherein $H=\underset{x}{\mathrm{\&Sigma;}}{[\&dtri;Y_k\frac{\&PartialD;m_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}]}^T\underset{x}{\mathrm{\&Sigma;}}[\&dtri;Y_k\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}],$ $\&dtri;Y_k=\frac{{\&PartialD;Y}_k}{{\&PartialD;m}_k^g(x;{\mathrm{\&theta;}}_k)},$ $\frac{{\&PartialD;m}_k^g}{{\&PartialD;\mathrm{\&theta;}}_k}=\left(\begin{array}{cccccc}{x}_u& 0& x_v& 0& 1& 0\\ 0& x_u& 0& x_v& 0& 1\end{array}\right);$

S323A, judge whether to meet || Δ θ |≤∈, threshold value 0≤∈≤0.01, if do not meet, θ _k← θ _k+ Δ θ, and return to step S322A, if meet, show parameter θ convergence, stop iteration, enter step S324A; S324A, obtain globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅).

As shown in Figure 3, solve local light stream for of the present invention

the process flow diagram of embodiment.Obtaining globe motion parameter θ=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅), after being also global motion field, if affine motion can be similar to the motion model of object well, just now can omit solving of local light flow field that sports ground is compensated.If but overall affine model fails to be accurately similar to whole sports ground, just need to solve local light flow field.But the global parameter sports ground having solved has approached mass motion field to a great extent, contribute to solving of ensuing local light flow field, make the local light flow field that solves more accurate.This has just embodied good scalability, high-precision estimation and the low operand of the compound motion model of the present invention's proposition.In order to try to achieve local light stream

solution, the present invention utilize Thomas Brox propose light stream solving method.Thomas Brox light stream solving method is mainly based on three hypothesis: gradation of image shape constancy hypothesis, image gradient shape constancy hypothesis, image segmentation smoothness assumption.

Solve local light stream

comprise the steps S31B to S34B:

S31B, utilize gradation of image shape constancy hypothesis and image gradient shape constancy hypothesis to obtain about local light stream

data degree of confidence energy function $E_{\mathrm{Data}}(u,v)=\underset{\mathrm{\&Omega;}}{\&Integral;}\mathrm{\&Psi;}({|Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-Y_k\left(\stackrel{\&OverBar;}{x}\right)|}^2+|\&dtri;Y_k(\stackrel{\&OverBar;}{x}+v+\mathrm{\&omega;})-\&dtri;Y_k{|}^2)d\stackrel{\&OverBar;}{x},$ Wherein, $\stackrel{\&OverBar;}{x}={(x,t)}^T={(x_u,x_v,1)}^T,v={(m_k^g{\left(x\right)}^T,0)}^T,\mathrm{\&omega;}={(m_k^l{\left(x\right)}^T,1)}^T={(u,v,1)}^T,$ Function $\mathrm{\&Psi;}\left(s^2\right)=\sqrt{s^2+{\mathrm{\&tau;}}^2},$ 0<τ≤0.01：

S32B, obtain level and smooth penalty according to the image segmentation smoothness assumption of light stream $E_{\mathrm{Smooth}}(u,v)=\underset{\mathrm{\&Omega;}}{\&Integral;}\mathrm{\&Psi;}\left({|\&dtri;u|}^2{+|\&dtri;v|}^2\right)d\stackrel{\&OverBar;}{x};$

S33B, to obtain whole energy function be E (u, v)=E _data+ α E _smooth, wherein α >0 is regularization parameter;

S34B, employing Nonlinear Numerical method for solving ask optimal value just to obtain local light stream to whole energy function

solution.

More than introduce a kind of super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation.The present invention is not limited to above embodiment, and any technical solution of the present invention that do not depart from only carries out to it improvement or change that those of ordinary skills know, within all belonging to protection scope of the present invention.

Claims (10)

1. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation, high-definition picture X is through degrading Procedure Acquisition p width low resolution observed image Y _k(k=1 ..., p), the size of every width observed image is m × n, it is characterized in that, the described p width low resolution observed image Y that utilizes _k(k=1 ..., the super resolution ratio reconstruction method of p) rebuilding high resolving power estimated image comprises the steps:

(1) in p width low resolution observed image, choose reference frame image Y _ref(1≤ref≤p) and non-reference frame image Y _k(k=1 ..., ref-1, ref+1 ..., p), the compound motion model that adopts global parameter motion and local light stream for the sub-pix motion between reference frame image and non-reference frame image, the relation table between reference frame image and non-reference frame image is shown:

wherein m _krepresent two dimensional motion field,

represent global parameter motion,

for local light stream campaign, θ _kfor globe motion parameter,

represent the reference frame image with non-reference frame image prediction, ε _k(x) represent residual image;

(2) solve globe motion parameter θ _k=(a ₀, ₁, a ₂, a ₃, a ₄, a ₅) and local light stream adopt globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) and local light stream

method carry out image registration, obtain the sports ground m of non-reference frame image with respect to reference frame image _k(x), utilize m _k(x) construct motion converter matrix M _k;

(3) computing reference two field picture Y _refr times of interpolation image X, non local priori parameter h _{i, j}the European threshold value of (0≤i<rm, 0≤j<rn) and similar image;

(4) utilize non local priori parameter, European threshold value, and with interpolation image

as the initial pictures of full resolution pricture X, calculate interpolation image in the similarity weight w of each pixel (i, j) and other pixels (s, t) _nLM[i, j; S, t], wherein 0≤i<rm, 0≤j<rn, 0≤s<rm, 0≤t<rn, utilizes similarity weight w _nLMbuild non local weight matrix S;

(5) utilize motion converter matrix M _ksolve cost functional with non local weight matrix S

wherein B _kfor observed image Y _kcorresponding clear function, the M of falling _kfor observed image Y _kwith respect to the sub-pix motion of reference frame image, non local weight matrix S is the non-local mean wave filter of a self-adaptation high-definition picture X, and ρ >0, adopts conjugate gradient iterative procedure to minimize cost functional, obtains the high resolving power estimated image of rebuilding.

2. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, it is characterized in that: described step (1) adopts the compound motion model of global parameter motion and local light stream for the sub-pix motion between reference frame image and non-reference frame image, wherein motion vector is m _k(x)=[m _{k, u}(x) m _{k, v}(x)], x=[x _ux _v], two dimensional motion field m _kbe expressed as

wherein

represent global parameter motion,

for local light stream campaign, θ _kfor globe motion parameter.

3. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: described step (2) solves globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) comprise the steps:

(21A) adopt a ₀, a ₁, a ₂, a ₃, a ₄, a ₅the affined transformation of six parameters is as global parameter motion model:

(22A) set up least square standard

solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅).

4. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 3, is characterized in that: described step (22A) is set up least square standard

solve globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅) comprise

Following steps:

(221A) by θ _kwrite as θ _k+ Δ θ;

(222A) least square cost functional is carried out to Taylor expansion and obtain the linear function about Δ θ;

(223A) function after launching being carried out to a series of arithmetic operations obtains wherein

(224A) judge whether to meet || Δ θ ||≤e, threshold value 0≤∈≤0.01, if do not meet, θ _k← θ _k+ Δ θ, and return to step (223A); If meet, show parameter θ _kconvergence, stops iteration, obtains globe motion parameter θ _k=(a ₀, a ₁, a ₂, a ₃, a ₄, a ₅), finally can obtain global parameter motion .

5. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: described step (2) solves local light stream

comprise the steps:

(21B) utilize gradation of image shape constancy hypothesis and image gradient shape constancy hypothesis to obtain about local light stream

data degree of confidence energy function

wherein,

function

wherein 0< τ≤0.01;

(22B) according to image segmentation, smoothness assumption obtains level and smooth penalty

(23B) obtaining whole energy function is E (u, v)=E _data+ α E _smooth, wherein α >0 is regularization parameter;

(24B) adopt Nonlinear Numerical method for solving to whole energy function E (u, v)=E _data+ α E _smoothask optimal value just to obtain local light stream

solution.

6. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 5, is characterized in that: described step (2) is utilized m _k(x) construct motion converter matrix M _kcomprise following steps:

(21C) calculate the sports ground m of non-reference frame image with respect to reference frame image _k(x)=m _k ^g(x)+m _k ^l(x), and by sports ground m _k(x) carry out r times of linear interpolation, obtain the sports ground m after interpolation _k(x), described r is the enlargement factor of the final high-definition image of rebuilding;

(22C) calculate relative displacement Δ x _k=m _k(x)-x=(Δ x _{k, u}-Δ x _{k, v}), and d _k=Δ x _{k, u}-floor (Δ x _{k, u}), e _k=Δ x _{k, v}-floor (Δ x _{k, v}), wherein operational character floor (.) represents to get the maximum integer that is less than or equal to designated value;

(23C) calculate motion converter matrix M _kin the value of each element: M _k(j*m+i, floor (Δ x _{k, u})+x _u+ ceil (Δ x _{k, v}+ x _v+ 1) * m)=d _k* (1-e _k), M _k(j*m+i, ceil (Δ x _{k, u})+x _u+ ceil (Δ x _{k, v}+ x _v+ 1) * m)=d _k* e _k, M _k(j*m+i, floor (Δ x _{k, u})+x _u+ floor (Δ x _{k, v}+ x _v+ 1) * m)=(1-d _k) * (1-e _k), M _k(j*m+i, ceil (Δ x _{k, u})+x _u+ floor (Δ x _{k, v}+ x _v+ 1) * m+x)=(1-d _k) * e _k, wherein operational character ceil (.) represents to get the smallest positive integral that is greater than or equal to designated value, for matrix M _kother element values except these four row in capable of j*m+i be zero.

7. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: described step (3) is calculated non local priori parameter h _{i, j},

wherein std (N _{i, j}) be centered by pixel (i, j), region of search is N _{i, j}standard deviation, β is greater than zero constant (1< β <5), σ ²for reference frame image Y _refnoise variance, r is image enlargement factor, described reference frame image Y _refnoise variance σ ²be estimated as:

the pixel sum that wherein n is image,

n (x _i) be pixel x _ineighbours territory.

8. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: the maximum weighted Euclidean distance that in described step (3), the European threshold value of similar image adopts is 4 σ ², also:

9. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: described step (4) is utilized non local priori parameter, European threshold value, and with interpolation image

as the initial pictures of full resolution pricture X, calculate interpolation image

in the similarity weight w of each pixel (i, j) and other pixels (i, j) _nLM[i, j; S, t], utilize similarity weight w _nlMbuild non local weight matrix S and comprise following steps:

(41) calculate the similarity of the block of pixels centered by coordinate (i, j) in each block of pixels centered by coordinate (i, j) and its neighborhood N (i, j)

, wherein R _{i, j}the image block that is pixel centered by (i, j) extracts operator, the weighted euclidean distance that represents two image blocks, wherein a>0 is the standard deviation of gaussian kernel function, h _{i, j}for filter smoothing parameter, depend on noise size and the image itself of image, f is the normal function that depends on the geometric distance of two central pixel point;

(42) all pixel similarity values in normalization pixel (i, j) and neighborhood N (i, j),

(43) build non local weight matrix

10. the super resolution ratio reconstruction method based on compound motion and the non local priori of self-adaptation according to claim 1, is characterized in that: in described step (5), work as

time, iteration stops, and wherein n is iterations.

Images