"IMAGING SYSTEM AND METHOD"
Technical Field This invention relates to imaging systems and methods .
As used herein the expressions "imaging system" and "imaging method" refer to systems and methods in which radiation is detected and an image produced. As used herein the expression "radiation" is to be given a broad meaning and includes the emission of any rays, wave motion, or particles from a source. It includes light and other form of electromagnetic radiation. As used herein the expression "image" is also to be given a broad meaning and includes all forms of representation of detected radiation including mathematical, digital and analog models representing an actual image, wherein the image is represented by a plurality of distributed signals. The invention has application to improving the resolution of high definition imaging systems and methods . The invention has particular but not exclusive application to imaging systems such as those which utilise charge coupled devices (CCDs) ie digital cameras, camcorders etc and for illustrative purposes reference will be made herein to such devices. However it will be appreciated that the invention is also applicable to other imaging systems such as spectrometry for example.
Background of Invention
It is known that some high definition cameras achieve a higher resolution than the detector element array used in their detection system by masking each detector element array element with an opaque mask which allows light through a small central aperture. By moving this aperture around an area the size of the detector element, it is possible to progressively build a full sampling of the pixel area at the resolution defined by
the aperture .
It is also known that imaging systems exist which improve the signal to noise in an image by using a plurality of different masks on each detector element and combining the results from these masks to construct an image at higher resolution than the detector system. These systems are generally derived from Hadamard matrices and applied in spectroscopic imaging.
Summary of Invention
The present invention aims to provide an alternative to known imaging systems in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, and to known methods of improving the resolution of such imaging systems .
This invention in one aspect resides broadly in a method of improving the resolution of an imaging system in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, the method including:- improving the signal to noise ratio of the image. In another aspect this invention resides broadly in an imaging system in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, the system including:- means for improving the signal to noise ratio of the image .
Masking matrices can be constructed which produce a low uniform error. However these so-called S matrices have inconvenient dimensions (3, 7, 15, 19, 23, 31, 35,
43,47, 63, 71, 79, 83 ) none of which are power of two (2k) and none of which are square integer (k2) . This has disadvantages in that n x n square masks and hierarchical refinement of the resolution cannot be utilised. Accordingly it is preferred that the masks are chosen so as to allow for square (n x n) arrays of aperture elements for each detector and so that the apertures can be arranged as an embedded set which
facilitate heirarchical refinement of the resolution.
Most preferably the distribution of the signal to noise ratio of the image is uniform.
In an embodiment in which the radiation is detected by a charge coupled device having a two-dimensional array of light sensitive detector elements and resolution is enhanced by sequentially masking discrete areas of each detector element with an apertured opaque mask, it is preferred that the signal to noise ratio of the image is improved by means of a coded aperture system.
As used herein the expression "coded aperture system" means an aperture system which utilises a series of apertures to sample an image or other extended signal. (The apertures modulate the signal on the detector element such that the detector integrates the modulated result over the aperture. By combining the results from a number of different apertures or from displaced versions of the same aperture, the signal can be reconstructed at a higher resolution than the aperture dimensions would normally allow) .
It is preferred that the size of the apertured opaque mask substantially corresponds with that of the light sensitive detector elements and that the mask comprises a two-dimensional array of apertures substantially aligned with each detector element, each aperture comprising a two-dimensional array of transparent and/or opaque aperture portions whereby the configuration of the mask can be represented mathematically by a masking matrix, the transparent and opaque aperture portions being represented in the matrix by +1 and 0 respectively.
The two dimensional arrays may be other than square arrays and the sub-elements or pixel array need not necessarily be constrained to a power of two configuration, but it is preferred that there are N x N sub-elements or pixels in the mask where N = 2n with n an integer. It will be appreciated that the analysis which can be applied to a 4 x 4 array can also be applied to an
8 x 2 or l6 x l array .
In a preferred embodiment the configuration of the apertured opaque mask is determined by stochastically searching the space of sets of apertures achieved by complementing individual apertures in the set. As used herein the expression "complementing" means changing opaque regions to transparent regions and vice versa. It is preferred that the stochastic searching commences with a canonical set of apertures which does not produce spatially uniform signal-to-noise response in the reconstructed image and progresses until a suitable set of apertures is achieved.
In another aspect this invention resides broadly in a method of determining the configuration of an apertured opaque mask for improving the resolution of an imaging system in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, the size of the mask substantially corresponding with that of the light sensitive detector elements and the mask comprising a two-dimensional array of apertures substantially aligned with each detector element, each aperture comprising a two-dimensional array of transparent and/or opaque aperture portions, the method including:- representing the configuration of the mask mathematically by a masking matrix in which said transparent and opaque portions are represented by +1 and 0 respectively; inverting the masking matrix, and stochastically searching the space of masking matrices achieved by complementing rows of the masking matrix.
In a further aspect this invention also resides in an apertured opaque mask for improving the resolution of an imaging system in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, the mask including:- a two-dimensional array of apertures each comprising
a two-dimensional array of transparent and/or opaque aperture portions, and the size of the mask substantially corresponding with that of the light sensitive detector elements; wherein the configuration of the transparent and opaque portions is determined in accordance with the method defined above.
Description of Drawings In order that this invention may be more easily understood and put into practical effect, reference will now be made to the accompanying drawings which illustrate a preferred embodiment of the invention, wherein:-
FIGS 1,3 and 5 illustrate the Haar wave-packet basis (also known as the Hadamard- alsh basis) for 2 x 2, 4 4 and 8 x 8 image blocks respectively;
FIGS 2 and 4 illustrate the division of the CCD elements into pixels and the relationship of the pixels to the transform coefficients for the 2 x 2 and 4 x 4 image blocks respectively;
FIG 6 illustrates an arrangement of masked detector elements for a linear scanning detector, and
FIG 7 illustrates a set of 4 x 4 pixel apertures with uniform noise sensitivity.
Description of Preferred Embodiment of Invention
For the purpose of the description of the preferred embodiment of the invention and an explanation of the underlying theory, each CCD pixel or element is considered to consist of an array of higher resolution sub-pixels or sub-elements (say N x N where N = 2π with n an integer). This high resolution image block contained within each CCD pixel is sampled by means of the Haar wave-packet expansion. Throughout the specification in general, the term "element" will be used to refer to the CCD detector elements and the term "pixel" used to refer to the N x N sub-elements into which the CCD element is divided.
Haar wave-packet expansion If the N x N pixels are broken into 2 x 2 blocks, each block can be represented in terms of a basis illustrated in FIG 1. In each of these four weighting functions, the weights are either +1 (black) or -1 (white). Applying these weights to the 2 2 image block will give four coefficients from which the image block samples can be exactly reconstructed.
The basis for the 2 x 2 Haar wave-packet transform can be represented with a matrix in which the rows of the matrix are the basis vectors. The division of the CCD element into pixels and the relationship of the pixels to the transform coefficients is illustrated for the case above by FIG 2. The shaded coefficients are drawn from the shaded set of pixels by the transformation in equation 1. The reconstruction of the pixel values from the basis coefficients for a single block is given by equation 2.
The N x N pixel image block can therefore be represented as four N/2 x N/2 sub-band image blocks. Each N/2 x N/2 sub-band block can itself be encoded as four N/4 x N/4 sub-band blocks, using the same 2 x 2 block coding scheme. Alternatively this can be viewed as a single step encoding of the original N x N block by breaking the block into 4 x 4 pixel blocks and filtering
with the weighting filters illustrated in FIG 3 which is equivalent to a two-dimensional Hadamard/ alsh basis.
In the basis illustrated in FIG 3 the weighting functions only take the values of 1 or -1, but now each weighting function consists of a 4 x 4 array. The blocking of the pixels in the CCD element is illustrated in FIG 4. The pixels in the shaded region are used to generate the shaded coefficients. The matrix form of the 4 x 4 block coding is given in equation 3.
An unusual symmetry is observed between the row/column ordered coefficients of the matrix and the masks in FIG 3. A row of the matrix corresponds to a single block in the mask (one of the 4 x 4 regions) and yet the whole matrix has the same pattern as in the mask.
This process can be continued until there are N2 weighting functions each of N x N values, and the resulting sub-bands consist of single values. The full set for N=8 is illustrated in FIG 5. The matrix form has the same pattern of 1 and -1 elements as the set of masks, with black interpreted as 1 and white interpreted as -1. Because the coefficients are single values the
masks can be used to sample the image, one coefficient value being derived from a detector masked by one of each of the masks. Given this sampling of an image block, the image block can be constructed from the samples by using the inverse of the matrix.
The matrix is easily constructed by a process which starts with just the first row vector, which is all l's. The process is illustrated with the Matlab source code below: -
n = 4; xSz = 2Λn; n = log(xSz) /log(2) ; ySz = 2; jO = 2; k = 2;
A = ones ( l,xSz) ;
A(2,:) = [A(l,l:2:xSz) ,-A( 1, 1 : 2 :xSz ) ]; for j = l:n-l, for i = jO:ySz, k= k+1;
A(k,:) = [A(i,l:2:xSz) ,A(i, 1: 2 :xSz ) ] ; k = k+1;
A(k,:) = [A(i,l:2:xSz) , -A(i, 1:2 :xSz ) ] ; end jO = ySz+1; ySz = 2*ySz; end
The family of matrices generated by this algorithm is referred to as ΛN2. The family of matrices can serve as either a one-dimensional linear basis for vectors of length N2 or, through the unusual symmetry mentioned above, as a two-dimensional basis for blocks of N x N pixels .
The inverse of the matrix is the same as the matrix except for a factor of l/N2.
Masking
In principle an image block can be sampled with multiplicative intensity masks corresponding to an orthogonal basis for images within the block. However,
because the intensity measured by the detector element is strictly positive, negative values in the intensity mask are difficult to support. Practical difficulties also mean that a continuously variable mask is difficult to achieve with any accuracy. However the Haar wave-packet basis described above is strictly binary valued with a weighting factor of 1 or -1. If the basis is modified slightly and all the -1 masking factors set to zero, the resulting mask can easily be achieved with opaque masking.
This set of weighting functions is a complete (although non-orthogonal) basis for the image block. The matrix representation of the resulting basis for 4 4 blocks is given in equation 5.
(5)
Equation 5 is more succinctly written as
C-k = ^16 (6) where
= ^ + V/2 (7)
where 016 is a 16 x 16 matrix of l's.
'It can be seen by inspection that the inverse transformation is given by
Zk ^~l6-c k (9)
The inverse matrix M AXT is in general
Where Q -.N 2 is an Ν2 x Ν2 matrix with zeros everywhere except in the top left coefficient which is 1. This can be verified by checking the product
a^* = 2(A-TA)/W2 ' (A+ °A2
I N2
_ -j- (11)
N2 2 N2
- V 2 + QV ~ ~2~
=
where Ik is the k x k identity matrix.
Sampling Process
The sampling process for the kth n x n block of pixels in the image (of the size of a CCD element) consists of integrating the intensity through each of the N2 masks represented by the matrix vi 2 to derive the set of coefficients c^ The pixel values can then be recovered by simply applying the transform
Pk = M~^ck . (12)
The arrangement of the blocks for most efficient acquisition of the image depends on the application. For many applications, the most efficient acquisition is achieved when the masks are arranged so that each sequence of N2 masks is in a row with the masks in each column being identical. This sequence could be repeated to reduce the scan size if necessary. However if the total acquisition time is not critical the scanning head need only be N2 elements wide and could be as high as is necessary to cover the image. The array is moved in a linear scan, as indicated in FIG 6, in steps of the CCD element separation. The output of each element can be recorded at each step to build a set of coefficients for each block.
The masks in FIG 6 are arranged in scale order so that the first 4 elements can be used to reconstruct to a resolution of 1/2 the CCD element size; the first 16 elements can be used to reconstruct to a resolution of 1/4 the CCD element size etc. This allows variable (dyadic) resolution scanning with identical scanning
geometry from the same detector.
Noise Sensitivity
Prior art sampling processes use masks which effectively have only one pixel area exposed. This corresponds to using the identity matrix instead of M -, in the sampling and produces a trivial reconstruction problem. The advantages of the sampling process in accordance with the present invention become evident when noise sensitivity is considered.
For the purposes of simplifying the analysis it is assumed that each detector element has associated with it zero-mean additive Gaussian-distributed noise. This is a reasonable approximation to the behaviour of a real detector because much of the noise is due to thermal processes and is substantially independent of the light falling on the detector. The signal then looks like
where Pk is a vector of length N2 with zero mean Gaussian distributed random values _ with a scalar variance of σ2 • The reconstructed image will be
Pk = M-^ck + M^ k • (14)
= ? + MXpk So the noise in the reconstructed image is
If the matrix M , is written as
At
where R
k is the k
th row of M
~ 2 , then the noise in the reconstructed image becomes
1 = 1
where ei is a column vector which is zero everywhere except the ith element which is 1. But if the noise in each coefficient is independent zero-mean Gaussian noise with uniform variance σ~, then the noise in the reconstructed signal will also be Gaussian with a variance in element i of ||Λ 2σ2 . The mean squared error in the signal is then simply
where *tø is the sum of the square of the coefficients of the matrix M~ . But from equation 10 it is clear that all of the elements of N—2 ~ have a magnitude of 1 except the f rst element whic 2h h Na~s a magnitude of N1 1 . The mean square error for the reconstructed image block is then
N4 -
CN4' ] +r 1 /N2
(19)
= A.(5N2 - 4) c N4
The mean squared error in the "reconstructed" image pixels from a masked camera which uses masks which cover all but the area of one pixel, as is the case with known prior art devices is
Each pixel is subject to the noise from a single
measurement. For the system disclosed here there is a root mean square gain in the signal to noise ratio for constant illumination of
Remarkably, for a system with N=2 (2 2 pixels per CCD element) there is no gain in the sensitivity. For a system with N=4 (4 4 pixels per CCD element) this would represent a sensitivity gain of 5.3dB over a single pixel masked system with the same resolution enhancement, the same illumination and the same number of exposures per element.
Spatially Variable Sensitivity
To obtain an accurate picture of the response of the system to noise, it is necessary to look at the distribution of the noise in the reconstructed image as well as the mean square error. The variance p in the reconstructed pixels will be
where the variance in the individual coefficients vc is simply σ2 for each coefficient and -V is a matrix with each of its elements being the square of the elements of M~ N,2 .
However this is simply
The variance on the reconstructed image is then
The variance of the reconstructed components is clearly not uniform except for N=2. This presents a problem. Although most of the pixels actually have slightly lower noise than the means square error, the first component has no signal to noise gain over a pixel asking scheme .
Correcting the Spatially Variable Sensitivity
This spatially variable noise can be corrected without affecting the mean square response by redistributing the noise on the reconstructed image.
This is achieved by selectively complementing individual masking patterns .
Thus the N2 x N2 matrix XN2 , Ϊ when multiplied on the left of the masking matrix M^. complements the ith masking vectors ie the complementing process turns l's to 0 ' s and 0 ' s to l's in one row of MN. - Xnτ, i will be the identity matrix with the ih row modified so that the first element is 1 and the diagonal element is -1. This produces a modified masking matrix
^N i ~ XN i^N- (25)
The inverse of this modified masking matrix will be
*tø.« = MjkXjh. i (26)
The intuitive result that
is its own inverse can easily be verified and thus
Xjh. l = M XfP. i (27)
Any of the basis vectors can be complemented (except the first for which the complement would be zero). The new matrix is thus
Ntf 'O.'i .,] = X N , ■ XN , i,XN- ■ M Ό AΓ- (28)
and the inverse will be
AΛ/2, A* = ~ M M NXixN- X X 3, .X A2, (29)
Empirical observation suggests that complementing any combination of the rows of NZ (except the first row) has no effect on the mean squared error in the reconstruction. Complementing row i of N2 is equivalent to multiplying MN2 on the right by the matrix XN2 k . This adds the kth column of M~ NZ. to the first column and then multiplies the kth column by -1. Complementing several of the masks (rows of ^μ ) has the effect of adding those columns of MN ~ 2 to the first column of ~ N2 and multiplying those columns by -1. It can be shown that this has no effect on the total squared error.
If the matrix MN2 is written as
^ ■1 = [0 c, ... cN2_ (30)
where C.- is the ith column of M~ n ' , then the total squared error in the reconstruction becomes
σ2 = (||^||2σ2)/N2
However the complementing operations change only the sign of columns 2 to N
2, and there is no change in the contribution of these columns to |ffl^UI
2. Thus the change in the total error in the reconstruction is simply that due to the first column which gives
Now
So
But
where 1 is vector with all elements equal to 1. This vector and the column vectors Q- (i=2...Ν2) are all mutually orthogonal so
N2 (36)
— 0 i > \ , j > 1, i≠y
σc 2 N2 N2 (37)
= 0
thus proving that complementing the basis functions in this way does not change the mean square error in the reconstruction .
Given this result, constraints can be found which must be satisfied for complemented masks to produce uniform noise sensitivity in the reconstruction.
It is assumed that I of the basis masks in an Ν2 x
Ν2 matrix ^-N2, [j j2 j,] have been complemented, and that this set of masks produces a uniform noise sensitivity. Therefore all elements of the first column must have the same magnitude. But if the mean squared error in the reconstruction is unchanged after complementing the basis vectors then the magnitude of the first elements
However these values all result from summing integers so N must be an even integer. Given that it has been assumed for the purposes of the construction of the basis that N is a power of 2, this is not a restrictive constraint. (This analysis has only been tested for N equal to a power of 2 and it is less clear what the constraints would be for more general Hadamard matrices . If I basis masks have been complemented then the first element of the first column must be (l-N/2+1) which gives
N2 N 1 (38) —
and thus for a given n there are only two possible
choices for the number of basis masks which must be complemented to achieve a uniform noise sensitivity.
Configuring the masks
The above analysis illustrates that there are constraints on the sets of masks which must be satisfied to produce uniform noise sensitivity.
To find a set of masks a search is performed over the space of possible masks. This search is constrained by the result embodied in equation 39. Random rows (excepting the first) of a masking matrix N2 are complemented with the constraint that the number of complemented rows satisfies the constraint given by equation 39. This constraint is derived from consideration of the first element of the first column of the inverse of the masking matrix. Further constraints could be identified by considering the other elements of the first column of this matrix. An exhaustive search could then be performed, however for coding efficiency a stochastic search is performed. The inverse of this matrix is calculated and tested for uniform magnitude of the elements of the first column. A row, i, of the masking matrix which has already been complemented is then randomly chosen together with a row, j, of the masking matrix which has not been complemented. These two rows are then effectively complemented leaving the masking matrix with the same total number of complemented rows with respect to M .
The first column of the inverse of the masking matrix is then checked and if it has the same magnitude for all components, a set of masks has been derived which produce the desired uniform noise sensitivity in the reconstruction.
An example of a set of masks configured by this process is illustrated in FIG 7. It can be seen that the masks have the same form as the original masks illustrated in FIG 3 except that five of them are the complements of the original masks.
It will be appreciated that the method and system of the present invention has a number of advantages over known imaging systems in which radiation is detected and an image which is represented by a plurality of distributed signals is produced, and over known methods of improving the resolution of such imaging systems .
High definition cameras which achieve a higher resolution than the CCD pixel array used in their detection system by sequentially masking each CCD array element with an apertured opaque mask suffer because the signal from each element is reduced in proportion to the aperture size. Such cameras therefore require very bright lighting to operate. As the CCD elements are sensitive to the infra-red, heating of the opaque mask by the intense lighting required can degrade signal to noise performance or lead to relatively long exposures.
On the other hand the performance of high definition cameras incorporating the system and method of the present invention has the advantage of using the same number of samples but in such a way that each aperture transmits at least 50% of the incoming flux
onto the CCD. This reduces the lighting requirements and at the same time improves the signal to noise ratio.
The present invention permits the use of a linearly scanned imaging detector with larger sensing elements than the required resolution. This allows the use of coarser features in the integrated circuits of the detector than would be required for a fixed resolution detector of equivalent effective resolution. This results in higher yields and cheaper production technology for the chips, whereby apparatus in accordance with the invention can be fabricated at lower cost than standard devices with similar resolution.
In comparison with known scanned imaging technology, other advantages of the present invention include:
* The signal to noise performance is improved and results in reduced exposure time or reduced lighting requirements.
* The scanning of the detector consists of a single linear scan of the device across the image plane. This simpler geometry has advantages in terms of the mechanical system required and the speed with which it can be effectively scanned, thus allowing the device to take advantage of the improved signal to noise performance.
* The set of masks can be arranged such that the first four masks can be used to reconstruct an image with twice the resolution and the first 16 masks can be used to construct an image with four times the resolution etc . This embedded
construction allows for adaptive resolution changes with a single set of masks.
* Reduced exposure times means that the device can more easily be used on live subjects and in more normal lighting conditions than prior art technology.
* The linear scan means that the technology is suitable for use in scanning imaging devices such as flat bed scanners, drum scanners, or film scanners .
* The nature of the signal produced makes it more amenable to hardware/software based compression of the resulting image with potential for continuously variable fidelity in a lossy scheme.
It will of course be realised that whilst the above has been given by way of an illustrative example of this invention, all such and other modifications and variations hereto, as would be apparent to persons skilled in the art, are deemed to fall within the broad scope and ambit of this invention as is herein set forth.