WO1997034423A1

WO1997034423A1 - Image sensor with multiple resolution and compression capability

Info

Publication number: WO1997034423A1
Application number: PCT/US1997/003998
Authority: WO
Inventors: Andrew K. Juenger
Original assignee: Polaroid Corporation
Priority date: 1996-03-14
Filing date: 1997-03-14
Publication date: 1997-09-18
Also published as: EP0886974A1

Abstract

An electronic still camera and a method associated therewith is disclosed, which is useful with a sensor of rectangular pixel geometry. The invention provides dual resolution modes for producing square pixels for display. In one resolution mode, neighboring pairs of pixels are averaged in the direction normal to the major axis of the rectangular pixels within each color plane such that one half as many square pixel RGB triplets are formed as there were original single color rectangular pixels, while in the second resolution mode new RGB triplets are interjected by interpolation in the direction parallel to the major axis of the rectangular pixels such that twice as many square pixel RGB triplets are formed as there were original single color rectangular pixels. A method of compression for stored images is also provided.

Description

IMAGE SENSOR WITH MULTIPLE RESOLUTION AND COMPRESSION CAPABI LITY

Background of the Invention

The present invention relates generally to an apparatus and method for interpolating and compressing image data and, more particularly, to an apparatus and method for sensing color sub-sampled image data and thereafter interpolating for non-color sub-sampled image data that substantially reduces color fringing and preserves detail while allowing selection of the overall quantity of image data.

Electronic imaging cameras for recording still images are well known in the art. Such cameras can record a plurality of still images on a single magnetic disk or tape in either analog or digital format for subsequent playback on an output device, such as any well-known cathode ray tube (CRT) viewing device for example. Printers may also be utilized with such cameras in a well-known manner to provide hard copies of the recorded images.

Such electronic imaging still cameras often utilize two-dimensional image sensing arrays such as charge coupled devices (CCDs) which integrate incident scene light over a predetermined time to provide an electronic information signal corresponding to the scene light intensity incident on the array. Such two- dimensional image sensing arrays comprise a predetermined number of discrete image sensing elements or pixels arranged in a two-dimensional array, in which each image sensing element responds to incident illumination to provide an electronic information signal corresponding to the intensity of the incident of illumination. In order to record color images, the incident illumination to the two dimensional image sensing array is filtered so that different image sensing elements receive different colored illumination. The filters are arranged in well known patterns across the face of an image sensing array, such as a repeating pattern of red, green, and blue stripes. Alternatively, individual image sensing elements or pixels across each line may be filtered in a repeating pattern of red, green, blue, and green filters in a two- by-two checkerboard, as is well known in the art. Since each image sensing element can only detect one color of illumination, the color information for the other colors not detected by that image sensing element must be filled in. Filling in the missing color information is generally accomplished by inteφolating the detected image data for each color to determine color values for all the colors for each image sensing element.

Conventional types of inteφolation, however, can provide images with objectionable color aliasing artifacts such as "color fringes" near shaφ edges. The conventional approach to solve this problem is to prevent the color fringes at the expense of image shaφness by blurring, otherwise known as anti-aliasing, the picture so that the edges are not shaφ enough to create a color fringe. Blurring the image in this manner, however, has obvious disadvantages, resulting in a reduction in resolution and a so-called "fuzzy" picture.

The missing color inteφolation also increases the number of pixels by a factor of three from the raw image data sensed by the CCD. Storing the image data locally

_* then oecomes a potential problem. The increased number of pixels means that a file size for storing the image, on local magnetic media for example, leads to a requirement for increased storage capacity. The pixels on a CCD can be rectangular in shape particularly when striped color filter arrays are employed. Thus, outputting the image on an output device that has substantially square pixels distorts the image. Since most conventional output devices such as printers and CRT's require square pixels, it is necessary to form substantially square pixels from the rectangular pixels prior to outputting the image. A result is to get the desired pixels geometry.

Summary

The aforementioned and other objects are achieved by the invention which provides, in one aspect, an electronic still camera and a method associated therewith for image processing to form a processed image with reduced color artifacts, where an image is represented by a plurality of image signals in three colors. The method is useful with an electronic still camera, and comprises the steps of separating the plurality of image signals into a first color plane, a second color plane, and a third color plane. For example, many conventional CCDs use colored stripes formed onto the CCD to filter colors, thereby making individual pixels associated with only one color. These individual pixels are broken into color planes. Therefore, if colored stripes of red, green, and blue (RGB) are attached to the CCD, then the pixels associated with each individual color are broken into a red color plane, a green color plane, and a blue color plane.

Since these colors have now been separated, voids now exist where the previous colors used to be. That is, in the red color plane, the pixel locations where green and blue had to have existed previously are now left void. Therefore, the next step is to interject inteφolated image signals in each of the color planes that correspond to the image signal locations left void by the separation step. This now forms a triplet for each color location in the image. Previously, a single pixel location would contain only information on one color, red for example. Now, with the inteφolated data, each pixel location has RGB information associated therewith.

Next, two difference signals are created at each pixel location that represent the differences between one channel and a second, and the first channel and the third within the RGB triplet. In the previous example using RGB, those difference signals can be any combination of the red, the green, or the blue, as long as from those three colors, two different difference signals are created. In the preferred embodiment, R- G and B-G are used.

The difference signals are then filtered using a median filter which substantially reduces or removes color artifacts. The image is subsequently reconstructed from the original image signals and the filtered difference signals to form a processed image which has substantially preserved detail and reduced color artifacts. The improved RGB triplets thus formed from linearly inteφolated RGB triplets have the same pixel geometry as the original single color pixels. In the case of rectangular pixel geometry, some method of producing pixels of square geometry must be employed for use with any display device that renders images on a raster of square geometry.

In the case of rectangular pixels of aspect ratio height twice the width, the invention provides dual resolution modes for easily producing square pixels for display. In one resolution mode, neighboring pairs of pixels are averaged in the direction normal to the major axis of the rectangular pixels within each color plane such that one half as many square pixel RGB triplets are formed as there were original single color rectangular pixels, while in the second resolution mode new RGB triplets are interjected by inteφolation in the direction parallel to the major axis of the rectangular pixels such that twice as many square pixel RGB triplets are formed as there were original single color rectangular pixels. In the case of rectangular pixels of aspect ratio other than height twice width, multi resolution mode can be supported with more complicated down sampling and up sampling schemes.

Finally, if the image data is to be stored, then compression becomes an essential characteristic. In the invention, the stored image signals are compressed versions of the two filtered difference signals and a compressed version of the single color signal common to both difference signals reconstructed from the filtered difference signals and the original sampled pixel values. In the preferred embodiment, compressed, filtered R-G and B-G are used as well as a compressed version of the G signal formed from the filtered R-G and B-G and the original RGB samples. The difference signals are highly compressible while the G signal formed from the filtered R-G and B-G and the original RGB samples provides the complete detail signal originally sensed, and the overall image size can be drastically compressed. Upon decompression of the compressed image data, pixels of square geometry can be reconstructed in either of the dual resolution modes. The aforementioned and other aspects of the invention are evident in the drawings and in the description that follows.

Brief Description of the Drawings

The foregoing and other objects of this invention, the various features thereof, as well as the invention itself, may be more fully understood from the following description, when read together with the accompanying drawings in which:

Figs. IA and IB show, respectively, pixel locations as found on a CCD, and a representative example; Figs. 2A and 2B show the pixel locations of the example of Figs. IA and IB, respectively, after linear inteφolation is performed;

Fig. 3 shows the representative example of Figs. IB and 2B during performance of median filtering;

Fig. 4 shows the result of the median filter on the example of Fig. 3;

Fig. 5 shows reconstruction of pixels of Fig. 3; and

Figs. 6 A, 6B, and 6C show, respectively, the pixel values reconstructed after median filtration of the color difference signals, the pixel values after averaging in the direction normal to the major axis of the rectangular pixels, and the pixel values after inteφolation along the major axis of the rectangular pixels.

Detailed Description

While the present invention retains utility within a wide variety of photographic devices, such as video cameras for example, and may be embodied in several different forms, it is advantageously employed in connection with an electronic still camera. Though this is the form of the preferred embodiment and will be described as such, this embodiment should be considered illustrative and not restrictive.

As previously discussed, electronic imaging cameras conventionally record color images by using an image sensing array comprising a predetermined number of discrete image sensing elements or pixels arranged in a two-dimensional array in which the image sensing elements respond to incident illumination to provide an electronic information signal corresponding to the intensity of the incident illumination. Such an image sensing arrays may be a CCD of the frame transfer type. It is well known to sense color images using a single two-dimensional CCD array by filtering the illumination incident to the image sensing array so that different groups of the image sensing elements arranged in well-known patterns across the image sensing array receive different wavelengths for colored illumination. Thus, each color of illumination is sampled by each group of image sensing elements, and thereafter inteφolated to provide color values corresponding to the other groups of image sensing elements. The full color image is therefore estimated or inteφolated between the different groups of image sensing elements or pixels to fill in all colors for each image sensing element or pixel.

Conventional types of inteφolation provide images with objectionable aliasing artifacts such as color fringes near shaφ edges. An example of how a shaφ edge in a sample to be recorded can create color fringes, when the image of the subject is reconstructed using conventional inteφolation methods, will be discussed herein.

As previously described, a typical CCD arrangement includes color filter stripes thereon such that individual stripes of pixels measure an intensity of light for only a single color. Figure 1 A shows a typical arrangement of a CCD using color stripes, where individual pixels are marked by R, G, or B representing red, green, and blue colors. Numerical designations associated with the several colors, i.e., Rj Gi and Bι, together are referred to a triplet.

In order to illustrate the invention, an example is shown for each step. The example is of an image having a shaφ contrast such as a shaφ white-to-black transition. The example begins in Fig. IB where a light intensity level is shown graphically for a white-to-black transition in an image. For the graph, the abscissa is c _, pixel number across the CCD and the ordinate is by signal level, often measured in volts. Keys to Fig. IB is as follows: the solid line represents the edge transition in the original continuous image of the scene before sampling, the circle represents an original red sample, the triangle represents an original green sample, and the square represents an original blue sample. It can be seen from the illustration that for the original continuous image the falloff during the transition is substantially vertical. This indicates a shaφ transition. A slope in the falloff would be indicative of a more gradual, or less shaφ, transition.

Fig. 2A again shows a pixel layout of the pixels of Fig. IA, but now the pixels have gone through the step of inteφolation. In the preferred embodiment, this is a linear inteφolation, as will be later hereinafter described. The convention used is that a capital letter represents original raw data and a lower case letter represents inteφolated data. Therefore, the colors inteφolated between R] and R are shown as r,¹ and ri". Using linear inteφolation, ' contains 2/3 of the intensity of R and 1/3 of the intensity of R₂ Likewise, H" contains 1/3 the intensity of R₍ and 2/3 the intensity of R . By linearly inteφolating values throughout for red, green, and blue, the colors can be broken into three color planes such that there now exists triplets for each individual pixel location, where previously there only existed raw data in one color for each location.

Continuing the example, Fig. 2B shows the step of linear inteφolation on the white-to-black transition shown in Fig. IB. As shown in Fig. IB, the lines indicate paths of the linear inteφolation where the circles, triangles, and squares represent original red, green, and blue samples and the x's represent inteφolated values. Figure 2B illustrates how the inteφolation tends to blur the shaφ transition previously seen in the image by inteφolating where the dashed line represents red, the dotted line represents green, and the solid line represents blue. The slope of the line is indicative of a less shaφ transition, thus blurring the image. Fig. 2B also illustrates how the inteφolation tends to create color fringe artifacts. In the original white-to-black transition the red, green, and blue intensity transitioned at the same spatial location while it can be seen in Fig. 2B that the red signal after inteφolation initiates transition before the green which initiates transition before the blue. An image reconstructed from the data represented in Fig. 2B would show a pronounced white-to-cyan-to-puφle-to-blue-to-black color fringe.

A next step is to create two difference signals at each location. The color fringe artifact at a white-to-black transition in the scene followed by inteφolation within color planes would appear as a sudden rise in a color difference signal followed by a corresponding sudden fall or as a sudden fall followed by a sudden rise. It is this rapid increase and decrease in the difference between the colors which is a characteristic of objectionable color fringing, and is not simply a sudden rise in the difference between colors, which is indicative of a change from one color to a different color. Thus, it is unlikely that a real scene would result in the creation of such a color spike, and it is not desirable to create such a color spike as a result of a method of inteφolation chosen.

Fig. 3 illustrates the aforementioned color spikes, where in the preferred embodiment, the difference signals are performed by subtracting the intensity of green from the intensity of red and subtracting the intensity of green from the intensity of blue, that is R-G and B-G. One skilled in the art will realize that the actual choice of colors that determine the difference signals is somewhat arbitrary. Consistency after the choice is made is of primary importance. Therefore, other color difference signals can be chosen without detriment to the invention. Fig. 3(i) graphically illustrates the results of the difference signal for R-G, for the signal shown in Figure 2B. Fig. 3(iii) graphically illustrates B-G.

Various methods can now be employed to remove the color aberration. In the preferred embodiment, a median filter is employed, such as that shown in commonly assigned patents 4,663,655, issued May 5, 1987, 4,774,565, issued September 27, 1988, and 4,724,395, issued February 8, 1988, all to William T. Freeman, which are each incoφorated herein by reference. Simply stated, the median filter takes a series of pixels, such as those shown in Fig. 3(ii), and replaces the pixel value at the center of the filter region with the median value of all the pixels within the region. Though various numbers of pixels can be used, in the preferred embodiment, the median filter uses nine pixels and returns the median value of the nine pixels. Therefore, as the filter is worked horizontally across the signal, it can be seen that unless the spike is longer than one half the pixel length, the spike in the signal will be essentially eliminated.

Figure 4 illustrates the results of passing the median filter across the difference signals, where 4(i) corresponds to the R-G difference signal shown in 3(i), and 4(ii) corresponds to the B-G difference signal shown in 3(iii), where these signals can now be referred to as (R-G)¹ and (B-G)', respectively. The differences are now flat and the reconstructed colors will now remain substantially constant with respect to each other before, during and after the color transition. Reference is again made to Figure IB as an example of this.

A next step is shown in Figure 5, where the pixels are reconstructed from original raw RGB data and filtered (R-G)' and (B-G)' difference signals. The colors are restored using the following relationships:

R@R=R

R@G=G+(R-G)'

R@B=B-(B-G)'+(R-G)'

G@R=R-(R-G)'

G@G=G G@B = B - (B-G)'

B@R = R - (R-G)' + (B-G)^*

B@G = G + (B-G)'

B@B = B

These relationships illustrate the properties of preserving the original sampled values and extracting detail at every pixel regardless of its original color. The detail extracting property is accomplished by removing the sudden spikes in the color difference signals which effectively produce local color sameness, so in regions of shaφ edge transitions, when the original sampled pixels values change in response to edge transitions, the reconstructed missing color values are made to follow the original values. This property injects the detail information into all three channels. Figure 5 illustrates the reconstruction, where now instead of having diagonal lines showing the transition between colors, illustrating loss of shaφness and color fringing, now the circle representing red, the triangle representing green, and the square representing blue at the transition points are located in a substantially identical location such that the shaφness is retained and color fringes are avoided.

In order to display the resulting image on most commercially available CRTs and most commercially available printers, a square pixel must now be attained. Since many CCDs, and specifically that of the preferred embodiment, have rectangular pixels, generally having a 2 to 1 aspect ratio, a way of solving this problem is to average two neighboring pixels, R] and τ , for example, to form a single square pixel at that location. Fig. 6A shows pixels values from an RGB striped sensor with 2:1 aspect ratio rectangular pixels after reconstruction from original sampled color values and median filtered color difference signals. Figure 6B shows pixel values on a square grid after averaging neighboring values. This would take an image resolution that has one million pixels, for example, and create an image that is displayable to a user having 500,000 three color pixels. An alternative method of creating square pixels is optionally used in the preferred embodiment. Rather than averaging horizontally, the pixels are inteφolated vertically, as is shown in Fig. 6C.

In Fig. 6C(i), an original line of RGB data is shown, and in Fig. 6C(iii) a second line of RGB data is shown, separated by Fig. 6C(ii) which is an inteφolated line of RGB data. The inteφolation between these lines can be as simple as nearest neighbor inteφolation, which, in essence, duplicates a previous line to create a new line. The preferred methods include linear inteφolation, bi-cubic convolution inteφolation, or frequency domain inteφolation, such as Fourier, DCT, wavelet, et cetera.

The inteφolated line of Fig. 6C(ii) creates additional pixels available for display. Thus, in the one million pixel example, two million three color pixels would now be available. That is, if the image was 1600 * 600 * 3 where the three represents the three color planes, inteφolation would result in 1600 x 1200 x 3 available pixels.

In contrast to the prior art method of beginning and ending with square pixels, the method of the invention does not, per se, result in square pixels. The pixel geometry has actually not changed. The method works since output devices do not actually know the geometry of the pixels coming into the device, but instead simply displays according to an electronic signal. Therefore, rectangular pixels are displayed either separated by an inteφolated line which carries color information that makes the transition between the rectangular pixels or effectively formed into squares by averaging. This serves to eliminate the distortion that would otherwise result and simulates having a square pixels geometry. The resulting increase in pixels bodes well for output image quality but presents a difficulty for data storage. Increasing the image pixel count by two requires an increase in storage capacity for an image by two. Therefore, compression is used to decrease image storage requirements.

A characteristic that makes data highly compressible is redundancy. The actual amount of compression is determined by the compression method chosen. In the preferred embodiment, DCT compression is used which is described in commonly assigned patent application number 08/441,000 filed May 15, 1995 which is incoφorated herein by reference.

Signals that have a high degree of redundancy are the filtered difference signals as can be seen in Figure 4. The signals are compressed along with the reconstructed data of one color. The one color must be chosen such that the one color can be used to derive the remaining colors with the difference signals. In this case, green is used since both difference signals incoφorate green. Therefore, the other colors can be derived by the following equations where the superscripted plus indicates that reconstructed data is incorporated therein:

G = G⁺

R = G⁺ + (R-G)'

B = G⁺ + (B-G)'

The detail is carried in the green G⁺ signal which when the image is reconstructed is injected into the R and B signals by the above equations.

In the preferred embodiment, visually lossless compression is used though lossless compression can be used without detriment to the invention. Visually lossless is preferred since higher compression ratios can be obtained, on the order of 10:1, while by definition changes to the reconstructed image are not detected to the human eye. Furthermore, in the preferred embodiment, generation of RGB pixel values with square geometry in either of two resolutions is accomplished during decompression as described in commonly assigned patent application number 08/441,000.

The invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims

ClaimsThe embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method of processing an image stored in an electronic still camera in a first resolution having a first pixel geometry and the electronic still camera is in communication with a display device having a second pixel geometry, the method comprising: processing the image to alter the first pixel geometry to form a processed image having the second pixel geometry; and transmitting the processed image to the display device for display thereon;

2. The method according to claim 1 wherein said step of processing further comprises the steps of: separating the plurality of image signals into a first color plane, a second color plane and a third color plane; interjecting inteφolated image signals into each color plane corresponding to image signal locations left void by said separation step to form a triplet for each location in the image; creating a first difference signal and a second difference signal representing a difference between two or more image signals for each triplet; filtering the first and second difference signals to create filtered difference signals; and reconstructing the image from the plurality of image signals and said filtered difference signals to form the processed image having the second pixel geometry.

3. A method according to claim 2 wherein said step of reconstructing the image comprises generating pixel values according to the following criteria:

R@R=R; R@G=G+(R-G)'; R@B=B-(B-G)'+(R-G)' where (B-G)' and (R-G)' are the first and second difference signals, respectively.

4. The method according to claim 2 further comprising the step of reducing a file size of the processed image for storage by compressing said difference signals and one color common to the difference signals.

5. The method according to claim 1 further comprising the steps of: receiving a resolution signal from the display device; and altering the image such that the processed image has a second resolution not equal to the first resolution.

6. The method according to claim 5 wherein said step of altering the image comprises the step of averaging pixels in the image horizontally to form the processed image.

7. The method according to claim 4 wherein said step of altering the image comprises the step of inteφolating pixels in the image vertically to form the processed image.

8. The method according to claim 7 wherein said step of inteφolating pixels in the image uses linear inteφolation.

9. The method according to claim 7 wherein said step of inteφolating pixels in the image uses bi-cubic convolution inteφolation

10. The method according to claim 7 wherein said step of inteφolating pixels in the image uses frequency domain interpolation

11. The method according to claim 7 wherein said step of inteφolating pixels in the image uses nearest neighbor inteφolation.

12. An electronic still camera having a sensor with rectangular pixels which captures image-bearing light to form an image thereon, said camera being connectable to a computer having a display with square pixels, said camera further comprising processor means for converting rectangular pixels in the image to produce square pixels such that the image can be displayed on the display of the computer.

13. The electronic still camera according to claim 12 wherein said processing means is selectable to transmit the image to the computer in one of a plurality of resolutions.

14. The electronic still camera according to claim 12 wherein said processing means is selectable to transmit the image to the computer in a first resolution or a second resolution mode where the first resolution mode is less than the second resolution mode.

15. The electronic still camera according to claim 12 wherein said processing means averages neighboring pairs of the rectangular pixels in the direction normal to a major axis of the rectangular pixels to form the first resolution mode.

16. The electronic still camera according to claim 12 wherein half as many square pixel RGB triplets are formed as there were original single color rectangular pixels.

17. The electronic still camera according to claim 12 wherein said processing means interjects inteφolated pixels in the direction parallel to the major axis of the rectangular pixels to form the second resolution mode.

18. The electronic still camera according to claim 12 wherein twice as many square pixel RGB triplets are formed as there were original single color rectangular pixels.

19. The electronic still camera according to claim 12 wherein the image is compressed prior to storage.

20. A method of storing an image captured by an electronic imaging device comprising the steps of: separating the image into a first color plane, a second color plane and a third color plane; interjecting interpolated pixels into each color plane corresponding to image signal locations left void by the separation step to form a triplet for each location in the image to produce complete color planes; creating a first difference signal and a second difference signal representing a difference between two or more image signals for each triplet; compressing the first difference signal, the second difference signal and one of the complete color planes where the one of the complete color planes is chosen as being a color common to the first and second difference signals.

21. The method according to claim 20 further comprising the step of decompressing the first difference signal, the second difference signal and one of the complete color planes to form a decompressed image substantially similar to the image.

22. The method according to claim 20 wherein said step of decompressing performs reconstruction according to the following formula where G is representative of the one of the complete color planes common to the first and second difference signals:

G = G⁺

R = G⁺ + (R-G)'

B = G⁺ + (B-G)' and where the superscripted plus indicates that reconstructed data is incoφorated therein.

23. The method according to claim 22 wherein the G⁺ signal contains color detail which when reconstructed is injected into the R and B signals by the above equations.