Adaptive red eye correction

1

ADAPTIVE RED EYE CORRECTION

CROSS REFERENCE TO RELATED PATENTS
AND APPLICATIONS

5

The following copending applications, the disclosures of which are expressly incorporated herein by reference in their entireties, are mentioned:

U.S. application Ser. No. 11/145,710, filed Jun. 3, 2005, entitled "RED EYE DETECTION AND CORRECTION," 10 by Lixin Fan, et al.; and

U.S. application Ser. No. 11/311,991, filed Dec. 20, 2005, entitled "RED EYE DETECTION AND CORRECTION," by Jutta Willamowski, et al.

15

BACKGROUND

The present exemplary embodiment relates to image processing. It finds particular application in connection with the automated correction of digital images for red-eye. 20

Red-eye is a common problem in photographic images and can occur whenever a flash is used. Light reflecting from the human retina makes the eyes' pupils appear red instead of their natural color. Recognizing this problem, camera manufacturers have attempted to minimize or inhibit red-eye by 25 equipping cameras with the ability to emit one or more preflashes of light immediately prior to completion of the actual photograph. These pre-flashes are intended to constrict the subject's pupils to minimize light incident on the retina and reflected therefrom. Although cameras equipped with pre- 30 flash hardware can alleviate red-eye problems, they are not always well received since the red-eye artifact is not always prevented. They also tend to consume much more energy, induce a significant delay between pushing the button and taking the photograph, and result in people blinking the eyes. 35 Red-eye has become more prevalent and severe as cameras have been made smaller with integrated flashes. The small size coupled with the built-in nature of the flash requires placement of the flash in close proximity to the objective lens. Thus, a greater portion of the reflected light from a subject's 40 retinas enters the object lens and is recorded.

The automatic correction of the red-eye effect in digital photography generally involves two steps. First, a detection step distinguishes red-eye from non red-eye (or other) regions in an image. Second, a correction step attempts to reverse the 45 red-eye effect. Recently, the focus has been on the detection stage. However, even where the detection is performed with reasonable accuracy, it is not uncommon for the correction step to degrade the image to a point where it would have been better to leave the red-eye unaltered. 50

In one method, an operator visually scans all images and marks those images including red-eye for further processing. The processing typically involves modifying the red pixels in the identified red-eye. Efforts to eliminate or reduce operator involvement have resulted in automated processes that 55 attempt to detect red-eye based upon color, size, and shape criteria. When a red-eye is detected, the automated process applies a correction to the red area. To correct red-eyes, most approaches desaturate the pixels in the detected red-eye regions. Some also modify their luminance. In some 60 approaches, the red-eye regions are smoothed around the edges to provide a more natural looking transition between the corrected and uncorrected portions of the image.

In correcting red-eyes, the altering of chromatic information on the eye may result in a degradation of the original 65 image and the final result is not acceptable for the user. One reason for this is that the correction may remove glint. Glint

2

refers to the typically small, white specular reflections in the eye region of a photograph which give the eye a sparkle. Without the sparkle, eyes can appear dead or flat rendering the photograph unappealing. Glint results from the curvature of the cornea and typically occurs under flash light although it is also found in photographs taken in natural lighting.

Recognizing the importance of glint, some methods artificially insert a glint in the photograph if, after the standard correction, the corrected eye lacks a glint. In one approach a template of an eye is used. The template is composed of a pupil, an iris and possibly other elements, such as glint or sclera. These components are first located within the red-eye region and then corrected differently and separately.

INCORPORATION BY REFERENCE

The following references, the disclosures of which are expressly incorporated herein in their entireties by reference, are mentioned:

Methods for recognizing or addressing red-eye are disclosed, for example, in U.S. Pat. Nos. 5,432,863, 5,990,973, 6,009,209, 6,016,354, 6,278,401, 6,278,491, 6,718,051, 6,728,401, Published Application Nos. 2004/0184670, 2006/ 0098867, 2006/0120599, and the references cited therein.

U.S. Pat. No. 6,895,112 entitled RED EYE DETECTION BASED ON RED REGION DETECTION WITH EYE CONFIRMATION, by Chen, et al., discloses a system which initially identifies pixels that correspond to the color of redeye within an image. A determination is then made as to whether these identified pixels and surrounding areas are part of an eye or not part of an eye. Those identified pixels that are determined to be part of an eye are the detected red-eye regions. The system includes a filter which identifies groups of pixels, based on a set of rules, which are to be output to an eye confirmation module as detected regions. An eye confirmation module receives the detected regions from the region detection module and identifies, for each of the detected regions, whether the detected region is part of an eye.

U.S. Pat. No. 6,980,691, entitled CORRECTION OF "RED-EYE" EFFECTS IN IMAGES, by Nesterov, et al. discloses a method of automatically identifying a red-eye defect in a region of an image which includes classifying pixels within the region according to a ratio of the respective values of a first color channel and a second color channel. A red-eye defect is identified when the value of the ratio exceeds a predetermined value. Templates of colors or templates for features (such as glint, pigmentation, or artistic features) may be added to identified regions to be corrected for red-eye.

U.S. Publication No. 2004/0184670, published Sep. 23, 2004, entitled "DETECTION CORRECTION OF RED-EYE FEATURES IN DIGITAL IMAGES," by Jarman, et al, discloses a method of correcting red-eye features in a digital image which includes generating a list of possible features by scanning through each pixel in the image searching for saturation and/or lightness profiles characteristic of red-eye features . For each feature in the list, an attempt is made to find an isolated area of correctable pixels which could correspond to a red-eye feature. Each successful attempt is recorded in a list of areas. Each area is then analyzed to calculate statistics and record properties of that area, and validated using the calculated statistics and properties to determine whether or not that area is caused by red-eye. Areas not caused by red-eye and overlapping areas are removed from the list. Each area remaining is corrected to reduce the effect of red-eye. More than one type of feature may be identified in the initial search for features.

3

U.S. Published Patent Application 2004/0228542, published Nov. 18, 2004, entitled MODIFICATION OF RED EYE-EFFECT IN DIGITAL IMAGE, by Zhang, et al. discloses a process for automatic artifact compensation in a digital representation of an image. The process includes 5 detecting, by a processor, regions corresponding to facial images within the digital representation. Red-eye regions within the detected regions are located by the processor and the located red-eye regions are automatically modified.

U.S. Published Patent Application 2005/0047655, entitled 10 DETECTING AND CORRECTING REDEYE IN AN IMAGE, by Luo, et al., discloses a method of processing an input image which includes detecting red-eye pixel areas in the input image, segmenting glowing red-eye pixel areas from non-glowing red-eye pixel areas, and re-coloring regions of 15 the segmented glowing red-eye pixel areas. A given red-eye pixel area is segmented as a glowing red-eye pixel area when the relative numbers of red-eye pixels and non-red-eye pixels in an oval glint correction region inscribed in the given redeye pixel area exceeds a predetermined threshold. 20

Lei Zhang, Yanfeng Sun, Mingjing Li, Hongjiang Zhang, 'AUTOMATED RED EYE DETECTION AND CORRECTION IN DIGITAL PHOTOGRAPHS," ICIP 2004, discloses an automatic approach for detecting and correcting red eyes in digital images. In order to detect red eyes in a picture, a 25 heuristic algorithm is first adopted to detect a group of candidate red regions, and then an eye classifier is utilized to confirm whether each candidate region is a human eye. Thereafter, for each detected red eye, a correction algorithm applies a correction. In cases where a red eye cannot be detected 30 automatically, another algorithm is also provided to detect red eyes manually with the user's interaction by clicking on an eye.

BRIEF DESCRIPTION 35

Aspects of the exemplary embodiment relate to a method and apparatus for processing an image. In one aspect, an image processing method includes, for a patch of an image where a candidate red-eye has been detected, assigning the 40 patch to a first class or to one or more second classes. The assignment includes classifying the patch with a classifier with a classifier trained to assign a patch to a first class of patches associated with a standard correction of the candidate red-eye or to one or more second classes of patches associated 45 with modified corrections of the candidate red-eye. The modified correction is designed to reduce a risk of degradation of the image for the second class of patches. Where the patch is assigned to one or more second classes, the modified correction is applied to the candidate red-eye. Where the 50 patch is assigned to the first class, the standard correction is applied to the candidate red-eye.

In another aspect, an image processing apparatus includes a detection component which, for a plurality of pixels in a digital image comprising an original image or a modified 55 resolution image derived therefrom, assigns a probability that a pixel is within a red-eye, and identifies whether a patch of the image contains a candidate red-eye based on the assigned probabilities. A classifier classifies a patch of the digital image in which a candidate red-eye is identified into one of a 60 first class of images and at least one second class of images, the first class being associated with a standard correction of the candidate red-eye, the at least one second class being associated with a modified correction of the candidate redeye, the modified correction designed to reduce a risk of 65 degradation of the digital image for the second class of images and optionally assigns a level of confidence to the

4

classification. A correction component, based on the classification by the classifier, applies either the modified correction or the standard correction to the candidate red-eye.

In another aspect, an image processing method includes, for each of a plurality of pixels of a digital image, assigning a probability to the pixel of the pixel being in a red eye as a function of a color of the pixel. The method further includes identifying whether a patch of the digital image includes a candidate red-eye based on the assigned probabilities of pixels in the patch. For a patch which is identified as including a candidate red-eye, the method includes automatically classifying the patch to a first class of images associated with a standard correction of the candidate red-eye or to at least one of a plurality of second classes of images, each of the second classes being associated with a modified correction of the candidate red-eye and determining a confidence level associated with the classification. Where the image is classified into the at least one second class and where determined, the confidence level exceeds a threshold value, the modified correction is applied to the patch. Otherwise, the method includes applying the standard correction to the patch.

In another aspect, a classifier for digital images is provided which takes as input a digital image in which a patch includes a candidate red-eye, the classifier having been trained to identify images for which a modified correction of the candidate red-eye results in less degradation of the patch of the image than a standard correction of the candidate red-eye and to classify the image into either a first class for which the standard correction is to be applied or at least a second class for which the modified correction is applied.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified flow diagram illustrating one aspect of an exemplary method for correction of red-eye in digital images;

FIG. 2 is a functional block diagram of an exemplary apparatus for detection and correction of red-eyes in accordance with another aspect of the exemplary embodiment; and

FIG. 3 illustrates a patch of an image containing a red-eye which extends beyond the pupil and after correction with a standard red-eye correction method;

FIG. 4 illustrates a patch of an image containing a red-eye which is poorly defined around a border of a pupil, and after correction with the standard red-eye correction method;

FIG. 5 illustrates a patch of an image containing a red-eye which looses its glint during correction with the standard red-eye correction method;

FIG. 6 is a flow diagram illustrating a method of developing the apparatus of FIG. 2; and

FIG. 7 is a flow diagram illustrating a method of detection and correction of red-eyes in accordance with one aspect of the exemplary method.

DETAILED DESCRIPTION

Aspects of the exemplary embodiment relate to an apparatus and to a method for correction of red-eyes in photography and to a classifier suitable for use in such an apparatus and method.

The automatic correction of the red-eye effect in a digital image includes a detection stage which distinguishes red-eye from non red-eye (or other) regions in an image prior to a correction stage which is applied with the object of reducing the red-eye effect. Red-eye correction techniques often suffer from a tendency to cause image degradation. This can be due