CA1315206C - System and method for color image enhancement - Google Patents
System and method for color image enhancementInfo
- Publication number
- CA1315206C CA1315206C CA000573741A CA573741A CA1315206C CA 1315206 C CA1315206 C CA 1315206C CA 000573741 A CA000573741 A CA 000573741A CA 573741 A CA573741 A CA 573741A CA 1315206 C CA1315206 C CA 1315206C
- Authority
- CA
- Canada
- Prior art keywords
- hue
- saturation
- gray
- values
- luminance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/11—Scanning of colour motion picture films, e.g. for telecine
Abstract
Abstract A system and method for color enhancing an image or a series of images such as a motion picture by digitally capturing the images, interactively defining masks corresponding to objects in the images having similar hues, creating regions from these masks, and for each region, defining a color transfer function for converting image gray-scale information to unique values of Hue, Luminance, and Saturation. The gray-scale values within each region are then processed through that region's color transfer function, and the resulting colors applied to the image and stored for later retrieval and display.
Description
.. -- 1 --S~TE~5 ~IID ~lET~OD FO~ COLOP~ IHlGE ENH~leE~lE~lT
The present invention pertai~s to a system and method for the color enhancement o~ monochrome or multi-color images by relation of digitized gray-scale information from the original image to unique values ttriplets) of hue, luminance, and ~aturation derived from a color tran~fer function applied to the achromatic gray-scale information.
~Q~Il~ gf the ~
In strictly mathematical terms, the image ~roduced on black and white film is one dimensional Ibrightness), while an original scene can be viewed as three dimensional (using the HLS model of color, where hue, luminance, and saturation are the dimension~). The chemical reaction of the film emulsion, and the camera mechanisms and filters9 essentially reduce the three dimensional informat~on to one dimen ion by complex mapping, and forever obliterate the orlginal hue, luminance, and saturation information.
Note: It ~hould be pointed ou~that luminance and brightne3s are DQ~ the same. Since brightne3~ of the black and ~hlte image re~ult~ from the co~plex interactions of the three element3 of the HLS ~y~tem, there i~ only a ~tatl~tical correlation between brightne~3 and luminance, but they are not ldentical.
.
Color Representation Systems Color is both an absol~te quality of light (its wavelength or frequency) and a relative quality which depends on the human perceiver. For this reason, various color r'gamuts" have been proposed which allow perceived colors to be quantified. Among these models is the CIE chromaticity scale9 which permits quantification of color by combinations of so-called primaries (which are, by themselves, not visible to human observers). (See Foley & VanDam, Fundament~ls n~
Interactive Compl~ter ~L~hin~, Addison Wesley, Reading, MA 1984 pp. 606-608.) In addition to theoretical models of color (CIE Chromaticity, Hue-Luminance-Saturation, and Red-Green-Blue), there are many representational systems for color reproduction in various media such as the RGB system for cathode ray tubes, the process color system for printing, and the NTSC, PAL, and S~CAM systems for broadcast video. Each of these models is, in fact, a relatively small subset of both the theoretical absolute range of colors and of the actual range of humanly perceivable colors.
One of the theoretical models (Ostwald, W.,ColQ~ ~i~n~_, Winsor & Winson, London, 1931) which is especially useful for analysis of color is the HLS
model of hue, luminance, and saturation. This model uses a three dimensional cylindrical universe (the absolute theoretical color space), inside of which is a space bounded by a pair of cones having a common base (the perception space). The cylinder's axis (from apex to apex of the cones) is the luminance axis, with the lower apex having a luminance of zero (all black) and the apex of the upper cone having a luminance of 1 (all white). Distance from the luminance axis (normal to the axis) determines saturation (proportion of a color which is made up of its complementary color), with the axis representing zero saturation (equal amounts of the primary color and its complement and therefore no color) and the surface of the cone representing maximum perceived saturation for a given luminance. Finally, angular displacement around the luminance a~is represents hue, as with a conventional color wheel, with complements separated by 180 degrees.
The solid space defined by the HLS conic model is pointed at its top and bottom to represent the fact that as colors become whiter or blacker, they are perceived to lose hue. Saturation is a characteristic which depends chiefly on the narrowness or purity of the band of wavelengths comprising a color. At the exact center of the space is a medium gray which ' ' ' ' ., ~ ' ' . ~
' !
exhibits a moderate emittance or reflectance for all wavelengths of light.
The RGB system, i~ used, for example, in CRT
displays, where each of the three electron guns (red, green, and blue) is controlled directly by a signal indicating its respective intensity. HLS color may be converted to the RGB representation (see Foley &
VanDam, pp. 648-9.), however there are many HLS
combinations which must be "clipped" to be represented in RGB. The same is true of NTSC video (also known as YIQ), which has an even more restricted range than RGB.
Thus, not every "real world" color can be reproduced by video systems using either RGB or NTSC standards.
According to Foley and VanDam, nneither the V in HSV
[Hue, Saturation, Value] nor the L in HLS correspond to luminance in the YIQ model, so two different colors defined in either space can easily have the same luminance, and be indistinguishable on black and white TV or videotape." (Foley and VanDam p. 618) Since approximately 1970, several techniques ~lave been developed for the addition of color to monochrome images. Often, these techniques rely on video technology to augment the video signal in a defined manner, typically by adding a chrominance component signal to the pre-existing luminance signal.
This technique results in an appearance not unlike that achieved by Thomas Edison in his early efforts to add tint to black and white film by painting the frames: a colored area over a black and white picture.
Because of their reliance on video technology, all of these sytems suffer from lack of vividness of colors, and from an inability to accurately reproduce colors.
Brief ~escription Q~ ~h~ Invention The present invention comprises a system and method for color enhancing an image or a series of images such as a motion picture by digitally capturing the images, interactively defining areas or regions corresponding to objects or regions in the images having similar hues, and for each region, defining a color transfer function for converting image gray-scale information for each picture element to unique values of Hue, Luminance, and Saturation. The gray-scale values within each region are then processed through that region's color transfer function, and the resulting colors created in an image memory and stored for later retrieval and display.
1 ~1 520~
~rief ~s~i~lgn nf the Figure~
Figure 1 depicts in schematic form the creation of two adjacent frames of graphic information according to the method of the present invention.
Figure 2 is a schematic representation of a portion of the image memory structure according to one embodiment of the method of the present invention.
Figure 3 depicts the replication of a the region memory portion of the image memory structure of the present irvention.
Figure 4 depicts the correction of a value stored in the replicated region memory portion of Fig.
3.
Figure 5 depicts the application of the HLS
color model to the coloring of an array of picture elements stored in the image memory structure of Figs.
The present invention pertai~s to a system and method for the color enhancement o~ monochrome or multi-color images by relation of digitized gray-scale information from the original image to unique values ttriplets) of hue, luminance, and ~aturation derived from a color tran~fer function applied to the achromatic gray-scale information.
~Q~Il~ gf the ~
In strictly mathematical terms, the image ~roduced on black and white film is one dimensional Ibrightness), while an original scene can be viewed as three dimensional (using the HLS model of color, where hue, luminance, and saturation are the dimension~). The chemical reaction of the film emulsion, and the camera mechanisms and filters9 essentially reduce the three dimensional informat~on to one dimen ion by complex mapping, and forever obliterate the orlginal hue, luminance, and saturation information.
Note: It ~hould be pointed ou~that luminance and brightne3s are DQ~ the same. Since brightne3~ of the black and ~hlte image re~ult~ from the co~plex interactions of the three element3 of the HLS ~y~tem, there i~ only a ~tatl~tical correlation between brightne~3 and luminance, but they are not ldentical.
.
Color Representation Systems Color is both an absol~te quality of light (its wavelength or frequency) and a relative quality which depends on the human perceiver. For this reason, various color r'gamuts" have been proposed which allow perceived colors to be quantified. Among these models is the CIE chromaticity scale9 which permits quantification of color by combinations of so-called primaries (which are, by themselves, not visible to human observers). (See Foley & VanDam, Fundament~ls n~
Interactive Compl~ter ~L~hin~, Addison Wesley, Reading, MA 1984 pp. 606-608.) In addition to theoretical models of color (CIE Chromaticity, Hue-Luminance-Saturation, and Red-Green-Blue), there are many representational systems for color reproduction in various media such as the RGB system for cathode ray tubes, the process color system for printing, and the NTSC, PAL, and S~CAM systems for broadcast video. Each of these models is, in fact, a relatively small subset of both the theoretical absolute range of colors and of the actual range of humanly perceivable colors.
One of the theoretical models (Ostwald, W.,ColQ~ ~i~n~_, Winsor & Winson, London, 1931) which is especially useful for analysis of color is the HLS
model of hue, luminance, and saturation. This model uses a three dimensional cylindrical universe (the absolute theoretical color space), inside of which is a space bounded by a pair of cones having a common base (the perception space). The cylinder's axis (from apex to apex of the cones) is the luminance axis, with the lower apex having a luminance of zero (all black) and the apex of the upper cone having a luminance of 1 (all white). Distance from the luminance axis (normal to the axis) determines saturation (proportion of a color which is made up of its complementary color), with the axis representing zero saturation (equal amounts of the primary color and its complement and therefore no color) and the surface of the cone representing maximum perceived saturation for a given luminance. Finally, angular displacement around the luminance a~is represents hue, as with a conventional color wheel, with complements separated by 180 degrees.
The solid space defined by the HLS conic model is pointed at its top and bottom to represent the fact that as colors become whiter or blacker, they are perceived to lose hue. Saturation is a characteristic which depends chiefly on the narrowness or purity of the band of wavelengths comprising a color. At the exact center of the space is a medium gray which ' ' ' ' ., ~ ' ' . ~
' !
exhibits a moderate emittance or reflectance for all wavelengths of light.
The RGB system, i~ used, for example, in CRT
displays, where each of the three electron guns (red, green, and blue) is controlled directly by a signal indicating its respective intensity. HLS color may be converted to the RGB representation (see Foley &
VanDam, pp. 648-9.), however there are many HLS
combinations which must be "clipped" to be represented in RGB. The same is true of NTSC video (also known as YIQ), which has an even more restricted range than RGB.
Thus, not every "real world" color can be reproduced by video systems using either RGB or NTSC standards.
According to Foley and VanDam, nneither the V in HSV
[Hue, Saturation, Value] nor the L in HLS correspond to luminance in the YIQ model, so two different colors defined in either space can easily have the same luminance, and be indistinguishable on black and white TV or videotape." (Foley and VanDam p. 618) Since approximately 1970, several techniques ~lave been developed for the addition of color to monochrome images. Often, these techniques rely on video technology to augment the video signal in a defined manner, typically by adding a chrominance component signal to the pre-existing luminance signal.
This technique results in an appearance not unlike that achieved by Thomas Edison in his early efforts to add tint to black and white film by painting the frames: a colored area over a black and white picture.
Because of their reliance on video technology, all of these sytems suffer from lack of vividness of colors, and from an inability to accurately reproduce colors.
Brief ~escription Q~ ~h~ Invention The present invention comprises a system and method for color enhancing an image or a series of images such as a motion picture by digitally capturing the images, interactively defining areas or regions corresponding to objects or regions in the images having similar hues, and for each region, defining a color transfer function for converting image gray-scale information for each picture element to unique values of Hue, Luminance, and Saturation. The gray-scale values within each region are then processed through that region's color transfer function, and the resulting colors created in an image memory and stored for later retrieval and display.
1 ~1 520~
~rief ~s~i~lgn nf the Figure~
Figure 1 depicts in schematic form the creation of two adjacent frames of graphic information according to the method of the present invention.
Figure 2 is a schematic representation of a portion of the image memory structure according to one embodiment of the method of the present invention.
Figure 3 depicts the replication of a the region memory portion of the image memory structure of the present irvention.
Figure 4 depicts the correction of a value stored in the replicated region memory portion of Fig.
3.
Figure 5 depicts the application of the HLS
color model to the coloring of an array of picture elements stored in the image memory structure of Figs.
2-4.
Figure 6 depicts a schematic representation of the "histogram slide" operation according to the method of the present invention.
Figure 7 depicts a schematic representation of the nhistogram multiplication downt' operation according to the method of the preaent invention.
Figure 8 depicts a schematic representation of the "histogram multiplication up" operation according to the method of the present invention.
Fi~ure 9 depicts a schematic represention of the "color bending" operation according to the method of the present invention.
Detailed DesQription of ~he In~ention General Process Overview The film to be colored or color enhanced is first digitally recorded at a preselected resolution.
In instances where the final product will be placed on conventional video tape, appropriate resolutions are employed. (E.g. NTSC=525 lines, PAL-640, HDTV=1125-1500) Where the final product will be used for theatrical presentation; resolutlons of 1024 lines or more may be employed. The digiti~ation process - : . '' : .
;: , . ,, :
, . ' ., ' : :. : ' ~. - . ; .
1 31 520h (depicted in Fig. 1) comprises projecting each frame of the original (typically black and white) film through a liquid gate device which masks scratches and other imperfections. The projected frame is captured using a video camera such as a saticon tube and the video signal is digitized and stored in a random access memory frame buffer. The digitization process allows adjustment of the intensity of light falling on the film in order to achieve the best contrast for the particular frame being digitized (gamma correction).
After acquisition of a frame, the digital representation of the frame is stored in non- volatile storage for further processing. The digitization process is repeated on a frame-by-frame basis, marking each frame with a time and sequence indication (SMPTE
time code).
For each scene (sequence of frames), a frame is selected as being representative of the majority of the elements present in the sequence. This "key frame" is used as a template for the enhancement of other frames within the scene sequence. For this reason, the greatest care is applied to the processing of key frames.
After key frames are color-enhanced and defined, they are passed with the rest of the frames comprising a scene to a work station where the regions defining the application of color are tracked from frame to frame. After each frame has been enhanced, it is re-stored in non-volatile storage and sent to a central storage were all enhanced scenes are eventually reassembled according to their original sequence as a final product motion picture.
Key Frame Coloring The first task of the key frame colorist is to identify areas of the image which comprise objects which will be designated to be of similar hue. Several software tools are available to the colorist to aid in this computer-interactive definition process.
A nfree hand" mode allows drawing and erasing as ~ell as control over a variably-sized cursor. A
pointing device controls an interactive cursor displayed on a screen overlaying the key frame to be processed. Movement of the pointing device effects an analogous movement of the cursor and the picture elements selected by this movement (which are displayed on a graphics plane known as the "mask plane") are u~ed to define the edge of an area. Because the cursor size may be reduced to only one pixel, the smallest region which may be produced is also one pixel.
A second drawing tool available to the colorist is one which defines polygons, rather than completely free-hand shapes. Using the pointing devices, the colorist may indicate the precis~
placement of vertices which are intended to define the perimeter of a polygonal area. Each vertex may be assigned or removed and, when complete, the vertices may be completed to define a closed path.
After definition of a closed perimeter by free-hand or polygon methods, the enclosed area may be assigned to the region (by setting the bits in the mask plane) by using one of two filling algorithms. An "area fill" assigns all picture elements within the closed shape and up to the defined ed~e to the area. A "cartoon fill" combines the defined edges with edge recognition algorithms in order to fill an area which is ~ounded by a border line such as the black pen lines which are typical of cartoons. Cartoon fill mode fills up to, or slightly over such heavy border lines.
After areas have been defined for all objects having a common hue, the final step is the selection and application of a given hue to that area~ Colors may be selected by interactively pointing to a color wheel displayed on a video screen, and then by manipulating control indicators on the video screen which allow refinement of the gray-scale information for the area as described below. After selection, the color may be displayed by its application to the areas (which are collectively Xnown as a color mask.) After a color mask has been defined, its associated picture elements are identified as a "region", the mask plane is cleared, and additional masks are defined in a similar manner, one for each hue which will be displayed in the frame.
Once all regions are defined, the regions, the underlying frame, and the colors assigned to each region are stored for later retrieval by colorists who will reiterate the process using the subsequent frames of each scene and the key frame which has oeen defined.
An additional tool for defining regions is the threshold region definition tool. The threshold region definition tool permits the automatic defini-tion of regions to be colored. First, a brush size (a form of cursor) is selected. Next, an upper or lower threshold gray-scale value is selected. ~he upper/lower gray-scale value is representative of :
' ' ' ' ' , .
.
.
- 6a -the edge of a contiguous group of picture elements having common brightness. An upper gray-scale value is selected if the threshold region definition tool is to be used to select picture elements below a given upper threshold value. Alternatively, a lower threshold gray-scale Yalue is selected to designate a region above said lower threshold value. The picture elements located within the brush tip are analyzed to determine the medium brightness of those picture elements. If the medium brightness within the brush tip is within the designated threshold region (either below the upper threshold region or above the lower threshold region), the picture elements are included in the designated region. The picture elements above an upper threshold region or below a lower threshold region are excluded from the new region.
The process of extrapolating color enhancement from key frames to entire scenes is performed by identifying and tracking motion as well as appearances and disappearances of objects.
Generally fewer than 5% of the picture elements in a frame of a scene change in the succeeding frame.
Because motion ~ I
-- , "
pictures are filmed at 24 frames per second, increment~
of motion on all but the most rapidly moving objects are extremely small and easily followed. Once motion has been identified, adjustment of the regions and application of the key frames~ predefined colors to those regions is all that is required.
Implicit in all frame-to-frame tracking is the ability to copy predefined parts of the preceding or key frame's region memory contents and to use those parts in subsequent frame processing. After copying, small adjustments may be made to the new region memory contents and they may be saved with the new frame.
Importantly, because the digitization process as~ures consistency of brightness and gray scale, and because colors are predefined by an art director working with key frames, overall consistency from frame-to-frame, and scene-to-scene is also assured.
The simplest form of tracking from frame to frame is where motion in the frame is apparent only as a result of a camera pan. In such instances, the entire region memory may be copied from that of the preceding frame and "shiftedn in the direction of the pan. Such a shift will create an undefined area at the end opposite the motion. This area must then be added to the existing regions in order to completely apply color.
When only some of the objects of a frame move, a corresponding portion of the region memory may be translated as well. In this operation, that portion of the region memory which is to be moved is designated by the operator and is moved manually to its new loca-tion, thus overwriting the former contents of the region memory at the new location. The old location must then be re-defined and/or added to one or more pre~existing regions in order to apply color.
In highly complex scenes, where several types of motion are present, (especially compound motion of subject and camera) a ~differencing~ procedure is em-ployed. A frame for which areas have been defined and to which color has been applied is subtracted from a next subsequent frame in order to reveal those picture elements which have changed (in gray-scale value).
These areas of difference are then displayed as a contrasting overlay on the subsequent frame to be pro-cessed. Alternatively, an algorithm for edge detection in gray-scale images may be used and may be correlated to the previous frame's region memory. Detected dif-ferences may then cause the subsequent fram~'s regions ~ ' , ~_ -8-to be adjusted or deleted, as required. By examining the coincidence of this overlay and the region struc-ture which has been duplicated from the preceding frame, the production colorist is able to quickly designate areas which should be added to the region structure, and those which must be removed from a given region and reassigned.
The final operation is to assure that no objects have either completely appeared or disappeared from the frame. If an object disappears, its region must be removed. Similarly, newly appearing obJects must have a region defined for them "o" the fly" and a color assigned as well. After adjusting the region for the next subsequent image9 the colorist again stores that image together with its region memory and repeats the process for each subsequent frame.
To permit quality control and review of a sequence of color-enhanced frames, two modes of viewing are used: Cine mode and Cycle mode. In cine mode, a series of frames is sequentially displayed, shuttling first in a forward direction, and then immediately in reverse. The speed may be controlled by the operator.
In cycle mode, display direction and speed are directly tied to the pointing device to allow fine control over viewing frame-to-frame transitions.
Hard~are OYerYie~
The method of the present invention is implemented on a special purpose version of an RTI-Station image processing system in~erfaced to a personal computer (such as the IB~* personal computer AT). The RTI image processing subsystem (available from Recognition Technology, Inc. of Holliston, MA) comprises a video digitizer, a pipeline architecture pixel processor, and image memory which is all interconnected both by a host computer bus and by a sychronous video bus. The video digitizer takes its input from a v$deo camera such as that described above for digitizin`g film frames, and outputs its informatlon to a high definition color CRT-(though not necessarily one interfaced by conventional NTSC video circuitry).
The host computer bu~ permits the image processing sub~y~tem to communicate with the user through the host CPU, its memory, as~ociated mass storage devices, and I~O devices including pointing devices and keyboards.
The image processing subsy~tem of the preferred embodiment of the present invention compri~e~
an image memory having 512 x 512 spatial resolution, 8 * Trade Mark --. ,, g bit of brightness reYolution, and 4 bits for region identification. Alternatively, the same system may be configured for 6 bits of brightness resolution and 6 bits of region identification or for 7 bits of brightness resolution and 5 bits of region identification. (One additional bit is reserved for use by the system in construction of the mask bit plane.) General information regarding the structure and operation of the RTI Station and its associated software library (RTILIB) may be found in the RTILIB~500 Tutorial Revision 1.00 and the RTILIB~500 User Manual Revision A, which are published by Recognition Technology, Inc.
Although the currently used implementation employs a 512 line spatial resolution, it will be recognized by those skilled in the art that increases to and above 2000 line spatial resolution are within the scope of the present invention. Additional~y, the image memory may be a part of a larger frame store memory which may be, for instance, 4096 x 4096 x 16 bits. The display of frames may then be controlled by a memory pointer into this frame store.
For the purpose of color selection, the host CPU is provided with a 24 bit RGB video board (such as the Targa*~4, a product of AT&T Information Systems, Inc.) A color wheel which corresponds to the common base of the HLS cone is displayed on a ~ideo screen~
Using a pointing device, the colorist selects a particular color on the wheel, thus defining both hue (angular position on the color wheel) and saturation (distance from the center). As described more fully below, this unique 24 bit RGB value is translated immediately into the HLS coordinate space, correlated to the modified image gray-scale, and retranslated for output on an RGB monitor.
Softuare Overview Referring now to Figs. 1-4, according to the method of the present invention, a series of digitized monochrome images may be stored in a random access memory of a computer as an array of gray-scale values.
For example~ 8 bits of information may be used to represent each picture element, thus permitting a total of 256 unique gray-scale values to be represented. (Of course, more bits may be used to achieve better resolution and dynamic range for the gray-scale information.) The digitization ~tep may optionally include the u~e of look-up tables in order to enhance contrast, adjust gamma, enhance edge definition or * Trade Mark 13~5~06 AFT-011 -10~
"stretch" the gray-scale of a faded image. Associated with the memory used to store the gray-scale values is a region memory used to uniquely identify each picture element with one of a number of regions (depicted as a four bit memory structure in Figs. 2-5). In addition to the image memory and region memory structures, a mask plane memory is also associated with the image for use in identifying picture elements. Through conventional techniques, the contents or parts of the contents of these memory structures may be copied from one structure to another (for example, from the region memory to the mask plane memory) or from one frame's memory to another (as depicted in Fig. 3 for the transfer of frame 1 region memory to frame 2 region memory.) The copied contents may, of course, be altered by the operator to conform to the underlying frame contents as depicted in Fig. 4.
As depicted in Figs. 5-9, the colorist has at his or her disposal a number o~ software-implemented tools which control the construction of the color transfer function. For computational efficiency, the luminance axis of HLS cylindrical space is represented as an integer range of [0..1000] (as opposed to the conventional representation as a real number in the range [0..1.0]). The gray-scale of a frame is linearly related to the integer range by defaulting a gray-scale value of 128 to a luminance value of 500, and corresponding each discrete gray-scale unit to each integer of luminance. (Thus, 255 gray-scale corresponds by default to a luminance of 627.) In order to affect the color transfer function constructed by the system, the colorist actually manipulates the frame gray-scale information by applying histogramming techniques such as translation, compression, and stretching.
Specifically, the entire gray-scale may be made to appear either more or less luminous by a simple "slide" of the gray-scale (GS) with respect to the luminance scale (Fig. 6). Instead of corresponding GS
= 128 to L = 500, the operator may instead choose to briBhten the frame by choosing L = 750. Similarly, the frame may be darkened by choosing L = 350. As depicted in Fig. 6, the portion of the color transfer function falling within the HLS conic space which corresponds to the gray-scale is controlled by this slide operation.
At its extremes, the operation creates a threshhold in the transfer function beyond which all gray-scale values are displayed as either white (above the upper threshhold), or as black (below the lower one).
Listing 1 void slide_map(); t~slide brightness mapping ~/
void stretchLmap~ stretch or compress brightness mapping */
void bend_on(); /~enable color bending */
void bend_off~); /*disable color bending */
void slide_bend(); /*slide region of color bending */
void stretch_bend(); /#stretch region of color bending */
/~LOCAL OONSTANTS ~ ---*/
#define TRUE 1 $define FALSE 0 /~Local St~tic Data-- --------*/
static int brt_l = 0; /*brightness value for start of ramp static int brt_u = 1000; /~brightness value for end of ramp ~/
static int bend = FALSE; /%logical - TRUE if color bend enabled ~/
static int markl; /~lower color bending overlap marker ff/
static int mark2; /~upper color bending overlap marker */
/~Global Data (and externals)~
extern int hue_1; /*primary hue value - [0.. 360] ~/
extern int sat_1; /*primary saturation value - [0.. 1000] *~
extern int hue_2; /*secondary hue value - [0.. 360] */
extern int sat_2; /*secondary saturation value - ~0.. 1000] */
/*secondaries valid for bend = TRUE only */
void slide_n~p (dbrt) int dbrt; /*change in brightness - [-99g.. 999] */
/*make sure that the requested change is in range ~/
dbrt = max ( dbrt, -9g9);
dbrt = min ( dbrt, 999);
/*check for limiting case ~/
if ( ( dbrt < 0) && (brt_l < -dbrt) ) {
stretchLmap ( -brt_1, 1000-brt_u);
return;
}
/*Simple case: apply the change */
brt_l += dbrt;
brt_u ~= dbrt;
if ( bend) {
markl ~= dbrt;
mark2 ~= dbrt;
) End Slide map ~/
Another operation available to the colorist is a ratiometric multiplication of the gray-scale by a value (depicted in Figs. 7-8). When the multiplier value is a fraction less than one, a ratiometric compression of the gray-scale occurs which relates the 256 discrete gray-~cale values to fewer than 256 integer luminance values. When the multiplier value is greater than unity, a "stretch" of the gray-scale occurs which results in a correspondence of the 256 gray-scal e values to a range of luminance greater than 256. These ~ultiplications may also occur using either end of the gray-scal e as a fixed point, yiel ding 4 variations as depicted in Figs. 7-8.
void stretch_map( db_lcw, db_up) int db_low; /* change in lcwer ramp point ~
int d~_up; ~# change in upper ramp point ~/
int newbrt_1, r~wbrt_u 7 newmark_1, newmark_u;
double fmark1, fmark2, rlen, newrlen;
/~ Check that change parameters are within range ~/
db_low = min( db_low, 999~;
db_low = max( db_low, -999);
db_up = min( db_up, 999);
db_up = max( db_up, -999);
/* Apply change to lower ramp point*/
newbrt_1 = brt_1 + db_low;
newbrt_1 = max( newbrt_1, 0);
newbrt_1 = min( newbrt_1, 999);
/~ Apply change to upper ramp point ~/
newbrt_u = brt_u ~ db_up;
newbrt_u = max( newbrt_u, 1);
newbrt_u = min( newbrt_u, 1000);
/* Make sure that the new ccmbination is valid */
if( (db_up == 0) && (newbrt_1 >- newbrt_u) ) newbrt_1 = newbrt_u - 1;
else if( newbrt_u <= newbrt_1) newbr~_u = newbrt_1 + 1;
/* Reset color-bend markers for new ramp */
if( bend){
rlen = ~double)( brt_u - brt_1); /* current ramplen ~/
newrlen - (double)( newbrt_u - newbrt_1); /*new ramplen */
fmarkl - (double)( mark1 - brt_1) / rlen;
fmark2 - (double)( mark2 - brt_1) / rlen;
markl = (int)( fmark1 * newrlen) + newbrt_1;
mark2 - tint)( fmark2 ~ newrlen) + newbrt_1;
}
Move new ramp points into official ramp points */
brt_1 = newbrt_1;
brt_u - newbrt_u;
}/~End of stretch-map ~/
In depicting reflected light (either originating from a colored source, reflecting from a colored surfac.e, or both), it is neces~ary to "bend"
the col or transfer function. As shown in Fig. ~, the col orist has available the option of setting a pair of "bending points" for the function which determine the nature and extent of the bending. These points, depicted in Fig. 9, (as pointq A and B) define the endpoints of three line segments (OB, BA, and A1). Of these segments, the first and l ast each hav e associated with them unique, operator-selected values of hue, whil e the AB segment represents a mixture of hues and saturations. The locations OI points A and B are defaulted to 25% and 75~ of the current gra~scale range, thus providing color mixing over one-half of the gray-scale. These locations, however, may be changed by the operator to change the '?rate" of col or bending.
I.isting ~
void ben~on( ) int midpoint, ramplen;
if( bend) return; t* already done, do nothing */
ramplen = (brt_u - brt_1 + 1); /~ lengt~ of current ramp ~/
midpoint = brt_1 + ranplen /2; /* midpoint of current ramp ~/
markl = midpoint- ranplent4; /* bottcm of color overlap */
mark2 - midpoint + ranplen/4; /~ top of color overlap ~/
} /* End of ben~on ~/
void bend_off( ) bend = FALSE;
} /~End of bend-off ~/
void slide_bend( dbend) int dbend: /* change in region of bend - -999 to 999~/
{
int ovlplen;
/* If bend not enabled, do nothing ~/
if( lbend ) return;
/* Make sure that there is room to go in the requested direction ~/
if( ( dbend < O) && (mark1 == brt_1 ) ) return;
`if( (dbend > O) && (mark2 -= brt_u) ) return;
/~ Make sure that the requested change is within range ~/
dbend = tdbend < O) ? max ( dbend, brt_1 - mark1):
min( dbend, brt_u - mark2);
Apply the change *t mark1 += dbend; /~ new markl position ~/
mark2 += dbend; /* new mark2 position ~/
} /~t End of slide_bend ~/
void stretc~bend( ~fact) int sfact; /~ stretch factor (-100 to 100)5~/
double fact, ~narklen, markfact, marklen, midpoint, ramplen;
/* Canpute a floating point stretch factor ~/
fact = (sfact ~- 0~ ? 1.0 + (double)sfact / 100.0;
1 31 5~06 1.0 - (double) (-sfact)~100.0;
Get ~oating point ramp, overlap length and midpoint of bend ~/
ramplen = (double) ( brt_u - brt_1 + 1);
mar~en = (double) ( mark2 - markl + 1);
midpoint = (double)mark1 + mar~en/2.0;
/~ Compute and adjust ~e percentage overlap */
markfact = mar~en / ramplen;
markfact ~= fact: /* apply ch~ge */
hmarklen = (ramplen * markfact) / 2.0; /* half n~ mark leng~ */
/* Compute new mark points */
mark1 = max( (int)( midpoint - hmar~en), brt_1);
mark2 = min( (int)( midpoint + hmar~en), brt_u);
} /~ End of stretchLbend ~/
As an example of bending, consider the selection of a OB hue as flesh tone, and the A1 hue as orange (as would be the case if a face were viewed by reflected fire light). The BA segment would contain hues which are flesh tones in the less bright areas?
but for brighter ones, take on more and more orange appearance. Of course, it will be appreciated that the selection of the locations of points A and B or the inclusion of additional bending points are fully within the spirit of the technique.
Because regions are defined so as to include all objects of a similar hue, regions contain a wide range of picture element gray-scale values. In order to accurately represent the gray-scale values in the final colored image, a color transfer function must be defined for converting the one--dimensional gray-scale to a set of three dimensional HLS values. From the point on the displayed color wheel chosen by a key frame colorist, a line segment is extended vertically parallel to the axis of the cylindrical HLS universe.
This line segment passes through the points in the HLS
cones which represent the colors to be applied within the region, and comprises the color transfer function.
For example, an 8 bit picture element having a gray-scale value of 190 is represented on the gray-scale of Figure 5. The corresponding point on the upper segment of the color transfer function defines a unique HLS
value which will be applied to those picture elements within the region having the 190 gray-scale value.
Where the line segment does not fall within the HLS
conic space, the representation is either pure white (in the region above L = 0.5) or pure black (in the region below L = 0.5). It will be recognized that because the color transfer function traverses a portion of the altitude of HLS coordinate space, points which fall at either extreme of the gray-scale are correspondingly less saturated when displayed in color.
1 31 520h This renders the d~splayed colors much more realistic than systems using constant saturations.
Specular highlights are those areas of particularly bright reflection due to angle of incidence effects. The production of colorless specular highlights can be produced by applying the histogram stretch technique described above to "pull"
brighter gray-scale values outside of the ~ILS cone, thus making them appear white. The extent of this stretch controls the appearance of the specular highlight.
Referring again to Figure 9, there is depicted a schematic representation of the method by which both specular highlights and reflection of colored light sources may be achieved. Although human perception is, in fact, best represented by the HLS
double cone, the actual physics of color and light are best represented by an HLS cylinder which has luminance as its axis, and saturation as its radius. Within this cylinder is inscribed the conic perceptual space. This con~c space defines the limits of human perception, and the precise shape and size of the cones varies from person to person. (A colorblind person, for example, has a cone with a "flattened" side, while one with night-blindness has an extremely truncated lower cone.) By manipulating gray-scale information representing an image, the color transfer function by which gray~scale is transformed into hue, luminance, and saturation is affected, and a variety of effects are produced.
Glossary of Terms Image - monochrome or color analog picture Picture Elements - discrete locations in an image Brightness - the absolute amount of light transmitted through a monochrome image (dependent on incident light intensity) Capturing an Image Digitally - recording digital data representative of an image and from which an image can be re-created.
Gray-Scale Value - relati~e amount of light transmitted through a point on a monochrome image ~dependent on incident light intensity) 1315~06 Luminance - the absolute amount of light emitted by7 or transmitted through a point on a color image Hue - the peak or primary wavelength emitted by or transmitted through a point on a color image Saturation - the relative number of primary to complementary points in a given area of a color image (more primary = higher saturation) Region - a data structure in memory used to identify a group of picture elements to which a defined color transfer function is to be applied.
Mask - a data structure (commonly a graphics plane) in memory used to identify a group of picture elements to be acted upon, such as by their inclusion within a region.
Map - a data structure (commonly a multi-bit graphics plane) for identifying one or more regions of an image, each such region comprising picture elements representative of one or more objects or image areas having the common property.
Specular ~lighlight - region of high brightness (monochrome) or high luminance (color) brought about by direct (angle of incidence) reflection of a light source.
Statement Q~ Industrial ~llitY
The method and apparatus of the present invention is useful in the color enhancement of achromatic images such as still photographs, motion pictures, video taped images, telemetry data, and the like.
Figure 6 depicts a schematic representation of the "histogram slide" operation according to the method of the present invention.
Figure 7 depicts a schematic representation of the nhistogram multiplication downt' operation according to the method of the preaent invention.
Figure 8 depicts a schematic representation of the "histogram multiplication up" operation according to the method of the present invention.
Fi~ure 9 depicts a schematic represention of the "color bending" operation according to the method of the present invention.
Detailed DesQription of ~he In~ention General Process Overview The film to be colored or color enhanced is first digitally recorded at a preselected resolution.
In instances where the final product will be placed on conventional video tape, appropriate resolutions are employed. (E.g. NTSC=525 lines, PAL-640, HDTV=1125-1500) Where the final product will be used for theatrical presentation; resolutlons of 1024 lines or more may be employed. The digiti~ation process - : . '' : .
;: , . ,, :
, . ' ., ' : :. : ' ~. - . ; .
1 31 520h (depicted in Fig. 1) comprises projecting each frame of the original (typically black and white) film through a liquid gate device which masks scratches and other imperfections. The projected frame is captured using a video camera such as a saticon tube and the video signal is digitized and stored in a random access memory frame buffer. The digitization process allows adjustment of the intensity of light falling on the film in order to achieve the best contrast for the particular frame being digitized (gamma correction).
After acquisition of a frame, the digital representation of the frame is stored in non- volatile storage for further processing. The digitization process is repeated on a frame-by-frame basis, marking each frame with a time and sequence indication (SMPTE
time code).
For each scene (sequence of frames), a frame is selected as being representative of the majority of the elements present in the sequence. This "key frame" is used as a template for the enhancement of other frames within the scene sequence. For this reason, the greatest care is applied to the processing of key frames.
After key frames are color-enhanced and defined, they are passed with the rest of the frames comprising a scene to a work station where the regions defining the application of color are tracked from frame to frame. After each frame has been enhanced, it is re-stored in non-volatile storage and sent to a central storage were all enhanced scenes are eventually reassembled according to their original sequence as a final product motion picture.
Key Frame Coloring The first task of the key frame colorist is to identify areas of the image which comprise objects which will be designated to be of similar hue. Several software tools are available to the colorist to aid in this computer-interactive definition process.
A nfree hand" mode allows drawing and erasing as ~ell as control over a variably-sized cursor. A
pointing device controls an interactive cursor displayed on a screen overlaying the key frame to be processed. Movement of the pointing device effects an analogous movement of the cursor and the picture elements selected by this movement (which are displayed on a graphics plane known as the "mask plane") are u~ed to define the edge of an area. Because the cursor size may be reduced to only one pixel, the smallest region which may be produced is also one pixel.
A second drawing tool available to the colorist is one which defines polygons, rather than completely free-hand shapes. Using the pointing devices, the colorist may indicate the precis~
placement of vertices which are intended to define the perimeter of a polygonal area. Each vertex may be assigned or removed and, when complete, the vertices may be completed to define a closed path.
After definition of a closed perimeter by free-hand or polygon methods, the enclosed area may be assigned to the region (by setting the bits in the mask plane) by using one of two filling algorithms. An "area fill" assigns all picture elements within the closed shape and up to the defined ed~e to the area. A "cartoon fill" combines the defined edges with edge recognition algorithms in order to fill an area which is ~ounded by a border line such as the black pen lines which are typical of cartoons. Cartoon fill mode fills up to, or slightly over such heavy border lines.
After areas have been defined for all objects having a common hue, the final step is the selection and application of a given hue to that area~ Colors may be selected by interactively pointing to a color wheel displayed on a video screen, and then by manipulating control indicators on the video screen which allow refinement of the gray-scale information for the area as described below. After selection, the color may be displayed by its application to the areas (which are collectively Xnown as a color mask.) After a color mask has been defined, its associated picture elements are identified as a "region", the mask plane is cleared, and additional masks are defined in a similar manner, one for each hue which will be displayed in the frame.
Once all regions are defined, the regions, the underlying frame, and the colors assigned to each region are stored for later retrieval by colorists who will reiterate the process using the subsequent frames of each scene and the key frame which has oeen defined.
An additional tool for defining regions is the threshold region definition tool. The threshold region definition tool permits the automatic defini-tion of regions to be colored. First, a brush size (a form of cursor) is selected. Next, an upper or lower threshold gray-scale value is selected. ~he upper/lower gray-scale value is representative of :
' ' ' ' ' , .
.
.
- 6a -the edge of a contiguous group of picture elements having common brightness. An upper gray-scale value is selected if the threshold region definition tool is to be used to select picture elements below a given upper threshold value. Alternatively, a lower threshold gray-scale Yalue is selected to designate a region above said lower threshold value. The picture elements located within the brush tip are analyzed to determine the medium brightness of those picture elements. If the medium brightness within the brush tip is within the designated threshold region (either below the upper threshold region or above the lower threshold region), the picture elements are included in the designated region. The picture elements above an upper threshold region or below a lower threshold region are excluded from the new region.
The process of extrapolating color enhancement from key frames to entire scenes is performed by identifying and tracking motion as well as appearances and disappearances of objects.
Generally fewer than 5% of the picture elements in a frame of a scene change in the succeeding frame.
Because motion ~ I
-- , "
pictures are filmed at 24 frames per second, increment~
of motion on all but the most rapidly moving objects are extremely small and easily followed. Once motion has been identified, adjustment of the regions and application of the key frames~ predefined colors to those regions is all that is required.
Implicit in all frame-to-frame tracking is the ability to copy predefined parts of the preceding or key frame's region memory contents and to use those parts in subsequent frame processing. After copying, small adjustments may be made to the new region memory contents and they may be saved with the new frame.
Importantly, because the digitization process as~ures consistency of brightness and gray scale, and because colors are predefined by an art director working with key frames, overall consistency from frame-to-frame, and scene-to-scene is also assured.
The simplest form of tracking from frame to frame is where motion in the frame is apparent only as a result of a camera pan. In such instances, the entire region memory may be copied from that of the preceding frame and "shiftedn in the direction of the pan. Such a shift will create an undefined area at the end opposite the motion. This area must then be added to the existing regions in order to completely apply color.
When only some of the objects of a frame move, a corresponding portion of the region memory may be translated as well. In this operation, that portion of the region memory which is to be moved is designated by the operator and is moved manually to its new loca-tion, thus overwriting the former contents of the region memory at the new location. The old location must then be re-defined and/or added to one or more pre~existing regions in order to apply color.
In highly complex scenes, where several types of motion are present, (especially compound motion of subject and camera) a ~differencing~ procedure is em-ployed. A frame for which areas have been defined and to which color has been applied is subtracted from a next subsequent frame in order to reveal those picture elements which have changed (in gray-scale value).
These areas of difference are then displayed as a contrasting overlay on the subsequent frame to be pro-cessed. Alternatively, an algorithm for edge detection in gray-scale images may be used and may be correlated to the previous frame's region memory. Detected dif-ferences may then cause the subsequent fram~'s regions ~ ' , ~_ -8-to be adjusted or deleted, as required. By examining the coincidence of this overlay and the region struc-ture which has been duplicated from the preceding frame, the production colorist is able to quickly designate areas which should be added to the region structure, and those which must be removed from a given region and reassigned.
The final operation is to assure that no objects have either completely appeared or disappeared from the frame. If an object disappears, its region must be removed. Similarly, newly appearing obJects must have a region defined for them "o" the fly" and a color assigned as well. After adjusting the region for the next subsequent image9 the colorist again stores that image together with its region memory and repeats the process for each subsequent frame.
To permit quality control and review of a sequence of color-enhanced frames, two modes of viewing are used: Cine mode and Cycle mode. In cine mode, a series of frames is sequentially displayed, shuttling first in a forward direction, and then immediately in reverse. The speed may be controlled by the operator.
In cycle mode, display direction and speed are directly tied to the pointing device to allow fine control over viewing frame-to-frame transitions.
Hard~are OYerYie~
The method of the present invention is implemented on a special purpose version of an RTI-Station image processing system in~erfaced to a personal computer (such as the IB~* personal computer AT). The RTI image processing subsystem (available from Recognition Technology, Inc. of Holliston, MA) comprises a video digitizer, a pipeline architecture pixel processor, and image memory which is all interconnected both by a host computer bus and by a sychronous video bus. The video digitizer takes its input from a v$deo camera such as that described above for digitizin`g film frames, and outputs its informatlon to a high definition color CRT-(though not necessarily one interfaced by conventional NTSC video circuitry).
The host computer bu~ permits the image processing sub~y~tem to communicate with the user through the host CPU, its memory, as~ociated mass storage devices, and I~O devices including pointing devices and keyboards.
The image processing subsy~tem of the preferred embodiment of the present invention compri~e~
an image memory having 512 x 512 spatial resolution, 8 * Trade Mark --. ,, g bit of brightness reYolution, and 4 bits for region identification. Alternatively, the same system may be configured for 6 bits of brightness resolution and 6 bits of region identification or for 7 bits of brightness resolution and 5 bits of region identification. (One additional bit is reserved for use by the system in construction of the mask bit plane.) General information regarding the structure and operation of the RTI Station and its associated software library (RTILIB) may be found in the RTILIB~500 Tutorial Revision 1.00 and the RTILIB~500 User Manual Revision A, which are published by Recognition Technology, Inc.
Although the currently used implementation employs a 512 line spatial resolution, it will be recognized by those skilled in the art that increases to and above 2000 line spatial resolution are within the scope of the present invention. Additional~y, the image memory may be a part of a larger frame store memory which may be, for instance, 4096 x 4096 x 16 bits. The display of frames may then be controlled by a memory pointer into this frame store.
For the purpose of color selection, the host CPU is provided with a 24 bit RGB video board (such as the Targa*~4, a product of AT&T Information Systems, Inc.) A color wheel which corresponds to the common base of the HLS cone is displayed on a ~ideo screen~
Using a pointing device, the colorist selects a particular color on the wheel, thus defining both hue (angular position on the color wheel) and saturation (distance from the center). As described more fully below, this unique 24 bit RGB value is translated immediately into the HLS coordinate space, correlated to the modified image gray-scale, and retranslated for output on an RGB monitor.
Softuare Overview Referring now to Figs. 1-4, according to the method of the present invention, a series of digitized monochrome images may be stored in a random access memory of a computer as an array of gray-scale values.
For example~ 8 bits of information may be used to represent each picture element, thus permitting a total of 256 unique gray-scale values to be represented. (Of course, more bits may be used to achieve better resolution and dynamic range for the gray-scale information.) The digitization ~tep may optionally include the u~e of look-up tables in order to enhance contrast, adjust gamma, enhance edge definition or * Trade Mark 13~5~06 AFT-011 -10~
"stretch" the gray-scale of a faded image. Associated with the memory used to store the gray-scale values is a region memory used to uniquely identify each picture element with one of a number of regions (depicted as a four bit memory structure in Figs. 2-5). In addition to the image memory and region memory structures, a mask plane memory is also associated with the image for use in identifying picture elements. Through conventional techniques, the contents or parts of the contents of these memory structures may be copied from one structure to another (for example, from the region memory to the mask plane memory) or from one frame's memory to another (as depicted in Fig. 3 for the transfer of frame 1 region memory to frame 2 region memory.) The copied contents may, of course, be altered by the operator to conform to the underlying frame contents as depicted in Fig. 4.
As depicted in Figs. 5-9, the colorist has at his or her disposal a number o~ software-implemented tools which control the construction of the color transfer function. For computational efficiency, the luminance axis of HLS cylindrical space is represented as an integer range of [0..1000] (as opposed to the conventional representation as a real number in the range [0..1.0]). The gray-scale of a frame is linearly related to the integer range by defaulting a gray-scale value of 128 to a luminance value of 500, and corresponding each discrete gray-scale unit to each integer of luminance. (Thus, 255 gray-scale corresponds by default to a luminance of 627.) In order to affect the color transfer function constructed by the system, the colorist actually manipulates the frame gray-scale information by applying histogramming techniques such as translation, compression, and stretching.
Specifically, the entire gray-scale may be made to appear either more or less luminous by a simple "slide" of the gray-scale (GS) with respect to the luminance scale (Fig. 6). Instead of corresponding GS
= 128 to L = 500, the operator may instead choose to briBhten the frame by choosing L = 750. Similarly, the frame may be darkened by choosing L = 350. As depicted in Fig. 6, the portion of the color transfer function falling within the HLS conic space which corresponds to the gray-scale is controlled by this slide operation.
At its extremes, the operation creates a threshhold in the transfer function beyond which all gray-scale values are displayed as either white (above the upper threshhold), or as black (below the lower one).
Listing 1 void slide_map(); t~slide brightness mapping ~/
void stretchLmap~ stretch or compress brightness mapping */
void bend_on(); /~enable color bending */
void bend_off~); /*disable color bending */
void slide_bend(); /*slide region of color bending */
void stretch_bend(); /#stretch region of color bending */
/~LOCAL OONSTANTS ~ ---*/
#define TRUE 1 $define FALSE 0 /~Local St~tic Data-- --------*/
static int brt_l = 0; /*brightness value for start of ramp static int brt_u = 1000; /~brightness value for end of ramp ~/
static int bend = FALSE; /%logical - TRUE if color bend enabled ~/
static int markl; /~lower color bending overlap marker ff/
static int mark2; /~upper color bending overlap marker */
/~Global Data (and externals)~
extern int hue_1; /*primary hue value - [0.. 360] ~/
extern int sat_1; /*primary saturation value - [0.. 1000] *~
extern int hue_2; /*secondary hue value - [0.. 360] */
extern int sat_2; /*secondary saturation value - ~0.. 1000] */
/*secondaries valid for bend = TRUE only */
void slide_n~p (dbrt) int dbrt; /*change in brightness - [-99g.. 999] */
/*make sure that the requested change is in range ~/
dbrt = max ( dbrt, -9g9);
dbrt = min ( dbrt, 999);
/*check for limiting case ~/
if ( ( dbrt < 0) && (brt_l < -dbrt) ) {
stretchLmap ( -brt_1, 1000-brt_u);
return;
}
/*Simple case: apply the change */
brt_l += dbrt;
brt_u ~= dbrt;
if ( bend) {
markl ~= dbrt;
mark2 ~= dbrt;
) End Slide map ~/
Another operation available to the colorist is a ratiometric multiplication of the gray-scale by a value (depicted in Figs. 7-8). When the multiplier value is a fraction less than one, a ratiometric compression of the gray-scale occurs which relates the 256 discrete gray-~cale values to fewer than 256 integer luminance values. When the multiplier value is greater than unity, a "stretch" of the gray-scale occurs which results in a correspondence of the 256 gray-scal e values to a range of luminance greater than 256. These ~ultiplications may also occur using either end of the gray-scal e as a fixed point, yiel ding 4 variations as depicted in Figs. 7-8.
void stretch_map( db_lcw, db_up) int db_low; /* change in lcwer ramp point ~
int d~_up; ~# change in upper ramp point ~/
int newbrt_1, r~wbrt_u 7 newmark_1, newmark_u;
double fmark1, fmark2, rlen, newrlen;
/~ Check that change parameters are within range ~/
db_low = min( db_low, 999~;
db_low = max( db_low, -999);
db_up = min( db_up, 999);
db_up = max( db_up, -999);
/* Apply change to lower ramp point*/
newbrt_1 = brt_1 + db_low;
newbrt_1 = max( newbrt_1, 0);
newbrt_1 = min( newbrt_1, 999);
/~ Apply change to upper ramp point ~/
newbrt_u = brt_u ~ db_up;
newbrt_u = max( newbrt_u, 1);
newbrt_u = min( newbrt_u, 1000);
/* Make sure that the new ccmbination is valid */
if( (db_up == 0) && (newbrt_1 >- newbrt_u) ) newbrt_1 = newbrt_u - 1;
else if( newbrt_u <= newbrt_1) newbr~_u = newbrt_1 + 1;
/* Reset color-bend markers for new ramp */
if( bend){
rlen = ~double)( brt_u - brt_1); /* current ramplen ~/
newrlen - (double)( newbrt_u - newbrt_1); /*new ramplen */
fmarkl - (double)( mark1 - brt_1) / rlen;
fmark2 - (double)( mark2 - brt_1) / rlen;
markl = (int)( fmark1 * newrlen) + newbrt_1;
mark2 - tint)( fmark2 ~ newrlen) + newbrt_1;
}
Move new ramp points into official ramp points */
brt_1 = newbrt_1;
brt_u - newbrt_u;
}/~End of stretch-map ~/
In depicting reflected light (either originating from a colored source, reflecting from a colored surfac.e, or both), it is neces~ary to "bend"
the col or transfer function. As shown in Fig. ~, the col orist has available the option of setting a pair of "bending points" for the function which determine the nature and extent of the bending. These points, depicted in Fig. 9, (as pointq A and B) define the endpoints of three line segments (OB, BA, and A1). Of these segments, the first and l ast each hav e associated with them unique, operator-selected values of hue, whil e the AB segment represents a mixture of hues and saturations. The locations OI points A and B are defaulted to 25% and 75~ of the current gra~scale range, thus providing color mixing over one-half of the gray-scale. These locations, however, may be changed by the operator to change the '?rate" of col or bending.
I.isting ~
void ben~on( ) int midpoint, ramplen;
if( bend) return; t* already done, do nothing */
ramplen = (brt_u - brt_1 + 1); /~ lengt~ of current ramp ~/
midpoint = brt_1 + ranplen /2; /* midpoint of current ramp ~/
markl = midpoint- ranplent4; /* bottcm of color overlap */
mark2 - midpoint + ranplen/4; /~ top of color overlap ~/
} /* End of ben~on ~/
void bend_off( ) bend = FALSE;
} /~End of bend-off ~/
void slide_bend( dbend) int dbend: /* change in region of bend - -999 to 999~/
{
int ovlplen;
/* If bend not enabled, do nothing ~/
if( lbend ) return;
/* Make sure that there is room to go in the requested direction ~/
if( ( dbend < O) && (mark1 == brt_1 ) ) return;
`if( (dbend > O) && (mark2 -= brt_u) ) return;
/~ Make sure that the requested change is within range ~/
dbend = tdbend < O) ? max ( dbend, brt_1 - mark1):
min( dbend, brt_u - mark2);
Apply the change *t mark1 += dbend; /~ new markl position ~/
mark2 += dbend; /* new mark2 position ~/
} /~t End of slide_bend ~/
void stretc~bend( ~fact) int sfact; /~ stretch factor (-100 to 100)5~/
double fact, ~narklen, markfact, marklen, midpoint, ramplen;
/* Canpute a floating point stretch factor ~/
fact = (sfact ~- 0~ ? 1.0 + (double)sfact / 100.0;
1 31 5~06 1.0 - (double) (-sfact)~100.0;
Get ~oating point ramp, overlap length and midpoint of bend ~/
ramplen = (double) ( brt_u - brt_1 + 1);
mar~en = (double) ( mark2 - markl + 1);
midpoint = (double)mark1 + mar~en/2.0;
/~ Compute and adjust ~e percentage overlap */
markfact = mar~en / ramplen;
markfact ~= fact: /* apply ch~ge */
hmarklen = (ramplen * markfact) / 2.0; /* half n~ mark leng~ */
/* Compute new mark points */
mark1 = max( (int)( midpoint - hmar~en), brt_1);
mark2 = min( (int)( midpoint + hmar~en), brt_u);
} /~ End of stretchLbend ~/
As an example of bending, consider the selection of a OB hue as flesh tone, and the A1 hue as orange (as would be the case if a face were viewed by reflected fire light). The BA segment would contain hues which are flesh tones in the less bright areas?
but for brighter ones, take on more and more orange appearance. Of course, it will be appreciated that the selection of the locations of points A and B or the inclusion of additional bending points are fully within the spirit of the technique.
Because regions are defined so as to include all objects of a similar hue, regions contain a wide range of picture element gray-scale values. In order to accurately represent the gray-scale values in the final colored image, a color transfer function must be defined for converting the one--dimensional gray-scale to a set of three dimensional HLS values. From the point on the displayed color wheel chosen by a key frame colorist, a line segment is extended vertically parallel to the axis of the cylindrical HLS universe.
This line segment passes through the points in the HLS
cones which represent the colors to be applied within the region, and comprises the color transfer function.
For example, an 8 bit picture element having a gray-scale value of 190 is represented on the gray-scale of Figure 5. The corresponding point on the upper segment of the color transfer function defines a unique HLS
value which will be applied to those picture elements within the region having the 190 gray-scale value.
Where the line segment does not fall within the HLS
conic space, the representation is either pure white (in the region above L = 0.5) or pure black (in the region below L = 0.5). It will be recognized that because the color transfer function traverses a portion of the altitude of HLS coordinate space, points which fall at either extreme of the gray-scale are correspondingly less saturated when displayed in color.
1 31 520h This renders the d~splayed colors much more realistic than systems using constant saturations.
Specular highlights are those areas of particularly bright reflection due to angle of incidence effects. The production of colorless specular highlights can be produced by applying the histogram stretch technique described above to "pull"
brighter gray-scale values outside of the ~ILS cone, thus making them appear white. The extent of this stretch controls the appearance of the specular highlight.
Referring again to Figure 9, there is depicted a schematic representation of the method by which both specular highlights and reflection of colored light sources may be achieved. Although human perception is, in fact, best represented by the HLS
double cone, the actual physics of color and light are best represented by an HLS cylinder which has luminance as its axis, and saturation as its radius. Within this cylinder is inscribed the conic perceptual space. This con~c space defines the limits of human perception, and the precise shape and size of the cones varies from person to person. (A colorblind person, for example, has a cone with a "flattened" side, while one with night-blindness has an extremely truncated lower cone.) By manipulating gray-scale information representing an image, the color transfer function by which gray~scale is transformed into hue, luminance, and saturation is affected, and a variety of effects are produced.
Glossary of Terms Image - monochrome or color analog picture Picture Elements - discrete locations in an image Brightness - the absolute amount of light transmitted through a monochrome image (dependent on incident light intensity) Capturing an Image Digitally - recording digital data representative of an image and from which an image can be re-created.
Gray-Scale Value - relati~e amount of light transmitted through a point on a monochrome image ~dependent on incident light intensity) 1315~06 Luminance - the absolute amount of light emitted by7 or transmitted through a point on a color image Hue - the peak or primary wavelength emitted by or transmitted through a point on a color image Saturation - the relative number of primary to complementary points in a given area of a color image (more primary = higher saturation) Region - a data structure in memory used to identify a group of picture elements to which a defined color transfer function is to be applied.
Mask - a data structure (commonly a graphics plane) in memory used to identify a group of picture elements to be acted upon, such as by their inclusion within a region.
Map - a data structure (commonly a multi-bit graphics plane) for identifying one or more regions of an image, each such region comprising picture elements representative of one or more objects or image areas having the common property.
Specular ~lighlight - region of high brightness (monochrome) or high luminance (color) brought about by direct (angle of incidence) reflection of a light source.
Statement Q~ Industrial ~llitY
The method and apparatus of the present invention is useful in the color enhancement of achromatic images such as still photographs, motion pictures, video taped images, telemetry data, and the like.
Claims (19)
1. A method of color enhancing an image comprising the steps of:
a. capturing the image digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. defining one or more masks, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator-designated hue;
c. defining a color transfer function for each of said masks identifying each of said picture element gray-scale values within each of said masks with a unique combination of hue, luminance, and saturation values;
d. applying said color transfer function for each of said masks to said corresponding picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray-scale value; and e. assigning said derived hue, saturation, and luminance values to said picture elements corresponding to each of said masks.
a. capturing the image digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. defining one or more masks, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator-designated hue;
c. defining a color transfer function for each of said masks identifying each of said picture element gray-scale values within each of said masks with a unique combination of hue, luminance, and saturation values;
d. applying said color transfer function for each of said masks to said corresponding picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray-scale value; and e. assigning said derived hue, saturation, and luminance values to said picture elements corresponding to each of said masks.
2. The method of Claim 1, step (c) further comprising the steps of:
a. selecting an arbitrary operator-defined hue value intended for application to the picture elements corresponding to each of said masks;
b. selecting a range of luminance values representative of the brightness of said picture elements corresponding to each of said masks;
c. selecting a range of saturation values representative of the saturations of the picture elements corresponding to each of said masks;
d. constructing a predetermined color transfer function relating said range of luminance values to said range of saturation values.
a. selecting an arbitrary operator-defined hue value intended for application to the picture elements corresponding to each of said masks;
b. selecting a range of luminance values representative of the brightness of said picture elements corresponding to each of said masks;
c. selecting a range of saturation values representative of the saturations of the picture elements corresponding to each of said masks;
d. constructing a predetermined color transfer function relating said range of luminance values to said range of saturation values.
3. The method of claim 2 further comprising:
for at least one operator-defined mask, for which an operator defined arbitrarily selected hue value has been designated in accordance with step (a) of claim 2, selecting at least one arbitrary second hue value representative of an operator-designated hue of an illumination source; and wherein step (d) of claim 2 further comprises constructing a predetermined hue-luminance-saturation transfer function relating said ranges of saturation and luminance values to said first and second hue values.
for at least one operator-defined mask, for which an operator defined arbitrarily selected hue value has been designated in accordance with step (a) of claim 2, selecting at least one arbitrary second hue value representative of an operator-designated hue of an illumination source; and wherein step (d) of claim 2 further comprises constructing a predetermined hue-luminance-saturation transfer function relating said ranges of saturation and luminance values to said first and second hue values.
4. The method of Claim 1, step (b) wherein at least one of said regions is defined by the steps of:
a. selecting an upper or lower threshold gray-scale value representative of the edge of a contiguous group of picture elements having common brightness;
b. analyzing the picture elements in a predefined area (brush tip) of said captured image to determine their median brightness; and c. if the median brightness of the picture elements within said predefined area is above said lower threshold or below said upper threshold, then designating said picture elements as being included in said region.
a. selecting an upper or lower threshold gray-scale value representative of the edge of a contiguous group of picture elements having common brightness;
b. analyzing the picture elements in a predefined area (brush tip) of said captured image to determine their median brightness; and c. if the median brightness of the picture elements within said predefined area is above said lower threshold or below said upper threshold, then designating said picture elements as being included in said region.
5. The method of Claim 1 wherein at least one of said regions is defined by manually designating the bounds of one or more areas using a pointing device to define a mask.
6. A method of color enhancing a series of image frames comprising the steps of:
a. capturing and storing each image frame digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. designating one of said captured frames as a key frame;
c. using said key frame, defining one or more maps, each comprising one or more closed regions of picture elements representative of one or more objects designated to have the same hue;
d. defining for at least one of said maps a color transfer function identifying said picture element gray-scale values with unique combinations of hue, luminance, and saturation values;
e. correlating said picture element gray-scale values to said color transfer function, thereby deriving unique hue, saturation, and luminance values corresponding to each gray-scale value, and f. assigning said hue, saturation, and luminance values to said picture elements comprising said map;
g. storing said map in association with said key frame;
h. sequentially recalling said stored image frames, and for each frame:
i. comparing said recalled frame with the preceding frame, to determine differences in the locations of said objects;
j. establishing a modified map by modifying a copy of each of said maps by selective deletion of picture elements which no longer define said objects, and addition of picture elements which define said objects, in accordance with said determined differences in the locations of said objects; and k. storing said modified maps in association with said recalled frames.
a. capturing and storing each image frame digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. designating one of said captured frames as a key frame;
c. using said key frame, defining one or more maps, each comprising one or more closed regions of picture elements representative of one or more objects designated to have the same hue;
d. defining for at least one of said maps a color transfer function identifying said picture element gray-scale values with unique combinations of hue, luminance, and saturation values;
e. correlating said picture element gray-scale values to said color transfer function, thereby deriving unique hue, saturation, and luminance values corresponding to each gray-scale value, and f. assigning said hue, saturation, and luminance values to said picture elements comprising said map;
g. storing said map in association with said key frame;
h. sequentially recalling said stored image frames, and for each frame:
i. comparing said recalled frame with the preceding frame, to determine differences in the locations of said objects;
j. establishing a modified map by modifying a copy of each of said maps by selective deletion of picture elements which no longer define said objects, and addition of picture elements which define said objects, in accordance with said determined differences in the locations of said objects; and k. storing said modified maps in association with said recalled frames.
7. A system for color enhancing an image comprising:
a. means for capturing the image digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. means for defining one or more masks, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator designated hue;
c. means for defining a color transfer function for each of said masks identifying said picture element gray-scale values within each of said masks with unique combinations of hue, luminance, and saturation values;
d. means for applying said color transfer function for each of said masks to said corresponding picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray scale value; and e. means for assigning said derived hue, saturation, and luminance values to said picture elements corresponding to each of said masks.
a. means for capturing the image digitally, as a series of picture elements, each having a gray-scale value representative of its brightness;
b. means for defining one or more masks, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator designated hue;
c. means for defining a color transfer function for each of said masks identifying said picture element gray-scale values within each of said masks with unique combinations of hue, luminance, and saturation values;
d. means for applying said color transfer function for each of said masks to said corresponding picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray scale value; and e. means for assigning said derived hue, saturation, and luminance values to said picture elements corresponding to each of said masks.
8. A method of color enhancing a source image comprising the steps of:
a. recording said source image digitally, as a first series of picture elements, each having a gray-scale value representative of its brightness;
b. defining a first series of one or more masks within said source image, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator-designated hue;
c. defining a color transfer function for each of said masks identifying said picture element gray-scale values with unique combinations of hue, luminance, and saturation values;
d. applying said color transfer function to each of said picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray-scale value; and e. assigning said derived hue, saturation, and luminance values to a second series of picture elements in a second series of masks, each such mask corresponding in shape, size and position to a mask in said first series of masks, and storing said assigned values for said picture elements of said second series of masks as a recording of an output image.
a. recording said source image digitally, as a first series of picture elements, each having a gray-scale value representative of its brightness;
b. defining a first series of one or more masks within said source image, each comprising one or more closed regions of picture elements representative of one or more operator-perceived objects within the image having the same operator-designated hue;
c. defining a color transfer function for each of said masks identifying said picture element gray-scale values with unique combinations of hue, luminance, and saturation values;
d. applying said color transfer function to each of said picture element gray-scale values, thereby deriving unique combinations of hue, saturation, and luminance values corresponding to each gray-scale value; and e. assigning said derived hue, saturation, and luminance values to a second series of picture elements in a second series of masks, each such mask corresponding in shape, size and position to a mask in said first series of masks, and storing said assigned values for said picture elements of said second series of masks as a recording of an output image.
9. The method of claim 1, step (d) further comprising the steps of:
a) determining whether said unique saturation value which corresponds to said picture element gray-scale value is beyond a maximum saturation value limit which approximates the maximum, visually perceivable saturation value for said unique luminance value which corresponds to said picture element gray-scale value; and b) assigning said maximum saturation value as a new saturation value for said picture element if said unique saturation value is beyond the maximum saturation value for said unique luminance values which corresponds to said picture element gray-scale value.
a) determining whether said unique saturation value which corresponds to said picture element gray-scale value is beyond a maximum saturation value limit which approximates the maximum, visually perceivable saturation value for said unique luminance value which corresponds to said picture element gray-scale value; and b) assigning said maximum saturation value as a new saturation value for said picture element if said unique saturation value is beyond the maximum saturation value for said unique luminance values which corresponds to said picture element gray-scale value.
10. The method of claim 9, wherein said maximum saturation values for all unique luminance values defines a boundary for a double conic space in a hue, luminance, saturation (HLS) coordinate system.
11. The method of claim 10, wherein said double conic space is symmetrical about a central point located midway between a maximum unique luminance value and a minimum unique luminance value.
12. The method of claim 10, wherein said double conic space is asymmetrical about said central point.
13. The method of claim 3 wherein said predetermined hue-luminance-saturation transfer function is adapted to cause the hue values of said operator defined mask to be displayed in brighter portions thereof as if said operator defined mask hue value was the hue value of said operator designated hue value of an illumination source.
14. A system for coloring a monochrome image having a plurality of regions comprising:
a. means for digitally storing said image as a multiplicity of pixels, each represented by a specific gray scale value;
b. means for storing a product of a separate color transfer function for each of said regions; and c. means for digitally storing, for each pixel, a region identifier by which one of said color transfer functions is associated with said pixel.
a. means for digitally storing said image as a multiplicity of pixels, each represented by a specific gray scale value;
b. means for storing a product of a separate color transfer function for each of said regions; and c. means for digitally storing, for each pixel, a region identifier by which one of said color transfer functions is associated with said pixel.
15. The system of claim 14 wherein said color transfer function is constructed of operator manipulated relationships of hue, luminance, saturation and gray-scale values.
16. The system of claim 14 wherein (a) and (c) are part of a multi-bit memory structure comprising a first memory structure and a second memory structure.
17. The system of claim 16 wherein for each pixel of said image, said first memory structure contains an 8 bit gray-scale value and said second memory structure contains a 4 bit region identifier value.
18. The system of claim 16 further comprising an additional memory structure adapted to contain pixel identification information designating which pixels are identified with a particular region.
19. The system of claim 14 wherein a. said color transfer functions are constructed of operator manipulated relationships of hue, luminance, saturation and gray-scale values;
b. (a) and (c) of claim 17, are part of a multi-bit memory structure comprising, for each pixel, a first memory structure containing an 8 bit of gray-scale value and a second memory structure containing a 4 bit region identifier value; and c. an additional one-bit memory structure adapted to contain, for each pixel, pixel identification information designating which pixels are identified with a particular region.
b. (a) and (c) of claim 17, are part of a multi-bit memory structure comprising, for each pixel, a first memory structure containing an 8 bit of gray-scale value and a second memory structure containing a 4 bit region identifier value; and c. an additional one-bit memory structure adapted to contain, for each pixel, pixel identification information designating which pixels are identified with a particular region.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA000616362A CA1324321C (en) | 1987-08-03 | 1992-04-27 | System and method for color image enhancement |
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US8111687A | 1987-08-03 | 1987-08-03 | |
| US081,116 | 1987-08-03 | ||
| US222,543 | 1988-07-25 | ||
| US07/222,543 US4984072A (en) | 1987-08-03 | 1988-07-25 | System and method for color image enhancement |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000616362A Division CA1324321C (en) | 1987-08-03 | 1992-04-27 | System and method for color image enhancement |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1315206C true CA1315206C (en) | 1993-03-30 |
Family
ID=26765215
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000573741A Expired - Lifetime CA1315206C (en) | 1987-08-03 | 1988-08-03 | System and method for color image enhancement |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US4984072A (en) |
| AU (1) | AU611466B2 (en) |
| CA (1) | CA1315206C (en) |
Families Citing this family (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8396328B2 (en) | 2001-05-04 | 2013-03-12 | Legend3D, Inc. | Minimal artifact image sequence depth enhancement system and method |
| US7907793B1 (en) | 2001-05-04 | 2011-03-15 | Legend Films Inc. | Image sequence depth enhancement system and method |
| US5155589A (en) * | 1990-05-22 | 1992-10-13 | Gere David S | Storage and retrieval of images from a grey image having artificially enhanced color regions |
| JPH0514654A (en) * | 1991-07-04 | 1993-01-22 | Kowa Co | Picture signal processor |
| CA2093448C (en) * | 1992-07-17 | 1999-03-09 | Albert D. Edgar | Expert system for image enhancement |
| US5534915A (en) * | 1992-09-30 | 1996-07-09 | American Film Technologies, Inc. | Method of color enhancing a monochrome image using multiple base colors for selected regions of the monochrome image |
| IL103763A (en) * | 1992-11-16 | 1999-03-12 | Technion Res & Dev Foundation | Apparatus and method for enhancing color images |
| DE69420874T2 (en) * | 1993-02-12 | 2000-05-04 | Eastman Kodak Co | Procedure for cross-device color image calibration and improvement |
| GB9619117D0 (en) * | 1996-09-12 | 1996-10-23 | Pandora Int Ltd | Digital image processing |
| US6271926B1 (en) * | 1996-10-10 | 2001-08-07 | Xerox Corporation | Printing system with print job programming capability |
| US6400830B1 (en) * | 1998-02-06 | 2002-06-04 | Compaq Computer Corporation | Technique for tracking objects through a series of images |
| US6775406B1 (en) * | 1998-08-25 | 2004-08-10 | Douglas L. Watson | Colorizing a black-and-white image to facilitate the identification of a pattern in the image |
| US20030091229A1 (en) * | 2000-03-31 | 2003-05-15 | Imation Corp. | Color image display accuracy using comparison of complex shapes to reference background |
| US7119760B2 (en) * | 2000-03-31 | 2006-10-10 | Kodak Graphic Communications Canada Company | Color image display accuracy using green-limited gamma estimate |
| US20040227769A9 (en) * | 2000-03-31 | 2004-11-18 | Imation Corp. | Color image display accuracy using comparison of colored objects to dithered background |
| KR100413341B1 (en) * | 2001-03-20 | 2003-12-31 | 주식회사 코난테크놀로지 | Method for abstracting histogram of HSV Color |
| US8401336B2 (en) | 2001-05-04 | 2013-03-19 | Legend3D, Inc. | System and method for rapid image sequence depth enhancement with augmented computer-generated elements |
| US9286941B2 (en) | 2001-05-04 | 2016-03-15 | Legend3D, Inc. | Image sequence enhancement and motion picture project management system |
| US7181081B2 (en) * | 2001-05-04 | 2007-02-20 | Legend Films Inc. | Image sequence enhancement system and method |
| US9031383B2 (en) | 2001-05-04 | 2015-05-12 | Legend3D, Inc. | Motion picture project management system |
| US8897596B1 (en) | 2001-05-04 | 2014-11-25 | Legend3D, Inc. | System and method for rapid image sequence depth enhancement with translucent elements |
| US6897426B2 (en) * | 2001-08-09 | 2005-05-24 | Ralph Roque Etienne-Cummings | Self-contained integrated color processing imager |
| JP3816882B2 (en) * | 2003-03-05 | 2006-08-30 | 株式会社東芝 | Display font memory |
| US20040114053A1 (en) * | 2003-08-29 | 2004-06-17 | Legend Films Inc. | System and method for color enhancing images |
| US7372507B2 (en) | 2005-02-08 | 2008-05-13 | Texas Instruments Incorporated | Real-time content based gamma adjustment for digital video display |
| TWI338514B (en) * | 2006-01-20 | 2011-03-01 | Au Optronics Corp | Image processing method for enhancing contrast |
| JP5003196B2 (en) * | 2007-02-19 | 2012-08-15 | ソニー株式会社 | Image processing apparatus and method, and program |
| WO2009029671A1 (en) * | 2007-08-27 | 2009-03-05 | University Of Chicago | Systems and methods for image colorization |
| WO2011163480A2 (en) * | 2010-06-24 | 2011-12-29 | Tk Holdings Inc. | Vehicle display enhancements |
| US20120105584A1 (en) | 2010-10-28 | 2012-05-03 | Gallagher Andrew C | Camera with sensors having different color patterns |
| US20120188409A1 (en) | 2011-01-24 | 2012-07-26 | Andrew Charles Gallagher | Camera with multiple color sensors |
| US8730232B2 (en) | 2011-02-01 | 2014-05-20 | Legend3D, Inc. | Director-style based 2D to 3D movie conversion system and method |
| US9288476B2 (en) | 2011-02-17 | 2016-03-15 | Legend3D, Inc. | System and method for real-time depth modification of stereo images of a virtual reality environment |
| US9282321B2 (en) | 2011-02-17 | 2016-03-08 | Legend3D, Inc. | 3D model multi-reviewer system |
| US9113130B2 (en) | 2012-02-06 | 2015-08-18 | Legend3D, Inc. | Multi-stage production pipeline system |
| US9407904B2 (en) | 2013-05-01 | 2016-08-02 | Legend3D, Inc. | Method for creating 3D virtual reality from 2D images |
| US9241147B2 (en) | 2013-05-01 | 2016-01-19 | Legend3D, Inc. | External depth map transformation method for conversion of two-dimensional images to stereoscopic images |
| US9007365B2 (en) | 2012-11-27 | 2015-04-14 | Legend3D, Inc. | Line depth augmentation system and method for conversion of 2D images to 3D images |
| US9547937B2 (en) | 2012-11-30 | 2017-01-17 | Legend3D, Inc. | Three-dimensional annotation system and method |
| US9007404B2 (en) | 2013-03-15 | 2015-04-14 | Legend3D, Inc. | Tilt-based look around effect image enhancement method |
| US9438878B2 (en) | 2013-05-01 | 2016-09-06 | Legend3D, Inc. | Method of converting 2D video to 3D video using 3D object models |
| US9609307B1 (en) | 2015-09-17 | 2017-03-28 | Legend3D, Inc. | Method of converting 2D video to 3D video using machine learning |
| CN110463197B (en) | 2017-03-26 | 2021-05-28 | 苹果公司 | Enhancing spatial resolution in stereoscopic camera imaging systems |
| CN111724297B (en) * | 2019-03-21 | 2023-07-25 | 杭州海康威视数字技术股份有限公司 | Image processing method and device |
Family Cites Families (48)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US2593925A (en) * | 1948-10-05 | 1952-04-22 | Sheldon Edward Emanuel | Device for color projection of invisible rays |
| US2804500A (en) * | 1953-10-01 | 1957-08-27 | Rca Corp | Color interpretation system |
| US2799722A (en) * | 1954-04-26 | 1957-07-16 | Adalia Ltd | Reproduction with localized corrections |
| US2790844A (en) * | 1954-05-11 | 1957-04-30 | Adalia Ltd | Color correction selector |
| US2874212A (en) * | 1955-07-29 | 1959-02-17 | Rca Corp | Generator of color images from monochrome television signals |
| US2883763A (en) * | 1956-09-28 | 1959-04-28 | Otto F Schaper | Carrier landing trainer |
| US2974190A (en) * | 1957-12-09 | 1961-03-07 | Columbia Broadcasting Syst Inc | Electronic matting apparatus |
| US2982814A (en) * | 1958-02-14 | 1961-05-02 | Philips Corp | Apparatus for determining the composition and condition of a specimen of material |
| US3005042A (en) * | 1958-04-17 | 1961-10-17 | David S Horsley | Electronic motion picture printer |
| DE1143227B (en) * | 1960-01-20 | 1963-02-07 | Nippon Electric Co | Method for generating an artificial color television signal |
| US3258528A (en) * | 1963-06-18 | 1966-06-28 | Gen Precision Inc | Converter for changing a black and white television signal to a color television signal |
| US3551589A (en) * | 1967-03-23 | 1970-12-29 | Ward Electronic Ind | Apparatus for converting monochrome television signals to color signals |
| US3558811A (en) * | 1967-05-25 | 1971-01-26 | Xerox Corp | Graphic communication electrical interface system |
| US3486242A (en) * | 1967-05-29 | 1969-12-30 | Us Navy | Assault boat coxswain trainer |
| US3560644A (en) * | 1968-02-29 | 1971-02-02 | Us Navy | Multiple projection television system |
| US3595987A (en) * | 1969-02-20 | 1971-07-27 | Ass Motion Picture Tv Prod | Electronic composite photography |
| US3617626A (en) * | 1969-05-16 | 1971-11-02 | Technicolor | High-definition color picture editing and recording system |
| US3612755A (en) * | 1969-07-03 | 1971-10-12 | Dorothea Weitzner | Color pattern generator |
| US3761607A (en) * | 1969-11-03 | 1973-09-25 | Technicolor | Video monochrom to color conversion |
| US3603962A (en) * | 1970-03-18 | 1971-09-07 | Rca Corp | Color display for computer terminal |
| US3647942A (en) * | 1970-04-23 | 1972-03-07 | Eric J Siegel | Video color synthesizer |
| US3710011A (en) * | 1970-12-04 | 1973-01-09 | Computer Image Corp | System for automatically producing a color display of a scene from a black and white representation of the scene |
| US3673317A (en) * | 1970-12-30 | 1972-06-27 | Westinghouse Electric Corp | Comparitive display of images in color |
| US3772465A (en) * | 1971-06-09 | 1973-11-13 | Ass Of Motion Picture Televisi | Image modification of motion pictures |
| US3742125A (en) * | 1971-06-11 | 1973-06-26 | Electronic Visions Inc | Color video abstract synthesizer |
| US3706841A (en) * | 1971-09-17 | 1972-12-19 | Joseph F Novak | Method and apparatus for converting monochrome pictures to multi-color pictures electronically |
| US3784736A (en) * | 1971-09-17 | 1974-01-08 | J Novak | Method and apparatus for converting monochrome pictures to multi-color pictures electronically |
| US3770885A (en) * | 1971-10-21 | 1973-11-06 | Us Navy | Color electronic periscope view simulator |
| US3770884A (en) * | 1972-02-07 | 1973-11-06 | Us Navy | Luminance control circuit for multi-color periscope view simulator |
| US3769458A (en) * | 1972-05-23 | 1973-10-30 | Us Navy | Color electronic synthesizer |
| US3848856A (en) * | 1973-10-01 | 1974-11-19 | Hazeltine Corp | Local correction apparatus for a color previewer |
| US3972067A (en) * | 1975-01-17 | 1976-07-27 | The Singer Company | Color video synthesizer with monochrome input |
| GB1547666A (en) * | 1975-04-05 | 1979-06-27 | Nippon Electric Co | Video signal coding system |
| US3971068A (en) * | 1975-08-22 | 1976-07-20 | The United States Of America As Represented By The Secretary Of The Navy | Image processing system |
| US4189743A (en) * | 1976-12-20 | 1980-02-19 | New York Institute Of Technology | Apparatus and method for automatic coloration and/or shading of images |
| US4149185A (en) * | 1977-03-04 | 1979-04-10 | Ralph Weinger | Apparatus and method for animated conversion of black and white video to color |
| US4183046A (en) * | 1978-08-17 | 1980-01-08 | Interpretation Systems Incorporated | Electronic apparatus for converting digital image or graphics data to color video display formats and method therefor |
| US4258385A (en) * | 1979-05-15 | 1981-03-24 | Combined Logic Company | Interactive video production system and method |
| DE3024459A1 (en) * | 1979-07-03 | 1981-01-08 | Crosfield Electronics Ltd | PYRAMID INTERPOLATION |
| EP0024862A3 (en) * | 1979-09-04 | 1981-03-25 | Harold Charles Taylor | Video apparatus for visualing effects of selected juxtaposed colours |
| US4318121A (en) * | 1980-05-06 | 1982-03-02 | Jason Taite | Interior decor composition and display systems |
| US4549172A (en) * | 1982-06-21 | 1985-10-22 | Motorola, Inc. | Multicolor display from monochrome or multicolor control unit |
| US4606625A (en) * | 1983-05-09 | 1986-08-19 | Geshwind David M | Method for colorizing black and white footage |
| US4608596A (en) * | 1983-09-09 | 1986-08-26 | New York Institute Of Technology | System for colorizing video with both pseudo-colors and selected colors |
| US4590511A (en) * | 1984-01-03 | 1986-05-20 | Honeywell Inc. | Circuit for converting the phase encoded hue information of a quadrature modulated color subcarrier into distinct analog voltage levels |
| US4642682A (en) * | 1984-04-27 | 1987-02-10 | Vta Technologies, Inc. | Phase responsive composite video signal control system |
| US4694330A (en) * | 1984-08-21 | 1987-09-15 | Sony Corporation | Color correction circuit |
| US4725879A (en) * | 1987-03-27 | 1988-02-16 | Honeywell Inc. | Chroma responsive inspection apparatus selectively producing analog voltage levels based on the luminance, the phase of the chrominance subcarrier, or the amplitude of the chrominance subcarrier |
-
1988
- 1988-07-25 US US07/222,543 patent/US4984072A/en not_active Expired - Fee Related
- 1988-08-03 AU AU20390/88A patent/AU611466B2/en not_active Ceased
- 1988-08-03 CA CA000573741A patent/CA1315206C/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| AU2039088A (en) | 1989-02-09 |
| US4984072A (en) | 1991-01-08 |
| AU611466B2 (en) | 1991-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1315206C (en) | System and method for color image enhancement | |
| EP0302454B1 (en) | System and method for color image enhancement | |
| US5093717A (en) | System and method for digitally coloring images | |
| US3772465A (en) | Image modification of motion pictures | |
| JP3095818B2 (en) | Method and apparatus for mapping a color image to a black and white image | |
| US5739819A (en) | Method and apparatus for generating an artificial shadow in a two dimensional color image | |
| US6351557B1 (en) | Method and apparatus for color manipulation | |
| US7471823B2 (en) | Color correction control graphical user interface | |
| US6983082B2 (en) | Reality-based light environment for digital imaging in motion pictures | |
| US6445816B1 (en) | Compositing video image data | |
| US6049628A (en) | Polygon reshaping in picture colorization | |
| US3784736A (en) | Method and apparatus for converting monochrome pictures to multi-color pictures electronically | |
| JPH05506520A (en) | computer video processing system | |
| CN101088277A (en) | Method of electronic color image saturation processing | |
| CA2270035A1 (en) | Processing image data | |
| Sloan Jr et al. | Color map techniques | |
| US6744440B1 (en) | Image processing apparatus, recording medium, and program | |
| CA1324321C (en) | System and method for color image enhancement | |
| JPH05336346A (en) | Image processing method and storage device | |
| JPH0571099B2 (en) | ||
| JPH0341570A (en) | Color image processing method | |
| JP2844573B2 (en) | Image processing method | |
| JP3468887B2 (en) | Image creation method and image creation device | |
| JPH06180573A (en) | Image forming method | |
| Lee et al. | A New Image Based Lighting Method: Practical Shadow-Based Light Reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEX | Expiry |