US20010045984A1 - Hue adjusting method and hue adjusting device - Google Patents

Hue adjusting method and hue adjusting device Download PDF

Info

Publication number
US20010045984A1
US20010045984A1 US08/827,646 US82764697A US2001045984A1 US 20010045984 A1 US20010045984 A1 US 20010045984A1 US 82764697 A US82764697 A US 82764697A US 2001045984 A1 US2001045984 A1 US 2001045984A1
Authority
US
United States
Prior art keywords
matrix
coefficient
hue
matrix coefficient
parameter
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US08/827,646
Other versions
US6424374B2 (en
Inventor
Hiroyuki Itakura
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Assigned to SONY CORPORATION reassignment SONY CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ITAKURA, HIROYUKI
Publication of US20010045984A1 publication Critical patent/US20010045984A1/en
Application granted granted Critical
Publication of US6424374B2 publication Critical patent/US6424374B2/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00Details of colour television systems
    • H04N9/64Circuits for processing colour signals
    • H04N9/643Hue control means, e.g. flesh tone control

Definitions

  • the present invention relates to a hue adjusting method for adjusting the hue by executing matrix conversion processing of a color signal and so on.
  • the hue is adjusted by executing matrix conversion to, for example, color signals of the three primary colors.
  • the matrix X comprises six variable matrix coefficients of s, t, u, v, w and x and the color signals are related each other, when one matrix coefficient is changed, another matrix coefficients are also changed. Therefore, a user had to input six matrix coefficients every time when the hue was adjusted and it was inconvenient.
  • the present invention was performed considering said problems, and an object of the present invention is to provide a hue adjusting method and a hue adjusting device that are possible to adjust simply and easily the hue.
  • FIG. 1 is a block diagram showing the schematic structure of a camera apparatus according to the present invention.
  • FIG. 2 is a block diagram showing the concrete structure of a process circuit of said camera apparatus.
  • FIG. 3 is an explanatory view of the concept of adjusting the hue by using said camera apparatus.
  • FIG. 4 is a flow chart for describing actuation of the first matrix adjusting mode of said camera apparatus.
  • FIG. 5 is a flow chart for describing actuation of the second matrix adjusting mode of said camera apparatus.
  • a camera apparatus adjusts finely the hue by adjusting six matrix coefficients by a parameter when multiplying color signals obtained corresponding to image pick-up light of a subject by the six matrix coefficients.
  • Said camera apparatus includes, for example as shown in FIG. 1: a lens 1 gathering the image pick-up light of the subject; a CCD image sensor 2 outputting an image pick-up signal according to said image pick-up light; a video amplifier 3 for amplifying an image pick-up signal from the CCD image sensor 2 , holding a sample, adjusting gain and so on; an A/D converter 4 for outputting image pick-up data by digitalizing the image pick-up signal from the video amplifier 3 ; a process circuit 5 for executing, that is, camera process to the image pick-up data from the A/D converter 4 ; and an encoder 6 for generating picture data of a standard TV system from the image pick-up data from the process circuit 5 and outputting it through a D/A converter 7 .
  • the CCD image sensor 2 receives the image pick-up light from the subject through the lens 1 , and supplies the image pick-up signal corresponding to the image pick-up light through the video amplifier 3 , the A/D converter 4 to the process circuit 5 .
  • said image-pick-up signal includes R (red), G (green) and B (blue) of color signals of the three primary colors
  • the process circuit 5 performs the camera process of gamma correction, pedestal clamp, blanking inserting and so on to color signals R, G and B respectively.
  • the color signals to which the camera process is executed are converted to, for example, image signals of the NTSC system and are outputted through the D/A converter 7 .
  • said camera apparatus includes a synchronous signal generating circuit 8 for generating a synchronous pulse, a timing generator 9 for generating a horizontal and a vertical synchronous signals according to the synchronous pulse from the synchronous signal generating circuit 8 , and a driver 10 for driving the CCD image sensor 2 corresponding to the synchronous signal from the timing generator 9 .
  • the process circuit 5 and the encoder 6 actuates corresponding to the synchronous pulse from the synchronous signal generating circuit 8 .
  • said camera apparatus includes an operation part 11 for executing operation such as input of six matrix coefficients and so on, ROM (Read Only Memory) 12 in which programs such as a matrix operation has been stored in advance, RAM (Random Access Memory) 13 in which the inputted matrix coefficient and so on are temporarily stored, a character generator 14 for generating a character and so on for urging input instruction of the matrix coefficients, an image mixing circuit 15 for mixing the generated character with image data from the encoder, a view finder 17 for showing the image being taken and the character by supplying the image data from the image mixing circuit 15 through the a D/A converter 16 , and the CPU (Central Processing Unit) 18 for controlling the whole camera apparatus.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • a character generator 14 for generating a character and so on for urging input instruction of the matrix coefficients
  • an image mixing circuit 15 for mixing the generated character with image data from the encoder
  • a view finder 17 for showing the image being taken and the character by supplying the image data from the image mixing circuit 15
  • the CPU 18 makes the character generator 14 generate the fixed character at the mode for adjusting the matrix coefficients and indicate the character on the view finder 17 through the image mixing circuit 15 and the D/A converter 16 .
  • the process circuit includes, for example as shown in FIG. 2, a matrix circuit 20 for executing matrix conversion to the supplied color signals R, G and B and outputting the color signals R′, G′ and B′, gamma correction circuits 33 R, 33 G and 33 B for executing gamma correction to the color signals R′, G′ and B′ from the matrix circuit 20 respectively, and pedestal circuits 34 R, 34 G and 34 B for clamping the black levels of the color signals R′, G′ and B′ from the gamma correction circuits 33 R, 33 G and 33 B and for adding pedestal values.
  • the matrix circuit 20 is supplied with each of color signals R, G and B, and includes a multiplication circuit 21 for multiplying the color signal R by the matrix coefficient w, a multiplication circuit 22 for multiplying the color signal G by the matrix coefficient x, a multiplication circuit 23 for multiplying the color signal R by the matrix coefficient u, a multiplication circuit 24 for multiplying the color signal B by the matrix coefficient v, a multiplication circuit 25 for multiplying the color signal G by the matrix coefficient s, a multiplication circuit 26 for. multiplying the color signal B by the matrix coefficient t.
  • these multiplication circuits 21 through 26 it is possible to change setting of the matrix coefficients corresponding to control of the CPU 18 .
  • the matrix circuit 20 includes an addition circuit 27 for adding the color signals from the multiplication circuits 21 and 22 , an addition circuit 28 for adding the color signals from the multiplication circuits 23 and 24 , an addition circuit 29 for adding color signals from the multiplication circuits 25 and 26 , an addition circuit 30 for adding the color signal R and the color signal from the addition circuit 29 , an addition circuit 31 for adding the color signal G and the color signal from the addition circuit 28 , and an addition circuit 32 for adding the color signal B and the color signal from the addition circuit 27 .
  • the camera apparatus constructed as above mentioned is set to the matrix adjusting modes 1 and 2 corresponding to operation of the operating part 11 .
  • an operator inputs central matrix coefficients of s, t, u, v, w and x which are the central values within the variable range of the matrix coefficients in the following fine adjustment and the matrix coefficients of s1, t1, u1, v1, w1 and x1 instead of inputting the matrix coefficient that is actually set actually.
  • the central matrix X is decided by deciding the central matrix coefficients s, t, u, v, w and x
  • the matrix A is decided by inputting the matrix coefficients of s1, t1, u1, v1, w1 and x1.
  • the matrix coefficient which is actually set, as shown in FIG. 3, is adjustable between the matrixes B and A that are symmetrical as the central matrix X is the center.
  • said camera apparatus is possible to move an arbitrary point Q of routs Q 1 through Q 2 , to Q 1 and Q 2 which are obtained by substrating the matrixes A and B for the matrix X of the expression (1), thus being able to adjust finely the hue.
  • step S 1 makes the character generator 14 generate the character for urging the input of the central matrix X, supplies the character to the view finder 17 with the image data outputted from the encoder 6 , and proceeds to a step S 2 . Then, for example, the image of the subject and “Central Matrix ?” at the corner are indicated in the view finder 17 .
  • a step S 2 the CPU 18 detects whether instruction to choose the preset value that is set in advance is inputted or not, in case of being inputted, sets the matrix coefficient comprising the preset value, that is, stores into the RAM 13 , and proceeds to a step S 4 .
  • the preset value R is stored into the ROM 12 and the RAM 13 in advance.
  • the CPU 18 proceeds to a step S 3 and detects whether the coefficient of the matrix X is inputted or not.
  • the CPU 18 proceeds to a step S 4 , and in case of not being inputted, returns to the step S 2 and repeats this process.
  • the CPU 18 makes the character generator 14 generate the character for urging inputting of the off set matrix A, supplies the character to the view finder 17 with image data outputted from the encoder 6 and proceeds to a step S 5 . Then, for example, the image of the subject and “Central Matrix A ?” at the corner of it are indicated in the view finder 17 .
  • the CPU 18 waits the off set matrix A to be inputted, when the off set matrix A is inputted, the CPU 18 stores the matrix coefficient into the RAM 13 , and proceeds to the step S 6 .
  • the CPU 18 outputs the matrix coefficients of the central matrix X and the off set matrix from the RAM 13 one after another, calculates the off set matrix B that is a pair of the matrix A to the central matrix X and proceeds to a step S 7 .
  • the CPU 18 calculates the off set matrix B by the expression (2).
  • the CPU 18 calculates the matrix coefficient of the off set matrix B by the following expression.
  • the CPU 18 makes the character generator 14 generate the character for urging inputting of a parameter a, supplies the character to the view finder 17 with image data outputted from the encoder 6 and proceeds to a step S 8 . Then, for example, the image of the subject and “Parameter a ?” at the corner of it are indicated in the view finder 17 .
  • the CPU 18 detects whether the parameter a is inputted, or whether the instruction for setting the matrix coefficient is inputted, when the parameter a is inputted, the CPU 18 stores the value a into the RAM 13 and proceeds to a step S 9 and when the instruction for setting the matrix coefficient is inputted, the CPU 18 proceeds to a step 11 .
  • the CPU 18 outputs the matrix coefficients of the off set matrixes A and B and the parameter a from RAM 13 one after another, and calculates the matrix X′ according to the off set matrixes A and B and the parameter a. And the CPU 18 stores the matrix coefficient of the matrix X′ into the RAM 13 and proceeds a step 10 .
  • the CPU 18 calculates the matrix X′ by an expression (3).
  • the CPU 18 sets the matrix coefficient of the matrix X′ as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20 .
  • the CPU 18 sets the matrix coefficient of the matrix X as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20 .
  • the embodiment will be described as follows.
  • the matrix circuit 20 the matrix conversion is executed by the set matrix coefficient, and the color signals R, G and B are outputted through the gamma correction circuit 33 and the pedestal circuit 34 .
  • Said camera apparatus calculates the center matrix X when the off set matrixes A and B are inputted at the mode 2.
  • Said camera device as shown in FIG. 3, is possible to move an arbitrary point Q of the routes Q 1 through Q 2 by using a parameter to Q 1 and Q 2 which are obtained by substituting the matrixes A and B for the matrix X of the expression (1) and is possible to adjust finely the hue.
  • the CPU 18 proceeds to a step S 101 when it is set to the mode 2, makes the character generator 14 generate the character for urging inputting of the off set matrix A, supplies the character to the view finder 17 with the image data outputted from the encoder 6 , and proceeds to a step 102 . Then, for example, the image of the subject and “Off Set Matrix A ?” at the corner of it are shown in the view finder 17 .
  • the CPU 18 waits inputting of the off set matrix A; when the off set matrix A is inputted, the CPU 18 stores the matrix coefficient of it into the RAM 13 and proceeds to a step S 103 .
  • the CPU 18 makes the character generator 14 generate the character for urging inputting of the off set matrix B, supplies the character to the view finder 17 with the image data outputted from the encoder 6 , and proceeds to a step 104 . Then, for example, the image of the subject and “Off Set Matrix B ?” are shown in the view finder 17 .
  • the CPU 18 waits inputting of the off set matrix B; when the off set matrix B is inputted, the CPU 18 stores the matrix coefficient of it into the RAM 13 and proceeds to a step S 105 .
  • the CPU 18 outputs the matrix coefficients of the off set matrixes A and B from the RAM 13 one after another and calculates the central matrix. That is, the CPU 18 makes the value that the sum of each matrix coefficient is divided by 2 to the off set matrixes A and B be the matrix coefficient of the central matrix. That is, the central matrix is found as follows.
  • each matrix coefficient is found as follows.
  • the CPU 18 makes the character generator 14 generate the character of said matrix coefficient, indicates the character in the view finder 17 and proceeds to a step 106 .
  • the CPU 18 detects whether a defined button (not shown) of the operating part 11 is pressed or not, and proceeds to a step 107 when pressed; when the decided button is not pressed and a reset button is pressed, the CPU 18 returns to the step S 101 and waits inputting of the off set matrix A again.
  • the CPU 18 makes the character generator 14 generate the character for urging inputting of the parameter a, supplies the character to the view finder 17 with the image data outputted from the encoder 6 , and proceeds to a step 108 . Then, for example, the image of the subject and “Parameter a ?” at the corner of it are shown in the view finder 17 .
  • An a step S 108 the CPU 18 detects whether the parameter a is inputted, or whether the instruction of setting the matrix coefficient is inputted, when the parameter a is inputted, the CPU 18 stores the value of the parameter a into the RAM 13 and proceeds to a step S 109 and when the instruction of setting the matrix coefficient is inputted, the CPU 18 proceeds to a step 112 .
  • the CPU 18 outputs the matrix coefficients of the off set matrixes A and B and the parameter a from the RAM 13 one after another, and calculates the matrix X′ according to the off set matrixes A and B and the parameter a, and proceeds to a step S 110 .
  • the CPU 18 calculates the matrix coefficient of the matrix X′ similarly to the step S 9 .
  • the CPU 18 sets the matrix coefficient of the matrix X′ as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20 . And at a step S 112 , the CPU 18 sets the matrix coefficient of the matrix X as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20 .
  • the color signals R, G and B are supplied to the process circuit 5
  • the matrix circuit 20 the matrix conversion is executed by the set matrix coefficient, and the color signals R, G and B are outputted through the gamma correction circuit 33 and the pedestal circuit 34 .
  • the parameters a are inputted from the operating part 11 one by one according to the present embodiment, the present invention is not limited to this embodiment, for example, the parameters a may be continuously inputted by turning a dial and the hue may be adjusted while observing the image shown in the view finder 17 .
  • the matrix X and the off set matrix A have been set in advance without inputting any component of the matrix again every time when adjusting the hue, it is possible to adjust easily the hue by changing only the parameter a. Further, in said hue adjusting method, it is possible to change the range for adjusting the hue without changing the reference of the matrix conversion by changing the off set matrix A.
  • the hue adjusting method according to the present invention, if the off set matrixes A and B have been set in advance without inputting any component of the matrix again every time when adjusting the hue, it is possible to adjust easily the hue by changing only the parameter a. Further, according to said hue adjusting method, it is possible to input the off set matrixes A and B in which the fixed off set has added to the matrix X in advance, therefore, it is possible to set the range that the matrix conversion is executed in advance and possible to avoid adjusting the hue that deviates from the object of the present invention.

Abstract

An object of the Invention is to provide a hue adjusting method that is simply and easily possible to adjust finely the hue and a hue adjusting device. The CPU 18 calculates an off set matrix that is in a pair with an off set matrix to said central matrix X when a matrix X and an off set matrix A are inputted from an operating part 11. Then the CPU 18 carries out the following operation when a parameter a is inputted from the operating part 11.
X′=(a·A+(1−a)B)/2
It is possible to execute matrix conversion to adjust the hue by setting a matrix coefficient of each component of the matrix X′ to multiplication circuit 21 through 26 in the matrix 20.

Description

    BACKGROUND OF THE INVENTION
  • 1. Field of the Invention [0001]
  • The present invention relates to a hue adjusting method for adjusting the hue by executing matrix conversion processing of a color signal and so on. [0002]
  • 2. Description of the Related Art [0003]
  • At a processing part used for a camera device and so on, the hue is adjusted by executing matrix conversion to, for example, color signals of the three primary colors. Here, if the color signals of the three primary colors R, G and B, color signals R′, G′ and B′, to which the matrix conversion is executed, and a matrix X are as follows: [0004] P = ( R G B ) Q = ( R G B ) X = ( 1 s t u 1 v w x 1 )
    Figure US20010045984A1-20011129-M00001
  • the matrix conversion is represented by an expression (1). [0005]
  • Q=X·P  (1)
  • As the matrix X comprises six variable matrix coefficients of s, t, u, v, w and x and the color signals are related each other, when one matrix coefficient is changed, another matrix coefficients are also changed. Therefore, a user had to input six matrix coefficients every time when the hue was adjusted and it was inconvenient. [0006]
  • The present invention was performed considering said problems, and an object of the present invention is to provide a hue adjusting method and a hue adjusting device that are possible to adjust simply and easily the hue. [0007]
  • SUMMARY OF THE INVENTION
  • According to the present invention, a hue adjusting method, for adjusting the hue of a color image signal indicated by the three primary color signals, includes the steps of: inputting a central matrix coefficient deciding a central matrix X and that is the center of the variable range of a matrix coefficient; storing said central matrix coefficient; inputting a first matrix coefficient deciding a first matrix A and that is one end of the variable range of the matrix coefficient; storing said first matrix coefficient; inputting a parameter a for adjusting said central matrix coefficient; storing said parameter a; calculate X′ by X′={a·A+(1−a)B}/2 when the second matrix B is B=2?X−A according to the stored central matrix coefficient, the first matrix coefficient and the parameter a; and setting X′ as the coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals. [0008]
  • According to the present invention, a hue adjusting apparatus, for adjusting the hue of a color image signal indicated by three primary color signals, includes: a central matrix coefficient input means for inputting a central matrix coefficient deciding a central matrix X and that is the center of the variable range of a matrix coefficient; a central matrix coefficient storing means for storing said central matrix coefficient; a first matrix coefficient input means for inputting the first matrix coefficient deciding a first matrix A and that is one end of the variable range of the matrix coefficient; a first matrix coefficient storing means for storing said first matrix coefficient; a parameter input means for inputting a parameter a for adjusting said central matrix coefficient; a parameter storing means for storing said parameter a; a calculating means for calculating X′ by X′={a·A+(1−a)B}/2 when the second matrix B is B=2·X−A according to the stored central matrix coefficient, the first matrix coefficient and the parameter a; a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals; and a setting means for setting X′ as a coefficient of huge conversion of said matrix means. [0009]
  • According to the present invention, a hue adjusting method, for adjusting the hue of a color image signal indicated by three primary color signals, includes the steps of: inputting first matrix coefficient deciding a first matrix A and that is an end of the variable range of a matrix coefficient; storing said first matrix coefficient; inputting a second matrix coefficient deciding a second matrix B and that is the other end of the variable range of the matrix coefficient; storing said second matrix coefficient; inputting a parameter a for adjusting said matrix coefficients between the matrix coefficients of said first matrix A and said second matrix B; storing said parameter a; calculating X′ by X′={a·A+(1−a)B}/2 according to the stored first and second matrix coefficients and the parameter a; and setting X′ as the coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals. [0010]
  • According to the present invention, a hue adjusting apparatus, for adjusting the hue of a color image signal indicated by three primary color signals, includes: a first matrix coefficient input means for inputting a first matrix coefficient deciding a first matrix A and that is an end of the variable range of a matrix coefficient; a first matrix coefficient storing means for storing said first matrix coefficient; a second matrix coefficient input means for inputting a second matrix coefficient deciding a second matrix B and that is the other end of the variable range of the matrix coefficient; a second matrix coefficient storing means for storing said second matrix coefficient; a parameter input means for inputting a parameter a for adjusting matrix coefficients between the matrix coefficients of said first matrix A and said second matrix B; a parameter storing means for storing said parameter a; a calculation means for calculating X′ by X′={a·A+(1−a)B}/2 according to the stored first and second matrix coefficients and the parameter a; and a setting means for setting X′ as a coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three colors' signals.[0011]
  • BRIEF DESCRIPTION OF DRAWINGS
  • FIG. 1 is a block diagram showing the schematic structure of a camera apparatus according to the present invention. [0012]
  • FIG. 2 is a block diagram showing the concrete structure of a process circuit of said camera apparatus. [0013]
  • FIG. 3 is an explanatory view of the concept of adjusting the hue by using said camera apparatus. [0014]
  • FIG. 4 is a flow chart for describing actuation of the first matrix adjusting mode of said camera apparatus. [0015]
  • FIG. 5 is a flow chart for describing actuation of the second matrix adjusting mode of said camera apparatus.[0016]
  • DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • An embodiment according to the present invention will be described referring to drawings. [0017]
  • A camera apparatus according to the present invention adjusts finely the hue by adjusting six matrix coefficients by a parameter when multiplying color signals obtained corresponding to image pick-up light of a subject by the six matrix coefficients. [0018]
  • Said camera apparatus includes, for example as shown in FIG. 1: a [0019] lens 1 gathering the image pick-up light of the subject; a CCD image sensor 2 outputting an image pick-up signal according to said image pick-up light; a video amplifier 3 for amplifying an image pick-up signal from the CCD image sensor 2, holding a sample, adjusting gain and so on; an A/D converter 4 for outputting image pick-up data by digitalizing the image pick-up signal from the video amplifier 3; a process circuit 5 for executing, that is, camera process to the image pick-up data from the A/D converter 4; and an encoder 6 for generating picture data of a standard TV system from the image pick-up data from the process circuit 5 and outputting it through a D/A converter 7.
  • The [0020] CCD image sensor 2 receives the image pick-up light from the subject through the lens 1, and supplies the image pick-up signal corresponding to the image pick-up light through the video amplifier 3, the A/D converter 4 to the process circuit 5. Here, said image-pick-up signal includes R (red), G (green) and B (blue) of color signals of the three primary colors, and the process circuit 5 performs the camera process of gamma correction, pedestal clamp, blanking inserting and so on to color signals R, G and B respectively. The color signals to which the camera process is executed are converted to, for example, image signals of the NTSC system and are outputted through the D/A converter 7.
  • Further, said camera apparatus includes a synchronous [0021] signal generating circuit 8 for generating a synchronous pulse, a timing generator 9 for generating a horizontal and a vertical synchronous signals according to the synchronous pulse from the synchronous signal generating circuit 8, and a driver 10 for driving the CCD image sensor 2 corresponding to the synchronous signal from the timing generator 9. The process circuit 5 and the encoder 6 actuates corresponding to the synchronous pulse from the synchronous signal generating circuit 8.
  • Further, said camera apparatus includes an [0022] operation part 11 for executing operation such as input of six matrix coefficients and so on, ROM (Read Only Memory) 12 in which programs such as a matrix operation has been stored in advance, RAM (Random Access Memory) 13 in which the inputted matrix coefficient and so on are temporarily stored, a character generator 14 for generating a character and so on for urging input instruction of the matrix coefficients, an image mixing circuit 15 for mixing the generated character with image data from the encoder, a view finder 17 for showing the image being taken and the character by supplying the image data from the image mixing circuit 15 through the a D/A converter 16, and the CPU (Central Processing Unit) 18 for controlling the whole camera apparatus.
  • The [0023] CPU 18 makes the character generator 14 generate the fixed character at the mode for adjusting the matrix coefficients and indicate the character on the view finder 17 through the image mixing circuit 15 and the D/A converter 16.
  • Here, the process circuit includes, for example as shown in FIG. 2, a matrix circuit [0024] 20 for executing matrix conversion to the supplied color signals R, G and B and outputting the color signals R′, G′ and B′, gamma correction circuits 33R, 33G and 33B for executing gamma correction to the color signals R′, G′ and B′ from the matrix circuit 20 respectively, and pedestal circuits 34R, 34G and 34B for clamping the black levels of the color signals R′, G′ and B′ from the gamma correction circuits 33R, 33G and 33B and for adding pedestal values.
  • The matrix circuit [0025] 20, as concretely shown in FIG. 2, is supplied with each of color signals R, G and B, and includes a multiplication circuit 21 for multiplying the color signal R by the matrix coefficient w, a multiplication circuit 22 for multiplying the color signal G by the matrix coefficient x, a multiplication circuit 23 for multiplying the color signal R by the matrix coefficient u, a multiplication circuit 24 for multiplying the color signal B by the matrix coefficient v, a multiplication circuit 25 for multiplying the color signal G by the matrix coefficient s, a multiplication circuit 26 for. multiplying the color signal B by the matrix coefficient t. In these multiplication circuits 21 through 26, it is possible to change setting of the matrix coefficients corresponding to control of the CPU 18.
  • Further, the matrix circuit [0026] 20 includes an addition circuit 27 for adding the color signals from the multiplication circuits 21 and 22, an addition circuit 28 for adding the color signals from the multiplication circuits 23 and 24, an addition circuit 29 for adding color signals from the multiplication circuits 25 and 26, an addition circuit 30 for adding the color signal R and the color signal from the addition circuit 29, an addition circuit 31 for adding the color signal G and the color signal from the addition circuit 28, and an addition circuit 32 for adding the color signal B and the color signal from the addition circuit 27.
  • Here, if vector comprising the color signals R, G and B is P, vector comprising the color signals R[0027]
    Figure US20010045984A1-20011129-P00900
    , G
    Figure US20010045984A1-20011129-P00900
    and B
    Figure US20010045984A1-20011129-P00900
    is Q and the central matrix is X: P = ( R G B ) Q = ( R G B ) X = ( 1 s t u 1 v w x 1 )
    Figure US20010045984A1-20011129-M00002
  • relation between the color signals R, G and B and the color signals R′, G′ and B′ which matrix conversion is executed is shown by an expression (1). [0028]
  • Q=X·p  (1)
  • The camera apparatus constructed as above mentioned, is set to the [0029] matrix adjusting modes 1 and 2 corresponding to operation of the operating part 11.
  • At the [0030] matrix adjusting mode 1, an operator inputs central matrix coefficients of s, t, u, v, w and x which are the central values within the variable range of the matrix coefficients in the following fine adjustment and the matrix coefficients of s1, t1, u1, v1, w1 and x1 instead of inputting the matrix coefficient that is actually set actually. Still, the central matrix X is decided by deciding the central matrix coefficients s, t, u, v, w and x, and the matrix A is decided by inputting the matrix coefficients of s1, t1, u1, v1, w1 and x1.
  • The matrix coefficient which is actually set, as shown in FIG. 3, is adjustable between the matrixes B and A that are symmetrical as the central matrix X is the center. [0031]
  • Further, said camera apparatus, as shown in FIG. 3, is possible to move an arbitrary point Q of routs Q[0032] 1 through Q2, to Q1 and Q2 which are obtained by substrating the matrixes A and B for the matrix X of the expression (1), thus being able to adjust finely the hue.
  • As concretely shown in FIG. 4, after the [0033] CPU 18 is set to the mode 1, it proceeds to a step S1, makes the character generator 14 generate the character for urging the input of the central matrix X, supplies the character to the view finder 17 with the image data outputted from the encoder 6, and proceeds to a step S2. Then, for example, the image of the subject and “Central Matrix ?” at the corner are indicated in the view finder 17.
  • At a step S[0034] 2, the CPU 18 detects whether instruction to choose the preset value that is set in advance is inputted or not, in case of being inputted, sets the matrix coefficient comprising the preset value, that is, stores into the RAM 13, and proceeds to a step S4. The preset value R is stored into the ROM 12 and the RAM 13 in advance. In case of not being inputted at the step S2, the CPU 18 proceeds to a step S3 and detects whether the coefficient of the matrix X is inputted or not. In case of being inputted, the CPU 18 proceeds to a step S4, and in case of not being inputted, returns to the step S2 and repeats this process.
  • At the step S[0035] 4, the CPU 18 makes the character generator 14 generate the character for urging inputting of the off set matrix A, supplies the character to the view finder 17 with image data outputted from the encoder 6 and proceeds to a step S5. Then, for example, the image of the subject and “Central Matrix A ?” at the corner of it are indicated in the view finder 17.
  • At a step S[0036] 5, the CPU 18 waits the off set matrix A to be inputted, when the off set matrix A is inputted, the CPU 18 stores the matrix coefficient into the RAM 13, and proceeds to the step S6.
  • At a step S[0037] 6, the CPU 18 outputs the matrix coefficients of the central matrix X and the off set matrix from the RAM 13 one after another, calculates the off set matrix B that is a pair of the matrix A to the central matrix X and proceeds to a step S7.
  • Here, if the off set matrixes A and B are as follows: [0038] A = ( 1 s1 t1 u1 1 v1 w1 x1 1 ) B = ( 1 s2 t2 u2 1 v1 w2 x2 1 )
    Figure US20010045984A1-20011129-M00003
  • relation between the central X and the off set matrixes A and B is as follows: [0039]
  • X=(A+B)/2
  • That is, the relation between matrix coefficients of the matrix X, the off set matrix A and B, for example, the relation between s, s[0040] 1 and s2 are as follows: s = s1 + s2 2
    Figure US20010045984A1-20011129-M00004
  • Still, the relation between other matrix coefficients t, u, v, w, x and so on is similar. [0041]
  • Therefore, the [0042] CPU 18 calculates the off set matrix B by the expression (2).
  • B=X−A  (2)
  • That is, the [0043] CPU 18 calculates the matrix coefficient of the off set matrix B by the following expression.
  • z2=2·z−z1
  • However, z=s, t, u, v, w and x [0044]
  • At the step S[0045] 7, the CPU 18 makes the character generator 14 generate the character for urging inputting of a parameter a, supplies the character to the view finder 17 with image data outputted from the encoder 6 and proceeds to a step S8. Then, for example, the image of the subject and “Parameter a ?” at the corner of it are indicated in the view finder 17.
  • At a step S[0046] 8, the CPU 18 detects whether the parameter a is inputted, or whether the instruction for setting the matrix coefficient is inputted, when the parameter a is inputted, the CPU 18 stores the value a into the RAM 13 and proceeds to a step S9 and when the instruction for setting the matrix coefficient is inputted, the CPU 18 proceeds to a step 11.
  • At a step S[0047] 9, the CPU 18 outputs the matrix coefficients of the off set matrixes A and B and the parameter a from RAM 13 one after another, and calculates the matrix X′ according to the off set matrixes A and B and the parameter a. And the CPU 18 stores the matrix coefficient of the matrix X′ into the RAM 13 and proceeds a step 10.
  • Here, the [0048] CPU 18 calculates the matrix X′ by an expression (3).
  • X′={a·A+(1−a)B}/2  (3)
  • That is, the [0049] CPU 18 calculates the matrix coefficient of the matrix X′, for example s, by using the following expression. Still, other matrix coefficients t, u, v, w and x are similarly calculated. s = s1 × a + s2 × ( 1 - a ) 2
    Figure US20010045984A1-20011129-M00005
  • At a step S[0050] 10, the CPU 18 sets the matrix coefficient of the matrix X′ as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20. And at a step S11, the CPU 18 sets the matrix coefficient of the matrix X as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20. S till, though finding the matrix B by operation at the step S6 in said example, it is possible that B expresses X and A as variables as clearly from the expression (2), so that X′ may be found in the step (3) without finding the matrix B by operation.
  • Here, the embodiment will be described as follows. When the color signals R, G and B are supplied to the [0051] process circuit 5, in the matrix circuit 20, the matrix conversion is executed by the set matrix coefficient, and the color signals R, G and B are outputted through the gamma correction circuit 33 and the pedestal circuit 34.
  • That is, when said camera apparatus adjusts the hue, it is easily possible to adjust the hue by setting the central matrix X and the off set matrix A and by changing only the parameter a without inputting again all of six matrix coefficients in which the hue is changed with being related to each other. Said camera apparatus, as shown in FIG. 3, it is possible to adjust the hue by changing the routes Q[0052] 1 through Q2 without changing the reference point Q0 determined by the central matrix X by changing the off set matrix A.
  • Next, the actuating state of the camera apparatus at the [0053] mode 2 will be described.
  • Said camera apparatus calculates the center matrix X when the off set matrixes A and B are inputted at the [0054] mode 2. Said camera device, as shown in FIG. 3, is possible to move an arbitrary point Q of the routes Q1 through Q2 by using a parameter to Q1 and Q2 which are obtained by substituting the matrixes A and B for the matrix X of the expression (1) and is possible to adjust finely the hue.
  • Concretely to say, as shown in FIG. 5, the [0055] CPU 18 proceeds to a step S 101 when it is set to the mode 2, makes the character generator 14 generate the character for urging inputting of the off set matrix A, supplies the character to the view finder 17 with the image data outputted from the encoder 6, and proceeds to a step 102. Then, for example, the image of the subject and “Off Set Matrix A ?” at the corner of it are shown in the view finder 17.
  • At the step S[0056] 102: the CPU 18 waits inputting of the off set matrix A; when the off set matrix A is inputted, the CPU 18 stores the matrix coefficient of it into the RAM 13 and proceeds to a step S103.
  • At the step [0057] 103, the CPU 18 makes the character generator 14 generate the character for urging inputting of the off set matrix B, supplies the character to the view finder 17 with the image data outputted from the encoder 6, and proceeds to a step 104. Then, for example, the image of the subject and “Off Set Matrix B ?” are shown in the view finder 17.
  • At the step S[0058] 104, the CPU 18 waits inputting of the off set matrix B; when the off set matrix B is inputted, the CPU 18 stores the matrix coefficient of it into the RAM 13 and proceeds to a step S105.
  • At the step [0059] 105, the CPU 18 outputs the matrix coefficients of the off set matrixes A and B from the RAM 13 one after another and calculates the central matrix. That is, the CPU 18 makes the value that the sum of each matrix coefficient is divided by 2 to the off set matrixes A and B be the matrix coefficient of the central matrix. That is, the central matrix is found as follows.
  • X=(A+B)/2
  • That is, each matrix coefficient is found as follows. [0060]
  • z=(z1+z2)/2
  • Where t, u, v, w and x. [0061]
  • Further, the [0062] CPU 18 makes the character generator 14 generate the character of said matrix coefficient, indicates the character in the view finder 17 and proceeds to a step 106.
  • At the step [0063] 106: the CPU 18 detects whether a defined button (not shown) of the operating part 11 is pressed or not, and proceeds to a step 107 when pressed; when the decided button is not pressed and a reset button is pressed, the CPU 18 returns to the step S 101 and waits inputting of the off set matrix A again.
  • At the step [0064] 107, the CPU 18 makes the character generator 14 generate the character for urging inputting of the parameter a, supplies the character to the view finder 17 with the image data outputted from the encoder 6, and proceeds to a step 108. Then, for example, the image of the subject and “Parameter a ?” at the corner of it are shown in the view finder 17.
  • An a step S [0065] 108, the CPU 18 detects whether the parameter a is inputted, or whether the instruction of setting the matrix coefficient is inputted, when the parameter a is inputted, the CPU 18 stores the value of the parameter a into the RAM 13 and proceeds to a step S 109 and when the instruction of setting the matrix coefficient is inputted, the CPU 18 proceeds to a step 112.
  • At a step S [0066] 109, the CPU 18 outputs the matrix coefficients of the off set matrixes A and B and the parameter a from the RAM 13 one after another, and calculates the matrix X′ according to the off set matrixes A and B and the parameter a, and proceeds to a step S 110. Here, the CPU 18 calculates the matrix coefficient of the matrix X′ similarly to the step S 9.
  • At a step S [0067] 110, the CPU 18 sets the matrix coefficient of the matrix X′ as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20. And at a step S 112, the CPU 18 sets the matrix coefficient of the matrix X as the matrix coefficient of each of multiplication circuits 21 through 26 of the matrix circuit 20. When the color signals R, G and B are supplied to the process circuit 5, in the matrix circuit 20, the matrix conversion is executed by the set matrix coefficient, and the color signals R, G and B are outputted through the gamma correction circuit 33 and the pedestal circuit 34.
  • That is, every time when said camera apparatus adjusts the hue, it is easily possible to adjust the hue by setting the central matrix X and the off set matrix A and by changing only the parameter a without inputting again all of six matrix coefficients in which the hue is changed being related each other. Further, said camera apparatus is possible to input the off set matrixes A and B in which the predetermined off set is added to the central matrix X, therefore, it is possible to set the variable range of the matrix coefficient and to avoid executing adjustment of the hue that deviates from the original object. [0068]
  • Though the parameters a are inputted from the operating [0069] part 11 one by one according to the present embodiment, the present invention is not limited to this embodiment, for example, the parameters a may be continuously inputted by turning a dial and the hue may be adjusted while observing the image shown in the view finder 17.
  • As above mentioned, according to the present invention, if the matrix X and the off set matrix A have been set in advance without inputting any component of the matrix again every time when adjusting the hue, it is possible to adjust easily the hue by changing only the parameter a. Further, in said hue adjusting method, it is possible to change the range for adjusting the hue without changing the reference of the matrix conversion by changing the off set matrix A. [0070]
  • Further, in the hue adjusting method according to the present invention, if the off set matrixes A and B have been set in advance without inputting any component of the matrix again every time when adjusting the hue, it is possible to adjust easily the hue by changing only the parameter a. Further, according to said hue adjusting method, it is possible to input the off set matrixes A and B in which the fixed off set has added to the matrix X in advance, therefore, it is possible to set the range that the matrix conversion is executed in advance and possible to avoid adjusting the hue that deviates from the object of the present invention. [0071]

Claims (8)

What is claimed is:
1. A hue adjusting method for adjusting the hue of a color image signal indicated by three primary color signals, said hue adjusting method comprising the steps of:
inputting a central matrix coefficient deciding a central matrix X and that is the center of the variable range of a matrix coefficient;
storing said central matrix coefficient;
inputting a first matrix coefficient deciding a first matrix A and that is one end of the variable range of the matrix coefficient;
storing said first matrix coefficient;
inputting a parameter a for adjusting said central matrix coefficient;
storing said parameter a;
calculate X′ by X′={a·A+(1−a)B}/2 when the second matrix B is B=2·X−A according to the stored central matrix coefficient, the first matrix coefficient and the parameter a; and
setting X′ as the coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals.
2. A hue adjusting method according to
claim 1
, wherein a step for inputting said central matrix coefficient comprises the step of indicating information for urging to input said central matrix coefficient on an indicating means.
3. A hue adjusting method according to
claim 1
, wherein a step for inputting said parameter comprises the step of;
indicating information for urging to input said parameter on an indicating means.
4. A hue adjusting apparatus for adjusting the hue of a color image signal indicated by three primary color signals, said hue adjusting apparatus comprising:
a central matrix coefficient input means for inputting the central matrix coefficient deciding a central matrix X and that is the center of the variable range of a matrix coefficient;
a central matrix coefficient storing means for storing said central matrix coefficient;
a first matrix coefficient input means for inputting a first matrix coefficient deciding a first matrix A and that is one end of the variable range of the matrix coefficient;
a first matrix coefficient storing means for storing said first matrix coefficient;
a parameter input means for inputting a parameter a for adjusting said central matrix coefficient;
a parameter storing means for storing said parameter a;
a calculating means for calculating X′ by X′={a·A+(1−a)B}/2 when the second matrix B is B=2·X−A according to the stored central matrix coefficient, the first matrix coefficient and the parameter a;
a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals; and
a setting means for setting X′ as a coefficient of hue conversion of said matrix means.
5. A hue adjusting method for adjusting the hue of a color image signal indicated by three primary color signals, said hue adjusting method comprising the steps of:
inputting a first matrix coefficient deciding a first matrix A and that is an end of the variable range of a matrix coefficient;
storing said first matrix coefficient;
inputting a second matrix coefficient deciding a second matrix B and that is the other end of the variable range of the matrix coefficient;
storing said second matrix coefficient;
inputting a parameter a for adjusting said matrix coefficients between the matrix coefficients of said first matrix A and said second matrix B;
storing said parameter a;
calculating X′ by X′={a·A+(1−a)B}/2 according to the stored first and second matrix coefficients and the parameter a; and
setting X′ as the coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three primary color signals.
6. The hue adjusting method according to
claim 5
, wherein a step for inputting said first matrix coefficient comprises the step of;
indicating information for urging to input said first matrix coefficient on an indicating means.
7. The hue adjusting method according to
claim 5
, wherein a step for inputting said parameter comprises the step of;
indicating information for urging to input said parameter on an indicating means.
8. A hue adjusting apparatus for adjusting the hue of a color image signal indicated by three primary color signals, said hue adjusting apparatus comprising:
a first matrix coefficient input means for inputting a first matrix coefficient deciding a first matrix A and that is an end of the variable range of a matrix coefficient;
a first matrix coefficient storing means for storing said first matrix coefficient;
a second matrix coefficient input means for inputting a second matrix coefficient deciding a second matrix B and that is the other end of the variable range of the matrix coefficient;
a second matrix coefficient storing means for storing said second matrix coefficient;
a parameter input means for inputting a parameter a for adjusting matrix coefficients between the matrix coefficients of said first matrix A and said second matrix B;
a parameter storing means for storing said parameter a; a calculating means for calculating X′ by X′={a·A+(1−a)B}/2 according to the stored first and second matrix coefficients and the parameter a; and
a setting means for setting X′ as a coefficient of a matrix converting means for converting the hue of a color image signal indicated by said three colors' signals.
US08/827,646 1996-04-13 1997-04-10 Hue adjusting method and hue adjusting device Expired - Lifetime US6424374B2 (en)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
JP08-115411 1996-04-13
JP11541196 1996-04-13
JP8-115411 1996-04-13
JP04666797A JP3603528B2 (en) 1996-04-13 1997-02-28 Hue adjustment method and hue adjustment device
JP9-046667 1997-02-28

Publications (2)

Publication Number Publication Date
US20010045984A1 true US20010045984A1 (en) 2001-11-29
US6424374B2 US6424374B2 (en) 2002-07-23

Family

ID=26386777

Family Applications (1)

Application Number Title Priority Date Filing Date
US08/827,646 Expired - Lifetime US6424374B2 (en) 1996-04-13 1997-04-10 Hue adjusting method and hue adjusting device

Country Status (2)

Country Link
US (1) US6424374B2 (en)
JP (1) JP3603528B2 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030160743A1 (en) * 2002-02-27 2003-08-28 Hitoshi Yasuda Color organic EL display device
US20040212783A1 (en) * 2001-03-26 2004-10-28 Osamu Wada Image processing system, information storage medium, and image processing method
US20170142275A1 (en) * 2015-07-29 2017-05-18 Kabushiki Kaisha Toshiba Image processing apparatus, image processing method and image forming apparatus

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3560521B2 (en) * 1999-12-09 2004-09-02 三菱電機株式会社 Image display device
JP3652194B2 (en) * 1999-12-09 2005-05-25 三菱電機株式会社 Image display device
JP4489262B2 (en) * 2000-07-28 2010-06-23 日立プラズマディスプレイ株式会社 Color reproduction correction circuit for color display and color display
DE10208285A1 (en) * 2002-02-26 2003-09-18 Koenig & Bauer Ag Process for correcting the image data of a camera system
JP4262059B2 (en) * 2003-11-14 2009-05-13 キヤノン株式会社 Color correction processing circuit for each hue
KR101127905B1 (en) * 2012-01-04 2012-03-22 국방과학연구소 An infrared camera applying optical and mechanical athermalization mechanisms

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR920001287B1 (en) * 1988-02-12 1992-02-10 니혼 호오소오 고오카이 Digital video signal processing apparatus
JP2569691B2 (en) 1988-02-16 1997-01-08 ソニー株式会社 Color difference signal forming circuit
JP2936080B2 (en) * 1991-04-05 1999-08-23 富士写真フイルム株式会社 Video signal color correction device
US5349381A (en) 1992-02-28 1994-09-20 Sony Corporation Video camera with aperture correction having reduced power consumption
JPH06119455A (en) * 1992-10-06 1994-04-28 Matsushita Electric Ind Co Ltd Automatic color compounding device
JP2856386B2 (en) * 1994-10-26 1999-02-10 松下電器産業株式会社 Color adjustment device and method

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040212783A1 (en) * 2001-03-26 2004-10-28 Osamu Wada Image processing system, information storage medium, and image processing method
US6899431B2 (en) 2001-03-26 2005-05-31 Seiko Epson Corporation Image processing system, projector, information storage medium, and image processing method
US20030160743A1 (en) * 2002-02-27 2003-08-28 Hitoshi Yasuda Color organic EL display device
US20170142275A1 (en) * 2015-07-29 2017-05-18 Kabushiki Kaisha Toshiba Image processing apparatus, image processing method and image forming apparatus
US9888140B2 (en) * 2015-07-29 2018-02-06 Kabushiki Kaisha Toshiba Image processing apparatus, image processing method and image forming apparatus

Also Published As

Publication number Publication date
JPH09331541A (en) 1997-12-22
JP3603528B2 (en) 2004-12-22
US6424374B2 (en) 2002-07-23

Similar Documents

Publication Publication Date Title
US6256062B1 (en) Color correction apparatus for matching colors in a signal output from a first image apparatus with those from a second image apparatus
US5223921A (en) White balance adjusting apparatus for automatically adjusting white balance on the basis of a color information signal obtained from an image-sensing device
US5414537A (en) Color image processing method and apparatus
JP2841301B2 (en) Color TV camera color correction device
JP4974586B2 (en) Microscope imaging device
EP0530028B1 (en) Video signal processing
US6424374B2 (en) Hue adjusting method and hue adjusting device
US4979225A (en) Method and apparatus for detecting corresponding regions between picture images
EP0516460B1 (en) Digital signal processing system for color camera apparatus
US5381174A (en) Method of and arrangement for correcting vignetting
JPH0738812A (en) Image pickup device equipped with image pickup element for multichannel output
EP0533488A2 (en) Image sensing apparatus having two image sensing portions and one signal processing portion and allowing improved white balance control
US20030223001A1 (en) Image capturing apparatus
EP0455220A2 (en) Image sensing apparatus
US5132783A (en) Device for adjusting white balance by peak detection and smoothing
GB2382510A (en) Digital dynamic convergence correction system
JPH11168761A (en) System and apparatus for digital camera adjustment
JPH0634509B2 (en) White balance correction circuit
JPH11113007A (en) Digital camera
JPH11262021A (en) Color balance adjustment device
JPH01241276A (en) Video camera
KR100238492B1 (en) Method for setting a gamma correction data adapted for user monitor device specification in digital camera system
CN101370149A (en) Video processing apparatus and method
JPH11168734A (en) Digital camera adjustment system
EP0386785B1 (en) Gain adjuster for color camera of sequential system

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ITAKURA, HIROYUKI;REEL/FRAME:010331/0894

Effective date: 19971001

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

FEPP Fee payment procedure

Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FPAY Fee payment

Year of fee payment: 8

FPAY Fee payment

Year of fee payment: 12