CA2156944A1 - Method and apparatus for a region-based approach to coding a sequence of video images - Google Patents
Method and apparatus for a region-based approach to coding a sequence of video imagesInfo
- Publication number
- CA2156944A1 CA2156944A1 CA002156944A CA2156944A CA2156944A1 CA 2156944 A1 CA2156944 A1 CA 2156944A1 CA 002156944 A CA002156944 A CA 002156944A CA 2156944 A CA2156944 A CA 2156944A CA 2156944 A1 CA2156944 A1 CA 2156944A1
- Authority
- CA
- Canada
- Prior art keywords
- region
- motion vector
- pel
- arbitrarily
- regions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/649—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding the transform being applied to non rectangular image segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/103—Selection of coding mode or of prediction mode
- H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/537—Motion estimation other than block-based
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/537—Motion estimation other than block-based
- H04N19/543—Motion estimation other than block-based using regions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
Abstract
The present invention discloses a method and encoder for coding sequences of digital images for transmission or storage. Frames in the sequence are segmented into multiple regions of arbitrary shape each of which has a corresponding motion vector relative to a previous decoded frame. In a preferred embodiment, a hierarchical multi resolution motion estimation and segmentation technique, which segments the frame into multiple blocks and which assigns a best motion vector to each block is used.
Blocks having the same or similar motion vector are then merged to form the arbitrarily-shaped regions. The shape of each region is coded, and a decision is made to code additional image data of each region in one of three modes. In a first inter-frame mode, a motion vector associated with a region is encoded. In a second inter-frame mode, a prediction error for the region is also encoded. In an intra-frame mode, the intensity of each picture element in the region is encoded. A region interior coder with frequency domain region-zeroing and space domain region-enforcing operations is employed for effectively coding the interior image data of the arbitrarily-shaped regions. The region interior coder uses an iterative technique based on the theory of successive projection onto convex sets (POCS) to find the best values for a group of selected transform coefficients. The coded information, including the shape of the region, the choice of the mode, and the motion vector and/or the region's interior image data, may then be transmitted to a decoder where the image can be reconstructed.
Blocks having the same or similar motion vector are then merged to form the arbitrarily-shaped regions. The shape of each region is coded, and a decision is made to code additional image data of each region in one of three modes. In a first inter-frame mode, a motion vector associated with a region is encoded. In a second inter-frame mode, a prediction error for the region is also encoded. In an intra-frame mode, the intensity of each picture element in the region is encoded. A region interior coder with frequency domain region-zeroing and space domain region-enforcing operations is employed for effectively coding the interior image data of the arbitrarily-shaped regions. The region interior coder uses an iterative technique based on the theory of successive projection onto convex sets (POCS) to find the best values for a group of selected transform coefficients. The coded information, including the shape of the region, the choice of the mode, and the motion vector and/or the region's interior image data, may then be transmitted to a decoder where the image can be reconstructed.
Description
-- 21569~4 ~k1~W AND APPARA~U~ FOR A 1~ 'A7-~S13D
APPP~ C ~ rl~ A ~rR OF V~ ~3S
FT~n OF ~ 1NV~W
The present invention relates generally to a method and apparatus for co~;ng a video sequence, and, in particular, to a region-oriented method and apparatus for ccoA;n.~ arbitrarily-shaped regions of video images.
~:~ OF ~R TNv~
Many appro~h~ to Pnco~;n~ a sequence of digital video images are known in the art. One classical approach is to divide each frame in the sP~l~nc-e into square blocks of ~r~cle~en~;neA size also known as macroblocks. Each macroblock is then assigned a motion vector relative to a previous A~c~A frame, where the m~tion vector represents the offset between the CU~L~1L macroblock and a block of pixels of the same size in a previous reconstructed frame that fonms a best match. The motion vector is transmitted to a cl~c~l~r which can then reconstruct the c~u~e~l~ frame based upon the previous A~co~A frame, the motion vector and a prediction error. Block-h~ A te~hn;ques, however, can lead to distortions such as blocking and mosquito effects in low bit-rate applications.
A m~re complex object-oriented, or region-oriented, approach ~ c~ arbitrarily-.~h~pPA regions instead of rectangular or square blocks. While block-ori~nteA coding techniques typically transmit tw~ parameter sets, specifically the motion and color of each block, an object-oriented approach requires that the shape of each region be transmitted as well in order to allow reconstruction of the image. For example, in M. Hotter, "Object-Oriented Analysis-Synthesis ~o~;ng Based On Moving Two-Dimensional Objects," Signal Proc~ing: l~age ~m~7nication, Vol. 2, pp. 409-428 (1990), an encoder `_ 215694i which ~n~o~q arbitrarily-qh~rP~ regions is pr~q~nte~, where objects are described by three parameter sets defining their motion, shape and color. A priority control determines in which of two modes the coded information will be sent h~.q~ upon the success or failure of the motion estimation t~hn;que for a particular region. me shape coding t~hn;que considered in the afor~ ,L;oned article a~r~ximates the shape of each region by a ccmbination of polygon and spline repr~c~nt~;on of the shape. U.S. Patent No. 5,295,201 also discloses an object-oriented ~n~o~r which includes an apparatus for d~roximating the shape of an arbitrarily-shaped region to a polygon. The vertices of the polygon are deter~;n~A, and the coordinate values of the vertices are calculated and transmitted.
One color co~;ng t~hn;que for use in object-oriented d~r~h~q is preq~nte~ in Gilge et al., "Co~;ng of Arbitrarily .~h~ Image .~"~-.~q Based On A Generalized Ortho~n~l Transform," SiDn~7 Proc~ ;n~: Image C~m~mication, Vol. 1, pp. 153-180 (1989). According to the t~hn;que disclosed in this article, an intensity function inside each region is d~roximated by a weighted sum of basis functions which are orthogonal with respect to the shape of the region to be coded. While this technique may be theoretically useful, it is not practicable for implPmPnt~tion in a real-time system.
Due to the pot~nt;~l adv~nt~es of an object-oriented d~oach, there exists a need for an object-oriented ~nC~r which provides powerful .qchem~q for sf~ L;~ an imlage into arbitrarily-.qh~P~ regions, each of which has a correqrnnA;ng motion vector, and for representing the s~ L c~nt~t in a m~n~er which can be readily impl~m~ntP~ for use in real-time. It is also desirable to have an P~nc~r which can encode a generic scene or sequence of imlages, the contPnt of which is not known 215~
beforehand, in contrast to the requi~"~r,L.~ of the prior art. It is further desirable to provide an encoder which permits additional functionalities, such as tracking objects moving from one area of a scene to another between images in a sequence.
SUNNaRY OF -~ TNVRNTTnN
The present invention discloses an encoder for encoding a sequence of video frames. The ~n~o~er ~ ises a se~"~L~tion unit which ~"æ~,ls a c~Lr~lL.
frame in the video sequence into a plurality of arbitrarily-.~h~e~ regions, where each of the plurality of arbitrarily-sh~r~ regions is assigned a motion vector.
The encoder also has a ~co~P~ frame ll~l~r~ for storing a previously ~co~P~ frame in the video sequence and a prediction unit c~nnected to the .~"~.,l~tion unit and the ~C~f~ frame II~I~L~ for predicting image data of the CULr~1L frame based upon a previously decoded frame and h~ upon the motion vector assigned to one of the plurality of arbitrarily-.~h~e~ regions. The ~o~r further ~ ises a region shape co~;ng unit for encoding the shape of each of the arbitrarily-shaped regions.
A mode decision unit determines in which one of a plurality of modes image data from each of the plurality of arbitrarily-shaped regions is to be encoded. The plurality of modes ~I~Lises a first inter-frame mode in which a motion vector associated with one of the plurality of arbitrarily-shaped regions is ~nco~ and a second inter-frame mode in which a motion vector and a motion c~l~ sated prediction error associated with one of the plurality of arbitrarily-Rh~reA regions are ~n~o~. A
third mode is an intra-frame mode in which the intensity of each pel in one of the plurality of arbitrarily-shaped regions is ~nco~. A mode coding unit then encodes the '- 21S6944 mode in which each of the plurality of arbitrarily-shaped regions is to be encoded.
The ~nCo~er includes a motion ro~ unit for encoding motion vectors associated with the plurality of arbitrarily-.~hA~e~ regions. In addition, the encoder c~,~ises a region interior coder which ~nco~ a motion c~l~ sated prediction error associated with one of the plurality of arbitrarily-~hA~F~ regions if the region is to be ~nco~e~ in the second inter-frame mode, and which ~nro~s the intensity of each pel in one of the plurality of arbitrarily-shaped regions if the region is to be ~nC~P~ in the intra-frame mode.
A buffer serves as an interface for transmitting ~nco~ infonmation between the ~nco~r and a tr~n~ Rion or storage medium. Finally, a rate controller receives signals from the buffer. The rate controller then sends control signals to the ~J"~ tion unit, the mode decision unit, the region interior unit and a frame skip unit in re~rDn~e to the signals received from the buffer.
Other features and adv~nta~es of the present invention will be readily apparent by reference to the following detailed description and ~c~ ing drawings.
n~ -nN OF TU~ nRA~gE
FIG. 1 is a block diagram of an ~n~o~er for perfonming motion estimation, s~J"~r,l~tion and coding of a video sequence with regions of arbitrary shape and size.
FIG. 2 is a flow chart showing steps of the image co~ method according to the principles of the present invention FIG. 3 is an exemplary graph for use in determ;n;ng the mode in which the coded image data is to be sent.
FIG. 4 depicts a preferred embo~;m~nt of a motion estimation and s~"PrlL~tion unit for use in the encc~er in FIG. 1.
215~94~
FIG. 5 is a simplified block diagram showing one embo~;mPnt of a joint motion estimation and se[~"P~ ttion unit for generating arbitrarily-shaped regions with correspon~;ng motion vectors.
FIG. 6 is a flow chart showing the steps for selecting a best motion vector for a square block.
FIG. 7 shows a simplified block diagram of a region interior coder for use in the encoder in FIG. 1.
FIG. 8 illustrates an exemplary arbitrarily-shaped region circumscribed by a rectangular block.
DETPTLED ~-v,v-ll~N OF TR~ ~NT TNV~NnT~N
FIG. 1 is a block diagram of an encoder 100 for performing motion estimation, s~J,,~.,I~tion and coding of a video sequence with regions of arbitrary shape and size.
The encoder 100 has a buffer 105 which serves as an interface between the ~nco~r 100 and a tr~n.~m;~.~ion or storage medium. The buffer 105 sends signals to a rate controller 108 via line 106 indic~tt;ng the ftlllnP~s of the buffer 105. In r~qro~.~e to the signals received from the buffer 105, the rate controller 108 controls the rate of flow of information from the ~n~o~r 100 to the - tr~nqm;~ion or storage medium by providing control signals to other c~pnn~tlt~ of the ~nCo~er 100 via line 109. The ~nco~r 100 further has a frame skip unit 110 for reeeiving a sequence of digital video frames one at a time from an input line 112 and for determ;n;ng whether a particular frame in the sequenee should be skipped.
A .~e~"~ tion unit 120 is ineluded in the encoder 100 for s~ ;ng a frame into a plurality of regions of arbitrary shape and size. The .~"~.,(~tion unit 120 also determines a motion veetor associated with eaeh region for predieting the cu~LeIlL frame from a previous deeoded frame whieh may be stored in a ~eco~ frame memory unit 125.
Details of the s~3mPnt~tion unit 120 are provided below.
2156~1 me ~nco~r 100 comprises a prediction unit 130 for predicting a region of the c~LLenL frame based upon a motion vector received from the ~3m~nt~tion unit 120 and a previous decoded frame received from the ~co~ frame memory unit 125.
A region shape co~;n~ unit 140 is also included in the ~n~er 100 for ~nCo~;n~ the shape of each region of the C~LL~1L frame. The encoder 100 further ~ ises a mode decision unit 150 for deciding in which of three modes image data of a region will be coded and transmitted. These three modes are ~ ¢sed in greater detail below. A mode and motion vector coding unit 160 is included in the ~co~er 100 for encoding the particular mode in which the information about a region will be sent and for ~nCo~;n~ the motion vector associated with the region. Although shown as a single unit in FIG. 1, the mode and motion unit 160 may c~l~ise separate units for performing the functions of ~nco~;n~ the mode and encoding the motion vector. S;~ rly, a region interior coder 170, described more fully below, is included for ~nco~;ng either the prediction error or the intensity value of a particular region dep~n~;n~ upon the mode in which the region is to be ~n~o~f~ The ~nco~r 100 also has a multiplexer 180 for passing the enco~ infonmation from the various co~ units 140, 160 and 170 to the buffer 105 in a predefined order. Finally, a control unit 101 is ~o.~r~e~ted to the other units so as to control the interaction and flow of infonmation between the other units of the ~n~ r 100. The control unit 101 may be, for example, a suitably ~r~yL~ mi~Lo~o~essor or other suitably configured hardware, as well as implemented in software.
FIG. 2 is a flow chart showing steps of the image coding method according to the principles of the present invention. As shown in step 205, a c~LeIlL frame in a sequence of frames is received by the frame skip unit 110 via line 112. I~e frame skip unit 110 determines whether the cu~e~l~ frame should be skipped or not, as shown in 210. me decision whether to skip the c~,enL frame is 5 deterrn;n~, for exa~le, by a control signal received on line 111 from the rate controller 108, which receives and processes information from the buffer 105 via line 106 indicating the flllln.o~ of the buffer. If the c~e~l~
frame is skipped, then the next frame in the sequence is 10 received by ret~ to step 205.
If the c~enL frame is not sk;E~l, then, as indicated by step 215, the CU~L`~1L. frame is s~J"~rlLed or divided into a plurality of regions each having a motion vector associated with it. In particular, the regions may 15 be of a shape and size which is not known a priori, thereby resulting in arbitrarily-~h~l regions. The c~lL. frame is divided into regions by grouping or merging together adjacent pels having the same or a s;m;l~r ;rlt.on~ity or by grouping together adjacent pels 20 having the same or a s;m;l~r motion vector. The m~ r of regions into which the frame is divided may be limited or controlled by a control signal on line 121 received from the rate controller 108. In order to determine the motion vector for each region, the se~ "l~tion unit 120 receives 25 the previous ~-oc~l frame from the ~ r unit 125 as an input.
Ihe output of the ~"~ tion unit 120 includes a description indicating the region to which each pel in the frame belongs and an array indic~t;ng the motion vector 30 assigned to each region. Once a frame has been s~rr~nted into a plurality of regions, and a motion vector has been assigned to each region, each region is processed until all the regions have been proc~ as indicated by step 220. If all the regions in the cuLLell~ frame have been 35 processed and coded, then the next frame is received in 2156g4~
step 205. If all the regions in the cu~r~lL frame have not been processed and coded, then the process continues to process and code the next region as shown in step 225.
As shown in step 230, the shape of each region is coded by the region shape co~;ng unit 140. For this purpose, the region shape co~;ng unit 140 receives from the .~"~ tion unit 120 a description indicating the region to which each pel in the frame belongs. This description may be, for example, a simple array of region l~hPl.~, a binary array of boundaries, a series of chain codes, a tree, or any other suitable se~"~lL~tion representation. The shape coding unit 140 ~n~o~ this description according to any one of a m ~hPr of lossless co~ t~chn;ques, such as arithmetic co~;ng. It should be noted that the step of enc~;ng the shape of the arbitrarily-~h~r~ regions may be performed prior~to step 220.
Next, as shown in step 235, a predicted region is ccmputed h~ upon the motion vector assigned to the cul,~lL region. The predicted region is computed in the prediction unit 130 which receives from the ll~m~ unit 125 the previous ~co~ frame. The description indicating the region to which each pel in the frame belongs and the array indicating the motion vector assigned to each region are sent from the se~ tion unit 120 to the prediction unit 130 as well.
In step 240, the mode decision unit 150 determines in which one of at least three modes the image data of each region is to be coded and transmitted. The first of the three possible modes is an inter-frame motion ~ e~lsated mode in which the motion vector associated with a particular region is ~nco~. In this first mode, no prediction error is encoded. A ~co~er then would recover the image of the region by using a previous decoded frame and the motion vector. The second mode is also inter-~15S9~4 frame motion coT~Pn~ted. A motion ~AI~Pn~ted predictionerror for each pel in the region, however, is also encoded along with the motion vector in order to improve the quality of the image intensity in the region. The prediction error represents the difference between the C~LL~1L image ~-P'JIl~PllL and the motion compensated segment obtA;ne~ from the prediction unit 130. The third mode is an intra-frame mode in which the image data in the region is treated indepPn~Pntly of previously decoded or reconstructed frames. From the st~n~roint of m;n;m;~;ng the amount of coded information generated by the P~co~er 100, the first mode is preferred bPc~ e it is likely to require the least ~m~l~nt of infonmation to be coded. Gn the other hand, the inter-frame motion ~,4~1sated modes may result in excessive noise or may fail to accurately predict a region in the CULL~1L frame under certain circumstA~P.~. In such situations, it is necessary to Pnco~e the intensity, in other words, the ll~;n~nce and chro~;n~n~P, of each pel in the region.
In one embo~;ment, the decision to code the image data in a particular mode may ~pren~ for example, upon the calculated values of the following two normalized sums of absolute differences for the region under consideration:
NSADI ~ ~ i~R
NSPI~ R Ieil where N is the total m ~hPr of pels in the region, i is a given pel in the region, and R is the set of all the pels in the region. Also, in the above equations, Ii is the intensity of the particular pel i, m is the mean value of the intensity of all the pels in the region, and e designates the motion co~pPn~ted prediction error associated with the pel i. For the above purposes, the mode decision unit 150 receives the same seg~Pnt~tion and 215G~ ~4 motion information that is sent to the prediction unit 130. In addition, the mode decision unit 150 receives the CUUL~1~ frame from the frame skip unit 110 and a predicted region from the prediction unit 130.
FIG. 3 is an exemplary graph plotting the values of NSPI~ versus NSADI for use in determ;n;ng the mode in which the coded image data is to be sent. The graph is divided by the solid lines into three sections corr~rDnA;ng to the three modes. When the value of NSADp is less than a threshold value c, then the image data is coded and transmitted in the first mode. When the value of N5ADp e~cPe~ the threshold value c and is less than the value of NSADI by at least a threshold value of b, then the image data is coded and transmitted in the s~con~
mode. Otherwise, the image data is coded and transmitted in the third mode. me value of b is, therefore, chosen to distribute the coded regions between the intra-frame mode and the sec~n~ inter-frame mode. The value of c is chosen to control the ~l~hPr of regions which are to be co~A in the first inter-frame motion c~,~ensated mode.
me values of b and c depend upon the noise present in the frame and the status of the rate control unit 108. For this purpose, for example, a control signal may be received on line 151 from the rate controller 108. Once the mode has been deter~;ne~, the choice of the mode is ~nco~A, for example, in binary code by the mode coding unit 160, as indicated in step 245.
It should be noted that there may be other modes different from or in addition to the three modes discussed above. Also, each of the three modes discussed above may permit or require other parameter choices to be made, thereby resulting in secon~ry modes. For example, a quantization step size, ~ e~ further below in the c~nt~xt of the region interior coder 170, may be 21S69~
adjustable. me setting of such parameters or secon~ry modes would also be ~n~o~ by the mode decision unit.
me next step depends upon whether the intra-frame coding mode was selected for the region under consideration, as shown by 250. If the intra-frame coding mode was not selected, then the motion vector of the region is also coded by the co~;n~ unit 160 as indicated in step 255. me next step ~r~n~ upon whether the first inter-frame motion ~m~n~ted mode was selected as shown in 260. If the first inter-frame motion compensated mode was not selected, then, as indicated in step 265, the image data for the region's interior is coded by the region interior coder 170. As expl~;n~ above, the interior coding for the inter-frame motion ~l4~1sated mode using prediction errors requires that a prediction error for each pel in the region under consideration be co~ and transmitted. In contrast, the interior coding for the intra-frame mode requires that the ;nt~nRity Of each pel in the region be coded and transmitted. Details of the region interior coder 170 and the step of co~;ng the image data for the region's interior are explained in greater detail below.
me co~ infonmation for the region under consideration, including the shape of the region, the choice of mode, and the motion vector and/or the image data for the region's interior, is then transmitted via the multiplexer 180 and the buffer 105 to a decoder via a tr~n.~ sion medium as indicated in step 270.
Alternatively, the coded information may be stored in a suitable storage medium, such as a CD-ROM, for subsequent decoding at a decoder. The steps 230 through 270 are performed for each region until all the regions in the current frame have been coded and transmitted. The ~co~r can then use the coded information to reconstruct the current frame.
215S9~4 FIG. 4 depicts a preferred ~mh~;m~nt of the se~ tion unit 120. The unit 120 reeeives the c~e~lL
frame and the previous decoded frame from the frame skip unit 110 and the ~co~ frame memory unit 125 via lines 5 401 and 402, respectively. The received e~LellL frame and previous ~eo~ frame data is routed, in r~p~ e to a first eontrol signal on line 406, to an intensity ~gmPnt~tion unit 420 via a switch 405. Alternatively, the received C~U~1L frame and previous decoded frame data is routed, in response to a second control signal on line 406, to a joint motion estimation and s~"~,ll~tion unit 450 via the switeh 405. The first and seeond eontrol signals are generated by the eontrol unit 101. The int~n~ity s~J"~"~tion unit 420 divides the C~UL~1L frame 15 into a plurality of arbitrarily-shaped intensity regions by grouping together pels that have the same or s;m;l~r ;nt~ ity features. The initial frame in a sequenee of images, for example, is always sent to the ;ntPn.~ity ~ "~,~ ;on unit 420 and coded in the intra-frame mode 20 hPC~ e motion est;m~;on and ~g~Pnt~tion is not applieable when there is no previous deeoded frame. A
deseription indieating the region to whieh eaeh pel in the frame belongs is sent via lead 422 to a switch 465. The switeh 465 is also eontrolled by a eontrol signal on line 25 466 that allows the infonmation indieating the region to whieh eaeh pel in the frame belongs to pass from the intensity .~"~ tion 420 or the joint motion estimation unit 450 to the other units in the eneoder 100 depending upon whieh unit, 450 or 420, reeeived the CUkre~l~ frame.
30 In other words, the eontrol signal on line 466 is synehronized with the control signal on line 406. The infonmation indicating the region to which each pel in the frame belongs may then be sent to the region shape coding unit 140, the prediction unit 130 and the mcde deeision 35 unit 150.
Other frames may also be selected and s~"~~ by the ;ntPn~ity .~"~ t;on unit 420 and sent in the intra-frame mode. If, for example, a scene ~h~n~e occurs which results in objects or image areas that did not appear in 5 previous frames, a decision is made to send the frame to the ;nt~n.~ity ~ tion unit 420. Similarly, in order to ease editing of the sequence at some later time, periodic frames, such as one frame every few secon~, may be S~J~ by the ;nt~n.~ity ~e~l~rl~tion unit 420. In addition, a particular frame may be sf~"~,l~ed by the ;nt~ ity .~"~---~tion unit 420 to resynchronize the ~nco~r 100 and the decoder.
The inter-frame modes can also be performed by the use of the ;nt~nqity .~ tion unit 420. In such a 15 situation, a motion vector is deter~'n~ for each region in a region based matching unit 425 using a region matching t~hn;que s;m;l~r to well-known block matching tP~hniques employed with rectangular or square regions.
In other words, each region is com~red to the previous 20 ~CO~ frame, and a best match is found which ~;n;m;zes the total prediction error for the region. A motion vector then is used to indicate the relative difference in position between the c~L~ellL region and the matched region in the previous decoded frame. The motion vectors are 25 then sent, via line 427, to a switch 475. The switch 475 is also controlled by a control signal on line 476 that allows the motion vector assigned to each region to pass from the ;nt~n.~ity s~yl~ ation unit 420 or the joint motion estimation and se~"~~ tion unit 450 to the other 30 ~ X~ nt~ in the ~nco~r 100 ~Pn~;n~ upon which unit, 450 or 420, received the C~Lr~1~ frame. The control signal on line 476 iS, therefore, also synchronized with the control signals on lines 406 and 466. The control signals on lines 466 and 476 are also sent from the 35 control unit 101 and allow image data to pass from the '- 2156~44 ;~t~n.~ity s~"~ tion unit 420 and the joint motion estimation and segmP~tAtion unit 450 to other units in the encoder 100 as expl Ai n~ above. The information indicating the motion vector associated with each region in the frame is sent to the prediction unit 130 and the mode decision unit 150.
Typically, however, frames other than the initial frame in the sequence are s~"~ by the joint motion estimation and segm~ntAtion unit 450. In a preferred e~ho~-m~nt, the joint motion estimation and se~"P~Ation unit 450 divides a CULL~1L frame into a plurality of regions and assigns a motion vector to each region according to the method and apparatus described in U.S.
Patent Application No. 08/271,308, entitled "Method and Apparatus For Motion Field Estimation, Sey"~Ation and Coding," which is assigned to the assignee of the present invention and which is in~o,~orated by reference herein.
The motion estimation and s~"P~IlAtion t~hn;que described in the afo~ Lioned application uses a hierarchical approach in which a frame is divided into a plurality of regions, and in which a motion vector updating routine is performed with respect to multiple levels of smaller and smaller regions. In a preferred embo~;~Pnt, the motion vector l~At;ng routine is performed with respect to smaller and smaller square blocks of predeter~;n~ size.
The motion vector updating routine ll~A~ the motion vector of each smaller block by assigning to it a best motion vector selected from among an initial motion vector assigned to the smaller block, motion vectors of neighboring blocks, and an updated or matched motion vector obtA;n~ by performing a block matching technique for the smaller block. The initial motion vector assigned to blocks in the first s~"P.,~Ation level is typically zero, whereas the initial motion vector assigned to each block in subsequent seylllenLation levels is the motion ` '_ 2156-944 vector of its parent block from which it was obt~;ne~.
The best motion vector for each block is selected according to a priority scheme and a predetenm;n~
threshold value.
FIG. 5 is a simplified block diagram showing the joint motion estimation and .~P~I~r~-~tion unit 450 that is used for determ;n;~ the best motion vector associated with each square block and for generating the arbitrarily-shaped regions having corresp~n~;n~ motion vectors. A
motion vector c~n~ te unit 510 computes and stores motion vectors that serve as candidates for the best motion vector associated with a particular square block.
These motion vectors include the initial motion vector (PV) assigned to the block, the motion vectors (V0 through V7) of up to eight neighboring blocks, and the updated motion vector (NV) ob~;n~ by performing a block matching ~Pchn;que for the block. A prediction error computation unit 520 computes and stores the motion co~n~ted prediction or matching error corr~rnn~ to each of the candidate motion vectors. A ~;n;~ ~ prediction error unit 530 determines the smallest prediction error (MIN) from ~ y the prediction errors computed by the prediction error unit 520. The motion vector candidates, the corr~.~r~n~;ng prediction errors, and the ~;n;~ predicted error then are sent to a best motion vector selection unit 540 which selects the best motion vector for the block under consideration.
A basic idea h~h;n~ the selection of the best motion vector for each block, according to the technique disclosed in the afoL~,l~.lL;o~ patent application No.
08/271,308, is to substitute the motion vector of one of the neighboring blocks or an updated motion vector obt~;ne~ by the block matching t~ch~;que for the cu~e~lL
block only if such a substitution yields a significant impro~ lL. in the matching or prediction error for that ' ~ 21~6g~
block. Furthermore, there is a preference for the motion vectors of the neighboring blocks relative to the updated motion vector of the cu~re~lL block ob~;ne~ from the block matching te~hn;que.
FIG. 6 shows an exemplary process, starting in step 600, by which the best motion vector is selected by the best motion vector selection unit 540 for a block which has, for example, eight neighboring blocks. In FIG. 6, PE, and E0 through E7 refer, respectively, to the prediction errors that result from assigning the motion vectors PV, and V0 through V7 to the block under consideration. In step 605, it is determin~ whether the absolute difference between the prediction error (PE) and the ~mallest prediction error (MIN) is less than a 15 predeterm;ne~ threshold (THR). If the absolute value in step 605 is less than the threshold value, this determ;n~t;on serves as an indication that substituting any one of the motion vectors V0 through V7, or NV, would not result in a significant im~ro~ lL in the prediction 20 error. As indicated by step 610, the motion vector PV is, therefore, selected as the best motion vector for the block under consideration.
If, however, the absolute difference determ;n~ in step 605 is not less than THR, then the process continues 25 with step 615. In each of steps 615, 625, 635, 645, 655, 665, 675 and 685, it is determ;n~ whether the absolute difference between MrN and a respective one of the prediction errors E0 through E7 is less than THR. If the absolute difference in a particular step is less than THR, 30 then the motion vector corre~r~ g to the particular prediction error is selected as the best motion vector as indicated by steps 620, 630, 640, 650, 660, 670, 680 and 690, respectively. The steps 615, 625, 635, 645, 655, 665, 675 and 685, are performed se~l~nt;~lly until a best 35 motion vector is selected. If none of the afo~e,.~lLioned '~ 21S6~44 absolute differenceæ is less than THR, then, as shown by step 695, this determination indicates that NV should be used as the best motion vector for the bloek under consideration.
Once a best motion vector is obta;n~ for each square bloek of the e~LLenL frame, eaeh square bloek is segmented into four smaller square bloeks of equal size, and the motion veetor updating routine is repeated for each smaller block until a stop condition is rea~h~. The stop condition may be, for example, a lower limit on the size of the blocks for which the motion vector updating routine is performed or a predeterm;n~ m ~h~r of eyeles of the motion vector updating routine.
When the stop condition is reAehPA, and a best motion vector has been assigned to each square bloek in the eu~.L .q~gmPntAtion level, a r~ "~ tion unit 550 performs a merging proeess which merges adjacent square regions having the same or s;m;lAr motion vectors to form a set of non-overlapping merged regions, each of whieh may be arbitrarily-shaped and have different ~imen~ions. It should be understood that, in some applications, some or all of the arbitrarily-shaped regions that result from the merging proeess may be square bloeks having the same or different ~;m~n.~ions.
me re.~ t;on unit 550 also assigns to eaeh pel in the euuf~lL frame a region label indieating to which region it belongs. The region lAh~l~ are then sent via line 452 to the switeh 465, and the motion veetors asso~;At~ with each region are sent via line 457 to the switch 475. As eXplA;n~A above, eontrol signals on lines 466 and 476 allow the region l~hel~ and motion veetors to pass to other ~l4~l,~nt~ of the eneoder 100 for further ~o~essing and ~nco~;n~.
In a preferred emb~;ment of the present invention, the region interior eoder 170 of the ~neo~er 100 _ 215~9'14 implem~nt.~, for example, the method described in U.S.
Patent Application No. 08/138,295, entitled "Block Transform Coder For Arbitrarily Shaped Image S~"~-,~s,~
which is assigned to the assignee of the present invention, and which is in~o~o~ated by reference herein.
The method described in Application No. 08/138,295 uses block transfonms with frequency ~ ;n region-zeroing and space ~m~;n region-enforcing operations for effectively co~;ng the image data of arbitrarily-shaped regions. The method uses an iterative t~rhn;que based on the theory of sll~ce~ive projection onto convex sets (POCS) to find the best values for a group of selected transform coefficients.
A simplified block diagram of an exemplary region interior coder 170 for implem~t;ng the afore~ Lioned iterative POCS t~chn;que is depicted in FIG. 7. An image circumscription unit 710 receives the image data of an arbitrarily-.~h~r~ region 802 from the mode decision unit 150. As shown in FIG. 8, a rectangular region block 801 is circumscribed around the arbitrarily-shaped region 802.
An original internal pel set 803 which lies within the arbitrarily-shaped region 802 is thereby defined.
S;mil~rly, an original ext~r~l pel set 804 which lies outside the arbitrarily-shAr~A region 802 and within the region block 801 is thereby defined. An extrapolator 720 extrapolates the pel values of the ;nter~l pel set 803 to initl~ e the pel values of the external pel set 804.
Examples of extrapolation methods include pel repetition, mirroring and ll~llological dilations.
Okher ~,5~ nt.~ of the transfonm coder 170 shown in FIG. 7 perform a POCS iteration loop on the image data.
It should be understood that the image data upon which the POCS iteration loop is perfonmed ~r~n~-~ upon the mode in which the image data of the region is to be coded, and transmitted or stored. In the second inter-frame motion ~ 215~9~1 c~.4~lsated mode in which the prediction errors are coded, the image data coded by the POCS iteration locp is the prediction error associated with the pels in the region under consideration. If, on the other hand, the intra-frame mode is to be used, then the image data coded by thePOCS iteration loop includes the ;nt~n.~ity of each pel in the region under consideration.
The POCS iteration loop begins with the application of a forward transform, such as the discrete cosine transform (DCT), by a forward transform unit 730 to generate transform coefficients. A transform coefficient set (TCS) is generated by a TCS generator 740 which selects and retains transform coefficients having high energy according to the energy com~ction property of transform coefficients. The re~;n;n~ transform coefficients are set to zero. The number of selected coefficients is deter~;ne~ by the rate controller 108 which establ;~h~ a threshold energy h~ , for example, upon the size of the arbitrarily-.~h~ region. Next, an inverse transform unit 750 perfonms an inverse transform on the TCS to generate a cc~ t~ region block having c~.~uted pel values. An interior pel repl~ ~n~ unit 760 replaces those ccmputed pels corr~r~n~;ng to the internal pel set with original pel values to form a modified ccmputed r~;on block. A forward transform is ~g~;n perfor~f~ on the modified computed region block (MCRB), and a new transform coefficient set is generated.
If a particular transform coefficient set (TCS) represents optimal transform coefficients (OTC), then the TCS is qll~nt;~ and coded using, for example, variable length coding. The step size of quantization may be deter~;nf~, for example, by a signal on line 171 received from the rate controller 108. The coded, qll~nt;~ed values of the optimal transfonm coefficients are then sent as outputs to the multiplexer 180. A particular TCS may 21569~4 represent optimal transform coefficients when a predeterm;n~ number of POCS loop iterations is reached, when the exterior pels do not ~h~n~e, or when the mean squared difference of the exterior pels between iterations is within a pre-defined threshold. If the TCS does not represent the optimal transform coefficients, then the POCS loop is reiterated until optimal transform coefficients are obt~;ne~. At the decoded frame memory unit 125 or at a ~co~r, an inverse quantization process and inverse transform process is applied to reconstruct the image data for the region prior to storing the decoded region.
me region-based video ~ncq~r 100 of the present invention is particularly adv~nt~ous for systems using low bit-rates. Com~ m;cation applications include, for example, video tele~l~u~l~, personal c~m~ln;cation, mult;mP~;~, education, entert~ nt and remote sensing where the tr~n~m;s~ion or storage c~c;ty of the system is limited.
Although the present invention has been described with reference to .~c;fic ~mh~;ment.~ it will be appreciated that other a~ly~.~r~L~ within the spirit and scope of present invention will be readily apparent to persons of ordinary skill in the art. The present invention is, therefore, limited only by the appended cl~;m.~.
APPP~ C ~ rl~ A ~rR OF V~ ~3S
FT~n OF ~ 1NV~W
The present invention relates generally to a method and apparatus for co~;ng a video sequence, and, in particular, to a region-oriented method and apparatus for ccoA;n.~ arbitrarily-shaped regions of video images.
~:~ OF ~R TNv~
Many appro~h~ to Pnco~;n~ a sequence of digital video images are known in the art. One classical approach is to divide each frame in the sP~l~nc-e into square blocks of ~r~cle~en~;neA size also known as macroblocks. Each macroblock is then assigned a motion vector relative to a previous A~c~A frame, where the m~tion vector represents the offset between the CU~L~1L macroblock and a block of pixels of the same size in a previous reconstructed frame that fonms a best match. The motion vector is transmitted to a cl~c~l~r which can then reconstruct the c~u~e~l~ frame based upon the previous A~co~A frame, the motion vector and a prediction error. Block-h~ A te~hn;ques, however, can lead to distortions such as blocking and mosquito effects in low bit-rate applications.
A m~re complex object-oriented, or region-oriented, approach ~ c~ arbitrarily-.~h~pPA regions instead of rectangular or square blocks. While block-ori~nteA coding techniques typically transmit tw~ parameter sets, specifically the motion and color of each block, an object-oriented approach requires that the shape of each region be transmitted as well in order to allow reconstruction of the image. For example, in M. Hotter, "Object-Oriented Analysis-Synthesis ~o~;ng Based On Moving Two-Dimensional Objects," Signal Proc~ing: l~age ~m~7nication, Vol. 2, pp. 409-428 (1990), an encoder `_ 215694i which ~n~o~q arbitrarily-qh~rP~ regions is pr~q~nte~, where objects are described by three parameter sets defining their motion, shape and color. A priority control determines in which of two modes the coded information will be sent h~.q~ upon the success or failure of the motion estimation t~hn;que for a particular region. me shape coding t~hn;que considered in the afor~ ,L;oned article a~r~ximates the shape of each region by a ccmbination of polygon and spline repr~c~nt~;on of the shape. U.S. Patent No. 5,295,201 also discloses an object-oriented ~n~o~r which includes an apparatus for d~roximating the shape of an arbitrarily-shaped region to a polygon. The vertices of the polygon are deter~;n~A, and the coordinate values of the vertices are calculated and transmitted.
One color co~;ng t~hn;que for use in object-oriented d~r~h~q is preq~nte~ in Gilge et al., "Co~;ng of Arbitrarily .~h~ Image .~"~-.~q Based On A Generalized Ortho~n~l Transform," SiDn~7 Proc~ ;n~: Image C~m~mication, Vol. 1, pp. 153-180 (1989). According to the t~hn;que disclosed in this article, an intensity function inside each region is d~roximated by a weighted sum of basis functions which are orthogonal with respect to the shape of the region to be coded. While this technique may be theoretically useful, it is not practicable for implPmPnt~tion in a real-time system.
Due to the pot~nt;~l adv~nt~es of an object-oriented d~oach, there exists a need for an object-oriented ~nC~r which provides powerful .qchem~q for sf~ L;~ an imlage into arbitrarily-.qh~P~ regions, each of which has a correqrnnA;ng motion vector, and for representing the s~ L c~nt~t in a m~n~er which can be readily impl~m~ntP~ for use in real-time. It is also desirable to have an P~nc~r which can encode a generic scene or sequence of imlages, the contPnt of which is not known 215~
beforehand, in contrast to the requi~"~r,L.~ of the prior art. It is further desirable to provide an encoder which permits additional functionalities, such as tracking objects moving from one area of a scene to another between images in a sequence.
SUNNaRY OF -~ TNVRNTTnN
The present invention discloses an encoder for encoding a sequence of video frames. The ~n~o~er ~ ises a se~"~L~tion unit which ~"æ~,ls a c~Lr~lL.
frame in the video sequence into a plurality of arbitrarily-.~h~e~ regions, where each of the plurality of arbitrarily-sh~r~ regions is assigned a motion vector.
The encoder also has a ~co~P~ frame ll~l~r~ for storing a previously ~co~P~ frame in the video sequence and a prediction unit c~nnected to the .~"~.,l~tion unit and the ~C~f~ frame II~I~L~ for predicting image data of the CULr~1L frame based upon a previously decoded frame and h~ upon the motion vector assigned to one of the plurality of arbitrarily-.~h~e~ regions. The ~o~r further ~ ises a region shape co~;ng unit for encoding the shape of each of the arbitrarily-shaped regions.
A mode decision unit determines in which one of a plurality of modes image data from each of the plurality of arbitrarily-shaped regions is to be encoded. The plurality of modes ~I~Lises a first inter-frame mode in which a motion vector associated with one of the plurality of arbitrarily-shaped regions is ~nco~ and a second inter-frame mode in which a motion vector and a motion c~l~ sated prediction error associated with one of the plurality of arbitrarily-Rh~reA regions are ~n~o~. A
third mode is an intra-frame mode in which the intensity of each pel in one of the plurality of arbitrarily-shaped regions is ~nco~. A mode coding unit then encodes the '- 21S6944 mode in which each of the plurality of arbitrarily-shaped regions is to be encoded.
The ~nCo~er includes a motion ro~ unit for encoding motion vectors associated with the plurality of arbitrarily-.~hA~e~ regions. In addition, the encoder c~,~ises a region interior coder which ~nco~ a motion c~l~ sated prediction error associated with one of the plurality of arbitrarily-~hA~F~ regions if the region is to be ~nco~e~ in the second inter-frame mode, and which ~nro~s the intensity of each pel in one of the plurality of arbitrarily-shaped regions if the region is to be ~nC~P~ in the intra-frame mode.
A buffer serves as an interface for transmitting ~nco~ infonmation between the ~nco~r and a tr~n~ Rion or storage medium. Finally, a rate controller receives signals from the buffer. The rate controller then sends control signals to the ~J"~ tion unit, the mode decision unit, the region interior unit and a frame skip unit in re~rDn~e to the signals received from the buffer.
Other features and adv~nta~es of the present invention will be readily apparent by reference to the following detailed description and ~c~ ing drawings.
n~ -nN OF TU~ nRA~gE
FIG. 1 is a block diagram of an ~n~o~er for perfonming motion estimation, s~J"~r,l~tion and coding of a video sequence with regions of arbitrary shape and size.
FIG. 2 is a flow chart showing steps of the image co~ method according to the principles of the present invention FIG. 3 is an exemplary graph for use in determ;n;ng the mode in which the coded image data is to be sent.
FIG. 4 depicts a preferred embo~;m~nt of a motion estimation and s~"PrlL~tion unit for use in the encc~er in FIG. 1.
215~94~
FIG. 5 is a simplified block diagram showing one embo~;mPnt of a joint motion estimation and se[~"P~ ttion unit for generating arbitrarily-shaped regions with correspon~;ng motion vectors.
FIG. 6 is a flow chart showing the steps for selecting a best motion vector for a square block.
FIG. 7 shows a simplified block diagram of a region interior coder for use in the encoder in FIG. 1.
FIG. 8 illustrates an exemplary arbitrarily-shaped region circumscribed by a rectangular block.
DETPTLED ~-v,v-ll~N OF TR~ ~NT TNV~NnT~N
FIG. 1 is a block diagram of an encoder 100 for performing motion estimation, s~J,,~.,I~tion and coding of a video sequence with regions of arbitrary shape and size.
The encoder 100 has a buffer 105 which serves as an interface between the ~nco~r 100 and a tr~n.~m;~.~ion or storage medium. The buffer 105 sends signals to a rate controller 108 via line 106 indic~tt;ng the ftlllnP~s of the buffer 105. In r~qro~.~e to the signals received from the buffer 105, the rate controller 108 controls the rate of flow of information from the ~n~o~r 100 to the - tr~nqm;~ion or storage medium by providing control signals to other c~pnn~tlt~ of the ~nCo~er 100 via line 109. The ~nco~r 100 further has a frame skip unit 110 for reeeiving a sequence of digital video frames one at a time from an input line 112 and for determ;n;ng whether a particular frame in the sequenee should be skipped.
A .~e~"~ tion unit 120 is ineluded in the encoder 100 for s~ ;ng a frame into a plurality of regions of arbitrary shape and size. The .~"~.,(~tion unit 120 also determines a motion veetor associated with eaeh region for predieting the cu~LeIlL frame from a previous deeoded frame whieh may be stored in a ~eco~ frame memory unit 125.
Details of the s~3mPnt~tion unit 120 are provided below.
2156~1 me ~nco~r 100 comprises a prediction unit 130 for predicting a region of the c~LLenL frame based upon a motion vector received from the ~3m~nt~tion unit 120 and a previous decoded frame received from the ~co~ frame memory unit 125.
A region shape co~;n~ unit 140 is also included in the ~n~er 100 for ~nCo~;n~ the shape of each region of the C~LL~1L frame. The encoder 100 further ~ ises a mode decision unit 150 for deciding in which of three modes image data of a region will be coded and transmitted. These three modes are ~ ¢sed in greater detail below. A mode and motion vector coding unit 160 is included in the ~co~er 100 for encoding the particular mode in which the information about a region will be sent and for ~nCo~;n~ the motion vector associated with the region. Although shown as a single unit in FIG. 1, the mode and motion unit 160 may c~l~ise separate units for performing the functions of ~nco~;n~ the mode and encoding the motion vector. S;~ rly, a region interior coder 170, described more fully below, is included for ~nco~;ng either the prediction error or the intensity value of a particular region dep~n~;n~ upon the mode in which the region is to be ~n~o~f~ The ~nco~r 100 also has a multiplexer 180 for passing the enco~ infonmation from the various co~ units 140, 160 and 170 to the buffer 105 in a predefined order. Finally, a control unit 101 is ~o.~r~e~ted to the other units so as to control the interaction and flow of infonmation between the other units of the ~n~ r 100. The control unit 101 may be, for example, a suitably ~r~yL~ mi~Lo~o~essor or other suitably configured hardware, as well as implemented in software.
FIG. 2 is a flow chart showing steps of the image coding method according to the principles of the present invention. As shown in step 205, a c~LeIlL frame in a sequence of frames is received by the frame skip unit 110 via line 112. I~e frame skip unit 110 determines whether the cu~e~l~ frame should be skipped or not, as shown in 210. me decision whether to skip the c~,enL frame is 5 deterrn;n~, for exa~le, by a control signal received on line 111 from the rate controller 108, which receives and processes information from the buffer 105 via line 106 indicating the flllln.o~ of the buffer. If the c~e~l~
frame is skipped, then the next frame in the sequence is 10 received by ret~ to step 205.
If the c~enL frame is not sk;E~l, then, as indicated by step 215, the CU~L`~1L. frame is s~J"~rlLed or divided into a plurality of regions each having a motion vector associated with it. In particular, the regions may 15 be of a shape and size which is not known a priori, thereby resulting in arbitrarily-~h~l regions. The c~lL. frame is divided into regions by grouping or merging together adjacent pels having the same or a s;m;l~r ;rlt.on~ity or by grouping together adjacent pels 20 having the same or a s;m;l~r motion vector. The m~ r of regions into which the frame is divided may be limited or controlled by a control signal on line 121 received from the rate controller 108. In order to determine the motion vector for each region, the se~ "l~tion unit 120 receives 25 the previous ~-oc~l frame from the ~ r unit 125 as an input.
Ihe output of the ~"~ tion unit 120 includes a description indicating the region to which each pel in the frame belongs and an array indic~t;ng the motion vector 30 assigned to each region. Once a frame has been s~rr~nted into a plurality of regions, and a motion vector has been assigned to each region, each region is processed until all the regions have been proc~ as indicated by step 220. If all the regions in the cuLLell~ frame have been 35 processed and coded, then the next frame is received in 2156g4~
step 205. If all the regions in the cu~r~lL frame have not been processed and coded, then the process continues to process and code the next region as shown in step 225.
As shown in step 230, the shape of each region is coded by the region shape co~;ng unit 140. For this purpose, the region shape co~;ng unit 140 receives from the .~"~ tion unit 120 a description indicating the region to which each pel in the frame belongs. This description may be, for example, a simple array of region l~hPl.~, a binary array of boundaries, a series of chain codes, a tree, or any other suitable se~"~lL~tion representation. The shape coding unit 140 ~n~o~ this description according to any one of a m ~hPr of lossless co~ t~chn;ques, such as arithmetic co~;ng. It should be noted that the step of enc~;ng the shape of the arbitrarily-~h~r~ regions may be performed prior~to step 220.
Next, as shown in step 235, a predicted region is ccmputed h~ upon the motion vector assigned to the cul,~lL region. The predicted region is computed in the prediction unit 130 which receives from the ll~m~ unit 125 the previous ~co~ frame. The description indicating the region to which each pel in the frame belongs and the array indicating the motion vector assigned to each region are sent from the se~ tion unit 120 to the prediction unit 130 as well.
In step 240, the mode decision unit 150 determines in which one of at least three modes the image data of each region is to be coded and transmitted. The first of the three possible modes is an inter-frame motion ~ e~lsated mode in which the motion vector associated with a particular region is ~nco~. In this first mode, no prediction error is encoded. A ~co~er then would recover the image of the region by using a previous decoded frame and the motion vector. The second mode is also inter-~15S9~4 frame motion coT~Pn~ted. A motion ~AI~Pn~ted predictionerror for each pel in the region, however, is also encoded along with the motion vector in order to improve the quality of the image intensity in the region. The prediction error represents the difference between the C~LL~1L image ~-P'JIl~PllL and the motion compensated segment obtA;ne~ from the prediction unit 130. The third mode is an intra-frame mode in which the image data in the region is treated indepPn~Pntly of previously decoded or reconstructed frames. From the st~n~roint of m;n;m;~;ng the amount of coded information generated by the P~co~er 100, the first mode is preferred bPc~ e it is likely to require the least ~m~l~nt of infonmation to be coded. Gn the other hand, the inter-frame motion ~,4~1sated modes may result in excessive noise or may fail to accurately predict a region in the CULL~1L frame under certain circumstA~P.~. In such situations, it is necessary to Pnco~e the intensity, in other words, the ll~;n~nce and chro~;n~n~P, of each pel in the region.
In one embo~;ment, the decision to code the image data in a particular mode may ~pren~ for example, upon the calculated values of the following two normalized sums of absolute differences for the region under consideration:
NSADI ~ ~ i~R
NSPI~ R Ieil where N is the total m ~hPr of pels in the region, i is a given pel in the region, and R is the set of all the pels in the region. Also, in the above equations, Ii is the intensity of the particular pel i, m is the mean value of the intensity of all the pels in the region, and e designates the motion co~pPn~ted prediction error associated with the pel i. For the above purposes, the mode decision unit 150 receives the same seg~Pnt~tion and 215G~ ~4 motion information that is sent to the prediction unit 130. In addition, the mode decision unit 150 receives the CUUL~1~ frame from the frame skip unit 110 and a predicted region from the prediction unit 130.
FIG. 3 is an exemplary graph plotting the values of NSPI~ versus NSADI for use in determ;n;ng the mode in which the coded image data is to be sent. The graph is divided by the solid lines into three sections corr~rDnA;ng to the three modes. When the value of NSADp is less than a threshold value c, then the image data is coded and transmitted in the first mode. When the value of N5ADp e~cPe~ the threshold value c and is less than the value of NSADI by at least a threshold value of b, then the image data is coded and transmitted in the s~con~
mode. Otherwise, the image data is coded and transmitted in the third mode. me value of b is, therefore, chosen to distribute the coded regions between the intra-frame mode and the sec~n~ inter-frame mode. The value of c is chosen to control the ~l~hPr of regions which are to be co~A in the first inter-frame motion c~,~ensated mode.
me values of b and c depend upon the noise present in the frame and the status of the rate control unit 108. For this purpose, for example, a control signal may be received on line 151 from the rate controller 108. Once the mode has been deter~;ne~, the choice of the mode is ~nco~A, for example, in binary code by the mode coding unit 160, as indicated in step 245.
It should be noted that there may be other modes different from or in addition to the three modes discussed above. Also, each of the three modes discussed above may permit or require other parameter choices to be made, thereby resulting in secon~ry modes. For example, a quantization step size, ~ e~ further below in the c~nt~xt of the region interior coder 170, may be 21S69~
adjustable. me setting of such parameters or secon~ry modes would also be ~n~o~ by the mode decision unit.
me next step depends upon whether the intra-frame coding mode was selected for the region under consideration, as shown by 250. If the intra-frame coding mode was not selected, then the motion vector of the region is also coded by the co~;n~ unit 160 as indicated in step 255. me next step ~r~n~ upon whether the first inter-frame motion ~m~n~ted mode was selected as shown in 260. If the first inter-frame motion compensated mode was not selected, then, as indicated in step 265, the image data for the region's interior is coded by the region interior coder 170. As expl~;n~ above, the interior coding for the inter-frame motion ~l4~1sated mode using prediction errors requires that a prediction error for each pel in the region under consideration be co~ and transmitted. In contrast, the interior coding for the intra-frame mode requires that the ;nt~nRity Of each pel in the region be coded and transmitted. Details of the region interior coder 170 and the step of co~;ng the image data for the region's interior are explained in greater detail below.
me co~ infonmation for the region under consideration, including the shape of the region, the choice of mode, and the motion vector and/or the image data for the region's interior, is then transmitted via the multiplexer 180 and the buffer 105 to a decoder via a tr~n.~ sion medium as indicated in step 270.
Alternatively, the coded information may be stored in a suitable storage medium, such as a CD-ROM, for subsequent decoding at a decoder. The steps 230 through 270 are performed for each region until all the regions in the current frame have been coded and transmitted. The ~co~r can then use the coded information to reconstruct the current frame.
215S9~4 FIG. 4 depicts a preferred ~mh~;m~nt of the se~ tion unit 120. The unit 120 reeeives the c~e~lL
frame and the previous decoded frame from the frame skip unit 110 and the ~co~ frame memory unit 125 via lines 5 401 and 402, respectively. The received e~LellL frame and previous ~eo~ frame data is routed, in r~p~ e to a first eontrol signal on line 406, to an intensity ~gmPnt~tion unit 420 via a switch 405. Alternatively, the received C~U~1L frame and previous decoded frame data is routed, in response to a second control signal on line 406, to a joint motion estimation and s~"~,ll~tion unit 450 via the switeh 405. The first and seeond eontrol signals are generated by the eontrol unit 101. The int~n~ity s~J"~"~tion unit 420 divides the C~UL~1L frame 15 into a plurality of arbitrarily-shaped intensity regions by grouping together pels that have the same or s;m;l~r ;nt~ ity features. The initial frame in a sequenee of images, for example, is always sent to the ;ntPn.~ity ~ "~,~ ;on unit 420 and coded in the intra-frame mode 20 hPC~ e motion est;m~;on and ~g~Pnt~tion is not applieable when there is no previous deeoded frame. A
deseription indieating the region to whieh eaeh pel in the frame belongs is sent via lead 422 to a switch 465. The switeh 465 is also eontrolled by a eontrol signal on line 25 466 that allows the infonmation indieating the region to whieh eaeh pel in the frame belongs to pass from the intensity .~"~ tion 420 or the joint motion estimation unit 450 to the other units in the eneoder 100 depending upon whieh unit, 450 or 420, reeeived the CUkre~l~ frame.
30 In other words, the eontrol signal on line 466 is synehronized with the control signal on line 406. The infonmation indicating the region to which each pel in the frame belongs may then be sent to the region shape coding unit 140, the prediction unit 130 and the mcde deeision 35 unit 150.
Other frames may also be selected and s~"~~ by the ;ntPn~ity .~"~ t;on unit 420 and sent in the intra-frame mode. If, for example, a scene ~h~n~e occurs which results in objects or image areas that did not appear in 5 previous frames, a decision is made to send the frame to the ;nt~n.~ity ~ tion unit 420. Similarly, in order to ease editing of the sequence at some later time, periodic frames, such as one frame every few secon~, may be S~J~ by the ;nt~n.~ity ~e~l~rl~tion unit 420. In addition, a particular frame may be sf~"~,l~ed by the ;nt~ ity .~"~---~tion unit 420 to resynchronize the ~nco~r 100 and the decoder.
The inter-frame modes can also be performed by the use of the ;nt~nqity .~ tion unit 420. In such a 15 situation, a motion vector is deter~'n~ for each region in a region based matching unit 425 using a region matching t~hn;que s;m;l~r to well-known block matching tP~hniques employed with rectangular or square regions.
In other words, each region is com~red to the previous 20 ~CO~ frame, and a best match is found which ~;n;m;zes the total prediction error for the region. A motion vector then is used to indicate the relative difference in position between the c~L~ellL region and the matched region in the previous decoded frame. The motion vectors are 25 then sent, via line 427, to a switch 475. The switch 475 is also controlled by a control signal on line 476 that allows the motion vector assigned to each region to pass from the ;nt~n.~ity s~yl~ ation unit 420 or the joint motion estimation and se~"~~ tion unit 450 to the other 30 ~ X~ nt~ in the ~nco~r 100 ~Pn~;n~ upon which unit, 450 or 420, received the C~Lr~1~ frame. The control signal on line 476 iS, therefore, also synchronized with the control signals on lines 406 and 466. The control signals on lines 466 and 476 are also sent from the 35 control unit 101 and allow image data to pass from the '- 2156~44 ;~t~n.~ity s~"~ tion unit 420 and the joint motion estimation and segmP~tAtion unit 450 to other units in the encoder 100 as expl Ai n~ above. The information indicating the motion vector associated with each region in the frame is sent to the prediction unit 130 and the mode decision unit 150.
Typically, however, frames other than the initial frame in the sequence are s~"~ by the joint motion estimation and segm~ntAtion unit 450. In a preferred e~ho~-m~nt, the joint motion estimation and se~"P~Ation unit 450 divides a CULL~1L frame into a plurality of regions and assigns a motion vector to each region according to the method and apparatus described in U.S.
Patent Application No. 08/271,308, entitled "Method and Apparatus For Motion Field Estimation, Sey"~Ation and Coding," which is assigned to the assignee of the present invention and which is in~o,~orated by reference herein.
The motion estimation and s~"P~IlAtion t~hn;que described in the afo~ Lioned application uses a hierarchical approach in which a frame is divided into a plurality of regions, and in which a motion vector updating routine is performed with respect to multiple levels of smaller and smaller regions. In a preferred embo~;~Pnt, the motion vector l~At;ng routine is performed with respect to smaller and smaller square blocks of predeter~;n~ size.
The motion vector updating routine ll~A~ the motion vector of each smaller block by assigning to it a best motion vector selected from among an initial motion vector assigned to the smaller block, motion vectors of neighboring blocks, and an updated or matched motion vector obtA;n~ by performing a block matching technique for the smaller block. The initial motion vector assigned to blocks in the first s~"P.,~Ation level is typically zero, whereas the initial motion vector assigned to each block in subsequent seylllenLation levels is the motion ` '_ 2156-944 vector of its parent block from which it was obt~;ne~.
The best motion vector for each block is selected according to a priority scheme and a predetenm;n~
threshold value.
FIG. 5 is a simplified block diagram showing the joint motion estimation and .~P~I~r~-~tion unit 450 that is used for determ;n;~ the best motion vector associated with each square block and for generating the arbitrarily-shaped regions having corresp~n~;n~ motion vectors. A
motion vector c~n~ te unit 510 computes and stores motion vectors that serve as candidates for the best motion vector associated with a particular square block.
These motion vectors include the initial motion vector (PV) assigned to the block, the motion vectors (V0 through V7) of up to eight neighboring blocks, and the updated motion vector (NV) ob~;n~ by performing a block matching ~Pchn;que for the block. A prediction error computation unit 520 computes and stores the motion co~n~ted prediction or matching error corr~rnn~ to each of the candidate motion vectors. A ~;n;~ ~ prediction error unit 530 determines the smallest prediction error (MIN) from ~ y the prediction errors computed by the prediction error unit 520. The motion vector candidates, the corr~.~r~n~;ng prediction errors, and the ~;n;~ predicted error then are sent to a best motion vector selection unit 540 which selects the best motion vector for the block under consideration.
A basic idea h~h;n~ the selection of the best motion vector for each block, according to the technique disclosed in the afoL~,l~.lL;o~ patent application No.
08/271,308, is to substitute the motion vector of one of the neighboring blocks or an updated motion vector obt~;ne~ by the block matching t~ch~;que for the cu~e~lL
block only if such a substitution yields a significant impro~ lL. in the matching or prediction error for that ' ~ 21~6g~
block. Furthermore, there is a preference for the motion vectors of the neighboring blocks relative to the updated motion vector of the cu~re~lL block ob~;ne~ from the block matching te~hn;que.
FIG. 6 shows an exemplary process, starting in step 600, by which the best motion vector is selected by the best motion vector selection unit 540 for a block which has, for example, eight neighboring blocks. In FIG. 6, PE, and E0 through E7 refer, respectively, to the prediction errors that result from assigning the motion vectors PV, and V0 through V7 to the block under consideration. In step 605, it is determin~ whether the absolute difference between the prediction error (PE) and the ~mallest prediction error (MIN) is less than a 15 predeterm;ne~ threshold (THR). If the absolute value in step 605 is less than the threshold value, this determ;n~t;on serves as an indication that substituting any one of the motion vectors V0 through V7, or NV, would not result in a significant im~ro~ lL in the prediction 20 error. As indicated by step 610, the motion vector PV is, therefore, selected as the best motion vector for the block under consideration.
If, however, the absolute difference determ;n~ in step 605 is not less than THR, then the process continues 25 with step 615. In each of steps 615, 625, 635, 645, 655, 665, 675 and 685, it is determ;n~ whether the absolute difference between MrN and a respective one of the prediction errors E0 through E7 is less than THR. If the absolute difference in a particular step is less than THR, 30 then the motion vector corre~r~ g to the particular prediction error is selected as the best motion vector as indicated by steps 620, 630, 640, 650, 660, 670, 680 and 690, respectively. The steps 615, 625, 635, 645, 655, 665, 675 and 685, are performed se~l~nt;~lly until a best 35 motion vector is selected. If none of the afo~e,.~lLioned '~ 21S6~44 absolute differenceæ is less than THR, then, as shown by step 695, this determination indicates that NV should be used as the best motion vector for the bloek under consideration.
Once a best motion vector is obta;n~ for each square bloek of the e~LLenL frame, eaeh square bloek is segmented into four smaller square bloeks of equal size, and the motion veetor updating routine is repeated for each smaller block until a stop condition is rea~h~. The stop condition may be, for example, a lower limit on the size of the blocks for which the motion vector updating routine is performed or a predeterm;n~ m ~h~r of eyeles of the motion vector updating routine.
When the stop condition is reAehPA, and a best motion vector has been assigned to each square bloek in the eu~.L .q~gmPntAtion level, a r~ "~ tion unit 550 performs a merging proeess which merges adjacent square regions having the same or s;m;lAr motion vectors to form a set of non-overlapping merged regions, each of whieh may be arbitrarily-shaped and have different ~imen~ions. It should be understood that, in some applications, some or all of the arbitrarily-shaped regions that result from the merging proeess may be square bloeks having the same or different ~;m~n.~ions.
me re.~ t;on unit 550 also assigns to eaeh pel in the euuf~lL frame a region label indieating to which region it belongs. The region lAh~l~ are then sent via line 452 to the switeh 465, and the motion veetors asso~;At~ with each region are sent via line 457 to the switch 475. As eXplA;n~A above, eontrol signals on lines 466 and 476 allow the region l~hel~ and motion veetors to pass to other ~l4~l,~nt~ of the eneoder 100 for further ~o~essing and ~nco~;n~.
In a preferred emb~;ment of the present invention, the region interior eoder 170 of the ~neo~er 100 _ 215~9'14 implem~nt.~, for example, the method described in U.S.
Patent Application No. 08/138,295, entitled "Block Transform Coder For Arbitrarily Shaped Image S~"~-,~s,~
which is assigned to the assignee of the present invention, and which is in~o~o~ated by reference herein.
The method described in Application No. 08/138,295 uses block transfonms with frequency ~ ;n region-zeroing and space ~m~;n region-enforcing operations for effectively co~;ng the image data of arbitrarily-shaped regions. The method uses an iterative t~rhn;que based on the theory of sll~ce~ive projection onto convex sets (POCS) to find the best values for a group of selected transform coefficients.
A simplified block diagram of an exemplary region interior coder 170 for implem~t;ng the afore~ Lioned iterative POCS t~chn;que is depicted in FIG. 7. An image circumscription unit 710 receives the image data of an arbitrarily-.~h~r~ region 802 from the mode decision unit 150. As shown in FIG. 8, a rectangular region block 801 is circumscribed around the arbitrarily-shaped region 802.
An original internal pel set 803 which lies within the arbitrarily-shaped region 802 is thereby defined.
S;mil~rly, an original ext~r~l pel set 804 which lies outside the arbitrarily-shAr~A region 802 and within the region block 801 is thereby defined. An extrapolator 720 extrapolates the pel values of the ;nter~l pel set 803 to initl~ e the pel values of the external pel set 804.
Examples of extrapolation methods include pel repetition, mirroring and ll~llological dilations.
Okher ~,5~ nt.~ of the transfonm coder 170 shown in FIG. 7 perform a POCS iteration loop on the image data.
It should be understood that the image data upon which the POCS iteration loop is perfonmed ~r~n~-~ upon the mode in which the image data of the region is to be coded, and transmitted or stored. In the second inter-frame motion ~ 215~9~1 c~.4~lsated mode in which the prediction errors are coded, the image data coded by the POCS iteration locp is the prediction error associated with the pels in the region under consideration. If, on the other hand, the intra-frame mode is to be used, then the image data coded by thePOCS iteration loop includes the ;nt~n.~ity of each pel in the region under consideration.
The POCS iteration loop begins with the application of a forward transform, such as the discrete cosine transform (DCT), by a forward transform unit 730 to generate transform coefficients. A transform coefficient set (TCS) is generated by a TCS generator 740 which selects and retains transform coefficients having high energy according to the energy com~ction property of transform coefficients. The re~;n;n~ transform coefficients are set to zero. The number of selected coefficients is deter~;ne~ by the rate controller 108 which establ;~h~ a threshold energy h~ , for example, upon the size of the arbitrarily-.~h~ region. Next, an inverse transform unit 750 perfonms an inverse transform on the TCS to generate a cc~ t~ region block having c~.~uted pel values. An interior pel repl~ ~n~ unit 760 replaces those ccmputed pels corr~r~n~;ng to the internal pel set with original pel values to form a modified ccmputed r~;on block. A forward transform is ~g~;n perfor~f~ on the modified computed region block (MCRB), and a new transform coefficient set is generated.
If a particular transform coefficient set (TCS) represents optimal transform coefficients (OTC), then the TCS is qll~nt;~ and coded using, for example, variable length coding. The step size of quantization may be deter~;nf~, for example, by a signal on line 171 received from the rate controller 108. The coded, qll~nt;~ed values of the optimal transfonm coefficients are then sent as outputs to the multiplexer 180. A particular TCS may 21569~4 represent optimal transform coefficients when a predeterm;n~ number of POCS loop iterations is reached, when the exterior pels do not ~h~n~e, or when the mean squared difference of the exterior pels between iterations is within a pre-defined threshold. If the TCS does not represent the optimal transform coefficients, then the POCS loop is reiterated until optimal transform coefficients are obt~;ne~. At the decoded frame memory unit 125 or at a ~co~r, an inverse quantization process and inverse transform process is applied to reconstruct the image data for the region prior to storing the decoded region.
me region-based video ~ncq~r 100 of the present invention is particularly adv~nt~ous for systems using low bit-rates. Com~ m;cation applications include, for example, video tele~l~u~l~, personal c~m~ln;cation, mult;mP~;~, education, entert~ nt and remote sensing where the tr~n~m;s~ion or storage c~c;ty of the system is limited.
Although the present invention has been described with reference to .~c;fic ~mh~;ment.~ it will be appreciated that other a~ly~.~r~L~ within the spirit and scope of present invention will be readily apparent to persons of ordinary skill in the art. The present invention is, therefore, limited only by the appended cl~;m.~.
Claims (49)
1. An encoder for encoding a sequence of video frames comprising:
a segmentation unit which segments a current frame in said video sequence into a plurality of arbitrarily-shaped regions, each of said plurality of arbitrarily-shaped regions being assigned a motion vector;
a decoded frame memory for storing a previously decoded frame in said video sequence;
a prediction unit connected to said segmentation unit and said decoded frame memory for predicting image data of said current frame based upon a previously decoded frame and based upon the motion vector assigned to one of said plurality of arbitrarily-shaped regions;
a region shape coding unit for encoding the shape of each of said arbitrarily-shaped regions;
a mode decision unit which determines in which one of a plurality of modes image data from each of said plurality of arbitrarily-shaped regions is to be encoded, where said plurality of modes comprises an intra-frame mode in which the intensity of each pel in one of said plurality of arbitrarily-shaped regions is encoded;
a mode coding unit which encodes the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
a motion coding unit for encoding motion vectors associated with said plurality of arbitrarily-shaped regions;
a region interior coder which encodes the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said intra-frame mode;
a buffer which serves as an interface for transmitting encoded information from said encoder; and a rate controller which receives signals from said buffer, where said rate controller sends control signals to said segmentation unit, said mode decision unit, and said region interior unit in response to the signals received from said buffer.
a segmentation unit which segments a current frame in said video sequence into a plurality of arbitrarily-shaped regions, each of said plurality of arbitrarily-shaped regions being assigned a motion vector;
a decoded frame memory for storing a previously decoded frame in said video sequence;
a prediction unit connected to said segmentation unit and said decoded frame memory for predicting image data of said current frame based upon a previously decoded frame and based upon the motion vector assigned to one of said plurality of arbitrarily-shaped regions;
a region shape coding unit for encoding the shape of each of said arbitrarily-shaped regions;
a mode decision unit which determines in which one of a plurality of modes image data from each of said plurality of arbitrarily-shaped regions is to be encoded, where said plurality of modes comprises an intra-frame mode in which the intensity of each pel in one of said plurality of arbitrarily-shaped regions is encoded;
a mode coding unit which encodes the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
a motion coding unit for encoding motion vectors associated with said plurality of arbitrarily-shaped regions;
a region interior coder which encodes the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said intra-frame mode;
a buffer which serves as an interface for transmitting encoded information from said encoder; and a rate controller which receives signals from said buffer, where said rate controller sends control signals to said segmentation unit, said mode decision unit, and said region interior unit in response to the signals received from said buffer.
2. The encoder of claim 1 wherein said plurality of modes further comprises a first inter-frame mode in which a motion vector associated with one of said plurality of arbitrarily-shaped regions is encoded and a second inter-frame mode in which a motion vector and a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions are encoded; and said region interior coder encodes a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said second inter-frame mode.
3. The encoder of claim 1 wherein said segmentation unit comprises a joint motion estimation and segmentation unit for perform1ng the following functions:
(a) dividing said current frame into a plurality of smaller regions of predetermined shape and size;
(b) perfonming for each of said smaller regions a motion vector updating routine by assigning to the smaller region a best motion vector selected from among an initial motion vector assigned to the smaller region, motion vectors of neighboring regions, and an updated motion vector obtained by performing a block matching technique for the smaller region, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(c) dividing each of said smaller regions into a plurality of smaller regions of predetermined shape and size;
(d) repeating step (b) for each of the smaller regions resulting from step (c);
(e) iteratively repeating steps (c) and (d) for each smaller region resulting from step (d) until a stop condition is reached; and (f) merging adjacent regions having similar motion vectors to form said arbitrarily-shaped regions.
(a) dividing said current frame into a plurality of smaller regions of predetermined shape and size;
(b) perfonming for each of said smaller regions a motion vector updating routine by assigning to the smaller region a best motion vector selected from among an initial motion vector assigned to the smaller region, motion vectors of neighboring regions, and an updated motion vector obtained by performing a block matching technique for the smaller region, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(c) dividing each of said smaller regions into a plurality of smaller regions of predetermined shape and size;
(d) repeating step (b) for each of the smaller regions resulting from step (c);
(e) iteratively repeating steps (c) and (d) for each smaller region resulting from step (d) until a stop condition is reached; and (f) merging adjacent regions having similar motion vectors to form said arbitrarily-shaped regions.
4. The encoder of claim 3 wherein said joint motion estimation and segmentation unit comprises a best motion vector selection unit which performs the following functions:
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value; and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value; and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value; and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value; and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
5. The encoder of claim 3 wherein the step of dividing each of said smaller regions into a plurality of smaller regions comprises the step of dividing each of said smaller regions into four smaller square blocks of equal size.
6. The encoder of claim 5 wherein said region interior coder uses an iterative technique with frequency domain region-zeroing and space domain region-enforcing operations to transform an arbitrarily-shaped image into optimal transform coefficients (OTC).
7. The encoder of claim 6 wherein said region interior coder comprises:
an image circumscription unit for receiving image data of an arbitrarily-shaped region, and circumscribing a rectangular block around said arbitrarily-shaped region, thereby defining an original internal pel set and an original external pel set;
an extrapolator which extrapolates pel values of said internal pel set to initialize pel values of said external pel set;
a forward transform which transforms said image to transform coefficients;
a TCS generator which generates a transform coefficient set (TCS) from said transform coefficients, said TCS generator outputs said TCS when said TCS
represents said OTC, and sends said TCS to an inverse transform when said TCS does not represent said OTC;
an inverse transform which transforms said TCS to a computed region block having computed pel values; and a replacer which replaces those computed pel values corresponding to said interior pel set with said original pel values to form a modified computed region block (MCRB), said replacer sends the modified computed region block to the forward transform for re-iteration.
an image circumscription unit for receiving image data of an arbitrarily-shaped region, and circumscribing a rectangular block around said arbitrarily-shaped region, thereby defining an original internal pel set and an original external pel set;
an extrapolator which extrapolates pel values of said internal pel set to initialize pel values of said external pel set;
a forward transform which transforms said image to transform coefficients;
a TCS generator which generates a transform coefficient set (TCS) from said transform coefficients, said TCS generator outputs said TCS when said TCS
represents said OTC, and sends said TCS to an inverse transform when said TCS does not represent said OTC;
an inverse transform which transforms said TCS to a computed region block having computed pel values; and a replacer which replaces those computed pel values corresponding to said interior pel set with said original pel values to form a modified computed region block (MCRB), said replacer sends the modified computed region block to the forward transform for re-iteration.
8. The encoder of claim 7 wherein said TCS
generator generates said TCS by selecting an retaining those transform coefficients which have high energy according to the energy compaction property of transform coefficients, and by zeroing all the non-selected transform coefficients.
generator generates said TCS by selecting an retaining those transform coefficients which have high energy according to the energy compaction property of transform coefficients, and by zeroing all the non-selected transform coefficients.
9. The encoder of claim 8 further comprising a frame skip unit which receives said sequence of frames and which determines whether each frame in said sequence should be skipped.
10. The encoder of claim 9 further comprising a multiplexer for passing encoded information from said region shape coding unit, said mode coding unit, said motion coding unit, and said region interior coder to said buffer in a predefined order.
11. The encoder of claim 2 wherein said mode decision unit determines in which one of said plurality of modes to encode image data of a particular one of said arbitrarily-shaped regions based upon the values of the following normalized sums of absolute differences for the particular region:
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, Ii is the intensity of the given pel i, m is the mean value of the intensity of all the pels in the particular region, and e? designates the motion compensated prediction error associated with the given pel i.
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, Ii is the intensity of the given pel i, m is the mean value of the intensity of all the pels in the particular region, and e? designates the motion compensated prediction error associated with the given pel i.
12. The encoder of claim 3 wherein said segmentation unit further comprises:
an intensity segmentation unit which divides a frame into a plurality of arbitrarily-shaped intensity regions by grouping together pels that have a similar intensity features;
a region based matching unit for determining a motion vector indicating the relative difference in position between one of said plurality of intensity regions and a matched region in a previously decoded frame;
a switch for sending a received frame to said intensity segmentation unit in response to a first control signal and to said joint motion estimation and segmentation unit in response to a second control signal;
and a plurality of switches that allow image data information to pass from said intensity segmentation unit and said joint motion estimation and segmentation unit to other of said units in said encoder in response to control signals synchronized with said first and second control signals, respectively.
an intensity segmentation unit which divides a frame into a plurality of arbitrarily-shaped intensity regions by grouping together pels that have a similar intensity features;
a region based matching unit for determining a motion vector indicating the relative difference in position between one of said plurality of intensity regions and a matched region in a previously decoded frame;
a switch for sending a received frame to said intensity segmentation unit in response to a first control signal and to said joint motion estimation and segmentation unit in response to a second control signal;
and a plurality of switches that allow image data information to pass from said intensity segmentation unit and said joint motion estimation and segmentation unit to other of said units in said encoder in response to control signals synchronized with said first and second control signals, respectively.
13. The encoder of claim 1 wherein said region interior coder uses an iterative technique with frequency domain region-zeroing and space domain region-enforcing operations to transfonm an arbitrarily-shaped image into optimal transfonm coefficients (OTC).
14. The encoder of claim 13 wherein said region interior coder comprises:
an image circumscription unit for receiving image data of an arbitrarily-shaped region, and circumscribing a rectangular block around said arbitrarily-shaped region, thereby defining an original internal pel set and an original external pel set;
an extrapolator which extrapolates pel values of said internal pel set to initialize pel values of said external pel set;
a forward transform which transfonms said image to transfonm coefficients;
a TCS generator which generates a transform coefficient set (TCS) from said transform coefficients, said TCS generator outputs said TCS when said TCS
represents said OTC, and sends said TCS to an inverse transform when said TCS does not represent said OTC;
an inverse transform which transforms said TCS to a computed region block having computed pel values; and a replacer which replaces those computed pel values corresponding to said interior pel set with said original pel values to form a modified computed region block (MCRB), said replacer sends the modified computed region block to the forward transform for re-iteration.
an image circumscription unit for receiving image data of an arbitrarily-shaped region, and circumscribing a rectangular block around said arbitrarily-shaped region, thereby defining an original internal pel set and an original external pel set;
an extrapolator which extrapolates pel values of said internal pel set to initialize pel values of said external pel set;
a forward transform which transfonms said image to transfonm coefficients;
a TCS generator which generates a transform coefficient set (TCS) from said transform coefficients, said TCS generator outputs said TCS when said TCS
represents said OTC, and sends said TCS to an inverse transform when said TCS does not represent said OTC;
an inverse transform which transforms said TCS to a computed region block having computed pel values; and a replacer which replaces those computed pel values corresponding to said interior pel set with said original pel values to form a modified computed region block (MCRB), said replacer sends the modified computed region block to the forward transform for re-iteration.
15. The encoder of claim 1 further comprising a frame skip unit which receives said sequence of frames and which determines whether each frame in said sequence should be skipped.
16. The encoder of claim 15 further comprising a multiplexer for passing encoded information from said region shape coding unit, said mode coding unit, said motion coding unit, and said region interior coder to said buffer in a predefined order.
17. The encoder of claim 15 wherein said rate controller sends control signals to said frame skip unit in response to the signals received from said buffer.
18. A method of encoding a frame in a video sequence comprising the steps of:
(a) segmenting the frame into a plurality of arbitrarily-shaped regions each having a corresponding motion vector;
(b) encoding the shape of each arbitrarily-shaped region;
(c) determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded, where said plurality of modes includes a first mode in which the motion vector corresponding to an arbitrarily-shaped region is encoded, a second mode in which the motion vector and a motion compensated prediction error associated with an arbitrarily-shaped region are encoded, and a third intra-frame mode in which the intensity of each pel in an arbitrarily-shaped region is encoded;
(d) encoding the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
(e) encoding the motion vector corresponding to one of said plurality of arbitrarily-shaped regions if the region is to be encoded in either said first mode or said second mode;
(f) encoding a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said second mode;
(g) encoding the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said third mode; and (h) storing information encoded in steps (b), (d), (e), (f) and (g).
(a) segmenting the frame into a plurality of arbitrarily-shaped regions each having a corresponding motion vector;
(b) encoding the shape of each arbitrarily-shaped region;
(c) determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded, where said plurality of modes includes a first mode in which the motion vector corresponding to an arbitrarily-shaped region is encoded, a second mode in which the motion vector and a motion compensated prediction error associated with an arbitrarily-shaped region are encoded, and a third intra-frame mode in which the intensity of each pel in an arbitrarily-shaped region is encoded;
(d) encoding the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
(e) encoding the motion vector corresponding to one of said plurality of arbitrarily-shaped regions if the region is to be encoded in either said first mode or said second mode;
(f) encoding a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said second mode;
(g) encoding the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said third mode; and (h) storing information encoded in steps (b), (d), (e), (f) and (g).
19. The method of claim 18 wherein the step of segmenting said frame comprises the steps of:
(a) dividing the frame into a plurality of smaller regions of predetermined shape and size to form a first segmentation level;
(b) assigning to each of said plurality of smaller regions an initial motion vector;
(c) performing for each of said plurality of smaller regions a motion vector updating routine which updates the motion vector of a smaller region by assigning to it a best motion vector selected from among the initial motion vector assigned to the smaller region, an updated motion vector abtained by performing a block matching technique for the smaller region, and motion vectors of the smaller region's neighboring regions, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(d) dividing each smaller region in the previous segmentation level into a plurality of smaller regions of predetermined shape and size to form a subsequent segmentation level;
(e) assigning to each of the plurality of smaller regions in the subsequent segmentation level an initial motion vector equal to the motion vector of its parent region;
(f) performing the motion vector updating routine for each of said plurality of smaller regions in the subsequent segmentation level;
(g) iteratively performing the steps (d), (e) and (f) until a stop condition is reached;
(h) merging adjacent smaller regions having similar motion vectors to form said plurality of arbitrarily-shaped regions.
(a) dividing the frame into a plurality of smaller regions of predetermined shape and size to form a first segmentation level;
(b) assigning to each of said plurality of smaller regions an initial motion vector;
(c) performing for each of said plurality of smaller regions a motion vector updating routine which updates the motion vector of a smaller region by assigning to it a best motion vector selected from among the initial motion vector assigned to the smaller region, an updated motion vector abtained by performing a block matching technique for the smaller region, and motion vectors of the smaller region's neighboring regions, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(d) dividing each smaller region in the previous segmentation level into a plurality of smaller regions of predetermined shape and size to form a subsequent segmentation level;
(e) assigning to each of the plurality of smaller regions in the subsequent segmentation level an initial motion vector equal to the motion vector of its parent region;
(f) performing the motion vector updating routine for each of said plurality of smaller regions in the subsequent segmentation level;
(g) iteratively performing the steps (d), (e) and (f) until a stop condition is reached;
(h) merging adjacent smaller regions having similar motion vectors to form said plurality of arbitrarily-shaped regions.
20. The method of claim 19 wherein the motion vector updating routine comprises the steps of:
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
21. The method of claim 20 wherein the steps of encoding the prediction error and encoding the intensity of each pel comprise the steps of:
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said arc;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said arc;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
22. The method of claim 21 wherein said step of performing a forward transform comprises the step of performing a discrete cosine transform (DCT).
23. me method of claim 22 wherein the step of generating a TCS comprises the step of quantizing said transform coefficients.
24. The method of claim 23 wherein the step of generating said TCS further comprises the steps of selecting and retaining those transform coefficients which have high energy according to the energy compaction property of transform coefficients, and zeroing the non-selected transform coefficients.
25. The method of claim 18 wherein the steps of encoding the prediction error and encoding the intensity of each pel comprise the steps of:
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initiating pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initiating pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
26. The method of claim 25 wherein said step of performing a forward transform comprises the step of performing a discrete cosine transform (DCT).
27. The method of claim 26 wherein the step of generating a TCS comprises the step of quantizing said transform coefficients.
28. The method of claim 27 further comprising the step of deciding whether to skip said frame where said step of deciding depends upon the fullness of a buffer which serves as an interface to said decoder.
29. The method of claim 28 wherein the step of determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded is based upon the values of the following normalized sums of absolute differences for the particular region:
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, I? is the intensity of the given pel i m is the mean value of the intensity of all the pels in the particular region, and ei designates the motion compensated prediction error associated with the given pel i.
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, I? is the intensity of the given pel i m is the mean value of the intensity of all the pels in the particular region, and ei designates the motion compensated prediction error associated with the given pel i.
30. The method of claim 29 wherein image data of the particular region is encoded in said first mode if the value of NSADp is less than a threshold value c and in said second mode when the value of NSADp exceeds the threshold value c and is less than the value of NSADI by at least a threshold value b.
31. The method of claim 30 wherein said threshold values b and c depend upon the fullness of a buffer which serves as an interface to said decoder.
32. The method of claim 24 wherein the number of selected transform coefficients is based upon the size of the particular arbitrarily-shaped region being encoded.
33. The method of claim 23 wherein the step of quantizing said transform coefficients uses a quantization step size which depends upon the fullness of a buffer which serves as an interface to a decoder.
34. A method of encoding a frame in a video sequence comprising the steps of:
(a) segmenting the frame into a plurality of arbitrarily-shaped regions each having a corresponding motion vector;
(b) encoding the shape of each arbitrarily-shaped region;
(c) determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded, where said plurality of modes includes a first mode in which the motion vector corresponding to an arbitrarily-shaped region is encoded, a second mode in which the motion vector and a motion compensated prediction error associated with an arbitrarily-shaped region are encoded, and a third intra-frame mode in which the intensity of each pel in an arbitrarily-shaped region is encoded;
(d) encoding the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
(e) encoding the motion vector corresponding to one of said plurality of arbitrarily-shaped regions if the region is to be encoded in either said first mode or said second mode;
(f) encoding a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said second mode;
(g) encoding the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said third mode; and (h) transmitting information encoded in steps (b), (d), (e), (f) and (g) to a decoder.
(a) segmenting the frame into a plurality of arbitrarily-shaped regions each having a corresponding motion vector;
(b) encoding the shape of each arbitrarily-shaped region;
(c) determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded, where said plurality of modes includes a first mode in which the motion vector corresponding to an arbitrarily-shaped region is encoded, a second mode in which the motion vector and a motion compensated prediction error associated with an arbitrarily-shaped region are encoded, and a third intra-frame mode in which the intensity of each pel in an arbitrarily-shaped region is encoded;
(d) encoding the mode in which each of said plurality of arbitrarily-shaped regions is to be encoded;
(e) encoding the motion vector corresponding to one of said plurality of arbitrarily-shaped regions if the region is to be encoded in either said first mode or said second mode;
(f) encoding a motion compensated prediction error associated with one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said second mode;
(g) encoding the intensity of each pel in one of said plurality of arbitrarily-shaped regions if the region is to be encoded in said third mode; and (h) transmitting information encoded in steps (b), (d), (e), (f) and (g) to a decoder.
35. The method of claim 34 wherein the step of segmenting said frame comprises the steps of:
(a) dividing the frame into a plurality of smaller regions of predetermined shape and size to form a first segmention level;
(b) assigning to each of said plurality of smaller regions an initial motion vector;
(c) performing for each of said plurality of smaller regions a motion vector updating routine which updates the motion vector of a smaller region by assigning to it a best motion vector selected from among the initial motion vector assigned to the smaller region, an updated motion vector obtained by performing a block matching technique for the smaller region, and motion vectors of the smaller region's neighboring regions, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(d) dividing each smaller region in the previous segmentation level into a plurality of smaller regions of predetermined shape and size to form a subsequent segmentation level;
(e) assigning to each of the plurality of smaller regions in the subsequent segmentation level an initial motion vector equal to the motion vector of its parent region;
(f) performing the motion vector updating routine for each of said plurality of smaller regions in the subsequent segmentation level;
(g) iteratively perfonming the steps (d), (e) and (f) until a stop condition is reached;
(h) merging adjacent smaller regions having similar motion vectors to form said plurality of arbitrarily-shaped regions.
(a) dividing the frame into a plurality of smaller regions of predetermined shape and size to form a first segmention level;
(b) assigning to each of said plurality of smaller regions an initial motion vector;
(c) performing for each of said plurality of smaller regions a motion vector updating routine which updates the motion vector of a smaller region by assigning to it a best motion vector selected from among the initial motion vector assigned to the smaller region, an updated motion vector obtained by performing a block matching technique for the smaller region, and motion vectors of the smaller region's neighboring regions, wherein the best motion vector is selected according to a priority scheme and a predetermined threshold value;
(d) dividing each smaller region in the previous segmentation level into a plurality of smaller regions of predetermined shape and size to form a subsequent segmentation level;
(e) assigning to each of the plurality of smaller regions in the subsequent segmentation level an initial motion vector equal to the motion vector of its parent region;
(f) performing the motion vector updating routine for each of said plurality of smaller regions in the subsequent segmentation level;
(g) iteratively perfonming the steps (d), (e) and (f) until a stop condition is reached;
(h) merging adjacent smaller regions having similar motion vectors to form said plurality of arbitrarily-shaped regions.
36. The method of claim 35 wherein the motion vector updating routine comprises the steps of:
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
determining the smallest matching error from among the matching errors obtained respectively by assigning to the smaller region the following motion vectors:
(a) the initial motion vector assigned to the smaller region;
(b) the updated motion vector obtained by performing a block matching technique for the smaller region; and (c) the motion vectors of the smaller region's neighboring regions;
selecting the initial motion vector as the best motion vector if the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is less than the predetermined threshold value;
selecting the motion vector of one of the neighboring regions as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and the matching error obtained by assigning to the smaller region the motion vector of the neighboring region is less than the predetermined threshold value; and selecting the matched motion vector as the best motion vector if:
(a) the absolute value of the difference between the smallest matching error and the matching error obtained by using the initial motion vector is not less than the predetermined threshold value;
and (b) the absolute value of the difference between the smallest matching error and each of the matching errors obtained by assigning to the smaller region the motion vector of one of the neighboring region is not less than the predetermined threshold value.
37. The method of claim 36 wherein the steps of encoding the prediction error and encoding the intensity of each pel comprise the steps of:
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transform coefficients;
(c) performing an inverse transform on said TCS
thereby generating a computed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
38. The method of claim 37 wherein said step of performing a forward transform comprises the step of performing a discrete cosine transform (DCT).
39. The method of claim 38 wherein the step of generating a TCS comprises the step of quantizing said transform coefficients.
40. The method of claim 39 wherein the step of generating said TCS further comprises the steps of selecting and retaining those transform coefficients which have high energy according to the energy compaction property of transform coefficients, and zeroing the non-selected transfonm coefficients.
41. The method of claim 34 wherein the steps of encoding the prediction error and encoding the intensity of each pel comprise the steps of:
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transfonm coefficients;
(c) performing an inverse transform on said TCS
thereby generating a ccmputed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
generating original pel values by:
(a) circumscribing said arbitrarily-shaped region with a rectangular region block, thereby creating an internal pel set which lies within said arbitrarily-shaped image and within said region block, and an external pel set which lies outside said arbitrarily-shaped region and within said region block; and, (b) initializing pel values of said external pel set by extrapolating the pel values of said internal pel set; and calculating optimal transform coefficients (OTC) by:
(a) performing a forward transform on said region block to generate transform coefficients;
(b) generating a transform coefficient set (TCS) from said transfonm coefficients;
(c) performing an inverse transform on said TCS
thereby generating a ccmputed region block having computed pel values;
(d) replacing those computed pel values corresponding to said internal pel set with original pel values to form a modified computed region block (MCRB);
(e) determining whether said TCS represents said OTC;
(f) reiterating steps (a) and (b) on said modified computed region block and outputting said TCS
when said TCS represents said OTC; and, (g) reiterating steps (a) through (g) on said modified computed region block when said TCS
values do not represent said OTC.
42. The method of claim 41 wherein said step of performing a forward transform comprises the step of performing a discrete cosine transform (DCT).
43. The method of claim 42 wherein the step of generating a TCS comprises the step of quantizing said transform coefficients.
44. The method of claim 43 further comprising the step of deciding whether to skip said frame where said step of deciding depends upon the fullness of a buffer which serves as an interface to said decoder.
45. me method of claim 44 wherein the step of determining in which of a plurality of modes image data of each arbitrarily-shaped region is to be encoded is based upon the values of the following normalized sums of absolute differences for the particular region:
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, Ii is the intensity of the given pel i, m is the mean value of the intensity of all the pels in the particular region, and ei designates the motion compensated prediction error associated with the given pel i.
where N is the total number of pels in the particular region, i is a given pel in the region, R is the set of all pels in the particular region, Ii is the intensity of the given pel i, m is the mean value of the intensity of all the pels in the particular region, and ei designates the motion compensated prediction error associated with the given pel i.
46. The method of claim 45 wherein image data of the particular region is encoded in said first mode if the value of NSADp is less than a threshold value c and in said second mode when the value of NSADp exceeds the threshold value c and is less than the value of NSADI by at least a threshold value b.
47. The method of claim 46 wherein said threshold values b and c depend upon the fullness of a buffer which serves as an interface to said decoder.
48. The method of claim 40 wherein the number of selected transform coefficients is based upon the size of the particular arbitrarily-shaped region being encoded.
49. The method of claim 39 wherein the step of quantizing said transform coefficients uses a quantization step size which depends upon the fullness of a buffer which serves as an interface to a decoder.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US08/322,893 US5608458A (en) | 1994-10-13 | 1994-10-13 | Method and apparatus for a region-based approach to coding a sequence of video images |
| US322,893 | 1994-10-13 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2156944A1 true CA2156944A1 (en) | 1996-04-14 |
Family
ID=23256900
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002156944A Abandoned CA2156944A1 (en) | 1994-10-13 | 1995-08-25 | Method and apparatus for a region-based approach to coding a sequence of video images |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US5608458A (en) |
| EP (1) | EP0707427A3 (en) |
| JP (1) | JPH08214318A (en) |
| CA (1) | CA2156944A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110300977A (en) * | 2017-02-17 | 2019-10-01 | 考吉森公司 | Method for image procossing and video compress |
Families Citing this family (119)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5828784A (en) * | 1992-07-13 | 1998-10-27 | Hitachi Denshi Kabushiki Kaisha | Data coding method and apparatus using a plurality of blocks of different length |
| JP3275423B2 (en) * | 1993-03-04 | 2002-04-15 | キヤノン株式会社 | Recording device |
| JP3050736B2 (en) * | 1993-12-13 | 2000-06-12 | シャープ株式会社 | Video encoding device |
| US6798834B1 (en) * | 1996-08-15 | 2004-09-28 | Mitsubishi Denki Kabushiki Kaisha | Image coding apparatus with segment classification and segmentation-type motion prediction circuit |
| DE69518128T2 (en) * | 1994-09-21 | 2001-03-29 | Toshiba Kawasaki Kk | Image data processing device with image data dividing function and method for processing image data |
| KR0178231B1 (en) * | 1995-08-10 | 1999-05-01 | 배순훈 | Method and apparatus for detecting motion vectors based on hierarchical motion estimation |
| US5978514A (en) * | 1994-11-10 | 1999-11-02 | Kabushiki Kaisha Toshiba | Image data coding and decoding system for efficiently compressing information using the shape and position of the image content |
| EP0719049B1 (en) * | 1994-12-20 | 2003-05-28 | Matsushita Electric Industries Co., Ltd. | Method and apparatus for image coding |
| KR100235345B1 (en) * | 1994-12-29 | 1999-12-15 | 전주범 | Moving picture estimation apparatus and method in divided region |
| JP3169783B2 (en) * | 1995-02-15 | 2001-05-28 | 日本電気株式会社 | Video encoding / decoding system |
| US5731840A (en) * | 1995-03-10 | 1998-03-24 | Kabushiki Kaisha Toshiba | Video coding/decoding apparatus which transmits different accuracy prediction levels |
| JPH08256266A (en) * | 1995-03-17 | 1996-10-01 | Mitsubishi Electric Corp | Image coding system |
| KR100249028B1 (en) * | 1995-03-20 | 2000-03-15 | 전주범 | Apparatus for effectively encoding/decoding video signals having stationary object |
| KR0171120B1 (en) * | 1995-04-29 | 1999-03-20 | 배순훈 | Method and apparatus for determining motion region in video coding technique using feature point based motion compensation |
| US6023301A (en) * | 1995-07-14 | 2000-02-08 | Sharp Kabushiki Kaisha | Video coding device and video decoding device |
| DE69615812T2 (en) * | 1995-08-02 | 2002-06-20 | Koninkl Philips Electronics Nv | METHOD AND SYSTEM FOR CODING A PICTURE SEQUENCE |
| JPH09121358A (en) * | 1995-10-25 | 1997-05-06 | Matsushita Electric Ind Co Ltd | Picture coding/decoding device and its method |
| CN1113541C (en) * | 1995-10-25 | 2003-07-02 | 皇家菲利浦电子有限公司 | Segmented picture coding method and system, and corresponding decoding method ans system |
| KR100211916B1 (en) * | 1995-10-26 | 1999-08-02 | 김영환 | Method for determination of coding type and mode in the object coding based on shape information |
| ES2241034T3 (en) | 1996-02-07 | 2005-10-16 | Sharp Kabushiki Kaisha | DEVICE FOR CODIFICATION OF MOVEMENT DECODIFICATION. |
| AU733569B2 (en) * | 1996-03-05 | 2001-05-17 | Telefonaktiebolaget Lm Ericsson (Publ) | Progressive image coding |
| US5778097A (en) * | 1996-03-07 | 1998-07-07 | Intel Corporation | Table-driven bi-directional motion estimation using scratch area and offset valves |
| DE19615492A1 (en) * | 1996-04-19 | 1997-10-23 | Philips Patentverwaltung | Image segmentation method |
| US6137920A (en) * | 1996-05-01 | 2000-10-24 | Hughes Electronics Corporation | Method and system for generating image frame sequences using morphing transformations |
| US5719642A (en) * | 1996-05-07 | 1998-02-17 | National Science Council Of R.O.C. | Full-search block matching motion estimation processor |
| EP1120971B1 (en) | 1996-05-17 | 2003-09-03 | Matsushita Electric Industrial Co., Ltd. | Video coding method for coding shape and texture signals using different modes |
| KR100403077B1 (en) * | 1996-05-28 | 2003-10-30 | 마쯔시다덴기산교 가부시키가이샤 | Image predictive decoding apparatus and method thereof, and image predictive cording apparatus and method thereof |
| KR100249029B1 (en) * | 1996-06-12 | 2000-03-15 | 전주범 | Method and apparatus for coding contour of video signals in image coding system |
| JP3474707B2 (en) * | 1996-07-04 | 2003-12-08 | シャープ株式会社 | Image encoding device and image decoding device |
| DE69725765T2 (en) | 1996-07-31 | 2004-04-29 | Matsushita Electric Industrial Co., Ltd., Kadoma | Image decoder and method for image decoding |
| FR2752474B1 (en) * | 1996-08-14 | 1998-12-31 | Iona Donescu | PROCESS FOR TRANSFORMING THE IMAGE SIGNAL ON ARBITRARY MEDIA |
| KR100235347B1 (en) * | 1996-09-19 | 1999-12-15 | 전주범 | Method and apparatus for encoding a video signal of a contour of an object |
| KR100424678B1 (en) * | 1996-09-25 | 2004-06-18 | 주식회사 팬택앤큐리텔 | Method for assigning coded block pattern with least bit to image after mergence when encoding image using mergence of object boundary blocks |
| KR100501902B1 (en) | 1996-09-25 | 2005-10-10 | 주식회사 팬택앤큐리텔 | Image information encoding / decoding apparatus and method |
| TW366648B (en) * | 1996-10-24 | 1999-08-11 | Matsushita Electric Ind Co Ltd | Method of supplementing pixel signal coding device, and pixel signal decoding device |
| JP4034380B2 (en) * | 1996-10-31 | 2008-01-16 | 株式会社東芝 | Image encoding / decoding method and apparatus |
| US6259738B1 (en) | 1996-10-31 | 2001-07-10 | Kabushiki Kaisha Toshiba | Video encoding apparatus and video decoding apparatus |
| TW358296B (en) | 1996-11-12 | 1999-05-11 | Matsushita Electric Ind Co Ltd | Digital picture encoding method and digital picture encoding apparatus, digital picture decoding method and digital picture decoding apparatus, and data storage medium |
| US6687405B1 (en) * | 1996-11-13 | 2004-02-03 | Koninklijke Philips Electronics N.V. | Image segmentation |
| GB9623573D0 (en) | 1996-11-13 | 1997-01-08 | Philips Electronics Nv | Image segmentation |
| US5864372A (en) * | 1996-12-10 | 1999-01-26 | United Microelectronics Corporation | Apparatus for implementing a block matching algorithm for motion estimation in video image processing |
| KR100239309B1 (en) * | 1997-01-15 | 2000-01-15 | 전주범 | Method and apparatus for coding contour image using vertex coding |
| WO1998035501A2 (en) * | 1997-02-06 | 1998-08-13 | Koninklijke Philips Electronics N.V. | Image segmentation and object tracking method and corresponding system |
| US5978034A (en) * | 1997-02-20 | 1999-11-02 | Sony Corporation | Moving picture encoding method and apparatus, moving picture decoding method and apparatus and recording medium |
| KR100457231B1 (en) * | 1997-03-20 | 2005-08-09 | 주식회사 팬택앤큐리텔 | Moving Compensation Prediction Apparatus and Method for Shape Encoding of Image Signal |
| US6633611B2 (en) * | 1997-04-24 | 2003-10-14 | Mitsubishi Denki Kabushiki Kaisha | Method and apparatus for region-based moving image encoding and decoding |
| US6078695A (en) * | 1997-06-13 | 2000-06-20 | Matsushita Electric Industrial Co., Ltd. | Shape coding method and shape decoding method |
| JP3191922B2 (en) * | 1997-07-10 | 2001-07-23 | 松下電器産業株式会社 | Image decoding method |
| WO1999007156A1 (en) * | 1997-07-28 | 1999-02-11 | Idt International Digital Technologies Deutschland Gmbh | Method and apparatus for multiresolution object-oriented motion estimation |
| KR100249223B1 (en) * | 1997-09-12 | 2000-03-15 | 구자홍 | Method for motion vector coding of mpeg-4 |
| SG116400A1 (en) * | 1997-10-24 | 2005-11-28 | Matsushita Electric Ind Co Ltd | A method for computational graceful degradation inan audiovisual compression system. |
| US6037987A (en) * | 1997-12-31 | 2000-03-14 | Sarnoff Corporation | Apparatus and method for selecting a rate and distortion based coding mode for a coding system |
| US6058214A (en) * | 1998-01-20 | 2000-05-02 | At&T Corp. | Compression of partially masked still images |
| US6714591B1 (en) * | 1998-01-27 | 2004-03-30 | Sharp Kabushiki Kaisha | Video image coding device and video image decoding device |
| US6621932B2 (en) | 1998-03-06 | 2003-09-16 | Matsushita Electric Industrial Co., Ltd. | Video image decoding and composing method and video image decoding and composing apparatus |
| US5838392A (en) * | 1998-03-19 | 1998-11-17 | United Microelectronics Corp. | Adaptive block-matching motion estimator with a compression array for use in a video coding system |
| US6507616B1 (en) * | 1998-10-28 | 2003-01-14 | Lg Information & Communications, Ltd. | Video signal coding method |
| GB2350512A (en) * | 1999-05-24 | 2000-11-29 | Motorola Ltd | Video encoder |
| US6785329B1 (en) * | 1999-12-21 | 2004-08-31 | Microsoft Corporation | Automatic video object extraction |
| CN1193620C (en) * | 2000-01-21 | 2005-03-16 | 诺基亚有限公司 | Motion estimation method and system for video coder |
| US6985527B2 (en) * | 2001-03-07 | 2006-01-10 | Pts Corporation | Local constraints for motion matching |
| FR2811832B1 (en) * | 2000-07-13 | 2003-09-26 | Canon Kk | ADAPTIVE OPTIMIZATION METHODS, DEVICES AND APPARATUS FOR THE TRANSMISSION OF CODED SIGNALS |
| US6842483B1 (en) | 2000-09-11 | 2005-01-11 | The Hong Kong University Of Science And Technology | Device, method and digital video encoder for block-matching motion estimation |
| EP1193978A1 (en) * | 2000-09-29 | 2002-04-03 | Koninklijke Philips Electronics N.V. | Display refresh method |
| CN1284373C (en) * | 2001-07-06 | 2006-11-08 | 皇家菲利浦电子有限公司 | Methods of and units for motion or depth estimation and image processing apparatus provided with such motion estimation unit |
| US7142695B2 (en) * | 2002-11-01 | 2006-11-28 | Pixart Imaging Inc. | Image qualification for optical navigation sensor |
| US7266247B2 (en) | 2002-09-30 | 2007-09-04 | Samsung Electronics Co., Ltd. | Image coding method and apparatus using spatial predictive coding of chrominance and image decoding method and apparatus |
| US9314691B2 (en) | 2002-12-10 | 2016-04-19 | Sony Computer Entertainment America Llc | System and method for compressing video frames or portions thereof based on feedback information from a client device |
| US8949922B2 (en) * | 2002-12-10 | 2015-02-03 | Ol2, Inc. | System for collaborative conferencing using streaming interactive video |
| US9108107B2 (en) * | 2002-12-10 | 2015-08-18 | Sony Computer Entertainment America Llc | Hosting and broadcasting virtual events using streaming interactive video |
| US20090118019A1 (en) | 2002-12-10 | 2009-05-07 | Onlive, Inc. | System for streaming databases serving real-time applications used through streaming interactive video |
| US8964830B2 (en) * | 2002-12-10 | 2015-02-24 | Ol2, Inc. | System and method for multi-stream video compression using multiple encoding formats |
| US9138644B2 (en) | 2002-12-10 | 2015-09-22 | Sony Computer Entertainment America Llc | System and method for accelerated machine switching |
| US9077991B2 (en) | 2002-12-10 | 2015-07-07 | Sony Computer Entertainment America Llc | System and method for utilizing forward error correction with video compression |
| JP4003128B2 (en) * | 2002-12-24 | 2007-11-07 | ソニー株式会社 | Image data processing apparatus and method, recording medium, and program |
| US7418134B2 (en) * | 2003-05-12 | 2008-08-26 | Princeton University | Method and apparatus for foreground segmentation of video sequences |
| CN1655620B (en) * | 2004-02-09 | 2010-09-22 | 三洋电机株式会社 | Image display apparatus |
| JP2005333609A (en) * | 2004-04-22 | 2005-12-02 | Sanyo Electric Co Ltd | Encoding control circuit and encoding circuit |
| US8467447B2 (en) * | 2004-05-07 | 2013-06-18 | International Business Machines Corporation | Method and apparatus to determine prediction modes to achieve fast video encoding |
| US7720295B2 (en) * | 2004-06-29 | 2010-05-18 | Sanyo Electric Co., Ltd. | Method and apparatus for coding images with different image qualities for each region thereof, and method and apparatus capable of decoding the images by adjusting the image quality |
| JP2006020095A (en) * | 2004-07-01 | 2006-01-19 | Sharp Corp | Motion vector detection circuit, image encoding circuit, motion vector detecting method and image encoding method |
| US9578345B2 (en) * | 2005-03-31 | 2017-02-21 | Euclid Discoveries, Llc | Model-based video encoding and decoding |
| US9532069B2 (en) | 2004-07-30 | 2016-12-27 | Euclid Discoveries, Llc | Video compression repository and model reuse |
| US9743078B2 (en) | 2004-07-30 | 2017-08-22 | Euclid Discoveries, Llc | Standards-compliant model-based video encoding and decoding |
| TWI250423B (en) | 2004-07-30 | 2006-03-01 | Ind Tech Res Inst | Method for processing video images |
| JP2006129248A (en) * | 2004-10-29 | 2006-05-18 | Sanyo Electric Co Ltd | Image encoding and decoding method and apparatus thereof |
| JP4656912B2 (en) * | 2004-10-29 | 2011-03-23 | 三洋電機株式会社 | Image encoding device |
| US8054882B2 (en) * | 2005-05-13 | 2011-11-08 | Streaming Networks (Pvt.) Ltd. | Method and system for providing bi-directionally predicted video coding |
| DE102005025634A1 (en) * | 2005-06-03 | 2006-12-07 | Micronas Gmbh | Method and device for determining motion vectors |
| JP2007206644A (en) * | 2006-02-06 | 2007-08-16 | Seiko Epson Corp | Image display system, image display method, image display program, recording medium, data processor, and image display device |
| JP4853199B2 (en) * | 2006-09-25 | 2012-01-11 | 株式会社ニコン | Image compression method, apparatus, electronic camera, and program |
| KR101365574B1 (en) * | 2007-01-29 | 2014-02-20 | 삼성전자주식회사 | Method and apparatus for video encoding, and Method and apparatus for video decoding |
| NZ585904A (en) * | 2007-12-05 | 2013-06-28 | Ol2 Inc | Streaming video game or interactive application video data using forward error correction based on the determined communication channel quality |
| JP2009147807A (en) * | 2007-12-17 | 2009-07-02 | Fujifilm Corp | Image processing apparatus |
| JP2009182623A (en) * | 2008-01-30 | 2009-08-13 | Panasonic Corp | Image encoding method |
| US20090244388A1 (en) * | 2008-03-27 | 2009-10-01 | Siou-Shen Lin | Motion estimation method and related apparatus for determining target motion vector according to motion of neighboring image blocks |
| JP2009267726A (en) * | 2008-04-24 | 2009-11-12 | Panasonic Corp | Moving image encoding apparatus, recorder, moving image encoding method, moving image encoding program |
| KR101517768B1 (en) * | 2008-07-02 | 2015-05-06 | 삼성전자주식회사 | Method and apparatus for encoding video and method and apparatus for decoding video |
| PL3567855T3 (en) * | 2009-03-23 | 2023-03-13 | Ntt Docomo, Inc. | Image predictive encoding device, image predictive encoding method, image predictive encoding program, image predictive decoding device, image predictive decoding method, and image predictive decoding program |
| JP5404918B2 (en) * | 2010-03-15 | 2014-02-05 | パナソニック株式会社 | Moving locus calculating method and apparatus, and region dividing method |
| KR101484281B1 (en) * | 2010-07-09 | 2015-01-21 | 삼성전자주식회사 | Method and apparatus for video encoding using block merging, method and apparatus for video decoding using block merging |
| AU2015201569B2 (en) * | 2010-07-09 | 2016-08-11 | Samsung Electronics Co., Ltd. | Method and apparatus for encoding video by using block merging, and method and apparatus for decoding video by using block merging |
| DK2897365T3 (en) * | 2010-07-09 | 2017-02-13 | Samsung Electronics Co Ltd | Device for decoding video using block merging |
| SG187094A1 (en) | 2010-07-20 | 2013-03-28 | Ntt Docomo Inc | Image prediction encoding device, image prediction encoding method, image prediction encoding program, image prediction decoding device, image prediction decoding method, and image prediction decoding program |
| KR101506446B1 (en) | 2010-12-15 | 2015-04-08 | 에스케이 텔레콤주식회사 | Code Motion Information Generating/Motion Information Reconstructing Method and Apparatus Using Motion Information Merge and Image Encoding/Decoding Method and Apparatus Using The Same |
| JP5247793B2 (en) * | 2010-12-21 | 2013-07-24 | キヤノン株式会社 | Image processing apparatus, image processing method, and program |
| US8797414B2 (en) | 2010-12-23 | 2014-08-05 | Samsung Electronics Co., Ltd. | Digital image stabilization device |
| KR101742449B1 (en) * | 2011-03-08 | 2017-05-31 | 가부시키가이샤 제이브이씨 켄우드 | Video encoding device, video encoding method, video encoding program, video decoding device, video decoding method, and video decoding program |
| CN102685504B (en) * | 2011-03-10 | 2015-08-19 | 华为技术有限公司 | The decoding method of video image, code device, decoding device and system thereof |
| EP3879834A1 (en) | 2011-05-31 | 2021-09-15 | JVCKENWOOD Corporation | Moving image encoding device, moving image encoding method and moving image encoding program, as well as moving image decoding device, moving image decoding method and moving image decoding program |
| CN102905123B (en) * | 2011-07-25 | 2017-04-12 | 乐金电子(中国)研究开发中心有限公司 | Intra-frame image predictive encoding method and video encoder |
| EP2769549A1 (en) * | 2011-10-21 | 2014-08-27 | Dolby Laboratories Licensing Corporation | Hierarchical motion estimation for video compression and motion analysis |
| JP6080277B2 (en) * | 2012-04-24 | 2017-02-15 | リリカル ラブズ ビデオ コンプレッション テクノロジー、エルエルシー | Macroblock partitioning and motion estimation using object analysis for video compression, video encoding method, video encoding computing system and program |
| JP6125808B2 (en) * | 2012-11-19 | 2017-05-10 | 任天堂株式会社 | Data compression apparatus, data compression program, data compression system, and data compression method |
| CA2942336A1 (en) | 2014-03-10 | 2015-09-17 | Euclid Discoveries, Llc | Continuous block tracking for temporal prediction in video encoding |
| US10097851B2 (en) | 2014-03-10 | 2018-10-09 | Euclid Discoveries, Llc | Perceptual optimization for model-based video encoding |
| US10091507B2 (en) | 2014-03-10 | 2018-10-02 | Euclid Discoveries, Llc | Perceptual optimization for model-based video encoding |
| WO2020035022A1 (en) * | 2018-08-17 | 2020-02-20 | Mediatek Inc. | Method and apparatus of simplified merge candidate list for video coding |
| US11503329B2 (en) | 2018-08-17 | 2022-11-15 | Hfi Innovation Inc. | Method and apparatus of simplified sub-mode for video coding |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2512894B2 (en) * | 1985-11-05 | 1996-07-03 | ソニー株式会社 | High efficiency coding / decoding device |
| DE3877105D1 (en) * | 1987-09-30 | 1993-02-11 | Siemens Ag, 8000 Muenchen, De | |
| FR2628864B1 (en) * | 1988-03-21 | 1990-06-15 | France Etat | METHOD FOR SEGMENTATION OF A FIELD OF SPEED VECTORS, IN PARTICULAR OF SPEEDS FOR MOVING POINTS OF AN IMAGE IN A SEQUENCE OF IMAGES |
| FR2633468B1 (en) * | 1988-06-24 | 1990-11-09 | France Etat | METHOD OF ENCODING ASSISTANCE DATA FOR THE RECONSTRUCTION OF SUB-SAMPLE ANIMATED ELECTRONIC IMAGES |
| DE3933346C1 (en) * | 1989-10-06 | 1991-04-04 | Ant Nachrichtentechnik Gmbh, 7150 Backnang, De | |
| US5164828A (en) * | 1990-02-26 | 1992-11-17 | Sony Corporation | Video signal transmission and method and apparatus for coding video signal used in this |
| JPH082107B2 (en) * | 1990-03-02 | 1996-01-10 | 国際電信電話株式会社 | Method and apparatus for moving picture hybrid coding |
| JP2514114B2 (en) * | 1991-01-31 | 1996-07-10 | 株式会社グラフィックス・コミュニケーション・テクノロジーズ | Video coding device |
| JP2514115B2 (en) * | 1991-02-15 | 1996-07-10 | 株式会社グラフィックス・コミュニケーション・テクノロジーズ | Image signal encoder |
| JP3068304B2 (en) * | 1992-01-21 | 2000-07-24 | 日本電気株式会社 | Video coding and decoding systems |
| US5241383A (en) * | 1992-05-13 | 1993-08-31 | Bell Communications Research, Inc. | Pseudo-constant bit rate video coding with quantization parameter adjustment |
| US5440346A (en) * | 1993-06-16 | 1995-08-08 | Intel Corporation | Mode selection for method and system for encoding images |
| US5422963A (en) * | 1993-10-15 | 1995-06-06 | At&T Corp. | Block transform coder for arbitrarily shaped image segments |
-
1994
- 1994-10-13 US US08/322,893 patent/US5608458A/en not_active Expired - Lifetime
-
1995
- 1995-08-25 CA CA002156944A patent/CA2156944A1/en not_active Abandoned
- 1995-10-03 EP EP95307030A patent/EP0707427A3/en not_active Withdrawn
- 1995-10-13 JP JP7265045A patent/JPH08214318A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110300977A (en) * | 2017-02-17 | 2019-10-01 | 考吉森公司 | Method for image procossing and video compress |
| CN110300977B (en) * | 2017-02-17 | 2024-04-16 | 英特尔公司 | Method for image processing and video compression |
Also Published As
| Publication number | Publication date |
|---|---|
| US5608458A (en) | 1997-03-04 |
| EP0707427A2 (en) | 1996-04-17 |
| EP0707427A3 (en) | 1997-10-29 |
| JPH08214318A (en) | 1996-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2156944A1 (en) | Method and apparatus for a region-based approach to coding a sequence of video images | |
| US11783512B2 (en) | Attribute value of reconstructed position associated with plural original points | |
| US11375208B2 (en) | Trisoup node size per slice | |
| EP1256238B1 (en) | Method for encoding and decoding video information, a motion compensated video encoder and a corresponding decoder | |
| Gerken | Object-based analysis-synthesis coding of image sequences at very low bit rates | |
| Chen et al. | Block transform coder for arbitrarily shaped image segments | |
| AU2020397870A1 (en) | Method and apparatus for point cloud coding | |
| CA2295689C (en) | Apparatus and method for object based rate control in a coding system | |
| Pickering et al. | A perceptually efficient VBR rate control algorithm | |
| JP2022502961A (en) | Motion estimation using 3D auxiliary data | |
| US20020009143A1 (en) | Bandwidth scaling of a compressed video stream | |
| US20040156437A1 (en) | Method for encoding and decoding video information, a motion compensated video encoder and a corresponding decoder | |
| US11451802B2 (en) | Method and apparatus for point cloud coding | |
| US20050053154A1 (en) | Video encoding apparatus and video decoding apparatus | |
| EP0649258A2 (en) | Block transform coder for arbitrarily shaped image segments | |
| EP3066832B1 (en) | Adaptive prediction of coefficients of a video block | |
| US6680974B1 (en) | Methods and apparatus for context selection of block transform coefficients | |
| EP1158806A1 (en) | Motion vector coding | |
| US11514610B2 (en) | Method and apparatus for point cloud coding | |
| EP0734168A2 (en) | Apparatus for encoding a moving image signal having a still object | |
| KR100571920B1 (en) | Video encoding method for providing motion compensation method based on mesh structure using motion model and video encoding apparatus therefor | |
| US20230245390A1 (en) | Manhattan layout estimation using geometric and semantic information | |
| Chen et al. | Linear rate-distortion models for MPEG-4 shape coding | |
| JPH09231377A (en) | Outline approximating method and outline encoding device | |
| Ostermann et al. | Coding of color parameters in an analysis-synthesis coder based on moving 3D-objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |