WO2007081899A2 - Techniques for creating facial animation using a face mesh - Google Patents
Techniques for creating facial animation using a face mesh Download PDFInfo
- Publication number
- WO2007081899A2 WO2007081899A2 PCT/US2007/000437 US2007000437W WO2007081899A2 WO 2007081899 A2 WO2007081899 A2 WO 2007081899A2 US 2007000437 W US2007000437 W US 2007000437W WO 2007081899 A2 WO2007081899 A2 WO 2007081899A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- face
- face mesh
- mesh
- image
- sequence
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
Definitions
- Embodiments of the present invention generally relate to computer graphics and more specifically to techniques for creating facial animation using a face mesh.
- changes in the facial expression may be manually entered to create the animation. For example, conditions need to be entered for the exact form, the exact timing of the motion of each part, and the change of the lightness and darkness on the skin that accompanies the change of form in the face. This requires skill and a lot of time. Even if the manually-entered changes create a realistic facial animation, the amount of time to input those changes may not make it worth it. Further, it is more difficult to create an animation when using only a single image of a user as input. Even when changes are manually entered, the resulting facial expression may not be realistic.
- Embodiments of the present invention generally relate to techniques for creating facial animation using a face mesh.
- the techniques include determining a face mesh sequence. A portion of the face mesh sequence is then fitted to an image of the face. For example, a first frame of the face mesh sequence is fitted to the image. The face image is then projected onto the portion of the face mesh based on the fitting. Also, a hair model may be generated and fitted onto the face mesh sequence. Additionally, color differences between sequences of the face mesh may be computed. The face mesh sequence is then played to create a facial animation sequence for the face image. The color differences that were created may also be added to the sequences as they are being played. Accordingly, the playing of the face mesh sequence creates a realistic facial animation of the image of the face.
- a method for creating facial animation comprises: determining a face mesh sequence; fitting a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; projecting the face image onto the face mesh as texture based on the fitting; adding the displacement values to vertices in subsequent face meshes in the face mesh sequence; and playing the face mesh sequence to create facial animation for the face image.
- a device for creating facial animation comprises: logic configured to determine a face mesh sequence; logic configured to fit a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; logic configured to project the face image onto the face mesh as texture based on the fitting; logic configured to add the displacement values for vertices in subsequent face meshes in the face mesh sequence; and logic configured to play the face mesh sequence to create facial animation for the face image.
- Fig. 1 depicts a simplified flow chart of a method for creating facial animation according to one embodiment of the present invention.
- FIG. 2 depicts a simplified flow chart of a method for fitting a face mesh of a face mesh sequence to the face image.
- FIG. 3 shows an example of a face image and feature points according to one embodiment of the present invention.
- FIG. 4 depicts a face mesh with feature points according to one embodiment of the present invention.
- Fig. 5 depicts an initial fitting according to one embodiment of the present invention.
- Fig. 6 depicts a face mesh that has been fitted by scaling, translation, and rotation.
- FIG. 7 shows the matching of face rnesh feature points onto face image feature points according to one embodiment of the present invention.
- Fig. 8 shows an image as projected as texture onto the face mesh.
- FIG. 9 depicts a simplified flow chart of a method for generating a hair model according to one embodiment of the present invention.
- Fig. 10 shows a mask image of a hair region according to one embodiment of the present invention.
- Fig. 11 shows a hair model projected onto the face mesh.
- Fig. 12 depicts a simplified flow chart of a method for computing the color differences according to one embodiment of the present invention.
- Fig. 13 shows the differences in color according to one embodiment of the present invention.
- Fig. 14 shows the differences when not using the differences in color for the vertices and using the differences in color.
- Fig. 15 shows a simplified flow chart for creating facial animation according to one embodiment of the present invention.
- Fig. 16 shows the result of the created animation according to one embodiment.
- Fig. 17 shows images with color differences added and without according to one embodiment.
- a face mesh sequence may be used to create facial animation.
- a face mesh sequence may be a set of face form meshes that are acquired between each time of expression changes or a deliverance of speech.
- the face mesh sequence is a grade, e.g., more than 20 frames per second, that looks smooth at the time of reproduction.
- Each face mesh of the sequence can express a form of the face in detail, such as about 20,000 vertices. The position, the amount of movements, and color data for each vertex can be acquired for each frame of the face mesh sequence.
- the face mesh sequence of a user is acquirable using known systems.
- the face mesh sequence may be of the user in which an image will be used to create a facial animation.
- a face mesh of another user may be used.
- the face mesh of the other user may then be modified to fit the image of the user.
- Fig. 1 depicts a simplified flow chart 100 of a method for creating facial animation according to one embodiment of the present invention.
- Step 102 determines a face mesh sequence.
- the face mesh sequence may be captured from a user.
- Step 104 fits a face mesh of a face mesh sequence to a face image.
- the face image may be an image (e.g. a picture) of the same user in which the face mesh sequence was captured. However, the face mesh sequence and face image may be captured from different users in other embodiments.
- the face mesh may be the first frame of the face mesh sequence. This first frame may correspond to the face image.
- the fitting process may deform the face mesh. Displacement values may be determined based on the face mesh and the deformed face mesh. These values are saved and will be used to deform all the face meshes in the face mesh sequence later in the playing of the face mesh sequence.
- Step 106 generates a hair model from the face mesh.
- the hair model creates a face mesh area for the hair. This process will be described in more detail below.
- Step 108 computes the difference in color for face meshes in the face mesh sequence.
- the difference in color may be computed for vertices in different face meshes in the sequence. The effect on including the differences in color will be described in more detail below but the result is that the change in lightness and darkness accompanying a change in form are displayed in the facial animation.
- Step 110 projects the face image on the face mesh and the hair mesh.
- the face mesh included shows the face image in addition to the hair.
- Step 112 plays the face mesh sequence with the hair model using displacement values and the computed differences in color.
- the displacement values may be added to all the face meshes in the face mesh sequence. Also, the color differences are added. Then, the face mesh sequence is played. In playing the face mesh sequence, an animation of the face image is provided. By adding the computed difference in color, change in lightness and darkness are shown.
- Fig.2 depicts a simplified flow chart 200 of a method for fitting a face mesh of a face mesh sequence to the face image. Although this method is described, it will be recognized that variations of the method will be appreciated.
- Step 202 loads a face image and feature points for the face image.
- the feature points may be any points on the face image.
- the feature points may be selected in places where distinct features of a face image are found, such as the eyes, mouth, nose, etc.
- the feature points may be acquired using known methods.
- Fig. 3 shows an example of a face image and feature points according to one embodiment of the present invention.
- feature points 302 are shown as white dots in face image 300.
- Feature points are placed near the eyes, mouth, ears, and chin in one embodiment. Although these feature points 302 are shown, it will be recognized that other feature points 302 may also be used.
- Step 204 places face image 300 on an XY plane.
- the XY plane may be displayed on any computing device.
- Step 206 loads a face mesh of the face mesh sequence and feature points of the face mesh.
- a face mesh that is loaded may be a portion of the face mesh sequence, such as a frame of the face mesh sequence.
- the first frame of the face mesh sequence is taken from the user found in face image 300.
- the face mesh may be a 3D mesh capture from the face mesh sequence.
- the feature points found on the face mesh may be determined using the same algorithm used in finding the feature points in face image 300.
- Fig. 4 depicts a face mesh 400 with feature points 402 according to one embodiment of the present invention.
- Feature points 402 may be vertices (e.g., a point in 3D space with a particular location) in face mesh 400.
- Feature points 402 may be determined by the user or by a process.
- Feature points 402 are placed near the eyes, mouth, ears, and chin in one embodiment.
- Step 208 roughly aligns face mesh 400 to face image 300.
- the alignment may be performed automatically or manually.
- Fig. 5 depicts an initial fitting according to one embodiment of the present invention.
- face mesh 400 has been roughly fitted over the face in face image 300.
- face mesh 400 does not completely cover the face.
- Step 210. minimizes a sum of the distance between the projected face mesh feature points 402 to the image plane and the image feature points 302 by scaling. Known equations may be used to minimize the distance between face mesh feature points 402 and image feature points 302. Scaling may, for example, increase or decrease the size of face mesh 400 to fit face image 300.
- Step 212 minimizes the sum of a distance between the projected face mesh feature points 402 to the image plane and the face image feature points 302 using translation and rotation.
- Known equations may be used to minimize the distance between face mesh feature points 402 and image feature points 302.
- the translation may move face mesh 400 to match face image 300.
- face mesh 400 may be rotated to match face image 300.
- Step 214 determines if the minimization is converged. That is, it is determined if the sum of distances between feature points 302 and projected points of the corresponding feature points 402 to the image plane is minimized. Steps 210 and 212 are repeated if it is determined that feature points 302 and feature points 402 are not converged.
- step 216 saves the parameters of scaling, translation, and rotation that were determined in steps 210 and 212. These parameters are the parameters that are used to scale, translate, and rotate face mesh 400.
- Step 218 moves feature points 402 of the face mesh 400 on the XY plane to match feature points 302 of face image 300. Minimizing the sum between face mesh feature points 402 and face image feature points 302 may not make them exactly match. Thus, this adjusting step is performed to match them.
- Step 220 moves all vertices according to the displacement of face mesh feature points 402 and face image feature points 402. These displacements can be computed using a radial basis function based on the displacement of the feature points obtained in step 222.
- a radial basis function is an interpolation method that can be used to deform an object in computer graphics. Other functions may also be appreciated.
- Fig. 6 depicts face mesh 400 that has been fitted by scaling, translation, and rotation. As shown, face mesh 400 roughly corresponds to the user's face. Face mesh 400 has been scaled by being made larger and also rotated and translated to match face image 300. However, face mesh feature points 402 may not exactly match image feature points 302. For example, around the eyes, feature points 402 do not exactly match feature points 302. Feature points 402 are then moved to match feature points 302. Also, the other vertices of face mesh 400 are then moved based on the movement of face mesh feature points 402.
- FIG. 7 shows the matching of feature points 402 onto feature points 302 of face image 300 according to one embodiment of the present invention. As shown, face mesh feature points 402 and image feature points 302 match each other.
- Step 222 saves the displacement values for the vertices between the original and deformed face mesh 400.
- the displacement values will be used when the face mesh sequence is later played.
- the displacement values are used align face mesh 400 to face image 300.
- Step 224 projects face image 300 as texture onto face mesh 400. Accordingly, the user's face image is projected onto the face mesh.
- Fig. 8 shows face image 300 as projected as texture onto face mesh 300. As shown, the face is shown as texture over face mesh 400.
- a hair model may also be generated.
- the hair model is generated because the face mesh may not have modeled the hair.
- a hair model is generated and included in the face meshes such that hair from face image 300 can be projected as texture on it.
- Fig. 9 depicts a simplified flow chart 900 of a method for generating a hair model according to one embodiment of the present invention.
- Step 902 loads face mesh 400 onto a square grid on an XY plane.
- the XY plane may be 100x100 grids, for example.
- Face mesh 400 is a 3D image and the XY plane is 2D.
- face mesh 400 needs to be places at some position in the Z space.
- face mesh 400 may be loaded such that the Z values of it are towards the positive axis (i.e., the XY plane is at the back of the face mesh or the front part of the face is facing forward from the XY plane if looking at a display screen).
- the hair model can also be created using 3D polygons rather than the square grid.
- step 904 determines for each grid point, if it is occluded (i.e., obstructed) by face mesh 702. If the grid point is occluded by face mesh 702, then step 906 moves the grid point to a point where it intersects with face mesh 400. For example, the grid point is moved in the positive direction (i.e., forward) until it intersects with the Z value of a corresponding face mesh 400 value.
- Step 908 determines if all grid points have been processed. If not, the process reiterates to step 904 to process another grid point.
- step 910 loads a mask image 1002 of a hair region.
- the hair region may be modeled using known methods.
- Fig. 10 shows a mask image 1002 of a hair region according to one embodiment of the present invention.
- mask image 1002 of the hair region models a hair region that may be captured.
- the outline of the hair in face image 300 may be taken.
- Mask image 1002 may then be set on face mesh 400.
- face mesh 400 There may be some areas of face mesh 400 that include hair areas. These areas are raised slightly in the positive Z direction for a point on face mesh 400. This is because in animation, hair may be raised from the skin for life-like animation. It will be recognized that the hair does not need to be raised in some embodiments.
- Step 912 determines for each grid point, if it is occluded by face mesh 400. If the grid point is not occluded by the face mesh, the process continues to step 918. Further, if the mask value in step 914 is not at a point that indicates it is hair, then the process proceeds to step 918, which determines if all grid points have been processed. If not, the process reiterates to step 912. [60] If the grid point is occluded by face mesh 400, step 914 determines if the mask value at the point indicates it is hair. For example, a value may be computed on face mesh 400 of the hair in a position corresponding to a grid pixel.
- step 916 if the value is 1 , the Z value of the point may be increased slightly.
- the vertex is projected to the mask image 1002 and the mask value is checked there. If the value is 1, the vertex is a part of hair. This is because the point is part of mask image 1002, which is the hair region.
- Step 918 determines if all grid points have.been processed. If not, the process reiterates to step 912.
- Step 920 determines for each grid point P if it is part of mask image 1002 that indicates a mask value of "hair" and is not occluded by face mesh 400. That is, is the grid point part of mask image 1002 but not part of face mesh 400. If not, nothing is performed and the process proceeds to step 928, described below.
- Step 922 computes the nearest occluded point (O).
- Step 924 determines along the line OP (but not in the segment OP), the boundary point, Q, between hair and not hair.
- Pz 5 Oz, and Qz are the Z values of P, O, and Q, respectively.
- w 1-OP/OQ A 2. This interpolates a Z value for the grid point P. It will be recognized that other methods of interpolating a Z value for grid point P will be appreciated.
- Step 928 determines if all grid points have been processed. If not, the process reiterates to step 920.
- the hair model is a mesh whose vertices are the grid points.
- Fig. 11 shows a hair model 1100.
- Fig. 12 depicts a simplified flow chart 1200 of a method for computing the color differences according to one embodiment of the present invention.
- Step 1202 loads the face mesh sequence.
- Step 1204 sets face mesh 400 (e.g., the first face mesh of a face mesh sequence) as a base face mesh.
- the base face mesh may be a first frame of the face mesh sequence.
- the base face mesh may be an expression that is close to the expression found in face image 300. Also, in another embodiment, the base face mesh may be the mean shape of all the face meshes.
- Step 1206 computes the color differences for each vertex between the base mesh and other face meshes in the sequence.
- the color differences may be computed using known methods.
- the difference may be a value in which the color should be increased or decreased from the base face mesh to another face mesh.
- a color difference may be calculated based on RGB (red, green, blue) model.
- the color difference may be the color value of a subsequent face mesh is subtracted from the color value of the base face mesh at a vertex.
- a positive color difference may be a lighter color and a negative color difference is a darker color.
- any positive color differences may be ignored in one embodiment. These differences may create white spots on an image. Thus, only negative color differences may be used. Also, if the resulting texture caused by the color differences does not look good (such as around the mouth or eye corners), color differences may be ignored in those regions.
- the color differences can be represented as grey scale rather than the difference in each color channel, such as an RGB color scheme.
- An exaggerated face in terms of color
- a 3D version of the expression ratio image can be used to represent the color differences.
- the option of adding color differences can be represented as texture blending rather than a vertex-wise operation. In this case, the color differences are rendered first. The resulting images are then added to the rendered image of the face mesh. This is kind of a image blending. The color differences can be added for each vertex (which was described earlier). Or, the color difference can be rendered in a "not difference-added" mesh and the same mesh colored with the color difference separately, and then both rendered images are added pixelwise. Either way may add color differences.
- Fig. 13 shows the differences in color according to one embodiment of the present invention.
- a top row 1302 shows a face mesh without color.
- Row 1304 shows an image that includes color.
- Row 1304 shows what an original capture of a sequence would look like for reference.
- Each face mesh 1306 and a corresponding face image 1308 are shown. Differences in color between face meshes 1306 may be calculated. For example, the color differences between face mesh 1306-1 and face mesh 1306-2, the difference between face mesh 1306-1 and face mesh 1306-3, etc. may be calculated.
- Step 1208 then stores the color differences for each vertex. The color differences will be used when the face mesh sequence is later played.
- Fig. 14 shows the differences in color when a face squints in the original face mesh.
- image 1404 shows the changes in color that occur in the expression when the expression is changed from image 1402.
- circle 1408 shows a darkness that occurs for a change in form from the image in circle 1406.
- Fig. 15 shows a simplified flow chart 1500 for creating facial animation according to one embodiment of the present invention.
- Step 1502 loads the face mesh sequence and the hair model.
- Step 1504 applies the scale, translation, and rotation for each face mesh in the face mesh sequence. These values were saved in step 216 of Fig. 2. This aligns the face meshes to face image 300 in the face mesh sequence.
- Step 1506 projects face image 302 to the base mesh and the hair model as texture. Thus, the face image is shown on the face mesh.
- Step 1508 adds the displacement of the vertices to all meshes. This is the displacement values that were saved in step 222 of Fig. 2. This matches the feature points and vertices to face image 300.
- the displacement difference from the base mesh to the other meshes may be used instead of having vertex positions and displacement values for every mesh.
- the base mesh may have a first displacement value for a vertex. The difference in value for the next face mesh may be stored instead of a new value.
- Step 1510 adds color differences that were saved in step 1206 of Fig. 12 to all face meshes.
- the color difference from the base mesh to the other meshes may be used instead of having vertex positions and color differences for every mesh.
- the base mesh may have a first color difference for a vertex.
- the difference in value for the next face mesh may be stored instead of a new value.
- Step 1512 then plays the face mesh sequence.
- Simple, natural, and very realistic face animation is created by playing the face mesh sequence. This animation is provided using a base image of only one sheet (such as a frame or an image) and the face mesh sequence.
- Fig. 16 shows the result of the created animation according to one embodiment. As shown, animation of the face looks very realistic. Different expressions are created on face image 300 using the face mesh sequence.
- Fig. 17 shows images with color differences added and without according to one embodiment.
- Image 1702 shows an image with the color differences when added and image 1704 shows an image without color differences.
- the expression in image 1702 shows the change in light and darkness that are created from the change in form of the facial expression.
- ridges and creases are more defined in image 1702.
- speech animations may be created using the above methods.
- a lip sync animation may be created by synchronizing face meshes of phonemes (called visimes) and speech audio.
- visimes face meshes of phonemes
- routines of embodiments of the present invention can be implemented using C, C++, Java, assembly language, etc.
- Different programming techniques can be employed such as procedural or object oriented.
- the routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different embodiments. In some embodiments, multiple steps shown as sequential in this specification can be performed at the same time.
- the sequence of operations described herein can be interrupted, suspended, or otherwise controlled by another process, such as an operating system, kernel, etc.
- the routines can operate in an operating system environment or as stand-alone routines occupying all, or a substantial part, of the system processing. Functions can be performed in hardware, software, or a combination of both. Unless otherwise stated, functions may also be performed manually, in whole or in part.
- a "computer-readable medium” for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, system or device.
- the computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
- Embodiments of the present invention can be implemented in the form of control logic in software or hardware or a combination of both.
- the control logic may be stored in an information storage medium, such as a computer-readable medium, as a plurality of instructions adapted to direct an information processing device to perform a set of steps disclosed in embodiments of the present invention.
- an information storage medium such as a computer-readable medium
- a person of ordinary skill in the art will appreciate other ways and/or methods to implement the present invention.
- a "processor” or “process” includes any human, hardware and/or software system, mechanism or component that processes data, signals or other information.
- a processor can include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor can perform its functions in "real time,” “offline,” in a “batch mode,” etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems.
- Embodiments of the invention may be implemented by using a programmed general purpose digital computer, by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nanoengineered systems, components and mechanisms may be used.
- the functions of embodiments of the present invention can be achieved by any means as is known in the art.
- Distributed, or networked systems, components and circuits can be used.
- Communication, or transfer, of data may be wired, wireless, or by any other means.
Abstract
Techniques for creating facial animation using a face mesh are provided. The techniques include determining a face mesh sequence. A portion of the face mesh sequence is then fitted to an image of the face. For example, a first frame of the face mesh sequence is fitted to the image. The face image is then projected onto the portion of the face mesh based on the fitting. Also, a hair model may be generated and fitted onto the face mesh sequence. Additionally, changes in color differences between sequences of the face mesh may be computed. The face mesh sequence is then played to create a facial animation sequence for the face image. The color differences that were created may also be added to the sequences as they are being played. Accordingly, the playing of the face mesh sequence creates a realistic facial animation of the image of the face.
Description
PATENT APPLICATION
TECHNIQUES FOR CREATING FACIAL ANIMATION
USING A FACE MESH
Cross References to Related Applications
This application claims priority from U.S. Provisional Patent Application No. 60/758,122, entitled "Method and Apparatus for Creating Facial Animation", filed January 10, 2006, and U.S. Patent Application No. 11/396,159, entitled "Techniques for Creating Facial Animation Using a Face Mesh", filed March 31, 2006 which are incorporated by reference in their entirety for all purposes.
Background of the Invention
[01] Embodiments of the present invention generally relate to computer graphics and more specifically to techniques for creating facial animation using a face mesh.
[02] Automated generation of realistic face animation is one of the most difficult problems in the computer graphics field. For example, creating a realistic natural expression for the changes of a person laughing (or performing any other action) is very difficult. This is because most people are very sensitive to a minute change in a facial expression. When people look at computer-generated graphics where the minute changes are not animated precisely, the animation does not look realistic.
[03] To create facial animation, changes in the facial expression may be manually entered to create the animation. For example, conditions need to be entered for the exact form, the exact timing of the motion of each part, and the change of the lightness and darkness on the skin that accompanies the change of form in the face. This requires skill and a lot of time. Even if the manually-entered changes create a realistic facial animation, the amount of time to input those changes may not make it worth it. Further, it is more difficult to create an animation when using only a single image of a user as input. Even
when changes are manually entered, the resulting facial expression may not be realistic.
Summary of Embodiments of the Invention
[04] Embodiments of the present invention generally relate to techniques for creating facial animation using a face mesh. In one embodiment, the techniques include determining a face mesh sequence. A portion of the face mesh sequence is then fitted to an image of the face. For example, a first frame of the face mesh sequence is fitted to the image. The face image is then projected onto the portion of the face mesh based on the fitting. Also, a hair model may be generated and fitted onto the face mesh sequence. Additionally, color differences between sequences of the face mesh may be computed. The face mesh sequence is then played to create a facial animation sequence for the face image. The color differences that were created may also be added to the sequences as they are being played. Accordingly, the playing of the face mesh sequence creates a realistic facial animation of the image of the face.
[05] In one embodiment, a method for creating facial animation is provided. The method comprises: determining a face mesh sequence; fitting a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; projecting the face image onto the face mesh as texture based on the fitting; adding the displacement values to vertices in subsequent face meshes in the face mesh sequence; and playing the face mesh sequence to create facial animation for the face image.
[06] In one embodiment, a device for creating facial animation is provided. The device comprises: logic configured to determine a face mesh sequence; logic configured to fit a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; logic configured to project the face image onto the face mesh as texture based on the fitting; logic configured to add the displacement values for vertices in subsequent face meshes in the face mesh sequence; and logic configured to play the face mesh sequence to create facial animation for the face image.
[07] A further understanding of the nature and the advantages of the inventions
disclosed herein may be realized by reference of the remaining portions of the specification and the attached drawings.
Brief Description of the Drawings
[08] Fig. 1 depicts a simplified flow chart of a method for creating facial animation according to one embodiment of the present invention.
[09] Fig. 2 depicts a simplified flow chart of a method for fitting a face mesh of a face mesh sequence to the face image.
[10] Fig. 3 shows an example of a face image and feature points according to one embodiment of the present invention.
[11] Fig. 4 depicts a face mesh with feature points according to one embodiment of the present invention.
[12] Fig. 5 depicts an initial fitting according to one embodiment of the present invention.
[13] Fig. 6 depicts a face mesh that has been fitted by scaling, translation, and rotation.
[14] Fig. 7 shows the matching of face rnesh feature points onto face image feature points according to one embodiment of the present invention.
[15] Fig. 8 shows an image as projected as texture onto the face mesh.
[16] Fig. 9 depicts a simplified flow chart of a method for generating a hair model according to one embodiment of the present invention.
[17] Fig. 10 shows a mask image of a hair region according to one embodiment of the present invention.
[18] Fig. 11 shows a hair model projected onto the face mesh.
[19] Fig. 12 depicts a simplified flow chart of a method for computing the color differences according to one embodiment of the present invention.
[20] Fig. 13 shows the differences in color according to one embodiment of the present invention.
[21] Fig. 14 shows the differences when not using the differences in color for the vertices and using the differences in color.
[22] Fig. 15 shows a simplified flow chart for creating facial animation according to one embodiment of the present invention.
[23] Fig. 16 shows the result of the created animation according to one embodiment.
[24] Fig. 17 shows images with color differences added and without according to one embodiment.
Detailed Description of Embodiments of the Invention
Overview
[25] A face mesh sequence may be used to create facial animation. A face mesh sequence may be a set of face form meshes that are acquired between each time of expression changes or a deliverance of speech. The face mesh sequence is a grade, e.g., more than 20 frames per second, that looks smooth at the time of reproduction. Each face mesh of the sequence can express a form of the face in detail, such as about 20,000 vertices. The position, the amount of movements, and color data for each vertex can be acquired for each frame of the face mesh sequence.
[26] In one embodiment, the face mesh sequence of a user (e.g. a human person) is acquirable using known systems. The face mesh sequence may be of the user in which an image will be used to create a facial animation. However, if a face mesh of the user cannot be taken, a face mesh of another user may be used. The face mesh of the other user may then be modified to fit the image of the user.
[27] Fig. 1 depicts a simplified flow chart 100 of a method for creating facial animation according to one embodiment of the present invention. Step 102 determines a face mesh sequence. The face mesh sequence may be captured from a user.
[28] Step 104 fits a face mesh of a face mesh sequence to a face image. The face image may be an image (e.g. a picture) of the same user in which the face mesh sequence was captured. However, the face mesh sequence and face image may be captured from different users in other embodiments. The face mesh may be the first frame of the face mesh sequence. This first frame may correspond to the face image.
[29] The fitting process may deform the face mesh. Displacement values may be determined based on the face mesh and the deformed face mesh. These values are saved and will be used to deform all the face meshes in the face mesh sequence later in the playing of the face mesh sequence.
[30] Step 106 generates a hair model from the face mesh. The hair model creates a face mesh area for the hair. This process will be described in more detail below.
[31] Step 108 computes the difference in color for face meshes in the face mesh sequence. The difference in color may be computed for vertices in different face meshes in the sequence. The effect on including the differences in color will be described in more detail below but the result is that the change in lightness and darkness accompanying a change in form are displayed in the facial animation.
[32] Step 110 projects the face image on the face mesh and the hair mesh. Thus, the face mesh included shows the face image in addition to the hair.
[33] Step 112 plays the face mesh sequence with the hair model using displacement values and the computed differences in color. The displacement values may be added to all the face meshes in the face mesh sequence. Also, the color differences are added. Then, the face mesh sequence is played. In playing the face mesh sequence, an animation of the face image is provided. By adding the computed difference in color, change in lightness and darkness are shown.
Face Mesh Fitting Process
[34] Fig.2 depicts a simplified flow chart 200 of a method for fitting a face mesh of a face mesh sequence to the face image. Although this method is described, it will be
recognized that variations of the method will be appreciated.
[35] Step 202 loads a face image and feature points for the face image. The feature points may be any points on the face image. For example, the feature points may be selected in places where distinct features of a face image are found, such as the eyes, mouth, nose, etc. The feature points may be acquired using known methods.
[36] Fig. 3 shows an example of a face image and feature points according to one embodiment of the present invention. As shown, feature points 302 are shown as white dots in face image 300. Feature points are placed near the eyes, mouth, ears, and chin in one embodiment. Although these feature points 302 are shown, it will be recognized that other feature points 302 may also be used.
[37] Step 204 places face image 300 on an XY plane. The XY plane may be displayed on any computing device.
[38] Step 206 loads a face mesh of the face mesh sequence and feature points of the face mesh. A face mesh that is loaded may be a portion of the face mesh sequence, such as a frame of the face mesh sequence. In one embodiment, the first frame of the face mesh sequence is taken from the user found in face image 300. The face mesh may be a 3D mesh capture from the face mesh sequence. The feature points found on the face mesh may be determined using the same algorithm used in finding the feature points in face image 300.
[39] Fig. 4 depicts a face mesh 400 with feature points 402 according to one embodiment of the present invention. Feature points 402 may be vertices (e.g., a point in 3D space with a particular location) in face mesh 400. Feature points 402 may be determined by the user or by a process. Feature points 402 are placed near the eyes, mouth, ears, and chin in one embodiment.
[40] Step 208 roughly aligns face mesh 400 to face image 300. The alignment may be performed automatically or manually. Fig. 5 depicts an initial fitting according to one embodiment of the present invention. As shown, face mesh 400 has been roughly fitted over the face in face image 300. As shown, face mesh 400 does not completely cover the face. Thus, further adjusting may be required.
[41] Step 210. minimizes a sum of the distance between the projected face mesh feature points 402 to the image plane and the image feature points 302 by scaling. Known equations may be used to minimize the distance between face mesh feature points 402 and image feature points 302. Scaling may, for example, increase or decrease the size of face mesh 400 to fit face image 300.
[42] Step 212 minimizes the sum of a distance between the projected face mesh feature points 402 to the image plane and the face image feature points 302 using translation and rotation. Known equations may be used to minimize the distance between face mesh feature points 402 and image feature points 302. The translation may move face mesh 400 to match face image 300. Similarly, face mesh 400 may be rotated to match face image 300.
[43] Step 214 determines if the minimization is converged. That is, it is determined if the sum of distances between feature points 302 and projected points of the corresponding feature points 402 to the image plane is minimized. Steps 210 and 212 are repeated if it is determined that feature points 302 and feature points 402 are not converged.
[44] If they are converged, step 216 saves the parameters of scaling, translation, and rotation that were determined in steps 210 and 212. These parameters are the parameters that are used to scale, translate, and rotate face mesh 400.
[45] Step 218 moves feature points 402 of the face mesh 400 on the XY plane to match feature points 302 of face image 300. Minimizing the sum between face mesh feature points 402 and face image feature points 302 may not make them exactly match. Thus, this adjusting step is performed to match them.
[46] Step 220 moves all vertices according to the displacement of face mesh feature points 402 and face image feature points 402. These displacements can be computed using a radial basis function based on the displacement of the feature points obtained in step 222. A radial basis function is an interpolation method that can be used to deform an object in computer graphics. Other functions may also be appreciated.
[47] Fig. 6 depicts face mesh 400 that has been fitted by scaling, translation, and rotation. As shown, face mesh 400 roughly corresponds to the user's face. Face mesh 400
has been scaled by being made larger and also rotated and translated to match face image 300. However, face mesh feature points 402 may not exactly match image feature points 302. For example, around the eyes, feature points 402 do not exactly match feature points 302. Feature points 402 are then moved to match feature points 302. Also, the other vertices of face mesh 400 are then moved based on the movement of face mesh feature points 402.
[48] Fig. 7 shows the matching of feature points 402 onto feature points 302 of face image 300 according to one embodiment of the present invention. As shown, face mesh feature points 402 and image feature points 302 match each other.
[49] Step 222 saves the displacement values for the vertices between the original and deformed face mesh 400. The displacement values will be used when the face mesh sequence is later played. The displacement values are used align face mesh 400 to face image 300.
[50] Step 224 projects face image 300 as texture onto face mesh 400. Accordingly, the user's face image is projected onto the face mesh. Fig. 8 shows face image 300 as projected as texture onto face mesh 300. As shown, the face is shown as texture over face mesh 400.
Hair Modeling Process
[51] After fitting face mesh 400 to face image 300, a hair model may also be generated. The hair model is generated because the face mesh may not have modeled the hair. Thus, a hair model is generated and included in the face meshes such that hair from face image 300 can be projected as texture on it. Fig. 9 depicts a simplified flow chart 900 of a method for generating a hair model according to one embodiment of the present invention.
[52] Step 902 loads face mesh 400 onto a square grid on an XY plane. The XY plane may be 100x100 grids, for example. Face mesh 400 is a 3D image and the XY plane is 2D. Thus, face mesh 400 needs to be places at some position in the Z space. For example, face mesh 400 may be loaded such that the Z values of it are towards the positive axis (i.e.,
the XY plane is at the back of the face mesh or the front part of the face is facing forward from the XY plane if looking at a display screen). Although a square grid is described, the hair model can also be created using 3D polygons rather than the square grid.
[53] Because face mesh 400 is a 3D model, grid points of the XY plane are moved in the Z direction to meet a point on face mesh 400. This creates a 3D model of the grid. Thus, step 904 determines for each grid point, if it is occluded (i.e., obstructed) by face mesh 702. If the grid point is occluded by face mesh 702, then step 906 moves the grid point to a point where it intersects with face mesh 400. For example, the grid point is moved in the positive direction (i.e., forward) until it intersects with the Z value of a corresponding face mesh 400 value.
[54] Step 908 then determines if all grid points have been processed. If not, the process reiterates to step 904 to process another grid point.
[55] If the grid point is not occluded by the face mesh 702, then the face mesh is not moved. This is the area outside of face mesh 400.
[56] Once all the grid points have been processed, step 910 loads a mask image 1002 of a hair region. The hair region may be modeled using known methods.
[57] Fig. 10 shows a mask image 1002 of a hair region according to one embodiment of the present invention. As shown, mask image 1002 of the hair region models a hair region that may be captured. For example, the outline of the hair in face image 300 may be taken. Mask image 1002 may then be set on face mesh 400.
[58] There may be some areas of face mesh 400 that include hair areas. These areas are raised slightly in the positive Z direction for a point on face mesh 400. This is because in animation, hair may be raised from the skin for life-like animation. It will be recognized that the hair does not need to be raised in some embodiments.
[59] Step 912 determines for each grid point, if it is occluded by face mesh 400. If the grid point is not occluded by the face mesh, the process continues to step 918. Further, if the mask value in step 914 is not at a point that indicates it is hair, then the process proceeds to step 918, which determines if all grid points have been processed. If not, the process reiterates to step 912.
[60] If the grid point is occluded by face mesh 400, step 914 determines if the mask value at the point indicates it is hair. For example, a value may be computed on face mesh 400 of the hair in a position corresponding to a grid pixel. In step 916, if the value is 1 , the Z value of the point may be increased slightly. To determine that a vertex on face mesh 400 has a value of 1, the vertex is projected to the mask image 1002 and the mask value is checked there. If the value is 1, the vertex is a part of hair. This is because the point is part of mask image 1002, which is the hair region.
[61] Step 918 determines if all grid points have.been processed. If not, the process reiterates to step 912.
[62] When all grid points have been processed, points occluded by mask image 1002 may be processed. Step 920 determines for each grid point P if it is part of mask image 1002 that indicates a mask value of "hair" and is not occluded by face mesh 400. That is, is the grid point part of mask image 1002 but not part of face mesh 400. If not, nothing is performed and the process proceeds to step 928, described below.
[63] If the mask value indicates it is hair, then the grid point is raised in a way to model a person's hair. For example, a person's hair may slightly scale backwards in the Z direction. This may be modeled using the following steps. Step 922 computes the nearest occluded point (O). Step 924 then determines along the line OP (but not in the segment OP), the boundary point, Q, between hair and not hair.
[64] Step 926 computes the Z value of P using an equation, such as the equation Pz = Qz + w * (Oz - Qz). Pz5 Oz, and Qz are the Z values of P, O, and Q, respectively. Also, w = 1-OP/OQA2. This interpolates a Z value for the grid point P. It will be recognized that other methods of interpolating a Z value for grid point P will be appreciated.
[65] Step 928 determines if all grid points have been processed. If not, the process reiterates to step 920.
[66] When all grid points have been processed, a hair model has been generated. The hair model is a mesh whose vertices are the grid points. Fig. 11 shows a hair model 1100.
Color Differences
[67] The difference in color for vertices in a face mesh sequence may also be calculated. This captures the difference in color from different expressions. For example, when a face moves from one expression to another, the color of certain areas may become lighter or darker. These color differences are calculated and later added for the facial animation. Fig. 12 depicts a simplified flow chart 1200 of a method for computing the color differences according to one embodiment of the present invention.
[68] Step 1202 loads the face mesh sequence. Step 1204 sets face mesh 400 (e.g., the first face mesh of a face mesh sequence) as a base face mesh. The base face mesh may be a first frame of the face mesh sequence. The base face mesh may be an expression that is close to the expression found in face image 300. Also, in another embodiment, the base face mesh may be the mean shape of all the face meshes.
[69] Step 1206 computes the color differences for each vertex between the base mesh and other face meshes in the sequence. The color differences may be computed using known methods. The difference may be a value in which the color should be increased or decreased from the base face mesh to another face mesh. A color difference may be calculated based on RGB (red, green, blue) model. The color difference may be the color value of a subsequent face mesh is subtracted from the color value of the base face mesh at a vertex. A positive color difference may be a lighter color and a negative color difference is a darker color.
[70] Any positive color differences may be ignored in one embodiment. These differences may create white spots on an image. Thus, only negative color differences may be used. Also, if the resulting texture caused by the color differences does not look good (such as around the mouth or eye corners), color differences may be ignored in those regions.
[71] In another embodiment, the color differences can be represented as grey scale rather than the difference in each color channel, such as an RGB color scheme. An exaggerated face (in terms of color) can be created by detecting the color difference that exceeds some threshold and is applied as different colors (e.g. black and white) to the base texture. A 3D version of the expression ratio image can be used to represent the color differences.
[72] Also, the option of adding color differences can be represented as texture blending rather than a vertex-wise operation. In this case, the color differences are rendered first. The resulting images are then added to the rendered image of the face mesh. This is kind of a image blending. The color differences can be added for each vertex (which was described earlier). Or, the color difference can be rendered in a "not difference-added" mesh and the same mesh colored with the color difference separately, and then both rendered images are added pixelwise. Either way may add color differences.
[73] Fig. 13 shows the differences in color according to one embodiment of the present invention. A top row 1302 shows a face mesh without color. Row 1304 shows an image that includes color. Row 1304 shows what an original capture of a sequence would look like for reference.
[74] Each face mesh 1306 and a corresponding face image 1308 are shown. Differences in color between face meshes 1306 may be calculated. For example, the color differences between face mesh 1306-1 and face mesh 1306-2, the difference between face mesh 1306-1 and face mesh 1306-3, etc. may be calculated.
[75] Step 1208 then stores the color differences for each vertex. The color differences will be used when the face mesh sequence is later played.
[76] Fig. 14 shows the differences in color when a face squints in the original face mesh. As shown, image 1404 shows the changes in color that occur in the expression when the expression is changed from image 1402. As shown, circle 1408 shows a darkness that occurs for a change in form from the image in circle 1406.
[77] Fig. 15 shows a simplified flow chart 1500 for creating facial animation according to one embodiment of the present invention. Step 1502 loads the face mesh sequence and the hair model.
[78] Step 1504 applies the scale, translation, and rotation for each face mesh in the face mesh sequence. These values were saved in step 216 of Fig. 2. This aligns the face meshes to face image 300 in the face mesh sequence.
[79] Step 1506 projects face image 302 to the base mesh and the hair model as texture. Thus, the face image is shown on the face mesh.
[80] Step 1508 adds the displacement of the vertices to all meshes. This is the displacement values that were saved in step 222 of Fig. 2. This matches the feature points and vertices to face image 300. Also, the displacement difference from the base mesh to the other meshes may be used instead of having vertex positions and displacement values for every mesh. For example, the base mesh may have a first displacement value for a vertex. The difference in value for the next face mesh may be stored instead of a new value.
[81] Step 1510 adds color differences that were saved in step 1206 of Fig. 12 to all face meshes. Also, the color difference from the base mesh to the other meshes may be used instead of having vertex positions and color differences for every mesh. For example, the base mesh may have a first color difference for a vertex. The difference in value for the next face mesh may be stored instead of a new value.
[82] Step 1512 then plays the face mesh sequence. Simple, natural, and very realistic face animation is created by playing the face mesh sequence. This animation is provided using a base image of only one sheet (such as a frame or an image) and the face mesh sequence.
[83] Fig. 16 shows the result of the created animation according to one embodiment. As shown, animation of the face looks very realistic. Different expressions are created on face image 300 using the face mesh sequence.
[84] Fig. 17 shows images with color differences added and without according to one embodiment. Image 1702 shows an image with the color differences when added and image 1704 shows an image without color differences. As shown, the expression in image 1702 shows the change in light and darkness that are created from the change in form of the facial expression. As shown, ridges and creases are more defined in image 1702.
[85] In other embodiments, speech animations may be created using the above methods. For example, a lip sync animation may be created by synchronizing face meshes of phonemes (called visimes) and speech audio. Thus, other animations will be appreciated by embodiments of the present invention.
[86] Although the invention has been described with respect to specific embodiments
thereof, these embodiments are merely illustrative, and not restrictive of the invention.
[87] Any suitable programming language can be used to implement the routines of embodiments of the present invention including C, C++, Java, assembly language, etc. Different programming techniques can be employed such as procedural or object oriented. The routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different embodiments. In some embodiments, multiple steps shown as sequential in this specification can be performed at the same time. The sequence of operations described herein can be interrupted, suspended, or otherwise controlled by another process, such as an operating system, kernel, etc. The routines can operate in an operating system environment or as stand-alone routines occupying all, or a substantial part, of the system processing. Functions can be performed in hardware, software, or a combination of both. Unless otherwise stated, functions may also be performed manually, in whole or in part.
[88] In the description herein, numerous specific details are provided, such as examples of components and/or methods, to provide a thorough understanding of embodiments of the present invention. One skilled in the relevant art will recognize, however, that an embodiment of the invention can be practiced without one or more of the specific details, or with other apparatus, systems, assemblies, methods, components, materials, parts, and/or the like. In other instances, well-known structures, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of embodiments of the present invention.
[89] A "computer-readable medium" for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, system or device. The computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
[90] Embodiments of the present invention can be implemented in the form of control logic in software or hardware or a combination of both. The control logic may be stored
in an information storage medium, such as a computer-readable medium, as a plurality of instructions adapted to direct an information processing device to perform a set of steps disclosed in embodiments of the present invention. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and/or methods to implement the present invention.
[91] A "processor" or "process" includes any human, hardware and/or software system, mechanism or component that processes data, signals or other information. A processor can include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor can perform its functions in "real time," "offline," in a "batch mode," etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems.
[92] Reference throughout this specification to "one embodiment", "an embodiment", or "a specific embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention and not necessarily in all embodiments. Thus, respective appearances of the phrases "in one embodiment", "in an embodiment", or "in a specific embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any specific embodiment of the present invention may be combined in any suitable manner with one or more other embodiments. It is to be understood that other variations and modifications of the embodiments of the present invention described and illustrated herein are possible in light of the teachings herein and are to be considered as part of the spirit and scope of the present invention.
[93] Embodiments of the invention may be implemented by using a programmed general purpose digital computer, by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nanoengineered systems, components and mechanisms may be used. In general, the functions of embodiments of the present invention can be achieved by any means as is known in the art. Distributed, or networked systems, components and
circuits can be used. Communication, or transfer, of data may be wired, wireless, or by any other means.
[94] It will also be appreciated that one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. It is also within the spirit and scope of the present invention to implement a program or code that can be stored in a machine-readable medium to permit a computer to perform any of the methods described above.
[95] Additionally, any signal arrows in the drawings/Figures should be considered only as exemplary, and not limiting, unless otherwise specifically noted. Furthermore, the term "or" as used herein is generally intended to mean "and/or" unless otherwise indicated. Combinations of components or steps will also be considered as being noted, where terminology is foreseen as rendering the ability to separate or combine is unclear.
[96] As used in the description herein and throughout the claims that follow, "a", "an", and "the" includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise.
[97] The foregoing description of illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the present invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the present invention in light of the foregoing description of illustrated embodiments of the present invention and are to be included within the spirit and scope of the present invention.
[98] Thus, while the present invention has been described herein with reference to particular embodiments thereof, a latitude of modification, various changes and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of embodiments of the invention will be employed without a
corresponding use of other features without departing from the scope and spirit of the invention as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the present invention. It is intended that the invention not be limited to the particular terms used in following claims and/or to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include any and all embodiments and equivalents falling within the scope of the appended claims.
Claims
1. A method for creating facial animation, the method comprising: determining a face rnesh sequence; fitting a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; projecting the face image onto the face mesh as texture based on the fitting; adding the displacement values to vertices in subsequent face meshes in the face mesh sequence; and playing the face mesh sequence to create facial animation for the face image.
2. The method of claim 1 , further comprising: computing a color difference between the face mesh and each subsequent face mesh of the face mesh sequence at a vertex; and adding the color difference for each subsequent face mesh at the vertex, wherein the color difference is shown in each subsequent face mesh when the face mesh sequence is played.
3. The method of claim 2, further comprising ignoring any positive color differences.
4. The method of claim 1 , further comprising: creating a hair model from the face mesh; and fitting the hair model on each face mesh in the face mesh sequence.
5. The method of claim 4, wherein projecting the face image comprises projecting hair of the face image onto the hair model for the face mesh.
6. The method of claim 1 , wherein the fitting the face mesh comprises determining scale, rotation, and/or translation values by minimizing a distance between first feature points of the face mesh to second feature points of the face image.
7. The method of claim 6, wherein the scale, rotation, and/or translation values are applied to the vertices in the subsequent face meshes.
8. The method of claim 1, wherein the face mesh sequence is a face mesh recording of a human user.
9. The method of claim 1, wherein the portion of the face mesh sequence is a still image that corresponds to the facial image.
10. A device for creating facial animation, the device comprising: logic configured to determine a face mesh sequence; logic configured to fit a face mesh of the face mesh sequence to an image of a face, wherein displacement values for vertices of the face mesh are determined in the fitting; logic configured to project the face image onto the face mesh as texture based on the fitting; logic configured to add the displacement values for vertices in subsequent face meshes in the face mesh sequence; and logic configured to play the face mesh sequence to create facial animation for the face image.
11. The device of claim 10, further comprising: logic configured to compute a color difference between the face mesh and each subsequent face mesh of the face mesh sequence at a vertex; and logic configured to add the color difference for each subsequent face mesh at the vertex, wherein the color difference is shown in each subsequent face mesh when the face mesh sequence is played.
12. The device of claim 11 , further comprising logic configured to ignore any positive color differences.
13. The device of claim 10, further comprising: logic configured to create a hair model from the face mesh; and logic configured to fit the hair model on each face mesh in the face mesh sequence.
14. The device of claim 13, wherein the logic configured to project the face image comprises projecting hair of the face image onto the hair model for the face mesh.
15. The device of claim 10, wherein the logic configured to fit the face mesh comprises logic configured to determine scale, rotation, and/or translation values by minimizing a distance between first feature points of the face mesh to second feature points of the face image.
16. The device of claim 15, wherein the scale, rotation, and/or translation values are applied to the vertices in the subsequent face meshes.
17. The device of claim 10, wherein the face mesh sequence is a face mesh recording of a human user.
18. The device of claim 10, wherein the portion of the face mesh sequence is a still image that corresponds to the facial image.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2007800086119A CN101401129B (en) | 2006-01-10 | 2007-01-04 | Techniques for creating facial animation using a face mesh |
JP2008550358A JP4872135B2 (en) | 2006-01-10 | 2007-01-04 | Technology to create face animation using face mesh |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US75812206P | 2006-01-10 | 2006-01-10 | |
US60/758,122 | 2006-01-10 | ||
US11/396,159 US7567251B2 (en) | 2006-01-10 | 2006-03-31 | Techniques for creating facial animation using a face mesh |
US11/396,159 | 2006-03-31 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2007081899A2 true WO2007081899A2 (en) | 2007-07-19 |
WO2007081899A3 WO2007081899A3 (en) | 2007-10-11 |
Family
ID=38134229
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2007/000437 WO2007081899A2 (en) | 2006-01-10 | 2007-01-04 | Techniques for creating facial animation using a face mesh |
Country Status (4)
Country | Link |
---|---|
US (1) | US7567251B2 (en) |
JP (1) | JP4872135B2 (en) |
CN (1) | CN101401129B (en) |
WO (1) | WO2007081899A2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8441485B2 (en) | 2008-12-03 | 2013-05-14 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for generating video animation |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110115798A1 (en) * | 2007-05-10 | 2011-05-19 | Nayar Shree K | Methods and systems for creating speech-enabled avatars |
AU2009330607B2 (en) * | 2008-12-04 | 2015-04-09 | Cubic Corporation | System and methods for dynamically injecting expression information into an animated facial mesh |
US8884968B2 (en) * | 2010-12-15 | 2014-11-11 | Microsoft Corporation | Modeling an object from image data |
CN103765479A (en) * | 2011-08-09 | 2014-04-30 | 英特尔公司 | Image-based multi-view 3D face generation |
US9886622B2 (en) * | 2013-03-14 | 2018-02-06 | Intel Corporation | Adaptive facial expression calibration |
US9710965B2 (en) * | 2014-09-10 | 2017-07-18 | Disney Enterprises, Inc. | Capturing and stylizing hair for 3D fabrication |
CN106447754B (en) * | 2016-08-31 | 2019-12-24 | 和思易科技(武汉)有限责任公司 | Automatic generation method of pathological animation |
KR102256110B1 (en) | 2017-05-26 | 2021-05-26 | 라인 가부시키가이샤 | Method for image compression and method for image restoration |
US10198860B1 (en) * | 2017-11-13 | 2019-02-05 | The Boeing Company | Automated three dimensional (3D) mesh UV coordinate transfer process |
US10991143B2 (en) | 2019-07-03 | 2021-04-27 | Roblox Corporation | Animated faces using texture manipulation |
US20230143019A1 (en) * | 2021-11-10 | 2023-05-11 | Evr Studio Co., Ltd. | Method of generating facial expression and three-dimensional (3d) graphic interface device using the same |
KR20230098089A (en) | 2021-12-23 | 2023-07-03 | 소프트뱅크 가부시키가이샤 | Avatar display device, avatar creation device and program |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5777619A (en) * | 1996-01-02 | 1998-07-07 | Silicon Graphics, Inc. | Method for simulating hair using particle emissions |
JP2974655B1 (en) * | 1998-03-16 | 1999-11-10 | 株式会社エイ・ティ・アール人間情報通信研究所 | Animation system |
IT1315446B1 (en) * | 1998-10-02 | 2003-02-11 | Cselt Centro Studi Lab Telecom | PROCEDURE FOR THE CREATION OF THREE-DIMENSIONAL FACIAL MODELS TO START FROM FACE IMAGES. |
IT1314671B1 (en) * | 1998-10-07 | 2002-12-31 | Cselt Centro Studi Lab Telecom | PROCEDURE AND EQUIPMENT FOR THE ANIMATION OF A SYNTHESIZED HUMAN FACE MODEL DRIVEN BY AN AUDIO SIGNAL. |
US6735566B1 (en) * | 1998-10-09 | 2004-05-11 | Mitsubishi Electric Research Laboratories, Inc. | Generating realistic facial animation from speech |
US6879946B2 (en) * | 1999-11-30 | 2005-04-12 | Pattern Discovery Software Systems Ltd. | Intelligent modeling, transformation and manipulation system |
US6807290B2 (en) * | 2000-03-09 | 2004-10-19 | Microsoft Corporation | Rapid computer modeling of faces for animation |
AU2001268702A1 (en) * | 2000-06-22 | 2002-01-02 | Auckland Uniservices Limited | Non-linear morphing of faces and their dynamics |
JP4316114B2 (en) * | 2000-06-30 | 2009-08-19 | コニカミノルタホールディングス株式会社 | Model deformation method and modeling apparatus |
US6850872B1 (en) * | 2000-08-30 | 2005-02-01 | Microsoft Corporation | Facial image processing methods and systems |
US6950104B1 (en) * | 2000-08-30 | 2005-09-27 | Microsoft Corporation | Methods and systems for animating facial features, and methods and systems for expression transformation |
US6664956B1 (en) * | 2000-10-12 | 2003-12-16 | Momentum Bilgisayar, Yazilim, Danismanlik, Ticaret A. S. | Method for generating a personalized 3-D face model |
US6975750B2 (en) * | 2000-12-01 | 2005-12-13 | Microsoft Corp. | System and method for face recognition using synthesized training images |
JP4126938B2 (en) * | 2002-03-22 | 2008-07-30 | セイコーエプソン株式会社 | Image processing apparatus and image output apparatus |
AU2003231911A1 (en) | 2002-06-12 | 2003-12-31 | Oded Luria | Emergency medical kit, respiratory pump, and face mask particularly useful therein |
JP2004048496A (en) * | 2002-07-12 | 2004-02-12 | Monolith Co Ltd | Image encoding method and device, image decoding method and device, and image distributing device |
US6919892B1 (en) * | 2002-08-14 | 2005-07-19 | Avaworks, Incorporated | Photo realistic talking head creation system and method |
US7239321B2 (en) * | 2003-08-26 | 2007-07-03 | Speech Graphics, Inc. | Static and dynamic 3-D human face reconstruction |
EP1510973A3 (en) * | 2003-08-29 | 2006-08-16 | Samsung Electronics Co., Ltd. | Method and apparatus for image-based photorealistic 3D face modeling |
US7804997B2 (en) * | 2004-06-10 | 2010-09-28 | Technest Holdings, Inc. | Method and system for a three dimensional facial recognition system |
-
2006
- 2006-03-31 US US11/396,159 patent/US7567251B2/en active Active
-
2007
- 2007-01-04 JP JP2008550358A patent/JP4872135B2/en active Active
- 2007-01-04 CN CN2007800086119A patent/CN101401129B/en active Active
- 2007-01-04 WO PCT/US2007/000437 patent/WO2007081899A2/en active Application Filing
Non-Patent Citations (4)
Title |
---|
CHOI K-H ET AL: "Automatic Creation of a Talking Head From a Video Sequence" IEEE TRANSACTIONS ON MULTIMEDIA, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 7, no. 4, August 2005 (2005-08), pages 628-637, XP011136184 ISSN: 1520-9210 * |
FUA P ET AL: "From synthesis to analysis: fitting human animation models to image data" COMPUTER GRAPHICS INTERNATIONAL, 1999. PROCEEDINGS CANMORE, ALTA., CANADA 7-11 JUNE 1999, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 7 June 1999 (1999-06-07), pages 4-11, XP010340026 ISBN: 0-7695-0185-0 * |
OSTERMANN J ET AL: "ANIMATED TALKING HEAD WITH PERSONALIZED 3D HEAD MODEL" JOURNAL OF VLSI SIGNAL PROCESSING SYSTEMS FOR SIGNAL, IMAGE, AND VIDEO TECHNOLOGY, SPRINGER, NEW YORK, NY, US, vol. 20, no. 1/2, 1 October 1998 (1998-10-01), pages 97-104, XP000786730 ISSN: 0922-5773 * |
ZHENGYOU ZHANG ET AL: "Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach" INTERNATIONAL JOURNAL OF COMPUTER VISION, KLUWER ACADEMIC PUBLISHERS, BO, vol. 58, no. 2, 1 July 2004 (2004-07-01), pages 93-119, XP019216414 ISSN: 1573-1405 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8441485B2 (en) | 2008-12-03 | 2013-05-14 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for generating video animation |
Also Published As
Publication number | Publication date |
---|---|
CN101401129A (en) | 2009-04-01 |
JP2009523288A (en) | 2009-06-18 |
CN101401129B (en) | 2013-06-05 |
WO2007081899A3 (en) | 2007-10-11 |
US20070159486A1 (en) | 2007-07-12 |
US7567251B2 (en) | 2009-07-28 |
JP4872135B2 (en) | 2012-02-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7567251B2 (en) | Techniques for creating facial animation using a face mesh | |
US11423556B2 (en) | Methods and systems to modify two dimensional facial images in a video to generate, in real-time, facial images that appear three dimensional | |
US20210390767A1 (en) | Computing images of head mounted display wearer | |
CN110650368B (en) | Video processing method and device and electronic equipment | |
US10089522B2 (en) | Head-mounted display with facial expression detecting capability | |
CN109410298B (en) | Virtual model manufacturing method and expression changing method | |
CN101055647B (en) | Method and device for processing image | |
CA2284348C (en) | A method of creating 3 d facial models starting from face images | |
US7116330B2 (en) | Approximating motion using a three-dimensional model | |
US8553037B2 (en) | Do-It-Yourself photo realistic talking head creation system and method | |
CN111971713A (en) | 3D face capture and modification using image and time tracking neural networks | |
US9196074B1 (en) | Refining facial animation models | |
KR101148101B1 (en) | Method for retargeting expression | |
KR20090098798A (en) | Method and device for the virtual simulation of a sequence of video images | |
US7277571B2 (en) | Effective image processing, apparatus and method in virtual three-dimensional space | |
US20230186550A1 (en) | Optimizing generation of a virtual scene for use in a virtual display environment | |
KR20200024105A (en) | Computer graphics synthesis system and method thereof | |
CN110580677A (en) | Data processing method and device and data processing device | |
US20220237751A1 (en) | Techniques for enhancing skin renders using neural network projection for rendering completion | |
JP3538263B2 (en) | Image generation method | |
Richter et al. | Real-time reshaping of humans | |
KR100422470B1 (en) | Method and apparatus for replacing a model face of moving image | |
US9792715B2 (en) | Methods, systems, and computer readable media for utilizing synthetic animatronics | |
CN112561784B (en) | Image synthesis method, device, electronic equipment and storage medium | |
Pauly | Realtime performance-based facial avatars for immersive gameplay |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 2008550358 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 200780008611.9 Country of ref document: CN |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 07709624 Country of ref document: EP Kind code of ref document: A2 |