WO2007044815A2

WO2007044815A2 - Generation of normalized 2d imagery and id systems via 2d to 3d lifting of multifeatured objects

Info

Publication number: WO2007044815A2
Application number: PCT/US2006/039737
Authority: WO
Inventors: Michael I. Miller
Original assignee: Animetrics Inc.
Priority date: 2005-10-11
Filing date: 2006-10-11
Publication date: 2007-04-19
Also published as: US20070080967A1; US20100149177A1; WO2007044815A3

Abstract

Image acquisition system (202) captures target image (402) and generates (206) best-fitting avatar (404) using the methods described above. Best-fitting avatar is geometrically normalized by backing out deformations corresponding to non-intrinsic and non-permanent features of the target head (406). The geometrically normalized avatar is lit with normal lighting (406), and projected into an image plane corresponding to a normal pose, such as a face-on view. The resulting image (408) is geometrically normalized with respect to shape (expressions and temporary surface alterations) and pose, as well as photometrically normalized with respect to lighting.

Description

GENERATION OF NORMALIZED 2D IMAGERY AND ID SYSTEMS VIA 2D TO 3D LIFTING OF MULTIFEATURED OBJECTS

TECHNICAL FIELD

[0001] This invention relates to object modeling and identification systems, and more particularly to the determination of 3D geometry and lighting of an object from 2D input using 3D models of candidate objects.

BACKGROUND

[0002] Facial identification (ID) systems typically function by attempting to match a newly captured image with an image that is archived in an image database. If the match is close enough, the system determines that a successful identification has been made. The matching takes place entirely within two dimensions, with the ID system manipulating both the captured image and the database images in 2D.

[0003] Most facial image databases store pictures that were captured under controlled conditions in which the subject is captured in a standard pose and under standard lighting conditions. Typically, the standard pose is a head-on pose, and the standard lighting is neutral and uniform. When a newly captured image to be identified is obtained with a standard pose and under standard lighting conditions, it is normally possible to obtain a relatively close match between the image and a corresponding database image, if one is present in the database. However, such systems tend to become unreliable as the image to be identified is captured under pose and lighting conditions that deviate from the standard pose and lighting. This is to be expected, because both changes in pose and changes in lighting will have a major impact on a 2D image of a three-dimensional object, such as a face. SUMMARY

[0004] Embodiments described herein employ a variety of methods to "normalize" captured facial imagery (both 2D and 3D) by means of 3D avatar representations so as to improve the performance of traditional ID systems that use a database of images captured under standard pose and lighting conditions. The techniques described can be viewed as providing a "front end" to a traditional ID system, in which an available image to be identified is preprocessed before being passed to the ID system for identification. The techniques can also be integrated within an ID system that uses 3D imagery, or a combination of 2D and 3D imagery.

[0005] The methods exploit the lifting of 2D photometric and geometric information to 3D coordinate system representations, referred to herein as avatars or model geometry. As used herein, the term lifting is taken to mean the estimation of 3D information about an object based on one or more available 2D projections (images) and/or 3D measurements. Photometric lifting is taken to mean the estimation of 3D lighting information based on the available 2D and/or 3D information, and geometric lifting is taken to mean the estimation of 3D geometrical (shape) information based on the available 2D and/or 3D information.

[0006] The construction of the 3D geometry from 2D photographs involves the use of a library of 3D avatars. The system calculates the closest matching avatar in the library of avatars. It may then alter 3D geometry, shaping it to more closely correspond to the measured geometry in the image. Photometric (lighting) information is then placed upon this 3D geometry in a manner that is consistent with the information in the image plane. In other words, the avatar is lit in such a way that a camera in the image plane would produce a photograph that approximates to the available 2D image.

[0007] When used as a preprocessor for a traditional 2D ID system, the 3D geometry can be normalized geometrically and photometrically so that the 3D geometry appears to be in a standard pose and lit with standard lighting. The resulting normalized image is then passed to the traditional ID system for identification. Since the traditional ID system is now attempting to match an image that has effectively been rotated and photometrically normalized to place it in correspondence with the standard images in the image database, the system should work effectively, and produce an accurate identification. This preprocessing serves to make traditional ID systems robust to variations in pose and lighting conditions. The described embodiment also works effectively with 3D matching systems, since it enables normalization of the state of the avatar model so that it can be directly and efficiently compared to standardized registered individuals in a 3D database.

[0008] In general, in one aspect, the invention features a method of estimating a 3D shape of a target head from at least one source 2D image of the head. The method involves searching a library of candidate 3D avatar models to locate a best-fit 3D avatar, for each 3D avatar model among the library of 3D avatar models computing a measure of fit between a 2D projection of that 3D avatar model and the at least one source 2D image, the measure of fit being based on at least one of (i) unlabeled feature points in the source 2D imagery, and (ii) additional feature points generated by imposing symmetry constraints, wherein the best-fit 3D avatar is the 3D avatar model among the library of 3D avatar models that yields a best measure of fit and wherein the estimate of the 3D shape of the target head is derived from the best-fit 3D avatar.

[0009] Other embodiments include one or more of the following features. A target image illumination is estimated by generating a set of notional lightings of the best-fit 3D avatar and searching among the notional lightings of the best-fit avatar to locate a best notional lighting that has a 2D projection that yields a best measure of fit to the target image. The notional lightings include a set of photometric basis functions and at least one of small and large variations from the basis functions. The best-fit 3D avatar is projected and compared to a gallery of facial images, and identified with a member of the gallery if the fit exceeds a certain value. The search among avatars also includes searching at least one of small and large deformations of members of the library of avatars. The estimation of 3D shape of a target head can be made from a single 2D image if the surface texture of the target head is known, or if symmetry constraints on the avatar and source image are imposed. The estimation of 3D shape of a target head can be made from two or more 2D images even if the surface texture of the target head is initially unknown.

[0010] In general, in another aspect, the invention features a method of generating a normalized 3D representation of a target head from at least one source 2D projection of the head. The method involves providing a library of candidate 3D avatar models, and searching among the candidate 3D avatar models and their deformations to locate a best-fit 3D avatar, the searching including, for each 3D avatar model among the library of 3D avatar models and each of its deformations, computing a measure of fit between a 2D projection of that deformed 3D avatar model and the at least one source 2D image, the deformations corresponding to permanent and non-permanent features of the target head, wherein the best-fit deformed 3D avatar is the deformed 3D avatar model that yields a best measure of fit; and generating a geometrically normalized 3D representation of the target head from the best-fit deformed 3D avatar by removing deformations corresponding to non-permanent features of the target head.

[0011] Other embodiments include one or more of the following features. The normalized 3D representation is projected into a plane corresponding to a normalized pose, such as a face-on view, to generate a geometrically normalized image. The normalized image is compared to members of a gallery of 2D facial images having a normal pose, and positively identified with a member of the gallery if a measure of fit between the normalized image and a gallery member exceeds a predetermined threshold. The best-fitting avatar can be lit with normalized (such as uniform and diffuse) lighting before being projected into a normal pose so as to generate a geometrically and photometrically normalized image.

[0012] In general, in yet another aspect, the invention features a method of estimating the 3D shape of a target head from source 3D feature points. The method involves searching a library of avatars and their deformations to locate the deformed avatar having the best fit to the 3D feature points, and basing the estimate on the best- fit avatar.

[0013] Other embodiments include matching to avatar feature points and their reflections in an avatar plane of symmetry, using unlabeled source 3D feature points, and using source 3D normal feature points that specify a head surface normal direction as well as position. Comparing the best-fit deformed avatar with each gallery member, yields a positive identification of the 3D head with a member of a gallery of 3D reference representations of heads if a measure of fit exceeds a predetermined threshold.

[0014] In general, in still another aspect, the invention features a method of estimating a 3D shape of a target head from a comparison of a projection of a 3D avatar and dense imagery of at least one source 2D image of a head.

[0015] In general, in a further aspect, the invention features positively identifying at least one source image of a target head with a member of a database of candidate facial images. The method involves generating a 3D avatar corresponding to the source imagery and generating a 3D avatar corresponding to each member of the database of candidate facial images using the methods described above. The target head is positively identified with a member of the database of candidate facial images if a measure of fit between the source avatar corresponding to the source imagery and an avatar corresponding to a candidate facial image exceeds a predetermined threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Fig. 1 is a flow diagram illustrating the principal steps involved in normalizing a source 2D facial image.

[0017] Fig. 2 illustrates photometric normalization of a source 2D facial image. [0018] Fig. 3 illustrates geometric normalization of a source 2D facial image.

[0019] Fig. 4 illustrates performing both photometric and geometric normalization of a source 2D facial image.

[0020] Fig. 5 illustrates removing lighting variations by spatial filtering and symmetrization of source facial imagery.

DETAILED DESCRIPTION

[0021] A traditional photographic ID system attempts to match one or more target images of the person to be identified with an image in an image library. Such systems perform the matching in 2D using image comparison methods that are well known in the art. If the target images are captured under controlled conditions, the system will normally identify a match, if one exists, with an image in its database because the system is comparing like with like, i.e., comparing two images that were captured under similar conditions. The conditions in question refer principally to the pose and shape of the subject and the photometric lighting. However, it is often not possible to capture target photographs under controlled conditions. For example, a target image might be captured by a security camera without the subject's knowledge, or it might be taken while the subject is fleeing the scene.

[0022] The described embodiment converts target 2D imagery captured under uncontrolled conditions in the projective plane and converts it into a 3D avatar geometry model representation. Using the terms employed herein, the system lifts the photometric and geometric information from 2D imagery or 3D measurements onto the 3D avatar geometry. It then uses the 3D avatar to generate geometrically and photometrically normalized representations that correspond to standard conditions under which the reference image database was captured. These standard conditions, also referred to as normal conditions, usually correspond to a head-on view of the face with a normal expression and neutral and uniform illumination. Once a target image is normalized, a traditional ID system can use it to perform a reliable identification. [0023] Since the described embodiment can normalize an image to match a traditional ID system's normal pose and lighting conditions exactly, the methods described herein also serve to increase the accuracy of a traditional ID system even when working with target images that were previously considered close enough to "normal" to be suitable for ID via such systems. For example, a traditional ID system might have a 70% chance of performing an accurate ID with a target image pose of 30° from head-on. However, if the target is preprocessed and normalized before being passed to the ID system, the chance of performing an accurate ID might increase to 90%.

[0024] The basic steps of the normalization process are illustrated in Fig. 1. The target image is captured (102) under unknown pose and lighting conditions. The following steps (104-110) are described in detail in U.S. Patent Application Serial Numbers 10/794,353 and 10/794,943, which are incorporated herein in their entirety.

[0025] The process starts with a process called jump detection, in which the system scans the target image to detect the presence of the feature points whose existence in the image plane are substantially invariant across different faces under varying lighting conditions and under varying poses (104). Such features include one or more of the following: points, such as the extremity of the mouth, curves, such an eyebrow; brightness order relationships; image gradients; edges, and subareas. For example, the existence in the image plane of the inside and outside of a nostril is substantially invariant under face, pose, and lighting variations. To determine the lifted geometry, the system only needs about 3-100 feature points. Each identified feature point corresponds to a labeled feature point in the avatar. Feature points are referred to as labeled when the correspondence is known, and unlabeled when the correspondence is unknown.

[0026] Since the labeled feature points being detected are a sparse sampling of the image plane and relatively small in number, jump detection is very rapid, and can be performed in real time. This is especially useful when a moving image is being tracked. [0027] The system uses the detected feature points to determine the lifted geometry by searching a library of avatars to locate the avatar whose invariant features, when projected into 2D at all possible poses, has a projection which yields the closest match to the invariant features identified in the target imagery (106). The 3D lifted avatar geometry is then refined via shape deformation to improve the feature correspondence (108). This 3D avatar representation may also be refined via unlabeled feature points, as well as dense imagery requiring diffusion or gradient matching along with the sparse landmark-based matching, and 3D labeled and unlabeled features.

[0028] In subsequent step 110, the deformed avatar is lit with the normal lighting parameters and projected into 2D from an angle that corresponds to the normal pose. The resulting "normalized" image is passed to the traditional ID system (1 12). Aspects of these steps that relate to the normalization process are described in detail below.

[0029] The described embodiment performs two kinds of normalization: geometric and photometric. Geometric normalizations include the normalization of pose, as referred to above. This corresponds to rigid body motions of the selected avatar. For example, a target image that was captured from 30° clockwise from head- on has its geometry and photometry lifted to the 3D avatar geometry, from which it is normalized to a head-on view by rotating the 3D avatar geometry by 30° anticlockwise before projecting it into the image plane.

[0030] Geometric normalizations also include shape changes, such as facial expressions. For example, an elongated or open mouth corresponding to a smile or laugh can be normalized to a normal width, closed mouth. Such expressions are modeled by deforming the avatar so as to obtain an improved key feature match in the 2D target image (step 108). The system later "backs out" or "inverts" the deformations corresponding to the expressions so as to produce an image that has a "normal" expression. Another example of shape change corresponding to geometric normalization inverts the effects of aging. A target image of an older person can be normalized to the corresponding younger face.

[0031] Photometric normalization includes lighting normalizations and surface texture/color normalizations. Lighting normalization involves converting a target image taken under non-standard illumination and converting it to normal illumination. For example, a target image may be lit with a point source of red light. Photometric normalization converts the image into one that appears to be taken under neutral, uniform lighting. This is performed by illuminating the selected deformed avatar with the standard lighting before projecting it into 2D (110).

[0032] A second type of photometric normalization takes account of changes in the surface texture or color of the target image compared to the reference image. An avatar surface is described by a set of normals N(x) which are 3D vectors representing the orientations of the faces of the model, and a reference texture called T_ref(x), that is a data structure, such as a matrix having an RGB value for each polygon on the avatar. Photometric normalization can involve changing the values of T_ref for some of the polygons that correspond to non-standard features in the target image. For example, a beard can change the color of a region of the face from white to black. In the idealized case, this would correspond to the RGB values changing from (256, 256, 256) for white to (0,0,0) for black. In this case, photometric normalization corresponds to restoring the face to a standard, usually with no facial hair.

[0033] As illustrated by 108 in Fig. 1 , the selected avatar is deformed prior to illumination and projection into 2D. Deformation denotes a variation in shape from the library avatar to a deformed avatar whose key features more closely correspond to the key features of the target image. Deformations may correspond to an overall head shape variation, or to a particular feature of a face, such as the size of the nose.

[0034] The normalization process distinguishes between small geometric or photometric changes performed on the library avatar and large changes. A small change is one in which the geometric change (be it a shape change or deformation) or photometric change (be it a lighting change to surface texture/color change) is such that the mapping from the library avatar to the changed avatar is approximately linear. Geometric transformation moves the coordinates according to the general mapping x e IR³ h-> φ(x) e R³ . For small geometric transformation, the mapping approximates to an additive linear change in coordinates, so that the original value x maps approximately under the linear relationship x <≡ R³ h-> φ(x) » x + u(x) e R³ . The lighting variation changes the values of the avatar function texture field values T(x) at each coordinate systems point x, and is generally of the multiplicative form F_ref(^χ) ¹^ ^l'^_ref(^χ) ^e ^³ • F°^r small variation lighting the change is also linearly

approximated by T_ref(x) i→ L(x)'T_ref(x) « ε (x) + T_ref(x) e R³.

[0035] Examples of small geometric deformations include small variations in face shape that characterize a range of individuals of broadly similar features and the effects of aging. Examples of small photometric changes include small changes in lighting between the target image and the normal lighting, and small texture changes, such as variations in skin color, for example a suntan. Large deformations refer to changes in geometric or photometric data that are large enough so that the linear approximations used above for small deformations cannot be used.

[0036] Examples of large geometric deformations include large variation in face shapes, such as a large nose compared to a small nose, and pronounced facial expressions, such as a laugh or display of surprise. Examples of large photometric changes include major lighting changes such as extreme shadows, and change from indoor lighting to outdoor lighting.

[0037] The avatar model geometry, from here on referred to as a CAD model (or by the symbol CAD) is represented by a mesh of points in 3D that are the vertices of the set of triangular polygons that approximate the surface of the avatar. Each surface point x e CAD has a normal direction N(x) e R'.jc e CAD . Each vertex is given a color value, called a texture T(x) e l',x e CAD , and each triangular face is colored according to an average of the color values assigned to its vertices. The color values are determined from a 2D texture map that may be derived using standard texture mapping procedures, which define a bijective correspondence (1-1 and onto) from the photograph used to create the reference avatar. The avatar is associated with a coordinate system that is fixed to it, and is indexed by three angular degrees of freedom (pitch, roll, and yaw), and three translational degrees of freedom of the rigid body center in three-space. To capture articulation of the avatar geometry, such as motion of the chin and eyes, certain subparts have their own local coordinates, which form part of the avatar description. For example, the chin can be described by cylindrical coordinates about an axis corresponding to the jaw. Texture values are represented by a color representation, such RGB values. The avatar vertices are connected to form polygonal (usually triangular) facets.

[0038] Generating a normalized image from a single or multiple target photographs requires a bijection or correspondence between the planar coordinates of the target imagery and the 3D avatar geometry. As introduced above, once the correspondences are found, the photometric and geometric information in the measured imagery can be lifted onto the 3D avatar geometry. The 3D object is manipulated and normalized, and normalized output imagery is generated from the 3D object. Normalized output imagery may be provided via OpenGL or other conventional rendering engines, or other rendering devices. Geometric and photometric lifting and normalization are now described.

2D to 3D Photometric Lifting to 3D Avatar Geometries

Nonlinear Least-Square Photometric Lifting

[0039] For photometric lifting, it is assumed that the 3D model avatar geometry with surface vertices and normals is known, along with the avatar's shape and pose parameters, and its reference texture T_ref{x),x e CAD . The lighting normalization involves the interaction of the known shape and normals on the surface of the CAD model. The photometric basis is defined relative to the midplane of the avatar geometry and the interaction of the normals indexed with the surface geometry and the luminance function representation. Generating a normalized image from a single or multiple target photographs requires a bijection or correspondence between the planar coordinates of the imagery I(p),p e [0,l]² and the 3D avatar geometry, denoted p e [0,l]² <-> x(p) e R³ ; for the correspondence between the multiple views f^'(p), v = \,...,V , the multiple correspondences becomes p e [0,l]² <H> x^v(p) e l³ . A set of photometric basis functions representing the entire lighting sphere for each I^v(p) is computed in order to represent the lighting of each avatar corresponding to the photograph, using principal components relative to the particular geometric avatars. The photometric variation is lifted onto the 3D avatar geometry by varying the photometric basis functions representing illumination variability to match optimally the photographic values between the known avatar and the photographs. By working in the log-coordinates, the luminance function, L(x),x e CAD , can be estimated in a closed-form least-squares solution for the photometric basis functions. The color of the illuminating light can also be normalized by matching the RGB values in the textured representation of the avatar to reflect lighting spectrum variations, such as natural versus artificial light, and other physical characteristics of the lighting source.

[0040] Once the lighting state has been fit to the avatar geometry, neutralized, or normalized versions of the textured avatar can be generated by applying the inverse transformation specified by the geometric and lighting features to the best- fit models. The system then uses the normalized avatar to generate normalized photographic output in the projective plane corresponding to any desired geometric or lighting specification. As mentioned above, the desired normalized output usually corresponds to a head-on pose viewed under neutral, uniform lighting.

[0041] Photometric normalization is now described via the mathematical equations which describe the optimum solution. Given a reference avatar texture field, the textured lighting field T(x),x e CAD is written as a perturbation of the original reference T_ref{x),x e CAD by luminance £(χ),i e CAD and color functions e'^Λ , e'^a , e'" . These luminance and color functions can in general be expanded in a basis which may be computed using principal components on the CAD model by varying all possible illuminations. It may sometimes be preferable to perform the calculation analytically based on any other complete orthonormal basis defined on surfaces, such as spherical harmonics, Laplace-Beltrami functions and other functions of the derivatives. In general, luminance variations cannot be additive, as the space of measured imagery is a positive function space. For representing large variation lighting, the photometric field T(x) is modeled as a multiplicative group acting on the reference textured object T_ref according to

where φ_x are orthogonal basis functions indexed over the face, and the coefficient vectors /, = (/,^Λ , /,^σ , /,^β ), I₂ = (/₂ ^Λ , /₂ ^σ , /₂ ^β ), ... represent the unknown basis function coefficients representing a different variation for each RGB within the multiplicative representation.

[0042] Here £(•) represents the luminance function indexed over the CAD model resulting from interaction of the incident light with the normal directions of the 3D avatar surface. Once the correspondence is defined between the observed photograph and the avatar representation p e [0,l]² O x{p) e R³ , there exists a correspondence between the photograph and the RGB texture values on the avatar. In this section it is assumed that the avatar texture T_ref (x) is known. In general, the overall color spectrum of the texture field may demonstrate variations as well. In this case, each RGB expansion coefficient solves for the separate channel random field variations requires solution of the minimum mean-squared error (MMSE) equations

(r (P) -I? (-W(X(P)))² . (2)

The system then uses non-linear least-squares algorithms such as gradient algorithms or Newton search to generate the minimum mean-squared error (MMSE) estimator of the lighting field parameters. It does this by solving the minimization over the

luminance fields in the span of the bases L^c = e^"¹ ' ' ^* , c=R,G,B. Other norms besides the 2-norm for positive functions may be used, including the Kullback-Liebler distance, Ll distance, or others. Correlation between the RGB components can be introduced via a covariance matrix between the lighting and color components.

[0043] For a lower-dimensional representation in which there is a single RGB tinting function - rather than one for each expansion coefficient - the model becomes simply

T(x) =

The MMSE corresponds to

Given the reference T_ref(x) , the non-linear least-squares algorithms such as gradient algorithms and Newton search, can be used for minimizing the least-squares equation.

Fast Photometric Lifting to 3D Geometries via the Log Metric

[0044] Since the space of lighting variations is very extensive, multiplicative photometric normalization is computationally intensive. A log transformation creates a robust, computationally effective, linear least-squares formulation. Converting the

The LSE's for the lighting functions becomes fory = l, ,.,,</

Φ,(^χ(^p))ΦM(^p))- (8) pe Σ[0,l

Small Variation Photometric Lifting to 3D Geometries

[0045] As discussed above, small variations in the texture field (corresponding, for example, to small color changes of the reference avatar) are approximately linear T_ιef{x) v- > ε{x) + T_ιef{x) , with the additive field modeled in the basis

ε(x) = ^T (ε' ₅ εf,ε^)φ_t(x) . For small photometric variations, the MMSE satisfies

The LLSE' s for the images directly (rather than their log) gives

_for ^c=^{R,G B,} ^ (r(_P)-τ;_e/(x(p)))φ_J(x(p)) = ∑l ε; ∑ Φ,MP))ΦM(P)) j - \,..., d _pe[0,l]² pe[0,l]²

(10)

Adding the color representation via the tinting function gives ε(x) = ∑_M(t^R + ε, ,t^G + ε, ,t^B + ε, )φ, (x) gives the color tints according to

Begin by assuming that only the sparse feature points are used for the geometric lifting, and that they are defined in correspondence between points on the avatar 3D geometry and the 2D projective imagery, concentrating on extracted features associated with points, curves, or subareas in the image plane. Given the starting imagery I(p), p e [0, 1]² , the set of X₁ — (X_j ,y_J, z_J),j = \,...,N features is defined on the candidate avatar and to a correspondence to a similar set of features in the projective imagery P₁ = (p_jX , p_J2) e [0, 1]², j = 1, ..., N . The projective geometry mapping is defined as either positive or negative z projecting along the z axis with rigid transformation of the form O, b : x H> Ox + b around object center

The search for the best-

fitting avatar pose (corresponding to the optimal rotation and translation for the selected avatar) uses the invariant features as follows. Given the projective points in the image plane p_pj = 1,2,..., N and a rigid transformation of the form

O,b : x

where id is the 3x3 identity matrix. As described in U.S. Patent Applications serial number 10/794,353, the cost function (a measure of the aggregate distance between the projected invariant points of the avatar and the corresponding points in the measured target image) is evaluated by exhaustively calculating the lifted Z₁J = 1,...,N . Using MMSE estimation, choosing the minimum cost function, gives the lifted z-depths corresponding to:

min ∑ \\θx, + b- Z₁Pg, = min ∑ (Ox₁ + b)^< Q₁ (Ox₁ + b). (22)

O,b 7 /~=1

[0053] Choosing a best-fitting predefined avatar involves the database of avatars, with CAD - a,a = \,2,... the number of total avatar models each with labeled features x",j = \,...,N . Selecting the optimum CAD model minimizes overall cost function, choosing the optimally fit CAD model.

where || v, ||^ is the Sobelev norm with v satisfying smoothness constraints associated with Il V₁ 11^ . The norm can be associated with a differential operator L representing the smoothness enforced on the vector fields, such as the Laplacian and other forms of derivatives so that || v, ||^=|| Lv₁ ||² ; alternatively smoothness is enforced by forcing the

Sobelev space to be a reproducing kernel Hubert space with a smoothing kernel. All of these are acceptable methods. Adding the rigid motions gives a similar minimization problem

O

+

[0055] Such large deformations can represent expressions, jaw motion as well as large deformation shape change, following U.S. Patent Application Serial No. 10/794,353. In another embodiment, the avatar may be deformed with small deformations only representing the large deformation according to the linear approximation x — > x + u(x), x e CAD :

[0056] Expressions and jaw motions can be added directly by writing the vector fields u in a basis representing the expressions as described in U.S. Patent Application Serial No. 10/794,353. In order to track such changes, the motions may be parametrically defined via an expression basis E₁ , E₂ , ... so that u(x) = ^ e₍E₍ (x) .

These are defined as functions that describe how a smile, eyebrow lift and other expressions cause the invariant features to move on the face. The coefficients e, , e₂, ... describing the magnitude of each expression, become the unknowns to be estimated. For example, jaw motion corresponds to a flow of points in the jaw following a rotation around the fixed jaw axis 0{γ) : x V- > 0{γ)x where O rotates the jaw points around the jaw axis γ .

2D to 3D Geometric Lifting Using Symmetry

[0057] For symmetric objects such as the face, the system uses a reflective symmetry constraint in both rigid motion and deformation estimation to gain extra power. Again the CAD model coordinates are centered at the origin such that its plane of symmetry is aligned with the yz -plane. Therefore, the reflection matrix is

simply and R : x \- > Rx is the reflection of x about the plane of

symmetry on the CAD model. Given the features X₁ = (x, , y_l , z.), i = 1, ..., N , the system defines σ : {\, ...,N} h-» {\,...,N} to be the permutation such that X₁ and x_σ{l) are symmetric pairs for all / = 1, ..., N . In order to enforce symmetry the system adds an identical set of constraints on the reflection of the original set of model points. In the case of rigid motion estimation, the symmetry requires that an observed feature in the projective plane matches both the corresponding point on the model (under the rigid motion) (O, b) : x \- > Ox₁ + b , as well as the reflection of the symmetric pair on the model, ORx_σ(l) + b . Similarly, the deformation, φ , applied to a point X₁ should be the same as that produced by the reflection of the deformation of the symmetric pair Rφ(x_σU)) ■ This amounts to augmenting the optimization to include two constraints for each feature point instead of one. The rigid motion estimation reduces to the same structure as in U.S. Patent Application Serial Nos. 10/794,353 and 10/794,943 with 2N instead of N constraints and takes a similar form as the two view problem, as described therein.

2D to 3D Geometric Lifting via Dense Imagery (Without Correspondence)

[0065] In another embodiment, as described in U.S. Patent Application Ser. No. 10/794,353, the geometric transformations are constructed directly from the dense set of continuous pixels representing the object, in which case observed N feature points may not be delineated in the projective imagery or in the avatar template models. In such cases, the geometrically normalized avatar can be generated from the dense imagery directly. Assume the 3D avatar is at orientation and translation (O,b) under the Euclidean transformation x h-> Ox + b , with associated texture field T(O,b). Define the avatar at orientation and position (O,b) the template T(O,b). Then model the given image I(p),p e [0,l]² as a noisy representation of the projection of the avatar template at the unknown position (O,b). The problem is to estimate the rotation and translation O, b which minimizes the expression

min ∑ \\I(P) - T(θ,b)(x(p))&> (48)

where x(p) indexes through the 3D avatar template. In the situation where targets are tracked in a series of images, and in some instances when a single image only is available, knowledge of the position of the center of the target will often be available. This knowledge is incorporated as described above, by adding the prior information via the position information

min ∑ \\l{p) - T{O,b){x(p))t + (b-μ)'∑-\b - μ) . (49)

[0066] This minimization procedure is accomplished via diffusion matching as described in U.S. Patent Application Ser. No. 10/794,353. Further including annotated features give rise to jump diffusion dynamics. Shape changes and expressions corresponding to large deformations with φ : x H> φ(x) satisfying

φ = φ_uφ, = f V_S (φ_s {x))ds + x, x e CAD are generated:

. (50)

As above in the small deformation equation, for small deformation φ : x I— » ^(JC) « x + u{x) . To represent expressions directly, the transformation can be written in the basis E_x , E₂ , ... as above with the coefficients e, , e₂ , ... describing the magnitude of each expression's contribution to the variables to be estimated.

[0067] The optimal rotation and translation may be computed using the techniques described above, by first performing the optimization for the rigid motion alone, and then performing the optimization for shape transformation. Alternatively, the optimum expressions and rigid motions may be computed simultaneously by searching over their corresponding parameter spaces simultaneously.

[0068] For dense matching, the symmetry constraint is applied in a similar fashion by applying the permutation to each element of the avatar according to

Photometric, Texture and Geometry Lifting

[0069] When the geometry and photometry and texture are unknown, then the lifting must be performed simultaneously. In this case, the images /^v,v = l,2,... , are available and the unknowns are the CAD models with their associated bijections p e [0,l]² <→ x^v(p) e R³,v = l,...,V defined by rigid motions O\b\v = 1,2,... , along with T_ιef being unknown and the unknown lighting fields L^v determining the color representations for each instance under the multiplicative model T^v = UT_ιef . When using such multiple views, the first step is to create a common coordinate system that accommodates the entire model geometry. The common coordinates are in 3D, based directly on the avatar vertices. To perform the photometric normalization and the texture field estimation for the multiple photographs there are multiple bijective correspondences p e [0,l]² <-> x^v(p) e E³, v = \,...,V between the CAD models and the planar images /^v,v = 1,... . The first step is to estimate the CAD models geometry either from labeled points in 2D or 3D or via unlabeled points or via dense matching. This follows the above sections for choosing and shaping the geometry of the CAD model to be consistent with the geometric information in the observed imagery, and determining the bisections between the observed imagery and the fixed CAD model. For one instance, if given the projective points in the image plane p_} , j = 1, 2, ... ,N

- P II² where id is the

3x3 identity matrix, and the cost function (a measure of the aggregate distance between the projected invariant points of the avatar and the corresponding points in the measured target image) using MMSE estimation, then a best-fitting predefined avatar can be chosen from the database of avatars, with CAD", a = 1,2,... ,each with labeled features x", j = 1, ..., N . Selecting the optimum CAD model minimizes the overall cost function:

CAD = min Y (Oc₁ ^* + b)' Q₁(Ox? + b).

CAD", O, b ~{

[0070] Alternatively, the CAD model geometry could be selected by symmetry, unlabeled points, or dense imagery, or any of the above methods for geometric lifting. Given the CAD model, the 3D avatar reference texture and lighting fields

T^v = UT_{1 ef} are obtained from the observed images by lifting the observed imagery color values to the corresponding vertices on the 3D avatar via the correspondences x^v{p) e K³, v = \,...,V defined by the geometric information. The problem of estimating the lighting fields and reference texture field becomes the MMSE of each according to

with the summation over the V separate available views, each corresponding to a different target image. Alternatively, the color tinting model or the log-normalization equations as defined above are used.

Normalization of Photometry and Geometry

Photometric Normalization of 3D Avatar Texture

[0071] The basic steps of photometric normalization are illustrated in Fig. 2. Image acquisition system 202 captures a 2D image 204 of the target head. As described above, the system generates (206) best fitting avatar 208 by searching through a library of reference avatars, and by deforming the reference avatars to accommodate permanent or intrinsic features as well as temporary or non-intrinsic features of the target head. Best- fitting generated avatar 208 is photometrically normalized (210) by applying "normal" lighting, which usually corresponds to uniform, white lighting.

[0072] For the fixed avatar geometry CAD model, the lighting normalization process exploits the basic model that the texture field of the avatar CAD model has the multiplicative relationship T(x(p)) = L(x(p))T_ref(x(p)) . For generating the photometrically normalized avatar CAD model with texture imagery T(x),x e CAD , the inverse of the MMSE lighting field L in the multiplicative group is applied to the texture field: L^~λ : T(x) h→ T^mm [X) = L^~] (x) ^■ T(x), x e CAD . (53)

For the vector version of the lighting field this corresponds to componentwise division of each component of the lighting field (with color) into each component of the vector texture field.

Photometric Normalization of 2D Imagery

[0073] Referring again to Fig. 2, best-fitting avatar 208 illuminated with normal lighting is projected into 2D to generate photometrically normalized 2D imagery 212.

[0074] For the fixed avatar geometry CAD model, generating normalized 2D projective imagery, the lighting normalization process exploits the basic model that the image I is in bijective correspondence with the avatar with the multiplicative relationship I(p) <-> T(x(p)) = L(x(p))T_ref(x(p)) ; for multiple images

P'(p) <-» T^v(x(p)) = L^v (x(p))T_ref (x(p)) . Thus normalized imagery can be generated by dividing out the lighting field. For the lighting model in which each component has a lighting function according to

then the normalized imagery is generated according to the direct relationship

In a second embodiment in which there is the common lighting field with separate color components T(x) = (>^Σ>'^W7;>), e'^c+Σ>' ^W7^(x), e'^>' ^wZj(*)) (56)

then the normalization takes the form

1

(57)

L(x(p)) ^v '

In a third embodiment, we view the change as small and additive, which implies that the general model becomes T(x) = ε(x) + T_ιef (x) . The normalization then takes the form

I_noιm(p) = (l^R(p)J^G{p),I^B(p)) - (ε^R(x{p)),ε^G(x(p)), ε^B{x(p))) . (58)

In such an embodiment the small deformation may have a single common shared basis

Nonlinear Spatial Filtering of Lighting Variations and Symmetrization

[0075] In general, the variations in the lighting across the face of a subject are gradual, resulting in large-scale variations. By contrast, the features of the target face cause small-scale, rapid changes in image brightness. In another embodiment, the nonlinear filtering and symmetrization of the smoothly varying part of the texture field is applied. For this, the symmetry plane of the models is used for calculating the symmetric pairs of points in the texture fields. These values are averaged, thereby creating a single texture field. This average may only be preferentially applied to the smoothly varying components of the texture field (which exhibit lighting artifacts).

[0076] Fig. 5 illustrates a method of removing lighting variations. Local luminance values L (506) are estimated (504) from the captured source image I (502). Each measured value of the image is divided (508) by the local luminance, providing a quantity that is less dependent on lighting variations and more dependent on the features of the source object. Small spatial scale variations, deemed to stem from source features, are selected by high pass filter 510 and are left unchanged. Large spatial scale variations, deemed to represent lighting variations, are selected by low pass filter 512, and are symmetrized (514) to remove lighting artifacts. The symmetrized smoothly varying component and the rapidly varying component are added together (516) to produce an estimate of the target texture field 518.

[0077] For the small variations in lighting, the local lighting field estimates can be subtracted from the captured source image values, rather than being divided into them.

Geometrically Normalized 3D Geometry

[0078] The basic steps of geometric normalization are illustrated in Fig. 3. Image acquisition system 202 captures 2D image 302 of the target head. As described above, the system generates (206) best fitting avatar 304 by searching through a library of reference avatars, and by deforming the reference avatars to accommodate permanent or intrinsic features as well as temporary or non-intrinsic features of the target head. Best-fitting avatar is geometrically normalized (306) by backing out deformations corresponding to non-intrinsic and non-permanent features of the target head. Geometrically normalized 2D imagery 308 is generated by projecting the geometrically normalized avatar into an image plane corresponding to a normal pose, such as a face-on view.

[0079] Given the fixed and known avatar geometry, as well as the texture field T(x) generated by lifting sparse corresponding feature points, unlabeled feature points, surface normals, or dense imagery, the system constructs normalized versions of the geometry by applying the inverse transformation.

[0080] From the rigid motion estimation O, b , the inverse transformation is applied to every point on the 3D avatar (O, b)^~l : x e CAD h-ϊ θ'(x - b) , as well as to every normal by rotating the normals O,b : N(x) i-> O'N(x) . This new collection of vertex points and normals forms the new geometrically normalized avatar model CAD"⁰'"' = {(y, N(y)) : y = O'(x - b), N(y) = O'N(x), x e CAD) . (59)

The rigid motion also carries all the texture field T(x),x e CAD of the original 3D avatar model according to

_T ^nom tø _{= T}φ_{χ + b}^_{X G} CAD^norm . (60)

The rigid motion normalized avatar is now in neutral position, and can be used for 3D matching as well as to generate imagery in normalized pose position.

From the shape change φ , the inverse transformation is applied to every point on the 3D avatar f' : i e CAD M> φ^~\x) as well as to every normal by rotating the normals by the Jacobian of the mapping at every point φ^~x : N(x) e

where Dφ is the Jacobian of the mapping. The shape change also carries all of the surface normals as well as the associated texture field of the avatar

7"^! °™ (x) = T(φ(x)\ x e CAD"⁰'"' . (61)

The shape normalized avatar is now in neutral position, and can be used for 3D matching as well to generate imagery in normalized pose position.

For the small deformation deformations φ(x) « x + u(x) , the approximate inverse transformation is applied to every point on the 3D avatar φ^~x : x e CAD H> x - u{x) . As well the normals are transformed via the Jacobian of the linearized part of the mapping Du , and the texture is transformed as above T"^orι" (x) = T(x + u(x)), x e CAD"^on" .

[0081] The photometrically normalized imagery is now generated from the geometrically normalized avatar CAD model with transformed normals and texture field as described in the photometric normalization section above. For normalizing the texture field photometrically, the inverse of the MMSE lighting field L in the multiplicative group is applied to the texture field. Combining with the geometric normalization gives

T"^orm (JC) = I^" ' {-)T{-){Ox + b),x e CAD"⁰"" . (62)

Adding the shape change gives the photometrically normalized texture field

T"^om (x) = r^! (-)T(-)(φ(x)), x e CAD"^om . (63)

Geometry Unknown, Photometric Normalization

[0082] In many settings the geometric normalization must be performed simultaneously with the photometric normalization. This is illustrated in Fig. 4. Image acquisition system 202 captures target image 402 and generates (206) best- fitting avatar 404 using the methods described above. Best-fitting avatar is geometrically normalized by backing out deformations corresponding to non-intrinsic and non-permanent features of the target head (406). The geometrically normalized avatar is lit with normal lighting (406), and projected into an image plane corresponding to a normal pose, such as a face-on view. The resulting image 408 is geometrically normalized with respect to shape (expressions and temporary surface alterations) and pose, as well as photometrically normalized with respect to lighting.

[0083] In this situation, the first step is to run the feature-based procedure for generating the selected avatar CAD model that optimally represents the measured photographic imagery. This is accomplished by defining the set of (i) labeled features, (ii) the unlabeled features, (iii) 3D labeled features, (iv) 3D unlabeled features, or (v) 3D surface normals. The avatar CAD model geometry is then constructed from any combination of these, using rigid motions, symmetry, expressions, and small or large deformation geometry transformation. [0084] If given multiple sets of 2D or 3D measurements, the 3D avatar geometry can be constructed from the multiple sets of features.

The rigid motion also carries all the texture field T(x),x <= CAD of the original 3D avatar model according to T'""^m(x) = T(Ox + b),x s CAD"⁰'^'"' , or alternatively T"^orm{x) = T(φ(x)),x e CAD"^01''" , where the normalized CAD model is

CAD"⁰'^''" = {(y,N(y)) : y = O'(x - b),N(y) = O'N(x),x e CAD} . (64)

The texture field of the avatar can be normalized by the lighting field as above according to

T"^orm (x) = L^~l OT(XOx + b),x e CAD'""^'"' . (65)

Adding the shape change gives the photometrically normalized texture field

T"^om (x) = L^~] (-)T(-)(φ(x)), x e CAD*⁰™ . (66)

The small variation representation can be used as well.

[0085] Once the geometry is known from the associated photographs, the 3D avatar geometry has the correspondence p e [0,l]² <-> x(p) e IR³ defined between it and the photometric information via the bijection defined by the rigid motions and shape transformation. For generating the normalized imagery in the projective plane from the original imagery, the imagery can be directly normalized in the image plane according to

Similarly, the direct color model can be used as well

I_nornXP) = _Υ7 ^} _rrAe-^hl\p\e-^lcI^G(p),e-'°I^B{p)) . (68)

ID Lifting

[0086] Identification systems attempt to identify a newly captured image with one of the images in a database of images of ID candidates, called the registered imagery. Typically the newly captured image, also called the probe, is captured with a pose and under lighting conditions that do not correspond to the standard pose and lighting conditions that characterize the images in the image database.

ID Lifting Using Labeled Feature Points in the Projective Plane

[0087] Given registered imagery and probes, ID or matching can be performed by lifting the photometry and geometry into the 3D avatar coordinates as depicted in Fig. 4. Given bisections between the registered image / and the 3D avatar model geometry, and between the probe image I _pιobe and its 3D avatar model geometry, the 3D coordinate systems can be exploited directly. For such a system, the registered imagery are first converted to 3D CAD models, call them CAD" , a = \,..., A , with textured model correspondences I _ιeΛp) <-> T₁ (x(p)),x e CAD — reg. These CAD models can be generated using any combination of 2D labeled projective points, unlabeled projective points, labeled 3D points, unlabeled 3D points, unlabeled surface normals, as well as dense imagery in the projective plane. In the case of dense imagery measurements, the texture fields T₀₁₁₃₀ generated using the bijections described in the previous sections are associated with the CAD models.

[0088] Performing ID amounts to lifting the measurements of the probes to the 3D avatar CAD models and computing the distance metrics between the probe measurements and the registered database of CAD models. Let us enumerate each of

/D= arg_Dmin_o min_o J lI v, ||_κ ² dt + fN{φ{f^»))^< κ{φ{c(f,^a)),φ{c(f!)))N{φ{fn)

+∑N(Rφ(f_J ^a ))' K ( Rφ(c(f^» )), Rφ{c{f^» ))) N(Rφ(f.^a ))

Removing symmetry involves removing the last 3 terms in the equations.

ID Lifting Using Textured Features

[0094] Given registered imagery and probes, ID can be performed by lifting the photometry and geometry into the 3D avatar coordinates. Assume that bijections between the registered imagery and the 3D avatar model geometry, and between the probe imagery and its 3D avatar model geometry are known. For such a system, the registered imagery is first converted to 3D CAD models CAD" , a = l,..., A with textured model correspondences 1_CAD« (p) <-> T _* (x(p)), x e CAD" . The 3D CAD models and correspondences between the textured imagery can be generated using any of the above geometric features in the image plane including 2D labeled projective points, unlabeled projective points, labeled 3D points, unlabeled 3D points, unlabeled surface normals, as well as dense imagery in the projective plane. In the case of dense imagery measurements, associated with the CAD models are the texture fields T (JAL_n)_a generated using the bijections described in the previous sections. Performing ID via the texture fields amounts to lifting the measurements of the probes to the 3D avatar CAD models and computing the distance metrics between the probe measurements and the registered database of CAD models. One or more probe images I_p ^v _robe(p), p <≡ [0, 1]² , v = 1, ..., V in the image plane are given. Also given are the geometries for each of the CAD models CAD", a = \,...,A , together with associated texture fields T_CA[)I, ,a = \,...,A . Determining the ID from the given images corresponds to choosing the CAD models with texture fields that minimize the distance to the probe:

/D (82)

with the summation over the V separate available views, each corresponding to a different version of the probe image. Performing ID using the single channel model with multiplicative color model takes the form

/D (83)

A fast version of the ID may be accomplished using the log-minimization:

ID

ID Lifting Using Geometric and Textured Features

[0095] ID can be performed by matching both the geometry and the texture features.

Claims

What is claimed is:

1. A method of estimating a 3D shape of a target head from at least one source 2D image of the head, the method comprising: providing a library of candidate 3D avatar models; and searching among the candidate 3D avatar models to locate a best-fit 3D avatar, said searching involving for each 3D avatar model among the library of 3D avatar models computing a measure of fit between a 2D projection of that 3D avatar model and the at least one source 2D image, the measure of fit being based on at least one of (i) a correspondence between feature points in a 3D avatar and feature points in the at least one source 2D image, wherein at least one of the feature points in the at least one source 2D image is unlabeled, and (ii) a correspondence between feature points in a 3D avatar and their reflections in an avatar plane of symmetry, and feature points in the at least one source 2D image, wherein the best-fit 3D avatar is the 3D avatar model among the library of 3D avatar models that yields a best measure of fit and wherein the estimate of the 3D shape of the target head is derived from the best-fit 3D avatar.

2. The method of claim 1, further comprising:

generating a set of notional lightings of the best-fit 3D avatar; searching among the notional lightings of the best-fit avatar to locate a best notional lighting, said searching involving for each notional lighting of the best-fit avatar computing a measure of fit between a 2D projection of the best-fit avatar under that lighting and the at least one source 2D image, wherein the best notional lighting is the lighting that yields a best measure of fit, and wherein an estimate of the lighting of the target head is derived from the best notional lighting.

3. The method of claim 2, wherein the set of notional lightings comprises a set of photometric basis functions and at least one of small and large variations from the photometric basis functions.

4. The method of claim 1, further comprising: generating a 2D projection of the best-fit avatar; comparing the 2D projection with each member of a gallery of 2D facial images;and positively identifying the target head with a member of the gallery if a measure of fit between the 2D projection and that member exceeds a pre-determined threshold.

5. The method of claim 1, further comprising: after locating the best-fit 3D avatar, searching among deformations of the best- fit 3D avatar to locate a best-fit deformed 3D avatar, said searching involving computing the measure of fit between each deformed best-fit avatar and the at least one 2D projection, wherein the best-fit deformed 3D avatar is the deformed 3D avatar model that yields a best measure of fit and wherein the 3D shape of the target head is derived from the best-fit deformed 3D avatar.

6. The method of claim 5, wherein the deformations comprise at least one of small deformations and large deformations.

7. The method of claim 5, further comprising:

generating a set of notional lightings of the deformed best-fit avatar; and searching among the notional lightings of the best-fit deformed avatar to locate a best notional lighting, said searching involving for each notional lighting of the best- fit deformed avatar computing a measure of fit between a 2D projection of the best-fit deformed avatar under that lighting and the at least one source 2D image, wherein the best notional lighting is the lighting that yields a best measure of fit, and wherein an estimate of the lighting of the target head is derived from the best notional lighting.

8. The method of claim 5, further comprising:

generating a 2D projection of the best-fit deformed avatar; comparing the 2D projection with each member of a gallery of 2D facial images; and positively identifying the target head with a member of the gallery if a measure of fit between the 2D projection and that member exceeds a pre-determined threshold.

9. A method of estimating a 3D shape of a target head from at least one source 2D image of the head, the method comprising:

providing a library of candidate 3D avatar models; and searching among the candidate 3D avatar models and among deformations of the candidate 3D avatar models to locate a best-fit 3D avatar, said searching involving, for each 3D avatar model among the library of 3D avatar models and each of its deformations, computing a measure of fit between a 2D projection of that deformed 3D avatar model and the at least one source 2D image, the measure of fit being based on at least one of (i) a correspondence between feature points in a deformed 3D avatar and feature points in the at least one source 2D image, wherein in at least one of the feature points in the at least one source 2D image is unlabeled, and (ii) a correspondence between feature points in a deformed 3D avatar and their reflections in an avatar plane of symmetry, and feature points in the at least one source 2D image, wherein the best-fit deformed 3D avatar is the deformed 3D avatar model that yields a best measure of fit and wherein the estimate of the 3D shape of the target head is derived from the best-fit deformed 3D avatar.

10. The method of claim 9, wherein the deformations comprise at least one of small deformations and large deformations

11. The method of claim 9, wherein the at least one source 2D projection comprises a single 2D projection and a 3D surface texture of the target head is known.

12. The method of claim 9, wherein the at least one source 2D projection comprises a single 2D projection, a 3D surface texture of the target head is initially unknown, and the measure of fit is based on the degree of correspondence between feature points in the best-fit deformed 3D avatar and their reflections in the avatar plane of symmetry, and feature points in the at least one source 2D image.

13. The method of claim 9, wherein the at least one source 2D projection comprises at least two projections, and a 3D surface texture of the target head is initially unknown.

14. A method of generating a geometrically normalized 3D representation of a target head from at least one source 2D projection of the head, the method comprising:

providing a library of candidate 3D avatar models; and searching among the candidate 3D avatar models and among deformations of the candidate 3D avatar models to locate a best- fit 3D avatar, said searching involving, for each 3D avatar model among the library of 3D avatar models and each of its deformations, computing a measure of fit between a 2D projection of that deformed 3D avatar model and the at least one source 2D image, the deformations corresponding to permanent and non-permanent features of the target head, wherein the best-fit deformed 3D avatar is the deformed 3D avatar model that yields a best measure of fit; and generating a geometrically normalized 3D representation of the target head from the best-fit deformed 3D avatar by removing deformations corresponding to non- permanent features of the target head.

15. The method of claim 14, wherein the avatar deformations comprise at least one of small deformations and large deformations.

16. The method of claim 14, further comprising generating a geometrically normalized image of the target head by projecting the normalized 3D representation into a plane corresponding to a normalized pose.

17. The method of claim 16, wherein the normalized pose corresponds to a face-on view.

18. The method of claim 16, further comprising:

comparing the normalized image of the target head with each member of a gallery of 2D facial images having the normal pose; and positively identifying the target 3D head with a member of the gallery if a measure of fit between the normalized image of the target head and that gallery member exceeds a pre-determined threshold.

19. The method of claim 14, further comprising generating a photometrically and geometrically normalized 3D representation of the target head by illuminating the normalized 3D representation with a normal lighting.

20. The method of claim 19, further comprising generating a geometrically and photometrically normalized image of the target head by projecting the geometrically and photometrically normalized 3D representation into a plane corresponding to a normalized pose.

21. The method of claim 20, wherein the normalized pose is a face-on view.

22. The method of claim 20, wherein the normal lighting corresponds to uniform, diffuse lighting.

23. A method of estimating a 3D shape of a target head from source 3D feature points of the head, the method comprising:

providing a library of candidate 3D avatar models; searching among the candidate 3D avatar models and among deformations of the candidate 3D avatar models to locate a best-fit deformed avatar, the best-fit deformed avatar having a best measure of fit to the source 3D feature points, the measure of fit being based on a correspondence between feature points in a deformed 3D avatar and the source 3D feature points, wherein the estimate of the 3D shape of the target head is derived from the best-fit deformed avatar.

24. The method of claim 23, wherein the measure if fit is based on a correspondence between feature points in a deformed 3D avatar and their reflections in an avatar plane of symmetry, and the source 3D feature points.

25. The method of claim 23, wherein at least one of the source 3D points is unlabeled.

26. The method of claim 23, wherein at least one of the source 3D feature points are normal feature points, wherein the normal feature points specify a head surface normal direction as well as a position.

27. The method of claim 23, further comprising:

comparing of the best-fit deformed avatar with each member of a gallery of 3D reference representations of heads; and positively identifying the target 3D head with a member of the gallery of 3D reference representations set if a measure of fit between the best-fit deformed avatar and that member exceeds a pre-determined threshold.

28. A method of estimating a 3D shape of a target head from at least one source 2D image of the head, the method comprising:

providing a library of candidate 3D avatar models; and searching among the candidate 3D avatar models and among deformations of the candidate 3D avatar models to locate a best-fit deformed avatar, the best-fit deformed avatar having a 2D projection with a best measure of fit to the at least one source 2D image, the measure of fit being based on a correspondence between dense imagery of a projected 3D avatar and dense imagery of the at least one source 2D image, wherein at least a portion of the dense imagery of the projected avatar is generated using a mirror symmetry of the candidate avatars, wherein the estimate of the 3D shape of the target head is derived from the best- fit deformed avatar.

29. A method of positively identifying at least one source image of a target head with a member of a database of candidate facial images, the method comprising:

providing a library of 3D avatar models; searching among the 3D avatar models and among deformations of the candidate 3D avatar models to locate a source best-fit deformed avatar, the source best-fit deformed avatar having a 2D projection with a best first measure of fit to the at least one source image; for each member of the database of candidate facial images, searching among the library of 3D avatar models and their deformations to locate a candidate best- fit deformed avatar having a 2D projection with a best second measure of fit to the member of the database of candidate facial images; positively identifying the target head with a member of the database of candidate facial images if a third measure of fit between the source best-fit deformed avatar and the member candidate best-fit deformed avatar exceeds a predetermined threshold.

30. The method of claim 29, wherein the first measure of fit is based at least in part on a degree of correspondence between feature points in the source best-fit deformed avatar and their reflections in the avatar plane of symmetry, and feature points in the at least one source 2D image.

31. The method of claim 29, wherein the second measure of fit is based at least in part on a degree of correspondence between feature points in the candidate best-fit deformed avatar and their reflections in the avatar plane of symmetry, and feature points in the member of the database of candidate facial images.