WO2010042990A1

WO2010042990A1 - Online marketing of facial products using real-time face tracking

Info

Publication number: WO2010042990A1
Application number: PCT/AU2009/001360
Authority: WO
Inventors: Timothy James Henry Edwards; Nicholas John Langdale-Smith
Original assignee: Seeing Machines Limited
Priority date: 2008-10-16
Filing date: 2009-10-15
Publication date: 2010-04-22

Abstract

Face tracking is used to display a live image of the person being tracked with a synthetically generated 3D model of eyewear (for example, sunglasses or spectacles) overlaid on the person's face to provide a virtual mirror for the person, for example to improve the sales and marketing of eyewear over the internet. The person need not remove their existing glasses; by tracking the outline of the glasses and using 2D image recovery any real glasses are removed and replaced with virtual ones. This ensures the synthetically rendered image is convincing. Optionally, since facial features are tracked in 3D with a high degree of accuracy and precision, the user can also select eyewear based on an appropriate fit to their facial proportions.

Description

Online marketing of facial products using real-time face tracking

Field of the Invention

[0001] The present invention relates to the field of real time facial tracking and, in particular, to the online marketing of facial products. Background

[0002] Standard online internet sales and marketing of facial products, such as eyewear, is limited as the customer cannot preview, as they would in a retail store, what they will look like wearing the product nor whether the product will fit satisfactorily. 2D-based approaches are inadequate as they do not provide a realistic experience for the customer. 3D-based approaches provide more realism, but current systems, such as those related to eyewear, have limitations.

[0003] Unites States Patent 6692127 to Abitbol et al. (US 6692127) discloses a spectacle fitting system and method using a 3D representation of the client's face and a 3D representation of a spectacle frame. The visualisation system is not based on live images of the client. The virtual head/face is created from a set of multiple images of the customer taken initially.

[0004] Unites States Patent 7016824 to Waupotitsch et al. (US 7016824) discloses systems and methods for an interactive try-on platform for eyeglasses for online commerce. A fully-textured 3D face model and a 3D representation of a selected pair of glasses are used. The visualisation system is not based on live images. The 3D face model is generated in advance.

[0005] United States Patent Application US 2005/0162419 to Kim et al. (US

2005/0162419) discloses a 3D virtual simulation system and method for selection and purchase of eyeglasses. The visualisation system is not based on live images. The 3D face model is created from photo images of the face with options to select hair models. Summary of the Invention

[0006] It is an object of the present invention to provide an improved an improved system and method for online marketing of facial products.

[0007] In accordance with a first aspect of the present invention, there is provided an interactive system comprising: a video capture device for capturing a video image of the face of a system user; a rendering system for rendering a synthetically generated 3D model of eyewear on the face of the video image; and a display system for displaying to the user a view of the rendered image. Preferably, the system operates in real time. The rendering system preferably can include tracking of the 3D location and orientation of a user's face.

[0008] In accordance with a further aspect of the present invention, there is provided an interactive system comprising: a video capture device for capturing a video image of the face of a system user; a rendering system for rendering a synthetically generated 3D model of an object a user may wear on the face of the video image; and display system for displaying to the user a view of the rendered image.

[0009] The object can in some embodiments, include one of a necklace, sunglasses, hair style, cosmetics, cosmetic procedures, tattoos, beauty treatments or any other procedure that affects the facial visual appearance of the user.

[0010] hi accordance with a further aspect of the present invention, there is provided a method of providing interactive feedback to a user of the projected facial appearance of an object, the method comprising the steps of: (a) capturing a real time image of the user's face; (b) projecting a 3D object on to the face within the image; (c) rendering the 3D object on the face to create a rendered image; (d) displaying the rendered image to the user in a real time feedback loop. A synthetic face can be rendered for certain cosmetic treatments or procedures where the texture and/or geometry of the face image require alteration. Brief Description of the Drawings

[0011] Benefits and advantages of the present invention will become apparent to those skilled in the art to which this invention relates from the subsequent description of exemplary embodiments and the appended claims, taken in conjunction with the accompanying drawings, in which:

[0012] Fig. 1 illustrates a customer using the system as a virtual mirror to preview sunglasses;

[0013] Fig. 2 illustrates a diagram of the system when used for online marketing; and

[0014] Fig. 3 illustrates a diagram of a generalized algorithm of virtual mirror system. Description of Preferred and Other Embodiments

[0015] The preferred embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings.

[0016] The preferred embodiments provide a system to display a live image of the customer being tracked with a synthetically generated 3D model of eyewear, for the purpose of seeing the item on the face to improve the sales and marketing of eyewear.

The system provides robust tracking with precise feature location and registration at large angles of head rotation.

[0017] In preferred embodiments, the customer need not remove their existing glasses during face tracking. The system can be used in conjunction with an online store, to allow the customer to virtually "try on" the eyewear in a realistic and convenient way.

Ideally, the preferred embodiments are robust at large angles of head rotation so that the customer can see themselves with eyewear from the side. Ideally, the virtual mirror is indistinguishable from a real mirror.

[0018] The preferred embodiment is directed to the skilled computer graphics programmer or skilled computer vision programmer and in particular, to the skilled 3D real time graphics programmer and familiarity with such system is assumed. In particular, familiarity with such systems as those described in United States Patent US - A -

7043056 and PCT Patent Application Publication Numbers WO 2003/081532, WO 2004/003849, WO 2007/062478, and WO 2008/106725, the contents of which are hereby incorporated by cross reference, is assumed.

[0019] The preferred embodiments utilize accurate calibrated 3D face-tracking technology with automatic feature recognition such as provided by faceAPI ™ (available from www.faceapi.com) in an online eyewear sales portal which allows customers to virtually "try on" eyewear.

[0020] Fig. 1 illustrates an operational environment of the preferred embodiment 1. hi this environment, a user 2 interacts with a computer application projected on screen 3. The application shows the user with a pair of sunglasses on 5. Imagery of the user is captured via the camera 4.

[0021] Fig. 2 illustrates schematically the operational environment of the preferred embodiment. The captured images 4 are fed to a face tracking application 7 running on computer 6. The application 7 is in turn interconnected to the internet 10 and a backend application 1 1 which in turn interacts with a database of 3D models of eyewear 12. It should be realized, the system could also function utilizing a local network between the application 7 and the backend application 11. Similarly, any of the interconnections between components of the system can also be network connections. The customer sits in front of a web-camera, which produce a video feed of their face. The application on computer 6 then finds and tracks the customer's facial coordinates as they move. The system renders a currently selected 3D model of eyewear over their face in real-time. The system ideally allows robust tracking and precise feature location and registration at large angles. Different 3D models can be downloaded for rendering on demand from database 12.

[0022] The 3D model rendering allows the customer to rotate their head and observe how the eyewear would appear on them from the side and other angles, much like they would do in a real store. Furthermore, it allows the customer to 'try on' as many of the eyewear articles listed in the online catalogue as they would like with ease. Furthermore, it allows the customer to send videos or pictures of their appearance, whilst wearing the virtual glasses, to friends for feedback. The catalogue of glasses can include highly realistic 3D models of the glasses for sale. The system further allows an indication of the frame/face fit to be calculated and presented to the customer. For instance, the customer could be presented with a selection of eyewear from the catalogue that would ensure an appropriate fit to their face.

[0023] The basics of the application within the computer 6 can be as shown in the flow chart of Fig. 3. The camera inputs a live image of the user 31. This is utilized by a face tracking system 32 to determine the 3D location data of the face 33. This data is then utilized in conjunction with a downloaded 3D model of the eyewear, to render the eyewear in real time on the user's face, with the output being displayed to the display 36.

[0024] Furthermore, the person need not remove their existing glasses; by tracking the glasses outline and using 2D image recovery, the real glasses can be removed and replaced with virtual ones, in real-time. This ensures the synthetically rendered image is convincing.

[0025] An embodiment includes tracking aspects of humans using image information obtained from a camera or set of cameras. These aspects can include any one or more of: the size, shape, texture and orientation of the onlooker's head, face, eyes (including eye features), and facial features; presence, shape and orientation of glasses, sunglasses or other head adornments (including jewellery, hats); skin colour and texture; and shadow geometry.

[0026] An another embodiment the system can be located in a kiosk or at a touch screen at a public place or in-store that allows the user to "try on" items that are suitable for their face type and proportions.

[0027] Another embodiment allows customers to preview other items or procedures they are considering purchasing like hair styles, cosmetics, cosmetic procedures, tattoos, beauty treatments or any other procedure that affects visual appearance. One or more beauty metrics for the customer, before and/or after the simulation, can be calculated and displayed for the customer.

[0028] In other embodiments, the virtual mirror can include a replay feature with the ability to pause and initiate other standard video editing capabilities, or one or more display screens positioned around the customer. This would allow the user to see themselves from angles of rotation that usually would not be possible with a standard mirror (such as the back of their own head for previewing hairstyles).

Interpretation [0029] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

[0030] Similarly it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention. [0031] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination. [0032] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention. [0033] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details, hi other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. [0034] As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner. [0035] hi the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements/features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements/features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising. [0036] Similarly, it is to be noticed that the term coupled, when used in the claims, should not be interpreted as being limitative to direct connections only. The terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression a device A coupled to a device B should not be limited to devices or systems wherein an output of device A is directly connected to an input of device B. It means that there exists a path between an output of A and an input of B which may be a path including other devices or means. "Coupled" may mean that two or more elements are either in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but yet still co-operate or interact with each other.

[0037] Although the present invention has been described with particular reference to certain preferred embodiments thereof, variations and modifications of the present invention can be effected within the spirit and scope of the following claims.

Claims

We Claim:

1. An interactive system comprising: a video input device for capturing a video image of the face of a system user;

a rendering system for rendering a synthetically generated 3D model of eyewear on the face of the video image; and

a display system for displaying to the user a view of the rendered image.

2. A system as claimed in claim 1 wherein the said system operates in real time.

3. A system as claimed in claim 1 wherein said rendering system includes tracking of the 3D location and orientation of a user's face.

4. An interactive system comprising:

a video input device for capturing a video image of the face of a system user;

a rendering system for rendering a synthetically generated 3D model of an object a user may wear on the face of the video image; and

a display system for displaying to the user a view of the rendered image.

5. A system as claimed in claim 4 wherein the object is one of a necklace, sunglasses, hair style, cosmetics, cosmetic procedures, tattoos, beauty treatments or any other procedure that affects the facial visual appearance of the user.

6. A method of providing interactive feedback to a user of the projected facial appearance of an object, the method comprising the steps of:

(a) capturing a real time image of the user's face;

(b) projecting a 3D object on to the face within the image;

(c) rendering the 3D object on the face to create a rendered image; (d) displaying the rendered image to the user in a real time feedback loop.

7. An interactive system for rendering objects on a user's face, substantially as herein described with reference to any one of the embodiments of the invention illustrated in the accompanying drawings and/or examples.

8. A method of providing interactive feedback to a user of the projected facial appearance of an object, the method substantially as herein described with reference to any one of the embodiments of the invention illustrated in the accompanying drawings and/or examples.