WO2007000685A1

WO2007000685A1 - Method of operating a camera for taking electronic images, camera for taking electronic images

Info

Publication number: WO2007000685A1
Application number: PCT/IB2006/051999
Authority: WO
Inventors: Rajendra S. Sisodia; Ranjith Gopalakrishnan
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2005-06-28
Filing date: 2006-06-21
Publication date: 2007-01-04

Abstract

The invention relates to a camera and a method of operating such a camera for taking electronic images wherein a facial emotive index is calculated for an electronic image comprising one or more human faces, said facial emotive index is coupled to the electronic image, said facial emotive index is displayed and a prompting signal is given to prompt the user to accept, keep, or delete said electronic image or take a further electronic image.

Description

Method of operating a camera for taking electronic images, camera for taking electronic images

The invention relates to a method of operating a camera for taking electronic images. The invention also relates to a camera comprising means for taking electronic images and a display for displaying said electronic images.

Electronic cameras are becoming more and more widely used. Electronic cameras have the advantage that the electronic images taken with the camera can be relatively easily sent to other users and relatively easily improved. As the use and the volume of electronic images grow, the task of managing an ever growing selection of electronic images becomes more and more difficult. Many users take large quantities of photos making it difficult if not impossible to keep track of all the photos.

To store the electronic images the cameras use storage device such as memory cards. Although the capabilities of such storage means are ever increasing, the users take as many pictures as they can shoot, resulting sooner or later, in the memory card being full. It is relatively easy, technically speaking, to delete images and thereby to create space for more images to be taken. However, choosing which images to delete from among a large collection of images is far from easy. Conventionally the camera has some means of displaying the images in a display window. Two options are usually given, one image on the display window, or a number of images on the display window. It is, however, very difficult to make a right decision in regards to managing taking images on the basis of the images displayed on the display window. When a number of images are presented on the display of the camera, the images are so small that little or nothing can be seen on the images, even if one can find its glasses. If one image is visible, the other images cannot be seen and it is impossible to make a proper comparison. The user is thus left to scan back and forth between the images to make a decision. More often than not, by the time the user has made the decision, the reason for wanting to take yet another picture has passed.

One solution may be thought to be to increase the storage capacity of the storage device, in other words to allow more and more images to be taken before any problem with storage capacity occurs. However, this may mean that it will take longer before the problem arises, but if it does, the problem is very large indeed, since the number of images stored is larger and thus it will take longer to make a decision if the problem occurs. Even if the problem of having to delete images does not occur, as the volume of electronic images grows, it becomes more and more difficult to manage the volume of images and to make any decision, whether it be deletion of particular images from the selection to make room for further images, selection of images for close inspection, finding the right images for display, whether or not yet another image should be taken etc. etc.

In other words the problem of managing the collection of images taken and/or stored by a electronic camera becomes ever greater.

It is an object of the invention, in its various aspects to provide a method and a camera in which the user is aided in managing a collection of electronic images.

To this end the method of operating a camera, in a first aspect, is characterized in that a facial emotive index is calculated for an electronic image comprising one or more human faces, said facial emotive index is coupled to the electronic image, said facial emotive index is displayed and a prompting signal is given to prompt the user to accept, keep, delete said image or take a further electronic image.

The problem described above i.e. managing a collection of electronic images, e.g. a full memory card and yet a lot of images to be taken, typically arises at social gatherings such as weddings, new year celebrations, end of Ramadan festivities, bar mitzvah's, high school reunions, funerals etc. At such social gatherings many images are taken. When the memory card is lull the problem arises: which images should be deleted to make room for new images? A similar problem arises when one wants to show or view the images taken at such social gathering. The sheer volume of the images taken usually means that it is very difficult to manage the collection in such a way that a meaningful selection of images is presented, resulting in boring, endless video or photo viewing sessions. Thus it becomes more and more important to manage the collection of images taken by the camera, wherein quality becomes more and more important as the quantity increases. The most effective way of managing the collection is at the source, i.e. in the camera. However, present day camera offer all kind of iunctionalities, but they do not offer a means for managing the collection of data other than taking a lot of images and afterwards deleting them by hand. Often, the user would like to preserve or take images that best capture the mood of the occasion, which will often, but not always, be happiness. It is virtually impossible to do so with conventional cameras. Such camera have a display. However, if the display of the camera displays a collection of images, the images are typically stamp sized, and simply too small to make any judgement. Even if the display is used for one image, the image is quite coarse and cannot be easily compared to other images. Known cameras do not offer any possibilities for managing the collection of images, i.e. preserve or take image, according to the mood of the occasion. By assigning a facial emotive index to the electronic images, and using a prompting signal based on the emotive index, the user is aided in his or her decisions in managing the collection, i.e. accepting, acquiring or deleting images form the collection in accordance with the mood of the occasion. The facial emotive index may be displayed separately or with the corresponding image. The latter is preferred, since it allows the user to establish a notion of the relationship between the emotive index and the actual images. This is especially useful e.g. if and when the camera comprises a means for setting a standard emotive index. Displayed with means that the user can see the image and the emotive index or an indication of the emotive index together. The emotive index can be displayed separately, in or near the image, or the image can be so displayed that the image itself indicates the emotive index. Displaying of the emotive index means that the user is provided with a visual or otherwise indication, such as sound, indicating the emotive index. Preferably a visual indication is given.

To calculate a facial emotion parameters is in itself known, for instance from the article "Emotion recognition Using a Cauchy Naϊve Bayes Classifier" by Nicu Sebe, Michael S. Lew, Ira Cohen, Ashotosh Garg and Thomas S. Huang 2002 IEEE, page 17-20. Another example is known from "Real World, Real-time Automatic Recognition of Facial Expressions" by Ying-Li Tian et al, www.research.ibm.com/peoplevision/PETS2003.pdf.

Based on the characteristic of the image of a face a number of emotions can be distinguished usually as by anger, disgust, fear, sadness, happiness and surprise.

Although techniques for extracting facial emotion parameters are thus known in prior art documents they are used solely for machine recognition of human expressions, for instance to judge in an experiment the facial expression of test person in experiments, or to judge mental illness for a patient. They do not offer any guidance to a person seeking to better manage a collection of images taken by an electronic camera.

The method of operating the camera to better manage the collection of images stored or taken by the camera can be executed in one or more of the following ways: 1. the emotive index of a taken electronic image is displayed with or in association with the electronic image and a prompting signal is given, for instance displayed on the display, to prompt the user to take a decision to store said electronic image or not. This allows the user to immediately judge whether or not to store the taken image and whether or not to take a further image. The emotive index is thus used as a managing parameter to manage the collection of images directly at the source, i.e. whether or not to store or keep the image in storage or to take a further image based on the displayed emotive index. The prompting signal may be a signal separate from the indication of the emotive index, or it may be incorporated in the indication of the emotive index. For instance a small green square may indicate that the emotive index is up to standard and, if the user does not take any action within a few seconds the image is accepted. For simple decisions the indication of emotive index and the prompting signal may thus be incorporated in one signal, for instance a green or red flashing signal reading "OK?' (green) or "reject?'(red).

2. as a further embodiment of the above aspect: wherein the emotive index of the last taken image is compared to a preset emotive index and the comparison is shown. This allows the user to judge whether the emotive index of the image is up to the standard the user wishes to impose on his or her collection. He/she can then reject or accept the image for storaging the image, or may decide to take another image. The camera in accordance with this embodiment is preferably provided with a setting means for presetting of the emotional preset by the user, which are then converted into an expected emotive index range. Based on the value of the emotive index of an image that has been taken, a decision about accepting or rejecting the image will be taken and/or a decision to take a further image.

3. a number of images of a collection of images are sorted by ascending or descending facial emotive index and are shown to the user in sorted order and the prompting signal is displayed to prompt the user select one or more images among the collection for keeping or deleting from the collection. Sorting one or more images on emotive index allows the user to select and keep in storage those images that adhere to his or her standards.

Thus 'managing of the collection of images' comprises in this aspect of the invention three points which, within the concept of the invention, may be present alone or in any combination: a. accepting or rejecting an image taken prompted by a prompting signal which is based on the emotive index (acceptance of taken images on the basis of emotive index) b. Take yet another image on the same basis (acquisition of further images on the basis of emotive index of taken images) c. Sorting the images in the collection in accordance with ascending or descending emotive index, and deciding to keep or erase one or more images prompted by a prompting signal ("cleaning up' of the collection on the basis of emotive index of taken images)

The following steps are common to this aspect of the invention: calculating an emotive index and assigning the emotive index to the image taken displaying said emotive index, which may be either direct, as a number or e.g. colour indication (point 1 above), may be indirect by reference to a preset emotive index (point 2 above), or implicit by displaying the images in a sorted order (point 3) prompting the user to make a decision to store, keep or reject the image or to take another image, i.e. to manage the collection of electronic images.

The camera in accordance with the first aspect of the invention is characterized in that it comprises a calculator for calculating for an electronic image comprising one or more faces a facial emotive index, a coupler for coupling said facial emotive index to the electronic image, a display, and means for displaying said facial emotive index with the electronic image on the display, and means for giving a prompting signal on the display.

In the first aspect of the invention as described above, a prompting signal based on the calculated emotive index is given and the user reacts to the prompting signal. In point 2 above one step towards a more controlling role of the user is taken by comparing the emotive index to a preset value for the emotive index. In the latter case the camera has means to set the preset value. These means may be simple setting procedure, i.e. the user sets a value, or it may be a more complex procedure in which the camera is taught which level the user wants to set. This is for instance possible by showing the user a collection of prerecorded images and having him/her choose which are acceptable and which are not acceptable. In a further aspect of the invention the method and camera provides the user with a more proactive role within the same basic inventive idea. The user may set a preset emotive index and any image which is not up to that standard is rejected, even without a prompting signal being given.

The method according to this second aspect of the invention is characterized in that for an electronic image comprising an image of one or more faces a facial emotive index is calculated, said facial emotive index is compared to a preset emotive index, and, if the calculated emotive index is below the preset emotive index, the image is rejected.

The camera is characterized in that it comprises a calculator for calculating for an electronic image comprising an image of one or more faces a facial emotive index, a comparator for comparing the calculated emotive index to a preset emotive index and providing a comparison signal, and a rejector/storer for rejecting or storing the electronic image coupler in dependence of the comparator signal.

Within the invention a method and camera both aspects may be incorporated in a single method or camera. For instance there may be more than one preset emotive index, a first preset value indicating a green or blue range for which any image is accepted, even without a prompting signal, a second indicating an orange range, for which a prompting signal is given and the user decides after inspection of the image to accept or reject the image, and a third preset value indicating a red range for which the images are outright rejected, without a prompting signal.

In the first and second aspect of the invention described above the method and camera are arranged to aid the user in collecting images for a collection or deleting image from a collection. The managing is done at the source, i.e. the camera, to help the user to increase the 'emotional' quality (as supposed to mere quantity) of the collection of electronic images with the use of a facial emotive index.

In yet a third aspect of the invention, the user is aided in finding within a collection of images, which may be a collection of pictures or for instance a video sequence, those image that are most pleasing to him/her. Thus the object is aid the user to find within an already taken collection of electronic images subcollections of high emotional quality. This may be done in the camera itself or in other devices such as a personal computer. This also allows a better management of the collection since it allows to select or earmark in a large collection or lengthy video sequence those images that are most pleasing to the viewer.

In accordance with this third aspect of the invention, a method for selecting electronic images within a collection of electronic images is characterized in that for an electronic image comprising one or more human faces a facial emotive index is calculated, said facial emotive index is coupled to the image, and for a collection of images, the facial emotion indices of the collection of images are compared to a preset emotive index, and a subcollection is made of accepted images for which the emotive index is above the preset emotive index. This aspect of the invention allows a user to find within the collection of images, those images that are most pleasing.

In a further embodiment of the invention, wherein the collection of images is a video sequence, the method is characterized in that it is established where, within the video sequence, the accepted images are present, and to the subcollection images intermediate between accepted images within the video sequence are added.

This allows constructing a time track within a video sequence. For each of the three aspects of the invention, in both method and camera, the facial emotive index or indices may be calculated on basis of: the facial emotion indices of all persons within the electronic image. A collective facial emotive index can be made by calculating for each of the human faces within a group the individual facial emotive index and constructing a collective facial emotive index. In one embodiment the facial emotive index is equal to an average emotive index, in another embodiment the collective emotive index equals the minimum facial emotive index. Yet another collective facial emotive index may be the 25% mark (or in general , i.e. the lowest 25% of the facial emotion indices). The prompt signal or acceptance or selection is then dependent on whether or not on average the persons depicted in the electronic image are in the right mood (average), whether nobody is sad (minimum), or at least, enough people are in the right mood (x% mark).

A iacial emotive index of a select group of persons, independent of the facial emotion of any further persons within the image. For instance only the bride and groom when the subject of the collection of images is a wedding, independent of the facial emotive index of other persons within the party. It is remarked that it is known to scan a collection of images to find a characteristic face, such as the face of a particular person.

These and further aspects of the invention will be explained in greater detail by way of example and with reference to the accompanying drawings and examples.

In the drawings:

Fig.1 shows a group photo with a display of a comparison between a set emotive index and a calculated average emotive index.

Fig. 2 illustrates a camera in accordance with the invention.

The present invention will now be described more fully hereinafter with reference to the accompanying drawings and examples. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout. Fig. 1 shows a group photo. In this example the camera has a setting means and one of the settings is 'Cheerful'. If the camera emotive mode is set at 'cheerful" and majority of people are cheerful during photo snap-shot, then the user is intimated about the COMPUTED EMOTIVE INDEX = 7 (say) and if CHEERFUL is determined as EMOTIVE INDEX > 6, the photo is accepted and user is prompted "OK" (or green light), If not the user is prompted that the computed emotive index of the current image is not as expected because too many people are not cheerful enough and gives user an option to re-try or accept the photo as it is. If and when the photo is accepted the computed emotive index becomes attached to the photo (for instance as meta-data).

This solves for instance the following problem:

Often during or for a certain social gathering photos are made which are then sent to others. As an example: for a 50-year wedding anniversary a festivity is planned. Almost always invitation cards are made to invite guests to the festivities. Of course, one could use standard invitation cards, but personalized cards are much better suited. Recent photos of the married couple (with or without their children) are often used for such invitation cards. These are invitation cards, so preferably the photos are inviting, i.e. cheerful.

Often tasks are divided; one of the children takes the photos; another one takes care of making the invitation card. However, usually many pictures are taken, so the photographer has to make a selection of the pictures to be sent (via e-mail) to the other. Often in the pictures sent either one of the parents smiles, but rarely both of them. So the problem arises that it is difficult to find a good picture for the invitation card. The invention solves this problem, either directly because when taking the pictures a pre-selection could have taken place (i.e. it would have been apparent that never both of the parents smiled) or, afterwards, it is possible, using the meta-data on collective emotive index to sent the 'best selection' (i.e. with the highest average 'smile quotient'). Even if the best picture would not have been good enough, at least it is known that this is the case! Presently this is not possible. By making such a pre-selection it is also much easier to agree on the photo to be chosen, instead of a collection of hundreds of photos, may be 10-20 photos need to be considered.

A further example of the method and device of the invention is:

Consider that we have a movie in which the faces of several individuals are seen. This might be a short clip where some activity is taking place (like the cutting of a birthday cake, or the bride and groom kissing after the wedding). In the various frames associated with the movie, different people are having different emotions. In this context, we would like to extract out the frame (or frames), in which the maximum number of people have the desired emotion (either in the foreground or background). For example, in the birthday cake example, we want the frame that has the maximum number of people smiling. Additionally, we might identify some people (objects) in the video and we want to select the frame in which those people most important for the occasion, for instance at a wedding the bride and groom, are having the right emotion. The method and camera in accordance with the invention would select, by identifying the emotive index and selecting those frames with the highest (or higher than a preset values) emotive index, and prompting the user to store or put a label on such frames. The method in accordance with the invention facilitates this scenario, too. It helps to identify the facial emotive index associated with each frame, and hence, select the best frame.

It is remarked that facial recognition methods, i.e. methods for recognizing certain faces within an image or frame are well known, for instance many automatic surveillance systems are based on facial recognition to identify and track the movements of certain individuals.

Fig. 2 illustrates schematically a camera in accordance with an embodiment of the invention. The camera 1 has a display 2 for displaying the image (either before it is actually shot or afterwards). The camera has an emotive index setting bar 3, which can be tuned upwards (high emotive index) and lower (low emotive index). In this example an indication bar of similar shape and size is positioned next to the setting bar. This indication bar indicates the actual emotive index (by a small indicator) and/or a colour indication (red, orange or green, depending on whether the computed emotive index is below, near of above the set emotive index). The indication bar may itself form the prompting signal and the prompting signal may then be an implicit signal (for instance when the user does nothing, the photo is stored or rejected in accordance with the comparison between the indicated emotive index and the set emotive index). The user is able to see whether the image is up to standards, he/she is also able to see whether the set emotive index is a realistic one for the occasion or group of persons. The camera has a means for coupling (e.g. as meta-data) the emotive index of the photo to the photo. The user may accept or reject the image, for instance by pressing on the top (acceptance) or bottom (rejection) of the indication or setting bar, or by pressing a knob on top of the camera.

It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. The invention resides in each and every novel characteristic feature and each and every combination of characteristic features. Reference numerals in the claims do not limit their protective scope. Use of the verb "to comprise" and its conjugations does not exclude the presence of elements other than those stated in the claims. Use of the article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

The present invention has been described in terms of specific embodiments, which are illustrative of the invention and not to be construed as limiting. The invention may be implemented in hardware, firmware or software, or in a combination of them. Other embodiments are within the scope of the following claims. The invention is also embodied in any computer program comprising program code means for performing a method in accordance with the invention when said program is run on a computer as well as in ay computer program product comprising program code means stored on a computer readable medium for performing a method in accordance with the invention.

Claims

CLAIMS:

1. A method of operating a camera for taking electronic images wherein a facial emotive index is calculated for an electronic image comprising one or more human faces, said facial emotive index is coupled to the electronic image, said facial emotive index is displayed and a prompting signal is given to prompt the user to accept, keep, or delete said electronic image or take a further electronic image.

2. A method as claimed in claim 1 , wherein the emotive index of a taken electronic image is displayed with or in association with the electronic image and a prompting signal is given to prompt the user to take a decision to store said electronic image or not.

3. A method as claimed in claim 1, wherein the emotive index of the electronic image is compared to a preset emotive index and the comparison is shown.

4. A method as claimed in claim 1 , wherein a number of images of a collection of images are sorted by ascending or descending facial emotive index and are shown to the user in sorted order and the prompting signal is displayed to prompt the user select one or more images among the collection for keeping or deleting from the collection.

5. A camera comprising means for taking electronic images and, preferably, a display (2) for displaying said images, wherein the camera comprises a calculator for calculating for an electronic image comprising an image of one or more faces a facial emotive index, a coupler for coupling said facial emotive index to the electronic image, a display, and means (4) for displaying said facial emotive index with the electronic image on the display, and means for giving a prompting signal.

6. A camera as claimed in claim 5, wherein the means for giving a prompting signal are for prompting the user to take a decision to store said electronic image or not.

7. A camera as claimed in claim 5, wherein the camera comprises means (3) for setting a preset facial emotive index and means for comparing an emotive index of an electronic image to the preset facial emotive index, and means to display the comparison.

8. A method of operating a camera for taking electronic images wherein for an electronic image comprising an image of one or more faces a facial emotive index is calculated, said facial emotive index is compared to a preset emotive index, and, if the calculated emotive index is below the preset emotive index, the image is rejected.

9. A camera comprising means for taking electronic images and a display for displaying said images, wherein the camera comprises a calculator for calculating for an electronic image comprising an image of one or more faces a facial emotive index, a comparator for comparing the calculated emotive index to a preset emotive index and providing a comparison signal, and a rejector/storer for rejecting or storing the electronic image coupler in dependence of the comparator signal.

10. A method for selecting electronic images within a collection of electronic images is characterized in that for an electronic image comprising one or more human faces a facial emotive index is calculated, said facial emotive index is coupled to the image, and for a collection of images, the facial emotion indices of the collection of images are compared to a preset emotive index, and a subcollection is made of accepted images for which the emotive index is above the preset emotive index. .

11. A method as claimed in claim 10, wherein the collection of electronic images is a video sequence and it is established where, within the video sequence, the accepted images are present, and to the subcollection images intermediate between accepted images within the video sequence are added.

12. A method as claimed in any of the claims 1 to 4, 8, 10 or 11, wherein the facial emotion indices of all persons within the electronic image are calculated and a collective emotive index is calculated.

13. Computer program comprising program code means for performing a method as claimed in any one of claims 1 to 8 and 10 to 12 when said program is run on a computer.

14. Computer program product comprising program code means stored on a computer readable medium for performing a method as claimed in any one of claims 1 to 8 and 10 to 12 when said program is run on a computer.