WO2005119515A1

WO2005119515A1 - Updating video summary

Info

Publication number: WO2005119515A1
Application number: PCT/IB2005/051725
Authority: WO
Inventors: Mauro Barbieri; Lalitha Agnihotri
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2004-06-01
Filing date: 2005-05-26
Publication date: 2005-12-15
Also published as: TW200608243A

Abstract

The present invention relates to a method and an electronic device (100) for enabling dynamically affecting a summary of a multimedia stream, where the method comprises the steps, presenting the summary to a user (step 202), obtaining user input related to topics appearing at least in the multimedia stream (step 204, 206, 208, 214, 216), from the user, and updating the summary (step 210), in relation to at least the obtained user input, so that the summary can be changed and presented to a user.

Description

UPDATING VIDEO SUMMARY

The present invention relates to affecting a summary in real time of a video stream, based on user input.

The interest for multimedia summarization such as video summarization has grown rapidly during the last couple of years as generating multimedia streams and presenting them to users, have become increasingly popular. As today, video summarizations algorithms and systems exist which are able to generate summaries of a certain user-specified duration, of for instance, a video stream. Other algorithms can generate video summaries of video streams, which video summaries are tailored according to the interests and preferences of a specific user. According to a study of Agnihotri et al. (Agnihotri, L., et al. "Study on requirement specifications for personized multimedia summarization" in Proc. of IEEE Conference on Multimedia and Expo, June 2003, Baltimore, MD, USA) an ideal summary is tailored to the specific needs of an individual or a group of users for not only a given context but also on a given task, time and environment. The patent application WO 01/27876 Al presents a hierarchical video summary description scheme, by which overview, navigation and browsing functionalities is enabled. By activating various representative frames, original video intervals can be accessed. By clicking on sound icons corresponding to a certain frame of a video summary, the sound of the corresponding video interval can be played. This patent application has the limitation of only offering video intervals or sound of the corresponding video intervals, which are pre-defined and level-based. During the consumption of a video summary the interest of the user might drift from what was initially specified in his/her user profile that was used for the generation of the video summary. The consumer might be enticed by a scene, by particular content elements or by some details that were not taken into proper consideration during the generation of the video summary. The consumer has no practical means to obtain information that is of his/her current interest when consuming a video summary, unless this information is present in the summary from the very beginning. The user must therefore make sure that his/her user profile is up-to-date. It is however very difficult to predict in advance with sufficient precision what information and with which level of detail a certain user wants to consume a multimedia summary, before consuming said multimedia summary. There is thus a need for a technique with which information that is of the user's current interest, when consuming a multimedia summary, could be made available to the user. The present invention relates to dynamically affecting a summary in real time of a multimedia stream, based on user input. An object of the present invention is to provide a method by which information, that is of interest to a user consuming a summary of a multimedia stream, can be made available to the user, in said summary. According to a first aspect of the present invention, this object is achieved by a method for enabling dynamically affecting a summary of a multimedia stream, comprising the steps, presenting the summary to a user, obtaining user input related to topics appearing at least in the multimedia stream, from the user, and updating the summary, in relation to at least the obtained user input, so that the summary can be changed and presented to a user. Another object of the invention is to provide an electronic device which enables information that is of interest to a user consuming a summary of a multimedia stream to be made available to the user, in said summary. According to a second aspect of the present invention, this object is achieved by an electronic device for dynamically affecting a summary of a multimedia stream, comprising a media stream reading unit, arranged to read the multimedia stream, a user input unit, arranged to obtain user input, a summary updating unit, arranged to update the summary, in relation to obtained user input, and a control unit, arranged control the updating of the summary, so that the summary can be changed and presented to the user. The present invention has the following advantages: It offers a consumer of a summary of a multimedia stream the possibility to change the summary that is being presented, according to input from the consumer. The present invention enables generating an up-to-date summary of a multimedia stream. Direction of the dependent claims and the advantages thereof: Claim 2 is directed toward making the updated summary available to the user. This claim has the advantage of showing the result of the user input to the user. Claim 3 is directed toward presenting the updated summary directly after having received user input. This claim has the advantage of enabling presenting the updated summary to the user in near real time after the user input is obtained. Claims 4-6 are directed to offering the user of the device a possibility to input information that is related to zooming, the duration of the summary and to the topic(s) chosen. These claims have the advantage of enabling the user to input his/her summary-related information. The general idea of this invention is to provide a consumer of a multimedia summary the possibility to change the summary in order to generate an up-to-date summary, dependent on input from the consumer.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. The present invention will now be described in more detail in relation to the enclosed drawings, in which: Fig. 1 shows a schematic representation of an electronic device for dynamically affecting a summary, according to the present invention, Fig. 2 presents a flow-chart of a method for enabling dynamically affecting a summary, according to the present invention, and Fig. 3 is a schematic view of a user input unit according to one embodiment of the present invention.

The invention relates in general to a method and a device for dynamically affecting a summary of a multimedia stream. The invention concerns the ability for a user to change the contents of a multimedia summary while he or she is consuming the multimedia summary. This change is performed by updating the multimedia summary in real time based on input information obtained from a user for whom the multimedia summary is presented. Based on feedback being received from the user, on for instance the momentary content of the multimedia summary that is being consumed, the device can update the summary in accordance with the user input. The multimedia summary is a summary of an original multimedia stream. This original multimedia stream may contain a basic multimedia stream having any type of data, such as graphical data, audio data and or text data etc, and a meta data stream containing descriptors of the content, such as topics, attributes of the basic multimedia stream, which meta data stream for this reason is associated with the basic multimedia stream. The meta data stream may be embedded in the basic multimedia stream, but may alternatively be comprised in a stream separate from the basic multimedia stream, thus being one separate stream in the original multimedia stream. According to one preferred embodiment of the present invention, the basic multimedia stream as mentioned herein is separate from the meta data stream decribing its content. The multimedia summary may also comprise a basic multimedia summary stream and a meta data summary stream that is associated with the basic multimedia summary stream. According to one preferred embodiment of the present invention, the multimedia summary comprises such a basic multimedia summary stream and such a meta data summary stream. The meta data summary stream may be embedded in the basic multimedia summary stream, forming the multimedia summary stream, but may also be a stream separate from the basic multimedia summary stream. The meta data summary stream is separate from the basic multimedia summary stream, according to one preferred embodiment of the present invention. According to an alternative embodiment of the present invention, the multimedia summary includes a meta data summary stream, but not a basic multimedia summary stream as such. This meta data summary stream is associated with the basic multimedia stream of the original stream directly, and hence identifies which information shall be comprised in a multimedia summary, from the original basic multimedia stream. According to one preferred embodiment of the present invention, the basic multimedia summary stream may contain any information from the original basic multimedia stream, that is any type of data, such as graphical data, audio data and/or text data etc. Other types of data or information from the basic multimedia stream may also be present in this basic multimedia summary stream, as the latter is a part of the former. The basic multimedia summary stream thus contains a sub-set of the entire multimedia information contained in the basic multimedia stream. This selection of parts, sub-parts, sections or the like, from the basic multimedia stream to be present in the basic multimedia summary stream is defined in the meta data summary stream. The meta data summary stream can thus contain information on which frames, which audio and text sections etc. shall be included in the basic multimedia summary stream. The meta data summary stream can also comprise information on the time stamps of the frames to be included in the basic multimedia summary stream or alternatively the time length of each media presentation and the beginning or the ending time of each such presentation of frames, of the basic multimedia summary stream, to mention a few examples only. A schematic representation of an electronic device for dynamically affecting a summary, according to one preferred embodiment of the present invention is shown in Fig. 1. With reference to said figure, fig. 1, the electronic device 100 can be seen to comprise a control unit 102, a summary agent 103, a multimedia stream reading unit 104, an associated user input unit 106, and a summary updating unit 108. Fig.l also contains a multimedia summary presenting unit 110, which is connected to the electronic device 100 and arranged to present the multimedia summary to the user of the electronic device 100. The user input unit 106 is arranged to obtain user input and is connected to the control unit 102. According to this embodiment the control unit 102 further comprises the summary agent 103. The importance of the summary agent 103 will be explained down below. In an alternative embodiment the summary agent is separate from, but still in contact with, the control unit 102. The multimedia stream reading unit 104 is arranged to read the meta data stream and the basic multimedia stream, of the original multimedia stream. The multimedia stream reading unit 104 is also connected to the control unit 102. Said control unit 102 is further connected to the summary updating unit 108 that is arranged to generate an updated multimedia summary stream, based on obtained user input data from the user input unit 106. The multimedia stream reading unit 104 is further connected to the summary updating unit 108. This summary updating unit 108 is moreover connected to a multimedia presenting device 110, which presenting unit 110 is provided for presenting the basic multimedia summary stream to the user and for making meta data from the meta data summary stream, available to the user, as will be described below. According to one preferred embodiment of the present invention the electronic device is a stationary electronic device. In an alternative embodiment, the electronic device may be a portable electronic device. The user input unit 106, which is associated with and detachable from the electronic device, is comprised in the electronic device, according to one preferred embodiment of the present invention. The user input unit does not always obtain user input as such, but may work more like a user interface, for example in the case in which the user input unit is a pointer. Here the pointer is hence a tool to communicate the user input. According to one preferred embodiment, the user input unit 106 may communicate wirelessly with the electronic device by IR-light. According to an alternative embodiment the user input unit 106 may communicate with the device using the Bluetooth wireless technique. In all cases of wireless communication between the user input unit and the electronic device, the electronic device is adapted to be able to communicate with the user unit 106, either with assistance of the presentation unit being equipped with a light sensitive unit, such as a camera, or not. According to another embodiment of the present invention the user input unit 106 is connected to the electronic device by a cord. The user input unit may be a number of different types of input units that contain one or more operatable buttons, such as joysticks, navigating units, etc. The user input may also be realised as a clickable mouse with which a cursor may be moved on a display, to mention a few examples of different types of user input units. The original multimedia stream can be provided from a memory unit, a storage device, via a network such as a LAN or from any other types of networks, such as wireless networks or by various techniques such as different casting techniques, such as broadcasting. The multimedia summary stream that is generated, may be formatted in a number of different ways, of which some are MPEG-2 or MPEG-4. Now, the user input unit 106 is provided to enable the user of the device to give input to the device according to the present invention so as to update the multimedia summary that is currently consumed by the user. With reference to fig. 3, schematically presenting a user input unit according to one embodiment of the present invention, the user input unit 106, 300 as such and the buttons it can offer, 302,304,306 will now be described. According to one preferred embodiment of the present invention the user input unit 106, 300 comprises three buttons, 302,304,306 each of which receives functions dependent on the level of operation of the user consuming a video summary stream, being one type of multimedia summary stream. By operating these buttons, input may be communicated to the electronic device about whether the user of the electronic device wants to zoom- in into or zoom-out from any specified content. Also, by using these buttons the user can influence the length of the updated basic multimedia summary stream. Moreover, the user is prompted to specify which topic he/she wants to zoom into or out from, by using the set of buttons. According to one preferred embodiment of the present invention the first button 302 of the set of three buttons of the user input unit, is a +/yes/up-button. The second button 304 of the set is a -/no/down-button and the third button 306 is an OK/enter-button. Other types or buttons may be comprised in the user input unit, in an alternative embodiment of the present invention. In the case a pointer is being used as the user input unit, the electronic device and/or the presentation unit are/is adapted to offer soft buttons, which may be selected by the user operating the pointer. A more detailed description of the way the user input unit 106, 300 can be used and the effect of it under operation will be described below, with reference to fig. 2, presenting a flow-chart of the method for enabling dynamically affecting a summary, according to the present invention and fig. 3, as described above. The method contains presenting a basic multimedia summary stream, such as a video summary, on the presentation unit 1 10 to the user, step 202. If the user of the device, that is the consumer of the video summary presentation, wishes to update the contents of the summary, the user activates the user input unit 106, 300. By activating one of the set of buttons, by for instance pressing the first button 302, at this level of operation, the control unit 102 interprets this as an order from the user to zoom- in into some information that may be available while presenting the video summary. The control unit 102 notices at which position or time of the presented video summary, the user input is received. If, on the other hand the user is satisfied with the video summary presentation, the device does not obtain any user input in step 204. If however the control unit 102 of the electronic device 100 detects user input in the form of an activated zoom-in button 302, step 204, the control unit 102 presents a prompt on the presenting unit 110 to the user, whether the user prefers extension of the updated video summary when zooming in into a topic or not, step 206. If the user presses the first button 302, the +/yes/up-button the control unit 102 of the electronic device 100 interprets this input as a yes. If the user presses the second button 304, that is the -/no/down- button as an answer to the question in step 206, the control unit interprets this as if the user does not want an extension of the summary stream. After having received user input on the desire to zoom in and whether or not an extension of the video summary is desired, the summary agent 103 prompts the user via the presenting unit 110 which topics or attributes he/she wants to zoom- in into, in this particular case, since the user has already pressed button 302, that is zoom-in. The presenting unit 110 shows a list of topics and/or attributes which were relevant at the time or at the position of the video summary, when the input was received from the user. As the meta data summary stream defines the basic multimedia summary stream that is presented, information on the topics and the attributes are obtained from the meta data summary stream, and presented to the user via the presenting unit 110. The list of topics and attributes that is presented to the user may also contain topics and attributes that are not comprised in the meta data summary stream, but only in the meta data stream of the orginal multimedia stream. The multimedia summary stream that is presented to the user did thus not contain information on these topics and attributes, at the time the user input was received. Rather, such information is only present in the original multimedia stream. Example of topics and attributes may be specific or group of actors, environments, activities, locations, events, to mention a few examples only. The summary agent 103 of the control unit 102 has an important function. According to one embodiment of the present invention the summary agent 103 has access to the meta data of the original video stream via the multimedia stream reading unit 104 as well as the current meta data of the video summary stream, that is the meta data summary stream, from the summary updating unit 108. The current contents of the video summary is listed in the meta data summary stream, whereas any additional information or material is listed in the original meta data stream only. In order to offer a zoom- in functionality to the user of the electronic device, the device has access to the original multimedia stream. The functionality to increase the amount of information on a certain topic, as selected in step 208, with or without extending the duration, as determined in step 206, is handled by the summary agent 103. If the user chooses to zoom in into a topic, information according to that selection is obtained from the basic multimedia stream, which information is to be included in the basic multimedia summary stream, of which the video summary stream is one example. By selecting any topic or attribute in the list showing the meta data the user can include more information on this topic in the video summary, under the control of the control unit 102 and the summary agent 103. By pressing the first button 302, the list is scrolled up and by pressing the second button 304 the list is scrolled down. By pressing the OK/enter-button 306 the currently marked topic is selected, step 208. Directly after the summary agent has obtained a topic selection from the user, the step of updating the summary follows, step 210. As soon as a presentable part of the updated video summary, as an example of a basic multimedia summary stream, is generated, the updated video summary this part is presented to the user, step 212. The summary agent 103 preferably comprises user profiles of a number of users. The basic multimedia summary stream is dependent on the user profile of each specific user. If a certain user has earlier specified that he/she only had interest in a certain type of actors, the summary updating unit 108 will generate a basic multimedia summary stream respecting this specific information, under the control of the summary agent 103/control unit 102 in which the user profile is comprised. As the updated video summary is presented to the user, the user profile is updated in accordance with the latest user input, according to one preferred embodiment of the present invention. Within step 214, if the user selects zoom out at the highest level of operation of this electronic device, by pressing the -/no/down-button 304, the control unit 102 orders the presentation unit 110 to prompt the user whether he/she wants the duration of the summary to be reduced or not. If the user presses the first button 302, the control unit interprets this as a yes and if the user presses the second button, the user input is interpreted as a no, in step 216. As described above, after step 206, a list of selectable topics and/or attributes is now presented to the user via the presentation unit 110. From this list topics and/or attributes already present in the summary, that is topics/attributes included in the meta data summary stream, the user can select a topic to zoom out of. If the user has become less interested in a certain actor, for instance Tom Cruise, less importance is then put on this topic by the summary agent, upon updating the summary, as performed by the multimedia updating unit, 108. Subsequent to the updating of the presented multimedia summary, the updated video summary is automatically presented to the user, step 212. It should be emphasized that this updating of the summary is performed in real time "on-the-fly" so as to immediately respond to the input signals sent by the user. This enables the updated summary to be consumed by the user, as soon as a small fraction of the entire summary is updated. The invention can be varied in more ways, for instance: The functions of the summary agent 103 may partly be performed by control unit 102, and the ones of the control unit 102 may be performed by the summary agent 103. The summary updating unit 108 may be comprised within a summary agent 103, that in this case preferably is separate from the control unit 102. The user input unit 106 may be a user interface, receiving user input in a number of different ways. According to another embodiment of the present invention the meta data stream has two types of data, of which one is active and one is passive. The meaning of this is that an active type of data, or flag, indicates that multimedia data associated with this meta data shall be included in a summary stream. On the other hand, a passive type of meta data, indicates that the multimedia data associated with that passive meta data shall not be included in the summary. In this way one meta data stream may be excluded. Alternatively, the summary stream may not need to include any basic multimedia summary stream, since the meta data summary stream may obtain the associated multimedia data directly from the original multimedia stream.

Claims

CLAIMS:

1. A method for enabling dynamically affecting a summary of a multimedia stream, comprising the steps: presenting the summary to a user (step 202), obtaining user input related to topics appearing at least in the multimedia stream, from the user (step 204,206,208,214,216) and updating the summary, in relation to at least the obtained user input (step 210) so that the summary can be changed and presented to a user.

2. The method according to claim 1, further comprising the step: - presenting the updated summary to the user (step 212).

3. The method according to claim 2, in which the step of obtaining (step 204,206,208,214,216) user input is performed during the step of presenting the summary to the user (step 202), so that the step of presenting the updated summary to the user (step 212) can be performed directly.

4. The method according to claim 1 , in which the step of obtaining user input comprises obtaining zoom-related information from the user (step 204,214).

5. The method according to claim 1, in which the step of obtaining user input comprises obtaining duration-related summary information, from the user (step 206,216).

6. The method according to claim 1, in which the step of obtaining user input comprises obtaining topic-related summary, from the user (step 208).

7. The method as claimed in claim 1, wherein the summary comprises segements comprising data on the topics, the user input indicates an interest in a specific topic and the step of updating of the summary includes the substep of including additional segments of the mulitmedia stream in the summary, the additional segments comprising data on the specific topic.

8. The method as claimed in claim 7, further comprising, when a segment is added to the summary, the step of removing at least one segment on a topic other than the specific topic from the summary.

9. The method as claimed in claim 8, wherein the size of the summary is preserved.

10. The method as claimed in claim 9, wherein the size of the summary is expressed in one of the following entities: data size; time; and - required storage space.

11. An electronic device (100) for dynamically affecting a summary of a multimedia stream, comprising: a media stream reading unit (104), arranged to read the multimedia stream, - a user input unit (106), arranged to obtain user input, a summary updating unit (108), arranged to update the summary, in relation to obtained user input, and a control unit (102), arranged control the updating of the summary, so that the summary can be changed and presented to the user.

12. The electronic device (100) according to claim 11 in which the multimedia stream reading unit (104) is arranged to read a meta data stream that is associated to the multimedia stream.

13. The electronic device (100) according to claim 11, in which the summary updating unit (108) is arranged to update a multimedia summary stream.

14. Computer programme product comprising computer executable instructions enabling a computer to carry our the method as claimed in claim 1.

15. Medium comprising the computer programme product as claimed in claim 14.

16. Programmed computer configured to carry out the method as claimed in claim 1.