WO2008103364A1

WO2008103364A1 - Systems and methods for sending, receiving and processing multimedia bookmarks

Info

Publication number: WO2008103364A1
Application number: PCT/US2008/002200
Authority: WO
Inventors: Sanghoon Sull; William J. Raduchel; Myounghoon Kim
Original assignee: Vmark, Inc.
Priority date: 2007-02-20
Filing date: 2008-02-20
Publication date: 2008-08-28

Abstract

By using a cellular phone or other mobile device to send a text message or e-mail, or by using a computer to send an e-mail, HTTP Get and Post methods or instant message, the invention allows for the marking of an exact scene in any AV stream being watched. The invention further allows the user to fine tune the bookmark of the exact scene to save for later personal use, share with friends or on a social network, or to share with the general public by posting on a bulletin board. Additionally, by using a type of 'bookmark clearinghouse', a bookmark relevant to a first AV stream can be mapped or transformed into a bookmark relevant to a second AV stream having content that is the same or comparable to the content of the first AV stream. Thus, accessing multimedia content at the exact same point across different devices can be accomplished using the invention.

Description

SYSTEMS AND METHODS FOR SENDING, RECEIVING, AND PROCESSING

MULTIMEDIA BOOKMARKS

[0001] The present application claims priority from copending U.S. Patent Application No.

60/902,228 filed February 20, 2007 and entitled Systems and Methods for Sending, Receiving, and Processing Multimedia Bookmark, which is hereby incorporated by reference for all purposes.

[0002] Cross Reference to Related Applications

[0003] Commonly-owned, copending U.S. Patent Application No. 09/911,293 filed

7/23/2001 (Publication No. 2002/0069218), the entirety of which is hereby incorporated by reference for all purposes, discloses methods and systems that include a multimedia bookmark for an AV stream as well as sending the multimedia bookmark over a wireless network.

[0004] Background

[0005] The following disclosure is directed generally to the field of multimedia content distribution, and more particularly to the saving, sending, and processing of bookmarks identifying particular positions in the multimedia content.

[0006] With the phenomenal growth of digital media, the Internet and digital broadcasting, the amount of multimedia content that can be accessed by the public has virtually exploded.

Finding such content has become more difficult as the amount of content has increased. There are occasions where a user who once accessed particular multimedia content needs or desires to access the content again at a later time, possibly at or from a different place. For example, a user who watches multimedia content on DTV might want to send a multimedia bookmark from DTV to his/her mobile device through a network to continue to view the content later from that exact same spot in the content, whether on the same original device or on an alternative device such as on a personal computer. Similarly, a user who once viewed multimedia content at work may want to continue to view the content at home.

[0007] Additionally, viewers may wish to share content with their friends, a social network or the general public. The current solutions with respect to broadcast TV are inadequate. The simplest method is to call, text, or e-mail ones friends that something interesting is happening.

Often, because of the delay, it is too late to have the same viewing experience as the original viewer. A second method is to locate such content on the Internet and send a link to such content.

The two primary problems are finding the content on the Internet and secondly the user has to then find the particular scene within the content being referenced.

[0008] A shortcoming of conventional bookmarks is that conventional bookmarks mark a document or video from the beginning. Conventional bookmarks mark a document such as a static web page for later retrieval by saving a link (address) to the document. For example, Internet browsers support a bookmark facility by saving an address called a Uniform Resource Identifier (URI) to a particular file. Internet Explorer, manufactured by the Microsoft Corporation of Redmond, Washington, uses the term "favorite" to describe a similar concept. [0009] Conventional bookmarks, however, store only the information related to the location of a file, such as the directory name with a file name or a Universal Resource Identifier (URI). The files referred to by conventional bookmarks are treated in the same way regardless of the data formats for storing the content. Typically, a simple link is used for multimedia content. For example, to link to a multimedia content file through the Internet, a URI is used. Each time the file is revisited using the bookmark, the multimedia content associated with the bookmark is always played from the beginning.

[0010] Access to the same position of a multimedia content indicated by a bookmark is not trivial. In case of the streams available from video on demand (VOD) service providers, users can easily access the same position of a multimedia content later by using both URI and media time because users are watching the same AV stream which resides in a server managed by VOD service providers. However, multimedia content is accessible to users in the form of different AV streams such as in a variety of bit rates, encoding/compression methods such as MPEG-2, H.264 and proprietary encoding solution from Microsoft, Inc., delivery mediums, service providers, and others. Accessing multimedia content at the exact same point across different devices can present numerous problems.

[0011] It would be useful if a technique were available for sending and receiving bookmarks over a network between a variety of devices in a manner that would enable processing of the bookmark in such a manner that the same position in the multimedia content could be accessed regardless of the source of the content stream, bit rate, encoding scheme, etc. [0012] Summary

[0013] The current invention is a simple and elegant solution to the problem of bookmarking particular locations in a multimedia stream. By using a cellular phone or other mobile device to send a text message or e-mail, or by using a computer to send an e-mail, HTTP Get and Post methods or instant message, the invention allows for the marking of an exact scene in any AV stream being watched. The invention further allows the user to fine tune the bookmark of the exact scene to save for later personal use, share with friends or on a social network, or to share with the general public by posting on a bulletin board. Additionally, by using a type of "bookmark clearinghouse", a bookmark relevant to a first AV stream can be mapped or transformed into a bookmark relevant to a second AV stream having content that is the same or comparable to the content of the first AV stream. Thus, accessing multimedia content at the exact same point across different devices can be accomplished using the invention.

[0014] Many companies focus on saving and sharing of content already on the Internet using the personal computer. This invention allows the use of a TV remote control, cellular, or other mobile device, personal computer, IM, e-mail or HTTP Get and Post methods to save and share content being broadcast on the television, without waiting for it to be posted on the Internet first.

Those, and other advantages and benefits will be apparent from the following description.

[0015] Brief Description of the Drawings

[0016] For the present disclosure to be easily understood and readily practiced, the present disclosure will now be described, for purposes of illustration and not limitation, in conjunction with the following figures wherein:

[0017] FIG. IA illustrates a system for sending, receiving and processing a multimedia bookmark between different types of client devices, according to an embodiment of the present disclosure.

[0018] FIG. IB illustrates an exemplary system for sending, receiving and processing a multimedia bookmark between two different types of client devices, e.g., DTV and PC, according to an embodiment of the present disclosure.

[0019] FIG. 1C is an exemplary block diagram for the media localization server for managing the information on the contents, according to an embodiment of the present disclosure.

[0020] FIG. 2 illustrates an exemplary timestamp relationship between a variety of AV streams which correspond to the same content, according to an embodiment of the present disclosure.

[0021] FIG. 3A is an exemplary flowchart illustrating a scheme for generating a multimedia bookmark according to an embodiment of the present disclosure.

[0022] FIG. 3B is an exemplary flowchart illustrating a scheme for creating and sending a bookmark by using a cellular phone text message according to the present disclosure;

[0023] FIG. 4 illustrates an exemplary scheme of generating the AV content information for the multimedia bookmark generated by a user, according to an embodiment of the present disclosure.

[0024] FIG. 5A is an exemplary flowchart illustrating a scheme for processing a received multimedia bookmark, according to an embodiment of the present disclosure.

[0025] FIG. 5B is an exemplary flowchart illustrating a method for processing a multimedia bookmark at the client devices, according to an embodiment of the present disclosure. [0026] FIG. 5C is an exemplary flowchart illustrating a method for processing a multimedia bookmark at the bookmark server, according to an embodiment of the present disclosure. [0027] FIG. 6 is an exemplary flowchart illustrating a process performed in the media localization server, according to an embodiment of the present disclosure. [0028] FIG. 7 illustrates an exemplary AV pattern matching process for locating a frame accurate bookmarked position, according to an embodiment of the present disclosure. [0029] Description

[0030] The present disclosure is directed to a process of saving, sending, receiving, and processing audiovisual bookmarks between a variety of consumer electronic devices to access the same position of a multimedia content regardless of the source, format, and bit rate, among others. The disclosed method is especially useful in enabling the viewing of content on different devices, such as starting to watch an Internet video on your work computer and then to complete your viewing of the same video at the exact spot you left off on your home television. Furthermore, the disclosed method is also especially useful in allowing for the ease of sharing of live TV content within a social network. Instead of calling friends to tell them about some exact scene viewed, a viewer can use a cellular phone or computer or other such devices to reference an exact scene and share such exact scene among friends and a social network without having to take an additional step to find such content on the Internet.

[0031] Unless otherwise noted, or as may be evident from the context of their usage, any terms, abbreviations, acronyms or scientific symbols and notations used herein are to be given their ordinary meaning in the technical discipline to which the disclosure most nearly pertains. The following terms, abbreviations and acronyms may be used in the description contained herein.

[0032] ATSC Advanced Television Systems Committee, Inc. (ATSC) is an international, non-profit organization developing voluntary standards for digital television. Countries such as U.S. and Korea adopted ATSC for digital broadcasting. A more extensive explanation of ATSC may be found at "ATSC Standard A/53C with Amendment No. 1 : ATSC Digital Television Standard, Rev. C," (see World Wide Web at atsc.org). More description may be found in "Data Broadcasting: Understanding the ATSC Data Broadcast Standard" (McGraw- Hill Professional, April 2001) by Richard S. Chernock, Regis J. Crinon, Michael A. Dolan, Jr., John R. Mick, Richard Chernock, Regis Crinonm, and may also be available in "Digital Television, DVB-T COFDM and ATSC 8-VSB" (Digitaltvbooks.com, October 2000) by Mark Massel. Alternatively, Digital Video Broadcasting (DVB) is an industry-led consortium committed to designing global standards that were adopted in European and other countries, for the global delivery of digital television and data services. [0033] AV Audiovisual.

[0034] AVC Advanced Video Coding (H.264) is the newest video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. An explanation of AVC may be found at "Overview of the H.264/ AVC video coding standard", Wiegand, T., Sullivan, G.J., Bjntegaard, G., Luthra, A., Circuits and Systems for Video Technology, IEEE Transactions on , Volume: 13 , Issue: 7 , July 2003, Pages:560 - 576; another may be found at "ISO/IEC 14496-10: Information technology - Coding of audio-visual objects - Part 10: Advanced Video Coding" (see World Wide Web at iso.org); Yet another description is found in "H.264 and MPEG-4 Video Compression" (Wiley) by Iain E.G. Richardson, all three of which are incorporated herein by reference. MPEG-I and MPEG-2 are alternatives or adjunct to AVC and are considered or adopted for digital video compression. [0035] Communication Network Any local or wide area network (wired, wireless) including Internet, Wi-Fi and Bluetooth.

[0036] DMB Digital Multimedia Broadcasting (DMB), first commercialized in

Korea, is a new multimedia broadcasting service providing CD-quality audio, video, TV programs as well as a variety of information (for example, news, traffic news) for portable (mobile) receivers (small TV, PDA and mobile phones) that can move at high speeds. [0037] DTV Digital Television (DTV) is an alternative audio-visual display device augmenting or replacing current analog television (TV) characterized by receipt of digital, rather than analog, signals representing audio, video and/or related information. Video display devices include Cathode Ray Tube (CRT), Liquid Crystal Display (LCD), Plasma and various projection systems. Digital Television is more fully described at "Digital Television: MPEG-I, MPEG-2 and Principles of the DVB System" (Butterworth-Heinemann, June, 1997) by Herve Benoit.

[0038] DVB Digital Video Broadcasting is a specification for digital television broadcasting mainly adopted in various countered in Europe adopt. A more extensive explanation of DVB may be found at "DVB: The Family of International Standards for Digital Video Broadcasting" by Ulrich Reimers (see World Wide Web at dvb.org). ATSC is an alternative or adjunct to DVB and is considered or adopted for digital broadcasting used in many countries such as the U.S. and Korea.

[0039] DVR Digital Video Recorder (DVR) is usually considered a Set Top Box

(STB) having recording capability, for example in associated storage or in its local storage or hard disk. A more extensive explanation of DVR may be found at "Digital Video Recorders:

The Revolution Remains On Pause" (MarketResearch.com, April 2001) by Yankee Group.

[0040] Media time Media time is defined from the start of a video stream or file (0) to the end (the length of video).

[0041] MPEG-2 Moving Picture Experts Group - Standard 2 (MPEG-2) is a digital video compression standard designed for coding interlaced/noninterlaced frames. MPEG-2 is currently used for DTV broadcast and DVD. A more extensive explanation of MPEG-2 may be found on the World Wide Web at mpeg.org and "Digital Video: An Introduction to MPEG-2 (Digital

Multimedia Standards Series)" (Springer, 1996) by Barry G. Haskell, Atul Puri, Arun N.

Netravali.

[0042] MPEG-4 Moving Picture Experts Group - Standard 4 (MPEG-4) is a video compression standard supporting interactivity by allowing authors to create and define the media objects in a multimedia presentation, how these can be synchronized and related to each other in transmission, and how users are to be able to interact with the media objects. A more extensive information of MPEG-4 can be found at "H.264 and MPEG-4 Video Compression" (John Wiley

& Sons, August, 2003) by Iain E. G. Richardson and "The MPEG-4 Book" (Prentice Hall PTR,

July, 2002) by Touradj Ebrahimi, Fernando Pereira.

[0043] PCR Program Clock Reference (PCR) in the Transport Stream (TS) indicates the sampled value of the system time clock that can be used for the correct presentation and decoding time of audio and video. A more extensive explanation of PCR may be found at

"Generic Coding of Moving Pictures and Associated Audio Information- Part 1 : Systems,"

ISO/IEC 13818-1 (MPEG-2), 1994 (http://iso.org). SCR (System Clock Reference) is an alternative or adjunct to PCR used in MPEG program streams.

[0044] STB Set-top Box (STB) is a display, memory, or interface device intended to receive, store, process, repeat, edit, modify, display, reproduce or perform any portion of a program, including personal computer (PC) and mobile device.

[0045] STT System Time Table (STT) is a small table that provides the current date and time of day information in ATSC. Digital Video Broadcasting (DVB) has a similar table called a Time and Date Table (TDT). A more extensive explanation of STT may be found at

"ATSC Standard A/65B: Program and System Information Protocol for Terrestrial Broadcast and

Cable", Rev. B, 18 March 2003 (see World Wide Web at atsc.org).

[0046] [0046] TV Television, generally a picture and audio presentation or output device; common type include cathode ray tube (CRT), plasma, liquid crystal and other projection and direct view systems, usually with associated speakers. [0047] [0047] XML extensible Markup Language (XML) defined by W3 C (World

Wide Web Consortium), is a simple, flexible text format derived from SGML. A more extensive explanation of XML may be found at "XML in a Nutshell" (O'Reilly, 2004) by Elliotte Rusty

Harold, W.Scott Means.

[0048] [0048] A multimedia bookmark for an AV stream delivered through a broadcasting network or the Internet can be generated by any of a wide variety of devices such as mobile devices, e.g., cell phones, PDAs, laptops, etc, or non-mobile devices such as PCs, phones on landlines, or the like, and can contain one or more of the following bookmark information:

[0049] [0049] 1. URI of a bookmarked stream;

[0050] [0050] 2. Bookmarked position;

[0051] [0051] 3. AV content information such as reduced-sized frame(s) or signal pattern captured at a bookmarked position;

[0052] [0052] 4. Textual annotations attached to a segment that contains the bookmarked position;

[0053] [0053] 5. Title of the bookmark;

[0054] [0054] 6. Metadata identification (ID) of the bookmarked stream;

[0055] [0055] 7. Descriptive information of the stream or program;

[0056] [0056] 8. Bookmark date/time

[0057] [0057] 9. Generator of bookmark (unique identifier and/or attributes of the generator (e.g., time zone where the generator lives, etc.))

[0058] [0058] 10. Area of interests of the bookmarked frame.

[0059] [0059] 11. Client environment

[0060] [0060] The bookmark information includes not only positional information (1 and

2) and content information (3, 4, 5, and 6), but also some other useful information, such as descriptive information of an AV stream, date/time, etc., as disclosed in the copending U.S.

Patent Application No. 09/91 1,213 filed 07/23/2001 (Publication No.2002/0069218) the entirety of which is hereby incorporated by reference, and U.S. Patent Application No. 11/071,894 filed

03/03/2005 (Publication No.2005/0210145) the entirety of which is hereby incorporated by reference. In case of broadcast streams, the positional information can be represented by using one of media locators or timestamps such as broadcasting time (for example, system_time in STT for ATSC) as disclosed in the copending U.S. Patent Application No. 10/369,333 filed 2/19/2003

(Publication No. 2003/0177503) the entirety of which is hereby incorporated by reference, U.S.

Patent Application No. 1 1/069,750 filed 3/01/2005 (Publication No. 2005/0193425) the entirety of which is hereby incorporated by reference, and U.S. Patent Application No. 1 1/221,397 filed 09/07/2005 (Publication No.2006/0064716) the entirety of which is hereby incorporated by reference.

[0061] [0061] The AV content information (3) may be composed of AV low-level features and textual features. The AV features are the information, for example, obtained by capturing or sampling the AV signal at or around the bookmarked position. In case of a video bookmark, the AV features can be thumbnail image(s) of the captured video frame(s) which could be encoded in JPEG, for example, and low-level visual feature vectors like color histogram for one or more of the frames. In the case of an audio bookmark, the AV features can also be the sampled audio signal (typically of short duration) and its visualized image. [0062] [0062] The textual features (4) are text information specified by the user, as well as delivered with the AV stream. Other aspects of the textual features may be obtained from metadata associated with of the AV stream. Also, the descriptive information of the program (7) can be obtained from broadcast EPG, if the multimedia stream is digitally broadcast, and comprises channel number, TV program title, start date/time/duration of the TV program, episode number if the program belongs to a series, synopsis, etc. The date and time (8) when the bookmark is generated and the information on the generator of a bookmark (9) such as name can be also included. Also, in case of video bookmark, a user can draw an area of interests on a bookmarked frame where such information could be also contained in multimedia bookmark information.

[0063] [0063] The client environment (11) contains information comprising what delivery medium was used for the bookmarked stream, service provider, and device type. The delivery medium could be one of terrestrial digital broadcasting such as ATSC or DVB, digital cable, digital satellite, mobile such as DMB, Internet, DVD (or Blue-Ray), and others. The service provider contains the information on the service provider for a client device such as name(s) of the local digital terrestrial/mobile broadcasting network, IPTV service provider, local digital cable network, and satellite service provider. The two types of information on delivery medium and service provider can be unified to one field which uniquely identify the service available to client, for example, "ATSC: ABC-LA", or other equivalents. The device type contains the information on the client device such as resolution and a type of AV decoder (for example, MPEG-2 or H.264) that are appropriate for the client device.

[0064] [0064] Additionally, a multimedia bookmark can be sent to a designated address in the form of a multimedia bookmark message containing the multimedia bookmark information along with user information such as sender and receiver. For example, if a multimedia bookmark is delivered via e-mail (or other equivalents such as messaging), email addresses of sender and receiver will be needed. In case a multimedia bookmark is delivered via Bluetooth, a combination of the Personal Identification Number (PIN) and a Bluetooth address can be used to identify other Bluetooth devices.

[0065] [0065] Suppose that a user of DTV with DVR generates a multimedia bookmark on the broadcast TV video program stored in their local or associated storages or live TV, sends the multimedia bookmark to a user of PC through communication network . In this case, the recipient of the multimedia bookmark cannot simply play the video from the bookmarked position using a broadcasting time unless the same video stream is available in the PC's local storage.

[0066] [0066] It is disclosed herein methods and systems of sending, receiving, and processing multimedia bookmarks to access the exact position of the multimedia content indicated by a multimedia bookmark between different types of client devices. In that manner, bookmarks may be coordinated amongst different platforms.

[0067] [0067] FIG. IA illustrates a system for sending, receiving and processing multimedia bookmarks between different types of client devices (DTV with DVR, Mobile, and PC with DTV tuner). The broadcaster 1 10 broadcasts video programs to client devices of TV viewers (client 120, 180 and 130) through the broadcasting network 160 such as the Internet, cable, satellite, mobile and terrestrial networks. While watching a recorded or live broadcast program, a user of each client device can make a multimedia bookmark on the program where the corresponding bookmark information is generated at the bookmarked position of the program. The generated multimedia bookmark might be stored in its local storage. The user can send a multimedia bookmark to a user of another client with a designated address through the communication network 170 such as the Internet. When the same stream pointed to by the received multimedia bookmark is already recorded in the received client, for example, from DTV to DTV, a user of the client who received the bookmark can access the stream from the bookmarked position by using the bookmark information. When the program pointed by the received multimedia bookmark was not recorded in the local storage of the client who received the bookmark, the client who received the bookmark sends a request along with the received bookmark information and its information on client environment to the media localization server 150 through the communication network 170. Then, the media localization server 150 returns the information on how or where the same content as the bookmarked stream can be accessed in the form of a URl, for example, and the modified bookmarked position corresponding to the same content back to the client who requested such information. The client who received access information from the media localization server 150 can access the content from one of the media host 140 through the communication network 170. The operation of the media localization server 150 is described in detail in FIG. 6. Briefly, the media localization server 150 may be thought of as a clearinghouse that maps or translates bookmarks from a first AV stream available to the first user to a second AV stream available to a second user. [0068] Alternatively, a user can also create a bookmark by using cellular phone text messaging (e.g., short message service (SMS)), personal computer instant messaging, e-mail, among others. While a user is watching a recorded or live TV program, the user can have a bookmark created and sent whenever needed by just sending a text message, e-mail, or the like, to him/herself or the user's friends through the bookmark server 190. The sender of the message is known to the bookmark server 190 via a pre-registered profile. The receiver of the bookmark generated by bookmark server 190 can be the sender him/herself by default or specified by the sender. The descriptive information and the client environment of the bookmark is automatically obtained or filled in at the bookmark server 190 by using a TV program or broadcaster-specific recipient mobile number (or short code, a 5 number text message recipient number) to identify what is being watched. Otherwise, the text itself such as a short word to identify the TV program (for example, "vmarkABC-LA") for a user to bookmark the ABC-LA channel can be used as the descriptive information of the bookmark. The positional information or broadcasting time for the bookmark can be automatically obtained from the short message service (SMS) received time at the bookmark server 190 wherein the positional information can be adjusted by considering the typical delay (1-2 second). The recipient of the bookmark then accesses the bookmark server 190 and selects the received bookmark. If the recipient of the received bookmark does not have access to the AV steam identified in the bookmark, the recipient can connect to the media localization server 150 to obtain a content URI from which the bookmark program is available (for example, by streaming) and an appropriate timestamp which is mapped from the SMS received time accounting for commercial breaks to determine the actual commercial-free playout time (i.e., what is the position when playing out from the file, which is pointed by the content URI, that has no commercials). Then, the recipient of the bookmark can play from the file pointed to by the content URI without the commercials.

[0069] It is also noted that a PC user can login to the media localization server 150 to play an AV file and then generate and send a multimedia bookmark to a user of other client devices. [0070] FIG. IB illustrates an exemplary system for sending, receiving and processing a multimedia bookmark between two different types of client devices DTV with DVR 120, and PC 130 with an optional DTV tuner 131. The broadcaster 1 10 broadcasts HDTV TV video programs (for example, in 1080i format) to the clients 120 and 130 through the broadcasting network 160 such as the Internet, cable, satellite, and terrestrial networks. The broadcast TV programs might be recorded in local storages 122 and 132 of the clients, and played with media players 124 and 134 on the corresponding display devices 125 and 135 whenever a user wants. [0071] While watching a recorded or live TV program, a user of DTV 120 can make a multimedia bookmark on the program where the corresponding bookmark information is generated and stored in its local storage 122 by the multimedia bookmark client 126. Also, the user can send the multimedia bookmark to the client PC 130 via the bookmark server 190, which the user selects among other clients, by using the bookmark client 126 through the communication network 170 such as Bluetooth or the Internet. When a user sends a multimedia bookmark to the user of PC 130, the multimedia bookmark is saved in the local storage 194 at the bookmark server 190. Then, a user of PC 130 can receive the a multimedia bookmark from bookmark manager 193 of the bookmark server 190.

[0072] If the program pointed by the multimedia bookmark sent by the client 120 has already been recorded in the local storage 132 of the client 130, the program can be played from the bookmarked position by using the information such as a URI and bookmarked position included in the multimedia bookmark if the clients 120 and 130 use the same file naming scheme and predefined directory for storing recorded streams. For example, all recorded programs that are broadcast on February 2005 are stored in a directory whose name is "200502," and a recorded program that was broadcast at 9:30 PM on 16^th February 2005 at channel 205 has a file name such as "20050216-2130-205.ts." The file name is used in the field "URI of a bookmarked file" of the bookmark information in the present disclosure, for example.

[0073] Alternatively, it is disclosed herein a method and system that do not require the same file naming scheme. The search agent (128, 138) at each client (120, 130) can resolve the locations of the stored programs by associating the descriptive information and client environment in the received multimedia bookmark with the corresponding information on the recorded streams in the local storage 132. When the program pointed to by the multimedia bookmark sent by the client 120 was not recorded in the local storage 132 of the client 130, the program can be recorded later when it is rebroadcast on the same channel or available from other channels by searching broadcast EPG for the same program based on the descriptive information (for example, program title, episode number, etc) included in the received multimedia bookmark, wherein the searching broadcast EPG for the same program is also performed by the search agent 138, which is requested by the multimedia bookmark client 136. It is noted that the positional information (for example, system_time in STT for ATSC) included in the received multimedia bookmark cannot be simply used to access the same position of the multimedia content because the broadcasting time of the program pointed by the received multimedia bookmark is different from that of the program to be recorded later. Thus, it is necessary for the search client 138 to connect to the media localization server 150 (service discovery) and send a request for a modified timestamp along with the received bookmark information and additional information on the client environment for the client 130 to the media localization server 150 through the communication network 170. The media localization server 150 maps or modifies the timestamp in the received bookmark information to the correct timestamp for the stream available at the client 130 and sends it back to the client 130 through the communication network 170. [0074] When the TV program pointed to by the multimedia bookmark sent by the client 120 was not recorded in the local storage 132 of the client 130, an AV stream corresponding to the same multimedia content as the TV program can be downloaded or streamed to the PC client 130 as follows: First, the search agent 138 connects to the media localization agent of 157 the media localization server 150 (service discovery) and sends a request for a content URI from which the AV stream corresponding to the same multimedia content as the TV program can be available and the modified timestamp for the stream to the media localization server 150 through the communication network 170 wherein the search agent 138 sends the bookmark information received from the user of the client 120 to the media localization server 150 (for example, the time-stamp for bookmarked position, program title, channel number, start date/time of the program, delivery medium such as ATSC and service provider such as ABC-LA) and additional information on the client environment for the client 130 (for example, digital cable and Comcast) in the form of a bookmark message. Then, the media localization server 150 analyzes the received information and sends the content URI and a timestamp or media time corresponding to the bookmarked position represented in broadcasting time (or system time in STT in case of ATSC) back to the search agent 138 in the form of an updated bookmark message. The search agent 138 connects to the media host 140 as directed by the content URI from the media localization server 150 and sends the request for the stream pointed by the content URI to the media host 140 through the communication network 170. The search engine 148 searches its database for an appropriate AV stream or file of the same multimedia content as the TV program that is downloaded or streamed from the storage 142 by the download server 144 or streaming server 146 of media host 140 to the client 130 through the communication network 170, respectively.

[0075] FIG. 1C is an exemplary block diagram for the media localization server 150 for managing the information on the contents available to users through a variety of the client environments wherein the information on the contents comprises descriptive information of the contents (for example, channel number, program title, start date/time and duration of the program, delivery medium, service provider, device type, etc.) and the location where the contents are available (for example, in the form of URI). The media localization server 150 includes modules for receiving and decoding broadcast streams (for example, tuners 151, and AV decoders 152). The EPG parser 154 extracts the EPG data such as channel number, program title, start date/time, duration, rating (if available) and synopsis from each channel and stores the information into the content & timestamp database 158. The timestamp processor 156 associates timestamps adequate to each program (for example, system_time for ATSC) with frames of the program and stores the information to the content & timestamp database 158. The media localization agent 157 receives a request from a client to the media localization server 150. The content locator 155 locates the stream or program from the content & timestamp database 158 by using the bookmark information such as descriptive information of the content and the information on the client environment of the client 120 as well as the information on the client environment of the client 130 who sent request to the media localization server 150. The content locator 155 outputs the URI of the located program from which the program can be streamed or downloaded. The time stamp processor 156 also maps the timestamp in the received bookmark to the timestamp of the program located by the content locator 155 by using the information in the contents & timestamp database 158. The AV analyzer 153 performs an AV signal pattern matching using one of the existing image processing techniques. When frame accurate access to the bookmarked position is needed, the AV analyzer 153 searches the AV stream which is first located by the content locator 155 for the exact frame that is matching to the content information such as reduced-image of the multimedia bookmark, and then outputs a frame-accurate timestamp corresponding to the bookmark information. The requested information such as URI and the mapped (or frame- accurate if needed) timestamp information is then sent back to the client who requested by the media localization agent 157.

[0076] FIG. 2 illustrates an exemplary timestamp relationships between a variety of AV streams, which correspond to the same content (for example, a football game), that could be in different encoding formats or available through different delivery mediums or service providers. The AV stream 201, 202, 203 and 204 correspond to the same AV content, but are delivered through mobile network by ABCm, terrestrial network by ABC-LA, terrestrial network by ABC- New York, and the Internet by ABCi, respectively. The timestamp for the AV stream 201 is represented by the local time delivered through the mobile network. The timestamp for the AV streams 202 and 203 is represented by the local broadcasting time delivered through the terrestrial network (for example, system_time for ATSC). The timestamp for the AV stream file 204 is represented by media time. The timestamps 205, 207, 209, and 211 correspond to the start position of the same content, and the timestamps 206, 204, 210, and 212 correspond to the end position of the same content. There is an offset 220 between the timestamps 205 and 207 due to the delay caused by the time it takes to transmit the content from a broadcaster to the mobile network although the same content is broadcast in the same local area. The streams 202 and 203 include two commercial breaks 214/215 and 216/217, respectively. The commercial breaks are not included in the AV stream file 204 which could be available later than the local broadcasting time 205 by the amount of delay 222, and thus the intervals (214/216 and 215/217) correspond to the media times 218 and 219, respectively. The frame-accurate relationship between the timestamps can be obtained by using a technique similar to the AV signal pattern matching technique described later. This type of timestamp relationship (start and end times of the programs, start and end time of commercial breaks, and offset) between a variety of AV streams is stored in the content & timestamp database 158 in the media localization server 150 to map the bookmarked position to a timestamp appropriate for the AV stream available to the client 130. [0077] FIG. 3A is an exemplary flowchart illustrating a scheme for generating a multimedia bookmark, according to an embodiment of the present disclosure. The operation of multimedia bookmark client 126 of DTV 120 starts at step 302. While a user of DTV 120 is watching a recorded program in its local storage or live TV, a user bookmarks a position of the program via a button on remote control for DTV 120 at step 304. Otherwise, the process returns at step 316. The multimedia bookmark client 126 generates the positional information represented by using one of media locators such as broadcasting time at step 306. Then, the AV content information is generated by capturing or sampling the AV signal at or around the bookmarked position at step 308. In case of a video bookmark, the AV features can be thumbnail image(s) of the captured video frame(s) which could be encoded in JPEG, for example, and low-level visual feature vectors like color histogram for one or more of the frames. It is noted that the use of one thumbnail image on the bookmark position could not be enough to find the accurate or unique position since some of AV streams include similarly-looking frames, and thus it would be desirable to additionally utilize the AV features near the bookmarked frame or position, which is described in detail later with FIGs. 4 and 7. The textual annotations could be obtained through the remote metadata server if needed at step 308. Also, the user can input an optional bookmark title at step 308. Then, the descriptive information and other information such as bookmarked date and client environment of the program are obtained at steps 310 and 312. Finally, the bookmark information is saved at local storage 122 at step 314. The details of generating the AV content information are described in FIG. 4. [0078] FIG. 3B is an exemplary flowchart illustrating a scheme for creating and sending a bookmark by using a cellular phone text messaging, according to an embodiment of the present disclosure. The process starts at step 322. While a user of DTV 120 is watching a recorded program in its local storage or live TV, the user can create a cellular phone text message at step 324. Otherwise, the process returns at step 326. At step 328, a user can input texts representing a short word to identify the TV program. Otherwise, the user can use a TV program or broadcaster- specific recipient mobile number (or short code, a 5 number text message recipient number) to identify what is being watched at step 330. Then, the text message is sent to the bookmark server 190. The recipient of the multimedia bookmark can access the bookmark received from the bookmark server 190 at the client 130 or can play the bookmark by remotely logging into the bookmark server 190.

[0079] FIG. 4 illustrates an exemplary scheme of generating the AV content information for the multimedia bookmark generated by a user of the client 120 at the frame 403 of a TV program delivered through ATSC. The bookmarked frame is assumed to be in the shot 409 ranging from the frame 401 to the frame 402 wherein shot boundaries are detected by using one of the existing image processing techniques. Then, a reduced-sized or thumbnail image corresponding to the frame 403 is generated by the AV analyzer 127. The temporal positions 406, 407 and 408 correspond to the broadcast system_times in STTs for ATSC, respectively. The system_time for the bookmarked frame 403 could be the system_time at 407 obtained by finding the first- occurring and nearest system_time preceding the frame 403. Because the accuracy of system time is usually in the order of seconds, the frame-accurate temporal localization to a specific frame may be achieved by interpolating the values of system_times in STTs in the resolution of preferably at least or about 30Hz by using a clock, such as PCR, or other equivalents. Alternatively, the use of more AV features could contribute to the accurate frame access. For example, the frames 404 and 405 look different from the bookmarked frame 403 since the frames 404 and 405 are included in the shots different from the shot 409. Thus, two thumbnail images for the frames 404 and 405 adjacent to the bookmark frame 403 and their respective distances (410 and 411) to the frame 403 in the unit of time or a number of frames are also generated for the multimedia bookmark and stored in the local storage. It is also possible to use two thumbnail images instead of three or even more than three thumbnail images for more accurate frame access. If the accurate frame access is not needed, then only one thumbnail image corresponding to the bookmark position is generated. The use of thumbnails for frame accurate access is described with FIG. 7. [0080] An exemplary scenario of sending a multimedia bookmark from a user of the client 120 to a user of the client 130 is as follows: If the user of the client 120 wants to send the bookmark selected from a list of multimedia bookmarks stored at the local storage to the user of the client 130, the user opens the stored multimedia bookmarks where the bookmark thumbnails and additional information are displayed. Then, the user selects a bookmark and sends it to other user by e-mail. Then, the multimedia bookmark client 126 generates the message of a multimedia bookmark. Table 1 shows an example of a bookmark message written in XML. After all, the bookmark message is sent to the client PC 130. <BookmarkMessage> TABLE 1

</PositionInformation> <ContentInformation> <Title>Best Super Bowl Scene</Title> <Annotation>Touchdown</Annotation> <AVContentInformation> <ThumbnailOfBookmarkedFrame> <MediaURI>20050216-221013-205 jpg</MediaURI> <ImageProfile horizontal" 160" vertical" 1207>

</ThumbnailOfBookmarkedFrame>

</Adj acentFrame>

<BroadcastingTime>2005-02- 16T22: 10: 15</BroadcastingTime> </Adj acentFrame> </AVContentInformation> </ContentInformation> <DescriptiveInformation> <ProgramTitle>2005 FootbalK/ProgramTitle> <CharinelNumber>205 </ChannelNumber>

<Name>admin</Name> <Email>admin@vmark.tv</Email> </Sender> <Receiver> <Name>hong</Name> <Email>hong@vmark.tv</Email> </Receiver> </UserInformation> </BookmarkMessage>

[0081] FIG. 5 A is an exemplary flowchart illustrating a scheme for processing a multimedia bookmark received at, for example, client PC 130 from the bookmark server 190 or other device of 120, 180 of FIG. IA, according to an embodiment of the present disclosure. The multimedia bookmark client 136 of client PC 130 starts at step 502. The multimedia bookmark client 136 receives a multimedia bookmark message at step 504. A search agent 138 searches the local storage for the AV stream pointed to by the received bookmark by using the bookmark information such as positional information, descriptive information, and client environment at step 506. If the AV stream is available in the local storage, the received bookmark is saved in the local storage at step 516 and the process returns at step 518. If the AV stream is not available in the local storage, the search agent 138 connects to the media localization agent (157 in FIG. 1C) of the media localization server 150 (service discovery) and sends a request for an AV stream URI from which the stream can be available and the timestamp for the stream to the media localization server 150 through the communication network 170 at step 508. Thus, the localization server 150 may be thought of as providing a "clearing house" type of service in which bookmarks created for one source of content are translated or mapped to bookmarks identifying the same location in a different source for the same or comparable (e.g., same program but without commercials) content,. The search agent 138 also sends the bookmark information and additional information on client environment for the PC 130 in the form of bookmark message to the media localization server 150. The search agent 138 then receives an updated bookmark message from the media localization agent 157 of the media localization server 150 wherein the updated bookmark message contains the information on the AV stream URI and an appropriate timestamp at step 510. The search agent 138 sends a request for downloading the AV stream from the media host 140 as directed by the AV stream URI received from the media localization server 150 at step 512 wherein an appropriate AV stream of the same multimedia content as the bookmarked stream at the client 120 is downloaded from the media host to the local storage of the PC 130. The updated bookmark message from the media localization server 150 is saved in the local storage of the client 130 at step 514 and the process returns at step 518. Then, the user of the client PC 130 can play the bookmarked AV stream whenever needed.

[0082] Alternatively, the process shown in FIG. 5 A can be performed when the user of the client 130 wants to play the bookmarked stream wherein the AV stream can be streamed to the client 130 instead of downloading the AV stream as shown at step 512. This process can be easily modified to handle the case when the same content are available in the storages of both the sending and receiving clients, but the same content is delivered in different way via ATSC and mobile, for example. In this case, the stream corresponding to the content does not need to be downloaded at step 512 to the bookmark receiving client 130 which just needs to send request for an appropriately modified timestamp for the bookmark position along with the received bookmark information and additional information on client environment for the client 130 to the media localization server 150.

[0083] FIG. 5B shows an exemplary flowchart illustrating a process of processing a multimedia bookmark from a multimedia bookmark list, according to an embodiment of the present disclosure. The multimedia bookmark client 136 of the client 130 starts at step 522. The multimedia bookmark client 136 reads the list of the multimedia bookmarks which are received from the bookmark server 190 and stored in the local storage 132, and displays it at step 524. When a user selects one of received bookmarks at step 526 and wants to play the bookmarked AV stream or program, the media player 134 of the client 130 starts to play from bookmarked position of the program at step 530. Otherwise, the process returns at step 528. [0084] FIG. 5C is an exemplary flowchart illustrating a scheme of processing a multimedia bookmark from a multimedia bookmark list or bookmark BBS (Bulletin Board System) by connecting to the bookmark server 190, according to an embodiment of the present disclosure. The process starts at step 542. A user of client devices without employing the multimedia bookmark client such as 136 can connect (or login) to the web server 191 of the bookmark server 190 at step 544, play the AV program pointed to by a multimedia bookmark or one of multimedia bookmarks posted on the bookmark BBS. If the user wants to see the multimedia bookmark list at step 546, the bookmark manager 193 of the bookmark server 190 reads the multimedia bookmark information from the local storage 194 and displays them at step 550. Otherwise, the bookmark BBS server 192 displays the bookmark BBS at step 548. When the user selects one of the multimedia bookmarks at step 552, then the search agent 195 connects to the media localization agent 157 of the media localization server 150 and sends a request for an AV stream URI from which the stream can be available and the appropriate timestamp for the stream to the media localization server 150 through the communication network 170 at step 556. The search agent 195 also sends the bookmark information and additional information on the client environment in the form of bookmark message to the media localization server 150 at step 556. The search agent 195 then receives an updated bookmark message from the media localization agent 157 of the media localization server 150 wherein the updated bookmark message contains the information on the AV stream URI and an on the client device such as resolution and a type of AV decoder (for example, MPEG-2 or H.264) timestamp at step 558. The search agent 195 sends a request for streaming the AV stream from the media host 140 as directed by the AV stream URI and the timestamp received from the media localization server 150 at step 560. At step 562, the media player of the client device starts to play from the updated bookmarked position of the AV stream. If a selected multimedia bookmark to play is sent via a text message generated by a cellular phone, the bookmarked position is not usually accurate and thus the media player of the client device can provide the control of auto-rewind to adjust the position. The process is terminated at step 554.

[0085] FIG. 6 is an exemplary flowchart illustrating a process performed in the media localization server 150 when the bookmarked AV stream is not recorded in the storage of the client 130. The process starts at step 602 with the receipt of a request from the client 130 (i.e., the requesting party) at step 604. If there is a request, the content locator 155 locates the AV stream from the contents & timestamp database 158 at step 606 by using the bookmark information such as descriptive information of the content and the information on the client environment of the client 120 as well as the information on the client environment of the client 130 who sent request to the media localization server 150. The content locator 155 outputs the URI of the located AV stream from which the program can be streamed or downloaded to the client 130. It is noted that the positional information (for example, broadcasting time) included in the received multimedia bookmark cannot be used to access the corresponding position of the AV stream from a media host directed by the AV stream URI found at step 606 because the AV stream of the same multimedia content that is to be downloaded or streamed from the select media host could use the timestamp different from the recorded TV program or stream in the local storage 122 (for example, broadcasting time versus media time) and further might not contain commercial breaks. By using descriptive information of the content and the information on the client environment included in the multimedia bookmark generated in the client 120 as well as the information on the located AV stream at step 606, the time stamp processor 156 accesses the content & timestamp database 158 and then transforms or maps the timestamp in the bookmark that is originally generated in the client 120 to the timestamp of the corresponding position of the stream located by the AV stream URI at step 608. When the timestamp processing step 608 does not provide a frame-accurate timestamp and a frame accurate access to the bookmarked position is needed, the AV analyzer 153 performs an AV signal pattern matching using one of the existing image processing techniques. The AV analyzer 153 searches the local version, which is stored in the local storage 159, of the AV stream located by the AV stream URI for the exact frame that is matching to the content information such as reduced-image of the multimedia bookmark generated in the client 120, and then outputs a frame-accurate timestamp corresponding to the bookmark information at step 610. Then, the requested information such as URI and the mapped timestamp information are sent back to the client 130 who requested at step 612. This process can be easily modified when the client 130 send a request for an appropriately modified timestamp for the bookmark position when the bookmarked AV stream is recorded in the local storage of the client 130 by omitting the step of locating content URI (step 606). The process returns at step 614.

[0086] FIG. 7 illustrates an exemplary AV pattern matching process for locating a frame accurate bookmarked position that is performed by the AV analyzer 168 of the media localization server 150 by using three thumbnails contained in the AV content information of the multimedia bookmark information although one or more thumbnail images instead of three or other AV low- level features or patterns can be used. The AV file 701 stored in the local storage 159 in the media localization server 150 is the same as the AV stream located by the URI by step 606 (or, it could be the transcoded version in low-bit rate having the same but reduced-size frames). The start of the AV file 701 corresponds to media time zero seconds and the end of the AV file 701 corresponds to the media time 703. The three thumbnails 707 consisting of the bookmarked frame 403 and its two adjacent thumbnails 404 and 405 are the AV content information in the multimedia bookmark. The media time 704 (the timestamp mapped at step 608) is not assumed to be frame accurate. The AV analyzer 168 searches the AV file 701 for three frames best matching three thumbnails 707 subject to the temporal distances 410 (or 708) and 411 (or 709) within the search range 706 centered at the media time 704 by using one of the existing image processing techniques, resulting in the frame accurate media time corresponding to the bookmarked frame 403. To speed up the matching process, only intra- frames of the video stream can be searched at the expense of frame accuracy. This process can be equally applied to the case of audio only streams.

[0087] It is noted that the AV pattern matching technique described in FIG. 7 can be performed by the AV analyzer such as 127 in client device to appropriately map the timestamp for a bookmark position between different types of client devices (for example, between DTV and mobile device) when the same content are available in those client devices without connecting to the media localization server 150.

[0088] The present disclosure is described in conjunction with various embodiments provided for the purpose of illustration and not limitation. The present disclosure is limited by the scope of the following claims and equivalents thereof and not by the disclosed embodiments.

Claims

What is claimed is:

1. A method of mapping a timestamp from a first AV stream available to a first platform to a second AV stream of comparable content available to a second platform, comprising: receiving a bookmark from a requesting party, said bookmark identifying a first AV stream and a position within said first AV stream; locating a second AV stream having content comparable to the content of said first AV stream based on information in said bookmark; transforming a timestamp in the received bookmark to a corresponding timestamp in said second AV stream; and when said second AV stream is available to the requesting party, sending said transformed timestamp to said requesting party; and when said second AV stream is not available to the requesting party, additionally sending access information for said second AV stream to said requesting party whereby said requesting party can view the second AV stream starting from a position originally identified by the bookmark to the first AV stream.

2. The method of claim 1 additionally comprising sending a message from a mobile device to a server containing profile information about the sender of the message, and generating a bookmark based on the message and the profile information.

3. The method of claim 2 wherein said generating a bookmark includes generating a bookmark in the form of a thumbnail image.

4. The method of claim 2 wherein said sending a message from a mobile device includes sending a message from a cell phone.

5. The method of claim 2 additionally comprising using an SMS short code as the method of designating an AV stream to bookmark.

6. The method of claim 2 additionally comprising using a description of an AV stream channel as the method of designating an AV stream to bookmark.

7. The method of claim 2 additionally comprising one of storing the bookmark or forwarding the bookmark or a bookmark message to the sender.

8. The method of claim 7 wherein said forwarding the bookmark or bookmark message to the sender includes forwarding to the sender's e-mail account, social networking page, or other managed account.

9. The method of claim 2 additionally comprising one of storing the bookmark or forwarding the bookmark or a bookmark message to another user or website.

10. The method of claim 9 wherein said forwarding to another website includes posting the bookmark on a bulletin board or on a social networking page.

1 1. The method of claim 1 additionally comprising sending a message from a computer to a server containing profile information about the sender of the message, and generating a bookmark based on the message and the profile information.

12. The method of claim 11 wherein said generating a bookmark includes generating a bookmark in the form of a thumbnail image.

13. The method of claim 11 additionally comprising one of storing the bookmark or forwarding the bookmark or a bookmark message to the sender.

14. The method of claim 13 wherein said forwarding the bookmark or bookmark message to the sender includes forwarding to the sender's e-mail account, social networking page, or other managed account.

15. The method of claim 11 additionally comprising one of storing the bookmark or forwarding the bookmark or a bookmark message to another user or website.

16. The method of claim 15 wherein said forwarding to another website includes posting the bookmark on a bulletin board or on a social networking page.

17. The method of claim 1 wherein said transforming the timestamp includes performing signal pattern matching to identify a location in the second AV stream corresponding to signal pattern information in the received bookmark.

18. A method of generating multimedia bookmarks, comprising: sending a message from a mobile device or a computer to a server containing profile information about the sender of the message; and generating a bookmark based on the message and the profile information.

19. The method of claim 18 additionally comprising one of storing the bookmark or forwarding the bookmark or a bookmark message.

20. The method of claim 18 wherein said generating a bookmark includes generating a bookmark in the form of a thumbnail image.

21. Apparatus for mapping a timestamp from a first AV stream available to a first platform to a second AV stream of comparable content available to a second platform, comprising: modules for receiving and decoding broadcast streams; a content locator responsive to a received bookmark for locating a broadcast stream; and a timestamp processor for mapping the timestamp in the received bookmark to a corresponding timestamp in said located broadcast stream.

22. The apparatus of claim 21 additionally comprising an AV analyzer responsive to said content locator.