WO2010132718A2 - Playing and editing linked and annotated audiovisual works - Google Patents

Playing and editing linked and annotated audiovisual works Download PDF

Info

Publication number
WO2010132718A2
WO2010132718A2 PCT/US2010/034808 US2010034808W WO2010132718A2 WO 2010132718 A2 WO2010132718 A2 WO 2010132718A2 US 2010034808 W US2010034808 W US 2010034808W WO 2010132718 A2 WO2010132718 A2 WO 2010132718A2
Authority
WO
WIPO (PCT)
Prior art keywords
video
storage medium
cue
file
metadata
Prior art date
Application number
PCT/US2010/034808
Other languages
French (fr)
Other versions
WO2010132718A3 (en
Inventor
David H. Kaiser
Bruce Schwartz
Carl Rosenberg
Original Assignee
Coincident.Tv , Inc.
Reyes, Luis
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Coincident.Tv , Inc., Reyes, Luis filed Critical Coincident.Tv , Inc.
Priority to EP10775570.4A priority Critical patent/EP2430833A4/en
Publication of WO2010132718A2 publication Critical patent/WO2010132718A2/en
Publication of WO2010132718A3 publication Critical patent/WO2010132718A3/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/235Processing of additional data, e.g. scrambling of additional data or processing content descriptors
    • H04N21/2353Processing of additional data, e.g. scrambling of additional data or processing content descriptors specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/74Browsing; Visualisation therefor
    • G06F16/748Hypervideo
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47End-user applications
    • H04N21/478Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4782Web browsing, e.g. WebTV
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85Assembly of content; Generation of multimedia applications
    • H04N21/854Content authoring

Definitions

  • the present disclosure generally relates to video playing, video editing, and displaying hyperlinked media.
  • World Wide Web sites more closely to the preferences of Internet users, based on collecting explicitly-specified preference data, based on a user profile, or by inferring preferences through collecting metadata that is derived as the Internet user selects pages or performs online actions.
  • Video editors such as Adobe Premiere Pro and Final Cut Pro enable users to select multiple video clips, join the clips, and annotate the clips by defining cue points and associating text notes with the cue points.
  • FIG. IA illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments.
  • FIG. IB illustrates a process of creating video programs, which are linked to metadata, which can control operation of a video player.
  • FIG. 1C illustrates a process of playing a video program that is linked to metadata.
  • FIG. 2 illustrates an example screen display that the video linking editor logic generates and causes displaying.
  • FIG. 3 graphically illustrates an example video linking arrangement.
  • FIG. 4 illustrates a screen display in the Adobe Premiere video editor in which a video file has been created with the segments and advertisements and appropriate cue points.
  • FIG. 5 illustrates a portion of a screen display showing a cue point list for the video of FIG. 3, FIG. 4.
  • FIG. 6 illustrates the metadata panel populated with data for the Start cue point of the example.
  • FIG. 7 illustrates the cue point data configured with values from user input that create such a cue point.
  • FIG. 8 illustrates a display generated at playback time based on the metadata that has been created in the present example.
  • FIG. 9 illustrates appropriate values of program- wide metadata for the present example.
  • FIG. 10 illustrates an example screen display that includes a directory.
  • FIG. 11 illustrates an example screen display that illustrates a player screen that may be generated and displayed in a computer display unit by metadata-capable video player logic.
  • FIG. 12 is a block diagram that illustrates a computer system upon which an embodiment of the invention may be implemented.
  • FIG. 13A illustrates an annotation coupled to a web service providing automated text messaging in association with an enriched video program.
  • FIG. 13B illustrates a frame of an enriched video program as displayed in a player window.
  • FIG. 14 illustrates a frame of a video program having a highlighted service icon.
  • FIG. 15A illustrates an annotation that provides a user choice.
  • FIG. 15B illustrates a frame of a video segment in a sequence for which Audrina is the featured character.
  • FIG. 16 illustrates concurrent playing of an enriched video program and displaying an associated web page.
  • FIG. 17A illustrates an example of playing an enriched audiovisual program with annotations that implement chapter selections.
  • FIG. 17B features a navigation animation, web integration icons, topic launch icons, and menu access link.
  • FIG. 17C illustrates a video window providing a menu of episodes in a collection or associated with a subscription.
  • FIG. 17D illustrates use of annotations to form elements of a main menu page for a video program subscription.
  • FIG. 18 A illustrates an example news program in which annotations may be used to provide a directory or menu of a plurality of news stores, features, segments, or related information.
  • FIG. 18B illustrates the news program of FIG. 18A after a viewer has selected a program link that is defined using an annotation having an association to a website.
  • FIG. 18C illustrates the browser window of FIG. 18B after the scroll bar has been moved.
  • FIG. 19 A illustrates playing a video program in which annotations are associated with multiple different responsive behavior types.
  • FIG. 19B illustrates an example of displaying a separate browser window below or behind the video window of the player window.
  • FIG. 20 illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments with a browser-based player for enriched video programs.
  • FIG. 21 illustrates an example screen display that the video linking editor logic generates and causes displaying and in which a Cue tab is selected.
  • FIG. 22 is a screen display diagram of the Metadata tab of an example Editor window.
  • FIG. 23 is a screen display diagram of an example Editor window in which an
  • Annotation tab is selected.
  • FIG. 24 is a screen display diagram of an example Editor window in which a Web tab is selected.
  • FIG. 25 is a screen display diagram of an example Editor window in which a Layout tab is selected.
  • Various embodiments provide an editor, a player, and a metadata format.
  • the editor implements a method of creating, for a video file consisting of multiple segments, metadata describing one or more display operations, decision operations, branching operations, video linking operations and web media linking operations, and associating the metadata with the video file.
  • the player implements a method of interpreting the metadata during playback of the video file and performing the operations in coordination with playback.
  • the metadata format comprises computer-readable data storage media encoded with tags and values which when interpreted cause performing particular display, decision, branching, video linking and web media linking operations. Metadata may comprise cue point type names for various cue point types, and attribute values associated with the cue point types that control the particular behavior of the player in performing the operations.
  • FIG. IA illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments.
  • a computer 102 is coupled directly or indirectly through one or more networks 120 to a web server 130 and optionally to a file server 132.
  • network 120 may comprise a local area network (LAN), wide area network (WAN), an internetwork, or a combination.
  • Web server 130 hosts one or more video files, HTML documents, HTTP servers or application servers, or other web content.
  • File server 132 stores or hosts video files 122, graphics files 124, and metadata files 126.
  • Computer 102 hosts or executes an operating system 104 that supervises I/O, storage management, and execution of application logic.
  • computer 102 further comprises a video editor 106.
  • video editor 106 include Adobe Premiere and Final Cut Pro.
  • computer 102 comprises a browser 108.
  • browser 108 include Firefox, Safari, Chrome and Internet Explorer.
  • computer 102 is coupled to storage 140, which broadly represents any data storage device, storage area network (SAN), network attached storage (NAS), or network file system (NFS) unit or server.
  • Storage 140 may reside on network 120 or on a server coupled to the network.
  • Storage 140 stores video files 122, graphics files 124, and metadata files 126.
  • computer 102 further comprises video linking editor logic 110 and metadata-capable video player logic 112.
  • computer 102 only comprises player logic 112 and does not have an editor; such an embodiment might be used by an end user who is viewing video programs that have been prepared by someone else. Thus, the use of video linking editor logic 110 is not required.
  • the video linking editor logic 110 is generally configured to cause one or more processors in computer 102 to receive user input specifying links between segments of a video file and other media such as other segments in the same file, other segments of other video files, graphics files, online content such as web sites or web applications, and other rich media content; to create representations of the links in metadata; and to store the metadata and link- related information in the metadata files 126 in association with related video files.
  • a user of computer 102 may interact with video linking editor logic 110 to select one or more of the video files 122, from storage 140 or file server 132, create links using editing functions that the editor logic provides, integrate graphics files 124 and references to content on web server 130, and then store metadata files 126 either at storage 140 or in file server 132.
  • the metadata files 126 identify the associated video files 122 and contain metadata defining links among segments, link types, and link-related information to support novel playback functions and other user experiences. Other more specific functions of video editor linking logic 110 are described in other sections herein.
  • the metadata-capable video player logic 112 is generally configured to open metadata files and associated video files, and to play the video files while interpreting and responding to links and related information and instructions in the associated metadata files. Other more specific functions of metadata-capable video player logic 112 are described in other sections herein.
  • the metadata-capable video player logic 112 may be implemented within a web browser and comprising a browser support library and browser-executable code, such as JavaScript, that is received in and executed by the browser at the time that an end user selects a video for playing.
  • the browser support library may be any video playing plug-in component for a browser. Examples include Macromedia Flash and Silverlight.
  • web browsers may use the VIDEO tag of HTML version 5 to render video and HTML and JavaScript to implement the player logic 112.
  • the player logic 112 may be partially implemented on server 132 or another server using dynamic AJAX techniques.
  • the server may convert data defining annotations into HTML to be displayed in the player.
  • the metadata-capable video player logic 112 is implemented as a standalone program application that may be installed locally in computer 102.
  • any software development kit (SDK) that is capable of displaying video could be used to implement the player. Examples include SDKs for Apple Mac OS X, Microsoft WINDOWS, and Linux.
  • Each of the computer 102, video linking editor logic 110 and metadata-capable video player logic 112 may be implemented in various embodiments using a computer, one or more application-specific integrated circuits (ASICs) or other digital electronic logic, one or more computer programs, modules, objects, methods, or other software elements.
  • computer 102 may comprise a special-purpose computer having particular logic configured to implement the elements and functions described herein.
  • service computer 102 may comprise a general purpose computer as in FIG. 12, loaded with one or more stored programs which transform the general purpose computer into a particular machine upon loading and execution.
  • video linking is facilitated by creating, in metadata files associated with video files, executable instructions and/or descriptive information that are linked to cue points in the video files.
  • a cue point generally comprises an association of a name to a position within a video file, wherein the position is typically expressed as a time value or timestamp.
  • cue points are created for a particular video file using video editor 106; the names and values of cue points become part of the video file through conventional operation of the video editor. Thereafter, user interaction with the video linking editor logic 110 can create links, operations and link-related metadata information for one or more of the cue points.
  • FIG. IB illustrates a process of creating video programs, which are linked to metadata, which can control operation of a video player.
  • FIG. 1C illustrates a process of playing a video program that is linked to metadata.
  • the video linking editor logic 110 is configured to perform at least selected functions of FIG. IB and the metadata-capable video player logic 112 is configured to perform the functions of FIG. 1C.
  • a video editor creates and stores one or more cue points in a video file.
  • FIG. IB presumes that at least one video file has been created and stored on a computer, such as computer 102.
  • Step 150 may comprise a user interacting with the video editor 106 to create and store named cue points in the video file as further described herein.
  • step 150 can involve a process or logic in computer 102, or another computer, creating cue points in a video file using programmatic techniques or electronic communication of messages to the computer.
  • the computer receives user input identifying a video file.
  • Step 152 may involve invoking the video linking editor logic 110 and specifying the file name of one of the video files 122, or specifying the name of one of the metadata files 126, which will include an internal reference to an associated one or more of the video files.
  • the video linking editor logic 110 reads the video file, finds or creates an associated metadata file, displays data for cue points in the video file, and displays any associated metadata relating to links to other segments or content. If one of the video files 122 is specified at step 152 and no existing metadata file is found, then the video linking editor logic 110 creates a related metadata file. If an existing related metadata file is found, then that file is read and metadata relating to cue points is displayed on a display unit that is coupled to computer 102. An example graphical user interface that may be generated to display the metadata is further described herein in connection with FIG. 2, but the approach of FIG. IB does not require that particular GUI.
  • the computer receives user input specifying, for a particular cue point, a cue point type. For example, interacting with the GUI of FIG. 2 or through other means, a user or external computer process or logic selects one of the previously created cue points of the video file and provides input specifying a cue point type value.
  • a cue point any of several types of operations may be defined to be performed at the time of playback using the metadata- capable video player logic 112.
  • a cue point within a video file and the operations performed at the cue point are sometimes collectively termed a cue point.
  • Cue points as defined herein can refer to video, coupled video-web contexts or non-temporal web locations (or "web points," as further described).
  • cue points enable a user at playback to jump forward and backward in time in a video, and jump between web content and video content. Since the user and the environment can change the order in which media is played, the metadata-capable video player logic 112 maintains data indicating the user's prior location so that the player can transfer control to a prior location.
  • web points define an end for web content that specify where to transfer the user when the user has reached the end of a navigation path.
  • Both video and web content can be displayed on the screen at the same time, overlaid over web content or using a picture-in-picture representation, and time can be running or paused.
  • selecting a back operation transfers control to a previously viewed page but when the earliest page is reached then a subsequent back operation transfers control away from web content and to the previously viewed video segment.
  • video is displayed, performing a back operation returns to the beginning of the current video segment.
  • FIG. 24 is a screen display diagram of an example Editor window in which a Web tab is selected.
  • the Web tab 2402 of the example Editor screen display 2102 may be used, in an embodiment, to create and store web points in association with a video program.
  • Web tab 2402 displays a list of all web points that have been defined for the video program that is previewed in video window 2403.
  • Selecting an Add web point control 2405 causes the editor logic 110 to display a data entry panel 2404 that may receive user input of parameter values defining attributes of a web point.
  • attributes include an interest URL 2406, query string 2408, target cue point, web view layout definition, description, and thumbnail graphic image.
  • interest URL 2406 refers to an online electronic document that is loaded and displayed at playback time if the user requests additional information about the data shown at the web point.
  • query string 2408 comprises a database query that is submitted to an online engine if the web point is selected, to generate a search result so that the user receives current search result information associated with the web point.
  • the target field defines a target cue point to which the user is directed at playback time after having viewed a web document associated with the web point.
  • the web view layout definition field identifies a layout format for the player to be used when displaying web information; in an embodiment, the layout format is one of the formats shown in FIG. 25, which is described further herein.
  • the description field is a text description of the web point to display
  • the thumbnail graphic image is a graphic image to display in the player to denote the web point.
  • Directory or Annotation a directory or annotation specifies one or more graphics files, web services, and associated links; at playback, the graphics files are selectable as hyperlinks to cause playback of other video segments and the web services may be invoked automatically to fire database queries, retrieve data, dispatch emails or text messages, or perform other communication functions as defined in the web services.
  • Metadata can specify that when a particular cue point is reached during playback, the player should jump to another cue point within the same video file.
  • the destination cue point may be earlier in time or later in time than the cue point from which a jump is made.
  • Metadata can specify that when a particular cue point is reached during playback, the computer 102 should connect to a web site at web server 130 and display a web page or invoke a web application. Typically the web site content is associated with or related to the video content at the cue point, but such an association is not required.
  • Change media under external process control metadata can specify that when a particular cue point is reached during playback, the computer 102 should switch to one of several pieces of media, as determined by a web service, and continue as specified in the media to which the switch was made.
  • Metadata can specify that when a particular cue point is reached during playback, the computer 102 should switch to one of several pieces of media, determined by user selection of an image associated with the media.
  • Overlay web content metadata can specify that when a particular cue point is reached during playback, the computer 102 should display a graphical and interactive overlay.
  • This overlay is logically and computationally associated with a web service.
  • the web service maintains databases (both session and persistent) that can be used to influence the course of playback, for example with an insertPt cue point.
  • the metadata-capable video player logic 112 invokes asynchronous web services to control user interaction with the overlaid web components.
  • the video linking editor logic 110 can define and store one or more web points comprising names with associated URLs, graphics and text. These web points can substitute for video cue points. For example, all targets for a user choice cue point can be either video cue points or web points. In this context web points also define an "end" action to be followed in circumstances paralleling reaching the end of a video segment.
  • video linking editor logic 110 can define and store, in the metadata file, one or more cue points that include data defining required associated web pages.
  • cue points are termed synchronized attributes or cue point attributes and refer to a specified point in a video that automatically creates a primary and secondary window.
  • any cue point can have an associated web page that is to be displayed while the cue point is active.
  • the associated web page is automatically loaded and displayed in a browser window that appears conceptually under the video player window.
  • under refers to Z-axis ordering of video and web content; in an embodiment, the video content is "over” the web content in the sense that the video content may obscure certain web content.
  • the size and positioning of the video and web content can also be specified in association with the cue point using the video linking editor logic.
  • an author can define a video that provides a synchronized display of an audiovisual work and Internet content such as web pages.
  • the web pages might comprise an advertisement or other online information that is relevant to a particular event on the TV program. For example, an author can set a cue point for a time in the video at which a character appears wearing particular distinctive clothing, and can associate a web page for a merchant of that clothing with the cue point.
  • a cue point can associate a time in the video at which an actor appears with a particular web page of the Internet Movie Database (IMDB) service, www.imdb.com, that contains background, filmography and other details for the actor.
  • IMDB Internet Movie Database
  • This approach may be economically attractive both to the broadcaster of the video and the associated web site; for example, the associated web site benefits from an additional page view while the broadcaster concurrently continues to have a viewer viewing the video. More broadly, this approach enables events occurring in temporal media such as video to cause automatic changes in state-based media such as web pages.
  • creating a synchronized attribute is performed by selecting the Cue tab 2106 in the editor screen display 2102 (FIG. 21), selecting a cue point to which a synchronized attribute should be associated, selecting an Other tab in a Parameters pane, selecting Browse and selecting an appropriate web point.
  • metadata can specify that when a particular cue point is reached during playback, the computer 102 should return to a previous segment from which a branch was taken.
  • web-based cue points define an end, even though web content is a non-temporal media, to specify where to go when the user has reached the end.
  • Both video content and web content can played or viewed on screen at the same time, overlaid or picture-in-picture, and time can be running or paused.
  • selecting a Back button while viewing web content causes backtracking among hyperlinks in the manner of a web browser, but when no other prior links exist then a selection of a Back operation transfers control to the video segment from which the web content was reached. In content, moving backward in video transfers control to the beginning of the video.
  • metadata-capable video player logic 112 interprets metadata such that when the user is watching video, a web link to some form of related content is always available. If the user selects the web link and views the web content, the player displays the video in a reduced size picture-in-picture form. Further description of the foregoing cue point types is provided in other sections of this disclosure.
  • Step 158 the video linking editor logic 110 updates the computer display based on the cue point type to provide input fields and display fields for metadata values that are associated with the specified cue point type.
  • Step 158 may also include receiving user input that indicates particular metadata values for the input fields. For example, if the cue point type provided at step 156 is "modal story branch," then at step 158 an input is received to specify two or more target cue points that represent branch destinations.
  • the video linking editor logic 110 creates and stores the cue point type and the associated metadata values in the metadata file that is associated with the video file.
  • the type and values may be stored in one or more XML script(s) within one of the metadata files 126.
  • XML is not required in all embodiments and the metadata files 126 may represent cue point names, types and metadata values in other forms that can be read by the metadata-capable video player logic 112 and used to control linking, branching, decisions, web interaction, and other content operations when cue points are reached.
  • Step 162 represents testing whether a user has requested to exit or discontinue using the video linking editor logic 110. If no exit request is received then control is transferred to step 156 or step 158 for the computer to await further user input relating to cue points. Alternatively the computer may perform an exit operation at step 164, for example, by closing the video file and metadata file.
  • a playback mechanism may be integrated into the process so that a user can play and view a video program or segment while determining what cue point types and values to specify.
  • the playback mechanism supports non-linear playback of video so that the player can execute branch operations, play one of a plurality of different alternative video segments at a branch point or decision point, return to a prior point and continue playing the next segment thereafter, and other complex operations consistent with the rich media authoring capabilities described herein.
  • the user may request playing a video segment or performing trick play functions such as fast forward or rewind.
  • selecting a different named cue point at step 156 causes the player mechanism to display a first frame of the video segment that starts at the selected cue point or to begin playing the video from that point.
  • a video file having internally stored named cue points becomes associated with a separate metadata file that specifies cue point types and metadata values relating to control functions for the video file, related networked content, and other user interactions.
  • the metadata-capable video player logic 112 is configured to play the video and, as each cue point is reached, perform the control functions based on the cue point types and metadata values that are specified in the metadata file.
  • FIG. 1C broadly represents a process involving opening a stored video file having one or more video segments and one or more cue points in the video file, and opening a metadata file that contains an internal reference to the video file; playing a first video segment of the video file; in response to reaching, during the playing, one of the cue points that are defined in the video file: receiving from the metadata file one or more metadata values identifying a cue point type, and one or more values of attributes that are associated with a particular cue point type, the cue point type and attribute values defining features of an action to perform at the cue point during playing of the video file; performing the action using the attribute values to determine and perform particular features, displays, or controls associated with the action.
  • the computer initiates executing the metadata-capable video player logic 112. Initiating execution may occur in response to user input, or in response to an instruction from other hardware logic or computer processes. For example, a user, logic, or process may select and invoke one of the metadata files 126 or video files 122, and in computer 102 the files may be associated with the metadata- capable video player logic 112 as an application that is launched when the files are invoked.
  • the metadata-capable video player logic 112 locates any existing metadata files and displays a list of the metadata files. Each metadata file may be represented visually in the list using a still image or other graphics file that is referenced within the metadata file.
  • the metadata-capable video player logic 112 may generate a display of thumbnail images, each image representing an associated metadata file.
  • the metadata-capable video player logic 112 receives user input selecting a metadata file from the list.
  • the graphic images may comprise selectable links and the user may select one of the images using a pointing device.
  • Steps 172 and 174 are described as optional because a selection of a metadata file may be unnecessary if the metadata-capable video player logic 112 is invoked by a user or process launching one of the metadata files 126 rather than launching or invoking the metadata-capable video player logic independently.
  • step 176 the selected metadata file is opened.
  • Each of the metadata files 126 is configured to internally name or reference at least one of the video files 122. Therefore, the metadata-capable video player logic 112 reads the selected metadata file, identifies the referenced video file, and opens the referenced video file at step 178.
  • the metadata-capable video player logic 112 enters a loop that begins when the video player logic plays the video file that was found and opened at step 178.
  • a test is performed to determine whether a cue point has been reached.
  • Step 182 represents the occurrence of an interrupt or other event indicating that a cue point was reached.
  • step 182 may be implemented by examining stored metadata values relating to a segment and setting timers that cause generic, non- video events to occur when the video events would have occurred. The timers are adjusted as the user moves among video segments and plays video segments, as the amount of time to a given video event changes as a result of such movement.
  • this approach enables content to play correctly even if the cue points have been removed from the video in the course of transmission or transcoding. For example, preparing video for the YouTube online player results in the YouTube system discarding the cue points and the present approaches enable video productions to play properly on YouTube.
  • the NO control path of step 182 represents continuing playback and waiting until the next cue point is reached.
  • the metadata-capable video player logic 112 determines the name of the cue point that has been reached.
  • the metadata-capable video player logic 112 reads and executes one or more metadata scripts and/or values associated with the current cue point, based on functions and behavior configured in the video player logic.
  • the metadata-capable video player logic 112 comprises logic or program instructions that define what functions are performed for all cue point types, and the metadata files 126 specify cue point types and attribute values that control how the functions are performed, such as specific video displays, graphical displays, user interactions, branches, links or other control functions.
  • step 186 control returns to step 180 to continue playing the current video segment.
  • the current video segment after step 186 may be a different video segment than earlier, depending on the cue point type and its associated metadata values.
  • the process of FIG. 1C and the metadata-capable video player logic 112 may be configured to asynchronously process user input requesting trick play functions or loading other video files or metadata files.
  • FIG. 1C enables playing a video with a rich set of controls and user interactions including branching to different video segments automatically, presenting a user with a branch selection menu and branching to particular video segments in response to user selection, determining a branch or different video segment using a web service, presenting web content that is related or associated with a video segment, and other controls and user interactions.
  • the video file does not require internal modification and can be used with other players that do not provide the controls and user interactions.
  • the controls and user interactions can be authored using an editing process as shown for FIG. IB, enabling video producers to rapidly create rich video productions without detailed knowledge of programming.
  • video linking editor logic 110 uses one or more cue points that have been previously defined for video files on which the video linking editor logic operates; in other embodiments as further described herein, cue points may be defined independently of the video, using the video linking editor logic, and are stored in metadata separate from the video files.
  • users create cue points and cue point names using the video editor 106.
  • a "video" is a single piece of video content (a file or a URL) typically with many cue points; within a video each "segment" begins and ends with a cue point without any cue points in between.
  • a “compound segment” or “inline” segment has cue points within it, i.e., cue points in addition to the beginning and ending cue points.
  • An external video, specified by a URL may also contain cue points, and depending upon their organization, these cue points may be segments or compound segments. The player can refer to internal and external cuePoints transparently.
  • video editor 106 is used to organize one or more video files into pieces each having a cue point at the start, at the end, and at any point to or from which the current time ("head") can jump.
  • Cue points have an attribute canBeDestination. If this is set to false, the cue point cannot be a destination of any action which causes the playhead to jump.
  • Cue points with canBeDestination set to false are typically used as markers for overlaying annotations but where the author does not want that point in the video to be a destination for a chapter jump.
  • cue points with cue type "null”. These are used to place markers in the video at precise points that the author may at some future time want to use. Null cue points require less processing. For example, when a logger (the first and least expensive person in the workflow on professional video shoots) logs the shots of the raw footage they can put Null cue points at every shot without adding undue computational overhead. After rendering, a step that takes many hours of computation, these cue points are all available and can selectively be changed into meaningful cue points like "regular” or "insertPt" without re-rendering. [0086] A user creates one or more cue points as desired using a cue point tool within the video editor 106.
  • a cue point is created by moving an icon representing a playback head to a particular point in the video file and selecting "Cue Point.”
  • a last video segment in a video file is supplemented with a terminal video segment denoted "endNote.”
  • an endNote may comprise short piece of junk video positioned about two seconds after the last usable video segment. The endNote is created with zero cue points to prevent confusion with an automatic, invisible cue point that the video editor 106 automatically inserts at the end of the last piece of media.
  • the endNote is positioned about two seconds after the last usable video segment to prevent reaching the actual end of the video file under certain conditions; user experience has determined that when the metadata-capable video player logic 112 issues a command to pause or stop, the virtual head keeps moving for a short time interval.
  • video linking editor logic 110 is configured to enable a user to define one or more cue points independent of the video for storage in metadata files 126.
  • Soft cue points allow the user to insert, delete, and change the time of cue points directly into a video that has already been imported into storage associated with the editor logic 110.
  • a soft cue point is created using editor logic 110 by selecting the Cue tab 2106 (FIG. 21) and selecting an add cue point control 2108. Selecting the add control 2108 causes editor logic 110 to create and store metadata for a new cue point at the default time of 00:00:00:00. Selecting the Description tab enables a user to insert a particular time for the cue point. The time can be determined by scrolling through the video using the trick play controls 2110.
  • a cue point is a named marker for a particular point in a video segment.
  • a cue points may comprise a name, a time value indicating the particular point, and other metadata that defines what actions occur when that point is reached during playing the video.
  • video player logic 112 continuously compares the time value of a current position of a logical playback head within a video segment, and determines if the current time value is equal to any soft cue point that has been previously defined and stored in the metadata file 126 that is associated with the video segment. When a cue point is reached, the video player logic 112 performs one or more particular operations that are defined in the metadata of the cue point.
  • an author can build a complete interactive video experience from existing video files without needing to use complex tools like Adobe Premiere or Final Cut to create cue points.
  • an author can select and use video files that are maintained on a third party video server or hosting site, such as YouTube, and streamed from that server or site to an end user using the video player logic 112 at the time of playback.
  • the video files do not need to have cue points previously defined for and stored in them. Instead, the user uses video linking editor logic 110 to create cue points and store the created cue points in the metadata files 126.
  • the metadata files 126 can be launched and can invoke the video player logic 112 to cause the video player logic to invoke streaming the video segments from the third party video server or hosting site while concurrently performing one or more operations as cue points are reached in playing the video segments.
  • video linking editor logic 110 is configured to enable a particular metadata file 126 to reference cue points that are defined in other metadata files 126.
  • a cue point may comprise a contained element termed a target, which specifies a cue point by name and optionally links it with an association attribute.
  • An attribute of a target may be a cue point reference, which may reference cue points that are in other metadata files.
  • a cue point reference is formed as a URL comprising a file location (path), file name, and a URL fragment that identifies a particular cue point.
  • the cue point reference "http://www.coincident.tv/cplfiles/foo.cpMDadArrivesHome” identifies a cue point named "DadArrivesHome” within a metadata file named "foo.cpl” that is stored in the folder or director "cplfiles" of the "coincident. tv” domain.
  • a cue point in any metadata file definition in which a cue point can be a target, for example, as the target of an annotation, insert point, goto cue point, or directory or user choice entry, that target can be in another file referenced by relative URL.
  • External cue points beneficially enable authors to work with cue points that otherwise might require multiple other steps to re-define for a particular audiovisual work. For example, a 2-hour video program might contain dozens of cue points, but a particular author might wish to reference only a few of the cue points. The author need not re-define the same cue points in a new metadata file for a new audiovisual project, but can reference previously defined cue points within other, previously created metadata files. Therefore, the author can create a cross-linked metadata control structure that can simplify video program development based on other files or segments. [0096] 5. AUTHORING VIDEO LINKS [0097] 5.1 EDITOR GUI OVERVIEW
  • video linking editor logic 110 generates and causes displaying a graphical user interface (GUI) on a computer display unit, and the GUI provides cue point editing functions that can be used to link video segments and other content in a plurality of ways.
  • GUI graphical user interface
  • the editor logic 110 is also configured to create and store, based on user input interacting with the editing functions and providing selections and values, metadata describing the links.
  • the metadata comprises one or more scripts expressed in a Cue Point Language (CPL).
  • CPL comprises an XML-based language that describes non- linear structures in a mixture of video and web media.
  • CPL can be embedded into digital video content that is available from a plurality of sources such as broadcast, DVR, DVD, broadband, game consoles.
  • CPL can be associated with web content also.
  • the resulting metadata may be played back with a CPL-capable player to create a playback experience that integrates video and interactive web-based graphic elements in such a manner that the sequence of playback is influenced by user interaction, run-time execution of code embedded in the video, run-time interaction with code referenced by data embedded in the video, and calls to remote web services in combination with jump tables authored in the editor and embedded (or embedded by reference) in the video.
  • the CPL may be viewed as an architecture rather than a user interface.
  • CPL implements a mechanism for a modal n-way branch
  • the author can use that mechanism to provide a video production that is graceful and easy to use, or confusing and user- hostile.
  • CPL is compatible with a variety of playback platforms, asset locations and video formats.
  • video content can be viewed using screens that are attached to processors, disks or network connections.
  • Platforms may consist of computers, game consoles, set- top boxes, or mobile devices.
  • CPL is format independent with the assumption that all digital video formats define cue points and have ways to associate events and text with the cue point.
  • CPL is location independent and can interoperate with video that originates from any desired source.
  • FIG. 2 illustrates an example screen display that the video linking editor logic generates and causes displaying.
  • screen display 200 generally comprises a video window 202, metadata panel 204, cue point list 206, a web point list, an annotation list and cue point data panel 208.
  • Video window 202 is configured to play and show one or more video segments representing a linked video project and comprises buttons 201 that are configured to receive user input selecting a playback function and trick play functions such as jumping to different segments that are forward or backward in time.
  • a "video project" refers to an association of a video file and a metadata file.
  • Metadata panel 204 receives and displays metadata values that pertain to a project as a whole.
  • metadata panel 204 comprises unique id field 212, video file field 214, and web service field 216.
  • the unique id field 212 is configured to receive a name, number, or other character sequence that uniquely identifies the current video project, and the unique id value is used in naming the metadata file that the editor creates and associates with a video file and to coordinate dynamic updates with a server.
  • the video file field 214 displays a name of a video file that has been loaded using the File menu 205 and previously created with cue points in a video editor.
  • the name may comprise a pathname in a filesystem accessible to the computer that is hosting the video linking editor logic 110, a URL identifying video in a web server, or another form of location identifier specifying a location of video.
  • selecting the File menu item 205 initiates a File Open dialog and after a file selection is made the logic 110 displays a value in the video file field 214 and opens and displays the named video file in video window 202.
  • a user may direct logic 110 to load a previously created metadata file, and in response, the video linking editor logic locates a video file that is referenced within the metadata file and displays the name of that referenced video file in video file field 214.
  • the web service field 216 is configured to receive user input identifying a web service in the form of a URL.
  • the specified web service may be hosted on computer 102 or on a remotely located computer.
  • the web service may comprise a web application or a script file.
  • the web service provides a control mechanism for interacting with insert points, overlays, and other types of cue points that are further described in other sections herein.
  • Cue point list 206 is configured to display a list of cue points that have been previously defined in the video that is shown in video window 202.
  • video linking logic 110 loads and displays the named video in video window 202 and concurrently reads and displays the cue point data that was embedded in the video file as a result of creating cue points using the video editor.
  • Cue points found in the video file are listed in one or more rows 218 of list 206 and each row includes time of the associated cue point in a time column 220 and a name in name column 222.
  • existing web points in the video are displayed in a separate list, and cue point annotations are displayed. The form and use of annotations are described more fully in the section herein entitled ANNOTATIONS and in the Appendix.
  • Video linking logic 110 is further configured to search for an existing cue point metadata file that may have been created in an earlier user session with video linking logic 110. If an existing cue point metadata file is found, then cue point data is loaded and the video linking logic displays, in cue point data panel 208, cue point data for the first cue point in list 206 that was automatically selected and highlighted.
  • Cue point data 208 is configured to receive user input specifying one or more metadata values relating to a particular link or transfer of control associated with one of the cue points in cue point list 206 that is currently selected or highlighted in the cue point list.
  • a user may operate a pointing device such as a mouse or trackball to select other cue points in list 206 and in response to selection of a different cue point the video linking logic 110 automatically updates cue point data panel 208 to display cue point metadata for the newly selected cue point.
  • Cue point data panel 208 comprises a cue name field 224 and cue time field 226 that reproduce the data shown in cue point list 206 for a selected cue point.
  • Cue point data panel 208 comprises a cue type combo box 228. Particular types of cue points are described further in other sections below.
  • Cue point data panel 208 is context-sensitive so that the particular fields displayed as part of the panel will vary according to the value of the cue type combo box 228 and a content type combo box 230.
  • the cue point data 208 comprises an interest URL field 232, query string field 234, story text field 236 and story picture field 238 as shown in the example of FIG. 2.
  • the interest URL field 232 is configured to receive user input specifying a website or other URL to which a viewer may be directed at playback time in response to receiving input indicating interest in other information relating to the video.
  • the query string field 234 is configured to receive user input specifying a search engine query string which, at playback time, the metadata-capable video player logic 112 may submit to an Internet search engine for the purpose of generating search results in which a viewer may have interest, or that relate to the video.
  • the story text field 236 is configured to receive user input specifying a story to display to a viewer using the player logic 112 at the time the video is played.
  • the story picture field 238 is configured to receive user input specifying a graphics file or still image, and a text string, to display to the viewer using the player logic 112 at the time the video is played.
  • FIG. 3 graphically illustrates an example video linking arrangement that can be configured using the mechanisms now described. For purposes of illustrating a clear example, FIG. 3 describes relatively few video segments and cue points; in a practical embodiment the techniques herein can be used to create video projects having any number of video segments and cue points.
  • the example of FIG. 3 represents a non-linear video program in which the viewer arrives at a choice point and selects one of three possible videos; at the end of the selected video, the video project continues with the program.
  • the video project comprises a first program video segment 302 having a start cue point 312 and ending in a modal story branch cue point 314, which is configured in video linking editor logic 110 to permit an N- way branch to other video or content but in the example of FIG. 3 is configured as a three-way branch.
  • a first branch leads to a first video advertisement 306 relating to hair care products.
  • a second branch leads to a second advertisement 308 relating to face products.
  • a third branch leads to a second program video segment 310.
  • a user activates video editor 106 and authors a video project that includes segments 302, 310 and advertisements 306, 308 in the same video file.
  • the user creates and stores a Flash navigation-type cue point with a name at a plurality of locations in the video file.
  • FIG. 4 illustrates a screen display in the Adobe Premiere video editor in which a video file has been created with the segments and advertisements and appropriate cue points. After creating the cue points, the user saves the video project in Premiere and encodes the video.
  • the user then activates video linking editor logic 110, and in response, the user interface of FIG. 2 is displayed.
  • FIG. 5 illustrates a portion of screen display 200 showing cue point list 206 for the video of FIG. 3, FIG. 4. Assume the user selects the Start cue point.
  • video linking editor logic displays metadata associated with that cue point in the metadata panel 208.
  • FIG. 6 illustrates the metadata panel populated with data for the Start cue point of the example. The user may edit the values in the metadata panel by selecting fields and entering new values, or selecting pull-down menus.
  • FIG. 7 illustrates the cue point data 308 configured with values from user input that create such a cue point.
  • a name may be entered in the Cue Name field.
  • the Cue Time field is not modified and shows the value obtained from the video file.
  • the cue type is selected as "modalStoryBranch.”
  • a branch cue type is associated with no content, so the Content Type field is grayed out.
  • a Targets list identifies possible destinations or targets to which control is transferred at the branch point.
  • a Background Picture field and Text field receive an identification of a picture to display to the user in a background area while the user is determining which selection to make, and a text string that can serve as a prompt.
  • FIG. 8 illustrates a display generated at playback time based on the metadata that has been created in the present example.
  • the user may create program wide metadata by entering values in the CPL metadata panel 204.
  • FIG. 9 illustrates appropriate values of program-wide metadata for the present example.
  • the user may then save the metadata using a Save function in the File menu 205 of the screen display 200.
  • selecting the Save function causes the video linking editor logic 110 to create and store an XML file containing the metadata and to store the XML file in a same directory or other storage location as the video file that is referenced in the metadata.
  • multiple cue point metadata files may reference and may be associated with a single video file.
  • the user may invoke the video linking editor logic 110, reload the metadata file, modify the cue points, save an updated metadata file with modified cue point data, and replay the video based on the updated metadata file.
  • Such updates may be performed without re-encoding the video, because the video file is maintained entirely separate from the metadata file.
  • a "goto" cue point may be defined and at playback, the goto cue point causes a jump to another video segment when the play head reaches the cue point.
  • the destination location for the jump is defined in a cue point group ("cpGroup") and discussed below.
  • a goto cue point has the following associated metadata:
  • a goto cue point has the following elements of interest:
  • An example script code excerpt including a goto cue point, which may be included in a metadata file, is:
  • a gotoAutoReturnButton cue point supports a mechanism for the user to obtain more information relating to a particular video. From the gotoAutoReturnButton until the next cue point, the player causes the video to be overlaid with a graphical button; user input selecting the button causes the player to perform a goto branch operation to reach another cue point with an automatic return. In an automatic return, at the end of the "more information" video segment, the player causes the playhead to jump back, reaching the beginning of a video segment that just fallows the end of the calling video segment.
  • a first video segment might comprise a 30-second automobile commercial; 10 seconds into it, the user selects the "more info” button, jumps to a 5 -minute extended commercial about the car; and at the end of the extended commercial the player jumps back to the programming that followed the original
  • the "gotoAutoReturnButton" cue point comprises the following attributes:
  • an example script code segment comprises:
  • an insertPt may be used to include one of several pieces of media at a certain point in time. A selection of one of the pieces of media is made by a call to a web service. When reached, the cue point at the end of a target piece of media determines what happens next. The cue point at the end may comprise a returnEnd, goto or progEnd cue point.
  • the group of media points is specified as a cpGroup.
  • the cpGroup must have uniform endings for the segments it contains. For example, every cue point identifies a contain segments (or compound segments) and every segment implies an ending cue point. For a cpGroup, all of the ending cue points are either goto cue points, returnEnd cue points, or progEnd cue points, or a mixture of these types of segment ending cue points.
  • the player when the player reaches an insertPt, the player invokes the web service specified in the progLevelMetadata element described below with an operation specified with the cpGroup. The result of this call is used to select which media to display (the "target").
  • the player calls a web service to obtain the user's zip code. Based on the received zip code value, the player selects from among Bronco, Escalade and Hummer commercials.
  • the cpGroup is stated in script code as:
  • the cpGroup is a table of targets in which an association attribute configured as a string is linked to a media point. The result of the web service call, a string, is tested against the association values until a match is found, and the first match is used.
  • the insertPt cue point has the following metadata attributes:
  • the cue point has the following elements:
  • a modal story branch cue point causes the player to pause the video and to present the user with an n-way branch. The user selects an image representing the cue point to go to that cue point.
  • the cue points can be either video or web points. The type for the cue points at the end of the targets are (for video) is goto or progEnd.
  • the cue point has the following attributes:
  • the cue point has the following elements:
  • FIG. 8 illustrates an example screen display resulting from the use of a modal story branch cue point and example script code follows.
  • an MXML (Macromedia extensible Markup Language) overlay cue point allows use of a web development tool to define overlays with web-aware bindings.
  • An example development tool is Flex from Adobe Systems, Inc., San Jose, California. Flex provides for content layout and code within an asynchronous architecture.
  • MXML code is passed to the player via the mxmllnCPL element in the cue point. The code is executed to make the UI element overlays.
  • the metadata-capable video player logic 112 is configured to read a MXML user interface markup language script from the metadata file, parse and interpret the MXML script, and generate and display one or more overlay graphical elements in the video window of the player GUI based on the parsing and interpreting.
  • User interaction is processed using a web service that is specified in the progLevelMetadata attribute.
  • User interaction with each component is handled by invoking an operation within the web service named "on" concatenated with the id property of the component.
  • the operation is called with the data relevant to the component.
  • tags that descend from UIComponent and RadioButtonGroup within Flex are used. MXML authoring is further described in Adobe developer network documents relating to Flex.
  • the cue point has the following attributes:
  • the cue point has the following elements:
  • progEnd end returnEnd cue points define the end of a video segment and upon reaching the cue points, the player stops playing video and does not provide a rewind option. There can multiple progEnd' s in a media program.
  • the returnEnd cue point is used at the end of a segment. Reaching a returnEnd causes a jump to the point that initiated the jump to the start of the segment.
  • the returnEnd and progEnd cue points have the following attributes:
  • the cue point has the following elements:
  • a webFocus cue point can specify a URL for a web point and, with a story element, associate an image and text (e.g., for a call out) with the web URL.
  • webFocus cue points can be used as targets in modalStoryBranch cue points and insertPt cue points.
  • webFocus cue points can appear in directories.
  • webFocus cue points can have a gotoWebPointEndName attribute value to specify what to show at the "end" of a webFocus.
  • a user indicates that the user is at the "end" of a webFocus by selecting a "back" browser function or by selecting a "TV" button. If the video media is in an operational state, the player switches to the video, maintaining the playhead time and play/pause status. If the video is not in an operational state because, for example, a zeroLen cue point has been reached, the player executes a goto to the media point specified by the gotoWebPointEndName.
  • the cue point has the following attributes:
  • the cue point has the following elements:
  • cuePt elements have the following attributes:
  • a cue point has the following contained elements:
  • a cpGroup is used anywhere a group of cue points is needed.
  • the cpGroups are made up of some attributes and a collection of targetSeg elements.
  • a targetSeg contains a cue point name and an optional association attribute.
  • Some cue points for example insertPt, use cpGroups where each cue point in the cpGroup has an association that is used to select the cue points.
  • the player searches the table to match the string provided as a key with the association attribute and then returns the cue point name contained in the first match.
  • the key may come from a web service as explained in the insertPt cue point section.
  • targetSeg' s in a cpGroup As an example, with the following targetSeg' s in a cpGroup:
  • the cpGroup has the following attributes:
  • the cpGroup has the following contained elements:
  • a targetSeg may be used to specify a cue point name and optionally associate it with an association attribute.
  • a group of cue points e.g., a modalStoryBlock where the user makes the choice
  • the association attribute can be omitted.
  • the association attribute is needed to determine which cue point to goto.
  • the association attribute can be of the form "aaa, bbb, ccc" where each substring would match the cuePointName. See cpGroup for an explanation of how the association attribute is used to select a cuePointName.
  • the targetSeg has the following elements:
  • a mxmllnCPL element may be used to hold executable MXML code.
  • a user can include anything that descends from the UIComponent in mx:MXML.
  • a progLevelMetadata element is required. It contains data associated with the overall program.
  • Example attributes include:
  • Example contained elements include:
  • a story element packages data used to display a cue point (web or video).
  • Example attributes include:
  • a gotoAutoReturnButton element and cue point support a "more info” operation.
  • Example attributes include:
  • an annotation element is used to display a graphic on screen.
  • the graphic can be actionable.
  • An annotation element is a graphic object that appears on screen starting at a cue point; when the next cue point is processed the annotation is removed (although it could be reapplied). It is used to overlay the display with a graphic while in video view and may optionally implement a goto behavior in response to a click.
  • the structure of the annotation element is similar to cuePoints in that it contains the same targetList and story elements. Clicks on an annotation can cause three things to happen, depending on the value of the clickBehavior attribute. See the attribute description.
  • Example attributes include:
  • an audiovisual work may include one or more annotations that specify interactions available to a viewer.
  • Annotations may comprise graphical images, buttons, text messages, labels, and other elements that may be displayed in a variety of locations overlaid on a video segment or near a video player window that is showing a video segment.
  • One or more annotations may be assigned to a cue point; when the cue point is reached during playing, the annotations are activated and remain active until the next cue point.
  • Annotations have flexible attributes relating to where they can be shown, what they can show, and how they behave.
  • Graphic images associated with annotations may include images such as PNG and JPEG files, or SWF files or any other files that can be rendered on the system on which the player logic 112 is hosted.
  • an annotation has one of four types: decoration; goto; returnEnd; and overlay. (Details of annotation types are disclosed in the Appendix.) Annotations may be displayed as static graphical images or animated graphics. Annotations may be positioned anywhere in the video windows that the player logic displays during playing. [0180] In an embodiment, annotation frames allow placement of an annotation outside of the video window; an annotation frame can provide a larger area outside the video in which annotations can appear, without covering up the video. In an embodiment, a user may use the video linking editor logic 112 to define an annotation frame as a rectangle within which the video window is placed. If the annotation frame is larger than the video frame, then space is displayed around the video and annotations can be placed in the resulting space without obscuring the video. With annotation frames, an author is not required to re-encode a video segment to create space to place annotations.
  • a "goto" annotation may be associated with a target and one of several different kinds of return behavior; a target specifies where the player branches when a viewer clicks on the annotation, and the return behavior specifies where the viewer returns after viewing the video or web page associated with the annotation.
  • the return behavior of a goto annotation may be set to "Skip.” With skip on return behavior, after a viewer returns from the annotation's target video segment or web point, the player skips to the next cue point after the one that includes the goto annotation.
  • An annotation of any type may be configured with modal behavior.
  • a modal cue point has two different playback modes comprising an initial entry mode and a return or overlay completion mode.
  • each annotation type causes the video player logic 112 to operate differently depending upon the then-current mode, as defined by how the player arrived at the associated cue point.
  • initial entry mode refers to the player arriving at the cue point via normal program flow, or as the result of a direct jump.
  • the video player logic 112 is configured to display all annotations that are configured as modal, pause, and wait for the user to select a non-decoration annotation, such as a "goto" annotation or a "returnEnd” annotation.
  • return or overlay completion model occurs when the player returns to the cue point via a returnEnd cue point or annotation after a jump from it, or when a viewer selects the Continue button to close an overlay data entry form.
  • FIG. 23 is a screen display diagram of an example Editor window in which an Annotation tab is selected.
  • screen display 2102 includes Annotation tab 2302.
  • Selecting an Add Annotation (+) control causes the editor logic 110 to display a default annotation name and type in fields 2304; user input may modify the annotation name and type, so that the annotation may be referenced in other metadata using a convenient name.
  • Editor logic 110 also displays a data entry panel 2306 that may receive values defining particular parameters of an annotation including screen position values (X position, Y position). Parameters also may include a text label for the annotation, a graphic image to display as the visible form of the annotation in the player, and mouse over image.
  • the mouse over image is a different graphic image to display if a user moves a pointing device over the annotation while using the player and when the annotation is displayed.
  • a Boolean parameter selected in the editor using a checkbox may specify whether to display an annotation icon.
  • a content type value associated in metadata with a cue point causes differentiated operation of the metadata-capable video player logic 112 at the time of playback.
  • the content type zeroLen is treated differently than all others (ad_Inline, segment_Inline, prog_Inline, prog_Segment).
  • ad_Inline and ad_Segment are used to skip advertising content coming back from an insertPt.
  • a computer program can create one or more cue points and store the cue points in a metadata file, rather than a user obtaining cue points from encoded video, or the user creating the cue points using the video linking editor logic 110.
  • cue points can be added, updated or completely replaced dynamically using web applications, processes, or other computers that are coupled to computer 102.
  • the unique identifier of a television program as specified by Society of Cable Telecommunications Engineers, could be used in an update message providing new cut points.
  • one or more computer programs can access video and other content databases and use the information gather to generate interactive video experiences based on the cue point language schema that is defined herein.
  • a Perl script may be configured to access YouTube metadata APIs to construct an interactive video experience based on playing all video matching a particular keyword.
  • the script may be configured to issue an HTTP-based query to a YouTube server, in which the query conforms to YouTube' s APIs, to retrieve a list of all stored videos that include a particular keyword in the metadata maintained by YouTube for the stored videos.
  • the YouTube server sends a responsive dataset.
  • the script may be configured to identify a URL for each video on the YouTube servers that is identified in the responsive dataset, and to write a metadata file 126 that specifies an audiovisual program consisting of a concatenation of all the matching videos.
  • the script could be configured to automatically generate a plurality of annotations, in which each annotation graphically represents a first frame of a different one of the matching videos. In this manner, at playback the user would see a visual menu of each matching video and could activate any desired video by selecting on the image associated with an annotation for one of the videos.
  • a program is configured to receive a user query for a particular keyword or phrase and to search a database of movie metadata for matches to the user query. For each match to the user query, an associated database record is selected and retrieved. From each database record, the program retrieves a URL of a video that is stored in third party hosted storage, such as YouTube. The program creates and stores a metadata file 126 that plays the matching videos. For example, the program could be configured to receive a user query to find all video clips in which a character says "Bond, James Bond", assuming such phrases are represented in the database of movie metadata.
  • a computer program may be configured to create multiple metadata files 126 based on a single video.
  • a Perl script may be configured to generate multiple versions metadata files 126 for a single video in which each metadata file 126 comprises definitions of annotations for subtitle data in a different language, and the subtitle data is displayed at playing time using the annotations as the subtitle display widget.
  • automatic creation of cue points may take user behavior into account to create customized cue points for a particular user based upon what is known about the user's behavior as represented in server-side stored data. User behavior can include information what previous cue points have been selected, the elapsed time between selections, whether certain video segments have been skipped, navigation paths as represented by user selections of different video segments in succession, etc.
  • embodiments provide flexible means to use output from a database, coupled to a script or other program, wherein the output is optionally selected based on matching user input or queries, to result in automatically creating and storing one or more metadata files 126 which, when played using the video player logic 112, result in displaying enriched interactive videos.
  • the program may cause displaying a concatenation of videos matching a query, concatenation is not required.
  • a program or script may have any level of complexity and may be configured to write a metadata file consisting of any number of cue points, annotations, or other information based upon the language description that is provided herein.
  • metadata may be created dynamically and transmitted to the player over a network connection without storing or saving the metadata in file format.
  • the examples provided herein are merely representative and countless other applications are possible.
  • a directory comprises, in one embodiment, a selectable, scrollable column on the right part of the video display that appears at cue point boundaries and for a specified period of time, such as four (4) seconds, in response to a movement of a pointing device.
  • FIG. 10 illustrates an example screen display that includes a directory.
  • Player logic 112 attempts to generate and display a cue point specific, non-modal directory on a cue point by cue point basis.
  • the media points (video and web) within the directory are specified as a cpGroup and must contain story elements if they are to appear in the directory. These points can be whatever the author chooses to make them and are an opportunity to guide the user into interesting, tangentially related information. For example, in a news show, when a story about Great Britain is shown the directory could contain the related online encyclopedia entry and several video segments; when the news program shifts to the next story, the cue point specific directory changes.
  • web services may be implemented using a ColdFusion web server.
  • web services are called with two string arguments comprising the called operation or function and the type of service.
  • the web service returns a string with three fields comprising an operation specific field (e.g., "serviced" for MXMLOverlay calls), a result, and the type of service string.
  • an author may customize the positioning of a video window and a web window within an overall player window.
  • dynamic layout is accomplished through user interaction with the video linking editor logic 110.
  • FIG. 21 illustrates an example screen display that the video linking editor logic generates and causes displaying.
  • An author selects a window size for the video to be displayed as part of an enriched video program. For example, a window size may be
  • a user selects a new layout control to create a new layout and assigns a unique name to the new layout.
  • the author selects a Positioning function and may select one of a plurality of predetermined layouts of the video window, web window, and static surrounding graphical display space.
  • the user may change the size of the video window or web video using an Advanced tab function.
  • the user may change dimensions in pixels for video width, video left position, video horizontal center position, and video right position.
  • the editor logic stores the changed values in association with the layout name. Changing numeric values of dimensions later results in changing the position of a video window when displayed using the player logic.
  • Each layout may have restrictions on repositioning based on the original layout; in an embodiment, the editor logic 110 prevents the user from entering data for parameters that do not fit a particular layout.
  • FIG. 25 is a screen display diagram of an example Editor window in which a Layout tab is selected.
  • example Editor window 2102 comprises a Layout tab 2502 that displays a list of names of selected layouts. Selecting an Add Layout control 2503 causes the editor logic 110 to add a new layout name to the list.
  • logic 110 can access stored data defining a plurality of predefined player window layouts, which are displayed in an information panel 2504. In each predefined layout, a relative size and position of a video window to be shown in the player window is indicated by a rectangle having a first color, and a relative size and position of a web browser window to be shown in the player window is indicated by a rectangle having a different, second color. In some layouts the video window has a reduced size as compared to a size of the browser window.
  • the video window is the same size as the browser window.
  • a background is defined that is logically behind or surrounds both the video window and browser window.
  • the video window is laterally or longitudinally adjacent to the browser window.
  • the video window is offset in a corner of the browser window, or centered.
  • selecting a predefined layout from panel 2504 causes editor logic
  • the author 110 to display an enlarged view 2510 of the selected layout in which the relative size and position of the browser window 2508 and video window 2506 are shown.
  • the author also can further customize the layout to obtain different effects using the parameters accessible using an
  • a layout may be linked to a particular cue point.
  • a user selects a
  • the user may select a Description tab in a Parameters pane and select a Browse button next to the Web View
  • Layout and the user may select the Layout that the user created.
  • an author has control over the location of a video window and web window. Further, a particular layout that the author deems aesthetically preferable for a particular combination of video and web content may be injected into the metadata so that the layout changes appropriately when a particular cue point is reached.
  • TABLE 1 presents an example of a complete metadata file of the type that can be created and stored as one of the metadata files 126.
  • the base element is MediaProgram and encloses all other elements.
  • the element progLevelMetadata is required and specifies information that applies to the whole MediaProgram.
  • the cueType is "insertPt" which jumps to a cue point (in this case D) while establishing a return point.
  • the target segment is specified within a cpGroup (a cue point group); in this case it has only one target and the association attribute is "default”. There is nothing to check and there is only one place to jump.
  • TABLE 2 presents an example schema for the Cue Point Language.
  • FIG. 11 illustrates an example screen display that illustrates a player screen that may be generated and displayed in a computer display unit by metadata-capable video player logic.
  • the video display unit may comprise a computer monitor, video monitor, digital TV, CRT, or other display unit that is driven from an appropriate output of computer 102.
  • the player screen display may be implemented as an application that is displayable within a web browser, or using standalone application program logic; in either case the player is not video specific and will work with various existing video formats (Flash, Silverlight, QuickTime, etc.) and can be adapted to new video formats as they are defined.
  • various existing video formats Flash, Silverlight, QuickTime, etc.
  • FIG. 20 illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments with a browser-based player for enriched video programs.
  • a computer 102 is coupled directly or indirectly through one or more networks 120 to a web server 130 and optionally to a file server 132.
  • network 120 may comprise a local area network (LAN), wide area network (WAN), an internetwork, or a combination.
  • Web server 130 hosts one or more video files, HTML documents, HTTP servers or application servers, or other web content.
  • File server 132 stores or hosts video files 122, graphics files 124, and metadata files 126, which may include or be associated with HTML and browser executable program code, such as JavaScript code.
  • file server 132 stores or hosts script files 2008 that can issue queries to a database 2010 and automatically generate the contents of one or more metadata files 126 based on result sets received from the database in response to the queries.
  • Computer 102 hosts or executes an operating system 104 that supervises I/O, storage management, and execution of application logic.
  • computer 102 optionally comprises a video editor 106; as indicated by broken lines, the video editor may be omitted.
  • computer 102 comprises a browser 108 that hosts or can access a support library 2002.
  • support library 2002 Commercially available examples of support library 2002 include Macromedia Flash and Silverlight.
  • computer 102 is coupled to storage 140, which broadly represents any data storage device, storage area network (SAN), network attached storage (NAS), or network file system (NFS) unit or server.
  • Storage 140 may reside on network 120 or on a server coupled to the network.
  • Storage 140 stores application programs but is not required to store video files or metadata files; instead, video files may be received through streaming video delivery from file server 132 and metadata files 126 may be received on the fly directly to browser 108 or support library 2002 under control of an instance of metadata-capable video player logic 112.
  • computer 102 optionally comprises video linking editor logic 110, which may be omitted entirely as indicated by broken lines.
  • a separate player control server 2004 comprises metadata-capable video player logic 112 and may comprise accounting logic 2006.
  • the metadata-capable video player logic 112 is generally configured to open metadata files and associated video files, and to play the video files while interpreting and responding to links and related information and instructions in the associated metadata files. Other more specific functions of metadata-capable video player logic 112 are described in other sections herein.
  • player control server 2004 controls delivery of instances of the player logic 112 to authorized clients, and in certain embodiments interactions with accounting logic 2006 determine whether a particular client in the form of computer 102 can receive an instance of the player logic. Additionally or alternatively, accounting logic 2006 determines amounts for invoices, other billing, or other charges to a video producer, studio, content owner, or other party that owns or controls the file server 132 and its contents.
  • computer 102 comprises player logic 112 and does not have an editor such as editor logic 110; such an embodiment might be used by an end user who is viewing video programs that have been prepared by someone else, and who does not use a browser to view video programs based on receiving the player logic over a network from a server computer as described above.
  • editor logic 110 such an embodiment might be used by an end user who is viewing video programs that have been prepared by someone else, and who does not use a browser to view video programs based on receiving the player logic over a network from a server computer as described above.
  • an end user or viewer invokes browser 108 and connects to web server 130, which offers links to play audiovisual media such as video files 122.
  • the viewer selects a link for a particular video file 122.
  • the browser 108 downloads from the file server 132 one or more elements of HTML and browser executable program code, such as JavaScript code, which the browser executes. Consequently, the browser 108 renders a page in the display unit of computer 102.
  • the rendered page includes code to invoke an instance of metadata-capable video player logic 112.
  • the player logic 112 accesses one or more metadata files 126, and accesses video files 122.
  • the video files 122 may be on file server 132, or stored in another networked location, or on a third party server or quasi-public hosting site such as YouTube. Based on instructions in the associated metadata files 126, the player logic 112 then streams the video files 122 and provides metadata from metadata files 126 to the support library 2002 of browser 108. As a result, one or more of the player screen displays described herein appears and can play video within the browser 108 in the manner described herein. [0218] In an embodiment, each time that browser 108 invokes use of the player logic 112 data is recorded at the player control server 2004, or at a third party server site, to indicate the invocation.
  • Invocation data may include data identifying a referring web site, that is, the web site at which the end user selected a video for playing, such as web server 130. Invocation data also may identify a producer of the video, if the producer is different than the owner or operator of the referring web site.
  • the end user of computer 102 may be denoted a first party; a second party may own or operate web server 130 at which the first party selects videos for playing, and the second party may comprise a producer of the videos; and a third party may owner or operate the player control server 2004 and may control delivery and use of instances of the player logic 112, and may be entitled to payment from the second party for each use of the player by the first party or each stream that the player facilitates delivering from the second party to the first party.
  • a copy of the player logic 112 or other browser executable code may be delivered from the third party to first party browsers only a specified maximum number of times per day, week, month or year in consideration for payment of a specified fee attributable to each day, week, month or year.
  • the third party may cease providing additional first parties with access to the browser executable code that implements or accesses the player.
  • the third party may deliver the browser executable code to an unlimited number of first parties who select videos at the second party's web site and may invoice the second party for a variable amount that is based upon or proportional to the actual number of first parties.
  • the invocation data is recorded in a database that is owned or operated by the third party.
  • the third party configures one or more computer programs to periodically analyze or compile invoicing data from the database, based on the number of streams that the second party delivered using the player or the number of first parties who connected and used the player. Based on the data analysis or compilation, the third party may invoice the second party.
  • the third party retains control over use of the metadata-capable video player logic 112 and its use by second party producers or first party end users, and the third party is entitled to collect fees or revenue from one or more of the second party and/or the first party in consideration for the use of the metadata-capable video player logic 112 to provide enriched videos to end users.
  • computer 102, logic 112, and a video display unit may form a special-purpose computer performing the functions described herein.
  • a player as in FIG. 11 comprises a video display window 1102, trick play icons 1104, a timeline 1106, and a web hyperlink 1108.
  • the video display window 1102 displays a video segment of a media program.
  • the trick play icons 1104 may be selected through user input from a pointing device or remote control.
  • trick play icons 1104 provide functions for video playback, fast forward at one or more speeds, and rewind at one or more speeds.
  • Other controls may be provided including an end playback or "eject" control, an audio volume adjustment control, and a video window size control.
  • the timeline 1106 provides a graphical indication of the player's current position within a video segment, the position of cue points, and the relationship of branches to other cue points and other video segments.
  • the timeline 1106 graphically displays cue points as dots or circles, branches to other cue points as arcs, and video segments as straight lines.
  • the lines, dots, and arcs are arranged in a temporal order so that the first video segment is arranged at the far left side of the display and the last cue point of the last video segment to which a branch can occur is displayed at the far right.
  • a graphical icon in the timeline 1106 moves from left to right in proportion to the time that has elapsed during playback or the amount of video that has been played.
  • the player logic 112 modifies the video display unit to update the timeline to indicate a user's current logical position in a media program as a physical icon shown among the lines, arcs and dots. Therefore, the timeline 1106 enables a user to visually identify upcoming cue points, branches, and branch destinations.
  • web hyperlink 1108 is continuously displayed in the screen display in an overlay manner over any video program that is shown in video window 1102. Thus, the web hyperlink 1108 is always available during operation of the player logic 112.
  • selecting the web hyperlink 1108 causes the player logic 112 to modify the display unit so that the video display window 1102 is redisplayed in a reduced size format, for example, in a small rectangular window at the bottom right corner of the screen display. Further, the video display window is overlaid on a web browser window that displays web content associated with the web hyperlink 1108. In this manner, the player logic 112 appears to generate a picture-in-picture form of display in which the background picture shows web content and the foreground, reduced size picture shows the video program. The video program continually runs during such a transition.
  • the screen display of FIG. 11 further comprises a TV button which, when selected, causes the player logic 112 to restore the video display window 1102 in a large size as seen in FIG. 11 and to discontinue displaying web content.
  • a TV button which, when selected, causes the player logic 112 to restore the video display window 1102 in a large size as seen in FIG. 11 and to discontinue displaying web content.
  • computer 102 uses either a remote control or a computer keyboard to provide user input to the metadata-capable video player logic 112.
  • user input selecting hot keys on the keyboard results in controlling playback.
  • the following key commands cause the metadata- capable video player logic 112 to perform the following functions:
  • FIG. 13A illustrates an annotation coupled to a web service providing automated text messaging in association with an enriched video program.
  • metadata-capable video player logic 112 displays a player window 1302 on a computer desktop or within a browser.
  • Player logic 112 is configured to generate the player window 1302 and to facilitate the functions that are further described herein for FIG. 13A.
  • Player window 1302 includes playing, segment, and chapter navigation buttons 1310 which when selected cause playing a video segment, performing trick play functions, or skipping to other defined chapters.
  • buttons 1310 may be associated with an HTML document that applies a specialized appearance or skin to the buttons 1310.
  • skinning buttons 1310 is performed using the editor logic 112 to display editor window 2102 (FIG. 21), selecting the Metadata tab 2112, selecting a Skin Buttons field and entering an HTML URL.
  • buttons 1310 may have a different appearance in different videos at playback; for example, comparing FIG. 13A, FIG. 17A shows buttons 1310 with different styles and appearance.
  • Player window 1302 includes an audio icon 1312 which when selected causes muting sound from the video and a full screen icon 1314 which when selected causes displaying the video in full screen mode.
  • metadata-capable video player logic 112 causes displaying an annotation 1300 that prompts a viewer to enter a viewer's name, phone number, and gender in data entry fields 1304, 1306, and using radio buttons.
  • metadata-capable video player logic 112 when a viewer enters values in the fields and selects the Go button, temporarily stores the values in memory for referencing and use by other logic when a particular cue point is reached that calls for invoking a text messaging function.
  • FIG. 13B illustrates a frame of an enriched video program as displayed in a player window.
  • player window 1302 as previously seen in FIG. 13A is displaying a video segment depicting a character 1320 who is using a text messaging device 1322.
  • player window 1302 further comprises show and character icons 1324, web site icons 1326, and service icons 1328.
  • a cue point associated with an invocation of a web service may be defined for a time point of the frame illustrated in FIG. 13B.
  • the metadata-capable video player logic 112 is configured to invoke a web service that can retrieve the stored value of the phone number that was received as user input at FIG. 13 A, and dispatch a specified text message to that phone number.
  • the specified text message may comprise information appearing to come from character 1320.
  • the video editor linking logic 110 may be used to define the cue point that can cause a specified text message to be sent automatically when the cue point is reached during playback.
  • show and character icons 1324 each comprise a graphical image that is associated with an annotation.
  • a first one of the show and character icons 1324 is an annotation associated with a URL for a web site of the show, which in the example of FIG. 13B is the MTV show "The Hills," that provides further information about the show.
  • second and third ones of the show and character icons 1324 each comprise annotations that are associated with sequences of video segments relating to the characters that are depicted in the icons. In the example of FIG.
  • selecting the "Heidi” icon causes the metadata-capable video player logic 112 to branch within the associated metadata file 126 to a point associated with a sequence of video segments that feature the character "Heidi.” Playing the video program then continues with the sequence of segments that feature "Heidi.”
  • selecting the "Audrina” icon causes the metadata-capable video player logic 112 to branch within the associated metadata file 126 to a point associated with a sequence of video segments that feature the character "Audrina.”
  • web site icons 1326 provide linkages to Internet sites that feature social networking and other services.
  • the video linking editor logic 110 may be used to create an annotation, symbolized by a Twitter icon, which is associated with the Twitter service and a web service to invoke the Twitter service.
  • the metadata-capable video player logic 112 at playing time, when a viewer selects the Twitter icon, the metadata-capable video player logic 112 generates and displays a new window that contains a feed of Twitter posts relating to the video program of FIG. 13B.
  • the other web site icons 1326 similarly each comprise an annotation that is associated in metadata files 126 with a web service, URL or other reference to executable code that can cause integration and use of the web service that is represented by the icon.
  • each of the service icons 1328 is an annotation represented by a graphic image that provides access to an external service or web site.
  • a music purchase icon may comprise an annotation that is associated with a web site that features downloads of songs, as further described herein for FIG. 14.
  • a commercial sponsor icon may comprise an annotation that is associated with a commercial advertising web site or online information about a commercial product.
  • the target of an annotation that is displayed as a commercial sponsor icon may be a video program segment comprising a commercial for a specified product.
  • selecting the Dos Equis service icon causes the metadata-capable video player logic 112 to branch to and play a video segment containing a commercial for Dos Equis brand beer.
  • FIG. 14 illustrates a frame of a video program having a highlighted service icon.
  • video window 1302 is displaying a frame 1402 of a program that includes background music at the time of playback.
  • a first service icon 1404 comprises an annotation that is associated with a highlighted graphic image as indicated by short lines radiating from the icon; in contrast, in the example of FIG. 13B, the same one of the service icons 1328 is not highlighted.
  • the highlighted icon signifies that the song that is then currently playing in the background of the scene of frame 1402 is available for purchase or downloading.
  • the metadata-capable video player logic 112 accesses and displays a web site that offers the associated song for downloading or purchase.
  • an author may use video linking editor logic 110 to associate a specified web service, URL, or program code with an annotation and graphic image for the service icons 1328.
  • the URL may be a complex URL that includes a domain name, service name or script name, and one or more embedded parameters or attributes. For example, an attribute of the URL may be set equal to a file name for the song that is known to play at the associated cue point.
  • selecting the first service icon 1404 causes the metadata-capable video player logic to invoke the URL specified in an associated annotation, effectively passing the name of the then-currently playing song to a third party web site, which extracts the song name and can offer the song identified in the parameter for purchase or downloading.
  • FIG. 15A illustrates an annotation that provides a user choice.
  • video window 1302 displays a plurality of annotations 1502 in a video window 1506.
  • First, second, and third annotations labeled Heidi, Spencer and Audrina are associated with static graphic images of the named characters and are linked to a target cue point for a sequence of video segments that feature the associated character.
  • a fourth annotation comprises a graphical image prompting the user to select one of the characters as a favorite character.
  • metadata-capable video player logic 112 branches within the metadata files 126 to instructions associated with playing a sequence of video segments that feature the selected character. For example, if Spencer is selected then the metadata-capable video player logic 112 branches and begins playing a first segment of video featuring the Spencer character, as represented by the frame of FIG. 13B.
  • the video segments featuring that particular character are also authored to include annotations identifying the other, non-selected characters, for possible future selection.
  • the show and character icons 1324 display only icons for annotations associated with other characters, namely Audrina and Heidi.
  • FIG. 15B illustrates a frame of a video segment in a sequence for which Audrina is the featured character; therefore, show and character icons 1324 depict Heidi and Spencer, but not Audrina, and the icons are associated with annotations which, when selected, cause playing sequences of video segments featuring Heidi or Spencer, respectively.
  • FIG. 15B also illustrates different service icons 1328 in which a third service icon is associated with a different commercial product or retailer.
  • an author using video linking editor logic 110 may define different annotations in the position of service icons 1328 for different commercial products, merchants, retailers, or other web sites or service providers in association with different cue points arising at different time points in a program.
  • an annotation associated with a graphic image or icon depicting a first merchant or product may be associated with a cue point at the start of a first scene of a video program that somehow involves, uses or shows the associated product, which a second merchant, product or service may be associated with a second cue point at another point in the program that shows, uses or involves the second merchant, product or service.
  • FIG. 16 illustrates concurrent playing of an enriched video program and displaying an associated web page.
  • the video player window 1302 comprises a video window 1602 that is overlaid on a browser window 1610.
  • the video window 1602, in the example of FIG. 16, is displayed in reduced size but contains the same navigation icons 1310 as in FIG. 13A.
  • the video window 1602 is configured to play a streaming video program.
  • the metadata-capable video player logic 112 is configured to concurrently cause playing a streaming video program in the video window 1602 and to display a web page 1612 associated with the video.
  • an author using the video linking editor logic 110 may define a cue point at the frame shown in FIG.
  • a cue point may associate the time of that frame with a URL for an Internet Movie Database (IMDB) page containing information for the named actor.
  • IMDB Internet Movie Database
  • the metadata-capable video player logic 112 may reach other cue points referencing other URLs.
  • the metadata- capable video player logic 112 accesses a referenced URL and causes the browser window 1610 to display the referenced web page. In this manner, cue points defined for a video segment can cause web content to be "pushed" to a browser window that underlies the video window. Content in the browser window thus changes dynamically as the video plays and as specified cue points are reached by the player.
  • embodiments provide the capability to display video in a "picture in picture" layout in which a video plays in a video window 1602 that is reduced in size in comparison to a browser window 1610 that is concurrently showing related web content.
  • the metadata-capable video player logic 112 is configured to allow the end user to watch video and scroll web content in the same screen without tabs or special windows.
  • the author of the metadata files 126 for the program has control of whether the video in video window 1602 plays or pauses, and what is rendered in the video window 1602 and the browser window 1610.
  • FIG. 16 shows a first rectangle comprising video window 1602 and a second rectangle comprising browser window 1610. In other embodiments, any number of rectangular display areas for video or browser content may be provided.
  • annotations can be configured so that invoking the Twitter web site icon 1326 causes the metadata-capable video player logic 112 to display a third rectangle to the right of the video window 1602, while maintaining a display of the browser window 1610 conceptually behind the other rectangles or windows.
  • the third rectangle displays a feed of Twitter posts using HTTP data transfers and rendering of HTML within the third rectangle.
  • a streaming video may be played at the same time that an HTML window is dynamically updated.
  • Both the video window 1602 and the browser window 1610 have equal conceptual weight within the player window 1302.
  • FIG. 17A illustrates an example of playing an enriched audiovisual program with annotations that implement chapter selections.
  • Player window 1302 displays a graphical background 1702 that surrounds a video window 1704 that displays a video program, based on an associated metadata file 126.
  • a plurality of enriched program navigation icons 1708 and chapter selection images 1706 are displayed over the video in the video window 1704.
  • each of the enriched program navigation icons 1708 and chapter selection images 1706 is an annotation as described herein, associated with a particular position, graphic image or animation, and operational behavior.
  • enriched program navigation icons 1708 include a Back navigation icon and a Home navigation icon, comprising annotations that associate static graphical images.
  • Selecting the Back navigation icon causes the metadata- capable video player logic 112 to branch to a prior video program that was previously played or a web page that had been previously displayed in a browser window area of the player window 1302.
  • Selecting the Home navigation icon causes the logic 112 to branch to a starting video segment representing a home position of the video program.
  • the chapter selection images 1706 each represent an annotation that is associated with a branch to a different cue point in the video program associated with a different video segment for a chapter, episode, or other discrete video element. During playing, selecting one of the chapter selection images 1706 causes the player logic 112 to branch to and start playing an associated video segment.
  • FIG. 17 A indicates an aspect of the flexibility inherent in the concept of annotations as described herein.
  • Both the icons 1706, 1708 can be represented using annotations that define different positions, graphic images and operational behavior.
  • annotations are different, an author is not required to learn and use a large number of different programming techniques; instead, the same features and functions are used to define all the annotations.
  • FIG.17B illustrates playing an audiovisual program in which annotations have multiple different forms and provide multiple different functions.
  • player window 1302 comprises the graphical background 1702, video window 1704, and enriched program navigation icons 1708 as described for FIG. 17A.
  • FIG. 17B features a navigation animation 1710, web integration icons 1712, topic launch icons 1714, and menu access link 1716.
  • navigation animation 1710 represents an annotation that is associated with an animated graphical object and a plurality of cue points for each of a plurality of characters shown in the animation at different positions within the animation.
  • a viewer can use a pointing device to cause a cursor to hover over the navigation animation 1710, and in response, the navigation animation scrolls graphically left or right in response to movement of the pointing device. Selecting a particular region showing a particular character causes the player logic 112 to branch to a chapter of the video program that features the selected character.
  • the web integration icons 1712 each represent an annotation that is associated with a static graphical image and an interactive operation relating to web content.
  • a Facebook icon represents an annotation that defines a link to a Facebook page for the program shown in the video window 1704.
  • selecting the Facebook icon causes the player logic 112 to redisplay the video window 1704 in smaller form and to access and display a Facebook page for the program in a browser window that is conceptually or logically under the video window 1704.
  • the topic launch icons 1714 represent annotations that define branching behavior to other video program segments relating to topics such as costumes used on the program or show and the history of the show. Additionally or alternatively, one or more of the topic launch icons 1714 may be associated with a web page; thus, selecting one of the topic launch icons can result in either playing a video segment or displaying web content in a browser window under the video window.
  • menu access link 1716 represents an annotation associated with branching behavior that causes the player logic 112 to branch to code in a metadata file 126 that causes displaying a list or menu of a plurality of video program episodes that are collected or associated with a subscription.
  • selecting the menu access link 1716 causes the player logic 112 to display a video window having the form of FIG. 17C.
  • FIG. 17C illustrates a video window providing a menu of episodes in a collection or associated with a subscription. The nature and use of subscriptions for video programs is further described below.
  • video window 1720 comprises a plurality of available episode icons 1722 and a plurality of unavailable icons 1724.
  • “Available,” in this context, means released by a producer or program owner for viewing by viewers who have purchased or otherwise validly accessed a subscription, and "unavailable" means not yet released and normally planned for the future.
  • An available episode icon 1722 represents an annotation that is associated with a static graphical image representing the particular episode, and associated with branching behavior that causes the player logic 112 to play the particular episode in video window 1720, replacing the icons 1722, 1724.
  • An unavailable episode icon 1724 represents an annotation that is associated with a static graphical image, or decoration, representing the particular episode that is unavailable. As decorations, unavailable episode icons 1724 are not selectable and not associated with branching behavior or other action.
  • the graphic images associated with unavailable episode icons 1724 may include an episode name and release date for the purpose of attracting viewer interest in future program material.
  • FIG. 17D illustrates use of annotations to form elements of a main menu page for a video program subscription.
  • a video window 1730 in a player window comprises a plurality of the web integration icons 1712 as previously discussed, located in a different position of the video window 1730.
  • the same annotations may be used to define the web integration icons 1712 as for FIG. 17C, with different values for screen position attributes.
  • video window 1730 further displays a plurality of program link graphics 1732, 1734, which comprise static images each associated with a different animation having different responsive operational behavior.
  • program link graphics 1732 represent annotations that are associated with static graphical images and branching behavior to cause the player logic to branch to metadata in a metadata file 126 that causes playing a bonus episode, or displaying a menu of other annotations having graphics representing short episodes or all available episodes.
  • Program link graphic 1734 represents an annotation which when selected causes branching in the metadata to code that causes the player logic 112 to play a particular video program episode.
  • annotations can be authored and associated with different graphics, branching behavior, and targets, including static graphics and video programs.
  • Annotations also can cause playing pages that consist solely of other annotations, to await selection of one of the annotations to cause other navigation or to cause playing various program materials.
  • FIG. 18 A illustrates an example news program in which annotations may be used to provide a directory or menu of a plurality of news stores, features, segments, or related information.
  • the player window comprises a video window 1804 surrounded by an undecorated background 1802.
  • background 1802 may carry advertisements, program logos, or other information.
  • a plurality of program links 1806 is arranged in a column 1808 at one side of the video window 1804. Each of the program links 1806 is associated with an annotation.
  • Each of the annotations defines a position, graphical image, and behavior in response to selection of the annotation.
  • Each annotation may be associated with a video program or an Internet site, so that selecting a particular annotation causes the player logic 112 to either play the associated video program or to access and display information from the Internet site in a new browser window, depending on the defined responsive behavior.
  • the graphical images may include a blend of images and text to indicate what kind of program or site is associated with the program link 1806.
  • FIG. 18B illustrates the news program of FIG. 18A after a viewer has selected a program link 1806 (FIG. 18A) that is defined using an annotation having an association to a website.
  • FIG. 18A illustrates the news program of FIG. 18A after a viewer has selected a program link 1806 (FIG. 18A) that is defined using an annotation having an association to a website.
  • player logic 112 obtains a URL for a web page from within the code of metadata files 126 that defines the annotation, issues an HTTP request for the URL, and generates a browser window 1810 that renders the resulting HTML.
  • player logic 112 redisplays the video window 1804 in a reduced size within the player window 1800.
  • the column 1808 of program links remains displayed in reduced size within the video window 1804.
  • the browser window 1810 may include a scroll bar 1814 that is movable in response to user input from a pointing device such as a mouse, touchpad or trackball. The scroll bar is scrollable to cause the web page in browser window 1810 to scroll up or down independent of the video window 1804.
  • FIG. 18C illustrates the browser window 1810 of HG. 18B after the scroll bar has been moved. While the content of the web page has moved downward in browser window 1810, the position of video window 1804 remains fixed within the player window 1800. In this approach, the video remains visible and the viewer can retain context for the associated web page content. At any time, the viewer can select a full screen icon 1816, which is also defined using an annotation. In response to selecting the full screen icon 1816, player logic 112 causes the browser window 1810 to close and the video window 1804 is redisplayed to occupy all of the player window 1800.
  • the browser window 1810 may comprise a plurality of browser navigation buttons 1818.
  • the browser navigation buttons 1818 include forward, backward, and magnification buttons. Selecting a backward navigation button causes the player logic 112 to redisplay, in the browser window 1810, a most recently displayed previous web page. If the most recently displayed previous web page was generated when the player logic 112 was playing a different video program, then it is possible that using the backward navigation button may cause displaying a web site that is unrelated to the current video program.
  • FIG. 19 A illustrates playing a video program in which annotations are associated with multiple different responsive behavior types.
  • player window 1900 comprises a video window 1902 that plays a video program.
  • a plurality of annotations defined in metadata files 126 are associated with graphic images displayed as page links 1904, video links 1906, and voting buttons 1908.
  • annotations for page links 1904 are associated with URLs for web pages that correspond to an individual who is depicted in the page link.
  • FIG. 18B Tim Gunn in a separate browser window in the manner shown for FIG. 18B, FIG. 18C.
  • FIG. 18B, FIG. 18C Tim Gunn in a separate browser window in the manner shown for FIG. 18B, FIG. 18C.
  • FIG. 19B illustrates an example of displaying a separate browser window 1920 below or behind the video window 1902 of the player window 1900.
  • the browser window is scrollable independent of the video window 1902, the video window is automatically displayed in a reduced size representation, and the video window may be restored to fully occupy the player window 1900 by selecting a full screen icon in the video window.
  • player logic 112 branches within the code of a metadata file 126 and causes playing an associated video segment. In the example of
  • the associated video segments may comprise commercials or infomercials associated with brands, products or merchants, but in other embodiments the video segments may be noncommercial.
  • voting buttons 1908 also represent annotations that cause the player logic 112 to invoke a web service that communicates a vote indicated by a particular voting button to a vote collecting server.
  • FIG. 19 A provides an example of how annotations may be used to link a viewer through interactive services to Internet servers that collect information or perform specified actions.
  • HTML and HTTP may be used to display a graphical format, termed a skin, for the player window 1302, for a background area of the player window 1302, and for various user interface elements such as annotation icons.
  • graphical backgrounds, skins, and UI elements all can be defined for, and thus synchronized at, any one or more of: video cue points; a metadata file 126 that describes a collection of video segments that are rendered into a single file; or a folder, directory, or collection of metadata files 126 making up a complex media presentation.
  • an author can configure cue-point level synchronization to show character background information as different characters come on stage.
  • the author can use file-level synchronization to have different backgrounds for commercials as compared to program content.
  • the author can use folder- or directory-level synchronization to change the color scheme used in backgrounds, windows and other UI elements on an episode-by-episode basis.
  • UI elements include annotations and associated graphic images.
  • a user may specify an HTML file to display in the background as the video is playing.
  • specifying a background skin is performed by a user accessing a Metadata tab 2112 of screen display 2102, as seen in FIG. 21, FIG. 22.
  • FIG. 22 is a screen display diagram of the Metadata tab.
  • a user enters a URL of an HTML document that contains a background image in a Background HTML field 2206.
  • the editor logic 110 stores the URL in metadata for the video.
  • the player logic 112 loads the URL and displays the contents as background behind or around the video window.
  • Background images may include graphics, text, product branding, or other information.
  • Metadata tab 2112 also displays and allows user entry of values for other parameters for other player functions that are described further herein.
  • a Video File field identifies a filename of a video file with which the user is currently working and that is associated with the other metadata.
  • a Video Size field identifies a size in pixels of a video window generated by the player logic 112 and that will display the video program at playback time.
  • a Web Service field 2202 displays a reference to a web service that can be invoked at one or more cue points to provide external functions or processing.
  • a Coincident Web Point field 2204 may receive user input of a synchronized web reference to display at a particular cue point.
  • a Skin Buttons field may receive a reference to an electronic document that defines an appearance for play and trick play buttons of the player.
  • video linking editor logic 110 may be used to author and configure, for playing using metadata-capable player logic 112, a live, refreshable collection of media with navigation metaphors.
  • a subscription video collection differs from a traditional magazine or program subscription in that time is an element of authoring; thus, the media elements that are available to a subscriber change over time. The media elements change over time not in the sense of an animation, which involves changes frame to frame, but for a season of a show.
  • the subscription may feature mixed HTML and video content, authored to incorporate additions, deletions and updates over time.
  • a subscription video collection is authored as at least a first video segment that comprises a plurality of annotations; each annotation may be represented by a graphic image or animation which, at playing time, is overlaid on the first video segment.
  • Each of the annotations is associated with a different one of a plurality of episodes or clips.
  • a show may have 22 planned episodes and at a particular time of year, there may be 8 of 22 episodes available for viewing to a subscriber.
  • An end user accesses a subscription at a web site associated with a producer or distributor of the show. The end user presents authentication credentials, such as user name and password, is authenticated and admitted to the subscription.
  • the metadata-capable video player logic 112 plays a first video segment that features 8 icons indicating episode names with graphics, and 14 icons indicating "Episode to be Available in the Future."
  • the annotations may be authored in the same single one of the metadata files 126 or may be in multiple different metadata files.
  • a first metadata file 126 for a show season may contain references to multiple other metadata files that contain actual annotations for each episode of the show. Selecting a particular episode to view is an invocation of the annotation associated with that episode and effectively causes a branch within the associated metadata file 126 to result in playing the selected video episode.
  • the techniques described herein are implemented by one or more special-purpose computing devices.
  • the special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application- specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination.
  • ASICs application-specific integrated circuits
  • FPGAs field programmable gate arrays
  • Such special- purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques.
  • the special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard- wired and/or program logic to implement the techniques.
  • FIG. 12 is a block diagram that illustrates a computer system 1200 upon which an embodiment of the invention may be implemented.
  • Computer system 1200 includes a bus 1202 or other communication mechanism for communicating information, and a hardware processor 1204 coupled with bus 1202 for processing information.
  • Hardware processor 1204 may be, for example, a general purpose microprocessor.
  • Computer system 1200 also includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204.
  • Main memory 1206 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1204. Such instructions, when stored in storage media accessible to processor 1204, render computer system 1200 into a special-purpose machine that is customized to perform the operations specified in the instructions.
  • Computer system 1200 further includes a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204.
  • ROM read only memory
  • a storage device 1210 such as a magnetic disk or optical disk, is provided and coupled to bus 1202 for storing information and instructions.
  • Computer system 1200 may be coupled via bus 1202 to a display 1212, such as a cathode ray tube (CRT), for displaying information to a computer user.
  • a display 1212 such as a cathode ray tube (CRT)
  • An input device 1214 is coupled to bus 1202 for communicating information and command selections to processor 1204.
  • cursor control 1216 is Another type of user input device
  • cursor control 1216 such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1204 and for controlling cursor movement on display 1212.
  • This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
  • Computer system 1200 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 1200 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 1200 in response to processor 1204 executing one or more sequences of one or more instructions contained in main memory 1206. Such instructions may be read into main memory 1206 from another storage medium, such as storage device 1210. Execution of the sequences of instructions contained in main memory 1206 causes processor 1204 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
  • Non-volatile media includes, for example, optical or magnetic disks, such as storage device 1210.
  • Volatile media includes dynamic memory, such as main memory 1206.
  • Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
  • Storage media is distinct from but may be used in conjunction with transmission media.
  • Transmission media participates in transferring information between storage media.
  • transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1202.
  • Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
  • Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 1204 for execution.
  • the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer.
  • the remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem.
  • a modem local to computer system 1200 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal.
  • An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 1202.
  • Bus 1202 carries the data to main memory 1206, from which processor 1204 retrieves and executes the instructions.
  • the instructions received by main memory 1206 may optionally be stored on storage device 1210 either before or after execution by processor 1204.
  • Computer system 1200 also includes a communication interface 1218 coupled to bus 1202.
  • Communication interface 1218 provides a two-way data communication coupling to a network link 1220 that is connected to a local network 1222.
  • communication interface 1218 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line.
  • ISDN integrated services digital network
  • communication interface 1218 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN.
  • LAN local area network
  • Wireless links may also be implemented.
  • communication interface 1218 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
  • Network link 1220 typically provides data communication through one or more networks to other data devices.
  • network link 1220 may provide a connection through local network 1222 to a host computer 1224 or to data equipment operated by an Internet Service Provider (ISP) 1226.
  • ISP 1226 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 1228.
  • Internet 1228 uses electrical, electromagnetic or optical signals that carry digital data streams.
  • the signals through the various networks and the signals on network link 1220 and through communication interface 1218, which carry the digital data to and from computer system 1200, are example forms of transmission media.
  • Computer system 1200 can send messages and receive data, including program code, through the network(s), network link 1220 and communication interface 1218.
  • a server 1230 might transmit a requested code for an application program through Internet 1228, ISP 1226, local network 1222 and communication interface 1218.
  • the received code may be executed by processor 1204 as it is received, and/or stored in storage device 1210, or other non-volatile storage for later execution.

Abstract

In one embodiment, a method includes obtaining metadata that relates to a video program and that defines, for one or more specified time points in the video program, one or more web services to be invoked at those time points; generating and displaying, on a computer, a video window that is configured to play the video program; during playing the video program on a computer, detecting that the video program is playing at one of the time points and determining an identifier of a particular web service associated with that one of the time points; in response to the detecting, causing the computer to invoke the particular web service.

Description

PLAYING AND EDITING LINKED AND ANNOTATED AUDIOVISUAL WORKS
COPYRIGHT NOTICE
[0001] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and
Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
Copyright © 2008-2010 Coincident. TV, Inc.
TECHNICAL FIELD
[0002] The present disclosure generally relates to video playing, video editing, and displaying hyperlinked media.
BACKGROUND
[0003] Commercial television broadcasting has been supported by advertising revenue since its inception. More recently, providers of video programs and video clips in Internet sites have embedded advertising within video programs or next to video programs in web pages at which the video programs are viewed. However, a continuing problem involved in these technologies is that the advertisements are not closely personalized for the viewer. Instead, commercial broadcasters attempt to define, in terms of rough demographic characteristics, a sub-population of a mass audience that is expected to be interested in a particular program; advertisers who believe that their products appeal to the same rough demographic will purchase advertising slots in the program. Unfortunately, a continuing result of this system is that at least some viewers, who do not fit the rough demographic, are shown advertisements that are irrelevant to the viewers' interests.
[0004] Internet technologies also have attempted to tailor advertisements, displayed in
World Wide Web sites, more closely to the preferences of Internet users, based on collecting explicitly-specified preference data, based on a user profile, or by inferring preferences through collecting metadata that is derived as the Internet user selects pages or performs online actions.
However, these technologies are not fully accurate because they rely on algorithms that attempt to match known characteristics of ads with user preferences that can be only roughly inferred from the data that the users provide.
[0005] Video editors such as Adobe Premiere Pro and Final Cut Pro enable users to select multiple video clips, join the clips, and annotate the clips by defining cue points and associating text notes with the cue points.
[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
BRIEF DESCRIPTION OF THE DRAWINGS
[0007] In the drawings:
[0008] FIG. IA illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments.
[0009] FIG. IB illustrates a process of creating video programs, which are linked to metadata, which can control operation of a video player.
[0010] FIG. 1C illustrates a process of playing a video program that is linked to metadata.
[0011] FIG. 2 illustrates an example screen display that the video linking editor logic generates and causes displaying.
[0012] FIG. 3 graphically illustrates an example video linking arrangement.
[0013] FIG. 4 illustrates a screen display in the Adobe Premiere video editor in which a video file has been created with the segments and advertisements and appropriate cue points.
[0014] FIG. 5 illustrates a portion of a screen display showing a cue point list for the video of FIG. 3, FIG. 4.
[0015] FIG. 6 illustrates the metadata panel populated with data for the Start cue point of the example.
[0016] FIG. 7 illustrates the cue point data configured with values from user input that create such a cue point.
[0017] FIG. 8 illustrates a display generated at playback time based on the metadata that has been created in the present example.
[0018] FIG. 9 illustrates appropriate values of program- wide metadata for the present example.
[0019] FIG. 10 illustrates an example screen display that includes a directory.
[0020] FIG. 11 illustrates an example screen display that illustrates a player screen that may be generated and displayed in a computer display unit by metadata-capable video player logic.
[0021] FIG. 12 is a block diagram that illustrates a computer system upon which an embodiment of the invention may be implemented.
[0022] FIG. 13A illustrates an annotation coupled to a web service providing automated text messaging in association with an enriched video program.
[0023] FIG. 13B illustrates a frame of an enriched video program as displayed in a player window.
[0024] FIG. 14 illustrates a frame of a video program having a highlighted service icon.
[0025] FIG. 15A illustrates an annotation that provides a user choice. [0026] FIG. 15B illustrates a frame of a video segment in a sequence for which Audrina is the featured character.
[0027] FIG. 16 illustrates concurrent playing of an enriched video program and displaying an associated web page.
[0028] FIG. 17A illustrates an example of playing an enriched audiovisual program with annotations that implement chapter selections.
[0029] FIG. 17B features a navigation animation, web integration icons, topic launch icons, and menu access link.
[0030] FIG. 17C illustrates a video window providing a menu of episodes in a collection or associated with a subscription.
[0031] FIG. 17D illustrates use of annotations to form elements of a main menu page for a video program subscription.
[0032] FIG. 18 A illustrates an example news program in which annotations may be used to provide a directory or menu of a plurality of news stores, features, segments, or related information.
[0033] FIG. 18B illustrates the news program of FIG. 18A after a viewer has selected a program link that is defined using an annotation having an association to a website.
[0034] FIG. 18C illustrates the browser window of FIG. 18B after the scroll bar has been moved.
[0035] FIG. 19 A illustrates playing a video program in which annotations are associated with multiple different responsive behavior types.
[0036] FIG. 19B illustrates an example of displaying a separate browser window below or behind the video window of the player window.
[0037] FIG. 20 illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments with a browser-based player for enriched video programs.
[0038] FIG. 21 illustrates an example screen display that the video linking editor logic generates and causes displaying and in which a Cue tab is selected.
[0039] FIG. 22 is a screen display diagram of the Metadata tab of an example Editor window.
[0040] FIG. 23 is a screen display diagram of an example Editor window in which an
Annotation tab is selected.
[0041] FIG. 24 is a screen display diagram of an example Editor window in which a Web tab is selected.
[0042] FIG. 25 is a screen display diagram of an example Editor window in which a Layout tab is selected. DETAILED DESCRIPTION
[0043] APPENDICES. Document appendices form a part of the disclosure herein and comprise the following elements: "Language Manual, Cue Point Language"; "CTV Editor User Guide"; "Addendum"; Schema Listing (XSD file). The appendices describe example embodiments and other embodiments may vary from the descriptions in the appendices. [0044] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention. Embodiments are described according to the following outline, although the following description does not reproduce, as section headings, each and every item in the outline.
1. Concept summary: Editor; Player; Metadata format
2. Overview of structural context
3. Overview of functional context: Directories; Jump to a destination; Get more information from a destination; Switching media based on a web service; Modal story branch; Overlay web content; Provide web points with associated URLs, graphics and text; Return from the end of a segment
4. Adding cue points and cue point names using a video editor: Overview of creating cue points; Definitions; endNote
5. Authoring video links using an editor
5.1 Editor GUI overview
5.2 Creating and modifying cue point metadata for particular cue point types
5.2.1 goto Cue Point
5.2.2 gotoAutoReturnButton Cue Point
5.2.3 insertPt Cue Point
5.2.4 modalStoryBranch Cue Point
5.2.5 MXMLOverlay Cue Point
5.2.6 progEnd Cue Point
5.2.7 webFocus Cue Point
5.3 Other language elements and attributes
5.4 Content types
5.5 Automatic creation of cue points
5.6 Directories
5.7 Web Services
5.8 Cue Point Language example 6. Playing video and linked media
6.1 Trick play functions, timeline, always-available web link
6.2 Keyboard controls
6.3 Subscription video
7. Implementation details — Hardware overview
* * *
[0045] 1. CONCEPT SUMMARY
[0046] Various embodiments provide an editor, a player, and a metadata format. In an embodiment, the editor implements a method of creating, for a video file consisting of multiple segments, metadata describing one or more display operations, decision operations, branching operations, video linking operations and web media linking operations, and associating the metadata with the video file. In an embodiment, the player implements a method of interpreting the metadata during playback of the video file and performing the operations in coordination with playback. In an embodiment, the metadata format comprises computer-readable data storage media encoded with tags and values which when interpreted cause performing particular display, decision, branching, video linking and web media linking operations. Metadata may comprise cue point type names for various cue point types, and attribute values associated with the cue point types that control the particular behavior of the player in performing the operations.
[0047] 2. OVERVIEW OF STRUCTURAL CONTEXT
[0048] FIG. IA illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments. In an embodiment, a computer 102 is coupled directly or indirectly through one or more networks 120 to a web server 130 and optionally to a file server 132. In various embodiments, network 120 may comprise a local area network (LAN), wide area network (WAN), an internetwork, or a combination. Web server 130 hosts one or more video files, HTML documents, HTTP servers or application servers, or other web content. File server 132 stores or hosts video files 122, graphics files 124, and metadata files 126. [0049] Computer 102 hosts or executes an operating system 104 that supervises I/O, storage management, and execution of application logic. In an embodiment, computer 102 further comprises a video editor 106. Commercially available examples of video editor 106 include Adobe Premiere and Final Cut Pro. In an embodiment, computer 102 comprises a browser 108. Commercially available examples of browser 108 include Firefox, Safari, Chrome and Internet Explorer.
[0050] In an embodiment, computer 102 is coupled to storage 140, which broadly represents any data storage device, storage area network (SAN), network attached storage (NAS), or network file system (NFS) unit or server. Storage 140 may reside on network 120 or on a server coupled to the network. Storage 140 stores video files 122, graphics files 124, and metadata files 126.
[0051] In an embodiment, computer 102 further comprises video linking editor logic 110 and metadata-capable video player logic 112. In other embodiments, computer 102 only comprises player logic 112 and does not have an editor; such an embodiment might be used by an end user who is viewing video programs that have been prepared by someone else. Thus, the use of video linking editor logic 110 is not required.
[0052] The video linking editor logic 110 is generally configured to cause one or more processors in computer 102 to receive user input specifying links between segments of a video file and other media such as other segments in the same file, other segments of other video files, graphics files, online content such as web sites or web applications, and other rich media content; to create representations of the links in metadata; and to store the metadata and link- related information in the metadata files 126 in association with related video files. For example, a user of computer 102 may interact with video linking editor logic 110 to select one or more of the video files 122, from storage 140 or file server 132, create links using editing functions that the editor logic provides, integrate graphics files 124 and references to content on web server 130, and then store metadata files 126 either at storage 140 or in file server 132. The metadata files 126 identify the associated video files 122 and contain metadata defining links among segments, link types, and link-related information to support novel playback functions and other user experiences. Other more specific functions of video editor linking logic 110 are described in other sections herein.
[0053] The metadata-capable video player logic 112 is generally configured to open metadata files and associated video files, and to play the video files while interpreting and responding to links and related information and instructions in the associated metadata files. Other more specific functions of metadata-capable video player logic 112 are described in other sections herein. The metadata-capable video player logic 112 may be implemented within a web browser and comprising a browser support library and browser-executable code, such as JavaScript, that is received in and executed by the browser at the time that an end user selects a video for playing. The browser support library may be any video playing plug-in component for a browser. Examples include Macromedia Flash and Silverlight. Alternatively, web browsers may use the VIDEO tag of HTML version 5 to render video and HTML and JavaScript to implement the player logic 112. In some embodiments, the player logic 112 may be partially implemented on server 132 or another server using dynamic AJAX techniques. For example, the server may convert data defining annotations into HTML to be displayed in the player. Alternatively, the metadata-capable video player logic 112 is implemented as a standalone program application that may be installed locally in computer 102. For such native applications any software development kit (SDK) that is capable of displaying video could be used to implement the player. Examples include SDKs for Apple Mac OS X, Microsoft WINDOWS, and Linux.
[0054] Each of the computer 102, video linking editor logic 110 and metadata-capable video player logic 112 may be implemented in various embodiments using a computer, one or more application-specific integrated circuits (ASICs) or other digital electronic logic, one or more computer programs, modules, objects, methods, or other software elements. For example, in one embodiment computer 102 may comprise a special-purpose computer having particular logic configured to implement the elements and functions described herein. In another embodiment, service computer 102 may comprise a general purpose computer as in FIG. 12, loaded with one or more stored programs which transform the general purpose computer into a particular machine upon loading and execution.
[0055] 3. OVERVIEW OF FUNCTIONAL CONTEXT AND OPERATION [0056] In an embodiment, video linking is facilitated by creating, in metadata files associated with video files, executable instructions and/or descriptive information that are linked to cue points in the video files. A cue point generally comprises an association of a name to a position within a video file, wherein the position is typically expressed as a time value or timestamp. In an embodiment, cue points are created for a particular video file using video editor 106; the names and values of cue points become part of the video file through conventional operation of the video editor. Thereafter, user interaction with the video linking editor logic 110 can create links, operations and link-related metadata information for one or more of the cue points. At any later time, the metadata-capable video player logic 112 may be invoked to play the video and to concurrently detect cue points, identify the previously created metadata information relating to links and operations, and execute the operations. [0057] FIG. IB illustrates a process of creating video programs, which are linked to metadata, which can control operation of a video player. FIG. 1C illustrates a process of playing a video program that is linked to metadata. In an embodiment, the video linking editor logic 110 is configured to perform at least selected functions of FIG. IB and the metadata-capable video player logic 112 is configured to perform the functions of FIG. 1C.
[0058] Referring first to FIG. IB, in one embodiment, at step 150 a video editor creates and stores one or more cue points in a video file. Thus, FIG. IB presumes that at least one video file has been created and stored on a computer, such as computer 102. Step 150 may comprise a user interacting with the video editor 106 to create and store named cue points in the video file as further described herein. Alternatively, step 150 can involve a process or logic in computer 102, or another computer, creating cue points in a video file using programmatic techniques or electronic communication of messages to the computer. [0059] In step 152, the computer receives user input identifying a video file. Step 152 may involve invoking the video linking editor logic 110 and specifying the file name of one of the video files 122, or specifying the name of one of the metadata files 126, which will include an internal reference to an associated one or more of the video files.
[0060] At step 154, the video linking editor logic 110 reads the video file, finds or creates an associated metadata file, displays data for cue points in the video file, and displays any associated metadata relating to links to other segments or content. If one of the video files 122 is specified at step 152 and no existing metadata file is found, then the video linking editor logic 110 creates a related metadata file. If an existing related metadata file is found, then that file is read and metadata relating to cue points is displayed on a display unit that is coupled to computer 102. An example graphical user interface that may be generated to display the metadata is further described herein in connection with FIG. 2, but the approach of FIG. IB does not require that particular GUI.
[0061] At step 156, the computer receives user input specifying, for a particular cue point, a cue point type. For example, interacting with the GUI of FIG. 2 or through other means, a user or external computer process or logic selects one of the previously created cue points of the video file and provides input specifying a cue point type value. At a cue point, any of several types of operations may be defined to be performed at the time of playback using the metadata- capable video player logic 112. In this document, a cue point within a video file and the operations performed at the cue point are sometimes collectively termed a cue point. Cue points as defined herein can refer to video, coupled video-web contexts or non-temporal web locations (or "web points," as further described).
[0062] In an embodiment, cue points enable a user at playback to jump forward and backward in time in a video, and jump between web content and video content. Since the user and the environment can change the order in which media is played, the metadata-capable video player logic 112 maintains data indicating the user's prior location so that the player can transfer control to a prior location.
[0063] In an embodiment, web points define an end for web content that specify where to transfer the user when the user has reached the end of a navigation path. Both video and web content can be displayed on the screen at the same time, overlaid over web content or using a picture-in-picture representation, and time can be running or paused. When web content is displayed, selecting a back operation transfers control to a previously viewed page but when the earliest page is reached then a subsequent back operation transfers control away from web content and to the previously viewed video segment. When video is displayed, performing a back operation returns to the beginning of the current video segment. [0064] FIG. 24 is a screen display diagram of an example Editor window in which a Web tab is selected. The Web tab 2402 of the example Editor screen display 2102 may be used, in an embodiment, to create and store web points in association with a video program. In an embodiment, Web tab 2402 displays a list of all web points that have been defined for the video program that is previewed in video window 2403. Selecting an Add web point control 2405 causes the editor logic 110 to display a data entry panel 2404 that may receive user input of parameter values defining attributes of a web point. In an embodiment, attributes include an interest URL 2406, query string 2408, target cue point, web view layout definition, description, and thumbnail graphic image.
[0065] In an embodiment, interest URL 2406 refers to an online electronic document that is loaded and displayed at playback time if the user requests additional information about the data shown at the web point. In an embodiment, query string 2408 comprises a database query that is submitted to an online engine if the web point is selected, to generate a search result so that the user receives current search result information associated with the web point. The target field defines a target cue point to which the user is directed at playback time after having viewed a web document associated with the web point. The web view layout definition field identifies a layout format for the player to be used when displaying web information; in an embodiment, the layout format is one of the formats shown in FIG. 25, which is described further herein. The description field is a text description of the web point to display, and the thumbnail graphic image is a graphic image to display in the player to denote the web point. [0066] In an embodiment, any of the following operations may be defined in the metadata for association with a cue point:
• Directory or Annotation — a directory or annotation specifies one or more graphics files, web services, and associated links; at playback, the graphics files are selectable as hyperlinks to cause playback of other video segments and the web services may be invoked automatically to fire database queries, retrieve data, dispatch emails or text messages, or perform other communication functions as defined in the web services.
• Jump to a destination — metadata can specify that when a particular cue point is reached during playback, the player should jump to another cue point within the same video file. The destination cue point may be earlier in time or later in time than the cue point from which a jump is made.
• Get more information from a destination — metadata can specify that when a particular cue point is reached during playback, the computer 102 should connect to a web site at web server 130 and display a web page or invoke a web application. Typically the web site content is associated with or related to the video content at the cue point, but such an association is not required. • Change media under external process control — metadata can specify that when a particular cue point is reached during playback, the computer 102 should switch to one of several pieces of media, as determined by a web service, and continue as specified in the media to which the switch was made.
• Modal story branch — metadata can specify that when a particular cue point is reached during playback, the computer 102 should switch to one of several pieces of media, determined by user selection of an image associated with the media.
• Overlay web content — metadata can specify that when a particular cue point is reached during playback, the computer 102 should display a graphical and interactive overlay. This overlay is logically and computationally associated with a web service. The web service maintains databases (both session and persistent) that can be used to influence the course of playback, for example with an insertPt cue point. In an embodiment, the metadata-capable video player logic 112 invokes asynchronous web services to control user interaction with the overlaid web components.
• The video linking editor logic 110 can define and store one or more web points comprising names with associated URLs, graphics and text. These web points can substitute for video cue points. For example, all targets for a user choice cue point can be either video cue points or web points. In this context web points also define an "end" action to be followed in circumstances paralleling reaching the end of a video segment.
• In an embodiment, video linking editor logic 110 can define and store, in the metadata file, one or more cue points that include data defining required associated web pages. Such cue points are termed synchronized attributes or cue point attributes and refer to a specified point in a video that automatically creates a primary and secondary window. For example, in an embodiment, any cue point can have an associated web page that is to be displayed while the cue point is active. In this embodiment, when the video segment defined by the cue point is being played and the cue point is reached, the associated web page is automatically loaded and displayed in a browser window that appears conceptually under the video player window. In this context, "under" refers to Z-axis ordering of video and web content; in an embodiment, the video content is "over" the web content in the sense that the video content may obscure certain web content. The size and positioning of the video and web content can also be specified in association with the cue point using the video linking editor logic. With this approach, an author can define a video that provides a synchronized display of an audiovisual work and Internet content such as web pages. The web pages might comprise an advertisement or other online information that is relevant to a particular event on the TV program. For example, an author can set a cue point for a time in the video at which a character appears wearing particular distinctive clothing, and can associate a web page for a merchant of that clothing with the cue point. Consequently, when the video plays and the character appears in the video, the merchant's web page for the associated clothing is automatically accessed and displayed in a browser window behind the player. As an another example, a cue point can associate a time in the video at which an actor appears with a particular web page of the Internet Movie Database (IMDB) service, www.imdb.com, that contains background, filmography and other details for the actor. This approach may be economically attractive both to the broadcaster of the video and the associated web site; for example, the associated web site benefits from an additional page view while the broadcaster concurrently continues to have a viewer viewing the video. More broadly, this approach enables events occurring in temporal media such as video to cause automatic changes in state-based media such as web pages. In an embodiment, creating a synchronized attribute is performed by selecting the Cue tab 2106 in the editor screen display 2102 (FIG. 21), selecting a cue point to which a synchronized attribute should be associated, selecting an Other tab in a Parameters pane, selecting Browse and selecting an appropriate web point. • Return from the end of segment — metadata can specify that when a particular cue point is reached during playback, the computer 102 should return to a previous segment from which a branch was taken. In an embodiment, web-based cue points define an end, even though web content is a non-temporal media, to specify where to go when the user has reached the end. Both video content and web content can played or viewed on screen at the same time, overlaid or picture-in-picture, and time can be running or paused. In an embodiment, selecting a Back button while viewing web content causes backtracking among hyperlinks in the manner of a web browser, but when no other prior links exist then a selection of a Back operation transfers control to the video segment from which the web content was reached. In content, moving backward in video transfers control to the beginning of the video. [0067] In an embodiment, metadata-capable video player logic 112 interprets metadata such that when the user is watching video, a web link to some form of related content is always available. If the user selects the web link and views the web content, the player displays the video in a reduced size picture-in-picture form. Further description of the foregoing cue point types is provided in other sections of this disclosure.
[0068] Referring again to FIG. IB, at step 158, the video linking editor logic 110 updates the computer display based on the cue point type to provide input fields and display fields for metadata values that are associated with the specified cue point type. Thus, a context-sensitive display of input fields and display fields is provided depending on the cue point type. Step 158 may also include receiving user input that indicates particular metadata values for the input fields. For example, if the cue point type provided at step 156 is "modal story branch," then at step 158 an input is received to specify two or more target cue points that represent branch destinations.
[0069] At step 160, the video linking editor logic 110 creates and stores the cue point type and the associated metadata values in the metadata file that is associated with the video file. As shown in optional step 161, the type and values may be stored in one or more XML script(s) within one of the metadata files 126. However, XML is not required in all embodiments and the metadata files 126 may represent cue point names, types and metadata values in other forms that can be read by the metadata-capable video player logic 112 and used to control linking, branching, decisions, web interaction, and other content operations when cue points are reached. [0070] Step 162 represents testing whether a user has requested to exit or discontinue using the video linking editor logic 110. If no exit request is received then control is transferred to step 156 or step 158 for the computer to await further user input relating to cue points. Alternatively the computer may perform an exit operation at step 164, for example, by closing the video file and metadata file.
[0071] As indicated in step 166, during all operations previously described for FIG. IB the method is configured to asynchronously process user input requesting video playback, trick play functions, or loading other video files or metadata files. Thus, in an embodiment, a playback mechanism may be integrated into the process so that a user can play and view a video program or segment while determining what cue point types and values to specify. The playback mechanism supports non-linear playback of video so that the player can execute branch operations, play one of a plurality of different alternative video segments at a branch point or decision point, return to a prior point and continue playing the next segment thereafter, and other complex operations consistent with the rich media authoring capabilities described herein. At any time during the process of FIG. IB, the user may request playing a video segment or performing trick play functions such as fast forward or rewind. In an embodiment, selecting a different named cue point at step 156 causes the player mechanism to display a first frame of the video segment that starts at the selected cue point or to begin playing the video from that point. [0072] As a result of the process of FIG. IB, a video file having internally stored named cue points becomes associated with a separate metadata file that specifies cue point types and metadata values relating to control functions for the video file, related networked content, and other user interactions. The metadata-capable video player logic 112 is configured to play the video and, as each cue point is reached, perform the control functions based on the cue point types and metadata values that are specified in the metadata file.
[0073] FIG. 1C broadly represents a process involving opening a stored video file having one or more video segments and one or more cue points in the video file, and opening a metadata file that contains an internal reference to the video file; playing a first video segment of the video file; in response to reaching, during the playing, one of the cue points that are defined in the video file: receiving from the metadata file one or more metadata values identifying a cue point type, and one or more values of attributes that are associated with a particular cue point type, the cue point type and attribute values defining features of an action to perform at the cue point during playing of the video file; performing the action using the attribute values to determine and perform particular features, displays, or controls associated with the action. [0074] In one embodiment of a playback process, at step 170 the computer initiates executing the metadata-capable video player logic 112. Initiating execution may occur in response to user input, or in response to an instruction from other hardware logic or computer processes. For example, a user, logic, or process may select and invoke one of the metadata files 126 or video files 122, and in computer 102 the files may be associated with the metadata- capable video player logic 112 as an application that is launched when the files are invoked. [0075] Optionally, in step 172, the metadata-capable video player logic 112 locates any existing metadata files and displays a list of the metadata files. Each metadata file may be represented visually in the list using a still image or other graphics file that is referenced within the metadata file. Thus, the metadata-capable video player logic 112 may generate a display of thumbnail images, each image representing an associated metadata file. At step 174, the metadata-capable video player logic 112 receives user input selecting a metadata file from the list. For example, the graphic images may comprise selectable links and the user may select one of the images using a pointing device. Steps 172 and 174 are described as optional because a selection of a metadata file may be unnecessary if the metadata-capable video player logic 112 is invoked by a user or process launching one of the metadata files 126 rather than launching or invoking the metadata-capable video player logic independently.
[0076] In step 176, the selected metadata file is opened. Each of the metadata files 126 is configured to internally name or reference at least one of the video files 122. Therefore, the metadata-capable video player logic 112 reads the selected metadata file, identifies the referenced video file, and opens the referenced video file at step 178.
[0077] At step 180, the metadata-capable video player logic 112 enters a loop that begins when the video player logic plays the video file that was found and opened at step 178. At step 182, a test is performed to determine whether a cue point has been reached. Step 182 represents the occurrence of an interrupt or other event indicating that a cue point was reached. As an alternative to interrupts, step 182 may be implemented by examining stored metadata values relating to a segment and setting timers that cause generic, non- video events to occur when the video events would have occurred. The timers are adjusted as the user moves among video segments and plays video segments, as the amount of time to a given video event changes as a result of such movement. However, this approach enables content to play correctly even if the cue points have been removed from the video in the course of transmission or transcoding. For example, preparing video for the YouTube online player results in the YouTube system discarding the cue points and the present approaches enable video productions to play properly on YouTube.
The NO control path of step 182 represents continuing playback and waiting until the next cue point is reached.
[0078] At step 184, when a cue point has been reached, the metadata-capable video player logic 112 determines the name of the cue point that has been reached. At step 186, based on the cue point name, the metadata-capable video player logic 112 reads and executes one or more metadata scripts and/or values associated with the current cue point, based on functions and behavior configured in the video player logic. Thus, in one embodiment, the metadata-capable video player logic 112 comprises logic or program instructions that define what functions are performed for all cue point types, and the metadata files 126 specify cue point types and attribute values that control how the functions are performed, such as specific video displays, graphical displays, user interactions, branches, links or other control functions. [0079] After step 186 control returns to step 180 to continue playing the current video segment. As a consequence of the processing in step 186, the current video segment after step 186 may be a different video segment than earlier, depending on the cue point type and its associated metadata values. As with FIG. IB, during any part of the loop from step 180 to step 186, the process of FIG. 1C and the metadata-capable video player logic 112 may be configured to asynchronously process user input requesting trick play functions or loading other video files or metadata files.
[0080] As a result, the approach of FIG. 1C enables playing a video with a rich set of controls and user interactions including branching to different video segments automatically, presenting a user with a branch selection menu and branching to particular video segments in response to user selection, determining a branch or different video segment using a web service, presenting web content that is related or associated with a video segment, and other controls and user interactions. The video file does not require internal modification and can be used with other players that do not provide the controls and user interactions. The controls and user interactions can be authored using an editing process as shown for FIG. IB, enabling video producers to rapidly create rich video productions without detailed knowledge of programming. [0081] 4. ADDING CUE POINTS
[0082] 4.1 ADDING CUE POINTS USING A VIDEO EDITOR
[0083] In one embodiment, video linking editor logic 110 uses one or more cue points that have been previously defined for video files on which the video linking editor logic operates; in other embodiments as further described herein, cue points may be defined independently of the video, using the video linking editor logic, and are stored in metadata separate from the video files. In an embodiment, users create cue points and cue point names using the video editor 106. For purposes of this document, a "video" is a single piece of video content (a file or a URL) typically with many cue points; within a video each "segment" begins and ends with a cue point without any cue points in between. A "compound segment" or "inline" segment has cue points within it, i.e., cue points in addition to the beginning and ending cue points. An external video, specified by a URL, may also contain cue points, and depending upon their organization, these cue points may be segments or compound segments. The player can refer to internal and external cuePoints transparently.
[0084] In an embodiment, video editor 106 is used to organize one or more video files into pieces each having a cue point at the start, at the end, and at any point to or from which the current time ("head") can jump. Cue points have an attribute canBeDestination. If this is set to false, the cue point cannot be a destination of any action which causes the playhead to jump. Cue points with canBeDestination set to false are typically used as markers for overlaying annotations but where the author does not want that point in the video to be a destination for a chapter jump.
[0085] There are also cue points with cue type = "null". These are used to place markers in the video at precise points that the author may at some future time want to use. Null cue points require less processing. For example, when a logger (the first and least expensive person in the workflow on professional video shoots) logs the shots of the raw footage they can put Null cue points at every shot without adding undue computational overhead. After rendering, a step that takes many hours of computation, these cue points are all available and can selectively be changed into meaningful cue points like "regular" or "insertPt" without re-rendering. [0086] A user creates one or more cue points as desired using a cue point tool within the video editor 106. For example, in Premiere, a cue point is created by moving an icon representing a playback head to a particular point in the video file and selecting "Cue Point." [0087] In an embodiment, a last video segment in a video file is supplemented with a terminal video segment denoted "endNote." For example, an endNote may comprise short piece of junk video positioned about two seconds after the last usable video segment. The endNote is created with zero cue points to prevent confusion with an automatic, invisible cue point that the video editor 106 automatically inserts at the end of the last piece of media. In an embodiment, the endNote is positioned about two seconds after the last usable video segment to prevent reaching the actual end of the video file under certain conditions; user experience has determined that when the metadata-capable video player logic 112 issues a command to pause or stop, the virtual head keeps moving for a short time interval. [0088] 4.2 SOFT CUE POINTS
[0089] In an embodiment, video linking editor logic 110 is configured to enable a user to define one or more cue points independent of the video for storage in metadata files 126. A cue point that is defined and stored in metadata, rather than stored within a video segment and previously created in the video segment using a separate video editor 106, may be termed a "soft" cue point. Soft cue points allow the user to insert, delete, and change the time of cue points directly into a video that has already been imported into storage associated with the editor logic 110.
[0090] In an embodiment, a soft cue point is created using editor logic 110 by selecting the Cue tab 2106 (FIG. 21) and selecting an add cue point control 2108. Selecting the add control 2108 causes editor logic 110 to create and store metadata for a new cue point at the default time of 00:00:00:00. Selecting the Description tab enables a user to insert a particular time for the cue point. The time can be determined by scrolling through the video using the trick play controls 2110.
[0091] In an embodiment, a cue point is a named marker for a particular point in a video segment. A cue points may comprise a name, a time value indicating the particular point, and other metadata that defines what actions occur when that point is reached during playing the video. During playing the video, video player logic 112 continuously compares the time value of a current position of a logical playback head within a video segment, and determines if the current time value is equal to any soft cue point that has been previously defined and stored in the metadata file 126 that is associated with the video segment. When a cue point is reached, the video player logic 112 performs one or more particular operations that are defined in the metadata of the cue point.
[0092] In this manner, an author can build a complete interactive video experience from existing video files without needing to use complex tools like Adobe Premiere or Final Cut to create cue points. For example, an author can select and use video files that are maintained on a third party video server or hosting site, such as YouTube, and streamed from that server or site to an end user using the video player logic 112 at the time of playback. The video files do not need to have cue points previously defined for and stored in them. Instead, the user uses video linking editor logic 110 to create cue points and store the created cue points in the metadata files 126. The metadata files 126 can be launched and can invoke the video player logic 112 to cause the video player logic to invoke streaming the video segments from the third party video server or hosting site while concurrently performing one or more operations as cue points are reached in playing the video segments. [0093] 4.3 EXTERNAL CUE POINTS
[0094] In an embodiment, video linking editor logic 110 is configured to enable a particular metadata file 126 to reference cue points that are defined in other metadata files 126. In an embodiment, a cue point may comprise a contained element termed a target, which specifies a cue point by name and optionally links it with an association attribute. An attribute of a target may be a cue point reference, which may reference cue points that are in other metadata files. In an embodiment, a cue point reference is formed as a URL comprising a file location (path), file name, and a URL fragment that identifies a particular cue point. For example, the cue point reference "http://www.coincident.tv/cplfiles/foo.cpMDadArrivesHome" identifies a cue point named "DadArrivesHome" within a metadata file named "foo.cpl" that is stored in the folder or director "cplfiles" of the "coincident. tv" domain. In this embodiment, in any metadata file definition in which a cue point can be a target, for example, as the target of an annotation, insert point, goto cue point, or directory or user choice entry, that target can be in another file referenced by relative URL.
[0095] External cue points beneficially enable authors to work with cue points that otherwise might require multiple other steps to re-define for a particular audiovisual work. For example, a 2-hour video program might contain dozens of cue points, but a particular author might wish to reference only a few of the cue points. The author need not re-define the same cue points in a new metadata file for a new audiovisual project, but can reference previously defined cue points within other, previously created metadata files. Therefore, the author can create a cross-linked metadata control structure that can simplify video program development based on other files or segments. [0096] 5. AUTHORING VIDEO LINKS [0097] 5.1 EDITOR GUI OVERVIEW
[0098] The structure and operation of an embodiment of video linking editor logic 110 is now described. In an embodiment, video linking editor logic 110 generates and causes displaying a graphical user interface (GUI) on a computer display unit, and the GUI provides cue point editing functions that can be used to link video segments and other content in a plurality of ways. The editor logic 110 is also configured to create and store, based on user input interacting with the editing functions and providing selections and values, metadata describing the links. In an embodiment, the metadata comprises one or more scripts expressed in a Cue Point Language (CPL). In an embodiment, CPL comprises an XML-based language that describes non- linear structures in a mixture of video and web media. CPL can be embedded into digital video content that is available from a plurality of sources such as broadcast, DVR, DVD, broadband, game consoles. CPL can be associated with web content also. The resulting metadata may be played back with a CPL-capable player to create a playback experience that integrates video and interactive web-based graphic elements in such a manner that the sequence of playback is influenced by user interaction, run-time execution of code embedded in the video, run-time interaction with code referenced by data embedded in the video, and calls to remote web services in combination with jump tables authored in the editor and embedded (or embedded by reference) in the video.
[0099] The CPL may be viewed as an architecture rather than a user interface. For example, while CPL implements a mechanism for a modal n-way branch, the author can use that mechanism to provide a video production that is graceful and easy to use, or confusing and user- hostile. CPL is compatible with a variety of playback platforms, asset locations and video formats. For example, in emerging systems video content can be viewed using screens that are attached to processors, disks or network connections. Platforms may consist of computers, game consoles, set- top boxes, or mobile devices. CPL is format independent with the assumption that all digital video formats define cue points and have ways to associate events and text with the cue point. CPL is location independent and can interoperate with video that originates from any desired source.
[0100] FIG. 2 illustrates an example screen display that the video linking editor logic generates and causes displaying. In an embodiment, screen display 200 generally comprises a video window 202, metadata panel 204, cue point list 206, a web point list, an annotation list and cue point data panel 208.
[0101] Video window 202 is configured to play and show one or more video segments representing a linked video project and comprises buttons 201 that are configured to receive user input selecting a playback function and trick play functions such as jumping to different segments that are forward or backward in time. In this context, a "video project" refers to an association of a video file and a metadata file.
[0102] Metadata panel 204 receives and displays metadata values that pertain to a project as a whole. In an embodiment, metadata panel 204 comprises unique id field 212, video file field 214, and web service field 216. The unique id field 212 is configured to receive a name, number, or other character sequence that uniquely identifies the current video project, and the unique id value is used in naming the metadata file that the editor creates and associates with a video file and to coordinate dynamic updates with a server. The video file field 214 displays a name of a video file that has been loaded using the File menu 205 and previously created with cue points in a video editor. The name may comprise a pathname in a filesystem accessible to the computer that is hosting the video linking editor logic 110, a URL identifying video in a web server, or another form of location identifier specifying a location of video. In an embodiment, selecting the File menu item 205 initiates a File Open dialog and after a file selection is made the logic 110 displays a value in the video file field 214 and opens and displays the named video file in video window 202. Alternatively, a user may direct logic 110 to load a previously created metadata file, and in response, the video linking editor logic locates a video file that is referenced within the metadata file and displays the name of that referenced video file in video file field 214.
[0103] The web service field 216 is configured to receive user input identifying a web service in the form of a URL. The specified web service may be hosted on computer 102 or on a remotely located computer. The web service may comprise a web application or a script file. The web service provides a control mechanism for interacting with insert points, overlays, and other types of cue points that are further described in other sections herein. [0104] Cue point list 206 is configured to display a list of cue points that have been previously defined in the video that is shown in video window 202. In an embodiment, in response to user input opening a video file, video linking logic 110 loads and displays the named video in video window 202 and concurrently reads and displays the cue point data that was embedded in the video file as a result of creating cue points using the video editor. Cue points found in the video file are listed in one or more rows 218 of list 206 and each row includes time of the associated cue point in a time column 220 and a name in name column 222. [0105] In an embodiment, existing web points in the video are displayed in a separate list, and cue point annotations are displayed. The form and use of annotations are described more fully in the section herein entitled ANNOTATIONS and in the Appendix. [0106] Further, in an embodiment the first cue point in list 206 is automatically selected and highlighted in the list. Video linking logic 110 is further configured to search for an existing cue point metadata file that may have been created in an earlier user session with video linking logic 110. If an existing cue point metadata file is found, then cue point data is loaded and the video linking logic displays, in cue point data panel 208, cue point data for the first cue point in list 206 that was automatically selected and highlighted.
[0107] Cue point data 208 is configured to receive user input specifying one or more metadata values relating to a particular link or transfer of control associated with one of the cue points in cue point list 206 that is currently selected or highlighted in the cue point list. In an embodiment, a user may operate a pointing device such as a mouse or trackball to select other cue points in list 206 and in response to selection of a different cue point the video linking logic 110 automatically updates cue point data panel 208 to display cue point metadata for the newly selected cue point.
[0108] Cue point data panel 208 comprises a cue name field 224 and cue time field 226 that reproduce the data shown in cue point list 206 for a selected cue point. Cue point data panel 208 comprises a cue type combo box 228. Particular types of cue points are described further in other sections below. Cue point data panel 208 is context-sensitive so that the particular fields displayed as part of the panel will vary according to the value of the cue type combo box 228 and a content type combo box 230. For example, when the cue type is Regular and the content type is ad_Inline (referring to an advertisement within a video segment) then the cue point data 208 comprises an interest URL field 232, query string field 234, story text field 236 and story picture field 238 as shown in the example of FIG. 2.
[0109] Alternatively, the fixed content types represented in FIG. 2 may be omitted and an author may tag cue points with arbitrary content types as further described in the Appendix. [0110] The interest URL field 232 is configured to receive user input specifying a website or other URL to which a viewer may be directed at playback time in response to receiving input indicating interest in other information relating to the video. The query string field 234 is configured to receive user input specifying a search engine query string which, at playback time, the metadata-capable video player logic 112 may submit to an Internet search engine for the purpose of generating search results in which a viewer may have interest, or that relate to the video. The story text field 236 is configured to receive user input specifying a story to display to a viewer using the player logic 112 at the time the video is played. The story picture field 238 is configured to receive user input specifying a graphics file or still image, and a text string, to display to the viewer using the player logic 112 at the time the video is played. [0111] An example of using the video linking editor logic 110 and interacting with the screen display 200 is now provided. FIG. 3 graphically illustrates an example video linking arrangement that can be configured using the mechanisms now described. For purposes of illustrating a clear example, FIG. 3 describes relatively few video segments and cue points; in a practical embodiment the techniques herein can be used to create video projects having any number of video segments and cue points.
[0112] The example of FIG. 3 represents a non-linear video program in which the viewer arrives at a choice point and selects one of three possible videos; at the end of the selected video, the video project continues with the program. The video project comprises a first program video segment 302 having a start cue point 312 and ending in a modal story branch cue point 314, which is configured in video linking editor logic 110 to permit an N- way branch to other video or content but in the example of FIG. 3 is configured as a three-way branch. A first branch leads to a first video advertisement 306 relating to hair care products. A second branch leads to a second advertisement 308 relating to face products. A third branch leads to a second program video segment 310.
[0113] To create a video project in which the foregoing logical structure is achieved at playback, a user activates video editor 106 and authors a video project that includes segments 302, 310 and advertisements 306, 308 in the same video file. The user creates and stores a Flash navigation-type cue point with a name at a plurality of locations in the video file. FIG. 4 illustrates a screen display in the Adobe Premiere video editor in which a video file has been created with the segments and advertisements and appropriate cue points. After creating the cue points, the user saves the video project in Premiere and encodes the video. [0114] The user then activates video linking editor logic 110, and in response, the user interface of FIG. 2 is displayed. The user selects a Load File function in screen display 200 and selects the video project that was created. In response, the video linking editor logic 110 loads the specified video file and displays data for cue points that are found in the file. FIG. 5 illustrates a portion of screen display 200 showing cue point list 206 for the video of FIG. 3, FIG. 4. Assume the user selects the Start cue point. In response, video linking editor logic displays metadata associated with that cue point in the metadata panel 208. FIG. 6 illustrates the metadata panel populated with data for the Start cue point of the example. The user may edit the values in the metadata panel by selecting fields and entering new values, or selecting pull-down menus.
[0115] Assume that the user wishes to create the modal story branch cue point 314. FIG. 7 illustrates the cue point data 308 configured with values from user input that create such a cue point. A name may be entered in the Cue Name field. The Cue Time field is not modified and shows the value obtained from the video file. The cue type is selected as "modalStoryBranch." A branch cue type is associated with no content, so the Content Type field is grayed out. A Targets list identifies possible destinations or targets to which control is transferred at the branch point. A Background Picture field and Text field receive an identification of a picture to display to the user in a background area while the user is determining which selection to make, and a text string that can serve as a prompt.
[0116] FIG. 8 illustrates a display generated at playback time based on the metadata that has been created in the present example. The user may create program wide metadata by entering values in the CPL metadata panel 204. FIG. 9 illustrates appropriate values of program-wide metadata for the present example. The user may then save the metadata using a Save function in the File menu 205 of the screen display 200. In an embodiment, selecting the Save function causes the video linking editor logic 110 to create and store an XML file containing the metadata and to store the XML file in a same directory or other storage location as the video file that is referenced in the metadata. In an embodiment, multiple cue point metadata files may reference and may be associated with a single video file.
[0117] At any point after creating and storing the metadata file, the user may invoke the video linking editor logic 110, reload the metadata file, modify the cue points, save an updated metadata file with modified cue point data, and replay the video based on the updated metadata file. Such updates may be performed without re-encoding the video, because the video file is maintained entirely separate from the metadata file. [0118] 5.2 CREATING AND MODIFYING CUE POINT METADATA FOR
PARTICULAR CUE TYPES
[0119] 5.2.1 GOTO CUE POINT
[0120] In an embodiment, a "goto" cue point may be defined and at playback, the goto cue point causes a jump to another video segment when the play head reaches the cue point. The destination location for the jump is defined in a cue point group ("cpGroup") and discussed below. In an embodiment, a goto cue point has the following associated metadata:
Figure imgf000023_0001
[0121] In an embodiment, a goto cue point has the following elements of interest:
Figure imgf000023_0002
[0122] An example script code excerpt including a goto cue point, which may be included in a metadata file, is:
<cuePt cueType="goto" interestURL="" nameCue="goSome Where" contentType="zeroLen"> <cpGroup>
<targetSeg cuePointName= "targetThisIsWhere YouGo "></targetSeg> </cpGroup> </cuePt>
[0123] 5.2.2 gotoAutoReturnButton cue point
[0124] In an embodiment, a gotoAutoReturnButton cue point supports a mechanism for the user to obtain more information relating to a particular video. From the gotoAutoReturnButton until the next cue point, the player causes the video to be overlaid with a graphical button; user input selecting the button causes the player to perform a goto branch operation to reach another cue point with an automatic return. In an automatic return, at the end of the "more information" video segment, the player causes the playhead to jump back, reaching the beginning of a video segment that just fallows the end of the calling video segment. For example, a first video segment might comprise a 30-second automobile commercial; 10 seconds into it, the user selects the "more info" button, jumps to a 5 -minute extended commercial about the car; and at the end of the extended commercial the player jumps back to the programming that followed the original
30 second commercial.
[0125] In an embodiment, the "gotoAutoReturnButton" cue point comprises the following attributes:
Figure imgf000024_0001
[0126] In an embodiment, the following elements are provided:
Figure imgf000024_0002
[0127] In an embodiment, an example script code segment comprises:
<cuePt cueType="gotoAutoReturnButton" interestURL="http://www.saabusa.com" nameCue="A" contentType="ad_Inline">
<gotoAutoReturnButton xCueName="C" xLabel="More about Saab!"/> </cuePt>
[0128] 5.2.3 INSERTPT CUE POINT
[0129] In an embodiment, an insertPt may be used to include one of several pieces of media at a certain point in time. A selection of one of the pieces of media is made by a call to a web service. When reached, the cue point at the end of a target piece of media determines what happens next. The cue point at the end may comprise a returnEnd, goto or progEnd cue point. [0130] In an embodiment, the media consists of one or more video segments with cueType="reg" to begin and a returnEnd, goto or progEnd to end; and one or more web points with cueType="webFocus" to begin and a valid cue point name specified in the gotoWebFocusEndName attribute.
[0131] The group of media points is specified as a cpGroup. The cpGroup must have uniform endings for the segments it contains. For example, every cue point identifies a contain segments (or compound segments) and every segment implies an ending cue point. For a cpGroup, all of the ending cue points are either goto cue points, returnEnd cue points, or progEnd cue points, or a mixture of these types of segment ending cue points. [0132] In an embodiment, when the player reaches an insertPt, the player invokes the web service specified in the progLevelMetadata element described below with an operation specified with the cpGroup. The result of this call is used to select which media to display (the "target"). [0133] For example, assume the user has provided zip code information when registering for NBC.com, and the user is watching an episode of "Saturday Night Live" using the player disclosed herein. At an insertPt for a commercial, the player calls a web service to obtain the user's zip code. Based on the received zip code value, the player selects from among Bronco, Escalade and Hummer commercials. In an embodiment, the cpGroup is stated in script code as:
<cuePt cueType=" insertPt" interestURL="" nameCue="B" contentType="ad_Inline"> <cpGroup operation="whichAdByZip">
<targetSeg association="94010, 94103" cuePointName="escalade"x/targetSeg> <targetSeg association="94611, 94612, 94708" cuePointName="hummer"x/targetSeg> <targetSeg association="94617, 95012, 99234, default" cuePointName="bronco"x/targetSeg> </cpGroup> </cuePt> [0134] In an embodiment, the cpGroup is a table of targets in which an association attribute configured as a string is linked to a media point. The result of the web service call, a string, is tested against the association values until a match is found, and the first match is used. The matching function implements a many-to-one matching as detailed in the cpGroup element description. If no match is found then the association= "default" is checked against the table. If there is no match for the string or for "default", then nothing is inserted and the video plays on. [0135] In an embodiment, the end of a video segment is its ending cue point. With cueType= "returnEnd," control returns to the calling point. A goto end cue point jumps to wherever specified and a progEnd stops playback. In an embodiment, the end of a cue point with cueType="webFocus" is explicitly specified. It is reached by user action ("back" or goto TV). In an embodiment, the insertPt cue point has the following metadata attributes:
Figure imgf000026_0001
[0136] In an embodiment, the cue point has the following elements:
Figure imgf000026_0002
[0137] An example of script code using an insertPt cue point is:
<cuePt cueType= "insertPt" interestURL="" nameCue="B" contentType="prog_Inline"> <cpGroup operation="whoAmI"> <targetSeg cuePointName="D" association="F18to35"x/targetSeg> <targetSeg cuePointName="F, default" association="oldMen">
</targetSeg>
</cpGroup>
</cuePt>
Further description on the details of behavior of the modalStoryB ranch and insertPt cue points, in an embodiment, is provided in the Appendix.
[0138] 5.2.4 MODAL STORY BRANCH CUE POINT
[0139] In an embodiment, a modal story branch cue point causes the player to pause the video and to present the user with an n-way branch. The user selects an image representing the cue point to go to that cue point. The cue points can be either video or web points. The type for the cue points at the end of the targets are (for video) is goto or progEnd. In an embodiment, the cue point has the following attributes:
Figure imgf000027_0001
[0140] In an embodiment, the cue point has the following elements:
Figure imgf000027_0002
[0141] In an embodiment, FIG. 8 illustrates an example screen display resulting from the use of a modal story branch cue point and example script code follows.
<cuePt cueType= "modalStoryBranch" interestURL="" nameCue="B" contentType="zeroLen"> <cpGroup backgroundPicLoc="stills/back.jpg" headerText="Pick your commercial and earn $!$!$!">
<targetSeg association^"' cuePointName="X"/>
<targetSeg association="" cuePointName="G"/> <targetSeg association="" cuePointName="I"/>
<targetSeg association="" cuePointName="K"/> </cpGroup> </cuePt>
[0142] 5.2.5 MXML OVERLAY CUE POINT
[0143] In an embodiment, an MXML (Macromedia extensible Markup Language) overlay cue point allows use of a web development tool to define overlays with web-aware bindings. An example development tool is Flex from Adobe Systems, Inc., San Jose, California. Flex provides for content layout and code within an asynchronous architecture. In an embodiment, in the MXMLOverlay cue point MXML code is passed to the player via the mxmllnCPL element in the cue point. The code is executed to make the UI element overlays. For example, the metadata-capable video player logic 112 is configured to read a MXML user interface markup language script from the metadata file, parse and interpret the MXML script, and generate and display one or more overlay graphical elements in the video window of the player GUI based on the parsing and interpreting.
[0144] User interaction is processed using a web service that is specified in the progLevelMetadata attribute. User interaction with each component, such as a button, is handled by invoking an operation within the web service named "on" concatenated with the id property of the component. In an embodiment, the operation is called with the data relevant to the component.
[0145] In an embodiment, tags that descend from UIComponent and RadioButtonGroup within Flex are used. MXML authoring is further described in Adobe developer network documents relating to Flex. In an embodiment, the cue point has the following attributes:
Figure imgf000028_0001
[0146] In an embodiment, the cue point has the following elements:
Figure imgf000028_0002
Figure imgf000029_0001
[0147] 5.2.6 PROGEND AND RETURNEND CUE POINTS
[0148] In an embodiment, progEnd end returnEnd cue points define the end of a video segment and upon reaching the cue points, the player stops playing video and does not provide a rewind option. There can multiple progEnd' s in a media program.
[0149] In an embodiment, the returnEnd cue point is used at the end of a segment. Reaching a returnEnd causes a jump to the point that initiated the jump to the start of the segment. In an embodiment, the returnEnd and progEnd cue points have the following attributes:
Figure imgf000029_0002
[0150] In an embodiment, the cue point has the following elements:
Figure imgf000029_0003
[0151] 5.2.7 WEBFOCUS CUE POINT
[0152] In an embodiment, a webFocus cue point can specify a URL for a web point and, with a story element, associate an image and text (e.g., for a call out) with the web URL. webFocus cue points can be used as targets in modalStoryBranch cue points and insertPt cue points. webFocus cue points can appear in directories. webFocus cue points can have a gotoWebPointEndName attribute value to specify what to show at the "end" of a webFocus. [0153] In an embodiment, during playback, a user indicates that the user is at the "end" of a webFocus by selecting a "back" browser function or by selecting a "TV" button. If the video media is in an operational state, the player switches to the video, maintaining the playhead time and play/pause status. If the video is not in an operational state because, for example, a zeroLen cue point has been reached, the player executes a goto to the media point specified by the gotoWebPointEndName. [0154] In an embodiment, the cue point has the following attributes:
Figure imgf000030_0001
[0155] In an embodiment, the cue point has the following elements:
Figure imgf000030_0002
[0156] 5.3 OTHER LANGUAGE ELEMENTS AND ATTRIBUTES [0157] In an embodiment, the cue point language defined herein has the following elements: [0158] cuePt elements have the following attributes:
Figure imgf000030_0003
[0159] In an embodiment, a cue point has the following contained elements:
Figure imgf000031_0001
[0160] In an embodiment, a cpGroup is used anywhere a group of cue points is needed. The cpGroups are made up of some attributes and a collection of targetSeg elements. A targetSeg contains a cue point name and an optional association attribute.
[0161] Some cue points, for example insertPt, use cpGroups where each cue point in the cpGroup has an association that is used to select the cue points. In operation, the player searches the table to match the string provided as a key with the association attribute and then returns the cue point name contained in the first match. Thus, a many-to-one matching is performed. The key may come from a web service as explained in the insertPt cue point section. As an example, with the following targetSeg' s in a cpGroup:
Figure imgf000031_0002
[0162] The following matching results would occur: [0163] J, Jack, ack all match A [0164] Jill matches B, and never gets to C [0165] John, Jerry, Jill, ill, Jer, err all match C [0166] ZZZZ (or anything or nothing) matches D (because after looking for the string key, the player attempts to match the string "default" as a key.
[0167] In an embodiment, the cpGroup has the following attributes:
Figure imgf000031_0003
Figure imgf000032_0001
[0168] In an embodiment, the cpGroup has the following contained elements:
Figure imgf000032_0002
[0169] In an embodiment, a targetSeg may be used to specify a cue point name and optionally associate it with an association attribute. When a group of cue points are needed, e.g., a modalStoryBlock where the user makes the choice, the association attribute can be omitted. In an insertPt the association attribute is needed to determine which cue point to goto. The association attribute can be of the form "aaa, bbb, ccc" where each substring would match the cuePointName. See cpGroup for an explanation of how the association attribute is used to select a cuePointName. [0170] In an embodiment, the targetSeg has the following elements:
Figure imgf000032_0003
[0171] In an embodiment, a mxmllnCPL element may be used to hold executable MXML code. There are no sub-elements and attributes defined. Instead, a user can include anything that descends from the UIComponent in mx:MXML. An example definition is now provided, followed by comments:
1. <mxmlInCPL> a. <mx:MXML id="whatever" xmlns:mx="http://www.adobe.com/2006/mxml">
1. <mx:Canvas xmlns:mx="http://www.adobe.com/2006/mxml" width="600" height="440" id="uberContainer">
2. <mx:Button label=" button 1" id="buttonl" click="onMXML(event) "/>
3. </mx:Canvas> b. </mx:MXML>
2. </mxmlInCPL>
Line Note
1 and 2 Enclose the MXML with the mxmllnCPL tag. l.a MXML, an element defined by Adobe; you need line l.a, as it is, but with whatever id you choose. l.a.l Any container (canvas, VBox, etc.) that will contain all the other components. With the fixed name (remember: quick and dirty) "uberContainer". l.a.2 Any id you want for the component and then the operation within the web service is "on"+ the id. Here you'd build a server side handler with the function name "onbuttonl". Any event can be handled (here it is "click") but all components call the fixed name "onMXML" as the handler.
[0172] In an embodiment, a progLevelMetadata element is required. It contains data associated with the overall program. Example attributes include:
Figure imgf000033_0001
[0173] Example contained elements include:
Figure imgf000033_0002
[0174] In an embodiment, a story element packages data used to display a cue point (web or video). Example attributes include:
Figure imgf000034_0001
[0175] In an embodiment, a gotoAutoReturnButton element and cue point support a "more info" operation. Example attributes include:
Figure imgf000034_0002
[0176] 5.6.2 ANNOTATIONS
[0177] In an embodiment, an annotation element is used to display a graphic on screen. The graphic can be actionable. An annotation element is a graphic object that appears on screen starting at a cue point; when the next cue point is processed the annotation is removed (although it could be reapplied). It is used to overlay the display with a graphic while in video view and may optionally implement a goto behavior in response to a click. The structure of the annotation element is similar to cuePoints in that it contains the same targetList and story elements. Clicks on an annotation can cause three things to happen, depending on the value of the clickBehavior attribute. See the attribute description. Example attributes include:
Figure imgf000034_0003
Figure imgf000035_0001
[0178] In an embodiment, an audiovisual work may include one or more annotations that specify interactions available to a viewer. Annotations may comprise graphical images, buttons, text messages, labels, and other elements that may be displayed in a variety of locations overlaid on a video segment or near a video player window that is showing a video segment. One or more annotations may be assigned to a cue point; when the cue point is reached during playing, the annotations are activated and remain active until the next cue point. Annotations have flexible attributes relating to where they can be shown, what they can show, and how they behave. Graphic images associated with annotations may include images such as PNG and JPEG files, or SWF files or any other files that can be rendered on the system on which the player logic 112 is hosted.
[0179] In an embodiment, an annotation has one of four types: decoration; goto; returnEnd; and overlay. (Details of annotation types are disclosed in the Appendix.) Annotations may be displayed as static graphical images or animated graphics. Annotations may be positioned anywhere in the video windows that the player logic displays during playing. [0180] In an embodiment, annotation frames allow placement of an annotation outside of the video window; an annotation frame can provide a larger area outside the video in which annotations can appear, without covering up the video. In an embodiment, a user may use the video linking editor logic 112 to define an annotation frame as a rectangle within which the video window is placed. If the annotation frame is larger than the video frame, then space is displayed around the video and annotations can be placed in the resulting space without obscuring the video. With annotation frames, an author is not required to re-encode a video segment to create space to place annotations.
[0181] A "goto" annotation may be associated with a target and one of several different kinds of return behavior; a target specifies where the player branches when a viewer clicks on the annotation, and the return behavior specifies where the viewer returns after viewing the video or web page associated with the annotation. For example, the return behavior of a goto annotation may be set to "Skip." With skip on return behavior, after a viewer returns from the annotation's target video segment or web point, the player skips to the next cue point after the one that includes the goto annotation.
[0182] An annotation of any type may be configured with modal behavior. A modal cue point has two different playback modes comprising an initial entry mode and a return or overlay completion mode. When an annotation is modal, each annotation type causes the video player logic 112 to operate differently depending upon the then-current mode, as defined by how the player arrived at the associated cue point. For example, initial entry mode refers to the player arriving at the cue point via normal program flow, or as the result of a direct jump. In initial entry mode, the video player logic 112 is configured to display all annotations that are configured as modal, pause, and wait for the user to select a non-decoration annotation, such as a "goto" annotation or a "returnEnd" annotation. In contrast, return or overlay completion model occurs when the player returns to the cue point via a returnEnd cue point or annotation after a jump from it, or when a viewer selects the Continue button to close an overlay data entry form. (Further details are provided in the Appendix.)
[0183] FIG. 23 is a screen display diagram of an example Editor window in which an Annotation tab is selected. In the example, screen display 2102 includes Annotation tab 2302. Selecting an Add Annotation (+) control causes the editor logic 110 to display a default annotation name and type in fields 2304; user input may modify the annotation name and type, so that the annotation may be referenced in other metadata using a convenient name. Editor logic 110 also displays a data entry panel 2306 that may receive values defining particular parameters of an annotation including screen position values (X position, Y position). Parameters also may include a text label for the annotation, a graphic image to display as the visible form of the annotation in the player, and mouse over image. The mouse over image is a different graphic image to display if a user moves a pointing device over the annotation while using the player and when the annotation is displayed. A Boolean parameter selected in the editor using a checkbox may specify whether to display an annotation icon. [0184] 5.4 CONTENT TYPES
[0185] A content type value associated in metadata with a cue point causes differentiated operation of the metadata-capable video player logic 112 at the time of playback. In particular, within the player the content type zeroLen is treated differently than all others (ad_Inline, segment_Inline, prog_Inline, prog_Segment). For example, ad_Inline and ad_Segment are used to skip advertising content coming back from an insertPt. [0186] 5.5 AUTOMATIC CREATION OF CUE POINTS
[0187] In an embodiment, a computer program can create one or more cue points and store the cue points in a metadata file, rather than a user obtaining cue points from encoded video, or the user creating the cue points using the video linking editor logic 110. In an embodiment, cue points can be added, updated or completely replaced dynamically using web applications, processes, or other computers that are coupled to computer 102. For example, the unique identifier of a television program, as specified by Society of Cable Telecommunications Engineers, could be used in an update message providing new cut points. [0188] In another example, one or more computer programs can access video and other content databases and use the information gather to generate interactive video experiences based on the cue point language schema that is defined herein. As one example, a Perl script may be configured to access YouTube metadata APIs to construct an interactive video experience based on playing all video matching a particular keyword. In this example, the script may be configured to issue an HTTP-based query to a YouTube server, in which the query conforms to YouTube' s APIs, to retrieve a list of all stored videos that include a particular keyword in the metadata maintained by YouTube for the stored videos. In response, the YouTube server sends a responsive dataset. The script may be configured to identify a URL for each video on the YouTube servers that is identified in the responsive dataset, and to write a metadata file 126 that specifies an audiovisual program consisting of a concatenation of all the matching videos. The script could be configured to automatically generate a plurality of annotations, in which each annotation graphically represents a first frame of a different one of the matching videos. In this manner, at playback the user would see a visual menu of each matching video and could activate any desired video by selecting on the image associated with an annotation for one of the videos. [0189] In another example, a program is configured to receive a user query for a particular keyword or phrase and to search a database of movie metadata for matches to the user query. For each match to the user query, an associated database record is selected and retrieved. From each database record, the program retrieves a URL of a video that is stored in third party hosted storage, such as YouTube. The program creates and stores a metadata file 126 that plays the matching videos. For example, the program could be configured to receive a user query to find all video clips in which a character says "Bond, James Bond", assuming such phrases are represented in the database of movie metadata.
[0190] In another example, a computer program may be configured to create multiple metadata files 126 based on a single video. For example, a Perl script may be configured to generate multiple versions metadata files 126 for a single video in which each metadata file 126 comprises definitions of annotations for subtitle data in a different language, and the subtitle data is displayed at playing time using the annotations as the subtitle display widget. Additionally or alternatively, automatic creation of cue points may take user behavior into account to create customized cue points for a particular user based upon what is known about the user's behavior as represented in server-side stored data. User behavior can include information what previous cue points have been selected, the elapsed time between selections, whether certain video segments have been skipped, navigation paths as represented by user selections of different video segments in succession, etc.
[0191] Thus, embodiments provide flexible means to use output from a database, coupled to a script or other program, wherein the output is optionally selected based on matching user input or queries, to result in automatically creating and storing one or more metadata files 126 which, when played using the video player logic 112, result in displaying enriched interactive videos. While certain examples have stated that the program may cause displaying a concatenation of videos matching a query, concatenation is not required. Instead, a program or script may have any level of complexity and may be configured to write a metadata file consisting of any number of cue points, annotations, or other information based upon the language description that is provided herein. In this approach, metadata may be created dynamically and transmitted to the player over a network connection without storing or saving the metadata in file format. Further, the examples provided herein are merely representative and countless other applications are possible.
[0192] 5.6 DIRECTORIES
[0193] A directory comprises, in one embodiment, a selectable, scrollable column on the right part of the video display that appears at cue point boundaries and for a specified period of time, such as four (4) seconds, in response to a movement of a pointing device. FIG. 10 illustrates an example screen display that includes a directory.
[0194] Player logic 112 attempts to generate and display a cue point specific, non-modal directory on a cue point by cue point basis. The media points (video and web) within the directory are specified as a cpGroup and must contain story elements if they are to appear in the directory. These points can be whatever the author chooses to make them and are an opportunity to guide the user into interesting, tangentially related information. For example, in a news show, when a story about Great Britain is shown the directory could contain the related online encyclopedia entry and several video segments; when the news program shifts to the next story, the cue point specific directory changes.
[0195] 5.7 WEB SERVICES
[0196] In one embodiment, web services may be implemented using a ColdFusion web server. In an embodiment, web services are called with two string arguments comprising the called operation or function and the type of service. The web service returns a string with three fields comprising an operation specific field (e.g., "serviced" for MXMLOverlay calls), a result, and the type of service string.
[0197] 5.8 DYNAMIC LAYOUT WITH MULTIPLE RECTANGLES
[0198] In an embodiment, an author may customize the positioning of a video window and a web window within an overall player window. In an embodiment, dynamic layout is accomplished through user interaction with the video linking editor logic 110.
[0199] A user selects a Dynamic Layout feature under a Layout tab 2104 of an editor screen display 2102 as seen in FIG. 21. FIG. 21 illustrates an example screen display that the video linking editor logic generates and causes displaying. An author selects a window size for the video to be displayed as part of an enriched video program. For example, a window size may be
1024 pixels wide by 763 pixels tall. Generally, a user selects a new layout control to create a new layout and assigns a unique name to the new layout. The author selects a Positioning function and may select one of a plurality of predetermined layouts of the video window, web window, and static surrounding graphical display space. The user may change the size of the video window or web video using an Advanced tab function. The user may change dimensions in pixels for video width, video left position, video horizontal center position, and video right position. The editor logic stores the changed values in association with the layout name. Changing numeric values of dimensions later results in changing the position of a video window when displayed using the player logic. Each layout may have restrictions on repositioning based on the original layout; in an embodiment, the editor logic 110 prevents the user from entering data for parameters that do not fit a particular layout.
[0200] FIG. 25 is a screen display diagram of an example Editor window in which a Layout tab is selected.
[0201] In an embodiment, example Editor window 2102 comprises a Layout tab 2502 that displays a list of names of selected layouts. Selecting an Add Layout control 2503 causes the editor logic 110 to add a new layout name to the list. In an embodiment, logic 110 can access stored data defining a plurality of predefined player window layouts, which are displayed in an information panel 2504. In each predefined layout, a relative size and position of a video window to be shown in the player window is indicated by a rectangle having a first color, and a relative size and position of a web browser window to be shown in the player window is indicated by a rectangle having a different, second color. In some layouts the video window has a reduced size as compared to a size of the browser window. In some layouts the video window is the same size as the browser window. In some layouts a background is defined that is logically behind or surrounds both the video window and browser window. In some layouts the video window is laterally or longitudinally adjacent to the browser window. In some layouts the video window is offset in a corner of the browser window, or centered.
[0202] In an embodiment, selecting a predefined layout from panel 2504 causes editor logic
110 to display an enlarged view 2510 of the selected layout in which the relative size and position of the browser window 2508 and video window 2506 are shown. The author also can further customize the layout to obtain different effects using the parameters accessible using an
Advanced tab of the editor window as shown in panel 2504.
[0203] A layout may be linked to a particular cue point. In an embodiment, a user selects a
Cue tab in the editor screen display and selects a cue point to link to the layout. The user may select a Description tab in a Parameters pane and select a Browse button next to the Web View
Layout and the user may select the Layout that the user created.
[0204] In this approach, an author has control over the location of a video window and web window. Further, a particular layout that the author deems aesthetically preferable for a particular combination of video and web content may be injected into the metadata so that the layout changes appropriately when a particular cue point is reached.
[0205] 5.9 CUE POINT LANGUAGE EXAMPLE
[0206] TABLE 1 presents an example of a complete metadata file of the type that can be created and stored as one of the metadata files 126.
TABLE 1 - CUE POINT LANGUAGE EXAMPLE
Figure imgf000041_0001
[0207] In an embodiment, the base element is MediaProgram and encloses all other elements. The element progLevelMetadata is required and specifies information that applies to the whole MediaProgram. In the sample code above, in the cue point named B, the cueType is "insertPt" which jumps to a cue point (in this case D) while establishing a return point. In B, the target segment is specified within a cpGroup (a cue point group); in this case it has only one target and the association attribute is "default". There is nothing to check and there is only one place to jump. In E, the target segment is a cueType=returnEnd which means it will return to where it came from (rather than goto another target). Further, anytime that no video immediately follows a cue point, the cue point has a contentType = "zeroLen" (cue point C is also zeroLen). [0208] TABLE 2 presents an example schema for the Cue Point Language.
TABLE 2 - EXAMPLE SCHEMA
Figure imgf000043_0001
Figure imgf000044_0001
Figure imgf000045_0001
Figure imgf000046_0001
Figure imgf000047_0001
6. PLAYING VIDEO AND LINKED MEDIA (THE COINCIDENT PLAYER)
[0209] 6.1 TRICK PLAY FUNCTIONS, TIMELINE, ALWAYS-AVAILABLE WEB
LINK
[0210] FIG. 11 illustrates an example screen display that illustrates a player screen that may be generated and displayed in a computer display unit by metadata-capable video player logic.
The video display unit may comprise a computer monitor, video monitor, digital TV, CRT, or other display unit that is driven from an appropriate output of computer 102.
[0211] In various embodiments, the player screen display may be implemented as an application that is displayable within a web browser, or using standalone application program logic; in either case the player is not video specific and will work with various existing video formats (Flash, Silverlight, QuickTime, etc.) and can be adapted to new video formats as they are defined.
[0212] FIG. 20 illustrates an example arrangement of digital computer elements that can be used to implement certain embodiments with a browser-based player for enriched video programs. In an embodiment, a computer 102 is coupled directly or indirectly through one or more networks 120 to a web server 130 and optionally to a file server 132. In various embodiments, network 120 may comprise a local area network (LAN), wide area network (WAN), an internetwork, or a combination. Web server 130 hosts one or more video files, HTML documents, HTTP servers or application servers, or other web content. File server 132 stores or hosts video files 122, graphics files 124, and metadata files 126, which may include or be associated with HTML and browser executable program code, such as JavaScript code. Optionally file server 132 stores or hosts script files 2008 that can issue queries to a database 2010 and automatically generate the contents of one or more metadata files 126 based on result sets received from the database in response to the queries.
[0213] Computer 102 hosts or executes an operating system 104 that supervises I/O, storage management, and execution of application logic. In an embodiment, computer 102 optionally comprises a video editor 106; as indicated by broken lines, the video editor may be omitted. In an embodiment, computer 102 comprises a browser 108 that hosts or can access a support library 2002. Commercially available examples of support library 2002 include Macromedia Flash and Silverlight.
[0214] In an embodiment, computer 102 is coupled to storage 140, which broadly represents any data storage device, storage area network (SAN), network attached storage (NAS), or network file system (NFS) unit or server. Storage 140 may reside on network 120 or on a server coupled to the network. Storage 140 stores application programs but is not required to store video files or metadata files; instead, video files may be received through streaming video delivery from file server 132 and metadata files 126 may be received on the fly directly to browser 108 or support library 2002 under control of an instance of metadata-capable video player logic 112.
[0215] In an embodiment, computer 102 optionally comprises video linking editor logic 110, which may be omitted entirely as indicated by broken lines. In an embodiment, a separate player control server 2004 comprises metadata-capable video player logic 112 and may comprise accounting logic 2006. The metadata-capable video player logic 112 is generally configured to open metadata files and associated video files, and to play the video files while interpreting and responding to links and related information and instructions in the associated metadata files. Other more specific functions of metadata-capable video player logic 112 are described in other sections herein. In an embodiment, player control server 2004 controls delivery of instances of the player logic 112 to authorized clients, and in certain embodiments interactions with accounting logic 2006 determine whether a particular client in the form of computer 102 can receive an instance of the player logic. Additionally or alternatively, accounting logic 2006 determines amounts for invoices, other billing, or other charges to a video producer, studio, content owner, or other party that owns or controls the file server 132 and its contents.
[0216] In another embodiment, computer 102 comprises player logic 112 and does not have an editor such as editor logic 110; such an embodiment might be used by an end user who is viewing video programs that have been prepared by someone else, and who does not use a browser to view video programs based on receiving the player logic over a network from a server computer as described above.
[0217] In one embodiment, an end user or viewer invokes browser 108 and connects to web server 130, which offers links to play audiovisual media such as video files 122. The viewer selects a link for a particular video file 122. In response, the browser 108 downloads from the file server 132 one or more elements of HTML and browser executable program code, such as JavaScript code, which the browser executes. Consequently, the browser 108 renders a page in the display unit of computer 102. The rendered page includes code to invoke an instance of metadata-capable video player logic 112. The player logic 112 accesses one or more metadata files 126, and accesses video files 122. The video files 122 may be on file server 132, or stored in another networked location, or on a third party server or quasi-public hosting site such as YouTube. Based on instructions in the associated metadata files 126, the player logic 112 then streams the video files 122 and provides metadata from metadata files 126 to the support library 2002 of browser 108. As a result, one or more of the player screen displays described herein appears and can play video within the browser 108 in the manner described herein. [0218] In an embodiment, each time that browser 108 invokes use of the player logic 112 data is recorded at the player control server 2004, or at a third party server site, to indicate the invocation. Invocation data may include data identifying a referring web site, that is, the web site at which the end user selected a video for playing, such as web server 130. Invocation data also may identify a producer of the video, if the producer is different than the owner or operator of the referring web site.
[0219] In an embodiment, the end user of computer 102 may be denoted a first party; a second party may own or operate web server 130 at which the first party selects videos for playing, and the second party may comprise a producer of the videos; and a third party may owner or operate the player control server 2004 and may control delivery and use of instances of the player logic 112, and may be entitled to payment from the second party for each use of the player by the first party or each stream that the player facilitates delivering from the second party to the first party. Thus, a copy of the player logic 112 or other browser executable code may be delivered from the third party to first party browsers only a specified maximum number of times per day, week, month or year in consideration for payment of a specified fee attributable to each day, week, month or year. In an embodiment, if the specified maximum number of first party video player invocations is reached, then the third party may cease providing additional first parties with access to the browser executable code that implements or accesses the player. Additionally or alternatively, the third party may deliver the browser executable code to an unlimited number of first parties who select videos at the second party's web site and may invoice the second party for a variable amount that is based upon or proportional to the actual number of first parties.
[0220] In this arrangement, the invocation data is recorded in a database that is owned or operated by the third party. The third party configures one or more computer programs to periodically analyze or compile invoicing data from the database, based on the number of streams that the second party delivered using the player or the number of first parties who connected and used the player. Based on the data analysis or compilation, the third party may invoice the second party. In all such arrangements, the third party retains control over use of the metadata-capable video player logic 112 and its use by second party producers or first party end users, and the third party is entitled to collect fees or revenue from one or more of the second party and/or the first party in consideration for the use of the metadata-capable video player logic 112 to provide enriched videos to end users.
[0221] In another embodiment, computer 102, logic 112, and a video display unit may form a special-purpose computer performing the functions described herein.
[0222] In one embodiment, a player as in FIG. 11 comprises a video display window 1102, trick play icons 1104, a timeline 1106, and a web hyperlink 1108. The video display window 1102 displays a video segment of a media program. The trick play icons 1104 may be selected through user input from a pointing device or remote control. In one embodiment, trick play icons 1104 provide functions for video playback, fast forward at one or more speeds, and rewind at one or more speeds. Other controls may be provided including an end playback or "eject" control, an audio volume adjustment control, and a video window size control. [0223] In an embodiment, the timeline 1106 provides a graphical indication of the player's current position within a video segment, the position of cue points, and the relationship of branches to other cue points and other video segments. For example, in one embodiment the timeline 1106 graphically displays cue points as dots or circles, branches to other cue points as arcs, and video segments as straight lines. The lines, dots, and arcs are arranged in a temporal order so that the first video segment is arranged at the far left side of the display and the last cue point of the last video segment to which a branch can occur is displayed at the far right. As the player plays video, a graphical icon in the timeline 1106 moves from left to right in proportion to the time that has elapsed during playback or the amount of video that has been played. As cue points are reached and branches are traversed, the player logic 112 modifies the video display unit to update the timeline to indicate a user's current logical position in a media program as a physical icon shown among the lines, arcs and dots. Therefore, the timeline 1106 enables a user to visually identify upcoming cue points, branches, and branch destinations. [0224] In an embodiment, web hyperlink 1108 is continuously displayed in the screen display in an overlay manner over any video program that is shown in video window 1102. Thus, the web hyperlink 1108 is always available during operation of the player logic 112. In an embodiment, selecting the web hyperlink 1108 causes the player logic 112 to modify the display unit so that the video display window 1102 is redisplayed in a reduced size format, for example, in a small rectangular window at the bottom right corner of the screen display. Further, the video display window is overlaid on a web browser window that displays web content associated with the web hyperlink 1108. In this manner, the player logic 112 appears to generate a picture-in-picture form of display in which the background picture shows web content and the foreground, reduced size picture shows the video program. The video program continually runs during such a transition.
[0225] In an embodiment, the screen display of FIG. 11 further comprises a TV button which, when selected, causes the player logic 112 to restore the video display window 1102 in a large size as seen in FIG. 11 and to discontinue displaying web content. [0226] 6.2 KEYBOARD CONTROLS
[0227] In an embodiment, computer 102 uses either a remote control or a computer keyboard to provide user input to the metadata-capable video player logic 112. [0228] In an embodiment, user input selecting hot keys on the keyboard results in controlling playback. In an embodiment, the following key commands cause the metadata- capable video player logic 112 to perform the following functions:
Figure imgf000051_0001
Figure imgf000052_0001
[0229] 6.3 PLAYBACK APPLICATIONS
[0230] Various embodiments facilitate production of enriched audiovisual programs that combine Internet web content and video content. Examples of playback applications are now described.
[0231] FIG. 13A illustrates an annotation coupled to a web service providing automated text messaging in association with an enriched video program. In an embodiment, metadata-capable video player logic 112 displays a player window 1302 on a computer desktop or within a browser. Player logic 112 is configured to generate the player window 1302 and to facilitate the functions that are further described herein for FIG. 13A. Player window 1302 includes playing, segment, and chapter navigation buttons 1310 which when selected cause playing a video segment, performing trick play functions, or skipping to other defined chapters.
[0232] In an embodiment, buttons 1310 may be associated with an HTML document that applies a specialized appearance or skin to the buttons 1310. In an embodiment, skinning buttons 1310 is performed using the editor logic 112 to display editor window 2102 (FIG. 21), selecting the Metadata tab 2112, selecting a Skin Buttons field and entering an HTML URL.
With button skinning, buttons 1310 may have a different appearance in different videos at playback; for example, comparing FIG. 13A, FIG. 17A shows buttons 1310 with different styles and appearance.
[0233] Player window 1302 includes an audio icon 1312 which when selected causes muting sound from the video and a full screen icon 1314 which when selected causes displaying the video in full screen mode. In response to appropriately defined annotations and cue points associated with a video program, which in this example is an excerpt from a program named "The Hills," metadata-capable video player logic 112 causes displaying an annotation 1300 that prompts a viewer to enter a viewer's name, phone number, and gender in data entry fields 1304, 1306, and using radio buttons. In an embodiment, when a viewer enters values in the fields and selects the Go button, metadata-capable video player logic 112 temporarily stores the values in memory for referencing and use by other logic when a particular cue point is reached that calls for invoking a text messaging function.
[0234] FIG. 13B illustrates a frame of an enriched video program as displayed in a player window. In an embodiment, player window 1302 as previously seen in FIG. 13A is displaying a video segment depicting a character 1320 who is using a text messaging device 1322. In the example of FIG. 13B, player window 1302 further comprises show and character icons 1324, web site icons 1326, and service icons 1328. In an embodiment, a cue point associated with an invocation of a web service may be defined for a time point of the frame illustrated in FIG. 13B. When the video program is played and the frame illustrated in FIG. 13B is reached, the metadata-capable video player logic 112 is configured to invoke a web service that can retrieve the stored value of the phone number that was received as user input at FIG. 13 A, and dispatch a specified text message to that phone number. The specified text message may comprise information appearing to come from character 1320. The video editor linking logic 110 may be used to define the cue point that can cause a specified text message to be sent automatically when the cue point is reached during playback.
[0235] In an embodiment, show and character icons 1324 each comprise a graphical image that is associated with an annotation. In an embodiment, a first one of the show and character icons 1324 is an annotation associated with a URL for a web site of the show, which in the example of FIG. 13B is the MTV show "The Hills," that provides further information about the show. In an embodiment, second and third ones of the show and character icons 1324 each comprise annotations that are associated with sequences of video segments relating to the characters that are depicted in the icons. In the example of FIG. 13B, selecting the "Heidi" icon causes the metadata-capable video player logic 112 to branch within the associated metadata file 126 to a point associated with a sequence of video segments that feature the character "Heidi." Playing the video program then continues with the sequence of segments that feature "Heidi." Similarly, selecting the "Audrina" icon causes the metadata-capable video player logic 112 to branch within the associated metadata file 126 to a point associated with a sequence of video segments that feature the character "Audrina."
[0236] In an embodiment, web site icons 1326 provide linkages to Internet sites that feature social networking and other services. For example, in an embodiment the video linking editor logic 110 may be used to create an annotation, symbolized by a Twitter icon, which is associated with the Twitter service and a web service to invoke the Twitter service. Thus, in one example embodiment, at playing time, when a viewer selects the Twitter icon, the metadata-capable video player logic 112 generates and displays a new window that contains a feed of Twitter posts relating to the video program of FIG. 13B. The other web site icons 1326 similarly each comprise an annotation that is associated in metadata files 126 with a web service, URL or other reference to executable code that can cause integration and use of the web service that is represented by the icon.
[0237] In an embodiment, each of the service icons 1328 is an annotation represented by a graphic image that provides access to an external service or web site. For example, in one embodiment, a music purchase icon may comprise an annotation that is associated with a web site that features downloads of songs, as further described herein for FIG. 14. In an embodiment, a commercial sponsor icon may comprise an annotation that is associated with a commercial advertising web site or online information about a commercial product. Additionally or alternatively, the target of an annotation that is displayed as a commercial sponsor icon may be a video program segment comprising a commercial for a specified product. In the example of FIG. 13B, selecting the Dos Equis service icon causes the metadata-capable video player logic 112 to branch to and play a video segment containing a commercial for Dos Equis brand beer.
[0238] FIG. 14 illustrates a frame of a video program having a highlighted service icon. In the example of FIG. 14, video window 1302 is displaying a frame 1402 of a program that includes background music at the time of playback. A first service icon 1404 comprises an annotation that is associated with a highlighted graphic image as indicated by short lines radiating from the icon; in contrast, in the example of FIG. 13B, the same one of the service icons 1328 is not highlighted. In the example of FIG. 14, the highlighted icon signifies that the song that is then currently playing in the background of the scene of frame 1402 is available for purchase or downloading. If a viewer selects the first service icon 1404, then in response, the metadata-capable video player logic 112 accesses and displays a web site that offers the associated song for downloading or purchase. To implement such a function, an author may use video linking editor logic 110 to associate a specified web service, URL, or program code with an annotation and graphic image for the service icons 1328. The URL may be a complex URL that includes a domain name, service name or script name, and one or more embedded parameters or attributes. For example, an attribute of the URL may be set equal to a file name for the song that is known to play at the associated cue point. Thus selecting the first service icon 1404 causes the metadata-capable video player logic to invoke the URL specified in an associated annotation, effectively passing the name of the then-currently playing song to a third party web site, which extracts the song name and can offer the song identified in the parameter for purchase or downloading.
[0239] FIG. 15A illustrates an annotation that provides a user choice. In the example of FIG. 15A, video window 1302 displays a plurality of annotations 1502 in a video window 1506. First, second, and third annotations labeled Heidi, Spencer and Audrina are associated with static graphic images of the named characters and are linked to a target cue point for a sequence of video segments that feature the associated character. A fourth annotation comprises a graphical image prompting the user to select one of the characters as a favorite character. In response to user input selecting one of the first, second or third annotations labeled Heidi, Spencer or Audrina, metadata-capable video player logic 112 branches within the metadata files 126 to instructions associated with playing a sequence of video segments that feature the selected character. For example, if Spencer is selected then the metadata-capable video player logic 112 branches and begins playing a first segment of video featuring the Spencer character, as represented by the frame of FIG. 13B.
[0240] In an embodiment, when a particular character is selected as a favorite character, then the video segments featuring that particular character are also authored to include annotations identifying the other, non-selected characters, for possible future selection. For example, as seen in FIG. 13B, in a video segment in which Spencer has been selected as featured character, the show and character icons 1324 display only icons for annotations associated with other characters, namely Audrina and Heidi.
[0241] In contrast, FIG. 15B illustrates a frame of a video segment in a sequence for which Audrina is the featured character; therefore, show and character icons 1324 depict Heidi and Spencer, but not Audrina, and the icons are associated with annotations which, when selected, cause playing sequences of video segments featuring Heidi or Spencer, respectively. FIG. 15B also illustrates different service icons 1328 in which a third service icon is associated with a different commercial product or retailer. Thus, an author using video linking editor logic 110 may define different annotations in the position of service icons 1328 for different commercial products, merchants, retailers, or other web sites or service providers in association with different cue points arising at different time points in a program. For example, an annotation associated with a graphic image or icon depicting a first merchant or product may be associated with a cue point at the start of a first scene of a video program that somehow involves, uses or shows the associated product, which a second merchant, product or service may be associated with a second cue point at another point in the program that shows, uses or involves the second merchant, product or service.
[0242] FIG. 16 illustrates concurrent playing of an enriched video program and displaying an associated web page. The video player window 1302 comprises a video window 1602 that is overlaid on a browser window 1610. The video window 1602, in the example of FIG. 16, is displayed in reduced size but contains the same navigation icons 1310 as in FIG. 13A. The video window 1602 is configured to play a streaming video program. The metadata-capable video player logic 112 is configured to concurrently cause playing a streaming video program in the video window 1602 and to display a web page 1612 associated with the video. [0243] For example, an author using the video linking editor logic 110 may define a cue point at the frame shown in FIG. 16, which is from a title scene in the show "Glee" that depicts the names of actors who portray characters in the show. At the frame of FIG. 16, the name of actor "Cory Monteith" is displayed. A cue point may associate the time of that frame with a URL for an Internet Movie Database (IMDB) page containing information for the named actor. As the video continues to play in video window 1602, the metadata-capable video player logic 112 may reach other cue points referencing other URLs. At each cue point, the metadata- capable video player logic 112 accesses a referenced URL and causes the browser window 1610 to display the referenced web page. In this manner, cue points defined for a video segment can cause web content to be "pushed" to a browser window that underlies the video window. Content in the browser window thus changes dynamically as the video plays and as specified cue points are reached by the player.
[0244] The foregoing applications and others provide the capability to display video over web content or to display web content in association with video in entirely new and different ways. As a first example, embodiments provide the capability to display video in a "picture in picture" layout in which a video plays in a video window 1602 that is reduced in size in comparison to a browser window 1610 that is concurrently showing related web content. The metadata-capable video player logic 112 is configured to allow the end user to watch video and scroll web content in the same screen without tabs or special windows. The author of the metadata files 126 for the program has control of whether the video in video window 1602 plays or pauses, and what is rendered in the video window 1602 and the browser window 1610. [0245] For purposes of illustrating a clear example, FIG. 16 shows a first rectangle comprising video window 1602 and a second rectangle comprising browser window 1610. In other embodiments, any number of rectangular display areas for video or browser content may be provided.
[0246] As another example, annotations can be configured so that invoking the Twitter web site icon 1326 causes the metadata-capable video player logic 112 to display a third rectangle to the right of the video window 1602, while maintaining a display of the browser window 1610 conceptually behind the other rectangles or windows. The third rectangle displays a feed of Twitter posts using HTTP data transfers and rendering of HTML within the third rectangle. In this manner, a streaming video may be played at the same time that an HTML window is dynamically updated. Both the video window 1602 and the browser window 1610 have equal conceptual weight within the player window 1302.
[0247] FIG. 17A illustrates an example of playing an enriched audiovisual program with annotations that implement chapter selections. Player window 1302 displays a graphical background 1702 that surrounds a video window 1704 that displays a video program, based on an associated metadata file 126. A plurality of enriched program navigation icons 1708 and chapter selection images 1706 are displayed over the video in the video window 1704. In an embodiment, each of the enriched program navigation icons 1708 and chapter selection images 1706 is an annotation as described herein, associated with a particular position, graphic image or animation, and operational behavior. In an embodiment, enriched program navigation icons 1708 include a Back navigation icon and a Home navigation icon, comprising annotations that associate static graphical images. Selecting the Back navigation icon causes the metadata- capable video player logic 112 to branch to a prior video program that was previously played or a web page that had been previously displayed in a browser window area of the player window 1302. Selecting the Home navigation icon causes the logic 112 to branch to a starting video segment representing a home position of the video program.
[0248] The chapter selection images 1706 each represent an annotation that is associated with a branch to a different cue point in the video program associated with a different video segment for a chapter, episode, or other discrete video element. During playing, selecting one of the chapter selection images 1706 causes the player logic 112 to branch to and start playing an associated video segment.
[0249] The example of FIG. 17 A indicates an aspect of the flexibility inherent in the concept of annotations as described herein. Both the icons 1706, 1708 can be represented using annotations that define different positions, graphic images and operational behavior. However, even though the annotations are different, an author is not required to learn and use a large number of different programming techniques; instead, the same features and functions are used to define all the annotations.
[0250] FIG.17B illustrates playing an audiovisual program in which annotations have multiple different forms and provide multiple different functions. In an embodiment, player window 1302 comprises the graphical background 1702, video window 1704, and enriched program navigation icons 1708 as described for FIG. 17A. Additionally FIG. 17B features a navigation animation 1710, web integration icons 1712, topic launch icons 1714, and menu access link 1716. In an embodiment, navigation animation 1710 represents an annotation that is associated with an animated graphical object and a plurality of cue points for each of a plurality of characters shown in the animation at different positions within the animation. For example, during playing, a viewer can use a pointing device to cause a cursor to hover over the navigation animation 1710, and in response, the navigation animation scrolls graphically left or right in response to movement of the pointing device. Selecting a particular region showing a particular character causes the player logic 112 to branch to a chapter of the video program that features the selected character.
[0251] The web integration icons 1712 each represent an annotation that is associated with a static graphical image and an interactive operation relating to web content. For example, a Facebook icon represents an annotation that defines a link to a Facebook page for the program shown in the video window 1704. During playing the program in the video window 1704, selecting the Facebook icon causes the player logic 112 to redisplay the video window 1704 in smaller form and to access and display a Facebook page for the program in a browser window that is conceptually or logically under the video window 1704. The topic launch icons 1714 represent annotations that define branching behavior to other video program segments relating to topics such as costumes used on the program or show and the history of the show. Additionally or alternatively, one or more of the topic launch icons 1714 may be associated with a web page; thus, selecting one of the topic launch icons can result in either playing a video segment or displaying web content in a browser window under the video window.
[0252] In an embodiment, menu access link 1716 represents an annotation associated with branching behavior that causes the player logic 112 to branch to code in a metadata file 126 that causes displaying a list or menu of a plurality of video program episodes that are collected or associated with a subscription. In an embodiment, during playing, selecting the menu access link 1716 causes the player logic 112 to display a video window having the form of FIG. 17C. [0253] FIG. 17C illustrates a video window providing a menu of episodes in a collection or associated with a subscription. The nature and use of subscriptions for video programs is further described below. In an embodiment, video window 1720 comprises a plurality of available episode icons 1722 and a plurality of unavailable icons 1724. "Available," in this context, means released by a producer or program owner for viewing by viewers who have purchased or otherwise validly accessed a subscription, and "unavailable" means not yet released and normally planned for the future.
[0254] An available episode icon 1722 represents an annotation that is associated with a static graphical image representing the particular episode, and associated with branching behavior that causes the player logic 112 to play the particular episode in video window 1720, replacing the icons 1722, 1724. An unavailable episode icon 1724 represents an annotation that is associated with a static graphical image, or decoration, representing the particular episode that is unavailable. As decorations, unavailable episode icons 1724 are not selectable and not associated with branching behavior or other action. In an embodiment, the graphic images associated with unavailable episode icons 1724 may include an episode name and release date for the purpose of attracting viewer interest in future program material.
[0255] FIG. 17D illustrates use of annotations to form elements of a main menu page for a video program subscription. In an embodiment, a video window 1730 in a player window comprises a plurality of the web integration icons 1712 as previously discussed, located in a different position of the video window 1730. The same annotations may be used to define the web integration icons 1712 as for FIG. 17C, with different values for screen position attributes. In an embodiment, video window 1730 further displays a plurality of program link graphics 1732, 1734, which comprise static images each associated with a different animation having different responsive operational behavior. For example, program link graphics 1732 represent annotations that are associated with static graphical images and branching behavior to cause the player logic to branch to metadata in a metadata file 126 that causes playing a bonus episode, or displaying a menu of other annotations having graphics representing short episodes or all available episodes. Program link graphic 1734 represents an annotation which when selected causes branching in the metadata to code that causes the player logic 112 to play a particular video program episode.
[0256] Thus, multiple different kinds of annotations can be authored and associated with different graphics, branching behavior, and targets, including static graphics and video programs. Annotations also can cause playing pages that consist solely of other annotations, to await selection of one of the annotations to cause other navigation or to cause playing various program materials.
[0257] FIG. 18 A illustrates an example news program in which annotations may be used to provide a directory or menu of a plurality of news stores, features, segments, or related information. In an embodiment, the player window comprises a video window 1804 surrounded by an undecorated background 1802. In other embodiments, background 1802 may carry advertisements, program logos, or other information. In an embodiment, a plurality of program links 1806 is arranged in a column 1808 at one side of the video window 1804. Each of the program links 1806 is associated with an annotation. Each of the annotations defines a position, graphical image, and behavior in response to selection of the annotation. Each annotation may be associated with a video program or an Internet site, so that selecting a particular annotation causes the player logic 112 to either play the associated video program or to access and display information from the Internet site in a new browser window, depending on the defined responsive behavior. The graphical images may include a blend of images and text to indicate what kind of program or site is associated with the program link 1806. [0258] FIG. 18B illustrates the news program of FIG. 18A after a viewer has selected a program link 1806 (FIG. 18A) that is defined using an annotation having an association to a website. During playback, in response to receiving user input that selects a particular program link 1806 (FIG. 18A) that is defined using an annotation having an association to a website, player logic 112 obtains a URL for a web page from within the code of metadata files 126 that defines the annotation, issues an HTTP request for the URL, and generates a browser window 1810 that renders the resulting HTML. At about the same time, player logic 112 redisplays the video window 1804 in a reduced size within the player window 1800. The column 1808 of program links remains displayed in reduced size within the video window 1804. [0259] The browser window 1810 may include a scroll bar 1814 that is movable in response to user input from a pointing device such as a mouse, touchpad or trackball. The scroll bar is scrollable to cause the web page in browser window 1810 to scroll up or down independent of the video window 1804.
[0260] FIG. 18C illustrates the browser window 1810 of HG. 18B after the scroll bar has been moved. While the content of the web page has moved downward in browser window 1810, the position of video window 1804 remains fixed within the player window 1800. In this approach, the video remains visible and the viewer can retain context for the associated web page content. At any time, the viewer can select a full screen icon 1816, which is also defined using an annotation. In response to selecting the full screen icon 1816, player logic 112 causes the browser window 1810 to close and the video window 1804 is redisplayed to occupy all of the player window 1800.
[0261] The browser window 1810 may comprise a plurality of browser navigation buttons 1818. In an embodiment, the browser navigation buttons 1818 include forward, backward, and magnification buttons. Selecting a backward navigation button causes the player logic 112 to redisplay, in the browser window 1810, a most recently displayed previous web page. If the most recently displayed previous web page was generated when the player logic 112 was playing a different video program, then it is possible that using the backward navigation button may cause displaying a web site that is unrelated to the current video program. [0262] FIG. 19 A illustrates playing a video program in which annotations are associated with multiple different responsive behavior types. In an embodiment, player window 1900 comprises a video window 1902 that plays a video program. A plurality of annotations defined in metadata files 126 are associated with graphic images displayed as page links 1904, video links 1906, and voting buttons 1908. In an embodiment, annotations for page links 1904 are associated with URLs for web pages that correspond to an individual who is depicted in the page link.
[0263] Thus, in the example of FIG. 19A a viewer who selects a page link 1904 for fashion critic Tim Gunn causes the player logic 112 to access and display a web page associated with
Tim Gunn in a separate browser window in the manner shown for FIG. 18B, FIG. 18C. FIG.
19B illustrates an example of displaying a separate browser window 1920 below or behind the video window 1902 of the player window 1900. As in FIG. 18B, 18C, the browser window is scrollable independent of the video window 1902, the video window is automatically displayed in a reduced size representation, and the video window may be restored to fully occupy the player window 1900 by selecting a full screen icon in the video window.
[0264] If the viewer selects one of the video links 1906, player logic 112 branches within the code of a metadata file 126 and causes playing an associated video segment. In the example of
FIG. 19A, the associated video segments may comprise commercials or infomercials associated with brands, products or merchants, but in other embodiments the video segments may be noncommercial.
[0265] In an embodiment, voting buttons 1908 also represent annotations that cause the player logic 112 to invoke a web service that communicates a vote indicated by a particular voting button to a vote collecting server. Thus, FIG. 19 A provides an example of how annotations may be used to link a viewer through interactive services to Internet servers that collect information or perform specified actions.
[0266] 6.4 SKINNING VIA HTML BACKGROUND PAGES
[0267] In an embodiment, HTML and HTTP may be used to display a graphical format, termed a skin, for the player window 1302, for a background area of the player window 1302, and for various user interface elements such as annotation icons. In an embodiment, graphical backgrounds, skins, and UI elements all can be defined for, and thus synchronized at, any one or more of: video cue points; a metadata file 126 that describes a collection of video segments that are rendered into a single file; or a folder, directory, or collection of metadata files 126 making up a complex media presentation.
[0268] For example, an author can configure cue-point level synchronization to show character background information as different characters come on stage. The author can use file-level synchronization to have different backgrounds for commercials as compared to program content. The author can use folder- or directory-level synchronization to change the color scheme used in backgrounds, windows and other UI elements on an episode-by-episode basis. In this context, UI elements include annotations and associated graphic images. [0269] In an embodiment, a user may specify an HTML file to display in the background as the video is playing. In an embodiment, specifying a background skin is performed by a user accessing a Metadata tab 2112 of screen display 2102, as seen in FIG. 21, FIG. 22. [0270] FIG. 22 is a screen display diagram of the Metadata tab. In an embodiment, a user enters a URL of an HTML document that contains a background image in a Background HTML field 2206. The editor logic 110 stores the URL in metadata for the video. At playback time, the player logic 112 loads the URL and displays the contents as background behind or around the video window. Background images may include graphics, text, product branding, or other information.
[0271] Metadata tab 2112 also displays and allows user entry of values for other parameters for other player functions that are described further herein. As an overview, a Video File field identifies a filename of a video file with which the user is currently working and that is associated with the other metadata. A Video Size field identifies a size in pixels of a video window generated by the player logic 112 and that will display the video program at playback time. A Web Service field 2202 displays a reference to a web service that can be invoked at one or more cue points to provide external functions or processing. A Coincident Web Point field 2204 may receive user input of a synchronized web reference to display at a particular cue point. A Skin Buttons field may receive a reference to an electronic document that defines an appearance for play and trick play buttons of the player. [0272] 6.5 SUBSCRIPTION VIDEO
[0273] In an embodiment, video linking editor logic 110 may be used to author and configure, for playing using metadata-capable player logic 112, a live, refreshable collection of media with navigation metaphors. A subscription video collection differs from a traditional magazine or program subscription in that time is an element of authoring; thus, the media elements that are available to a subscriber change over time. The media elements change over time not in the sense of an animation, which involves changes frame to frame, but for a season of a show. In a subscription video collection as provided herein, the subscription may feature mixed HTML and video content, authored to incorporate additions, deletions and updates over time.
[0274] In an embodiment, a subscription video collection is authored as at least a first video segment that comprises a plurality of annotations; each annotation may be represented by a graphic image or animation which, at playing time, is overlaid on the first video segment. Each of the annotations is associated with a different one of a plurality of episodes or clips. [0275] For example, a show may have 22 planned episodes and at a particular time of year, there may be 8 of 22 episodes available for viewing to a subscriber. An end user accesses a subscription at a web site associated with a producer or distributor of the show. The end user presents authentication credentials, such as user name and password, is authenticated and admitted to the subscription. In response, the metadata-capable video player logic 112 plays a first video segment that features 8 icons indicating episode names with graphics, and 14 icons indicating "Episode to be Available in the Future." The annotations may be authored in the same single one of the metadata files 126 or may be in multiple different metadata files. For example, a first metadata file 126 for a show season may contain references to multiple other metadata files that contain actual annotations for each episode of the show. Selecting a particular episode to view is an invocation of the annotation associated with that episode and effectively causes a branch within the associated metadata file 126 to result in playing the selected video episode.
[0276] 7. IMPLEMENTATION MECHANISMS— HARDWARE OVERVIEW [0277] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application- specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special- purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard- wired and/or program logic to implement the techniques.
[0278] For example, FIG. 12 is a block diagram that illustrates a computer system 1200 upon which an embodiment of the invention may be implemented. Computer system 1200 includes a bus 1202 or other communication mechanism for communicating information, and a hardware processor 1204 coupled with bus 1202 for processing information. Hardware processor 1204 may be, for example, a general purpose microprocessor.
[0279] Computer system 1200 also includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. Main memory 1206 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1204. Such instructions, when stored in storage media accessible to processor 1204, render computer system 1200 into a special-purpose machine that is customized to perform the operations specified in the instructions. [0280] Computer system 1200 further includes a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a magnetic disk or optical disk, is provided and coupled to bus 1202 for storing information and instructions.
[0281] Computer system 1200 may be coupled via bus 1202 to a display 1212, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 1214, including alphanumeric and other keys, is coupled to bus 1202 for communicating information and command selections to processor 1204. Another type of user input device is cursor control 1216, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1204 and for controlling cursor movement on display 1212. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. [0282] Computer system 1200 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 1200 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 1200 in response to processor 1204 executing one or more sequences of one or more instructions contained in main memory 1206. Such instructions may be read into main memory 1206 from another storage medium, such as storage device 1210. Execution of the sequences of instructions contained in main memory 1206 causes processor 1204 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
[0283] The term "storage media" as used herein refers to any media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 1210. Volatile media includes dynamic memory, such as main memory 1206. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
[0284] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1202. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. [0285] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 1204 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1200 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 1202. Bus 1202 carries the data to main memory 1206, from which processor 1204 retrieves and executes the instructions. The instructions received by main memory 1206 may optionally be stored on storage device 1210 either before or after execution by processor 1204.
[0286] Computer system 1200 also includes a communication interface 1218 coupled to bus 1202. Communication interface 1218 provides a two-way data communication coupling to a network link 1220 that is connected to a local network 1222. For example, communication interface 1218 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1218 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1218 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
[0287] Network link 1220 typically provides data communication through one or more networks to other data devices. For example, network link 1220 may provide a connection through local network 1222 to a host computer 1224 or to data equipment operated by an Internet Service Provider (ISP) 1226. ISP 1226 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 1228. Local network 1222 and Internet 1228 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1220 and through communication interface 1218, which carry the digital data to and from computer system 1200, are example forms of transmission media. [0288] Computer system 1200 can send messages and receive data, including program code, through the network(s), network link 1220 and communication interface 1218. In the Internet example, a server 1230 might transmit a requested code for an application program through Internet 1228, ISP 1226, local network 1222 and communication interface 1218. [0289] The received code may be executed by processor 1204 as it is received, and/or stored in storage device 1210, or other non-volatile storage for later execution. [0290] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
CUE POINT LANGUAGE SPECIFICATION
[0291] A Cue Point Language specification, set forth in another document that is concurrently submitted herewith, forms a part of this patent disclosure.
EDITOR SPECIFICATION
[0292] An editor specification, set forth in another document that is concurrently submitted herewith, forms a part of this patent disclosure..

Claims

CLAIMSWhat is claimed is:
1. A non- transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining metadata that relates to a video program and that defines, for one or more specified time points in the video program, one or more web services to be invoked at those time points; generating and displaying, on a computer, a video window that is configured to play the video program; obtaining and playing the video program on the computer; during playing the video program, detecting that the video program is at one of the time points and determining an identifier of a particular web service associated with that one of the time points; in response to the detecting, causing the computer to invoke the particular web service.
2. The computer-readable data storage medium of claim 1, instructions which when executed cause invoking a particular web service that is configured to obtain a cellular radiotelephone number and to dispatch a text message to the number.
3. The computer-readable data storage medium of claim 1, instructions which when executed cause obtaining metadata that further defines a database query, and invoking the particular web service by submitting the database query to the web service, including invoking the particular web service that is configured to execute the database query with a database, determine a result set, and return the result set in a response to the computer.
4. The computer-readable data storage medium of claim 1, instructions which when executed cause invoking a particular web service that is configured to submit information about the video program over a network to a social networking service.
5. The computer-readable data storage medium of claim 1, further comprising instructions which when executed cause obtaining the video program from any of local storage on the computer, or a first networked server computer that is separate from the computer.
6. The computer-readable data storage medium of claim 1, further comprising instructions which when executed cause obtaining the video program from any of local storage on the computer, or a first networked server computer that is separate from the computer, and obtaining the metadata from a different second networked server computer.
7. A non-transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining metadata that relates to a video program and that defines, for one or more specified time points in the video program, one or more annotations to be invoked at those time points, wherein each of the annotations comprises a stored association of a graphical image and a network location identifier; during playing the video program on a computer, detecting that the video program is playing at a particular time point among the time points; in response to the detecting, causing the computer to display the graphical image associated with the particular time point, obtain an online electronic document associated with the network location identifier, and display the online electronic document with the video program.
8. The computer-readable data storage medium of claim 7, further comprising instructions which when executed cause the computer, in response to the detecting, to display the video program in a reduced size video window logically layered above a browser window, and to display the online electronic document in the browser window concurrently.
9. The computer-readable data storage medium of claim 7, further comprising instructions which when executed cause obtaining the particular time point that is associated with a song playing as part of the video program, and in response to the detecting, causing the computer to display, as the online electronic document, online information about how to obtain or purchase the song.
10. The computer-readable data storage medium of claim 7, further comprising instructions which when executed cause obtaining the particular time point that is associated with an appearance of a person in the video program, and cause the computer, in response to the detecting, to display, as the online electronic document, online information about the person.
11. The computer-readable data storage medium of claim 7, further comprising instructions which when executed cause obtaining the online information about the person from a remotely located server computer over a network.
12. The computer-readable data storage medium of claim 7, further comprising instructions which when executed cause obtaining, from the metadata, an identifier of an electronic document for a background for the video window; loading the electronic document over a network; displaying the electronic document as a background of a player window that includes the video window.
13. A non- transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining a link to a stored video program; obtaining metadata that relates to the video program and that defines, for a specified time point in the video program, one or more annotations to be invoked at the specified time point, wherein each of the annotations comprises an association of a graphic image and a reference to a video program segment or an online electronic document; causing the computer to play the video program from the link; during playing the video program on a computer, detecting that the video program is playing at the specified time point; in response to the detecting, causing the computer to display one or more graphic images associated with the one or more annotations for the specified time point, obtain input selecting one of the graphic images, and initiate playing the video program segment that is associated with the selected one of the graphic images.
14. The computer-readable data storage medium of claim 13, further comprising instructions which when executed cause the computer, in response to the detecting, to display one or more graphic images associated with the one or more annotations for the specified time point, obtain input selecting one of the graphic images, obtain a copy of the online electronic document that is associated with the selected one of the graphic images, display the video program in a reduced size video window logically layered above a browser window, and to display the copy of the online electronic document in the browser window concurrently.
15. The computer-readable data storage medium of claim 13, wherein the video program segment is in the video program.
16. The computer-readable data storage medium of claim 13, wherein the reference to the video program segment is a particular time point in the video program.
17. The computer-readable data storage medium of claim 13, wherein the video program segment is a second video program that is obtainable over a network.
18. The computer-readable data storage medium of claim 13, further comprising instructions which when executed cause displaying the one or more graphic images associated with the one or more annotations for the specified time point continuously after the time point is reached.
19. The computer-readable data storage medium of claim 13, further comprising instructions which when executed cause displaying the one or more graphic images in a border rectangle that is adjacent to a video rectangle, wherein both the border rectangle and video rectangle are within a video window of a player window, and wherein the video program plays within the video rectangle.
20. The computer-readable data storage medium of claim 19, comprising instructions which when executed cause displaying the one or more graphic images in the border rectangle in a position laterally adjacent to the video rectangle.
21. The computer-readable data storage medium of claim 13, further causing instructions which when executed cause the computer to display the one or more graphic images only in response to determining that a user of the computer is a paid subscriber, authenticated, or authorized to view the graphic images.
22. The computer-readable data storage medium of claim 13, wherein the graphic images comprise at least one graphic image that is not selectable and that is associated with a video program segment that is to be available in the future.
23. The computer-readable data storage medium of claim 13, further comprising instructions which when executed cause obtaining the video program from any of local storage on the computer, or a first networked server computer that is separate from the computer, and obtaining the metadata from a different second networked server computer.
24. A non-transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining a stored video file having one or more video segments and one or more cue points in the video file; obtaining a metadata file that contains an internal reference to the video file; generating and displaying a video window on a display unit; playing a first video segment of the video file in the video window; during the playing, in response to reaching a particular cue point among the cue points that are defined in the video file: obtaining from the metadata file one or more metadata values identifying for the particular cue point a cue point type, and one or more values of attributes that are associated with a particular cue point type, the cue point type and attribute values defining features of an action to perform at the cue point during playing of the video file; performing the action using the attribute values to determine and perform particular features, displays, or controls associated with the action.
25. The storage medium of claim 24, further comprising instructions which when executed cause generating and displaying in the graphical user interface a graphical directory of the video file in which each of the cue points is a selectable directory element.
26. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by playing a second video segment of the video file that begins at a point other than the cue point.
27. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by displaying images representing a set of video segments or networked resources and prompting a selection of one of the video segments or network resources.
28. The storage medium of claim 27, wherein one of the images represents a web site.
29. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by automatically selecting from among a plurality of other video segments defined in the metadata file based upon a result of a call to a web service that is identified in the metadata file.
30. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by returning to a different cue point and playing another video segment starting at the different cue point.
31. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by displaying a graphical button overlaid in the video window; receiving user input selecting the graphical button; in response to the user input, playing a second video segment, then returning to an end of the first video segment and playing a next video segment after the end of the first video segment.
32. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by reading a MXML (Macromedia XML) user interface markup language script from the metadata file, parsing and interpreting the MXML script, and generating and displaying one or more overlay graphical elements in the video window based on the parsing and interpreting.
33. The storage medium of claim 24, further comprising instructions which when executed cause performing the action by automatically pausing the playing the video file, switching to displaying a web browser interface, connecting to a web site, retrieving one or more web pages, and displaying the one or more web pages.
34. The storage medium of claim 33, further comprising instructions which when executed cause, while displaying the web browser interface, receiving user input requesting resuming video playback, and in response to the user input, automatically switching to displaying the first video segment in the video window and resuming playing the first video segment at a point at which the switching to displaying the web browser interface occurred.
35. The storage medium of claim 24, further comprising instructions which when executed cause playing the first video segment, reaching a modal story branch cue point at a first end of the first video segment, displaying a plurality of graphic images representing a plurality of second video segments, receiving user input specifying a selected one of the second video segments, in response to the user input playing the selected one of the second video segments until reaching a "goto" type cue point, and playing a third video segment in response to reaching the "goto" type cue point.
36. The storage medium of claim 24, further comprising instructions which when executed cause displaying, in the graphical user interface, a timeline that includes a plurality of lines, each line representing one of the video segments, a plurality of points, each of the points representing one of the cue points, and one or more arcs, each of the arcs representing a branch from one of the cue points to one of the video segments.
37. The storage medium of claim 36, further comprising instructions which when executed cause displaying the arcs in the timeline of the graphical user interface based on the cue point type of the one of the cue points.
38. The storage medium of claim 24, further comprising instructions which when executed cause receiving user input requesting related web content, and in response, connecting to a web site that is identified in the metadata file, retrieving one or more web pages, and displaying the one or more web pages in the video window while continuing playing the video file.
39. The storage medium of claim 38, further comprising instructions which when executed cause playing the video file in a reduced size video window that is overlaid on the web pages.
40. A non-transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining first data identifying a stored video file having one or more video segments; creating and storing a list of cue points based upon either user input or predefined cue points in the video file; obtaining second data identifying one of the cue points of a first video segment; obtaining one or more metadata values identifying a cue point type, and one or more values of attributes that are associated with a particular cue point type, the cue point type and attribute values defining features of an action to perform at the cue point during playing of the video file; storing the first data, second data, and metadata values in association with one another in a metadata file that references the video file.
41. The data storage medium of claim 40 further comprising instructions which when executed cause generating and displaying a graphical user interface that comprises all of: a video player window configured to play the video file; a list of the cue points; a cue point type input field; and one or more input fields and one or more display fields associated with attributes of particular cue point types.
42. The storage medium of claim 41, further comprising instructions which when executed cause generating and displaying in the graphical user interface, at least one of: an annotation value field configured to receive annotation metadata values that specifies whether a player should generate and display a selectable graphical image with the video file; a web service field configured to receive a name of a web service that can control selection among a plurality of video segments.
43. The storage medium of claim 40, wherein the cue point type and attribute values specify an operation selected from the group consisting of: displaying a second video segment; displaying a set of video segments or networked resources and prompting a selection of one of the video segments or network resources; selecting from among a plurality of other video segments based upon a result of a call to a web service; and returning to a different cue point.
44. The storage medium of claim 43, further comprising instructions which when executed cause receiving a web service identifier value that specifies the web service, and storing the web service identifier value in the metadata file.
45. The storage medium of claim 40, wherein the cue point type and attribute values represent any of commands to a video player to play a different video file; commands to the video player to play a different portion of the same video file; commands to a web browser client to retrieve and display a file from a web server; commands to the video player to display a menu of user options.
46. The storage medium of claim 40, further comprising instructions which when executed cause playing and displaying the video file during any of the receiving steps.
47. The storage medium of claim 40 further comprising storing the cue point type and attribute values in the metadata file in extensible markup language (XML) format.
48. The storage medium of claim 40, further comprising instructions which when executed cause receiving an annotation value and storing the annotation value in the metadata file, wherein the annotation value specifies whether a player should generate and display a graphical annotation of the video file in which each of the cue points is a selectable annotation element.
49. The storage medium of claim 40, wherein the cue point type and attribute values specify playing a second video segment of the video file that begins at a point other than the cue point.
50. The storage medium of claim 40, wherein the cue point type and attribute values specify displaying images representing a set of video segments or networked resources and prompting a selection of one of the video segments or network resources.
51. The storage medium of claim 50, wherein one of the images represents a web site.
52. The storage medium of claim 40, wherein the cue point type and attribute values specify automatically selecting from among a plurality of other video segments defined in the metadata file based upon a result of a call to a web service that is identified in the metadata file.
53. The storage medium of claim 40, wherein the cue point type and attribute values specify returning to a different cue point and playing another video segment starting at the different cue point.
54. The storage medium of claim 40, wherein the cue point type and attribute values specify displaying a graphical button overlaid in the video window; receiving user input selecting the graphical button; in response to the user input, playing a second video segment, then returning to an end of the first video segment and playing a next video segment after the end of the first video segment.
55. The storage medium of claim 40, wherein the cue point type and attribute values specify reading a MXML (Macromedia XML) user interface markup language script from the metadata file, parsing and interpreting the MXML script, and generating and displaying one or more overlay graphical elements in the video window based on the parsing and interpreting.
56. The storage medium of claim 40, wherein the cue point type and attribute values specify automatically pausing the playing the video file, switching to displaying a web browser interface, connecting to a web site, retrieving one or more web pages, and displaying the one or more web pages.
57. A non- transitory computer-readable data storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform: obtaining first data identifying a stored video file having one or more video segments; obtaining second data specifying one or more cue points for the video file, wherein each of the cue points comprises an association of a playback time value, a cue point type, and one or more values of attributes that are associated with a particular cue point type, wherein the cue point type and attribute values define an action to perform at the cue point during playing of the video file, wherein the action includes at least one of: obtaining and displaying a networked electronic document, and invoking a web service; storing the first data and second data in association with one another in a metadata file.
58. The storage medium of claim 57, further comprising instructions which when executed cause: obtaining annotation data defining an annotation for one or more cue points, wherein the annotation data comprises an annotation name, annotation type, annotation position, and one or more of text, a graphical image, and a graphical image to display at the time of a user mouse over operation, and wherein the annotation defines information for a video player to display over a video program when the video program is at a particular one of the cue points; obtaining association data defining an association of the annotation to the particular one of the cue points; storing the annotation data and the association data in the metadata file.
59. The storage medium of claim 57, further comprising instructions which when executed cause: obtaining web reference data defining an online electronic document for association with one or more cue points, wherein the web reference data comprises at least a network resource identifier of an electronic document to display in a browser window of a video player at the time of that the one or more cue points is reached in playing the video program; obtaining association data defining an association of the web reference data to at least one of the cue points; storing the annotation data and the association data in the metadata file.
60. The storage medium of claim 59, further comprising instructions which when executed cause obtaining, as part of the web reference data, a query string for submitting to an internet search engine for generating search results in which a viewer may have interest.
61. The storage medium of claim 59, further comprising instructions which when executed cause obtaining, as part of the web reference data, an interest URL to which a viewer may be directed at playback time in response to receiving input at a player indicating interest in other information relating to the video program.
62. The storage medium of claim 59, further comprising instructions which when executed cause obtaining, as part of the web reference data, a target at which processing should continue in response to receiving user input indicating closing the electronic document or completing viewing the electronic document.
63. The storage medium of claim 62, wherein the target comprises a second URL.
64. The storage medium of claim 59, further comprising instructions which when executed cause obtaining, as part of the web reference data, web view layout data that defines a visual layout of a browser window and a video window within a player window for use at playback time to concurrently display both the electronic document and the video program.
65. The storage medium of claim 59, further comprising instructions which when executed cause obtaining player layout data that defines a visual layout of a player window for use at playback time to concurrently display both the electronic document and the video program, wherein the player layout data defines a size and position of a browser window and a video window within the player window.
66. The storage medium of claim 59, further comprising instructions which when executed cause obtaining player layout data that defines a visual layout of a player window for use at playback time to concurrently display both the electronic document and the video program, wherein the player layout data defines a size and position of a browser window, a video window, and a background within the player window.
67. The storage medium of claim 66 comprising instructions which when executed cause obtaining player layout data that defines the video window as reduced size and logically over the browser window.
68. The storage medium of claim 66 comprising instructions which when executed cause obtaining player layout data that defines the video window and the browser window as adjacent and both surrounded by the background.
69. The storage medium of claim 57, further comprising instructions which when executed cause storing the metadata file in a first location that is different than a second location of the stored video file.
70. A computer-readable data storage medium storing a metadata file produced by the process of: receiving first data identifying a stored video file having one or more video segments and one or more cue points in the video file; reading the video file and forming a list of the cue points that are defined in the video file; receiving second data identifying one of the cue points of a first video segment of the video data file; receiving one or more metadata values identifying a cue point type, and one or more values of attributes that are associated with a particular cue point type, wherein the cue point type and attribute values define features of an action to perform at the cue point during playing of the video file; storing the first data, second data, and metadata values in association with one another in a metadata file that references the video file.
71. The storage medium of claim 70, wherein the metadata file further comprises at least one of: an annotation value that specifies whether a player should generate and display a graphical annotation of the video file in which each of the cue points is a selectable annotation element; a name of a web service that can control selection among a plurality of video segments.
72. The storage medium of claim 70, wherein the cue point type and attribute values in the metadata file specify any selected from the group consisting of: displaying a second video segment; displaying a set of video segments or networked resources and prompting a selection of one of the video segments or network resources; selecting from among a plurality of other video segments based upon a result of a call to a web service; and returning to a different cue point.
73. The storage medium of claim 72, wherein the metadata file further comprises a web service identifier value that specifies the web service.
74. The storage medium of claim 70, wherein the cue point type and attribute values represent any of commands to a video player to play a different video file; commands to the video player to play a different portion of the same video file; commands to a web client to retrieve and display a file from a web server; commands to the video player to display a menu of user options.
75. The storage medium of claim 70 wherein the cue point type and attribute values are stored in the metadata file in extensible markup language (XML) format.
76. The storage medium of claim 70, wherein the metadata file further comprises the cue point type and attribute values specifying playing a second video segment of the video file that begins at a point other than the cue point.
77. The storage medium of claim 70, wherein the metadata file further comprises the cue point type and attribute values specifying displaying images representing a set of video segments or networked resources and prompting a selection of one of the video segments or network resources.
78. The storage medium of claim 78, wherein one of the images represents a web site.
79. The storage medium of claim 70, wherein the metadata file further comprises the cue point type and attribute values specifying automatically selecting from among a plurality of other video segments defined in the metadata file based upon a result of a call to a web service that is identified in the metadata file.
80. The storage medium of claim 70, wherein the cue point type and attribute values specify returning to a different cue point and playing another video segment starting at the different cue point.
81. The storage medium of claim 70, wherein the cue point type and attribute values specify displaying a graphical button overlaid in the video window; receiving user input selecting the graphical button; in response to the user input, playing a second video segment, then returning to an end of the first video segment and playing a next video segment after the end of the first video segment.
82. The storage medium of claim 70, wherein the cue point type and attribute values specify reading a MXML (Macromedia XML) user interface markup language script from the metadata file, parsing and interpreting the MXML script, and generating and displaying one or more overlay graphical elements in the video window based on the parsing and interpreting.
83. The storage medium of claim 70, wherein the cue point type and attribute values specify automatically pausing the playing the video file, switching to displaying a web browser interface, connecting to a web site, retrieving one or more web pages, and displaying the one or more web pages.
PCT/US2010/034808 2009-05-13 2010-05-13 Playing and editing linked and annotated audiovisual works WO2010132718A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP10775570.4A EP2430833A4 (en) 2009-05-13 2010-05-13 Playing and editing linked and annotated audiovisual works

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US17772609P 2009-05-13 2009-05-13
US61/177,726 2009-05-13
US32107610P 2010-04-05 2010-04-05
US61/321,076 2010-04-05

Publications (2)

Publication Number Publication Date
WO2010132718A2 true WO2010132718A2 (en) 2010-11-18
WO2010132718A3 WO2010132718A3 (en) 2011-02-03

Family

ID=43069363

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2010/034808 WO2010132718A2 (en) 2009-05-13 2010-05-13 Playing and editing linked and annotated audiovisual works

Country Status (3)

Country Link
US (2) US9170700B2 (en)
EP (1) EP2430833A4 (en)
WO (1) WO2010132718A2 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9326017B2 (en) 2012-11-19 2016-04-26 Thomson Licensing Method and apparatus for setting controlled events for network devices
US10817307B1 (en) 2017-12-20 2020-10-27 Apple Inc. API behavior modification based on power source health
US10983588B2 (en) 2014-08-06 2021-04-20 Apple Inc. Low power mode
US11088567B2 (en) 2014-08-26 2021-08-10 Apple Inc. Brownout avoidance
US11363133B1 (en) 2017-12-20 2022-06-14 Apple Inc. Battery health-based power management

Families Citing this family (160)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6543395B2 (en) 1998-10-13 2003-04-08 Gas Technologies, Inc. Bi-fuel control system and retrofit assembly for diesel engines
US8140973B2 (en) * 2008-01-23 2012-03-20 Microsoft Corporation Annotating and sharing content
WO2010132718A2 (en) 2009-05-13 2010-11-18 Coincident.Tv , Inc. Playing and editing linked and annotated audiovisual works
US9690866B2 (en) 2009-06-19 2017-06-27 Microsoft Technology Licensing, Llc Persistent media playback
US8979538B2 (en) 2009-06-26 2015-03-17 Microsoft Technology Licensing, Llc Using game play elements to motivate learning
US9465451B2 (en) 2009-12-31 2016-10-11 Flick Intelligence, LLC Method, system and computer program product for obtaining and displaying supplemental data about a displayed movie, show, event or video game
US9508387B2 (en) * 2009-12-31 2016-11-29 Flick Intelligence, LLC Flick intel annotation methods and systems
US8751942B2 (en) 2011-09-27 2014-06-10 Flickintel, Llc Method, system and processor-readable media for bidirectional communications and data sharing between wireless hand held devices and multimedia display systems
US10108620B2 (en) * 2010-04-29 2018-10-23 Google Llc Associating still images and videos
US9697500B2 (en) * 2010-05-04 2017-07-04 Microsoft Technology Licensing, Llc Presentation of information describing user activities with regard to resources
US8381108B2 (en) 2010-06-21 2013-02-19 Microsoft Corporation Natural user input for driving interactive stories
US20120209770A1 (en) * 2010-07-08 2012-08-16 Subramanian Peruvemba System and Method for Monetizing Video Content
JP2012050017A (en) * 2010-08-30 2012-03-08 Toshiba Corp Apparatus and method for displaying moving image
US9343110B2 (en) 2010-11-23 2016-05-17 Levels Beyond Dynamic synchronization tool
WO2012088468A2 (en) * 2010-12-22 2012-06-28 Coincident.Tv, Inc. Switched annotations in playing audiovisual works
EP2490213A1 (en) 2011-02-19 2012-08-22 beyo GmbH Method for converting character text messages to audio files with respective titles for their selection and reading aloud with mobile devices
US8539086B2 (en) 2011-03-23 2013-09-17 Color Labs, Inc. User device group formation
US20120246191A1 (en) * 2011-03-24 2012-09-27 True Xiong World-Wide Video Context Sharing
US9477574B2 (en) 2011-05-12 2016-10-25 Microsoft Technology Licensing, Llc Collection of intranet activity data
US8819009B2 (en) 2011-05-12 2014-08-26 Microsoft Corporation Automatic social graph calculation
CN102802055B (en) * 2011-05-25 2016-06-01 阿里巴巴集团控股有限公司 A kind of data interactive method based on Online Video and device
WO2012162803A1 (en) * 2011-06-03 2012-12-06 Gdial Inc. Systems and methods for authenticating and aiding in indexing of and searching for electronic files
US8577963B2 (en) 2011-06-30 2013-11-05 Amazon Technologies, Inc. Remote browsing session between client browser and network based browser
US8799412B2 (en) 2011-06-30 2014-08-05 Amazon Technologies, Inc. Remote browsing session management
US9621406B2 (en) 2011-06-30 2017-04-11 Amazon Technologies, Inc. Remote browsing session management
US20130007807A1 (en) * 2011-06-30 2013-01-03 Delia Grenville Blended search for next generation television
CN104025017A (en) * 2011-07-22 2014-09-03 谷歌公司 Linking content files
US9037696B2 (en) 2011-08-16 2015-05-19 Amazon Technologies, Inc. Managing information associated with network resources
US9195768B2 (en) 2011-08-26 2015-11-24 Amazon Technologies, Inc. Remote browsing session management
US10089403B1 (en) 2011-08-31 2018-10-02 Amazon Technologies, Inc. Managing network based storage
ITRM20110469A1 (en) * 2011-09-08 2013-03-09 Hyper Tv S R L SYSTEM AND METHOD FOR THE PRODUCTION BY A AUTHOR OF COMPLEX MULTIMEDIA CONTENT AND FOR THE USE OF SUCH CONTENT BY A USER
US20130074131A1 (en) * 2011-09-15 2013-03-21 Milestone Project, Inc. System and method for integrating and controlling web-based html players in a native context
US8412772B1 (en) 2011-09-21 2013-04-02 Color Labs, Inc. Content sharing via social networking
US9178955B1 (en) 2011-09-27 2015-11-03 Amazon Technologies, Inc. Managing network based content
US9641637B1 (en) 2011-09-27 2017-05-02 Amazon Technologies, Inc. Network resource optimization
US9152970B1 (en) 2011-09-27 2015-10-06 Amazon Technologies, Inc. Remote co-browsing session management
US8914514B1 (en) 2011-09-27 2014-12-16 Amazon Technologies, Inc. Managing network based content
US9383958B1 (en) 2011-09-27 2016-07-05 Amazon Technologies, Inc. Remote co-browsing session management
US8589385B2 (en) 2011-09-27 2013-11-19 Amazon Technologies, Inc. Historical browsing session management
US8849802B2 (en) * 2011-09-27 2014-09-30 Amazon Technologies, Inc. Historical browsing session management
US10693991B1 (en) 2011-09-27 2020-06-23 Amazon Technologies, Inc. Remote browsing session management
US8615431B1 (en) 2011-09-29 2013-12-24 Amazon Technologies, Inc. Network content message placement management
US9098533B2 (en) * 2011-10-03 2015-08-04 Microsoft Technology Licensing, Llc Voice directed context sensitive visual search
EP2766816A4 (en) * 2011-10-10 2016-01-27 Vivoom Inc Network-based rendering and steering of visual effects
US9031382B1 (en) * 2011-10-20 2015-05-12 Coincident.Tv, Inc. Code execution in complex audiovisual experiences
US9313100B1 (en) 2011-11-14 2016-04-12 Amazon Technologies, Inc. Remote browsing session management
US8972477B1 (en) 2011-12-01 2015-03-03 Amazon Technologies, Inc. Offline browsing session management
US9117002B1 (en) 2011-12-09 2015-08-25 Amazon Technologies, Inc. Remote browsing session management
US9009334B1 (en) 2011-12-09 2015-04-14 Amazon Technologies, Inc. Remote browsing session management
US9330188B1 (en) 2011-12-22 2016-05-03 Amazon Technologies, Inc. Shared browsing sessions
WO2013100978A1 (en) * 2011-12-28 2013-07-04 Intel Corporation Real-time natural language processing of datastreams
US20130174035A1 (en) * 2011-12-30 2013-07-04 United Video Properties, Inc. Systems and methods for representing a content dependency list
US9634903B2 (en) * 2012-01-18 2017-04-25 Nokia Technologies Oy Method and apparatus for distributing content to multiple devices
US9087024B1 (en) 2012-01-26 2015-07-21 Amazon Technologies, Inc. Narration of network content
US9336321B1 (en) 2012-01-26 2016-05-10 Amazon Technologies, Inc. Remote browsing and searching
US8627195B1 (en) 2012-01-26 2014-01-07 Amazon Technologies, Inc. Remote browsing and searching
US8839087B1 (en) 2012-01-26 2014-09-16 Amazon Technologies, Inc. Remote browsing and searching
US9509783B1 (en) 2012-01-26 2016-11-29 Amazon Technlogogies, Inc. Customized browser images
US9092405B1 (en) 2012-01-26 2015-07-28 Amazon Technologies, Inc. Remote browsing and searching
US9183258B1 (en) 2012-02-10 2015-11-10 Amazon Technologies, Inc. Behavior based processing of content
US9037975B1 (en) 2012-02-10 2015-05-19 Amazon Technologies, Inc. Zooming interaction tracking and popularity determination
US9137210B1 (en) 2012-02-21 2015-09-15 Amazon Technologies, Inc. Remote browsing session management
US10296558B1 (en) 2012-02-27 2019-05-21 Amazon Technologies, Inc. Remote generation of composite content pages
US9208316B1 (en) 2012-02-27 2015-12-08 Amazon Technologies, Inc. Selective disabling of content portions
US9374244B1 (en) 2012-02-27 2016-06-21 Amazon Technologies, Inc. Remote browsing session management
US9460220B1 (en) 2012-03-26 2016-10-04 Amazon Technologies, Inc. Content selection based on target device characteristics
US9307004B1 (en) 2012-03-28 2016-04-05 Amazon Technologies, Inc. Prioritized content transmission
US20140150029A1 (en) 2012-04-18 2014-05-29 Scorpcast, Llc System and methods for providing user generated video reviews
US8682809B2 (en) * 2012-04-18 2014-03-25 Scorpcast, Llc System and methods for providing user generated video reviews
US10506278B2 (en) * 2012-04-18 2019-12-10 Scorpoast, LLC Interactive video distribution system and video player utilizing a client server architecture
US9832519B2 (en) 2012-04-18 2017-11-28 Scorpcast, Llc Interactive video distribution system and video player utilizing a client server architecture
US20130328811A1 (en) * 2012-06-07 2013-12-12 Kontera Technologies, Inc. Interactive layer on touch-based devices for presenting web and content pages
US20150100569A1 (en) * 2012-06-28 2015-04-09 Google Inc. Providing a search results document that includes a user interface for performing an action in connection with a web page identified in the search results document
US10067730B2 (en) * 2012-07-10 2018-09-04 Recursive Labs, Inc. Systems and methods for enabling replay of internet co-browsing
US9423925B1 (en) 2012-07-11 2016-08-23 Google Inc. Adaptive content control and display for internet media
US9110562B1 (en) * 2012-07-26 2015-08-18 Google Inc. Snapping a pointing-indicator to a scene boundary of a video
US9772979B1 (en) 2012-08-08 2017-09-26 Amazon Technologies, Inc. Reproducing user browsing sessions
US8943197B1 (en) 2012-08-16 2015-01-27 Amazon Technologies, Inc. Automated content update notification
US8763041B2 (en) * 2012-08-31 2014-06-24 Amazon Technologies, Inc. Enhancing video content with extrinsic data
US20170024097A1 (en) * 2012-09-13 2017-01-26 Bravo Ideas Digital Co., Ltd. Method and Host Server for Creating a Composite Media File
TWI475504B (en) * 2012-09-13 2015-03-01 Bravo Ideas Digital Co Ltd A publish system and method for interactive video e-commerce
CN103780741B (en) * 2012-10-18 2018-03-13 腾讯科技(深圳)有限公司 Prompt the method and mobile device of network speed
WO2014067911A1 (en) * 2012-10-29 2014-05-08 Net Entertainment Ne Ab Architecture for multi-player, multi-game, multi- table, multi-operator & multi-jurisdiction live casino gaming
US9591339B1 (en) 2012-11-27 2017-03-07 Apple Inc. Agnostic media delivery system
US9774917B1 (en) 2012-12-10 2017-09-26 Apple Inc. Channel bar user interface
US10200761B1 (en) 2012-12-13 2019-02-05 Apple Inc. TV side bar user interface
US9532111B1 (en) 2012-12-18 2016-12-27 Apple Inc. Devices and method for providing remote control hints on a display
US10521188B1 (en) 2012-12-31 2019-12-31 Apple Inc. Multi-user TV user interface
US10701305B2 (en) * 2013-01-30 2020-06-30 Kebron G. Dejene Video signature system and method
WO2014127293A2 (en) * 2013-02-15 2014-08-21 Quick Eye Technologies Inc. An organizer for data that is subject to multiple criteria
US9305374B2 (en) * 2013-03-15 2016-04-05 Apple Inc. Device, method, and graphical user interface for adjusting the appearance of a control
US9374411B1 (en) * 2013-03-21 2016-06-21 Amazon Technologies, Inc. Content recommendations using deep data
US10152463B1 (en) 2013-06-13 2018-12-11 Amazon Technologies, Inc. System for profiling page browsing interactions
US9578137B1 (en) 2013-06-13 2017-02-21 Amazon Technologies, Inc. System for enhancing script execution performance
USD770462S1 (en) * 2013-08-07 2016-11-01 Bosch Automotive Service Solutions Inc. Display screen with an animated graphical user interface
US20150046812A1 (en) * 2013-08-12 2015-02-12 Google Inc. Dynamic resizable media item player
FR3009916A1 (en) * 2013-08-26 2015-02-27 Orange METHOD AND DEVICE FOR ENRICHING A COMMUNICATION
KR101524379B1 (en) * 2013-12-27 2015-06-04 인하대학교 산학협력단 System and method for the caption replacement of the released video for the interactive service
KR20150077774A (en) * 2013-12-30 2015-07-08 삼성전자주식회사 Method and Apparatus for Switching Display
WO2015103636A2 (en) * 2014-01-06 2015-07-09 Vinja, Llc Injection of instructions in complex audiovisual experiences
US9665251B2 (en) * 2014-02-12 2017-05-30 Google Inc. Presenting content items and performing actions with respect to content items
US20150271541A1 (en) 2014-03-19 2015-09-24 Time Warner Cable Enterprises Llc Apparatus and methods for recording a media stream
US9635041B1 (en) 2014-06-16 2017-04-25 Amazon Technologies, Inc. Distributed split browser content inspection and analysis
TWI585673B (en) 2014-06-24 2017-06-01 蘋果公司 Input device and user interface interactions
CN111782128B (en) 2014-06-24 2023-12-08 苹果公司 Column interface for navigating in a user interface
US10423703B1 (en) * 2014-06-26 2019-09-24 Amazon Technologies, Inc. Detection and rendering of endnotes in digital content
US9665241B2 (en) * 2014-06-30 2017-05-30 Verizon Patent And Licensing Inc. Media content search systems and methods
US9998518B2 (en) * 2014-09-18 2018-06-12 Multipop Llc Media platform for adding synchronized content to media with a duration
US10231033B1 (en) 2014-09-30 2019-03-12 Apple Inc. Synchronizing out-of-band content with a media stream
US10708391B1 (en) 2014-09-30 2020-07-07 Apple Inc. Delivery of apps in a media stream
US10001760B1 (en) 2014-09-30 2018-06-19 Hrl Laboratories, Llc Adaptive control system capable of recovering from unexpected situations
US10067955B1 (en) * 2014-12-08 2018-09-04 Conviva Inc. Custom video metrics management platform
US9671939B2 (en) * 2014-12-17 2017-06-06 Fuji Xerox Co., Ltd. Systems and methods for automatic generation and consumption of hypermeetings
US9940974B2 (en) * 2014-12-17 2018-04-10 Fuji Xerox Co., Ltd. Systems and methods for visualizing playback plans of hypermeetings
US9939983B2 (en) * 2014-12-17 2018-04-10 Fuji Xerox Co., Ltd. Systems and methods for plan-based hypervideo playback
US11231826B2 (en) * 2015-03-08 2022-01-25 Google Llc Annotations in software applications for invoking dialog system functions
US10375452B2 (en) * 2015-04-14 2019-08-06 Time Warner Cable Enterprises Llc Apparatus and methods for thumbnail generation
USD771072S1 (en) * 2015-04-27 2016-11-08 Lutron Electronics Co., Inc. Display screen or portion thereof with graphical user interface
US10382824B2 (en) * 2015-07-17 2019-08-13 Tribune Broadcasting Company, Llc Video production system with content extraction feature
US9665972B2 (en) * 2015-07-28 2017-05-30 Google Inc. System for compositing educational video with interactive, dynamically rendered visual aids
US9852770B2 (en) * 2015-09-26 2017-12-26 Intel Corporation Technologies for dynamic generation of a media compilation
KR20170083868A (en) 2016-01-11 2017-07-19 삼성전자주식회사 A data movement device and a data movement method
DK201670582A1 (en) 2016-06-12 2018-01-02 Apple Inc Identifying applications on which content is available
DK201670581A1 (en) 2016-06-12 2018-01-08 Apple Inc Device-level authorization for viewing content
US10652594B2 (en) 2016-07-07 2020-05-12 Time Warner Cable Enterprises Llc Apparatus and methods for presentation of key frames in encrypted content
US10555051B2 (en) 2016-07-21 2020-02-04 At&T Mobility Ii Llc Internet enabled video media content stream
US20180113579A1 (en) 2016-10-26 2018-04-26 Apple Inc. User interfaces for browsing content from multiple content applications on an electronic device
US10334264B2 (en) * 2016-11-18 2019-06-25 Ainsworth Game Technology Limited Method of encoding multiple languages in a video file for a gaming machine
US10313756B2 (en) 2017-03-03 2019-06-04 Rovi Guides, Inc. System and methods for recommending a media asset relating to a character unknown to a user
US10475349B2 (en) 2017-03-10 2019-11-12 SmartNoter Inc. System and method of producing and providing user specific educational digital media modules
CN109420338A (en) * 2017-08-31 2019-03-05 腾讯科技(深圳)有限公司 The mobile virtual scene display method and device of simulating lens, electronic equipment
US10726095B1 (en) 2017-09-26 2020-07-28 Amazon Technologies, Inc. Network content layout using an intermediary system
US10664538B1 (en) 2017-09-26 2020-05-26 Amazon Technologies, Inc. Data security and data access auditing for network accessible content
US10055508B1 (en) 2017-10-11 2018-08-21 Cgip Holdco, Llc Platform-agnostic thick-client system for combined delivery of disparate streaming content and dynamic content by combining dynamic data with output from a continuous queue transmitter
US10673771B2 (en) 2017-10-11 2020-06-02 Cgip Holdco, Llc Platform-agnostic thick-client system for combined delivery of disparate streaming content and dynamic content by combining dynamic data with output from a continuous queue transmitter
US10657380B2 (en) 2017-12-01 2020-05-19 At&T Mobility Ii Llc Addressable image object
US10726732B2 (en) 2018-01-16 2020-07-28 SmartNoter Inc. System and method of producing and providing user specific educational digital media modules augmented with electronic educational testing content
USD844021S1 (en) * 2018-01-30 2019-03-26 Xerox Corporation Display screen with graphical user interface
DK201870354A1 (en) 2018-06-03 2019-12-20 Apple Inc. Setup procedures for an electronic device
CN109144249B (en) * 2018-07-23 2021-09-14 Oppo广东移动通信有限公司 Screen sounding method and device, electronic device and storage medium
CN109062536B (en) * 2018-07-23 2021-08-17 Oppo广东移动通信有限公司 Screen sounding method and device, electronic device and storage medium
US10885094B2 (en) * 2018-09-13 2021-01-05 Legistek Corporation Method for cueing the display of active content to an audience
US11683565B2 (en) 2019-03-24 2023-06-20 Apple Inc. User interfaces for interacting with channels that provide content that plays in a media browsing application
US11467726B2 (en) 2019-03-24 2022-10-11 Apple Inc. User interfaces for viewing and accessing content on an electronic device
EP3928194A1 (en) 2019-03-24 2021-12-29 Apple Inc. User interfaces including selectable representations of content items
US11863837B2 (en) 2019-05-31 2024-01-02 Apple Inc. Notification of augmented reality content on an electronic device
CN113906380A (en) 2019-05-31 2022-01-07 苹果公司 User interface for podcast browsing and playback applications
US11082755B2 (en) * 2019-09-18 2021-08-03 Adam Kunsberg Beat based editing
CN110830813B (en) * 2019-10-31 2020-11-06 北京达佳互联信息技术有限公司 Video switching method and device, electronic equipment and storage medium
US11003713B1 (en) * 2019-11-25 2021-05-11 Sap Se Automated updating of video content
CN113434223A (en) * 2020-03-23 2021-09-24 北京字节跳动网络技术有限公司 Special effect processing method and device
US11843838B2 (en) 2020-03-24 2023-12-12 Apple Inc. User interfaces for accessing episodes of a content series
SE2050465A1 (en) * 2020-04-23 2021-10-24 Checkin Com Group Ab Method and system for generating a data collection process in a user device
CN112423109B (en) * 2020-05-19 2023-03-03 上海哔哩哔哩科技有限公司 Interactive video generation method and system, electronic equipment and storage medium
US11237708B2 (en) 2020-05-27 2022-02-01 Bank Of America Corporation Video previews for interactive videos using a markup language
US11461535B2 (en) 2020-05-27 2022-10-04 Bank Of America Corporation Video buffering for interactive videos using a markup language
US11899895B2 (en) 2020-06-21 2024-02-13 Apple Inc. User interfaces for setting up an electronic device
US11720229B2 (en) 2020-12-07 2023-08-08 Apple Inc. User interfaces for browsing and presenting content
US11894019B2 (en) * 2020-12-30 2024-02-06 Linearity Gmbh Time-lapse
CN113542818B (en) * 2021-07-16 2023-04-25 北京字节跳动网络技术有限公司 Video display method, video editing method and device

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3472659B2 (en) * 1995-02-20 2003-12-02 株式会社日立製作所 Video supply method and video supply system
US6505160B1 (en) * 1995-07-27 2003-01-07 Digimarc Corporation Connected audio and other media objects
US6199076B1 (en) * 1996-10-02 2001-03-06 James Logan Audio program player including a dynamic program selection controller
JP2001014417A (en) * 1999-03-31 2001-01-19 Canon Inc Synchronization of visual queue to multimedia presentation
US20030145338A1 (en) * 2002-01-31 2003-07-31 Actv, Inc. System and process for incorporating, retrieving and displaying an enhanced flash movie
US7945141B2 (en) * 2003-10-06 2011-05-17 Samsung Electronics Co., Ltd. Information storage medium including event occurrence information, and apparatus and method for reproducing the information storage medium
CN1942969A (en) * 2004-04-16 2007-04-04 松下电器产业株式会社 Playback device, method and program, and recording medium
US20060047722A1 (en) * 2004-09-01 2006-03-02 Walker Glenn A Metadata-based data storage in digital radio system
JP2007065356A (en) * 2005-08-31 2007-03-15 Toshiba Corp Device and method for composite object display, and program
CN101300621B (en) * 2005-09-13 2010-11-10 时空3D公司 System and method for providing three-dimensional graphical user interface
US20070150484A1 (en) * 2005-12-23 2007-06-28 Funge John L Systems and methods for multi-media transfer
US9554093B2 (en) * 2006-02-27 2017-01-24 Microsoft Technology Licensing, Llc Automatically inserting advertisements into source video content playback streams
US20070294292A1 (en) * 2006-06-14 2007-12-20 Microsoft Corporation Advertising transfer and playback on portable devices
US7881505B2 (en) * 2006-09-29 2011-02-01 Pittsburgh Pattern Recognition, Inc. Video retrieval system for human face content
US20080152300A1 (en) * 2006-12-22 2008-06-26 Guideworks, Llc Systems and methods for inserting advertisements during commercial skip
US8316302B2 (en) 2007-05-11 2012-11-20 General Instrument Corporation Method and apparatus for annotating video content with metadata generated using speech recognition technology
WO2009012235A1 (en) * 2007-07-13 2009-01-22 Spot Runner, Inc. Methods and systems for performing media searches, media creation and for secure distribution of media
US7938727B1 (en) * 2007-07-19 2011-05-10 Tim Konkle System and method for providing interactive content for multiple networked users in a shared venue
US9843774B2 (en) * 2007-10-17 2017-12-12 Excalibur Ip, Llc System and method for implementing an ad management system for an extensible media player
KR101348598B1 (en) * 2007-12-21 2014-01-07 삼성전자주식회사 Digital television video program providing system and digital television and contolling method for the same
US20090193475A1 (en) * 2008-01-29 2009-07-30 Corey Halverson Systems and Methods For Enabling Configurable Context-Specific Actions in Streaming Video
US8312486B1 (en) 2008-01-30 2012-11-13 Cinsay, Inc. Interactive product placement system and method therefor
US7840851B2 (en) 2008-02-15 2010-11-23 Red Hat, Inc. Annotating GUI test automation playback and debugging
US20100064220A1 (en) * 2008-03-27 2010-03-11 Verizon Data Services India Private Limited Method and system for providing interactive hyperlinked video
US8200587B2 (en) * 2008-04-07 2012-06-12 Microsoft Corporation Techniques to filter media content based on entity reputation
WO2009145848A1 (en) * 2008-04-15 2009-12-03 Pvi Virtual Media Services, Llc Preprocessing video to insert visual elements and applications thereof
US20100095211A1 (en) 2008-09-17 2010-04-15 Seth Kenvin Method and System for Annotative Multimedia
US9204181B2 (en) * 2008-12-12 2015-12-01 Genband Us Llc Content overlays in on-demand streaming applications
WO2010132718A2 (en) 2009-05-13 2010-11-18 Coincident.Tv , Inc. Playing and editing linked and annotated audiovisual works

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of EP2430833A4 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9326017B2 (en) 2012-11-19 2016-04-26 Thomson Licensing Method and apparatus for setting controlled events for network devices
US10983588B2 (en) 2014-08-06 2021-04-20 Apple Inc. Low power mode
US11088567B2 (en) 2014-08-26 2021-08-10 Apple Inc. Brownout avoidance
US10817307B1 (en) 2017-12-20 2020-10-27 Apple Inc. API behavior modification based on power source health
US11363133B1 (en) 2017-12-20 2022-06-14 Apple Inc. Battery health-based power management

Also Published As

Publication number Publication date
US20100293190A1 (en) 2010-11-18
US9462309B2 (en) 2016-10-04
EP2430833A4 (en) 2014-01-22
EP2430833A2 (en) 2012-03-21
US20160057458A1 (en) 2016-02-25
WO2010132718A3 (en) 2011-02-03
US9170700B2 (en) 2015-10-27

Similar Documents

Publication Publication Date Title
US9462309B2 (en) Playing and editing linked and annotated audiovisual works
US8526782B2 (en) Switched annotations in playing audiovisual works
US9936184B2 (en) Code execution in complex audiovisual experiences
US8978075B1 (en) Associating media using metadata and controlling multiple-device synchronization and rendering
US8903863B2 (en) User interface with available multimedia content from multiple multimedia websites
WO2015103636A9 (en) Injection of instructions in complex audiovisual experiences
US10602225B2 (en) System and method for construction, delivery and display of iTV content
US8745658B2 (en) System and method for construction, delivery and display of iTV content
EP1999953B1 (en) Embedded metadata in a media presentation
US8201201B2 (en) Inserting branding elements
US20100058220A1 (en) Systems, methods, and computer program products for the creation, monetization, distribution, and consumption of metacontent
US20110191684A1 (en) Method of Internet Video Access and Management
US20090327100A1 (en) Method of Internet Video Access and Management
WO2012083006A1 (en) Browser integration for a content system
US11277668B2 (en) Methods, systems, and media for providing media guidance
US11070890B2 (en) User customization of user interfaces for interactive television
TW484288B (en) Method and system for providing a user with active and passive access to cached content

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10775570

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2010775570

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2010775570

Country of ref document: EP