US20130174197A1 - Video preview module to enhance online video experience - Google Patents
Video preview module to enhance online video experience Download PDFInfo
- Publication number
- US20130174197A1 US20130174197A1 US13/771,563 US201313771563A US2013174197A1 US 20130174197 A1 US20130174197 A1 US 20130174197A1 US 201313771563 A US201313771563 A US 201313771563A US 2013174197 A1 US2013174197 A1 US 2013174197A1
- Authority
- US
- United States
- Prior art keywords
- video
- storyboard
- processor
- display
- vpm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/535—Tracking the activity of the user
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
Definitions
- the present application relates generally to video preview modules.
- storyboard approach in which a video storyboard presentation is added to a video link.
- a storyboard contains an array of frames that provide a useful guide to the content of the video.
- movement of a mouse along a video link activates the accompanying storyboard.
- current methods to generate storyboards are computationally intensive and therefore are typically pre-generated at the server, limiting its widespread use.
- the second approach to enhance the online video experience is playback of “key” frames of the video.
- a mouse action over the initial thumbnail representing the video initiates the playback of a subset of frames called key frames.
- Key frames are typically frames that can be decoded independently.
- the key frame generation is in real-time, so that the server need not pre-generate a key frame sequence apart from the stream itself.
- Availability is limited to only online video sites offering specific support for the functionality.
- a client device may be able to access key-frame playback on one site but not on a site that does not support the feature.
- Compatibility is limited to those client devices that meet the minimum system requirements (memory, graphics, screen size etc.) needed to support the server's presentation format.
- devices such as smart phones may fall short in meeting these requirements.
- neither approach gives consideration to a client device's available bandwidth. This may result in unacceptable latencies in presentation.
- a client-side apparatus includes a processor, a video display controlled by the processor, and a network interface by which the processor receives Internet video from a server over the Internet.
- a user input device communicates with the processor to send user selection signals to the processor.
- the processor executes a video preview module (VPM) to create an “on-the-fly” storyboard of the Internet video.
- the storyboard includes only a subset of frames in the video, and the processor presents the storyboard on the video display.
- the VPM includes a streamer block establishing one or more parallel connections to the server to obtain from the server the Internet video, and a decoder cooperating with the streamer block to stream in portions of video content from various locations of the Internet video.
- the decoder decodes frames from the Internet video.
- a frame selector operates on data from the streamer block to select a subset of frames from the stream for the storyboard for presentation of the storyboard on the display under control of a display block.
- the VPM further includes a controller block serving as overall manager of the VPM and executing initialization and run-time controls of all blocks, event handling, and responses.
- the controller block can receive a size and a resolution of the display and based thereon determine how many frames from the Internet video to use to establish the storyboard. Responsive to this the streamer block fetches “M” chunks of data for each of a sequence of non-overlapping sections of the Internet video. Each chunk of data can be established by an intraframe (I-frame) and the chunks of data can be equally spaced chunks from each other in a respective section of the Internet video, separated from each other by data in the Internet video.
- I-frame intraframe
- the streamer block establishes multiple parallel streaming connections a link associated with the Internet video.
- the decoder may receive frames from the streamer block, decode the frames, and send the frames into a queue for processing by the frame selector.
- the frame selector may use a feature selection vector to establish an initial storyboard using only a subset of the “M” chunks of data fetched by the streamer block.
- the initial storyboard is presented on the display and subsequently an expanded storyboard established by all of the “M” chunks of data fetched by the streamer block is presented on the display.
- the feature selection vector can be one or more of I-frame size and frame luminance histograms.
- a consumer electronics (CE) device has a housing, a display on the housing, a network interface, and a processor controlling the display and communicating through the network interface.
- the processor executes logic that includes receiving one or more parameters associated with the display. Based on the one or more parameters associated with the display an integer number “M” of data chunks to be extracted from each of a sequence of sections of a video is determined. Responsive to a user selection of a video link presented on the display, the processor communicates with a server on a wide area network to receive from the server a video associated with the video link and extracts “M” data chunks from plural sections of the video received from the server on the wide area network. The processor decodes the data chunks, establishing a storyboard using only the “M” data chunks from each of the plural sections of the video received from the server on the wide area network, and then presents the storyboard on the display.
- a Video Preview Module is contained on a non-transitory computer readable storage medium and is executable by a client side processor to generate an on-the-fly video storyboard that enhances online video browsing experiences of users.
- the VPM is scalable and adaptable in bringing a uniform online video browsing experience over multiple consumer devices.
- FIG. 1 is a block diagram of an example system in accordance with present principles
- FIG. 2 is a block diagram of an example architecture for a software-implemented video preview module (VPW), it being understood that the architecture alternatively may be implemented in hardware;
- VPN video preview module
- FIG. 3 is a schematic diagram showing four example overlapping sections of content in an audio-video stream, with the data chunks obtained by the VPW indicated by hatching;
- FIG. 4 is a screen shot showing a storyboard presentation in the grid mode
- FIG. 5 is a screen shot showing a storyboard presentation in the linear mode.
- FIG. 6 is a flow chart of example logic.
- a consumer electronics (CE) device 12 such as a TV, game player, video disk player, camera, digital clock radio, mobile telephone, personal digital assistant, laptop computer, personal computer (PC), etc. includes a portable lightweight plastic housing 14 bearing a digital processor 16 .
- the processor 16 can control a visual display 18 and an audible display such as one or more speakers.
- the processor 16 may access a media player module such that the CE device 12 has media decoding capability.
- the processor 16 may access one or more computer readable storage media 20 such as but not limited to RAM-based storage, a chip implementing dynamic random access memory (DRAM)) or flash memory or disk storage.
- Software code implementing present logic executable by the CE device 12 may be stored on one of the memories shown to undertake present principles.
- the processor 16 can receive user input signals from various input devices, including a wireless remote control (RC) 22 , a point and click device such as a mouse, a keypad, etc.
- a TV tuner 24 may be provided in some implementations particularly when the CE device is embodied by a TV to receive TV signals from a TV broadcast signal source 26 such as a set-top box, satellite receiver, cable head end, terrestrial TV signal antenna, etc.
- the TV tuner 24 may be implemented in a set top box separately housed from the TV and communicating therewith. In other embodiments, no TV tuner may be provided. Signals from the tuner 24 are sent to the processor 16 for presentation on the display 18 and speakers.
- a network interface 28 such as a wired and/or wireless modem communicates with the processor 16 to provide connectivity to content servers 30 on the Internet (only one server 30 shown in FIG. 1 for clarity).
- the server 30 has a processor 32 and a tangible non-transitory computer readable storage medium 34 such as disk-based and/or solid state storage.
- FIG. 2 shows an example architecture of a software-implemented Video Preview Module (VPM) 36 in accordance with present principles which executes the logic shown in FIG. 4 and described below.
- the VPM 36 is stored on the storage media 20 of the CE device 12 for execution by the CE device processor 16 and thus is a client-side algorithm.
- the VPM 36 is a fast algorithm to create an “on-the-fly” video storyboard of an online stream. Because, unlike the methods mentioned above, the VPM 36 executes on the client-device 12 , the user of the device 12 experiences a uniform enhanced video browsing experience across all supported video content on the Internet.
- the example VPM 36 is a sequential algorithm with a number of function-specific blocks that are managed by a central controller block 38 .
- a streamer block 40 makes one or more parallel connection to the server associated with the link.
- the streamer block 40 streams in portions of video content from various locations of the video stream.
- the decoder 42 and a frame selector 44 operate on the data from the streamer block 40 to select a subset of frames from the stream, typically Motion Picture Experts Group (MPEG) intra-coded frames (I-frames) and referred to herein as candidate frames, for the video storyboard.
- MPEG Motion Picture Experts Group
- I-frames intra-coded frames
- a display block 46 renders the selected frames as a video storyboard on the display 18 .
- the VPM 36 may use a number of open-source multimedia, graphics and AV-related libraries.
- the example software implementation enables the VPM to be portable across multiple client hardware platforms.
- the controller block 38 serves as the overall manager of the VPM 36 and is responsible for initialization and run-time controls of all blocks, event handling, and response of the module.
- the controller 38 functions to match the video storyboard creation to CE device 12 capability and network dynamics.
- the controller 38 records the display 18 size and resolution. This is considered along with the target online video link's format and length to determine the number of frames to be included in the storyboard.
- a set of initial parameters corresponding to the determined number of frames in the storyboard are set in the streamer 40 , decoder 42 , frame selector 44 , and display block 46 . This capability leads to different storyboard presentation forms on different CE devices for the same online video link based on the device capability and if desired network throughput.
- a CE device 12 embodied as a smart phone may present a more condensed video storyboard than a client device embodied TV for the same target video link.
- the number of candidate frames selected to compose the video storyboard varies inversely with greater network congestion (more congestion, fewer candidate frames) and directly with smaller display 18 size (smaller display, fewer candidate frames), and less display 18 resolution (less resolution, fewer frames, since low resolution devices would not benefit as much from an increased number of storyboard frames as higher resolution devices).
- the number of candidate frames selected to compose the video storyboard varies as above with network congestion and smaller display 18 size but may increase with less display 18 resolution, since less processing a needed to generate each lower-resolution candidate frame.
- the number of frames per video section determined by the VPM 36 for storyboard use may decrease as network bandwidth decreases and may increase or decrease depending on the video format of the target link.
- the precise number of candidate frames per device is heuristically determined as appropriate for the particular characteristics of that device and, if desired, network characteristics and target link video format.
- controller 38 receives and processes all key events.
- cursor left, right, up, and down navigation keys, display mode toggle key, and play and pause controls are executed by the controller 38 .
- the streamer block 40 connects with the target link of an associated online video server 30 and streams portions of the content therefrom.
- the streaming operation is controlled by the controller 38 and decoder 42 .
- the controller 38 determines the number of frames to be contained in the storyboard. This number is passed to the streamer 40 at, e.g., device 12 initialization.
- the streamer 40 accesses the target online video stream from the server 30 and divides the target online video stream into non-overlapping sections as shown in FIG. 3 .
- each chunk is established by a single respective I-frame and the I-frame (or in some embodiments, frames) of a chunk constitute the candidate frames for the video storyboard. While the data sections are continuous to each other, the individual chunks shown in cross-hatch are not continuous to each other as shown, so that additional frames of the video content exist between successive chunks in the stream.
- the streamer 40 can support hypertext transfer protocol (HTTP) and file-based access to the target online video link. Also, the streamer 40 may support user datagram protocol (UDP) and real time transport protocol (RTP). Additionally, the use of multiple parallel streaming connections to the video link to fetch data is an area targeted can be implemented, i.e., the streamer 40 may instantiate two connections simultaneously with the same video link so as to simultaneously access two identical versions of the same stream albeit in different portions of the stream (as by executing an automatic fast forward through one stream), as a way to more quickly obtain a temporal sequence of candidate frames for the storyboard.
- HTTP hypertext transfer protocol
- UDP user datagram protocol
- RTP real time transport protocol
- the decoder 42 may, in some implementations, use an open-source software library that support popular online video formats such as MPEG-2, MPEG-4, H.264 (also referred to as advanced video coding), and Windows Media Video (WMV) compression formats.
- the decoder 42 instructs the streamer 40 to fetch chunks of data selected along the video stream as shown in FIG. 3 .
- the decoder 42 receives the I-frames from the streamer 40 , decodes them, and pushes the decoded I-frames into a queue for processing by the frame selector block 44 . Audio segments corresponding to the decoded I-frames may also be decoded if desired, but in some implementations audio information need not be used.
- the VPM 36 may sacrifice some “subjective quality” for low-latency (speed) in the initial generation of the storyboard and then compensate any initial loss of the “subjective quality” by providing a fast visual seek and playback function to the user.
- the frame selector 44 in the example VPM 36 uses I-frame size as the feature vector during initial frame selection. Recall that a total of “M” (e.g., three) I-frames per each section of content are considered in an example embodiment. The I-frame of the largest size is selected to represent the section of content. The chunks used initially to constitute the storyboard may be further winnowed by selecting the largest I-frame of a section content only when it exceeds a predetermined size threshold.
- M e.g., three
- a frame luminance histogram may be used, with the I-frame having the histogram with the most levels being selected.
- the I-frames determined by the selection vector are used to establish an initial storyboard.
- a predetermined time e.g., a few seconds
- a predetermined event e.g., a mouse action or trick play command or other event
- additional I-frames are added to the storyboard incrementally or, if desired, all at once, i.e., the remaining I-frames from each group of “M” are added to the storyboard at once upon the elapse of the period or occurrence of the event.
- the VPM 36 in one embodiment supports both grid and linear display modes.
- the grid display can be set as the default and a user may elect to change to linear if desired.
- FIG. 4 shows the grid display mode, in which the I-frames of the storyboard are arranged in a grid of rows and columns of I-frames with the temporal sequence of the frames progressing from left to tight in each row and from top to bottom among the rows.
- the storyboard encompasses substantially the entire display 18 .
- a line of I-frames progresses from left to right across only a portion of the display with the remainder of the display presenting, e.g., the web page from which the video is selected.
- the processor 16 executing the VPM 36 receives the above-described display 18 parameters and if desired network information and target link video format, and based thereon determines “M” at block 50 in accordance with disclosure above. Responsive to a user selection of a video link presented on the display 18 , at block 52 the processor connects to the server 30 associated with the link and streams or otherwise receives the video from the server at block 54 .
- the processor 16 executing the VPM 36 fetches or otherwise extracts “M” frames from each section of video as described above and decodes them at block 58 .
- the initial storyboard may consist only of a subset of the “M” frames and that subset is arranged into a video storyboard such as shown in FIG. 4 or 5 at block 62 .
- the storyboard is presented on the client display 18 at block 64 . As also discussed above, after the initial presentation, more of the “M” frames can be added to the storyboard.
- the CE device 12 can be Linux Desktop and the processor 16 may be a three gigaHertz Dual-Core processor with one gigabyte of memory and supplied by Intel Corp.
- the CE device 12 can be a high definition TV (HDTV) such as a HDTV provided by Sony Corp. or an HDTV based on the apparatus provided by Intel Corp. under the trade names of “Canmore” with a type x86 core and ninety six megabytes of memory and running versions of Linux 2.6.1x operating system kernels.
- HDTV high definition TV
- a CE device 12 embodied as a desktop computer may use a graphics layer known as “Simple DirectMedia Layer”, a cross-platform multimedia abstraction layer, as its application framework using SDL functions for graphics, event handling and operating system services (threads, mutexes, timers etc.).
- An HDTV-based CE device 12 may use the graphics layer known as “DirectFB”, a cross-platform graphical abstraction layer, for its graphics operations. Event handling and OS services were handled using SDL similar to the Desktop Linux system.
- the VPM 36 may provide visual seek and playback in which a user can select, by means of the above-described input devices, to begin playback of the full video stream starting from any frame on the storyboard. This is equivalent to a visual seek.
- the user may be permitted to tag and comment any frame within the storyboard and share its link (the network address of the full video plus the position of the selected frame within the video) with other users, again by means of appropriately operating the input device of the CE device 12 . This enables another user to jump directly into the tagged frame within the video stream and is an attractive feature in a social network environment.
- video editing capability may be provided in which sections of the video as defined by the frames on the storyboard can be cropped, mixed and saved.
- multiple simultaneous streaming connections to the online link may be used to speed up the streaming module.
- the vectors for selection of frames may be in the compressed video domain so that frame selection may be effected without the need for video decoding.
Abstract
A Video Preview Module, a fast and adaptable software module designed to generate an on-the-fly video storyboard that enhances the online video browsing experience of the user. The VP module is a client-side implementation. This allows the module to be scalable and adaptable in bringing a uniform online video browsing experience over multiple consumer devices.
Description
- This application claims priority to U.S. provisional patent application 61/258,749, filed Nov. 6, 2009.
- The present application relates generally to video preview modules.
- People access online video through multiple devices. While the PC remains the most popular device of access, standalone internet video devices, set top boxes, smart phones, and TV's have recorded gains recently. Regardless of the type of device, sifting through online video listings remains a tedious task. At present, a text based approach is predominantly used to describe video, where a selection is determined by its title and any optional information such as a thumbnail and meta-data containing descriptions and/or user comments.
- Two improvements have been proposed, both of them server-centric. The first is the storyboard approach, in which a video storyboard presentation is added to a video link. A storyboard contains an array of frames that provide a useful guide to the content of the video. Typically, the movement of a mouse along a video link activates the accompanying storyboard. Unfortunately, current methods to generate storyboards are computationally intensive and therefore are typically pre-generated at the server, limiting its widespread use.
- The second approach to enhance the online video experience is playback of “key” frames of the video. In this approach, a mouse action over the initial thumbnail representing the video initiates the playback of a subset of frames called key frames. Key frames are typically frames that can be decoded independently. The key frame generation is in real-time, so that the server need not pre-generate a key frame sequence apart from the stream itself.
- As understood herein, both of the above-discussed server side enhancements impose restrictions in terms of availability, compatibility and performance. Availability is limited to only online video sites offering specific support for the functionality. For example, a client device may be able to access key-frame playback on one site but not on a site that does not support the feature. Compatibility is limited to those client devices that meet the minimum system requirements (memory, graphics, screen size etc.) needed to support the server's presentation format. As understood herein, devices such as smart phones may fall short in meeting these requirements. Finally, neither approach gives consideration to a client device's available bandwidth. This may result in unacceptable latencies in presentation.
- Accordingly, a client-side apparatus includes a processor, a video display controlled by the processor, and a network interface by which the processor receives Internet video from a server over the Internet. A user input device communicates with the processor to send user selection signals to the processor. The processor executes a video preview module (VPM) to create an “on-the-fly” storyboard of the Internet video. The storyboard includes only a subset of frames in the video, and the processor presents the storyboard on the video display. The VPM includes a streamer block establishing one or more parallel connections to the server to obtain from the server the Internet video, and a decoder cooperating with the streamer block to stream in portions of video content from various locations of the Internet video. The decoder decodes frames from the Internet video. A frame selector operates on data from the streamer block to select a subset of frames from the stream for the storyboard for presentation of the storyboard on the display under control of a display block.
- In example implementations the VPM further includes a controller block serving as overall manager of the VPM and executing initialization and run-time controls of all blocks, event handling, and responses. The controller block can receive a size and a resolution of the display and based thereon determine how many frames from the Internet video to use to establish the storyboard. Responsive to this the streamer block fetches “M” chunks of data for each of a sequence of non-overlapping sections of the Internet video. Each chunk of data can be established by an intraframe (I-frame) and the chunks of data can be equally spaced chunks from each other in a respective section of the Internet video, separated from each other by data in the Internet video.
- In some embodiments the streamer block establishes multiple parallel streaming connections a link associated with the Internet video. The decoder may receive frames from the streamer block, decode the frames, and send the frames into a queue for processing by the frame selector.
- Further, the frame selector may use a feature selection vector to establish an initial storyboard using only a subset of the “M” chunks of data fetched by the streamer block. In such an embodiment the initial storyboard is presented on the display and subsequently an expanded storyboard established by all of the “M” chunks of data fetched by the streamer block is presented on the display. The feature selection vector can be one or more of I-frame size and frame luminance histograms.
- In another aspect, a consumer electronics (CE) device has a housing, a display on the housing, a network interface, and a processor controlling the display and communicating through the network interface. The processor executes logic that includes receiving one or more parameters associated with the display. Based on the one or more parameters associated with the display an integer number “M” of data chunks to be extracted from each of a sequence of sections of a video is determined. Responsive to a user selection of a video link presented on the display, the processor communicates with a server on a wide area network to receive from the server a video associated with the video link and extracts “M” data chunks from plural sections of the video received from the server on the wide area network. The processor decodes the data chunks, establishing a storyboard using only the “M” data chunks from each of the plural sections of the video received from the server on the wide area network, and then presents the storyboard on the display.
- In another aspect, a Video Preview Module (VPM) is contained on a non-transitory computer readable storage medium and is executable by a client side processor to generate an on-the-fly video storyboard that enhances online video browsing experiences of users. The VPM is scalable and adaptable in bringing a uniform online video browsing experience over multiple consumer devices.
- The details of the present invention, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:
-
FIG. 1 is a block diagram of an example system in accordance with present principles; -
FIG. 2 is a block diagram of an example architecture for a software-implemented video preview module (VPW), it being understood that the architecture alternatively may be implemented in hardware; -
FIG. 3 is a schematic diagram showing four example overlapping sections of content in an audio-video stream, with the data chunks obtained by the VPW indicated by hatching; -
FIG. 4 is a screen shot showing a storyboard presentation in the grid mode; -
FIG. 5 is a screen shot showing a storyboard presentation in the linear mode; and -
FIG. 6 is a flow chart of example logic. - Referring initially to
FIG. 1 , a consumer electronics (CE)device 12 such as a TV, game player, video disk player, camera, digital clock radio, mobile telephone, personal digital assistant, laptop computer, personal computer (PC), etc. includes a portable lightweightplastic housing 14 bearing a digital processor 16. The processor 16 can control avisual display 18 and an audible display such as one or more speakers. The processor 16 may access a media player module such that theCE device 12 has media decoding capability. - To undertake present principles, the processor 16 may access one or more computer
readable storage media 20 such as but not limited to RAM-based storage, a chip implementing dynamic random access memory (DRAM)) or flash memory or disk storage. Software code implementing present logic executable by theCE device 12 may be stored on one of the memories shown to undertake present principles. - The processor 16 can receive user input signals from various input devices, including a wireless remote control (RC) 22, a point and click device such as a mouse, a keypad, etc. A
TV tuner 24 may be provided in some implementations particularly when the CE device is embodied by a TV to receive TV signals from a TVbroadcast signal source 26 such as a set-top box, satellite receiver, cable head end, terrestrial TV signal antenna, etc. TheTV tuner 24 may be implemented in a set top box separately housed from the TV and communicating therewith. In other embodiments, no TV tuner may be provided. Signals from thetuner 24 are sent to the processor 16 for presentation on thedisplay 18 and speakers. - As shown in
FIG. 1 , a network interface 28 such as a wired and/or wireless modem communicates with the processor 16 to provide connectivity tocontent servers 30 on the Internet (only oneserver 30 shown inFIG. 1 for clarity). Theserver 30 has aprocessor 32 and a tangible non-transitory computerreadable storage medium 34 such as disk-based and/or solid state storage. -
FIG. 2 shows an example architecture of a software-implemented Video Preview Module (VPM) 36 in accordance with present principles which executes the logic shown inFIG. 4 and described below. TheVPM 36 is stored on thestorage media 20 of theCE device 12 for execution by the CE device processor 16 and thus is a client-side algorithm. In overview, theVPM 36 is a fast algorithm to create an “on-the-fly” video storyboard of an online stream. Because, unlike the methods mentioned above, theVPM 36 executes on the client-device 12, the user of thedevice 12 experiences a uniform enhanced video browsing experience across all supported video content on the Internet. - As shown, the
example VPM 36 is a sequential algorithm with a number of function-specific blocks that are managed by acentral controller block 38. Given an online video link to access, a streamer block 40 makes one or more parallel connection to the server associated with the link. Under the control of adecoder 42 and thecontroller 38, the streamer block 40 streams in portions of video content from various locations of the video stream. Thedecoder 42 and aframe selector 44 operate on the data from the streamer block 40 to select a subset of frames from the stream, typically Motion Picture Experts Group (MPEG) intra-coded frames (I-frames) and referred to herein as candidate frames, for the video storyboard. Adisplay block 46 then renders the selected frames as a video storyboard on thedisplay 18. - The
VPM 36 may use a number of open-source multimedia, graphics and AV-related libraries. The example software implementation enables the VPM to be portable across multiple client hardware platforms. - With greater specificity, the
controller block 38 serves as the overall manager of theVPM 36 and is responsible for initialization and run-time controls of all blocks, event handling, and response of the module. Thecontroller 38 functions to match the video storyboard creation toCE device 12 capability and network dynamics. At initialization, thecontroller 38 records thedisplay 18 size and resolution. This is considered along with the target online video link's format and length to determine the number of frames to be included in the storyboard. A set of initial parameters corresponding to the determined number of frames in the storyboard are set in the streamer 40,decoder 42,frame selector 44, anddisplay block 46. This capability leads to different storyboard presentation forms on different CE devices for the same online video link based on the device capability and if desired network throughput. - As an example, a
CE device 12 embodied as a smart phone may present a more condensed video storyboard than a client device embodied TV for the same target video link. In one implementation, the number of candidate frames selected to compose the video storyboard varies inversely with greater network congestion (more congestion, fewer candidate frames) and directly withsmaller display 18 size (smaller display, fewer candidate frames), andless display 18 resolution (less resolution, fewer frames, since low resolution devices would not benefit as much from an increased number of storyboard frames as higher resolution devices). In another implementation, the number of candidate frames selected to compose the video storyboard varies as above with network congestion andsmaller display 18 size but may increase withless display 18 resolution, since less processing a needed to generate each lower-resolution candidate frame. Furthermore, the number of frames per video section determined by theVPM 36 for storyboard use may decrease as network bandwidth decreases and may increase or decrease depending on the video format of the target link. In any case, the precise number of candidate frames per device is heuristically determined as appropriate for the particular characteristics of that device and, if desired, network characteristics and target link video format. - Additionally, the
controller 38 receives and processes all key events. By way of non-limiting example, cursor left, right, up, and down navigation keys, display mode toggle key, and play and pause controls are executed by thecontroller 38. - The streamer block 40, on the other hand, connects with the target link of an associated
online video server 30 and streams portions of the content therefrom. The streaming operation is controlled by thecontroller 38 anddecoder 42. Based on the CE device capabilities mentioned above and if desired the network throughput, thecontroller 38 determines the number of frames to be contained in the storyboard. This number is passed to the streamer 40 at, e.g.,device 12 initialization. When a target video stream link from aserver 30 is subsequently selected by a user of theCE device 12, the streamer 40 accesses the target online video stream from theserver 30 and divides the target online video stream into non-overlapping sections as shown inFIG. 3 . - The streamer 40 fetches “M” equally spaced chunks of data per each section (in the example shown in
FIG. 3 , M=3). Shown in cross-hatch inFIG. 3 is an illustration of the data chunks obtained per section by the streamer 40. In one example, each chunk is established by a single respective I-frame and the I-frame (or in some embodiments, frames) of a chunk constitute the candidate frames for the video storyboard. While the data sections are continuous to each other, the individual chunks shown in cross-hatch are not continuous to each other as shown, so that additional frames of the video content exist between successive chunks in the stream. - The streamer 40 can support hypertext transfer protocol (HTTP) and file-based access to the target online video link. Also, the streamer 40 may support user datagram protocol (UDP) and real time transport protocol (RTP). Additionally, the use of multiple parallel streaming connections to the video link to fetch data is an area targeted can be implemented, i.e., the streamer 40 may instantiate two connections simultaneously with the same video link so as to simultaneously access two identical versions of the same stream albeit in different portions of the stream (as by executing an automatic fast forward through one stream), as a way to more quickly obtain a temporal sequence of candidate frames for the storyboard.
- The
decoder 42 may, in some implementations, use an open-source software library that support popular online video formats such as MPEG-2, MPEG-4, H.264 (also referred to as advanced video coding), and Windows Media Video (WMV) compression formats. In addition to audio and video decoding of candidate storyboard frames, thedecoder 42 instructs the streamer 40 to fetch chunks of data selected along the video stream as shown inFIG. 3 . - As mentioned above, only I-frames within the “M” data chunks (in hatched in
FIG. 3 ) which form “M” subsets of each non-overlapping section of content may be used in the storyboard. Accordingly, in such an implementation thedecoder 42 receives the I-frames from the streamer 40, decodes them, and pushes the decoded I-frames into a queue for processing by theframe selector block 44. Audio segments corresponding to the decoded I-frames may also be decoded if desired, but in some implementations audio information need not be used. - To achieve a subjective summary of the selected video content that enables the user to piece together the content story, all of the I-frames of the stream may be used so that none of the “key events” in the stream are missed, but as mentioned above, such comprehensive storyboarding is not suited for a real-time implementation particularly on a resource limited client device. Hence, the
VPM 36 may sacrifice some “subjective quality” for low-latency (speed) in the initial generation of the storyboard and then compensate any initial loss of the “subjective quality” by providing a fast visual seek and playback function to the user. - In one specific implementation of the above tradeoff, the
frame selector 44 in theexample VPM 36 uses I-frame size as the feature vector during initial frame selection. Recall that a total of “M” (e.g., three) I-frames per each section of content are considered in an example embodiment. The I-frame of the largest size is selected to represent the section of content. The chunks used initially to constitute the storyboard may be further winnowed by selecting the largest I-frame of a section content only when it exceeds a predetermined size threshold. - In another embodiment, instead of using I-frame size as a selection vector, a frame luminance histogram may be used, with the I-frame having the histogram with the most levels being selected.
- In any case, the I-frames determined by the selection vector are used to establish an initial storyboard. After a predetermined time, e.g., a few seconds, or a predetermined event, e.g., a mouse action or trick play command or other event, additional I-frames are added to the storyboard incrementally or, if desired, all at once, i.e., the remaining I-frames from each group of “M” are added to the storyboard at once upon the elapse of the period or occurrence of the event.
- The
VPM 36 in one embodiment supports both grid and linear display modes. The grid display can be set as the default and a user may elect to change to linear if desired.FIG. 4 shows the grid display mode, in which the I-frames of the storyboard are arranged in a grid of rows and columns of I-frames with the temporal sequence of the frames progressing from left to tight in each row and from top to bottom among the rows. InFIG. 4 , the storyboard encompasses substantially theentire display 18. In contrast, as shown inFIG. 5 in the linear mode a line of I-frames progresses from left to right across only a portion of the display with the remainder of the display presenting, e.g., the web page from which the video is selected. - Now referring to
FIG. 6 , atblock 48 the processor 16 executing theVPM 36 receives the above-describeddisplay 18 parameters and if desired network information and target link video format, and based thereon determines “M” atblock 50 in accordance with disclosure above. Responsive to a user selection of a video link presented on thedisplay 18, at block 52 the processor connects to theserver 30 associated with the link and streams or otherwise receives the video from the server atblock 54. - Moving to block 56, the processor 16 executing the
VPM 36 fetches or otherwise extracts “M” frames from each section of video as described above and decodes them atblock 58. Proceeding to block 60, as discussed previously the initial storyboard may consist only of a subset of the “M” frames and that subset is arranged into a video storyboard such as shown inFIG. 4 or 5 atblock 62. The storyboard is presented on theclient display 18 atblock 64. As also discussed above, after the initial presentation, more of the “M” frames can be added to the storyboard. - In example non-limiting implementations given for illustration only, the
CE device 12 can be Linux Desktop and the processor 16 may be a three gigaHertz Dual-Core processor with one gigabyte of memory and supplied by Intel Corp. In another implementation theCE device 12 can be a high definition TV (HDTV) such as a HDTV provided by Sony Corp. or an HDTV based on the apparatus provided by Intel Corp. under the trade names of “Canmore” with a type x86 core and ninety six megabytes of memory and running versions of Linux 2.6.1x operating system kernels. - It will be appreciated that while the VPM may remain unchanged device to device, the graphics layer may change depending on the
display 18 being used. As an example, aCE device 12 embodied as a desktop computer may use a graphics layer known as “Simple DirectMedia Layer”, a cross-platform multimedia abstraction layer, as its application framework using SDL functions for graphics, event handling and operating system services (threads, mutexes, timers etc.). An HDTV-basedCE device 12 may use the graphics layer known as “DirectFB”, a cross-platform graphical abstraction layer, for its graphics operations. Event handling and OS services were handled using SDL similar to the Desktop Linux system. - In addition to the above, the
VPM 36 may provide visual seek and playback in which a user can select, by means of the above-described input devices, to begin playback of the full video stream starting from any frame on the storyboard. This is equivalent to a visual seek. Furthermore, the user may be permitted to tag and comment any frame within the storyboard and share its link (the network address of the full video plus the position of the selected frame within the video) with other users, again by means of appropriately operating the input device of theCE device 12. This enables another user to jump directly into the tagged frame within the video stream and is an attractive feature in a social network environment. Furthermore, if desired video editing capability may be provided in which sections of the video as defined by the frames on the storyboard can be cropped, mixed and saved. - Still further, as noted above multiple simultaneous streaming connections to the online link may be used to speed up the streaming module. The vectors for selection of frames may be in the compressed video domain so that frame selection may be effected without the need for video decoding.
- While the particular VIDEO PREVIEW MODULE TO ENHANCE ONLINE VIDEO EXPERIENCE is herein shown and described in detail, it is to be understood that the subject matter which is encompassed by the present invention is limited only by the claims.
Claims (20)
1. Client-side apparatus comprising:
processor;
video display configured for being controlled by the processor;
network interface by which the processor is configured to receive Internet video from a server over the Internet;
user input device configured for communicating with the processor to send user selection signals to the processor;
the processor configured for executing a video preview module (VPM) to create an “on-the-fly” storyboard of the Internet video, the storyboard including only a subset of frames in the video, the processor presenting the storyboard on the video display;
wherein the VPM when executed by the processor uses a feature selection vector to establish an initial storyboard, and the feature selection vector is a frame luminance histogram.
2. The apparatus of claim 1 , wherein the VPM further includes a controller block serving as overall manager of the VPM and configured for executing initialization and run-time controls of all blocks, event handling, and responses.
3. The apparatus of claim 2 , wherein the controller block is configured to receive at least a size and a resolution of the display and based at least partially thereon determines how many frames from the Internet video to use to establish the storyboard.
4. The apparatus of claim 1 , wherein the VPM is configured to fetche “M” chunks of data for each of a sequence of non-overlapping sections of the Internet video.
5. The apparatus of claim 1 , wherein each chunk of data is established by an intraframe (I-frame).
6. The apparatus of claim 1 , wherein the chunks of data are equally spaced chunks from each other in a respective section of the Internet video and are separated from each other by data in the Internet video.
7. The apparatus of claim 1 , wherein the VPM is configured to establish multiple parallel streaming connections to a link associated with the Internet video.
8. The apparatus of claim 1 , wherein the VPM is configured to receive frames from the streamer block, decode the frames, and send the frames into a queue for processing.
9. The apparatus of claim 4 , wherein the VPM is configured to use the feature selection vector to establish an initial storyboard using only a subset of the “M” chunks of data.
10. The apparatus of claim 9 , wherein the initial storyboard is presented on the display and subsequently an expanded storyboard established by all of the “M” chunks of data fetched by the streamer block VPM is presented on the display.
11. The apparatus of claim 9 , a frame characterized by a histogram with a first number of levels is selected for the storyboard over a frame characterized by a histogram having a second number of levels based on the first number of levels being more than the second number of levels.
12. Consumer electronics (CE) device comprising:
display;
network interface;
processor configured for controlling the display and for communicating through the network interface;
the processor configured for executing logic including:
receiving one or more parameters associated with the display;
based on the one or more parameters associated with the display determining an integer number “M” of data chunks to be extracted from each of a sequence of sections of a video;
responsive to a user selection of a video link presented on the display, communicating with a server on a wide area network to receive from the server a video associated with the video link;
extracting “M” data chunks from at least plural sections of the video received from the server on the wide area network;
decoding the data chunks;
establishing a storyboard using only the “M” data chunks from each of the plural sections of the video received from the server on the wide area network; and
presenting the storyboard on the display wherein “M” varies with the format of the video.
13. The CE device of claim 12 , wherein the processor is configured for, prior to establishing a storyboard using only the “M” data chunks from each of the plural sections of the video received from the server on the wide area network establishing an initial storyboard using only of a subset of the “M” data chunks and presents the initial storyboard on the display.
14. The CE device of claim 13 , wherein the processor is configured to use a feature selection vector to establish the initial storyboard.
15. The CE device of claim 14 , wherein the initial storyboard is presented on the display and subsequently an expanded storyboard established by all of the “M” data chunks is presented on the display.
16. The CE device of claim 14 , wherein the feature selection vector is one or more of I-frame size, a frame luminance histogram.
17. A Video Preview Module (VPM) contained on a computer readable storage medium that is not a carrier wave, the VPM being executable by a processor to generate an on-the-fly video storyboard that enhances online video browsing experiences of users, the VPM when executed by the processor using frame luminance histogram information to select frames from a video, the frames selected from the video establishing the storyboard.
18. The VPM of claim 17 , comprising:
a streamer block which when executed configures the processor for establishing one or more parallel connections to an Internet server to obtain from the server an Internet video;
a decoder cooperating with the streamer block which when executed configures to the processor to stream in portions of video content from various locations of the Internet video, the decoder which when executed configures to the processor for decoding frames from the Internet video;
a frame selector which when executed configures to the processor for operating on data from the streamer block to select a subset of frames from the stream for the storyboard for presentation of the storyboard on a display under control of a display block.
19. The VPM of claim 18 , wherein the VPM further includes a controller block which when executed configures to the processor for serving as overall manager of the VPM and executing initialization and run-time controls of all blocks, event handling, and responses.
20. The VPM of claim 19 , wherein the controller block is configured to receive at least a size and a resolution of the display and based at least partially thereon determines how many frames from the Internet video to use to establish the storyboard.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/771,563 US20130174197A1 (en) | 2009-11-06 | 2013-02-20 | Video preview module to enhance online video experience |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US25874909P | 2009-11-06 | 2009-11-06 | |
US12/844,119 US8438484B2 (en) | 2009-11-06 | 2010-07-27 | Video preview module to enhance online video experience |
US13/771,563 US20130174197A1 (en) | 2009-11-06 | 2013-02-20 | Video preview module to enhance online video experience |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/844,119 Continuation US8438484B2 (en) | 2009-11-06 | 2010-07-27 | Video preview module to enhance online video experience |
Publications (1)
Publication Number | Publication Date |
---|---|
US20130174197A1 true US20130174197A1 (en) | 2013-07-04 |
Family
ID=43975076
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/844,119 Active 2031-02-22 US8438484B2 (en) | 2009-11-06 | 2010-07-27 | Video preview module to enhance online video experience |
US13/771,563 Abandoned US20130174197A1 (en) | 2009-11-06 | 2013-02-20 | Video preview module to enhance online video experience |
US13/917,479 Active US8818022B2 (en) | 2009-11-06 | 2013-06-13 | Associating cameras with users and objects in a social networking system |
US14/333,453 Active US8965041B2 (en) | 2009-11-06 | 2014-07-16 | Associating cameras with users and objects in a social networking system |
US14/596,483 Active US9485423B2 (en) | 2009-11-06 | 2015-01-14 | Associating cameras with users and objects in a social networking system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/844,119 Active 2031-02-22 US8438484B2 (en) | 2009-11-06 | 2010-07-27 | Video preview module to enhance online video experience |
Family Applications After (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/917,479 Active US8818022B2 (en) | 2009-11-06 | 2013-06-13 | Associating cameras with users and objects in a social networking system |
US14/333,453 Active US8965041B2 (en) | 2009-11-06 | 2014-07-16 | Associating cameras with users and objects in a social networking system |
US14/596,483 Active US9485423B2 (en) | 2009-11-06 | 2015-01-14 | Associating cameras with users and objects in a social networking system |
Country Status (1)
Country | Link |
---|---|
US (5) | US8438484B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140223482A1 (en) * | 2013-02-05 | 2014-08-07 | Redux, Inc. | Video preview creation with link |
US10528208B2 (en) | 2015-05-31 | 2020-01-07 | International Busines Machines Corporation | Instantaneous preview of a data stream with non-linear time warping |
Families Citing this family (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9185469B2 (en) | 2010-09-30 | 2015-11-10 | Kodak Alaris Inc. | Summarizing image collection using a social network |
US8606776B2 (en) * | 2011-02-18 | 2013-12-10 | Google Inc. | Affinity based ranked for search and display |
EP2689395B1 (en) * | 2011-03-25 | 2016-12-14 | Telefonaktiebolaget LM Ericsson (publ) | Method in streaming client for handling video streaming data, streaming client, computer programs and computer program products |
KR101781861B1 (en) * | 2011-04-04 | 2017-09-26 | 엘지전자 주식회사 | Image display device and method of displaying text in the same |
US9256859B2 (en) | 2011-07-26 | 2016-02-09 | Salesforce.Com, Inc. | Systems and methods for fragmenting newsfeed objects |
US20140082661A1 (en) * | 2012-03-06 | 2014-03-20 | Google Inc. | Low latency video storyboard delivery with selectable resolution levels |
US8756627B2 (en) * | 2012-04-19 | 2014-06-17 | Jumpercut, Inc. | Distributed video creation |
US9418079B2 (en) * | 2012-11-01 | 2016-08-16 | Google Inc. | Image comparison process |
US10009658B2 (en) * | 2013-03-11 | 2018-06-26 | Sony Corporation | Multiview TV template creation and display layout modification |
US10001904B1 (en) * | 2013-06-26 | 2018-06-19 | R3 Collaboratives, Inc. | Categorized and tagged video annotation |
US9686276B2 (en) * | 2013-12-30 | 2017-06-20 | AdMobius, Inc. | Cookieless management translation and resolving of multiple device identities for multiple networks |
US20150286865A1 (en) * | 2014-04-08 | 2015-10-08 | Sony Corporation | Coordination of object location data with video data |
US10552888B1 (en) * | 2014-09-30 | 2020-02-04 | Amazon Technologies, Inc. | System for determining resources from image data |
CA3012143A1 (en) | 2014-11-10 | 2016-05-19 | Swarms Ventures, Llc | Method and system for programmable loop recording |
US9946441B2 (en) * | 2015-05-05 | 2018-04-17 | Kai Austin | Computerized system and method for creative facilitation and organization |
AU2016291660B2 (en) | 2015-07-15 | 2021-10-21 | 15 Seconds of Fame, Inc. | Apparatus and methods for facial recognition and video analytics to identify individuals in contextual video streams |
CA3040856C (en) * | 2015-10-21 | 2024-01-02 | 15 Seconds of Fame, Inc. | Methods and apparatus for false positive minimization in facial recognition applications |
US10455270B2 (en) | 2016-03-15 | 2019-10-22 | Sony Corporation | Content surfing, preview and selection by sequentially connecting tiled content channels |
US10284900B2 (en) | 2016-03-15 | 2019-05-07 | Sony Corporation | Multiview as an application for physical digital media |
US20170272829A1 (en) * | 2016-03-15 | 2017-09-21 | Sony Corporation | Multiview tv environment that is curated by a broadcaster or service provider |
US9894404B2 (en) | 2016-03-15 | 2018-02-13 | Sony Corporation | Multiview TV custom display creation by aggregation of provider content elements |
US9928442B2 (en) * | 2016-03-22 | 2018-03-27 | Disney Enterprises, Inc. | Joint photo stream and blog post summarization and exploration |
US10586238B2 (en) | 2016-06-22 | 2020-03-10 | Microsoft Technology Licensing, Llc | Automation of image validation |
JP6708077B2 (en) * | 2016-09-16 | 2020-06-10 | カシオ計算機株式会社 | Display information generation device, information management system, display information generation method, and program |
US10447924B2 (en) | 2017-06-30 | 2019-10-15 | Microsoft Technology Licensing, Llc | Camera usage notification |
US10474988B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Predicting inventory events using foreground/background processing |
US10650545B2 (en) | 2017-08-07 | 2020-05-12 | Standard Cognition, Corp. | Systems and methods to check-in shoppers in a cashier-less store |
US10853965B2 (en) | 2017-08-07 | 2020-12-01 | Standard Cognition, Corp | Directional impression analysis using deep learning |
US11200692B2 (en) | 2017-08-07 | 2021-12-14 | Standard Cognition, Corp | Systems and methods to check-in shoppers in a cashier-less store |
US11023850B2 (en) | 2017-08-07 | 2021-06-01 | Standard Cognition, Corp. | Realtime inventory location management using deep learning |
US10474991B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Deep learning-based store realograms |
US11250376B2 (en) | 2017-08-07 | 2022-02-15 | Standard Cognition, Corp | Product correlation analysis using deep learning |
US11232687B2 (en) | 2017-08-07 | 2022-01-25 | Standard Cognition, Corp | Deep learning-based shopper statuses in a cashier-less store |
US10931854B2 (en) * | 2017-10-26 | 2021-02-23 | Facebook, Inc. | Aggregating video streams from cameras based on social connections in an online system |
US10924809B2 (en) | 2017-12-05 | 2021-02-16 | Silicon Beach Media II, Inc. | Systems and methods for unified presentation of on-demand, live, social or market content |
US11146845B2 (en) * | 2017-12-05 | 2021-10-12 | Relola Inc. | Systems and methods for unified presentation of synchronized on-demand, live, social or market content |
US10805367B2 (en) * | 2017-12-29 | 2020-10-13 | Facebook, Inc. | Systems and methods for sharing content |
WO2020023801A1 (en) * | 2018-07-26 | 2020-01-30 | Standard Cognition, Corp. | Systems and methods to check-in shoppers in a cashier-less store |
US10936856B2 (en) | 2018-08-31 | 2021-03-02 | 15 Seconds of Fame, Inc. | Methods and apparatus for reducing false positives in facial recognition |
US20200226171A1 (en) * | 2019-01-16 | 2020-07-16 | FinancialForce.com, Inc. | Automatic characterization of media files |
US11010596B2 (en) | 2019-03-07 | 2021-05-18 | 15 Seconds of Fame, Inc. | Apparatus and methods for facial recognition systems to identify proximity-based connections |
US11232575B2 (en) | 2019-04-18 | 2022-01-25 | Standard Cognition, Corp | Systems and methods for deep learning-based subject persistence |
US11341351B2 (en) | 2020-01-03 | 2022-05-24 | 15 Seconds of Fame, Inc. | Methods and apparatus for facial recognition on a user device |
CN111209490A (en) * | 2020-04-24 | 2020-05-29 | 深圳市爱聊科技有限公司 | Friend-making recommendation method based on user information, electronic device and storage medium |
US11237708B2 (en) | 2020-05-27 | 2022-02-01 | Bank Of America Corporation | Video previews for interactive videos using a markup language |
US11461535B2 (en) | 2020-05-27 | 2022-10-04 | Bank Of America Corporation | Video buffering for interactive videos using a markup language |
US11303853B2 (en) | 2020-06-26 | 2022-04-12 | Standard Cognition, Corp. | Systems and methods for automated design of camera placement and cameras arrangements for autonomous checkout |
US11361468B2 (en) | 2020-06-26 | 2022-06-14 | Standard Cognition, Corp. | Systems and methods for automated recalibration of sensors for autonomous checkout |
Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5956026A (en) * | 1997-12-19 | 1999-09-21 | Sharp Laboratories Of America, Inc. | Method for hierarchical summarization and browsing of digital video |
US20020075572A1 (en) * | 2000-12-14 | 2002-06-20 | John Boreczky | System and method for video navigation and client side indexing |
US20030167472A1 (en) * | 2002-03-04 | 2003-09-04 | Monique Barbanson | Systems and methods for displaying video streams |
US6647535B1 (en) * | 1999-03-18 | 2003-11-11 | Xerox Corporation | Methods and systems for real-time storyboarding with a web page and graphical user interface for automatic video parsing and browsing |
US20030210886A1 (en) * | 2002-05-07 | 2003-11-13 | Ying Li | Scalable video summarization and navigation system and method |
US20040128308A1 (en) * | 2002-12-31 | 2004-07-01 | Pere Obrador | Scalably presenting a collection of media objects |
US20050002452A1 (en) * | 1999-01-29 | 2005-01-06 | Frederic Dufaux | System for selecting a keyframe to represent a video |
US20050257152A1 (en) * | 2004-05-13 | 2005-11-17 | Sony Corporation | Image data processing apparatus, image data processing method, program, and recording medium |
US20070083883A1 (en) * | 2004-03-29 | 2007-04-12 | Deng Kevin K | Methods and apparatus to detect a blank frame in a digital video broadcast signal |
US20080192840A1 (en) * | 2007-02-09 | 2008-08-14 | Microsoft Corporation | Smart video thumbnail |
US20080320413A1 (en) * | 2007-06-25 | 2008-12-25 | Microsoft Corporation | Dynamic user interface for previewing live content |
US8307399B2 (en) * | 2006-12-20 | 2012-11-06 | Lg Electronics Inc. | Method of providing key frames of video in mobile terminal |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CA2067418C (en) * | 1991-07-22 | 1998-05-19 | Sung M. Choi | Frame buffer organization and control for real-time image decompression |
US5550965A (en) * | 1993-12-27 | 1996-08-27 | Lucent Technologies Inc. | Method and system for operating a data processor to index primary data in real time with iconic table of contents |
US5606365A (en) * | 1995-03-28 | 1997-02-25 | Eastman Kodak Company | Interactive camera for network processing of captured images |
US5884056A (en) | 1995-12-28 | 1999-03-16 | International Business Machines Corporation | Method and system for video browsing on the world wide web |
US6340971B1 (en) | 1997-02-03 | 2002-01-22 | U.S. Philips Corporation | Method and device for keyframe-based video displaying using a video cursor frame in a multikeyframe screen |
US6166735A (en) * | 1997-12-03 | 2000-12-26 | International Business Machines Corporation | Video story board user interface for selective downloading and displaying of desired portions of remote-stored video data objects |
US6154771A (en) * | 1998-06-01 | 2000-11-28 | Mediastra, Inc. | Real-time receipt, decompression and play of compressed streaming video/hypervideo; with thumbnail display of past scenes and with replay, hyperlinking and/or recording permissively intiated retrospectively |
US6636259B1 (en) * | 2000-07-26 | 2003-10-21 | Ipac Acquisition Subsidiary I, Llc | Automatically configuring a web-enabled digital camera to access the internet |
JP4214346B2 (en) * | 2000-07-31 | 2009-01-28 | 富士フイルム株式会社 | Communication device, server, service providing apparatus, service system and service method, service menu providing method and service menu providing system |
US8020183B2 (en) * | 2000-09-14 | 2011-09-13 | Sharp Laboratories Of America, Inc. | Audiovisual management system |
US7197531B2 (en) | 2000-12-29 | 2007-03-27 | Fotomedia Technologies, Llc | Meta-application architecture for integrating photo-service websites for browser-enabled devices |
US20020175917A1 (en) | 2001-04-10 | 2002-11-28 | Dipto Chakravarty | Method and system for streaming media manager |
US7433710B2 (en) * | 2001-04-20 | 2008-10-07 | Lightsurf Technologies, Inc. | System and methodology for automated provisioning of new user accounts |
US6999111B2 (en) * | 2001-06-26 | 2006-02-14 | Eastman Kodak Company | Electronic camera and system for transmitting digital over a communication network |
US6961908B2 (en) * | 2001-12-05 | 2005-11-01 | International Business Machines Corporation | System and method for navigating graphical images |
US7383508B2 (en) * | 2002-06-19 | 2008-06-03 | Microsoft Corporation | Computer user interface for interacting with video cliplets generated from digital video |
US7535492B2 (en) | 2002-07-02 | 2009-05-19 | Lightsurf Technologies, Inc. | Imaging system providing automated fulfillment of image photofinishing based on location |
US7333714B2 (en) * | 2004-02-10 | 2008-02-19 | Broadcom Corporation | Method and system for performing reverse play of SD MPEG video |
US20050228849A1 (en) * | 2004-03-24 | 2005-10-13 | Tong Zhang | Intelligent key-frame extraction from a video |
US8953908B2 (en) * | 2004-06-22 | 2015-02-10 | Digimarc Corporation | Metadata management and generation using perceptual features |
EP2201480A1 (en) * | 2007-08-28 | 2010-06-30 | Hewlett-Packard Development Company, L.P. | Method and apparatus for multimedia data management on mobile devices |
US8307395B2 (en) * | 2008-04-22 | 2012-11-06 | Porto Technology, Llc | Publishing key frames of a video content item being viewed by a first user to one or more second users |
US20100241961A1 (en) * | 2009-03-23 | 2010-09-23 | Peterson Troy A | Content presentation control and progression indicator |
US20120114196A1 (en) * | 2010-11-04 | 2012-05-10 | The Go Daddy Group, Inc. | Methods for Person's Verification Using Photographs on Identification Documents |
-
2010
- 2010-07-27 US US12/844,119 patent/US8438484B2/en active Active
-
2013
- 2013-02-20 US US13/771,563 patent/US20130174197A1/en not_active Abandoned
- 2013-06-13 US US13/917,479 patent/US8818022B2/en active Active
-
2014
- 2014-07-16 US US14/333,453 patent/US8965041B2/en active Active
-
2015
- 2015-01-14 US US14/596,483 patent/US9485423B2/en active Active
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5956026A (en) * | 1997-12-19 | 1999-09-21 | Sharp Laboratories Of America, Inc. | Method for hierarchical summarization and browsing of digital video |
US20050002452A1 (en) * | 1999-01-29 | 2005-01-06 | Frederic Dufaux | System for selecting a keyframe to represent a video |
US6647535B1 (en) * | 1999-03-18 | 2003-11-11 | Xerox Corporation | Methods and systems for real-time storyboarding with a web page and graphical user interface for automatic video parsing and browsing |
US20020075572A1 (en) * | 2000-12-14 | 2002-06-20 | John Boreczky | System and method for video navigation and client side indexing |
US7401351B2 (en) * | 2000-12-14 | 2008-07-15 | Fuji Xerox Co., Ltd. | System and method for video navigation and client side indexing |
US20030167472A1 (en) * | 2002-03-04 | 2003-09-04 | Monique Barbanson | Systems and methods for displaying video streams |
US20030210886A1 (en) * | 2002-05-07 | 2003-11-13 | Ying Li | Scalable video summarization and navigation system and method |
US20040128308A1 (en) * | 2002-12-31 | 2004-07-01 | Pere Obrador | Scalably presenting a collection of media objects |
US20070083883A1 (en) * | 2004-03-29 | 2007-04-12 | Deng Kevin K | Methods and apparatus to detect a blank frame in a digital video broadcast signal |
US20050257152A1 (en) * | 2004-05-13 | 2005-11-17 | Sony Corporation | Image data processing apparatus, image data processing method, program, and recording medium |
US8307399B2 (en) * | 2006-12-20 | 2012-11-06 | Lg Electronics Inc. | Method of providing key frames of video in mobile terminal |
US20080192840A1 (en) * | 2007-02-09 | 2008-08-14 | Microsoft Corporation | Smart video thumbnail |
US20080320413A1 (en) * | 2007-06-25 | 2008-12-25 | Microsoft Corporation | Dynamic user interface for previewing live content |
Non-Patent Citations (2)
Title |
---|
"Video Scene Decomposition with the Motion Picture Parser" by E. Deardorff et al., Proc. IS&T/SPIE Symposium on Electronic Imaging Science and Technology (Digital Video Compression and Processing on Personal Computers: Algorithms and Technologies), San Jose, February 1994, SPIE Vol. 2187, pp. 44-55. * |
Boreczky, Jr., et al., "A Hidden Markov Model Framework for Video Segmentation Using Audio and Image Features," In Proc. ICASSP '98, IEEE, Seattle, Washington (May 1998) pp. 3741-3744. * |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140223482A1 (en) * | 2013-02-05 | 2014-08-07 | Redux, Inc. | Video preview creation with link |
US9530452B2 (en) * | 2013-02-05 | 2016-12-27 | Alc Holdings, Inc. | Video preview creation with link |
US9589594B2 (en) | 2013-02-05 | 2017-03-07 | Alc Holdings, Inc. | Generation of layout of videos |
US9767845B2 (en) | 2013-02-05 | 2017-09-19 | Alc Holdings, Inc. | Activating a video based on location in screen |
US9852762B2 (en) | 2013-02-05 | 2017-12-26 | Alc Holdings, Inc. | User interface for video preview creation |
US9881646B2 (en) | 2013-02-05 | 2018-01-30 | Alc Holdings, Inc. | Video preview creation with audio |
US10373646B2 (en) | 2013-02-05 | 2019-08-06 | Alc Holdings, Inc. | Generation of layout of videos |
US10643660B2 (en) | 2013-02-05 | 2020-05-05 | Alc Holdings, Inc. | Video preview creation with audio |
US10528208B2 (en) | 2015-05-31 | 2020-01-07 | International Busines Machines Corporation | Instantaneous preview of a data stream with non-linear time warping |
Also Published As
Publication number | Publication date |
---|---|
US9485423B2 (en) | 2016-11-01 |
US20150124107A1 (en) | 2015-05-07 |
US20130282821A1 (en) | 2013-10-24 |
US8965041B2 (en) | 2015-02-24 |
US20140328542A1 (en) | 2014-11-06 |
US20110113336A1 (en) | 2011-05-12 |
US8438484B2 (en) | 2013-05-07 |
US8818022B2 (en) | 2014-08-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8438484B2 (en) | Video preview module to enhance online video experience | |
US10720188B2 (en) | Systems and methods of thumbnail generation | |
US9478256B1 (en) | Video editing processor for video cloud server | |
US9317188B2 (en) | Devices and methods for providing navigation images associated with adaptive bit rate video content | |
US10826963B2 (en) | Reducing latency for streaming video | |
US20230269441A1 (en) | Intermediate key frame selection and animation | |
CN111343504B (en) | Video processing method, video processing device, computer equipment and storage medium | |
US20170251042A1 (en) | High picture quality video streaming service method and system | |
US9705727B2 (en) | Remote viewing of media content using layered video encoding | |
US10462200B2 (en) | System for cloud streaming service, method for still image-based cloud streaming service and apparatus therefor | |
US20130291031A1 (en) | Systems and Methods for Adaptive Streaming with Augmented Video Stream Transitions Using a Media Server | |
KR102232899B1 (en) | System for cloud streaming service, method of cloud streaming service based on type of image and apparatus for the same | |
US20220329913A1 (en) | Smart player stream switching for live content playback and navigation based on user action | |
US11910038B2 (en) | Crop-based compression of videos | |
US20190200070A1 (en) | Streaming methods and systems using tuner buffers | |
Sung | A rich media framework for communication-broadcasting converged IPTV | |
US10904590B2 (en) | Method and system for real time switching of multimedia content | |
KR20160098927A (en) | Apparatus and method for providing streaming contents | |
CA3050636C (en) | Reducing latency for streaming video | |
EP3732888B1 (en) | Reducing latency for streaming video | |
US20230224521A1 (en) | Electronic device and operation method thereof | |
US20130287092A1 (en) | Systems and Methods for Adaptive Streaming with Augmented Video Stream Transitions | |
Sung et al. | Design and implementation of a ria framework for iptv set-top boxes | |
Percival | HTML5 Media |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION |