CA2411852A1 - Streaming panoramic video - Google Patents
Streaming panoramic video Download PDFInfo
- Publication number
- CA2411852A1 CA2411852A1 CA002411852A CA2411852A CA2411852A1 CA 2411852 A1 CA2411852 A1 CA 2411852A1 CA 002411852 A CA002411852 A CA 002411852A CA 2411852 A CA2411852 A CA 2411852A CA 2411852 A1 CA2411852 A1 CA 2411852A1
- Authority
- CA
- Canada
- Prior art keywords
- server
- client
- view window
- panorama
- slices
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
- H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
- H04N19/23—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44222—Analytics of user selections, e.g. selection of programs or purchase activity
- H04N21/44224—Monitoring of user activity on external systems, e.g. Internet browsing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
- H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1101—Session protocols
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Closed-Circuit Television Systems (AREA)
- Television Signal Processing For Recording (AREA)
Abstract
Streaming panoramic images from a server (100) to a client (150). The system utilizes a special program at the client and a special program in the server. The special program at the client communicates with the special program at the server (100) to direct which portion of the panorama should be streamed to the client. The special program at the client has the ability to accept data that represents a portion of a series of panoramic frames, to decompress the data, to select the data that constitutes an appropriate view window (216) and to render a portion of each frame on a screen or display. The special program at the server selects particular slices that constitute a region of interest in the panorama and these slices are sent to the client. When the location of the view window is changed by more than a threshold amount, the client sends a command back to the web server to adjusts the selection of the slices that are streamed from the server to the client.
Description
1 Streaming Panoramic Video 3 Field of the Invention:
4 The present invention relates to the transmission of panoramic images and more particularly to transferring portions of panoramic images from a server to a client using 6 "video streaming".
8 Background of the Invention:
9 It is well known that special provisions are required when viewing panoramic images on a computer display. If an entire panoramic image is projected on a computer display, the 11 image is necessarily distorted. Panoramic images are generally viewed using a viewer 12 program which renders (i.e. displays) a portion of the panorama on a screen or display.
13 The portion of the panorama that is displayed is generally termed a "view window".
14 Generally viewer programs provide a mechanism (such as a mouse) that can be used to select the desired portion of the panorama frame that constitutes the view window.
17 A panoramic video (or a panoramic movie) is a series of panoramic frames, each of which 18 contains a panoramic image. Co-pending U.S. patent application 09/310,715 filed 19 05/12/99 entitled "Panoramic Movies which Simulate Movement Through Multidimensional Space" describes a system for displaying a panoramic video by 21 displaying a view window that displays in sequence substantially the same view window 22 from a series of panoramic images. The view window only gradually changes location 23 between frames as the viewer chooses to change the direction of view.
Storing a panoramic video requires a great deal of storage, hence, a large amount of 26 bandwidth is required in order to stream panoramic images from a web server to a client.
27 The present invention is directed to reducing the bandwidth required to stream panoramic 28 images from a web server to a client. With the present invention panoramic images can 29 be streamed from a web server to a client over a lower bandwidth connection or with greater image quality, size, and/or frame rate.
32 The MPEG video compression standard provides a "slicing" mechanism. This mechanism 33 is generally used in order to facilitate error correction. The present invention utilizes the __1 __ 1 slicing mechanism in the MPEG video compression standard to reduce the bandwidth 2 required to stream panoramic video from a web server to a client.
4 Summary of the Invention.
The present invention streams panoramic images from a server to a client. The system 6 utilizes a special module at the client and a special module in the server.
The special 7 modules may be plug-ins for commercially available streaming programs. The special 8 module at the client provides the functions that are typically provided by a conventional 9 panorama viewer program and it also communicates with the module at the server to specify which portion of the panorama should be streamed to the client. The special 11 module at the client has the ability to accept data that represents a portion of a series of 12 panoramic frames, to decompress the data, to select the data that constitutes an 13 appropriate view window and to render (i.e. display) a portion of each frame on a screen 14 or display.
16 The server module selects particular slices that constitute a region of interest in the 17 panorama and these slices are sent to the client. At the client, the user may select 18 navigation commands such as pan left, pan right, pan up, pan down, roll left, roll right, 19 zoom in, zoom out or a combination of these or other commands to change the view window. When the location of the view window is changed by more than a threshold 21 amount, the client sends a command back to the web server. In response to the 22 commands from the client, the server module adjusts the selection of the slices that are 23 streamed from the server to the client. There may be many clients receiving information 24 from a particular server and for every client, the module at the server maintains session information and streams appropriate information to that client.
27 Brief Description of the Figures:
28 Figure 1 is a block diagram of first embodiment of the invention.
29 Figures 2A to 2D illustrate the movement of a region of interest and a view window in a panoramic image.
31 Figure 3 is a block diagram of the program in the browser plug in.
32 Figure 4 is a block diagram of the program in the server plug in.
33 Figure 5 illustrates the shape of a view window relative to the slices in a panorama.
34 Figure 6 shows an alternate form of panoramic image.
__2__ 1 Figure 7 shows an alternate embodiment of the invention wherein two different streams 2 are being transmitted from the server to different clients.
3 Figure 8 shows another alternate embodiment of the invention which utilizes a different 4 type of server.
Figure 9 shows an embodiment of the invention where the entire invention is operating on 6 a single computer.
8 Detailed Description:
9 A first preferred embodiment of the invention is shown in Figure 1. In this embodiment panoramic images are streamed from a server 100 to a client 150 over a network 120.
11 The network 120 could for example be the Internet. While only a single client 150 is 12 shown it will be understood by those skilled in the art that a single server 100 could 13 provide data to a large number of clients 150.
The data streamed from server 100 to client 150 could for example be data from a 16 panoramic movie of the type shown in co-pending application 09/310,715 filed 05/12/99 17 entitled "Panoramic Movies which Simulate Movement Through Multidimensional Space", 18 the content of which is herby incorporated by reference. A panoramic movie consists of a 19 series of panoramic images. Such a series of panoramic images could for example be a series of panoramas recorded by a multi lens camera which is moving along a street. A
21 panorama is normally displayed by allowing a user to select a view window (i.e. the 22 direction in which the user is looking). In a panoramic movie, this view window can 23 change direction as a series of frames is projected. That is, with a panoramic movie, the 24 user has the option of selecting the direction of view. The location of the view window in the panorama changes as the user changes direction of view.
27 With the present invention an entire panorama is not streamed from the server 100 to the 28 client 150. Only that portion of the panorama (called a region of interest) that includes the 29 view window and a surrounding region (i.e. a guard band) is streamed from the server 100 to the client 150. That is, the region of interest that is streamed from the server to the 31 client includes the view window and a guard band around the view window.
The user is 32 provided with controls (e.g. a mouse 159) whereby the user can change the location of the 33 view window in the panorama, that is, the user can change the area of the panorama that 34 is being displayed. When the user changes the location of the view window by more than __3__ 1 a threshold amount, the client sends a command to the server to change the location of 2 the region of interest.
4 Data in the entire region of interest is transmitted from the server 100 to the client 150.
The client therefore has the entire region of interest immediately available for display. The 6 guard band surrounding the view window provides data that is immediately available for 7 display at the client when the user moves the view window. Thus, the user can change 8 (to some degree) the location of the view window in the panorama and the data needed to 9 provide the changed display is immediately available without having to wait for the server to send different data.
12 Without the present invention, one could achieve the same result by streaming entire 13 panoramas from the server 100 to the client 150; however, this would require significantly 14 more bandwidth than is required by the present invention. Alternatively, only the data that is in the view window could be streamed from the server to the client;
however, if this were 16 done when the user gives a command to change the viewing direction (i.e.
the location of 17 the view window in the panorama), the command from the user would have to go from the 18 client to the server and the server would have to begin streaming different data to the 19 client 150. This would result in a delay between when the user gives a command and when the view window actually changes. It is noted that this delay is exacerbated by the 21 fact that streaming systems normally buffer data at the server and at the client. Buffering 22 is required for a number of reason including the need for multiple frames in order to 23 perform de-compression.
Figures 2A to 2D illustrate how changes in the location of the view window generates 26 changes in the area of interest that is streamed from the server to the client. Figure 2A
27 illustrates one panorama 214. The panorama is divided into areas 214A, 214B
etc.
28 There is a region of interest 215 and the view window is 216. It is noted that the size of 29 the areas in Figures 2A to 2D is exaggerated for purposes of illustration and they do not constitute actual MPEG slices. The actual sizes are explained later.
32 Figures 2A to 2D illustrate four frames in a panoramic video. It should be noted that the 33 four frames shown in Figures 2A to 2D are not necessarily adjacent sequential frames.
34 That is, out of a series of thirty frames, the frames (i.e. the panoramas) shown may be the __4__ 1 first, tenth, twentieth and thirtieth frames. The changes in the intermediate frames will be 2 a portion of the changes shown in Figures 2A to 2D.
4 For simplicity in illustration and ease of explanation in Figures 2A to 2D, the areas are shown as being square and the size of the view window is shown as coinciding with the 6 size of an area. The actual size of the areas and actual shapes will be explained later.
7 Furthermore, a panorama would normally include an image. For ease of illustration, in 8 Figures 2A to 2D the areas are shown without showing the actual image.
The entire panorama 214 is not transmitted from the web server 100 to the client 150.
11 Only a region of interest 215 from each frame is transmitted from the server to the 12 browser. The region of interest 215 includes the particular view window 216 that is being 13 displayed to the user.
When a user is looking at a particular view window in a panorama, the user might decide 16 to change the location of the view window in the panorama. That is, the user might want 17 to position the view window in a different part of the panorama so that a different part of 18 the panorama will be visible on the display. The term "pan" means that a user changes 19 the location of the view window in one direction or another.
21 Since the region of interest 215 includes a "guard band" surrounding the view window 22 216, and since the entire region of interest 215 is transmitted to the client 150, the data is 23 available at the client 150 to allow the user to change the location of the view window (i.e.
24 to change the portion of the panorama being displayed) without the need for any communication to the server 100.
27 Figure 2B illustrates the view window 216 moving to the right. As the user changes the 28 location of the view window 216, (i.e. as the user changes the portion of the panorama 29 being displayed) the region of interest 215 is changed as shown in Figure 2C. Motion by a user generally continues in the same direction for some time so the user might arrive at 31 the location shown in Figure 2D.
33 Each time the user changes the location of the view window by an amount which exceeds 34 a certain threshold (which can be set depending on factors discussed later), the client 150 __5__ 1 sends a message to the server 100 notifying the server of this change. When the server 2 receives a signal indicating that the location of the view window has changed, the server 3 changes (if appropriate) the particular slices being sent to the browser (i.e. the slices that 4 constitute the region of interest) so that the slices transmitted always include the view window plus a guard band. Thus, the server continues sending a particular region of 6 interest from each frame until notified to change by the client. A user can pan within this 7 region of interest without waiting for the server to change the portion of the panorama that 8 is being streamed from the server to the client.
Frames in a panoramic video are generally sent at a rate of thirty frames per second.
11 Thus, the region of interest from a significant number of frames may be transmitted before 12 the server receives and reacts to a command to change the region of interest. Since the 13 guard band surrounds the view window, the user can change the location of the view 14 window (to some extent) before the server has a chance to react to a command to change the location of the region of interest.
17 The size of the guard band does not have to be of a fixed size, or symmetrical around the 18 region of interest. The guard band may be larger in an expected or usual direction of 19 panning. For example, the guard band may be larger on the left and right sides of the view window, than at the top or bottom. The size of the guard band can be adjusted to an 21 appropriate amount by tracking the history of usage by each particular user and the 22 bandwidth available. Transmitting a larger region of interest requires more bandwidth.
23 Furthermore, the viewer program may limit the rate at which the image is panned. This 24 would be done to attempt to preserve smooth panning in return for a reduced pan rate.
26 The panoramic frames are compressed by the server 100 using standard MPEG
27 compression. The MPEG standard specifies that slices are always 16 pixels high and 28 that the width of a slice is a multiple of 16 pixels, up to the entire width of the 29 frame. With the present invention it has been found that with a frame that is 2K by 1 K, the frame can be divided into 8 slices horizontally each slice being 16 pixels 31 tall, and 256 pixels wide. Thus, there would be 512 slices for each frame.
__g__ 1 It is noted that "Slicing" is a term used in the MPEG 2 standard. In the 2 standard, the slicing mechanism is part of the error correction and concealment 3 section of the standard, and it is known as "inserting resynchronization markers", 4 or "resynchronization mechanism". While the terms used in the two standards differ somewhat the actual implementation is identical, since MPEG 4 carries over 6 all of MPEG 2's implementation. Herein the term "slice" from the MPEG 2 7 standard is used; however, it should be understood that as used herein the term 8 "slice" is intended to refer to "slices" from the MPEG 2 standard and to the 9 equivalent mechanism in other MPEG standards.
11 MPEG compression uses "I" frames (Intra frames), "P" frames, and/or "B"
frames. The I
12 frames contain all of the information needed to reconstruct a single image.
P (Predictive) 13 frames copy the closest matching block of pixels from the preceding I or P
frame, and add 14 a (hopefully small) correction to create blocks. B (Bi-directional) frames are similar to P
frames, but can also copy blocks from the future I or P frame, and/or can average a 16 preceding and future block to create a block in the frame being constructed. I frames are 17 relatively large, P frames are typically smaller, and B frames are usually the smallest. The 18 construction and definition of I frames, B frames, and P frames is set out in the publicly 19 available MPEG standards. The use of either B or P frames is chosen depending upon whether or not reverse motion is desired.
22 The I frames are considerably larger that the B or P frames. Thus, in the first 23 embodiment, only slices from the region of interest in the I frames is transmitted from the 24 client to the server and the entire B or P frames are transmitted.
Alternatively only slices in the region of interest from the B frames could be transmitted. However, it is noted that 26 the number of slices transmitted from the I or P frames may be larger than the number of 27 slices transmitted from the B frames. The reason for this is that only the slices in the B
28 frames that are in the region of interest need be transmitted. With respect to the I and P
29 frames, both the slices in the region of interest and the slices needed by their dependent P and B frames must be transmitted. This imposes a requirement that when encoding P
31 and B frames, blocks of pixels may only be copied from the corresponding slice of the 32 referenced I or P frame, and perhaps the adjacent slice as well.
__7__ 1 When motion is stopped and a user focuses on one frame, the bandwidth can be used to 2 transmit the additional information and to store this additional information in a buffer just in 3 case it is needed. In the situation where a user stops the motion of the video, freezing the 4 view window on a portion of one frame, the system can transmit the entire panorama (or a relatively large portion thereof) from the server to the browser, allowing the user full 6 freedom to pan, tilt, etc., at full speed within the current panorama without need to send 7 commands to the server. If the entire panorama (or a large portion thereof) is stored in a 8 buffer at the client machine, moving the view window can be changed over a larger region 9 more quickly.
11 In the first preferred embodiment of the invention shown in Figure 1, server 100 consists 12 of a convenfiional server platform with the "Microsoft Windows 2000"
operation system 13 101. The system includes the commercially available "Real System Server 8"
program 14 103 which is commercially available from ReaINetworks Inc. The system includes a memory subsystem 102 which stores panoramic videos. The overall streaming operation 16 is handled by the Real System Server 8; however, when the system is asked to stream a 17 panoramic video, the file is passed to plug-in 105. The system shown in figure 1 also 18 includes the Microsoft Internet Information Server 104. The Microsoft Internet Information 19 Server 104 is not used during the streaming operation; however, it may handle a web site that allows a user to request that a particular panoramic movie be streamed.
That is, a 21 web site may list a set of available panoramic movies. When a user clicks on one of the 22 listed movies, the system retrieves that files and begins sending the images to plug in 105 24 Figure 4 is a program block diagram showing the operation performed by plug-in 105.
The frames are stored in compressed format in memory system 102. When the system is 26 asked to stream a panoramic video, the panoramic frames are passed to the plug-in 105 27 from real player 8. The system starts by transmitting a default region of interest from the 28 panoramas with the view window located at a default location. Commands to change the 29 region of interest are received from the client as indicated by block 401.
As indicated by block 404, the slices which form the region of interest 216 are selected. As indicated by 31 block 405, the selected slices are passed to the Real System 8 for transmission to the 32 browser.
__g__ 1 In the embodiment shown in Figure 1, the client 150 consists of a personal computer 151 2 with the Microsoft Windows operating system 152, the Microsoft Internet Explorer Browser 3 153, and the Real Player 8 Plus program which is commercially available for Real 4 Networks Inc. The system includes a user input device 159 such as a mouse.
Finally the client 150 includes a plug in 155 which handles panoramic images.
7 Figure 3 is a block diagram of the program in plug in 155. Plug in 155 receives inputs 8 from the user and from Real Player 8 as indicated by blocks 301 and 302. As indicated by 9 block 303 the slices received from the server 100 are decompressed and stored. As indicated by block 304, the slices which constitute the view window are selected and this 11 image is rendered as indicated by block 305 and sent to the real player 8 port for display 12 as indicated by block 306. The view window from the panorama is rendered in a 13 perspectively correct manner using the transformation known in the prior art for this 14 purpose. Once the view window is determined the selection and rendering of the appropriate data is similar to the operation of many panoramic viewing programs.
17 The "Real System Server 8" and the "Real Player 8", that is units 103 and 154 shown in 18 Figure 1, have what is called a "back channel". The back channel is a communication 19 channel that is separate from the channel used to stream the video frames.
The back channel can accept data from the Real Player and send it to the Real System Server, or it 21 can accept data from the Real System Server and send it to the Real Player.
The back 22 channel is regularly used to send a command such as Stop and Reverse from the player 23 to the server. It is this back channel that is used to send data from client 150 to server 24 100 to instruct the server to change the region of interest. Naturally the plug-ins 105 and 155 includes the other conventional components that are specified by documentation for 26 the plug in specification for the Real Player 8 and the Real System Server 8.
28 It is noted that the size of a view window will typically be on the order of the size of about 29 twenty to eighty MPEG slices. As is know in the art, the actual size depends upon the size of the display and the characteristics of the particular viewer software.
The size of 31 the guard band around the view window will have a size in the range of 10 to 50 MPEG
32 slices. Thus the areas shown in Figures 2A to 2D are the size of about ten to fifty MPEG
33 slices.
__g__ 1 As indicated by block 307, the plug-in determines if different slices are required to 2 constitute the appropriate area of interest 215. This is done according to the following 3 logic where "t" "x", and "n" are variables the value of which is set as discussed below.
4 a) Has the view window changed by more a threshold amount "t" ?
b) If the location of the view window has changed determine direction of 6 movement.
7 c) When view window has moved by the threshold amount, move the region of 8 interest "n" slices in that direction.
9 d) No further movement of the region of interest is necessary until the view window has moved a distance equal to "x" amount.
11 e) When the view window has moved "x" amount, revert to step "a".
12 f) If direction of movement changes, revert to step "b".
13 g) If "action stopped" and user stops on a particular frame, instruct the server to 14 send other slices to in effect enlarge the region of interest available at the client. This data is stored at the client.
17 The variables "t", "x" and "n" can be initially set to default values and changed to suit the 18 actions of a particular user and system. For example, the value of "t", "x"
and "n" can be 19 in the order of the size of 5 to 50 slices. They can be set to one size and maintained at that size throughout a session or they can be changed during a session to make the 21 system react to existing conditions. Initially they may be set to the value which is the size 22 of 20 slices. if, for example, it is found that the system is experiencing a large amount of 23 latency from when a command is send from the client to the server and when the server 24 reacts, the values may be increased.
26 The above calculation takes place for both movement in the x direction and for movement 27 in the "y" direction. As indicated by block 309 the instructions to change the slices that 28 constitute the area of interest 215 are sent from the client 150 to the server 100.
As a specific example of how the system operates, consider a sequence of 500 31 panoramas in a panoramic move. Each panorama is 360 degrees in the horizontal 32 direction and 180 degrees in the vertical direction, represented as an image with 2,048 33 (2K) pixels in the horizontal direction and 1,024 (1 K) pixels in the vertical direction, for a 34 total of 2,097,152 (2M) pixels per panorama.
2 When compressed this movie might consist of one "I" frame followed by nine "B" frames, 3 followed by another "I" frame, nine "B" frames, etc. Each frame would be divided into 4 1024 slices, 16 slices horizontally by 64 slices vertically, each slice having a size of 16 pixels vertically by 128 pixels horizontally.
7 Assume a default view window centered vertically and horizontally within the panorama of 8 approximately 90 degrees horizontally by 45 degrees vertically. Ignoring, for simplicity, 9 the slight panoramic distortion that occurs about the horizon of the stored panoramic image, the view window would be represented by a region of 512 (2048/(360 degrees/90 11 degrees)) pixels horizontally by 256 (1024/(180 degrees/45 degrees)) pixels vertically, or 12 4 slices horizontally by 16 slices vertically. Assuming a guard band of one slice all the 13 way around the view window, the initial region of interest of each frame having a size of 6 14 (4+2) slices by 18 (16+2) slices would be transferred from the server to the client.
16 In a simple example, if the user moved the view window 45 degrees to the right, the client 17 would tell the server to shift the region of interest by two columns of slices to the right. If 18 the user moved the window only 10 degrees to the right, the client would tell the server to 19 add one additional column of slices on the right side of the region of interest, expanding the region of interest in order to preserve a guard band of at least one slice all the way 21 around the view window.
23 The above described embodiment does not take into account the rate at which the user is 24 panning. A more sophisticated embodiment could add additional computational ability to take into account the rate at which the user pans the view window. This added logic could 26 be added at either the server or the client. The following example is based on the logic for 27 rate being at the server. In such a situation the system would operate as follows:
28 Assume that the user starts panning to the right at a rate of 4.5 degrees per frame. The 29 client plug-in would communicate this rate back to the server.
Periodically, the client would also communicate back to the server the actual current position of the view window.
31 The server would use this information to predict the probable range of locations the view 32 window may have by the time each frame is actually displayed, and send the slices which 33 cover this range (plus a suitable guard band). Thus, when sending the first "I" frame, the 34 server would send the slices covering the current region of interest and all of the slices __ 11 __ 1 anticipated up to where the region of interest will probably be located at the time when the 2 next "I" frame is displayed.
4 In the above example, this would add two columns of slices to the right, since by the time the next "I" frame is reached, the panning may have progressed through 45 degrees. The 6 first "B" frame following this "I" frame will need to transmit only the same 6 by 18 slice 7 region as transmitted from the "I" frame, since the anticipated motion would not have 8 moved too far. For the next 4 "B" frames, the slices covering the 7 by 18 slice region 9 (adding an additional column to the right) would be sent, and the final 4 "B" frames would include all slices in the 8 by 18 slice region (adding two additional columns to the right).
11 The next "I" frame would need to include a 10 by 18 slice region, in anticipation that it 12 would need to cover the possible motion of the previous "B" frames as well as the future 13 "B" frames. As the server receives information on the actual position of the view window, 14 it may be able to reduce the number of slices transmitted by adjusting the size of the guard bands to correspond to the most recent actual, vs. predicted, position.
17 Figures 2A through 2D show rectangular view windows and guard bands.
Rectangular 18 shapes are shown to simplify the illustration and explanation. If a panoramic image is, for 19 example, stored in an equirectangular format, the view window and the guard band would typically have the shape shown in Figure 5. A common example of an equirectangular 21 image is that of a rectangular map of the surface of the earth. The trapezoidal-like area 22 shown in Figure 5, when perspectively corrected, will result in a rectangular view window.
23 The technique presented in this document can also be used if the image is stored in cubic 24 projection form such as that shown in Figure 6.
26 The embodiment of the invention described above utilizes I frames and B
frames. The 27 invention could also be applied using I frames and P frames. In another embodiment the 28 invention can be implemented using fractal compression techniques instead of MPEG
29 compression. Other streaming media platforms such as Microsoft's Windows Media or Apple's Quick Time or similar streaming media platforms could be used.
32 Figure 7 illustrates an embodiment of the invention, where the server has two sessions 33 operating and different streams are transmitted to two different client machines. In this 34 embodiment the server 701 has a real Networks server 702 which has two plug-ins 703 1 and 704. Each plug-in 703 and 704 can stream a different series of panoramic images to 2 browsers such as 723 and 724.
4 Another embodiment of the invention is illustrated in Figure 8. In the embodiment illustrated in Figure 8, the server 801 includes a conventional Apache Web server 802. A
6 module 803 termed the Streaming Panoramic Server Module streams slices as previously 7 described to the client 811. The client application in this embodiment is a standalone 8 application 812 that contains the functional capabilities of the client plug-in 155 in the first 9 embodiment.
11 Another embodiment of the invention is shown in Figure 9: In this embodiment a "Stand 12 Alone Panoramic Video Client" 902 is used. In this embodiment, the function of the server 13 module and the client plug-in are co-located on the same computer. The server 14 component 904 called the "Panoramic Media Access Module" retrieves and reads the desired panoramic video from a file system 905 that could be local hard drives, CDs, or a 16 networked file system. This module 904 slices the panoramic video frames in the same 17 way as described in the first embodiment and is functionally equivalent to the module 105 18 in the first embodiment. The "Panoramic Video Renderer" 903 takes the sliced video 19 frames and renderers the image to the screen in the same ways as the plug-in 155 in the first embodiment. The "Sliced Video Stream" is equivalent to that described in the first 21 embodiment. In this case, the stream is passed via an inter-process communications 22 mechanism that could include shared memory, pipes, sockets or an equivalent 23 mechanism instead of being streamed through a public or private network.
The "Session 24 Control Stream" is the same as the other embodiments and consists of instructions on how to slice the Video stream as it is read from the file system 27 While the invention has been shown and described with respect to preferred embodiments 28 thereof, it should be understood that a wide variety of changes may be made without 29 departing from the present invention. The scope of the invention is limited only by the appended claims:
8 Background of the Invention:
9 It is well known that special provisions are required when viewing panoramic images on a computer display. If an entire panoramic image is projected on a computer display, the 11 image is necessarily distorted. Panoramic images are generally viewed using a viewer 12 program which renders (i.e. displays) a portion of the panorama on a screen or display.
13 The portion of the panorama that is displayed is generally termed a "view window".
14 Generally viewer programs provide a mechanism (such as a mouse) that can be used to select the desired portion of the panorama frame that constitutes the view window.
17 A panoramic video (or a panoramic movie) is a series of panoramic frames, each of which 18 contains a panoramic image. Co-pending U.S. patent application 09/310,715 filed 19 05/12/99 entitled "Panoramic Movies which Simulate Movement Through Multidimensional Space" describes a system for displaying a panoramic video by 21 displaying a view window that displays in sequence substantially the same view window 22 from a series of panoramic images. The view window only gradually changes location 23 between frames as the viewer chooses to change the direction of view.
Storing a panoramic video requires a great deal of storage, hence, a large amount of 26 bandwidth is required in order to stream panoramic images from a web server to a client.
27 The present invention is directed to reducing the bandwidth required to stream panoramic 28 images from a web server to a client. With the present invention panoramic images can 29 be streamed from a web server to a client over a lower bandwidth connection or with greater image quality, size, and/or frame rate.
32 The MPEG video compression standard provides a "slicing" mechanism. This mechanism 33 is generally used in order to facilitate error correction. The present invention utilizes the __1 __ 1 slicing mechanism in the MPEG video compression standard to reduce the bandwidth 2 required to stream panoramic video from a web server to a client.
4 Summary of the Invention.
The present invention streams panoramic images from a server to a client. The system 6 utilizes a special module at the client and a special module in the server.
The special 7 modules may be plug-ins for commercially available streaming programs. The special 8 module at the client provides the functions that are typically provided by a conventional 9 panorama viewer program and it also communicates with the module at the server to specify which portion of the panorama should be streamed to the client. The special 11 module at the client has the ability to accept data that represents a portion of a series of 12 panoramic frames, to decompress the data, to select the data that constitutes an 13 appropriate view window and to render (i.e. display) a portion of each frame on a screen 14 or display.
16 The server module selects particular slices that constitute a region of interest in the 17 panorama and these slices are sent to the client. At the client, the user may select 18 navigation commands such as pan left, pan right, pan up, pan down, roll left, roll right, 19 zoom in, zoom out or a combination of these or other commands to change the view window. When the location of the view window is changed by more than a threshold 21 amount, the client sends a command back to the web server. In response to the 22 commands from the client, the server module adjusts the selection of the slices that are 23 streamed from the server to the client. There may be many clients receiving information 24 from a particular server and for every client, the module at the server maintains session information and streams appropriate information to that client.
27 Brief Description of the Figures:
28 Figure 1 is a block diagram of first embodiment of the invention.
29 Figures 2A to 2D illustrate the movement of a region of interest and a view window in a panoramic image.
31 Figure 3 is a block diagram of the program in the browser plug in.
32 Figure 4 is a block diagram of the program in the server plug in.
33 Figure 5 illustrates the shape of a view window relative to the slices in a panorama.
34 Figure 6 shows an alternate form of panoramic image.
__2__ 1 Figure 7 shows an alternate embodiment of the invention wherein two different streams 2 are being transmitted from the server to different clients.
3 Figure 8 shows another alternate embodiment of the invention which utilizes a different 4 type of server.
Figure 9 shows an embodiment of the invention where the entire invention is operating on 6 a single computer.
8 Detailed Description:
9 A first preferred embodiment of the invention is shown in Figure 1. In this embodiment panoramic images are streamed from a server 100 to a client 150 over a network 120.
11 The network 120 could for example be the Internet. While only a single client 150 is 12 shown it will be understood by those skilled in the art that a single server 100 could 13 provide data to a large number of clients 150.
The data streamed from server 100 to client 150 could for example be data from a 16 panoramic movie of the type shown in co-pending application 09/310,715 filed 05/12/99 17 entitled "Panoramic Movies which Simulate Movement Through Multidimensional Space", 18 the content of which is herby incorporated by reference. A panoramic movie consists of a 19 series of panoramic images. Such a series of panoramic images could for example be a series of panoramas recorded by a multi lens camera which is moving along a street. A
21 panorama is normally displayed by allowing a user to select a view window (i.e. the 22 direction in which the user is looking). In a panoramic movie, this view window can 23 change direction as a series of frames is projected. That is, with a panoramic movie, the 24 user has the option of selecting the direction of view. The location of the view window in the panorama changes as the user changes direction of view.
27 With the present invention an entire panorama is not streamed from the server 100 to the 28 client 150. Only that portion of the panorama (called a region of interest) that includes the 29 view window and a surrounding region (i.e. a guard band) is streamed from the server 100 to the client 150. That is, the region of interest that is streamed from the server to the 31 client includes the view window and a guard band around the view window.
The user is 32 provided with controls (e.g. a mouse 159) whereby the user can change the location of the 33 view window in the panorama, that is, the user can change the area of the panorama that 34 is being displayed. When the user changes the location of the view window by more than __3__ 1 a threshold amount, the client sends a command to the server to change the location of 2 the region of interest.
4 Data in the entire region of interest is transmitted from the server 100 to the client 150.
The client therefore has the entire region of interest immediately available for display. The 6 guard band surrounding the view window provides data that is immediately available for 7 display at the client when the user moves the view window. Thus, the user can change 8 (to some degree) the location of the view window in the panorama and the data needed to 9 provide the changed display is immediately available without having to wait for the server to send different data.
12 Without the present invention, one could achieve the same result by streaming entire 13 panoramas from the server 100 to the client 150; however, this would require significantly 14 more bandwidth than is required by the present invention. Alternatively, only the data that is in the view window could be streamed from the server to the client;
however, if this were 16 done when the user gives a command to change the viewing direction (i.e.
the location of 17 the view window in the panorama), the command from the user would have to go from the 18 client to the server and the server would have to begin streaming different data to the 19 client 150. This would result in a delay between when the user gives a command and when the view window actually changes. It is noted that this delay is exacerbated by the 21 fact that streaming systems normally buffer data at the server and at the client. Buffering 22 is required for a number of reason including the need for multiple frames in order to 23 perform de-compression.
Figures 2A to 2D illustrate how changes in the location of the view window generates 26 changes in the area of interest that is streamed from the server to the client. Figure 2A
27 illustrates one panorama 214. The panorama is divided into areas 214A, 214B
etc.
28 There is a region of interest 215 and the view window is 216. It is noted that the size of 29 the areas in Figures 2A to 2D is exaggerated for purposes of illustration and they do not constitute actual MPEG slices. The actual sizes are explained later.
32 Figures 2A to 2D illustrate four frames in a panoramic video. It should be noted that the 33 four frames shown in Figures 2A to 2D are not necessarily adjacent sequential frames.
34 That is, out of a series of thirty frames, the frames (i.e. the panoramas) shown may be the __4__ 1 first, tenth, twentieth and thirtieth frames. The changes in the intermediate frames will be 2 a portion of the changes shown in Figures 2A to 2D.
4 For simplicity in illustration and ease of explanation in Figures 2A to 2D, the areas are shown as being square and the size of the view window is shown as coinciding with the 6 size of an area. The actual size of the areas and actual shapes will be explained later.
7 Furthermore, a panorama would normally include an image. For ease of illustration, in 8 Figures 2A to 2D the areas are shown without showing the actual image.
The entire panorama 214 is not transmitted from the web server 100 to the client 150.
11 Only a region of interest 215 from each frame is transmitted from the server to the 12 browser. The region of interest 215 includes the particular view window 216 that is being 13 displayed to the user.
When a user is looking at a particular view window in a panorama, the user might decide 16 to change the location of the view window in the panorama. That is, the user might want 17 to position the view window in a different part of the panorama so that a different part of 18 the panorama will be visible on the display. The term "pan" means that a user changes 19 the location of the view window in one direction or another.
21 Since the region of interest 215 includes a "guard band" surrounding the view window 22 216, and since the entire region of interest 215 is transmitted to the client 150, the data is 23 available at the client 150 to allow the user to change the location of the view window (i.e.
24 to change the portion of the panorama being displayed) without the need for any communication to the server 100.
27 Figure 2B illustrates the view window 216 moving to the right. As the user changes the 28 location of the view window 216, (i.e. as the user changes the portion of the panorama 29 being displayed) the region of interest 215 is changed as shown in Figure 2C. Motion by a user generally continues in the same direction for some time so the user might arrive at 31 the location shown in Figure 2D.
33 Each time the user changes the location of the view window by an amount which exceeds 34 a certain threshold (which can be set depending on factors discussed later), the client 150 __5__ 1 sends a message to the server 100 notifying the server of this change. When the server 2 receives a signal indicating that the location of the view window has changed, the server 3 changes (if appropriate) the particular slices being sent to the browser (i.e. the slices that 4 constitute the region of interest) so that the slices transmitted always include the view window plus a guard band. Thus, the server continues sending a particular region of 6 interest from each frame until notified to change by the client. A user can pan within this 7 region of interest without waiting for the server to change the portion of the panorama that 8 is being streamed from the server to the client.
Frames in a panoramic video are generally sent at a rate of thirty frames per second.
11 Thus, the region of interest from a significant number of frames may be transmitted before 12 the server receives and reacts to a command to change the region of interest. Since the 13 guard band surrounds the view window, the user can change the location of the view 14 window (to some extent) before the server has a chance to react to a command to change the location of the region of interest.
17 The size of the guard band does not have to be of a fixed size, or symmetrical around the 18 region of interest. The guard band may be larger in an expected or usual direction of 19 panning. For example, the guard band may be larger on the left and right sides of the view window, than at the top or bottom. The size of the guard band can be adjusted to an 21 appropriate amount by tracking the history of usage by each particular user and the 22 bandwidth available. Transmitting a larger region of interest requires more bandwidth.
23 Furthermore, the viewer program may limit the rate at which the image is panned. This 24 would be done to attempt to preserve smooth panning in return for a reduced pan rate.
26 The panoramic frames are compressed by the server 100 using standard MPEG
27 compression. The MPEG standard specifies that slices are always 16 pixels high and 28 that the width of a slice is a multiple of 16 pixels, up to the entire width of the 29 frame. With the present invention it has been found that with a frame that is 2K by 1 K, the frame can be divided into 8 slices horizontally each slice being 16 pixels 31 tall, and 256 pixels wide. Thus, there would be 512 slices for each frame.
__g__ 1 It is noted that "Slicing" is a term used in the MPEG 2 standard. In the 2 standard, the slicing mechanism is part of the error correction and concealment 3 section of the standard, and it is known as "inserting resynchronization markers", 4 or "resynchronization mechanism". While the terms used in the two standards differ somewhat the actual implementation is identical, since MPEG 4 carries over 6 all of MPEG 2's implementation. Herein the term "slice" from the MPEG 2 7 standard is used; however, it should be understood that as used herein the term 8 "slice" is intended to refer to "slices" from the MPEG 2 standard and to the 9 equivalent mechanism in other MPEG standards.
11 MPEG compression uses "I" frames (Intra frames), "P" frames, and/or "B"
frames. The I
12 frames contain all of the information needed to reconstruct a single image.
P (Predictive) 13 frames copy the closest matching block of pixels from the preceding I or P
frame, and add 14 a (hopefully small) correction to create blocks. B (Bi-directional) frames are similar to P
frames, but can also copy blocks from the future I or P frame, and/or can average a 16 preceding and future block to create a block in the frame being constructed. I frames are 17 relatively large, P frames are typically smaller, and B frames are usually the smallest. The 18 construction and definition of I frames, B frames, and P frames is set out in the publicly 19 available MPEG standards. The use of either B or P frames is chosen depending upon whether or not reverse motion is desired.
22 The I frames are considerably larger that the B or P frames. Thus, in the first 23 embodiment, only slices from the region of interest in the I frames is transmitted from the 24 client to the server and the entire B or P frames are transmitted.
Alternatively only slices in the region of interest from the B frames could be transmitted. However, it is noted that 26 the number of slices transmitted from the I or P frames may be larger than the number of 27 slices transmitted from the B frames. The reason for this is that only the slices in the B
28 frames that are in the region of interest need be transmitted. With respect to the I and P
29 frames, both the slices in the region of interest and the slices needed by their dependent P and B frames must be transmitted. This imposes a requirement that when encoding P
31 and B frames, blocks of pixels may only be copied from the corresponding slice of the 32 referenced I or P frame, and perhaps the adjacent slice as well.
__7__ 1 When motion is stopped and a user focuses on one frame, the bandwidth can be used to 2 transmit the additional information and to store this additional information in a buffer just in 3 case it is needed. In the situation where a user stops the motion of the video, freezing the 4 view window on a portion of one frame, the system can transmit the entire panorama (or a relatively large portion thereof) from the server to the browser, allowing the user full 6 freedom to pan, tilt, etc., at full speed within the current panorama without need to send 7 commands to the server. If the entire panorama (or a large portion thereof) is stored in a 8 buffer at the client machine, moving the view window can be changed over a larger region 9 more quickly.
11 In the first preferred embodiment of the invention shown in Figure 1, server 100 consists 12 of a convenfiional server platform with the "Microsoft Windows 2000"
operation system 13 101. The system includes the commercially available "Real System Server 8"
program 14 103 which is commercially available from ReaINetworks Inc. The system includes a memory subsystem 102 which stores panoramic videos. The overall streaming operation 16 is handled by the Real System Server 8; however, when the system is asked to stream a 17 panoramic video, the file is passed to plug-in 105. The system shown in figure 1 also 18 includes the Microsoft Internet Information Server 104. The Microsoft Internet Information 19 Server 104 is not used during the streaming operation; however, it may handle a web site that allows a user to request that a particular panoramic movie be streamed.
That is, a 21 web site may list a set of available panoramic movies. When a user clicks on one of the 22 listed movies, the system retrieves that files and begins sending the images to plug in 105 24 Figure 4 is a program block diagram showing the operation performed by plug-in 105.
The frames are stored in compressed format in memory system 102. When the system is 26 asked to stream a panoramic video, the panoramic frames are passed to the plug-in 105 27 from real player 8. The system starts by transmitting a default region of interest from the 28 panoramas with the view window located at a default location. Commands to change the 29 region of interest are received from the client as indicated by block 401.
As indicated by block 404, the slices which form the region of interest 216 are selected. As indicated by 31 block 405, the selected slices are passed to the Real System 8 for transmission to the 32 browser.
__g__ 1 In the embodiment shown in Figure 1, the client 150 consists of a personal computer 151 2 with the Microsoft Windows operating system 152, the Microsoft Internet Explorer Browser 3 153, and the Real Player 8 Plus program which is commercially available for Real 4 Networks Inc. The system includes a user input device 159 such as a mouse.
Finally the client 150 includes a plug in 155 which handles panoramic images.
7 Figure 3 is a block diagram of the program in plug in 155. Plug in 155 receives inputs 8 from the user and from Real Player 8 as indicated by blocks 301 and 302. As indicated by 9 block 303 the slices received from the server 100 are decompressed and stored. As indicated by block 304, the slices which constitute the view window are selected and this 11 image is rendered as indicated by block 305 and sent to the real player 8 port for display 12 as indicated by block 306. The view window from the panorama is rendered in a 13 perspectively correct manner using the transformation known in the prior art for this 14 purpose. Once the view window is determined the selection and rendering of the appropriate data is similar to the operation of many panoramic viewing programs.
17 The "Real System Server 8" and the "Real Player 8", that is units 103 and 154 shown in 18 Figure 1, have what is called a "back channel". The back channel is a communication 19 channel that is separate from the channel used to stream the video frames.
The back channel can accept data from the Real Player and send it to the Real System Server, or it 21 can accept data from the Real System Server and send it to the Real Player.
The back 22 channel is regularly used to send a command such as Stop and Reverse from the player 23 to the server. It is this back channel that is used to send data from client 150 to server 24 100 to instruct the server to change the region of interest. Naturally the plug-ins 105 and 155 includes the other conventional components that are specified by documentation for 26 the plug in specification for the Real Player 8 and the Real System Server 8.
28 It is noted that the size of a view window will typically be on the order of the size of about 29 twenty to eighty MPEG slices. As is know in the art, the actual size depends upon the size of the display and the characteristics of the particular viewer software.
The size of 31 the guard band around the view window will have a size in the range of 10 to 50 MPEG
32 slices. Thus the areas shown in Figures 2A to 2D are the size of about ten to fifty MPEG
33 slices.
__g__ 1 As indicated by block 307, the plug-in determines if different slices are required to 2 constitute the appropriate area of interest 215. This is done according to the following 3 logic where "t" "x", and "n" are variables the value of which is set as discussed below.
4 a) Has the view window changed by more a threshold amount "t" ?
b) If the location of the view window has changed determine direction of 6 movement.
7 c) When view window has moved by the threshold amount, move the region of 8 interest "n" slices in that direction.
9 d) No further movement of the region of interest is necessary until the view window has moved a distance equal to "x" amount.
11 e) When the view window has moved "x" amount, revert to step "a".
12 f) If direction of movement changes, revert to step "b".
13 g) If "action stopped" and user stops on a particular frame, instruct the server to 14 send other slices to in effect enlarge the region of interest available at the client. This data is stored at the client.
17 The variables "t", "x" and "n" can be initially set to default values and changed to suit the 18 actions of a particular user and system. For example, the value of "t", "x"
and "n" can be 19 in the order of the size of 5 to 50 slices. They can be set to one size and maintained at that size throughout a session or they can be changed during a session to make the 21 system react to existing conditions. Initially they may be set to the value which is the size 22 of 20 slices. if, for example, it is found that the system is experiencing a large amount of 23 latency from when a command is send from the client to the server and when the server 24 reacts, the values may be increased.
26 The above calculation takes place for both movement in the x direction and for movement 27 in the "y" direction. As indicated by block 309 the instructions to change the slices that 28 constitute the area of interest 215 are sent from the client 150 to the server 100.
As a specific example of how the system operates, consider a sequence of 500 31 panoramas in a panoramic move. Each panorama is 360 degrees in the horizontal 32 direction and 180 degrees in the vertical direction, represented as an image with 2,048 33 (2K) pixels in the horizontal direction and 1,024 (1 K) pixels in the vertical direction, for a 34 total of 2,097,152 (2M) pixels per panorama.
2 When compressed this movie might consist of one "I" frame followed by nine "B" frames, 3 followed by another "I" frame, nine "B" frames, etc. Each frame would be divided into 4 1024 slices, 16 slices horizontally by 64 slices vertically, each slice having a size of 16 pixels vertically by 128 pixels horizontally.
7 Assume a default view window centered vertically and horizontally within the panorama of 8 approximately 90 degrees horizontally by 45 degrees vertically. Ignoring, for simplicity, 9 the slight panoramic distortion that occurs about the horizon of the stored panoramic image, the view window would be represented by a region of 512 (2048/(360 degrees/90 11 degrees)) pixels horizontally by 256 (1024/(180 degrees/45 degrees)) pixels vertically, or 12 4 slices horizontally by 16 slices vertically. Assuming a guard band of one slice all the 13 way around the view window, the initial region of interest of each frame having a size of 6 14 (4+2) slices by 18 (16+2) slices would be transferred from the server to the client.
16 In a simple example, if the user moved the view window 45 degrees to the right, the client 17 would tell the server to shift the region of interest by two columns of slices to the right. If 18 the user moved the window only 10 degrees to the right, the client would tell the server to 19 add one additional column of slices on the right side of the region of interest, expanding the region of interest in order to preserve a guard band of at least one slice all the way 21 around the view window.
23 The above described embodiment does not take into account the rate at which the user is 24 panning. A more sophisticated embodiment could add additional computational ability to take into account the rate at which the user pans the view window. This added logic could 26 be added at either the server or the client. The following example is based on the logic for 27 rate being at the server. In such a situation the system would operate as follows:
28 Assume that the user starts panning to the right at a rate of 4.5 degrees per frame. The 29 client plug-in would communicate this rate back to the server.
Periodically, the client would also communicate back to the server the actual current position of the view window.
31 The server would use this information to predict the probable range of locations the view 32 window may have by the time each frame is actually displayed, and send the slices which 33 cover this range (plus a suitable guard band). Thus, when sending the first "I" frame, the 34 server would send the slices covering the current region of interest and all of the slices __ 11 __ 1 anticipated up to where the region of interest will probably be located at the time when the 2 next "I" frame is displayed.
4 In the above example, this would add two columns of slices to the right, since by the time the next "I" frame is reached, the panning may have progressed through 45 degrees. The 6 first "B" frame following this "I" frame will need to transmit only the same 6 by 18 slice 7 region as transmitted from the "I" frame, since the anticipated motion would not have 8 moved too far. For the next 4 "B" frames, the slices covering the 7 by 18 slice region 9 (adding an additional column to the right) would be sent, and the final 4 "B" frames would include all slices in the 8 by 18 slice region (adding two additional columns to the right).
11 The next "I" frame would need to include a 10 by 18 slice region, in anticipation that it 12 would need to cover the possible motion of the previous "B" frames as well as the future 13 "B" frames. As the server receives information on the actual position of the view window, 14 it may be able to reduce the number of slices transmitted by adjusting the size of the guard bands to correspond to the most recent actual, vs. predicted, position.
17 Figures 2A through 2D show rectangular view windows and guard bands.
Rectangular 18 shapes are shown to simplify the illustration and explanation. If a panoramic image is, for 19 example, stored in an equirectangular format, the view window and the guard band would typically have the shape shown in Figure 5. A common example of an equirectangular 21 image is that of a rectangular map of the surface of the earth. The trapezoidal-like area 22 shown in Figure 5, when perspectively corrected, will result in a rectangular view window.
23 The technique presented in this document can also be used if the image is stored in cubic 24 projection form such as that shown in Figure 6.
26 The embodiment of the invention described above utilizes I frames and B
frames. The 27 invention could also be applied using I frames and P frames. In another embodiment the 28 invention can be implemented using fractal compression techniques instead of MPEG
29 compression. Other streaming media platforms such as Microsoft's Windows Media or Apple's Quick Time or similar streaming media platforms could be used.
32 Figure 7 illustrates an embodiment of the invention, where the server has two sessions 33 operating and different streams are transmitted to two different client machines. In this 34 embodiment the server 701 has a real Networks server 702 which has two plug-ins 703 1 and 704. Each plug-in 703 and 704 can stream a different series of panoramic images to 2 browsers such as 723 and 724.
4 Another embodiment of the invention is illustrated in Figure 8. In the embodiment illustrated in Figure 8, the server 801 includes a conventional Apache Web server 802. A
6 module 803 termed the Streaming Panoramic Server Module streams slices as previously 7 described to the client 811. The client application in this embodiment is a standalone 8 application 812 that contains the functional capabilities of the client plug-in 155 in the first 9 embodiment.
11 Another embodiment of the invention is shown in Figure 9: In this embodiment a "Stand 12 Alone Panoramic Video Client" 902 is used. In this embodiment, the function of the server 13 module and the client plug-in are co-located on the same computer. The server 14 component 904 called the "Panoramic Media Access Module" retrieves and reads the desired panoramic video from a file system 905 that could be local hard drives, CDs, or a 16 networked file system. This module 904 slices the panoramic video frames in the same 17 way as described in the first embodiment and is functionally equivalent to the module 105 18 in the first embodiment. The "Panoramic Video Renderer" 903 takes the sliced video 19 frames and renderers the image to the screen in the same ways as the plug-in 155 in the first embodiment. The "Sliced Video Stream" is equivalent to that described in the first 21 embodiment. In this case, the stream is passed via an inter-process communications 22 mechanism that could include shared memory, pipes, sockets or an equivalent 23 mechanism instead of being streamed through a public or private network.
The "Session 24 Control Stream" is the same as the other embodiments and consists of instructions on how to slice the Video stream as it is read from the file system 27 While the invention has been shown and described with respect to preferred embodiments 28 thereof, it should be understood that a wide variety of changes may be made without 29 departing from the present invention. The scope of the invention is limited only by the appended claims:
Claims (20)
1. A method of streaming a panorama from a server to a client, wherein a user can only see the portion of the panorama in a view window and the user can move the location of the view window in the panorama, said method comprising the steps of dividing the panorama into slices, transmitting from the server to the client slices of said panorama that contain the view window plus a guard band surrounding the view window, transmitting from the client to the server instructions to change the location of said guard band as said user moves said view window.
2. The method recited in claim 1 wherein said slices are the slices defined in the MPEG
standard.
standard.
3. The method recited in claim 1 where the streaming from the server to the client is handled by a streaming server and a plug-in to said server provides the slices in said guard band.
4. The method recited in claim 1 where the client and the server are located on the same physical machine.
5. The method of streaming data relative to a series of panoramic images from a server to a client, whereby a view window of said client can be displayed to a user, said method comprising the steps of:
dividing each of said panoramic images info areas, streaming a plurality of said areas from each area from said server to said client, said plurality of areas including said view window and a guard band around said view window, displaying said view window portion of said panorama at said client, accepting user directions to change the location of said view window, sending commands to said server to change said plurality of areas being streamed to said server when said view window is changed more than a threshold amount, and changing the areas streamed from said server to said client in response to said commands.
dividing each of said panoramic images info areas, streaming a plurality of said areas from each area from said server to said client, said plurality of areas including said view window and a guard band around said view window, displaying said view window portion of said panorama at said client, accepting user directions to change the location of said view window, sending commands to said server to change said plurality of areas being streamed to said server when said view window is changed more than a threshold amount, and changing the areas streamed from said server to said client in response to said commands.
6. The method recited in claim 5 wherein said areas are MPEG slices.
7. The method recited in claim 5 wherein said server is a Real Networks server.
8. The method recited in claim 5 wherein said panorama is displayed to said user in a perspectively correct manner.
9. The method recited in claim 5 wherein said server simultaneously streams portions of different panoramas to different clients.
10. The method recited in claim 5 wherein said server and said client are on the same physical machine.
11. A system for transmitting panoramic images from a server to a client, means at said server for dividing each panorama into areas, a plurality of said areas forming a region of interest of said panorama, said region of interest including a view window and a guard band around said view window, means for transmitting a region of interest from each panorama in a series of panoramas from said server to said client, means at said client for moving the location of said view window in said panorama, means for transmitting from said client to said server commands to change the location of said region of interest, and means at said server for changing the location of said region of interest which is streamed to said client.
12. The system recited in claim 11 where each of said areas comprise a plurality of MPEG
slices.
slices.
13. The system recited in claim 12 wherein said server simultaneously streams portions of different panoramas to different clients.
14. The system recited in claim 12 wherein said server and said client are on the same physical machine.
15. The system recited in claim 12 wherein all said means are physically located on one physical system.
16. A system for allowing a series of panoramic images stored at a server to be viewed by a user at a client, said system including, a streaming server at said server for streaming data to said client, a program at said server for providing said streaming server with an area of interest from each panorama to be streamed to said client, said area of interest including a view window and a guard band around said view window, and a program at said client for receiving said data and for selecting the data representing said view window and for displaying said view window to said user.
17. The system recited in claim 16 including a user input device whereby said user can move said view window in said area of interest, and a communication path from said client to said server whereby said client can instruct said server to change the location of said view window.
18. The system recited in claim 17 wherein said user input device is a computer mouse.
19. The system recited in claim 15 wherein said panoramic images are stored at said server using MPEG compression forming "I" and "B" or "P" frames.
20. The system recited in claim 19 wherein the region of interest from "I"
frames and entire "B" or "P" frames are transmitted from said server to said client.
frames and entire "B" or "P" frames are transmitted from said server to said client.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US21037400P | 2000-06-09 | 2000-06-09 | |
| US60/210,374 | 2000-06-09 | ||
| PCT/US2001/018731 WO2001095513A1 (en) | 2000-06-09 | 2001-06-08 | Streaming panoramic video |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2411852A1 true CA2411852A1 (en) | 2001-12-13 |
Family
ID=22782655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002411852A Abandoned CA2411852A1 (en) | 2000-06-09 | 2001-06-08 | Streaming panoramic video |
Country Status (7)
| Country | Link |
|---|---|
| US (1) | US20020021353A1 (en) |
| EP (1) | EP1297634A1 (en) |
| JP (1) | JP2003536319A (en) |
| AU (1) | AU2001275453A1 (en) |
| CA (1) | CA2411852A1 (en) |
| IL (1) | IL153164A0 (en) |
| WO (1) | WO2001095513A1 (en) |
Families Citing this family (78)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3472273B2 (en) * | 2001-03-07 | 2003-12-02 | キヤノン株式会社 | Image reproducing apparatus, image processing apparatus and method |
| US6747647B2 (en) | 2001-05-02 | 2004-06-08 | Enroute, Inc. | System and method for displaying immersive video |
| US20030016228A1 (en) * | 2001-05-02 | 2003-01-23 | Youngblood Paul A. | System and method for displaying seamless immersive video |
| FR2828754A1 (en) * | 2001-08-14 | 2003-02-21 | Koninkl Philips Electronics Nv | VISUALIZATION OF A PANORAMIC VIDEO EDITION BY APPLYING NAVIGATION COMMANDS TO THE SAME |
| WO2004004363A1 (en) * | 2002-06-28 | 2004-01-08 | Sharp Kabushiki Kaisha | Image encoding device, image transmission device, and image pickup device |
| GB0230328D0 (en) * | 2002-12-31 | 2003-02-05 | British Telecomm | Video streaming |
| FR2853797A1 (en) | 2003-04-09 | 2004-10-15 | Canon Kk | METHOD AND DEVICE FOR PRE-PROCESSING REQUESTS LINKED TO A DIGITAL SIGNAL IN A CUSTOMER-SERVER ARCHITECTURE |
| US7746373B2 (en) * | 2003-12-23 | 2010-06-29 | Telecom Italia S.P.A. | Device for viewing images, such as for videoconference facilities, related system, network and method of use |
| WO2006060846A1 (en) * | 2004-12-09 | 2006-06-15 | Real Estate Media Pty Ltd | Method and system for producing a moving picture which pans across an article |
| FR2884027B1 (en) * | 2005-04-04 | 2007-06-01 | Canon Kk | METHOD AND DEVICE FOR TRANSMITTING AND RECEIVING IMAGE SEQUENCES BETWEEN A SERVER AND A CUSTOMER |
| US9060101B2 (en) * | 2005-07-08 | 2015-06-16 | Activevideo Networks, Inc. | Video game system having an infinite playing field |
| US9061206B2 (en) * | 2005-07-08 | 2015-06-23 | Activevideo Networks, Inc. | Video game system using pre-generated motion vectors |
| US8118676B2 (en) * | 2005-07-08 | 2012-02-21 | Activevideo Networks, Inc. | Video game system using pre-encoded macro-blocks |
| US8284842B2 (en) | 2005-07-08 | 2012-10-09 | Activevideo Networks, Inc. | Video game system using pre-encoded macro-blocks and a reference grid |
| US8074248B2 (en) | 2005-07-26 | 2011-12-06 | Activevideo Networks, Inc. | System and method for providing video content associated with a source image to a television in a communication network |
| EP2477414A3 (en) * | 2006-09-29 | 2014-03-05 | Avinity Systems B.V. | Method for assembling a video stream, system and computer software |
| WO2008088741A2 (en) | 2007-01-12 | 2008-07-24 | Ictv, Inc. | Interactive encoded content system including object models for viewing on a remote device |
| US9826197B2 (en) | 2007-01-12 | 2017-11-21 | Activevideo Networks, Inc. | Providing television broadcasts over a managed network and interactive content over an unmanaged network to a client device |
| US8074241B2 (en) * | 2007-03-30 | 2011-12-06 | The Board Of Trustees Of The Leland Stanford Jr. University | Process for displaying and navigating panoramic video, and method and user interface for streaming panoramic video and images between a server and browser-based client application |
| US20090113505A1 (en) * | 2007-10-26 | 2009-04-30 | At&T Bls Intellectual Property, Inc. | Systems, methods and computer products for multi-user access for integrated video |
| JP4935632B2 (en) * | 2007-11-07 | 2012-05-23 | ソニー株式会社 | Image processing apparatus, image processing method, and image processing program |
| KR100891263B1 (en) * | 2007-11-15 | 2009-03-30 | 에스케이 텔레콤주식회사 | Method, system and server playing media using user equipment with motion sensor |
| US20100050221A1 (en) * | 2008-06-20 | 2010-02-25 | Mccutchen David J | Image Delivery System with Image Quality Varying with Frame Rate |
| JP5089658B2 (en) * | 2009-07-16 | 2012-12-05 | 株式会社Ｇｎｚｏ | Transmitting apparatus and transmitting method |
| US9258530B2 (en) * | 2010-04-09 | 2016-02-09 | Canon Kabushiki Kaisha | Method for accessing a spatio-temporal part of a compressed video sequence using decomposed access request |
| FR2959636B1 (en) * | 2010-04-28 | 2012-07-13 | Canon Kk | METHOD FOR ACCESSING A SPATIO-TEMPORAL PART OF A VIDEO SEQUENCE OF IMAGES |
| US8918737B2 (en) * | 2010-04-29 | 2014-12-23 | Microsoft Corporation | Zoom display navigation |
| US9232257B2 (en) | 2010-09-22 | 2016-01-05 | Thomson Licensing | Method for navigation in a panoramic scene |
| CA2814070A1 (en) | 2010-10-14 | 2012-04-19 | Activevideo Networks, Inc. | Streaming digital video between video devices using a cable television system |
| EP2695388B1 (en) | 2011-04-07 | 2017-06-07 | ActiveVideo Networks, Inc. | Reduction of latency in video distribution networks using adaptive bit rates |
| JP5918618B2 (en) * | 2011-06-03 | 2016-05-18 | 任天堂株式会社 | Information processing program, information processing apparatus, information processing system, and information processing method |
| US8850075B2 (en) * | 2011-07-06 | 2014-09-30 | Microsoft Corporation | Predictive, multi-layer caching architectures |
| WO2013014880A1 (en) | 2011-07-22 | 2013-01-31 | パナソニック株式会社 | Content regeneration device, content regeneration method, content regeneration program and content providing program |
| KR101282955B1 (en) * | 2011-08-31 | 2013-07-17 | 한국과학기술연구원 | Real-time Panorama Streaming System for High Resolution Panorama Videos and/or Images |
| US9723223B1 (en) | 2011-12-02 | 2017-08-01 | Amazon Technologies, Inc. | Apparatus and method for panoramic video hosting with directional audio |
| US9838687B1 (en) | 2011-12-02 | 2017-12-05 | Amazon Technologies, Inc. | Apparatus and method for panoramic video hosting with reduced bandwidth streaming |
| US20130141526A1 (en) | 2011-12-02 | 2013-06-06 | Stealth HD Corp. | Apparatus and Method for Video Image Stitching |
| WO2013106390A1 (en) | 2012-01-09 | 2013-07-18 | Activevideo Networks, Inc. | Rendering of an interactive lean-backward user interface on a television |
| US9800945B2 (en) | 2012-04-03 | 2017-10-24 | Activevideo Networks, Inc. | Class-based intelligent multiplexing over unmanaged networks |
| US9123084B2 (en) | 2012-04-12 | 2015-09-01 | Activevideo Networks, Inc. | Graphical application integration with MPEG objects |
| US9027052B2 (en) * | 2012-05-04 | 2015-05-05 | Adsparx USA Inc | System and method for in-stream advertising on an internet connected device |
| US20140188612A1 (en) * | 2012-12-31 | 2014-07-03 | Google Inc. | Directed content presentation |
| WO2014145921A1 (en) | 2013-03-15 | 2014-09-18 | Activevideo Networks, Inc. | A multiple-mode system and method for providing user selectable video content |
| US9219922B2 (en) | 2013-06-06 | 2015-12-22 | Activevideo Networks, Inc. | System and method for exploiting scene graph information in construction of an encoded video sequence |
| US9326047B2 (en) | 2013-06-06 | 2016-04-26 | Activevideo Networks, Inc. | Overlay rendering of user interface onto source video |
| US9294785B2 (en) | 2013-06-06 | 2016-03-22 | Activevideo Networks, Inc. | System and method for exploiting scene graph information in construction of an encoded video sequence |
| US10721530B2 (en) | 2013-07-29 | 2020-07-21 | Koninklijke Kpn N.V. | Providing tile video streams to a client |
| GB2520310A (en) * | 2013-11-15 | 2015-05-20 | Sony Corp | A method, server, client and software |
| US10015527B1 (en) | 2013-12-16 | 2018-07-03 | Amazon Technologies, Inc. | Panoramic video distribution and viewing |
| US9788029B2 (en) | 2014-04-25 | 2017-10-10 | Activevideo Networks, Inc. | Intelligent multiplexing using class-based, multi-dimensioned decision logic for managed networks |
| JP6440747B2 (en) | 2014-06-27 | 2018-12-19 | コニンクリーケ・ケイピーエヌ・ナムローゼ・フェンノートシャップ | Region of interest determination based on HEVC tiled video stream |
| US10694192B2 (en) | 2014-06-27 | 2020-06-23 | Koninklijke Kpn N.V. | HEVC-tiled video streaming |
| US10204658B2 (en) | 2014-07-14 | 2019-02-12 | Sony Interactive Entertainment Inc. | System and method for use in playing back panorama video content |
| EP3219100A1 (en) * | 2014-09-15 | 2017-09-20 | Dmitry Gorilovsky | A system comprising multiple digital cameras viewing a large scene |
| KR102313485B1 (en) * | 2015-04-22 | 2021-10-15 | 삼성전자주식회사 | Method and apparatus for transmitting and receiving image data for virtual reality streaming service |
| US9996895B2 (en) | 2015-05-01 | 2018-06-12 | Ricoh Company, Ltd. | Image display system, information processing apparatus, and image display method |
| GB2539027B (en) | 2015-06-04 | 2019-04-17 | Thales Holdings Uk Plc | Video compression with increased fidelity near horizon |
| JP2017022529A (en) * | 2015-07-09 | 2017-01-26 | キヤノン株式会社 | Communication system, communication device, communication method, and program |
| WO2017029400A1 (en) | 2015-08-20 | 2017-02-23 | Koninklijke Kpn N.V. | Forming one or more tile streams on the basis of one or more video streams |
| US10104286B1 (en) | 2015-08-27 | 2018-10-16 | Amazon Technologies, Inc. | Motion de-blurring for panoramic frames |
| US10609379B1 (en) | 2015-09-01 | 2020-03-31 | Amazon Technologies, Inc. | Video compression across continuous frame edges |
| US9843724B1 (en) | 2015-09-21 | 2017-12-12 | Amazon Technologies, Inc. | Stabilization of panoramic video |
| US10681335B2 (en) | 2015-09-23 | 2020-06-09 | Nokia Technologies Oy | Video recording method and apparatus |
| US10468066B2 (en) | 2015-09-23 | 2019-11-05 | Nokia Technologies Oy | Video content selection |
| US10674185B2 (en) | 2015-10-08 | 2020-06-02 | Koninklijke Kpn N.V. | Enhancing a region of interest in video frames of a video stream |
| US11089280B2 (en) | 2016-06-30 | 2021-08-10 | Sony Interactive Entertainment Inc. | Apparatus and method for capturing and displaying segmented content |
| CN106101847A (en) * | 2016-07-12 | 2016-11-09 | 三星电子（中国）研发中心 | The method and system of panoramic video alternating transmission |
| WO2018015806A1 (en) * | 2016-07-18 | 2018-01-25 | Glide Talk, Ltd. | System and method providing object-oriented zoom in multimedia messaging |
| KR102542515B1 (en) * | 2016-07-28 | 2023-06-12 | 삼성전자주식회사 | Image display apparatus and method for displaying image |
| WO2018057472A1 (en) | 2016-09-26 | 2018-03-29 | Dolby Laboratories Licensing Corporation | Content based stream splitting of video data |
| WO2018068213A1 (en) | 2016-10-10 | 2018-04-19 | 华为技术有限公司 | Video data processing method, and apparatus |
| US10887600B2 (en) * | 2017-03-17 | 2021-01-05 | Samsung Electronics Co., Ltd. | Method and apparatus for packaging and streaming of virtual reality (VR) media content |
| CN108476305A (en) * | 2017-03-21 | 2018-08-31 | 深圳市大疆创新科技有限公司 | A kind of image transfer method, device and equipment |
| WO2018212009A1 (en) * | 2017-05-15 | 2018-11-22 | Sharp Kabushiki Kaisha | Systems and methods for mapping sample locations to angular coordinates in virtual reality applications |
| WO2019120575A1 (en) * | 2017-12-22 | 2019-06-27 | Huawei Technologies Co., Ltd. | Vr 360 video for remote end users |
| US11323754B2 (en) | 2018-11-20 | 2022-05-03 | At&T Intellectual Property I, L.P. | Methods, devices, and systems for updating streaming panoramic video content due to a change in user viewpoint |
| CN111225293B (en) * | 2018-11-23 | 2023-10-03 | 深圳市中兴微电子技术有限公司 | Video data processing method and device and computer storage medium |
| US11523185B2 (en) | 2019-06-19 | 2022-12-06 | Koninklijke Kpn N.V. | Rendering video stream in sub-area of visible display area |
Family Cites Families (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4602279A (en) * | 1984-03-21 | 1986-07-22 | Actv, Inc. | Method for providing targeted profile interactive CATV displays |
| US4573072A (en) * | 1984-03-21 | 1986-02-25 | Actv Inc. | Method for expanding interactive CATV displayable choices for a given channel capacity |
| US4847699A (en) * | 1987-07-16 | 1989-07-11 | Actv, Inc. | Method for providing an interactive full motion synched compatible audio/visual television display |
| US4847700A (en) * | 1987-07-16 | 1989-07-11 | Actv, Inc. | Interactive television system for providing full motion synched compatible audio/visual displays from transmitted television signals |
| US4847698A (en) * | 1987-07-16 | 1989-07-11 | Actv, Inc. | Interactive television system for providing full motion synched compatible audio/visual displays |
| US4918516A (en) * | 1987-10-26 | 1990-04-17 | 501 Actv, Inc. | Closed circuit television system having seamless interactive television programming and expandable user participation |
| US7448063B2 (en) * | 1991-11-25 | 2008-11-04 | Actv, Inc. | Digital interactive system for providing full interactivity with live programming events |
| US5724091A (en) * | 1991-11-25 | 1998-03-03 | Actv, Inc. | Compressed digital data interactive program system |
| US5861881A (en) * | 1991-11-25 | 1999-01-19 | Actv, Inc. | Interactive computer system for providing an interactive presentation with personalized video, audio and graphics responses for multiple viewers |
| US20010013123A1 (en) * | 1991-11-25 | 2001-08-09 | Freeman Michael J. | Customized program creation by splicing server based video, audio, or graphical segments |
| US5648813A (en) * | 1993-10-20 | 1997-07-15 | Matsushita Electric Industrial Co. Ltd. | Graphical-interactive-screen display apparatus and peripheral units |
| US5537141A (en) * | 1994-04-15 | 1996-07-16 | Actv, Inc. | Distance learning system providing individual television participation, audio responses and memory for every student |
| US5632007A (en) * | 1994-09-23 | 1997-05-20 | Actv, Inc. | Interactive system and method for offering expert based interactive programs |
| US5682196A (en) * | 1995-06-22 | 1997-10-28 | Actv, Inc. | Three-dimensional (3D) video presentation system providing interactive 3D presentation with personalized audio responses for multiple viewers |
| US5774664A (en) * | 1996-03-08 | 1998-06-30 | Actv, Inc. | Enhanced video programming system and method for incorporating and displaying retrieved integrated internet information segments |
| US5778181A (en) * | 1996-03-08 | 1998-07-07 | Actv, Inc. | Enhanced video programming system and method for incorporating and displaying retrieved integrated internet information segments |
| US6459451B2 (en) * | 1996-06-24 | 2002-10-01 | Be Here Corporation | Method and apparatus for a panoramic camera to capture a 360 degree image |
| KR100238098B1 (en) * | 1996-09-16 | 2000-01-15 | 윤종용 | Apparatus for synchronously reproducing multi-angle data |
| US6043837A (en) * | 1997-05-08 | 2000-03-28 | Be Here Corporation | Method and apparatus for electronically distributing images from a panoptic camera system |
| US6064399A (en) * | 1998-04-03 | 2000-05-16 | Mgi Software Corporation | Method and system for panel alignment in panoramas |
| CA2371349A1 (en) * | 1998-05-13 | 1999-11-18 | Scott Gilbert | Panoramic movies which simulate movement through multidimensional space |
| US7668928B2 (en) * | 2001-03-13 | 2010-02-23 | Goldpocket Interactive Inc. | System and method for recording and playing back interactive content during a broadcast event |
-
2001
- 2001-06-08 CA CA002411852A patent/CA2411852A1/en not_active Abandoned
- 2001-06-08 EP EP01942165A patent/EP1297634A1/en not_active Withdrawn
- 2001-06-08 AU AU2001275453A patent/AU2001275453A1/en not_active Abandoned
- 2001-06-08 JP JP2002502936A patent/JP2003536319A/en active Pending
- 2001-06-08 US US09/877,166 patent/US20020021353A1/en not_active Abandoned
- 2001-06-08 WO PCT/US2001/018731 patent/WO2001095513A1/en not_active Application Discontinuation
- 2001-06-08 IL IL15316401A patent/IL153164A0/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| IL153164A0 (en) | 2003-06-24 |
| EP1297634A1 (en) | 2003-04-02 |
| WO2001095513A1 (en) | 2001-12-13 |
| AU2001275453A1 (en) | 2001-12-17 |
| JP2003536319A (en) | 2003-12-02 |
| US20020021353A1 (en) | 2002-02-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20020021353A1 (en) | Streaming panoramic video | |
| JP7029562B2 (en) | Equipment and methods for providing and displaying content | |
| JP4877852B2 (en) | Image encoding apparatus and image transmitting apparatus | |
| CN112204993B (en) | Adaptive panoramic video streaming using overlapping partitioned segments | |
| JP4414345B2 (en) | Video streaming | |
| US9756328B2 (en) | System, terminal, and method for dynamically adjusting video | |
| CN109891906A (en) | View perceives 360 degree of video streamings | |
| US11792463B2 (en) | Method of video transmission and display | |
| US20210227236A1 (en) | Scalability of multi-directional video streaming | |
| US9392303B2 (en) | Dynamic encoding of multiple video image streams to a single video stream based on user input | |
| KR101528863B1 (en) | Method of synchronizing tiled image in a streaming service providing system of panoramic image | |
| CN109362242A (en) | A kind of processing method and processing device of video data | |
| EP3434021B1 (en) | Method, apparatus and stream of formatting an immersive video for legacy and immersive rendering devices | |
| CN112753224B (en) | Apparatus and method for generating and rendering video streams | |
| JP5941000B2 (en) | Video distribution apparatus and video distribution method | |
| CN115580738B (en) | High-resolution video display method, device and system for on-demand transmission | |
| JP5594842B2 (en) | Video distribution device | |
| US20030179216A1 (en) | Multi-resolution video-caching scheme for interactive and immersive videos | |
| JP4241708B2 (en) | Video image presentation system, video image transmission device, video image presentation device, video image processing program | |
| KR20200076529A (en) | Indexing of tiles for region of interest in virtual reality video streaming | |
| EP4013059A1 (en) | Changing video tracks in immersive videos | |
| US20210352344A1 (en) | Content distribution system, content distribution method, and program | |
| WO2023194648A1 (en) | A method, an apparatus and a computer program product for media streaming of immersive media | |
| CN116137954A (en) | Information processing apparatus, information processing method, and information processing system | |
| JP2010016713A (en) | Content distribution apparatus and display device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FZDE | Discontinued | ||
| FZDE | Discontinued |
Effective date: 20050608 |