CA2281211C - Method and apparatus for receiving mpeg video over the internet - Google Patents
Method and apparatus for receiving mpeg video over the internet Download PDFInfo
- Publication number
- CA2281211C CA2281211C CA002281211A CA2281211A CA2281211C CA 2281211 C CA2281211 C CA 2281211C CA 002281211 A CA002281211 A CA 002281211A CA 2281211 A CA2281211 A CA 2281211A CA 2281211 C CA2281211 C CA 2281211C
- Authority
- CA
- Canada
- Prior art keywords
- high priority
- bytes
- priority partition
- packets
- partition information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/65—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience
- H04N19/66—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using error resilience involving data partitioning, i.e. separation of data into packets or partitions according to importance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
Abstract
In order to transmit an inter-frame coded video signal, such as an MPEG-coded video signal, over a packet-based network such as the Internet, the video signal associated with at least one video frame, is split into a high priority partition and a low priority partition. A systematic forward error erasure/correction (FEC) code, such as a Reed Solomon (n,k) code, is then applied to bytes in the high priority partition. The forward error/erasure corrected high priority partition bytes and the low priority partition bytes are then combined into n packets for transmission over the packet network to a receiver/decoder.
Each of the n transmitted packets contains a combination of both high priority partition data bytes and low priority partition information bytes. In k of those packets the high priority partition data bytes are all high priority partition information bytes and in n-k of those packets all the high priority partition data byte are parity bytes produced by the FEC coding. More specifically, for each high priority partition byte position within the n packets, the forward error/erasure correction code is applied using one high priority partition information byte from the same byte position in each of those k packets to determine n-k parity bytes, which are arranged, one byte per packet, in the n-k packets containing high priority partition parity bytes. If up to n-k packets are lost in transmission over the packet network to the receiver, then the high priority partition bytes in such lost packets can be recovered to applying FEC
decoding to the high partition bytes in the received packets. The most visually significant information is thus protected against packet loss over the network.
Each of the n transmitted packets contains a combination of both high priority partition data bytes and low priority partition information bytes. In k of those packets the high priority partition data bytes are all high priority partition information bytes and in n-k of those packets all the high priority partition data byte are parity bytes produced by the FEC coding. More specifically, for each high priority partition byte position within the n packets, the forward error/erasure correction code is applied using one high priority partition information byte from the same byte position in each of those k packets to determine n-k parity bytes, which are arranged, one byte per packet, in the n-k packets containing high priority partition parity bytes. If up to n-k packets are lost in transmission over the packet network to the receiver, then the high priority partition bytes in such lost packets can be recovered to applying FEC
decoding to the high partition bytes in the received packets. The most visually significant information is thus protected against packet loss over the network.
Description
METHOD AND APPARATUS FOR RECEIVING
MPEG VIDEO OVER THE INTERNET
Field of the Invention This invention relates to the transmission and reception of coded video s signals over the Internet, and more specifically, to the transmission and reception of video signals that have been coded using compression efficient inter-frame coding techniques such as those used in MPEG, MPEG-2, H.261, and H.263 standards.
Back4round of the Invention Io With the exploding popularity of the public Internet in the past several years for transporting all types of data, there has been much recent interest in transmitting digitally encoded real-time audio and video over the Internet using the Universal Datagram Protocol (UDP). Because UDP is an unreliable protocol, network packet losses will likely occur and, as a result, will adversely affect the is quality of the received audio and video. Recovery from packet losses may be performed solely by the receiver, or better quality can be achieved by involving both the sender and the receiver in the error recovery process. In networks that support prioritization, such as ATM, video quality can be improved in the Boyce 5 presence of packet loss by using scalable video coding (see, e.g., R. Aravind, M.
Civanlar, A. Reibman, "Packet Loss Resilience of MPEG-2 Scalable Video Coding Algorithms," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 6, No. 5, October 1996). There is currently, however, no s widespread support for prioritization on the public Internet. Overviews of proposed methods for error recovery for streaming of audio and video over the Internet, which involve both the sender and the receiver are disclosed by C. Perkins and O. Hodson in "Options for Repair of Streaming Media," Internet Engineering Task Force Internet RFC 2354, June 1987,and G. Carle and E.
io Biersack in "Survey of Error Recovery Techniques for IP-Based Audio-Visual Multicast Applications," IEEE Network, November/December 1997. While the general methods described in these overviews may be applicable to IP
transmission of both audio and video, most of the studies published where specific techniques have been implemented involve audio only. Because of its is higher data rates, and propagation of errors through inter-frame coding, it is more difficult to maintain video quality than audio, and audio techniques, therefore, cannot be directly applied to video signals.
Many of the currently popular schemes for transmitting digital video over the Internet, such as Motion-JPEG and wavelet-based schemes, use intra-frame 2o coding. Inter-frame coding techniques, such as those used in MPEG-1, MPEG-2, H.261, and H.263 standards, are generally more compression-efficient than intra-frame techniques. However, the inter-frame standards suffer more from Internet packet loss because errors in one frame may propagate for many frames. An MPEG video sequence includes intra-frame coded (I) frames, and 2s inter-frame predicted coded (P), and bi-directional inter-frame coded (B) frames.
I and P frames are used in the prediction of subsequent frames while B frames are not used in the prediction of subsequent frames. For example, consider an Royce 5 MPEG video sequence with I frames occurring every 15 frames. In MPEG
coding, because of inter-frame prediction, all predictive P and B frames rely upon the previous I frame. Thus, if an error occurs while transmitting the I frame, the effect persists for 15 frames, or 500 ms, which is quite noticeable to a viewer.
s The received video quality can be improved both through error concealment techniques that are applied at the decoder, and by error resilience techniques that are applied at the sender.
Error resilience techniques using Forward Error/Erasure Correction (FEC) add redundant data to a media stream prior to transmission, so that packet io losses can be repaired at the receiver without requiring contact with or re-transmissions from the sender. Forward Error/Erasure Correction techniques are well suited to multicast applications, because they avoid the use of re-transmissions. The same redundant data can be used to repair the loss of different packets at separate receivers in a multicast group. If re-transmission is were used instead, multiple re-transmission requests would have to be sent.
Forward Error/Erasure Correction techniques for multimedia generally fall into one of two categories, media-independent FEC and media-specific FEC ( see, e.g., C. Perkins and O. Hodson, "Options for Repair of Streaming Media,"
Internet Engineering Task Force Internet RFC 2354, June 1998).
2o In media-independent FEC, well-known information theory techniques for protecting any type of data are used. In, "Media-independent Error Correction using RTP,n Internet Engineering Task Force Internet Draft, May 1997 by D. Budge, R. McKenzie, W. Mills, and P. Long, several variations of exclusive-OR (XOR) operations are used to create parity packets from two or more data 2s packets. More complex techniques such as Reed Solomon (RS) coding can also be used (see, e.g., G. Carle and E. Biersack, "Survey of Error Recovery Techniques for IP-Based Audio-Visual Multicast Applications," IEEE Network, Boyce 5 Novemberfflecember 1997). Reed-Solomon encoding is an example of a systematic forward errorlerasure correction code. A systematic forward error/erasure correction code is one in which the information bytes are transmitted in the codeword without modification. Thus, in the absence of s channel errors, no Reed-Solomon decoding is necessary to recover the information bytes. When an RS(n,k) codeword is constructed from byte data, h parity bytes are created from k information bytes, and all n = k + h bytes are transmitted. Such a Reed Solomon decoder can correct up to any h/2 byte errors, or any h byte erasures, where an erasure is defined as an error in a Io known position. When RS coding is applied to protect packetized data against packet loss, k information packets of length j bytes are coded using j RS
codewords. For each RS codeword, k information bytes are taken from k different packets (one from each packet), and the h constructed parity bytes are placed into h separate parity packets, and all n = k + h packets are transmitted.
is Because the transmitted packets are numbered, and packets are assumed to be received perfectly or not at all, the receiver can determine which packets are missing, and thus a packet loss can be considered to be an erasure. Hence, if any h (or fewer) of the n transmitted packets are lost, the original k information packets can be recovered perfectly.
2o A key advantage of RS coding is its ability to protect against several consecutive errors, depending on the parameter choices. The overhead rate for RS coding is h/k, and it is most efficient for protection against burst errors for large values of k. For example, an RS(6,4) code and an RS(4,2) code both can protect against a burst length of 2 errors. But the RS(4,2) code has 100%
as overhead, while the RS(6,4) code has only 50% overhead. Reducing the overhead percentage by increasing the block length, however, leads to delay Boyce 5 because large block lengths require buffering of large amounts of data prior to transmission.
In media-specific FEC coding unlike in media-independent FEC coding where the multimedia stream is just treated as data, knowledge of the specific s type of multimedia stream to be transmitted is used. In "Simulation of FEC
Based Error Control for Packet Audio on the Internet," INFOCOM, March 1998, San Francisco, CA by M. Podolsky, C. Romer, and S. McCanne, and in 'Reliable Audio for Use over the Internet," Proc. INET '95, Honolulu, HI, pp. 171-178, June 1995, by V. Hardman, M.A. Sasse, M. Handley, and A. Watson. a redundant low-io bit rate audio stream is transmitted along with the standard audio stream, but delayed by one packet. If a standard audio packet is lost, the receiver uses the low-bit rate version of that audio instead, received in the next packet. This method protects against single packet losses.
In the aforenoted article by Perkins and Hodson, a suggestion is made to is combine media-specific and media-independent techniques by applying the media-independent FEC techniques to the most significant bytes of a coder's output, rather than applying FEC over the entire multimedia bitstream. No specific information about how this can be accomplished is given however.
A method for adding resilient information to inter-frame coded video, such as 2o MPEG video, in order to protect video quality against packet loss, but which has low overhead and low delay is desirable.
Summary of the Invention In accordance with the present invention, an inter-frame coded video 2s signal, such as an MPEG video signal, employs a data splitting function to split such a video stream into a high priority and a low priority partition.
Systematic Forward Error/Erasure Correction coding is then performed on only the data in Boyce 5 the high priority partition. The Forward Error/Erasure Corrected high priority partition data and the non-Forward Error/Erasure Corrected low priority partition data are then combined into packets and transmitted over the same network to a receiver, where they are decoded. Depending on the degree of protection s against errors or erasures offered by the particular FEC code that is used, the loss of one or more packets containing high priority data can be corrected with no loss of data in the high priority partition. The effect of the loss of the low partition data in the lost packet or packets, which low partition is not protected, has much less of a deleterious effect on the quality of the decoded video signal io than would the loss of data from the high priority partition data.
Advantageously, by limiting the application of the Forward Error/Erasure Correction to only the higher priority partition data, and thus protecting against loss only that "more important data", the overhead requirement is reduced for protection against a given packet loss.
is In the preferred embodiment, a Reed Solomon encoder is applied to the high priority data for an entire frame. For each RS(n,k) codeword, one information byte is taken from each of k packets and the constructed parity bytes are placed in h different packets, where n = k + h. Each individual frame's data is arranged in the n equal length packets that contain a combination of:
packet 2o headers; high priority data comprising one of either information bytes or parity bytes; and low priority data bytes, the latter comprising only information bytes since no error-correction coding is performed on the tow priority data. The same number of bytes of high priority data (information or parity in any one packet) are placed in each, of the n equal length packets, and the same number of bytes of 2s low priority data (information only) are placed in these same n packets, which together represent the video frame. Amongst these n equal length packets, k packets only contain high priority partition information bytes and h packets only i ' contain the high priority parity bytes. The parity byte in each high priority byte position in each of these h packets is formed from the RS(n,k) code as it is applied to the k high priority partition information bytes in a corresponding byte position in the k other high priority partition information-containing packets s associated with the frame. Advantageously, arranging the packets in this manner minimizes the amount of overhead and delay for a given packet loss protection rate.
A receiving decoder, upon receiving the packets associated with a frame separates the high priority partition bytes and low priority partition bytes in each ~o packet according to the numbers of such bytes or each type included within each packets, which numbers are transmitted in the packet headers. RS(n,k) decoding is applied byte position-by-byte position across the high priority partition portion within the received packets. If up to h of the n frame packets are lost, the RS decoding process recovers each high priority byte in the lost is packet or packets. Full reconstruction of the high priority partition information bytes that were transmitted in the k packets of the n packets that contained high priority partition data is thus effected. Although the low priority partition data in the lost packets is unrecoverable, the fully recovered high priority partition data enables the video picture to be decoded, albeit in what might be at a reduced 2o quality level for that frame or that portion of a frame in which only the high priority partition information is available.
In accordance with one aspect of the present invention there is provided a method of decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of 2s packets that are associated with at least one frame of the compression-coded video signal, the method comprising the steps of: receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes 7a associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted; determining which transmitted packets associated with the at least one frame of the video s sequence have not been received; applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the step of applying a FEC decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
io combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
is In accordance with another aspect of the present invention there is provided a decoder for decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the decoder comprising: a depacketizer connected to receive 2o packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into 2s low priority and high priority partitions before having been packetized and transmitted, the depacketizer separating the FEC-coded high priority partition data bytes and the low priority partition information bytes; a FEC decoder connected to receive and decode the FEC-coded high priority partition data bytes; and a video decompression decoder connected to receive the 7b depacketized low priority partition information bytes and the FEC-decoded high priority partition data bytes, the video decompression decoder combining and decompression-decoding the low priority partition information bytes and the FEC-decoded high priority partition data bytes.
s Brief Description of the Drawing FIG. I is a block diagram of first embodiment of a video encoder in accordance with the present invention which uses a data partitioning MPEG
encoder to code and split the coded video signal into HP and LP partitions;
Boyce 5 FIG:~2 the arrangement of HP data and parity information, and LP data within the packets associated with a frame for an example of RS(4,3) encoding;
FIG. 3 is a flow chart that details the steps for determining the n and k parameters associated with a frame as a function of the number of LP and HP
bytes in the frame;
FIG. 4 is a block diagram of a second embodiment of a video encoder in accordance with the present invention in which a standard MPEG encoder is used to encode the video signal and a data splitter divides the coded signal into HP and LP partitions;
to FIG. 5 is a block diagram of an embodiment of a video decoder in accordance with the present invention in which lost packets are reconstructed, HP and LP partitions reformed, and a data partitioning MPEG decoder is used to decode the video signal from the HP and LP partitions; and FIG. 6 is a block diagram of a second embodiment of a video decoder in is accordance with the present invention in which a data merger combines the HP
and LP partitions, which are then inputted to a standard MPEG decoder to decode the video signal.
Detailed Description 2o The MPEG-2 standard (ITU-T Recommendation H.262, "GENERIC
CODING OF MOVING PICTURES AND ASSOCIATED AUDIO INFORMATION", July 1995 ) includes several means for performing scalable video coding, including spatial scalability, SNR scalability, and data partitioning. In scalable video coding, receivers with different available bandwidths can receive and 2s decode appropriate representations of coded video, by receiving a base layer, and if bandwidth is available, receiving one or more enhancement layers. The more layers received, the higher the quality of the decoded video. In the Boyce 5 aforenoted-paper by Aravind, Civanlar and Reibman, data partitioning was applied to provide protection against packet loss for transmitting an MPEG-coded video signal in a network that supports prioritization such as ATM.
Specifically, protection against packet loss was shown to be achievable by s transmitting the base layer at a higher priority than the enhancement layer(s).
Since the public Internet does not currently support prioritization of packets this technique cannot be applied to the transmission of coded video over the Internet.
The inventor has recognized, however, that an advantage can be achieved by using, in a non-prioritized network, such as the public Internet, a Io data splitting functionality that is used in the prior art for packet protection over a network that does support prioritization. By incorporating a data splitting functionality together with a Forward Error/Erasure Correction functionality for transmitting an inter-frame compression-coded video signal over a non-prioritized public Internet, the amount of overhead needed for packet protection is is reduced, while achieving the improvement in the subjective video quality that packet protection affords. Further, by combining high priority and low priority data in the same packets, the delay for equal overhead and protection is advantageously reduced.
FIG. 1 is a first embodiment of an encoder in accordance with the present 2o invention. In this embodiment of the present invention, an encoder which is compliant with MPEG-standardized data partitioning is used to split an incoming video data stream into a high priority partition and a low priority partition.
With reference to FIG. 1, an incoming video data stream on 101 is inputted to a such a compliant data partitioning MPEG-standardized encoder 102 which, in 2s accordance with the standard, such as the MPEG-2 data partitioning standard, compression-codes the input video bitstream and splits the compression-coded bitstream into two output layers. The first layer is the base layer, referred to Boyce 5 10 herein as the high priority (HP) partition. The second layer is the enhancement layer, referred to herein as the low priority (LP) partition.
As is well known to one skilled in the art of MPEG coding, an MPEG-coded video bitstream includes headers at the sequence level, at the group-of s picture (GOP) level, at the picture (frame) level, and at the slice level.
As is well known, a slice consists of a group of adjacent macroblocks, where each macroblock in itself consists of a group of four adjacent luminance blocks of data and two chrominance blocks. At the picture level, a frame is classified as being as an intra-frame coded (I) frame, an inter-frame coded predictive (P) frame, or a to bi-directional inter-frame predictive (B) frame. At the macroblock level, for the predictive-type P and B type frames, information is included which indicates whether the macroblock is inter or intra coded with respect to another frame, as well as motion vector information. The information at the block level includes low frequency and high frequency discrete cosine transformation (DCT) coefficients is derived from actual pixel information.
In accordance with MPEG standards, data partitioning can be effected at a plurality of different priority breakpoints, such as above the macroblock layer, at the macroblock layer, or within the macroblock layer. Generally, the more critical parts of the coded bitstream, such as the headers, the motion vectors and 2o the low frequency DCT coefficients, are allocated to the HP partition, and the less critical data, such as the higher frequency DCT coefficients, are allocated to the LP partition. In accordance with the standard, a priority breakpoint can be chosen for each slice, which determines which codeword types are arranged into which partition.
2s In the embodiment of the invention in FIG. 1, which uses the standardized data-partitioning MPEG encoder 102, priority breakpoints are determined in accordance with the type of frame (1, P, or B) in the data stream that is being BOyCe 5 11 partitioned:- Specifically, in this embodiment, for each I frame, all the data is placed in the HP partition, with no data being placed in the LP partition.
Other embodiments may divide the data from each I frame between the HP and LP
partitions. For each B frame, as much frame data as the MPEG standards s relating to data partitioning allows is placed in the LP partition, with the remaining data placed in the HP partition. For each P frame, frame data is divided between the HP and LP partitions so that data elements through the first two DCT
coefficients of each block are placed in the HP partition, with the higher order DCT coefficients placed in the LP partition. Different priority breakpoints in the to three frame types between the LP and HP partitions other than described above for this embodiment could equally be used. It would be expected, however, that a higher breakpoint would be used for I frames than for P frames, which, in turn, would be higher than the breakpoint for B frames. In this standards-compliant embodiment, sequence, GOP, and picture headers are copied into both is partitions for error resilience.
In accordance with the invention, the HP partition is encoded with a systematic Forward Error/Erasure Correction code. Specifically, in this preferred embodiment, Reed Solomon coding is applied to the HP data for a single frame across the entire frame. For this embodiment, only data from a single frame is 20 operated upon by a Forward Error/Erasure Correction encoder/packetizer 103 at a time to minimize delay that would be incurred if data from multiple frames needed to be accumulated before being coded and transmitted. In other embodiments, B frames as well as their immediately following anchor frame (I
or P) may be operated upon together by the Forward Error/Erasure Correction 2s encoder/packetizer 103.
Reed Solomon encoding is applied to the HP data output of the MPEG
encoder 102 for the frame. As will be described in detail hereinafter, the number Boyce 5 t2 of bytes in the HP partition and the number of bytes in the LP partition are used together and separately, together with a maximum allowable packet length and a desired degree of packet protection, to determine the number of equal-length packets, n, into which, for that frame, the HP information bytes, the LP
s information bytes, and the protective HP parity bytes will fit. In accordance with the invention, each of the n equal-length packets contains a combination of both LP information bytes, and one of HP information and parity bytes. The parity bytes are determined by forming an RS(n, k) codeword by taking one HP
information byte from each of k packets that contain the HP information bytes to and placing the h parity bytes constructed from these k bytes, one byte at a time, into h different packets, where n = k + h. Thus, the parity bytes in those h packets in byte position m (where m varies between 1 and the number of HP
information bytes in a packet) are derived from the HP information bytes in that same byte position m in the k packets. Since the parity bytes in each byte is position in those h packets are calculated from the HP information bytes in the same byte positions in the other k packets, each equal-length packet contains an equal number of HP data bytes, the latter being either all information bytes or all parity bytes. It follows, therefore, that each equal-length packet then contains an equal number of LP information bytes. Because the number of HP information 2o bytes in the frame is not necessarily divisible by the number of k packets, padding bytes are added, as necessary, at the end of the kU' packet.
Similarly, padding bytes may be applied to the n~' packet for LP information bytes.
FIG. 2 shows an example of an arrangement of a frame group of packets for an RS (4, 3) code. As can be noted, three packets (k = 3) contain HP
2s information bytes information bytes and LP information bytes. One packet (h = 1) contains the HP parity bytes and the LP information bytes. Arranging the packets in the manner illustrated by this figure minimizes the amount of Royce 5 13 overhead for a given packet loss protection rate without adding to delay. The packet header in each packet contains the packet number, the number of the packets in the frame group of packets (n), the number of packets with parity data in the frame group of packets (h), the temporal reference value for the frame, the s frame type (I, P, or B), the HP/LP priority breakpoints, and the number of HP and LP bytes in each packet.
With reference back to FIG. 1, the output of the data partitioning MPEG
encoder 102 consists, for an input stream of video data associated with a frame, of two data streams, an HP partition data stream on output 104 and a LP
io partition data stream on output 105, both of which are inputted to the FEC
Coder/Packetizer 103. The number of bytes in the HP partition of the input frame, HP#, and the number of bytes in the LP partition of the input frame, LP#, are inputted to a processor 106 within coder/packetizer 103. As will be described in detail hereinafter, processor 106, using the values HP# and LP#, is together with a predetermined maximum number of bytes per frame, determines the number of packets, n, needed to transmit the entire frame, including packet headers, the HP partition information bytes, the HP partition parity bytes, and the LP partition information bytes. Further, processor 106 determines the number of packets, h (equal to n-k), which are needed to protect the HP partition ao information bytes in the other k packets for a desired level of protection.
Processor 106 also determines the byte-length of each of the n packets.
Processor 106 then determines the numbers of HP data or parity bytes and LP
data bytes to be apportioned into each of these n packets, with LP information bytes allocated to each of the n packets, HP information bytes being allocated to 2s the k packets, and HP parity bytes being allocated to the other h packets, as per the example shown in FIG. 2.
Royce 5 is Once processor 106 has determined all the parameters that determine the configuration of each packet and the number of such packets, a Reed Solomon RS(n,k) encoding is effected, byte by byte on the bytes within the HP
partition.
Specifically, the HP partition video stream is inputted to a byte interleaver 107, s which in response to the parameters determined by processor 106, forms sequential codewords that are inputted to a Reed Solomon encoder 108. For example, if the number of HP bytes per packet are calculated to be 650, the 1$', 651't, 1301, et seq., bytes in the HP stream are interleaved by interleaver 107 to form an input word to RS encoder 108 in order to determine the bytes for the first Io parity byte position for the h packets which are to contain HP parity bytes. The 2nd, 652"°, 1302"x, et seq., bytes are then interleaved to form the next input word to RS encoder 108 to determine the parity bytes for the second parity byte position for the h HP parity packets. In such a manner, the RS encoder 108 is sequentially presented with a series of k byte input words, each byte in each k is byte input word comprising one information byte from what will be, when re-interleaved, k different packets. For each k byte input word, h parity bytes are determined. Thus, for each k-byte input word, RS encoder 108 outputs an n-byte word containing k information bytes and h parity bytes.
In order to reassemble these vertically oriented codewords, each 2o containing bytes properly belonging in n different packets, back into a row-oriented packet format, a de-interleaver 109 sequentially places each byte from each n-byte output word into a separate one of n different rows. The first k of such n rows thus contain all HP information bytes and the last h of such rows contain all HP parity bytes. When the n rows are reconstructed, each such row 2s is combined by a packetizer 110 with a packet header and the calculated fixed number of LP partition information bytes apportioned to each packet from the LP
output 105 of the data partitioning MPEG encoder 102. Each packet, then in a Boyce 5 1 s format as shown in FIG. 2 for the illustrative example for an RS(4,3) code, is sequentially outputted on output 111 for transmission onto a UDP/IP network.
For the example of FIG. 2, if none of the packets containing the frame information are lost in transmission, the decoded video will be perfectly decoded s (i.e., identical to that encoded). If one of the packets (or up to h packets in the general case for R(n,k) coding) is lost, then all of the HP information in that packet is recoverable using a Reed Solomon decoder. The LP information data in the lost packet could not, however, be recovered since it is not protected.
The portions of the picture that correspond to the macroblocks whose LP data was to received will, however, be perfectly decoded, and those that correspond to macroblocks whose LP data is lost will decode only the HP data for those macroblocks. Those macroblocks which are decoded using only HP data may be noticeable to a viewer, but are unlikely to be visually offensive as long as the HP/LP priority breakpoint is properly chosen. The exact visual quality of is macroblocks decoded using only HP data depends on this HP/LP priority breakpoint, as well as the characteristics of the video source material. In the embodiment of FIG. 1 in which a standardized data partitioning MPEG encoder 102 is used to compression-code the input video stream and form the HP and LP
partitions, the lowest level header copied into both partitions is the picture 2o header. Thus, once any packet is lost, and with it the LP partition information contained therein, which cannot be recovered, the LP partition data that is received in the next received packets cannot be properly incorporated into the decoding process since there is no identifiable spatial entry point with which that received data can be associated until a next picture header is received. Thus, 2s with the embodiment of FIG. 1, the perceptual effect of lost LP partition information will extend until a packet is received that contains a next picture header within the received LP partition information. Once that picture header is Boyce 5 16 received, the LP partition data that follows can be properly incorporated with the received HP partition data.
As previously noted, the exact arrangement of the frame data into packets is a function of the number of bytes of HP partition information and LP
partition s information in the frame, the parameters of the RS(n,k) code, and the maximum packet size. For Internet Protocol (1P) transmission, the maximum packet size is set, for this embodiment, to the Ethernet Maximum Transport Unit (MTU) size of 1500 bytes. The choice of the RS(n,k) code parameters depends on the number of packets needed to transmit the frame, on the network loss conditions, and the to overhead percentage considered acceptable. Because Internet packet loss has a tendency to be somewhat bursty, the k/n ratio can be chosen to be lower for lower values of n and higher for higher values of n. Table 1 is an example of pairs of n and k values in which, based on the bursty nature of packet loss, the ratio kln is set lower for lower values of n and higher for higher values of n.
Boyce 5 Table 1 n k Given a list of (n,k) values such as that given in Table 1, the particular s (n,k) pair to use for each frame can be determined using an iterative process.
FIG. 3 is a flowchart that details the steps that processor 106 follows for determining the these parameters. After the data partitioning MPEG Encoder 102 splits the bitstream into the HP partition and the LP partition, there are HP#
bytes in the HP partition and LP# bytes in the LP partition. At step 301, an initial to estimate is formed for the number of packets, n, needed to transmit this information, assuming a maximum packet size PSmax , which in this embodiment Boyce 5 1 g is equal to the Ethernet MTU size minus the number of packet header bytes used. This initial estimate of n is formed from n = cell (HB# + LB#)lPSmeX, where ceil () is the ceiling function. This initial estimate of n is calculated without consideration of the HP parity bytes, which must be incorporated into the frame.
s The value of k that corresponds to the current estimate of n is retrieved, at step 302, using Table 1. At step 303, the necessary packet size for the HB#
information bytes is calculated from CurPacketSize = ceil (HB#lk) + ceil (LB#ln), because the HB# information bytes are split among k packets (with parity bytes in the remaining h packets) and the LB# information bytes are split among n io packets. At step 304, CurPacketSize is compared with PSm~ . If CurPacketSize is smaller than PSmaX , the current (n,k) parameters may be used. If not, the packet size is too large, and n is incremented at step 305. Table 1 may yield a new k for this incremented value of n. CurPacketSize is then recalculated using these new parameters. The process ends at step 306 when the resultant is parameters yields a CurPacketSize of less than PSmax . The actual number, HBP, of HP bytes (information or parity) in each of the n frames is then equal to ceil (HB#lk) and the actual number, LPB, of LP information bytes in each of the n frames is equal to ceil (LP#ln), the total number of bytes, less the length of the packet header, in each frame, TPB, thus being equal to HBP+LPB.
2o As an illustrative example, a P frame will be assumed to have a size of 5632 bytes, in which HB# = 2129 HP bytes and LB# = 3503 LP bytes. PSmax is assumed to be equal to 1475 bytes, derived from a 1500 byte MTU size minus 25 bytes for a packet header. The initial estimate of the number of packets is n=ceil ((2129+3503)/1475) = ceil (3.818) = 4. From Table 1, for n=4, it is seen Zs that k=3. Using these parameters, HPB, LPB and TPB are calculated:
HPB=ceil (2129/3)=710; LPB=ceil (3503/4)=876; and TPB= 710+876=1586, which is greater than PSmax of 1475. Thus, that set of parameters is not valid and Boyce 5 19 another iteration is necessary. For this next iteration, n is incremented to 5.
From Table 1, for n=5, it is seen that k=4. Using these parameters, HPB, LPB
and TPB are calculated: HPB=ceil(2129/4)=533; LPB=ceil(3503/5)=701; and TPB=533+701=1234, which is less than PSmaX of 1475.
s Once the parameters of n, k, HPB and LBP are determined for the frame, the aforedescribed Reed Solomon coding and arrangement of the HP
information and parity bytes, and LP information bytes into packets can be effected. Specifically, once processor 106 determines these parameters from the partitioned HP and LP data for each frame, that frame's data is interleaved, io RS encoded, de-interleaved and packetized for transmission over a UDP/IP
network, as previously described.
In the embodiment of FIG. 1, a data partitioning MPEG encoder 102 was used to split the bitstream into the HP and the LP partitions. FIG. 4 is a block diagram of an alternative embodiment which although not maintaining strict is MPEG compliance, in order to reduce overhead and improve performance, can still be used with standards compliant MPEG encoders and decoders. This embodiment may applied to MPEG-1 or MPEG-2 video, as well as other similar video coding standards such as H.261 and H.263. In this embodiment a standard MPEG encoder (or other standard encoder) 401 compression codes Zo the input video stream. A data splitter 402 then divides the compression-coded output of encoder 401 into HP and LP partitions. For purposes of the present invention, the combination of the standard MPEG encoder 401 with a data splitter 402 enables the splitting function to be performed more efficiently and with improved performance as compared to the data partitioning MPEG encoder 2s 102 in FIG. 1. Specifically, only slice headers are duplicated in both partitions and all other data is placed in one partition or the other, but not both.
Because of the frame alignment of packets and the containment of frame information in Boyce 5 20 the packet-headers themselves, it is not necessary to duplicate the picture headers and above in each of the partitions, thereby minimizing overhead. By providing slice headers in both the LP and HP partitions, however, a point of entry is provided for the LP data upon a packet loss. Thus, in a received packet s that follows a lost packet, the LP data can be incorporated into the decoded picture after the decoder receives, with the LP partition data, the next slice header. Advantageously, as compared the embodiment of FIG. 1, in which the point of insertion for the received LP data following a packet loss was the next frame, the embodiment of FIG. 4 provides a point of insertion for the LP data at io the next slice, thereby improving the picture quality of the decoded video signal.
As previously described, the data partitioning MPEG encoder 102 partitions I frames so that all frame data is placed in the HP partition to minimize the effect of an error in an I frame from propagating to other frames. Since B
frames are not used for prediction, as much data as is possible via the data is partitioning standard is placed in the LP partition. In the embodiment of FIG. 4, all data for all B frames is placed in the LP partition as opposed to the embodiment of FIG. 1 in which some data, in accordance with the standards, is required in the HP partition. In both the embodiment of FIG. 1 and FIG. 4, the P
frame data is divided between the HP and LP partitions in the manner described 2o above. Within each P frame macroblocks can be either inter-frame or intra-frame coded. In the embodiment of FIG. 4, different priority breakpoints are chosen for inter and intra coded macroblocks in each P frame, which the data partitioning MPEG encoder 102 in FIG. 1 could not do. Inter-coded macroblocks decoded using only the HP partition (and not the LP partition) may retain high 2s frequency information from the corresponding motion-compensated macroblocks in the previous frame, while intra-coded macroblocks may not. Hence, it is desirable to set the priority breakpoint for intra-coded macroblocks to include Boyce 5 21 more DCT coefficients in the HP partition than are included in inter-coded macroblocks. Advantageously, this reduces the overhead rate for given level of quality or improves the quality for the same overhead rate.
The HP and LP outputs of data splitter 402 are applied to a FEC
s coder /packetizer 403, which functions in the same manner as FEC
coder/packetizer does in FIG. 1, which functions have been described above.
The packets outputed by FEC codeNpacketizer 403 are then transmitted over a UDPIIP network.
Although the embodiment in FIG. 4 includes a separate standard MPEG
io encoder 401 and data splitter 402, it should be recognized that a data coder that supports the described data splitting operation could equally perform the same functions that encoder 401 and splitter 402 together do.
The encoder networks in FIG. 1 and FIG. 4 transmit output packets over a UDP/IP network such as the public Internet, to corresponding decoders is connected to that network. In the case of a multicasting encoder, the transmission may be simultaneously broadcast to a plurality of end-users who will each receive the transmitted information. Since different packets may be lost along the routes to different of such end-users, the FEC techniques employed in the present invention to protect against packet loss enable each individual 2o decoder to recover the particular packets that may have been lost on the path to that end-user up to, of course, the level of packet protection provided by the specific RS code used by the encoder.
A decoder network 500 that is associated with the encoder network of FIG. 1 is illustrated in FIG. 5. As a data partitioning MPEG encoder 102 is 2s incorporated within the encoder network of FIG. 1, the decoder network of FIG. 5 incorporates a complimentary data partitioning MPEG decoder 510. In FIG. 5, the serial packets transmitted by the encoder network over the UDP/IP network Boyce 5 22 501 are inputted to a de-packetizeNdecoder 502. De-packetizeNdecoder 502 includes a de-packetizer 503 which receives the serial packets and strips the header information in each packet and passes it to a processor 504. The packet header information includes: a packet number; a frame number; the type of s frame (I, B, P); the (n,k) parameters which define the packet structure of the frame; and the number of HP partition bytes, HPB, and LP partition bytes, LPB, in each packet. Processor 504, from the header information, determines the start of the frame, and from the parameter n, "knows" that that many packets are used to define the frame. Further, from the packet numbers received, processor to 504 determines which particular of those n packets, if any, are missing and their position within the sequence of these n packets. In response to receiving all such information from processor 504, de-packetizer 503 strips off the packet header of each packet within the frame and divides the data in each packet into its HP and LP portions. For those packets which are determined to be missing, Is de-packetizer 503 inserts "0" bytes or an error code in the HP and LP data streams. The HP serial byte stream output of de-packetizer 503 consists of n sub-packets-worth of HP data, each sub-packet containing HPB bytes. This HP
stream is inputted to an interleaver 505 to decode the RS(n,k) encoded words that exist across the sub-packets and to replace any missing data from up to h 20 lost packets. Thus, for each byte position across the sub-packet, a byte is selected from each such sub-packet at that byte position to form an input word to an RS decoder 506.
As an example, using the frame structure shown in FIG. 2 where n is equal to 4 and k is equal to 3, and with HPB being equal 650 bytes, interleaver 2s 505 selects the 1 St, 651 St, 1301 S' and 1951 S' bytes in the HP byte stream to form a 4-byte input word to RS decoder 506 to determine the 3-byte corresponding output word, each byte in the three-byte output word being an HP information Boyce 5 23 byte in the #irst byte position in each of the three information sub-packets.
Interleaver 505 then presents the 2"d, 652"d, 1302"d and 1952"d bytes in the HP
byte stream to RS decoder 506 to determine the bytes in the second byte position in each of the three information sub-packets. Interleaver 505 similarly s processes the 3~ through the 650'" bytes. The three information bytes for each byte position are thus sequentially outputted by RS decoder 506. Since the RS(4,3) code is capable of correcting up to one erasure, if one of the four packets is lost and thus not received by the decoder network 500, RS decoder 506 will determine the missing byte in each of the byte positions in the lost HP
io sub-packet. Thus, if processor 504 determines that the third packet is lost, "0"
bytes are inserted in each byte position from the 1301St through the 1950' bytes in the HP byte stream. Processor 504 passes the location of these missing bytes to RS decoder 506, which when sequentially presented with the bytes from the three received packets, recovers each missing byte in each byte location in is the third packet. Thus, RS decoder 504 is able to recover the entire sub-packet of HP data in the single lost third packet. If more than one packet is lost in this example, or more than h packets in the general case, then RS decoder is unable to recreate the missing data and a sequence error code that can be recognized by the data partitioning MPEG decoder 510 is inserted in the HP byte stream in 2o place of the recovered data.
Since RS decoder 506 outputs a k-byte word for each n-byte input word, in which each of the k bytes is associated with a different sub-packet, de-interleaver 507 re-sequences each k-byte output word, one byte at a time, into k separate HP sub-packets to reformulate the HP information in each transmitted Zs packet. The HP information in these k sub-packets that is re-created and, if necessary, recovered, is outputted by de-interleaver 507 and inputted on a first Boyce 5 24 input of the-data partitioning MPEG decoder 510. Where lost packets are unrecoverable, the data stream contains sequence error codes.
The LP information in the n LP sub-packets derived by de-packetizer 503 is simultaneously inputted on a second input of data partitioning MPEG decoder s 510. For those packets that are lost, the LP data cannot be recovered since forward error/erasure correction was not applied to the LP partition. For those packets, therefore, the LP output of de-packetizer 503 includes a codeword that will be recognized by the data partitioning MPEG decoder 510 as being indicative of missing data.
to The LP and HP data stream inputs to data partitioning MPEG decoder 510 are equivalent, in the absence of any lost packets, to the HP and LP data stream outputs of the data partitioning MPEG encoder 102 in FIG. 1. If up to and including h packets are lost, the HP data stream inputted to decoder 510 is equivalent to the HP data stream outputted by encoder 102 and the LP data is stream inputted to decoder 510 includes codewords marking lost data. When more than h packets are lost so that RS decoder 506 cannot recover the lost HP
data, both the HP and LP data stream inputted to data partitioning MPEG
decoder 510 have missing data, the HP data stream including sequence error codes indicating the absence of actual data and the LP data stream including the 2o recognizable codewords indicating lost data.
Data partitioning MPEG decoder 510, in response to the inputted LP
partition data and the HP partition data decompresses and reformulates the transmitted video data in accordance with its standardized algorithm. Where, with respect to specific pels in the frame, corresponding HP data is available (by 2s being actually received or recovered) but LP data is not available, the subjective video quality of the reconstructed video frame is degraded. Furthermore, that spatial portion of the reconstructed video frame that follows, in a scanning sense, Boyce 5 2s those pets +n the frame associated with the lost LP partition data, is also degraded since the LP partition data in the next received packet cannot be associated with specific spatial points within the frame and thus with the HP
data. As noted, only picture headers are included within both the HP and LP
s partitions. Thus, until the next picture header is received, all LP
partition data within the video frame that follows a lost packet will be unable to be combined with spatially corresponding HP data to decode and reconstruct the video signal.
Further, since the type of frames in this embodiment which are divided into separate HP and LP partitions are the P frames, which are used to predict the to next frame, the loss of LP data for reconstructing the remainder of the frame will have an effect on the quality of the next P and B frames, albeit at a substantially reduced level as compared to the effect that a total loss of data would have caused without the present invention in which the more important HP partition data has been protected. In the event that more than h packets are lost in is transmission, both HP and LP partition data is lost and not recoverable.
Standard error concealment techniques can then be used to minimize the degradation in video quality.
A decoder network 600 which is associated with the encoder network of FIG. 4 is illustrated in FIG. 6. In this embodiment of a decoder neiwork, a de-2o packetizer/decoder 601 receives from the UDP/IP network the data stream that was transmitted by encoder network of FIG. 4. De-packetizer/decoder 601 functions in a manner identical to that of de-packetizer/decoder 502 in FIG. 5 and includes the same elements as those described in connection with the description above of FIG. 5. The de-packetizer, interleaves, RS decoder, de-2s interleaves and processor will not, therefore, be discussed in detail. The outputs of de-packetizer/decoder 601 are the LP and HP partition data streams, as are the outputs of de-packetizer/decoder 502 in FIG. 5. In the absence of any lost ~ Boyce 5 26 packets in a frame, the LP and HP partitions are equivalent to the HP and LP
outputs of data splitter 402 in the encoder network of FIG. 4. If one through h packets are lost, then any lost HP data is recovered and the HP data stream is equivalent to the output of data splitter 402. The LI' stream outputted by de-s packetizer/decoder 601 includes, however, a codeword indicating that LP data has been lost. Where more than h packets are lost, then both the HP and LP
data streams include error codes. Alternatively, the de-packetizer/decoder 601 could send information to a data merger 602 indicating the positions of missing HP and LP information.
io The LP and HP data streams outputted by de-packetizer/decoder 601 are inputted to the data merger 602, which combines these data streams into a single data stream having a format which is decodable by a standard MPEG
decoder 603. MPEG decoder 603, using its standard algorithm, decompresses the coded video signal to reconstruct the digital video signal, which can then be is transformed to an analog video signal and displayed on a video terminal.
As earlier noted in connection with the description of the encoder in FIG. 4, slice headers are included in both the LP and HP partitions outputted by data splitter 402. Thus, the for decoder in FIG. 6, upon a loss of a packet and its LP data, a point of entry can be located for the LP data received in subsequent ao packets upon detection of the next slice header in such data. Thus, unlike the embodiment of the decoder in FIG. 5 in which, as noted, received LP data could only be reincorporated with HP data upon receipt of the next picture header for the next frame, the embodiment of FIG. 6 enables the LP data received in those packets following a lost packet to be incorporated with received or recovered HP
2s data at a point of entry at the next slice header. Therefore, only that portion of the frame that is associated with the lost LP data until the next received slice header will be degraded, rather than the entire remainder of the frame as in the Boyce 5 embodiment of FIG. 5. Further, by minimizing the visually degraded portion of the decoded frame, the degradation of the subsequent frames that may be predicted based on that frame is also minimized.
The preceding merely illustrates the principles of the invention. It will thus s be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.
Furthermore, all examples and conditional language that have been recited hereinabove are principally intended expressly to be only for pedagogical to purposes to aid the reader in understanding the principles of the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements that made hereinabove reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, Is are intended to encompass both structural and functional equivalents thereof.
Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the 2o block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer,or processor, whether or not such computer or processor is 2s explicitly shown.
The functions of the various elements shown in the FIGS. described above, including functional blocks labeled as "processors" may be provided - Boyce 5 through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which s may be shared. Moreover, explicit use of the term "processor" or "controller"
should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or io custom, may also be included. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
is In the claims hereof any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements which performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to 2o perform the function. The invention as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. Applicant thus regards any means which can provide those functionalities as equivalent to those shown herein.
MPEG VIDEO OVER THE INTERNET
Field of the Invention This invention relates to the transmission and reception of coded video s signals over the Internet, and more specifically, to the transmission and reception of video signals that have been coded using compression efficient inter-frame coding techniques such as those used in MPEG, MPEG-2, H.261, and H.263 standards.
Back4round of the Invention Io With the exploding popularity of the public Internet in the past several years for transporting all types of data, there has been much recent interest in transmitting digitally encoded real-time audio and video over the Internet using the Universal Datagram Protocol (UDP). Because UDP is an unreliable protocol, network packet losses will likely occur and, as a result, will adversely affect the is quality of the received audio and video. Recovery from packet losses may be performed solely by the receiver, or better quality can be achieved by involving both the sender and the receiver in the error recovery process. In networks that support prioritization, such as ATM, video quality can be improved in the Boyce 5 presence of packet loss by using scalable video coding (see, e.g., R. Aravind, M.
Civanlar, A. Reibman, "Packet Loss Resilience of MPEG-2 Scalable Video Coding Algorithms," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 6, No. 5, October 1996). There is currently, however, no s widespread support for prioritization on the public Internet. Overviews of proposed methods for error recovery for streaming of audio and video over the Internet, which involve both the sender and the receiver are disclosed by C. Perkins and O. Hodson in "Options for Repair of Streaming Media," Internet Engineering Task Force Internet RFC 2354, June 1987,and G. Carle and E.
io Biersack in "Survey of Error Recovery Techniques for IP-Based Audio-Visual Multicast Applications," IEEE Network, November/December 1997. While the general methods described in these overviews may be applicable to IP
transmission of both audio and video, most of the studies published where specific techniques have been implemented involve audio only. Because of its is higher data rates, and propagation of errors through inter-frame coding, it is more difficult to maintain video quality than audio, and audio techniques, therefore, cannot be directly applied to video signals.
Many of the currently popular schemes for transmitting digital video over the Internet, such as Motion-JPEG and wavelet-based schemes, use intra-frame 2o coding. Inter-frame coding techniques, such as those used in MPEG-1, MPEG-2, H.261, and H.263 standards, are generally more compression-efficient than intra-frame techniques. However, the inter-frame standards suffer more from Internet packet loss because errors in one frame may propagate for many frames. An MPEG video sequence includes intra-frame coded (I) frames, and 2s inter-frame predicted coded (P), and bi-directional inter-frame coded (B) frames.
I and P frames are used in the prediction of subsequent frames while B frames are not used in the prediction of subsequent frames. For example, consider an Royce 5 MPEG video sequence with I frames occurring every 15 frames. In MPEG
coding, because of inter-frame prediction, all predictive P and B frames rely upon the previous I frame. Thus, if an error occurs while transmitting the I frame, the effect persists for 15 frames, or 500 ms, which is quite noticeable to a viewer.
s The received video quality can be improved both through error concealment techniques that are applied at the decoder, and by error resilience techniques that are applied at the sender.
Error resilience techniques using Forward Error/Erasure Correction (FEC) add redundant data to a media stream prior to transmission, so that packet io losses can be repaired at the receiver without requiring contact with or re-transmissions from the sender. Forward Error/Erasure Correction techniques are well suited to multicast applications, because they avoid the use of re-transmissions. The same redundant data can be used to repair the loss of different packets at separate receivers in a multicast group. If re-transmission is were used instead, multiple re-transmission requests would have to be sent.
Forward Error/Erasure Correction techniques for multimedia generally fall into one of two categories, media-independent FEC and media-specific FEC ( see, e.g., C. Perkins and O. Hodson, "Options for Repair of Streaming Media,"
Internet Engineering Task Force Internet RFC 2354, June 1998).
2o In media-independent FEC, well-known information theory techniques for protecting any type of data are used. In, "Media-independent Error Correction using RTP,n Internet Engineering Task Force Internet Draft, May 1997 by D. Budge, R. McKenzie, W. Mills, and P. Long, several variations of exclusive-OR (XOR) operations are used to create parity packets from two or more data 2s packets. More complex techniques such as Reed Solomon (RS) coding can also be used (see, e.g., G. Carle and E. Biersack, "Survey of Error Recovery Techniques for IP-Based Audio-Visual Multicast Applications," IEEE Network, Boyce 5 Novemberfflecember 1997). Reed-Solomon encoding is an example of a systematic forward errorlerasure correction code. A systematic forward error/erasure correction code is one in which the information bytes are transmitted in the codeword without modification. Thus, in the absence of s channel errors, no Reed-Solomon decoding is necessary to recover the information bytes. When an RS(n,k) codeword is constructed from byte data, h parity bytes are created from k information bytes, and all n = k + h bytes are transmitted. Such a Reed Solomon decoder can correct up to any h/2 byte errors, or any h byte erasures, where an erasure is defined as an error in a Io known position. When RS coding is applied to protect packetized data against packet loss, k information packets of length j bytes are coded using j RS
codewords. For each RS codeword, k information bytes are taken from k different packets (one from each packet), and the h constructed parity bytes are placed into h separate parity packets, and all n = k + h packets are transmitted.
is Because the transmitted packets are numbered, and packets are assumed to be received perfectly or not at all, the receiver can determine which packets are missing, and thus a packet loss can be considered to be an erasure. Hence, if any h (or fewer) of the n transmitted packets are lost, the original k information packets can be recovered perfectly.
2o A key advantage of RS coding is its ability to protect against several consecutive errors, depending on the parameter choices. The overhead rate for RS coding is h/k, and it is most efficient for protection against burst errors for large values of k. For example, an RS(6,4) code and an RS(4,2) code both can protect against a burst length of 2 errors. But the RS(4,2) code has 100%
as overhead, while the RS(6,4) code has only 50% overhead. Reducing the overhead percentage by increasing the block length, however, leads to delay Boyce 5 because large block lengths require buffering of large amounts of data prior to transmission.
In media-specific FEC coding unlike in media-independent FEC coding where the multimedia stream is just treated as data, knowledge of the specific s type of multimedia stream to be transmitted is used. In "Simulation of FEC
Based Error Control for Packet Audio on the Internet," INFOCOM, March 1998, San Francisco, CA by M. Podolsky, C. Romer, and S. McCanne, and in 'Reliable Audio for Use over the Internet," Proc. INET '95, Honolulu, HI, pp. 171-178, June 1995, by V. Hardman, M.A. Sasse, M. Handley, and A. Watson. a redundant low-io bit rate audio stream is transmitted along with the standard audio stream, but delayed by one packet. If a standard audio packet is lost, the receiver uses the low-bit rate version of that audio instead, received in the next packet. This method protects against single packet losses.
In the aforenoted article by Perkins and Hodson, a suggestion is made to is combine media-specific and media-independent techniques by applying the media-independent FEC techniques to the most significant bytes of a coder's output, rather than applying FEC over the entire multimedia bitstream. No specific information about how this can be accomplished is given however.
A method for adding resilient information to inter-frame coded video, such as 2o MPEG video, in order to protect video quality against packet loss, but which has low overhead and low delay is desirable.
Summary of the Invention In accordance with the present invention, an inter-frame coded video 2s signal, such as an MPEG video signal, employs a data splitting function to split such a video stream into a high priority and a low priority partition.
Systematic Forward Error/Erasure Correction coding is then performed on only the data in Boyce 5 the high priority partition. The Forward Error/Erasure Corrected high priority partition data and the non-Forward Error/Erasure Corrected low priority partition data are then combined into packets and transmitted over the same network to a receiver, where they are decoded. Depending on the degree of protection s against errors or erasures offered by the particular FEC code that is used, the loss of one or more packets containing high priority data can be corrected with no loss of data in the high priority partition. The effect of the loss of the low partition data in the lost packet or packets, which low partition is not protected, has much less of a deleterious effect on the quality of the decoded video signal io than would the loss of data from the high priority partition data.
Advantageously, by limiting the application of the Forward Error/Erasure Correction to only the higher priority partition data, and thus protecting against loss only that "more important data", the overhead requirement is reduced for protection against a given packet loss.
is In the preferred embodiment, a Reed Solomon encoder is applied to the high priority data for an entire frame. For each RS(n,k) codeword, one information byte is taken from each of k packets and the constructed parity bytes are placed in h different packets, where n = k + h. Each individual frame's data is arranged in the n equal length packets that contain a combination of:
packet 2o headers; high priority data comprising one of either information bytes or parity bytes; and low priority data bytes, the latter comprising only information bytes since no error-correction coding is performed on the tow priority data. The same number of bytes of high priority data (information or parity in any one packet) are placed in each, of the n equal length packets, and the same number of bytes of 2s low priority data (information only) are placed in these same n packets, which together represent the video frame. Amongst these n equal length packets, k packets only contain high priority partition information bytes and h packets only i ' contain the high priority parity bytes. The parity byte in each high priority byte position in each of these h packets is formed from the RS(n,k) code as it is applied to the k high priority partition information bytes in a corresponding byte position in the k other high priority partition information-containing packets s associated with the frame. Advantageously, arranging the packets in this manner minimizes the amount of overhead and delay for a given packet loss protection rate.
A receiving decoder, upon receiving the packets associated with a frame separates the high priority partition bytes and low priority partition bytes in each ~o packet according to the numbers of such bytes or each type included within each packets, which numbers are transmitted in the packet headers. RS(n,k) decoding is applied byte position-by-byte position across the high priority partition portion within the received packets. If up to h of the n frame packets are lost, the RS decoding process recovers each high priority byte in the lost is packet or packets. Full reconstruction of the high priority partition information bytes that were transmitted in the k packets of the n packets that contained high priority partition data is thus effected. Although the low priority partition data in the lost packets is unrecoverable, the fully recovered high priority partition data enables the video picture to be decoded, albeit in what might be at a reduced 2o quality level for that frame or that portion of a frame in which only the high priority partition information is available.
In accordance with one aspect of the present invention there is provided a method of decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of 2s packets that are associated with at least one frame of the compression-coded video signal, the method comprising the steps of: receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes 7a associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted; determining which transmitted packets associated with the at least one frame of the video s sequence have not been received; applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the step of applying a FEC decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
io combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
is In accordance with another aspect of the present invention there is provided a decoder for decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the decoder comprising: a depacketizer connected to receive 2o packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into 2s low priority and high priority partitions before having been packetized and transmitted, the depacketizer separating the FEC-coded high priority partition data bytes and the low priority partition information bytes; a FEC decoder connected to receive and decode the FEC-coded high priority partition data bytes; and a video decompression decoder connected to receive the 7b depacketized low priority partition information bytes and the FEC-decoded high priority partition data bytes, the video decompression decoder combining and decompression-decoding the low priority partition information bytes and the FEC-decoded high priority partition data bytes.
s Brief Description of the Drawing FIG. I is a block diagram of first embodiment of a video encoder in accordance with the present invention which uses a data partitioning MPEG
encoder to code and split the coded video signal into HP and LP partitions;
Boyce 5 FIG:~2 the arrangement of HP data and parity information, and LP data within the packets associated with a frame for an example of RS(4,3) encoding;
FIG. 3 is a flow chart that details the steps for determining the n and k parameters associated with a frame as a function of the number of LP and HP
bytes in the frame;
FIG. 4 is a block diagram of a second embodiment of a video encoder in accordance with the present invention in which a standard MPEG encoder is used to encode the video signal and a data splitter divides the coded signal into HP and LP partitions;
to FIG. 5 is a block diagram of an embodiment of a video decoder in accordance with the present invention in which lost packets are reconstructed, HP and LP partitions reformed, and a data partitioning MPEG decoder is used to decode the video signal from the HP and LP partitions; and FIG. 6 is a block diagram of a second embodiment of a video decoder in is accordance with the present invention in which a data merger combines the HP
and LP partitions, which are then inputted to a standard MPEG decoder to decode the video signal.
Detailed Description 2o The MPEG-2 standard (ITU-T Recommendation H.262, "GENERIC
CODING OF MOVING PICTURES AND ASSOCIATED AUDIO INFORMATION", July 1995 ) includes several means for performing scalable video coding, including spatial scalability, SNR scalability, and data partitioning. In scalable video coding, receivers with different available bandwidths can receive and 2s decode appropriate representations of coded video, by receiving a base layer, and if bandwidth is available, receiving one or more enhancement layers. The more layers received, the higher the quality of the decoded video. In the Boyce 5 aforenoted-paper by Aravind, Civanlar and Reibman, data partitioning was applied to provide protection against packet loss for transmitting an MPEG-coded video signal in a network that supports prioritization such as ATM.
Specifically, protection against packet loss was shown to be achievable by s transmitting the base layer at a higher priority than the enhancement layer(s).
Since the public Internet does not currently support prioritization of packets this technique cannot be applied to the transmission of coded video over the Internet.
The inventor has recognized, however, that an advantage can be achieved by using, in a non-prioritized network, such as the public Internet, a Io data splitting functionality that is used in the prior art for packet protection over a network that does support prioritization. By incorporating a data splitting functionality together with a Forward Error/Erasure Correction functionality for transmitting an inter-frame compression-coded video signal over a non-prioritized public Internet, the amount of overhead needed for packet protection is is reduced, while achieving the improvement in the subjective video quality that packet protection affords. Further, by combining high priority and low priority data in the same packets, the delay for equal overhead and protection is advantageously reduced.
FIG. 1 is a first embodiment of an encoder in accordance with the present 2o invention. In this embodiment of the present invention, an encoder which is compliant with MPEG-standardized data partitioning is used to split an incoming video data stream into a high priority partition and a low priority partition.
With reference to FIG. 1, an incoming video data stream on 101 is inputted to a such a compliant data partitioning MPEG-standardized encoder 102 which, in 2s accordance with the standard, such as the MPEG-2 data partitioning standard, compression-codes the input video bitstream and splits the compression-coded bitstream into two output layers. The first layer is the base layer, referred to Boyce 5 10 herein as the high priority (HP) partition. The second layer is the enhancement layer, referred to herein as the low priority (LP) partition.
As is well known to one skilled in the art of MPEG coding, an MPEG-coded video bitstream includes headers at the sequence level, at the group-of s picture (GOP) level, at the picture (frame) level, and at the slice level.
As is well known, a slice consists of a group of adjacent macroblocks, where each macroblock in itself consists of a group of four adjacent luminance blocks of data and two chrominance blocks. At the picture level, a frame is classified as being as an intra-frame coded (I) frame, an inter-frame coded predictive (P) frame, or a to bi-directional inter-frame predictive (B) frame. At the macroblock level, for the predictive-type P and B type frames, information is included which indicates whether the macroblock is inter or intra coded with respect to another frame, as well as motion vector information. The information at the block level includes low frequency and high frequency discrete cosine transformation (DCT) coefficients is derived from actual pixel information.
In accordance with MPEG standards, data partitioning can be effected at a plurality of different priority breakpoints, such as above the macroblock layer, at the macroblock layer, or within the macroblock layer. Generally, the more critical parts of the coded bitstream, such as the headers, the motion vectors and 2o the low frequency DCT coefficients, are allocated to the HP partition, and the less critical data, such as the higher frequency DCT coefficients, are allocated to the LP partition. In accordance with the standard, a priority breakpoint can be chosen for each slice, which determines which codeword types are arranged into which partition.
2s In the embodiment of the invention in FIG. 1, which uses the standardized data-partitioning MPEG encoder 102, priority breakpoints are determined in accordance with the type of frame (1, P, or B) in the data stream that is being BOyCe 5 11 partitioned:- Specifically, in this embodiment, for each I frame, all the data is placed in the HP partition, with no data being placed in the LP partition.
Other embodiments may divide the data from each I frame between the HP and LP
partitions. For each B frame, as much frame data as the MPEG standards s relating to data partitioning allows is placed in the LP partition, with the remaining data placed in the HP partition. For each P frame, frame data is divided between the HP and LP partitions so that data elements through the first two DCT
coefficients of each block are placed in the HP partition, with the higher order DCT coefficients placed in the LP partition. Different priority breakpoints in the to three frame types between the LP and HP partitions other than described above for this embodiment could equally be used. It would be expected, however, that a higher breakpoint would be used for I frames than for P frames, which, in turn, would be higher than the breakpoint for B frames. In this standards-compliant embodiment, sequence, GOP, and picture headers are copied into both is partitions for error resilience.
In accordance with the invention, the HP partition is encoded with a systematic Forward Error/Erasure Correction code. Specifically, in this preferred embodiment, Reed Solomon coding is applied to the HP data for a single frame across the entire frame. For this embodiment, only data from a single frame is 20 operated upon by a Forward Error/Erasure Correction encoder/packetizer 103 at a time to minimize delay that would be incurred if data from multiple frames needed to be accumulated before being coded and transmitted. In other embodiments, B frames as well as their immediately following anchor frame (I
or P) may be operated upon together by the Forward Error/Erasure Correction 2s encoder/packetizer 103.
Reed Solomon encoding is applied to the HP data output of the MPEG
encoder 102 for the frame. As will be described in detail hereinafter, the number Boyce 5 t2 of bytes in the HP partition and the number of bytes in the LP partition are used together and separately, together with a maximum allowable packet length and a desired degree of packet protection, to determine the number of equal-length packets, n, into which, for that frame, the HP information bytes, the LP
s information bytes, and the protective HP parity bytes will fit. In accordance with the invention, each of the n equal-length packets contains a combination of both LP information bytes, and one of HP information and parity bytes. The parity bytes are determined by forming an RS(n, k) codeword by taking one HP
information byte from each of k packets that contain the HP information bytes to and placing the h parity bytes constructed from these k bytes, one byte at a time, into h different packets, where n = k + h. Thus, the parity bytes in those h packets in byte position m (where m varies between 1 and the number of HP
information bytes in a packet) are derived from the HP information bytes in that same byte position m in the k packets. Since the parity bytes in each byte is position in those h packets are calculated from the HP information bytes in the same byte positions in the other k packets, each equal-length packet contains an equal number of HP data bytes, the latter being either all information bytes or all parity bytes. It follows, therefore, that each equal-length packet then contains an equal number of LP information bytes. Because the number of HP information 2o bytes in the frame is not necessarily divisible by the number of k packets, padding bytes are added, as necessary, at the end of the kU' packet.
Similarly, padding bytes may be applied to the n~' packet for LP information bytes.
FIG. 2 shows an example of an arrangement of a frame group of packets for an RS (4, 3) code. As can be noted, three packets (k = 3) contain HP
2s information bytes information bytes and LP information bytes. One packet (h = 1) contains the HP parity bytes and the LP information bytes. Arranging the packets in the manner illustrated by this figure minimizes the amount of Royce 5 13 overhead for a given packet loss protection rate without adding to delay. The packet header in each packet contains the packet number, the number of the packets in the frame group of packets (n), the number of packets with parity data in the frame group of packets (h), the temporal reference value for the frame, the s frame type (I, P, or B), the HP/LP priority breakpoints, and the number of HP and LP bytes in each packet.
With reference back to FIG. 1, the output of the data partitioning MPEG
encoder 102 consists, for an input stream of video data associated with a frame, of two data streams, an HP partition data stream on output 104 and a LP
io partition data stream on output 105, both of which are inputted to the FEC
Coder/Packetizer 103. The number of bytes in the HP partition of the input frame, HP#, and the number of bytes in the LP partition of the input frame, LP#, are inputted to a processor 106 within coder/packetizer 103. As will be described in detail hereinafter, processor 106, using the values HP# and LP#, is together with a predetermined maximum number of bytes per frame, determines the number of packets, n, needed to transmit the entire frame, including packet headers, the HP partition information bytes, the HP partition parity bytes, and the LP partition information bytes. Further, processor 106 determines the number of packets, h (equal to n-k), which are needed to protect the HP partition ao information bytes in the other k packets for a desired level of protection.
Processor 106 also determines the byte-length of each of the n packets.
Processor 106 then determines the numbers of HP data or parity bytes and LP
data bytes to be apportioned into each of these n packets, with LP information bytes allocated to each of the n packets, HP information bytes being allocated to 2s the k packets, and HP parity bytes being allocated to the other h packets, as per the example shown in FIG. 2.
Royce 5 is Once processor 106 has determined all the parameters that determine the configuration of each packet and the number of such packets, a Reed Solomon RS(n,k) encoding is effected, byte by byte on the bytes within the HP
partition.
Specifically, the HP partition video stream is inputted to a byte interleaver 107, s which in response to the parameters determined by processor 106, forms sequential codewords that are inputted to a Reed Solomon encoder 108. For example, if the number of HP bytes per packet are calculated to be 650, the 1$', 651't, 1301, et seq., bytes in the HP stream are interleaved by interleaver 107 to form an input word to RS encoder 108 in order to determine the bytes for the first Io parity byte position for the h packets which are to contain HP parity bytes. The 2nd, 652"°, 1302"x, et seq., bytes are then interleaved to form the next input word to RS encoder 108 to determine the parity bytes for the second parity byte position for the h HP parity packets. In such a manner, the RS encoder 108 is sequentially presented with a series of k byte input words, each byte in each k is byte input word comprising one information byte from what will be, when re-interleaved, k different packets. For each k byte input word, h parity bytes are determined. Thus, for each k-byte input word, RS encoder 108 outputs an n-byte word containing k information bytes and h parity bytes.
In order to reassemble these vertically oriented codewords, each 2o containing bytes properly belonging in n different packets, back into a row-oriented packet format, a de-interleaver 109 sequentially places each byte from each n-byte output word into a separate one of n different rows. The first k of such n rows thus contain all HP information bytes and the last h of such rows contain all HP parity bytes. When the n rows are reconstructed, each such row 2s is combined by a packetizer 110 with a packet header and the calculated fixed number of LP partition information bytes apportioned to each packet from the LP
output 105 of the data partitioning MPEG encoder 102. Each packet, then in a Boyce 5 1 s format as shown in FIG. 2 for the illustrative example for an RS(4,3) code, is sequentially outputted on output 111 for transmission onto a UDP/IP network.
For the example of FIG. 2, if none of the packets containing the frame information are lost in transmission, the decoded video will be perfectly decoded s (i.e., identical to that encoded). If one of the packets (or up to h packets in the general case for R(n,k) coding) is lost, then all of the HP information in that packet is recoverable using a Reed Solomon decoder. The LP information data in the lost packet could not, however, be recovered since it is not protected.
The portions of the picture that correspond to the macroblocks whose LP data was to received will, however, be perfectly decoded, and those that correspond to macroblocks whose LP data is lost will decode only the HP data for those macroblocks. Those macroblocks which are decoded using only HP data may be noticeable to a viewer, but are unlikely to be visually offensive as long as the HP/LP priority breakpoint is properly chosen. The exact visual quality of is macroblocks decoded using only HP data depends on this HP/LP priority breakpoint, as well as the characteristics of the video source material. In the embodiment of FIG. 1 in which a standardized data partitioning MPEG encoder 102 is used to compression-code the input video stream and form the HP and LP
partitions, the lowest level header copied into both partitions is the picture 2o header. Thus, once any packet is lost, and with it the LP partition information contained therein, which cannot be recovered, the LP partition data that is received in the next received packets cannot be properly incorporated into the decoding process since there is no identifiable spatial entry point with which that received data can be associated until a next picture header is received. Thus, 2s with the embodiment of FIG. 1, the perceptual effect of lost LP partition information will extend until a packet is received that contains a next picture header within the received LP partition information. Once that picture header is Boyce 5 16 received, the LP partition data that follows can be properly incorporated with the received HP partition data.
As previously noted, the exact arrangement of the frame data into packets is a function of the number of bytes of HP partition information and LP
partition s information in the frame, the parameters of the RS(n,k) code, and the maximum packet size. For Internet Protocol (1P) transmission, the maximum packet size is set, for this embodiment, to the Ethernet Maximum Transport Unit (MTU) size of 1500 bytes. The choice of the RS(n,k) code parameters depends on the number of packets needed to transmit the frame, on the network loss conditions, and the to overhead percentage considered acceptable. Because Internet packet loss has a tendency to be somewhat bursty, the k/n ratio can be chosen to be lower for lower values of n and higher for higher values of n. Table 1 is an example of pairs of n and k values in which, based on the bursty nature of packet loss, the ratio kln is set lower for lower values of n and higher for higher values of n.
Boyce 5 Table 1 n k Given a list of (n,k) values such as that given in Table 1, the particular s (n,k) pair to use for each frame can be determined using an iterative process.
FIG. 3 is a flowchart that details the steps that processor 106 follows for determining the these parameters. After the data partitioning MPEG Encoder 102 splits the bitstream into the HP partition and the LP partition, there are HP#
bytes in the HP partition and LP# bytes in the LP partition. At step 301, an initial to estimate is formed for the number of packets, n, needed to transmit this information, assuming a maximum packet size PSmax , which in this embodiment Boyce 5 1 g is equal to the Ethernet MTU size minus the number of packet header bytes used. This initial estimate of n is formed from n = cell (HB# + LB#)lPSmeX, where ceil () is the ceiling function. This initial estimate of n is calculated without consideration of the HP parity bytes, which must be incorporated into the frame.
s The value of k that corresponds to the current estimate of n is retrieved, at step 302, using Table 1. At step 303, the necessary packet size for the HB#
information bytes is calculated from CurPacketSize = ceil (HB#lk) + ceil (LB#ln), because the HB# information bytes are split among k packets (with parity bytes in the remaining h packets) and the LB# information bytes are split among n io packets. At step 304, CurPacketSize is compared with PSm~ . If CurPacketSize is smaller than PSmaX , the current (n,k) parameters may be used. If not, the packet size is too large, and n is incremented at step 305. Table 1 may yield a new k for this incremented value of n. CurPacketSize is then recalculated using these new parameters. The process ends at step 306 when the resultant is parameters yields a CurPacketSize of less than PSmax . The actual number, HBP, of HP bytes (information or parity) in each of the n frames is then equal to ceil (HB#lk) and the actual number, LPB, of LP information bytes in each of the n frames is equal to ceil (LP#ln), the total number of bytes, less the length of the packet header, in each frame, TPB, thus being equal to HBP+LPB.
2o As an illustrative example, a P frame will be assumed to have a size of 5632 bytes, in which HB# = 2129 HP bytes and LB# = 3503 LP bytes. PSmax is assumed to be equal to 1475 bytes, derived from a 1500 byte MTU size minus 25 bytes for a packet header. The initial estimate of the number of packets is n=ceil ((2129+3503)/1475) = ceil (3.818) = 4. From Table 1, for n=4, it is seen Zs that k=3. Using these parameters, HPB, LPB and TPB are calculated:
HPB=ceil (2129/3)=710; LPB=ceil (3503/4)=876; and TPB= 710+876=1586, which is greater than PSmax of 1475. Thus, that set of parameters is not valid and Boyce 5 19 another iteration is necessary. For this next iteration, n is incremented to 5.
From Table 1, for n=5, it is seen that k=4. Using these parameters, HPB, LPB
and TPB are calculated: HPB=ceil(2129/4)=533; LPB=ceil(3503/5)=701; and TPB=533+701=1234, which is less than PSmaX of 1475.
s Once the parameters of n, k, HPB and LBP are determined for the frame, the aforedescribed Reed Solomon coding and arrangement of the HP
information and parity bytes, and LP information bytes into packets can be effected. Specifically, once processor 106 determines these parameters from the partitioned HP and LP data for each frame, that frame's data is interleaved, io RS encoded, de-interleaved and packetized for transmission over a UDP/IP
network, as previously described.
In the embodiment of FIG. 1, a data partitioning MPEG encoder 102 was used to split the bitstream into the HP and the LP partitions. FIG. 4 is a block diagram of an alternative embodiment which although not maintaining strict is MPEG compliance, in order to reduce overhead and improve performance, can still be used with standards compliant MPEG encoders and decoders. This embodiment may applied to MPEG-1 or MPEG-2 video, as well as other similar video coding standards such as H.261 and H.263. In this embodiment a standard MPEG encoder (or other standard encoder) 401 compression codes Zo the input video stream. A data splitter 402 then divides the compression-coded output of encoder 401 into HP and LP partitions. For purposes of the present invention, the combination of the standard MPEG encoder 401 with a data splitter 402 enables the splitting function to be performed more efficiently and with improved performance as compared to the data partitioning MPEG encoder 2s 102 in FIG. 1. Specifically, only slice headers are duplicated in both partitions and all other data is placed in one partition or the other, but not both.
Because of the frame alignment of packets and the containment of frame information in Boyce 5 20 the packet-headers themselves, it is not necessary to duplicate the picture headers and above in each of the partitions, thereby minimizing overhead. By providing slice headers in both the LP and HP partitions, however, a point of entry is provided for the LP data upon a packet loss. Thus, in a received packet s that follows a lost packet, the LP data can be incorporated into the decoded picture after the decoder receives, with the LP partition data, the next slice header. Advantageously, as compared the embodiment of FIG. 1, in which the point of insertion for the received LP data following a packet loss was the next frame, the embodiment of FIG. 4 provides a point of insertion for the LP data at io the next slice, thereby improving the picture quality of the decoded video signal.
As previously described, the data partitioning MPEG encoder 102 partitions I frames so that all frame data is placed in the HP partition to minimize the effect of an error in an I frame from propagating to other frames. Since B
frames are not used for prediction, as much data as is possible via the data is partitioning standard is placed in the LP partition. In the embodiment of FIG. 4, all data for all B frames is placed in the LP partition as opposed to the embodiment of FIG. 1 in which some data, in accordance with the standards, is required in the HP partition. In both the embodiment of FIG. 1 and FIG. 4, the P
frame data is divided between the HP and LP partitions in the manner described 2o above. Within each P frame macroblocks can be either inter-frame or intra-frame coded. In the embodiment of FIG. 4, different priority breakpoints are chosen for inter and intra coded macroblocks in each P frame, which the data partitioning MPEG encoder 102 in FIG. 1 could not do. Inter-coded macroblocks decoded using only the HP partition (and not the LP partition) may retain high 2s frequency information from the corresponding motion-compensated macroblocks in the previous frame, while intra-coded macroblocks may not. Hence, it is desirable to set the priority breakpoint for intra-coded macroblocks to include Boyce 5 21 more DCT coefficients in the HP partition than are included in inter-coded macroblocks. Advantageously, this reduces the overhead rate for given level of quality or improves the quality for the same overhead rate.
The HP and LP outputs of data splitter 402 are applied to a FEC
s coder /packetizer 403, which functions in the same manner as FEC
coder/packetizer does in FIG. 1, which functions have been described above.
The packets outputed by FEC codeNpacketizer 403 are then transmitted over a UDPIIP network.
Although the embodiment in FIG. 4 includes a separate standard MPEG
io encoder 401 and data splitter 402, it should be recognized that a data coder that supports the described data splitting operation could equally perform the same functions that encoder 401 and splitter 402 together do.
The encoder networks in FIG. 1 and FIG. 4 transmit output packets over a UDP/IP network such as the public Internet, to corresponding decoders is connected to that network. In the case of a multicasting encoder, the transmission may be simultaneously broadcast to a plurality of end-users who will each receive the transmitted information. Since different packets may be lost along the routes to different of such end-users, the FEC techniques employed in the present invention to protect against packet loss enable each individual 2o decoder to recover the particular packets that may have been lost on the path to that end-user up to, of course, the level of packet protection provided by the specific RS code used by the encoder.
A decoder network 500 that is associated with the encoder network of FIG. 1 is illustrated in FIG. 5. As a data partitioning MPEG encoder 102 is 2s incorporated within the encoder network of FIG. 1, the decoder network of FIG. 5 incorporates a complimentary data partitioning MPEG decoder 510. In FIG. 5, the serial packets transmitted by the encoder network over the UDP/IP network Boyce 5 22 501 are inputted to a de-packetizeNdecoder 502. De-packetizeNdecoder 502 includes a de-packetizer 503 which receives the serial packets and strips the header information in each packet and passes it to a processor 504. The packet header information includes: a packet number; a frame number; the type of s frame (I, B, P); the (n,k) parameters which define the packet structure of the frame; and the number of HP partition bytes, HPB, and LP partition bytes, LPB, in each packet. Processor 504, from the header information, determines the start of the frame, and from the parameter n, "knows" that that many packets are used to define the frame. Further, from the packet numbers received, processor to 504 determines which particular of those n packets, if any, are missing and their position within the sequence of these n packets. In response to receiving all such information from processor 504, de-packetizer 503 strips off the packet header of each packet within the frame and divides the data in each packet into its HP and LP portions. For those packets which are determined to be missing, Is de-packetizer 503 inserts "0" bytes or an error code in the HP and LP data streams. The HP serial byte stream output of de-packetizer 503 consists of n sub-packets-worth of HP data, each sub-packet containing HPB bytes. This HP
stream is inputted to an interleaver 505 to decode the RS(n,k) encoded words that exist across the sub-packets and to replace any missing data from up to h 20 lost packets. Thus, for each byte position across the sub-packet, a byte is selected from each such sub-packet at that byte position to form an input word to an RS decoder 506.
As an example, using the frame structure shown in FIG. 2 where n is equal to 4 and k is equal to 3, and with HPB being equal 650 bytes, interleaver 2s 505 selects the 1 St, 651 St, 1301 S' and 1951 S' bytes in the HP byte stream to form a 4-byte input word to RS decoder 506 to determine the 3-byte corresponding output word, each byte in the three-byte output word being an HP information Boyce 5 23 byte in the #irst byte position in each of the three information sub-packets.
Interleaver 505 then presents the 2"d, 652"d, 1302"d and 1952"d bytes in the HP
byte stream to RS decoder 506 to determine the bytes in the second byte position in each of the three information sub-packets. Interleaver 505 similarly s processes the 3~ through the 650'" bytes. The three information bytes for each byte position are thus sequentially outputted by RS decoder 506. Since the RS(4,3) code is capable of correcting up to one erasure, if one of the four packets is lost and thus not received by the decoder network 500, RS decoder 506 will determine the missing byte in each of the byte positions in the lost HP
io sub-packet. Thus, if processor 504 determines that the third packet is lost, "0"
bytes are inserted in each byte position from the 1301St through the 1950' bytes in the HP byte stream. Processor 504 passes the location of these missing bytes to RS decoder 506, which when sequentially presented with the bytes from the three received packets, recovers each missing byte in each byte location in is the third packet. Thus, RS decoder 504 is able to recover the entire sub-packet of HP data in the single lost third packet. If more than one packet is lost in this example, or more than h packets in the general case, then RS decoder is unable to recreate the missing data and a sequence error code that can be recognized by the data partitioning MPEG decoder 510 is inserted in the HP byte stream in 2o place of the recovered data.
Since RS decoder 506 outputs a k-byte word for each n-byte input word, in which each of the k bytes is associated with a different sub-packet, de-interleaver 507 re-sequences each k-byte output word, one byte at a time, into k separate HP sub-packets to reformulate the HP information in each transmitted Zs packet. The HP information in these k sub-packets that is re-created and, if necessary, recovered, is outputted by de-interleaver 507 and inputted on a first Boyce 5 24 input of the-data partitioning MPEG decoder 510. Where lost packets are unrecoverable, the data stream contains sequence error codes.
The LP information in the n LP sub-packets derived by de-packetizer 503 is simultaneously inputted on a second input of data partitioning MPEG decoder s 510. For those packets that are lost, the LP data cannot be recovered since forward error/erasure correction was not applied to the LP partition. For those packets, therefore, the LP output of de-packetizer 503 includes a codeword that will be recognized by the data partitioning MPEG decoder 510 as being indicative of missing data.
to The LP and HP data stream inputs to data partitioning MPEG decoder 510 are equivalent, in the absence of any lost packets, to the HP and LP data stream outputs of the data partitioning MPEG encoder 102 in FIG. 1. If up to and including h packets are lost, the HP data stream inputted to decoder 510 is equivalent to the HP data stream outputted by encoder 102 and the LP data is stream inputted to decoder 510 includes codewords marking lost data. When more than h packets are lost so that RS decoder 506 cannot recover the lost HP
data, both the HP and LP data stream inputted to data partitioning MPEG
decoder 510 have missing data, the HP data stream including sequence error codes indicating the absence of actual data and the LP data stream including the 2o recognizable codewords indicating lost data.
Data partitioning MPEG decoder 510, in response to the inputted LP
partition data and the HP partition data decompresses and reformulates the transmitted video data in accordance with its standardized algorithm. Where, with respect to specific pels in the frame, corresponding HP data is available (by 2s being actually received or recovered) but LP data is not available, the subjective video quality of the reconstructed video frame is degraded. Furthermore, that spatial portion of the reconstructed video frame that follows, in a scanning sense, Boyce 5 2s those pets +n the frame associated with the lost LP partition data, is also degraded since the LP partition data in the next received packet cannot be associated with specific spatial points within the frame and thus with the HP
data. As noted, only picture headers are included within both the HP and LP
s partitions. Thus, until the next picture header is received, all LP
partition data within the video frame that follows a lost packet will be unable to be combined with spatially corresponding HP data to decode and reconstruct the video signal.
Further, since the type of frames in this embodiment which are divided into separate HP and LP partitions are the P frames, which are used to predict the to next frame, the loss of LP data for reconstructing the remainder of the frame will have an effect on the quality of the next P and B frames, albeit at a substantially reduced level as compared to the effect that a total loss of data would have caused without the present invention in which the more important HP partition data has been protected. In the event that more than h packets are lost in is transmission, both HP and LP partition data is lost and not recoverable.
Standard error concealment techniques can then be used to minimize the degradation in video quality.
A decoder network 600 which is associated with the encoder network of FIG. 4 is illustrated in FIG. 6. In this embodiment of a decoder neiwork, a de-2o packetizer/decoder 601 receives from the UDP/IP network the data stream that was transmitted by encoder network of FIG. 4. De-packetizer/decoder 601 functions in a manner identical to that of de-packetizer/decoder 502 in FIG. 5 and includes the same elements as those described in connection with the description above of FIG. 5. The de-packetizer, interleaves, RS decoder, de-2s interleaves and processor will not, therefore, be discussed in detail. The outputs of de-packetizer/decoder 601 are the LP and HP partition data streams, as are the outputs of de-packetizer/decoder 502 in FIG. 5. In the absence of any lost ~ Boyce 5 26 packets in a frame, the LP and HP partitions are equivalent to the HP and LP
outputs of data splitter 402 in the encoder network of FIG. 4. If one through h packets are lost, then any lost HP data is recovered and the HP data stream is equivalent to the output of data splitter 402. The LI' stream outputted by de-s packetizer/decoder 601 includes, however, a codeword indicating that LP data has been lost. Where more than h packets are lost, then both the HP and LP
data streams include error codes. Alternatively, the de-packetizer/decoder 601 could send information to a data merger 602 indicating the positions of missing HP and LP information.
io The LP and HP data streams outputted by de-packetizer/decoder 601 are inputted to the data merger 602, which combines these data streams into a single data stream having a format which is decodable by a standard MPEG
decoder 603. MPEG decoder 603, using its standard algorithm, decompresses the coded video signal to reconstruct the digital video signal, which can then be is transformed to an analog video signal and displayed on a video terminal.
As earlier noted in connection with the description of the encoder in FIG. 4, slice headers are included in both the LP and HP partitions outputted by data splitter 402. Thus, the for decoder in FIG. 6, upon a loss of a packet and its LP data, a point of entry can be located for the LP data received in subsequent ao packets upon detection of the next slice header in such data. Thus, unlike the embodiment of the decoder in FIG. 5 in which, as noted, received LP data could only be reincorporated with HP data upon receipt of the next picture header for the next frame, the embodiment of FIG. 6 enables the LP data received in those packets following a lost packet to be incorporated with received or recovered HP
2s data at a point of entry at the next slice header. Therefore, only that portion of the frame that is associated with the lost LP data until the next received slice header will be degraded, rather than the entire remainder of the frame as in the Boyce 5 embodiment of FIG. 5. Further, by minimizing the visually degraded portion of the decoded frame, the degradation of the subsequent frames that may be predicted based on that frame is also minimized.
The preceding merely illustrates the principles of the invention. It will thus s be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.
Furthermore, all examples and conditional language that have been recited hereinabove are principally intended expressly to be only for pedagogical to purposes to aid the reader in understanding the principles of the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements that made hereinabove reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, Is are intended to encompass both structural and functional equivalents thereof.
Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the 2o block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer,or processor, whether or not such computer or processor is 2s explicitly shown.
The functions of the various elements shown in the FIGS. described above, including functional blocks labeled as "processors" may be provided - Boyce 5 through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which s may be shared. Moreover, explicit use of the term "processor" or "controller"
should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or io custom, may also be included. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
is In the claims hereof any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements which performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to 2o perform the function. The invention as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. Applicant thus regards any means which can provide those functionalities as equivalent to those shown herein.
Claims (52)
1. A method of decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the method comprising the steps of:
receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted;
determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the step of applying a FEC
decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted;
determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the step of applying a FEC
decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
2. The method of claim 1 wherein the packet-based network is the Internet.
3. The method of claim 1 wherein the received FEC-coded high priority partition data bytes are coded with a systematic forward error/erasure correction code, the received FEC-coded high priority partition data bytes comprising a combination of the high priority partition information bytes associated with the at least one frame and associated parity bytes.
4. The method claim 3 wherein each received packet has an equal length, the high priority partition data bytes in each received packet comprising either all high priority partition information bytes associated with the at least one frame or all parity bytes.
5. The method of claim 4 wherein an equal number of low priority partition information bytes are in each received packet and an equal number of high priority partition data bytes are in each received packet.
6. The method of claim 5 wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, are associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the FEC decoding being applied to the high priority partition information bytes and associated at least one parity byte at each high priority byte position, one byte position at a time, to determine high priority partition information bytes in each high priority byte position in a packet that was not received.
7. The method of claim 6 wherein the FEC-coded bytes are coded with a Reed Solomon code and the FEC decoding uses Reed Solomon decoding.
8. The method of claim 6 further comprising the steps of:
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
9. The method of claim 8 wherein the FEC decoding uses Reed Solomon (n, k) decoding.
10. The method of claim 8 wherein the received packets associated with the at least one video frame of the video signal are protected against a packet loss over the packet-based network of high priority partition information bytes in up to (n-k) packets.
11. The method of claim 6 wherein the step of decompression decoding the video signal uses MPEG decoding.
12. A decoder for decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the decoder comprising:
a depacketizer connected to receive packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before having been packetized and transmitted, the depacketizer separating the FEC-coded high priority partition data bytes and the low priority partition information bytes;
a FEC decoder connected to receive and decode the FEC-coded high priority partition data bytes; and a video decompression decoder connected to receive the depacketized low priority partition information bytes and the FEC-decoded high priority partition data bytes, the video decompression decoder combining and decompression-decoding the low priority partition information bytes and the FEC-decoded high priority partition data bytes.
a depacketizer connected to receive packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before having been packetized and transmitted, the depacketizer separating the FEC-coded high priority partition data bytes and the low priority partition information bytes;
a FEC decoder connected to receive and decode the FEC-coded high priority partition data bytes; and a video decompression decoder connected to receive the depacketized low priority partition information bytes and the FEC-decoded high priority partition data bytes, the video decompression decoder combining and decompression-decoding the low priority partition information bytes and the FEC-decoded high priority partition data bytes.
13. The decoder of claim 12 wherein the packet-based network is the Internet.
14. The decoder of claim 12 wherein the FEC-coded high priority partition data bytes are coded with a systematic FEC code, the received FEC-coded high priority partition data bytes comprising a combination of the high priority partition information bytes associated with the at least one frame and associated parity bytes.
15. The decoder of claim 14 wherein each received packet has an equal length, the high priority partition data bytes in each received packet comprising either all high priority partition information bytes associated with the at least one frame or all parity bytes.
16. The decoder of claim 15 wherein an equal number of low priority partition information bytes are in each received packet and an equal number of high priority data bytes are in each received packet.
17. The decoder of claim 16 wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, are associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the FEC decoder decoding the high priority partition information bytes and associated at least one parity byte at each high priority byte position, one byte position at a time, to determine high priority partition information bytes in each high priority byte position in a packet that was not received.
18. The decoder of claim 17 wherein the FEC-coded bytes are coded using a Reed Solomon code and the FEC decoding uses Reed Solomon decoding.
19. The decoder of claim 17 further comprising a processor, the processor:
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
20. The decoder of claim 19 wherein the FEC decoding uses Reed Solomon (n,k) decoding.
21. The decoder of claim 19 wherein the received packets associated with the at least one video frame of the video signal are protected against a packet loss over the packet-based network of high priority partition information bytes in up to (n-k) packets.
22. The decoder of claim 17 wherein the video decompression decoder uses MPEG decoding.
23. A decoder for decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the decoder comprising:
means for receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted;
means for determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
means for applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the means for applying a FEC decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
means for combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes;
and means for decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
means for receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both low priority partition information bytes and forward error/erasure correction (FEC)-coded high priority partition data bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into low priority and high priority partitions before being packetized and transmitted;
means for determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
means for applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, the means for applying a FEC decoding reconstructing high priority partition information bytes in those packets determined not to have been received;
means for combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes;
and means for decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reproduce the at least one frame of the video signal.
24. The decoder of claim 23 wherein the packet-based network is the Internet.
25. The decoder of claim 23 wherein the received FEC-coded high priority partition data bytes are coded with a systematic forward error/erasure correction code, the received FEC-coded high priority partition data byte comprising a combination of the high priority partition information bytes associated with the at least one frame and associated parity bytes.
26. The decoder of claim 25 wherein each received pack has an equal length, the high priority partition data bytes in each received packet comprising either all high priority partition information bytes associated with the at least one frame or all parity bytes.
27. The decoder of claim 26 wherein an equal number of low priority partition information bytes are in each received packet and an equal number of high priority partition data bytes are in each received packet.
28. The decoder of claim 27 wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, are associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the FEC decoding being applied to the high priority partition information bytes and associated at least one parity byte at each high priority byte position, one byte position at a time, to determine high priority partition information bytes in each high priority byte position in a packet that was not received.
29. The decoder of claim 28 wherein the FEC-coded bytes are coded with a Reed Solomon code and the FEC decoding uses Reed Solomon decoding.
30. The decoder of claim 28 further comprising processing means for:
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition Information bytes.
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition Information bytes.
31. The decoder of claim 30 wherein the forward error/erasure correction decoding uses Reed Solomon (n,k) decoding.
32. The decoder of claim 30 wherein the received packets associated with the at least one video frame of the video signal are protected against a packet loss over the packet-based network of high priority partition information bytes in up to (n-k) packets.
33. The decoder of claim 28 wherein the means for decompression decoding the video signal uses MPEG decoding.
34. A method of decoding a compression-coded video signal that has been packetized and transmitted over a packet-based network in a sequence of packets that are associated with at least one frame of the compression-coded video signal, the method comprising the steps of:
receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both forward error/erasure correction (FEC)-coded high priority partition data bytes that have been coded with a systematic FEC code, and low priority partition information bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into high priority and low priority partitions before having been packetized and transmitted;
determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, area associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the step of applying a FEC decoding further comprising applying a FEC decoding to the high priority partition information bytes and associated at least one parity byte at each high priority byte position, one byte position at a time, to reconstruct high priority partition information bytes in each high priority byte position in a packet that was determined not have been received;
combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reform the at least one frame of the video signal.
receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both forward error/erasure correction (FEC)-coded high priority partition data bytes that have been coded with a systematic FEC code, and low priority partition information bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into high priority and low priority partitions before having been packetized and transmitted;
determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, area associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the step of applying a FEC decoding further comprising applying a FEC decoding to the high priority partition information bytes and associated at least one parity byte at each high priority byte position, one byte position at a time, to reconstruct high priority partition information bytes in each high priority byte position in a packet that was determined not have been received;
combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes; and decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reform the at least one frame of the video signal.
35. The method of claim 34 wherein each packet has an equal length and an equal number of low priority partition information bytes are in each packet and an equal number of high priority partition information bytes or parity bytes are in each packet.
36. The method of claim 35 wherein the FEC-coded bytes are coded with a Reed Solomon code and the FEC decoder uses Reed Solomon decoding.
37 37. The method of claim 35 further comprising the steps of:
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
38. The method of claim 37 wherein the FEC decoding uses Reed Solomon (n,k) decoding.
39. The method of claim 37 wherein the received packets associated with the at least one video frame of the video signal are protected against a packet loss over the packet-based network of high priority partition information bytes in up to (n-k) packets.
40. The method of claim 34 wherein the step of decompression coding the video signal uses MPEG decoding.
41. A decoder for decoding a video signal for transmission over a packet-based network comprising:
means for receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both forward error/erasure correction (FEC)-coded high priority partition data bytes that have been coded with a systematic FEC code, and low priority partition information bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into high priority and low priority partitions before having been packetized and transmitted;
means for determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
means for applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, are associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the means for applying a FEC
decoding applying a FEC decoding to the high priority partition information bytes and associated at east one parity byte at each high priority byte position, one byte position at a time, to reconstruct high priority partition information bytes in each high priority byte position in a packet that was determined not have been received;
means for combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes;
and means for decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reform the at least one frame of the video signal.
means for receiving packets associated with the at least one frame of the compression-coded video signal, each of the received packets containing both forward error/erasure correction (FEC)-coded high priority partition data bytes that have been coded with a systematic FEC code, and low priority partition information bytes associated with the at least one frame of the compression-coded video signal, the compression-coded video signal having been split into high priority and low priority partitions before having been packetized and transmitted;
means for determining which transmitted packets associated with the at least one frame of the video sequence have not been received;
means for applying a FEC decoding to the FEC-coded high priority partition data bytes in the received packets to determine high priority partition information bytes associated with the at least one frame, wherein for each high priority byte position associated with the received packets, high priority partition information bytes in the same high priority byte position from each received packet containing high priority partition information bytes, one byte per packet, are associated with at least one parity byte in the same byte position in one or more received packets containing parity bytes, the means for applying a FEC
decoding applying a FEC decoding to the high priority partition information bytes and associated at east one parity byte at each high priority byte position, one byte position at a time, to reconstruct high priority partition information bytes in each high priority byte position in a packet that was determined not have been received;
means for combining the received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes;
and means for decompression-decoding the combined received low priority partition information bytes and the decoded and reconstructed high priority partition information bytes to reform the at least one frame of the video signal.
42. The decoder of claim 41 wherein each packet has an equal length and an equal number of low priority partition information bytes are in each packet and an equal number of high priority partition information bytes or parity bytes are in each packet.
43. The decoder of claim 42 wherein the FEC-coded bytes are coded with a Reed Solomon code and the FEC decoding used Reed Solomon decoding.
44. The decoder of claim 42 further comprising a processor, the processor:
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
determining the number of packets (n) in which the at least one frame of the compression-coded video signal has been packetized and transmitted; and determining of the n transmitted packets, the number of packets (k) transmitted containing both high priority partition information bytes associated with the at least one frame and low priority partition information bytes.
45. The decoder of claim 44 wherein the FEC decoding uses Reed Solomon (n,k) decoding.
46. The decoder of claim 44 wherein the received packets associated with the at least one video frame of the video signal are protected against a packet loss over the packet-based network of high priority partition information bytes in up to (n-k) packets.
47. The decoder of claim 41 wherein the means for decompression decoding the video signal uses MPEG decoding.
48. The method of claim 6 further comprising the step of providing to the step of decompression-decoding the low priority partition information bytes and the high priority partition information bytes, information indicating which low priority partition information bytes are missing as a result of having been transmitted in a packet determined not to have been received.
49. The decoder of claim 17 wherein the video decompression decoder is provided with information indicating which low priority partition information bytes are missing as a result of being transmitted in a packet not received.
50. The decoder of claim 28 wherein the means for decompression-decoding the low priority partition information bytes and the high priority partition information bytes is provided with information indicating which low priority information bytes are missing as a result of having been transmitted in a packet determined not to have been received.
51. The method of claim 34 further comprising the step of providing to the step of decompression-decoding the low priority partition information bytes and the high priority partition information bytes, information indicating which low priority partition information bytes are missing as a result of having been transmitted in a packet determined not to have been received.
52. The decoder of claim 41 wherein the means for decompression-decoding the low priority partition information bytes and the high priority partition information bytes is provided with information indicating which low priority information bytes are missing as a result of having been transmitted in a packet determined not to have been received.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US09/176,982 US6490705B1 (en) | 1998-10-22 | 1998-10-22 | Method and apparatus for receiving MPEG video over the internet |
| US09/176,982 | 1998-10-22 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2281211A1 CA2281211A1 (en) | 2000-04-22 |
| CA2281211C true CA2281211C (en) | 2003-07-15 |
Family
ID=22646693
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002281211A Expired - Fee Related CA2281211C (en) | 1998-10-22 | 1999-08-31 | Method and apparatus for receiving mpeg video over the internet |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US6490705B1 (en) |
| EP (1) | EP0996292B1 (en) |
| JP (1) | JP3831559B2 (en) |
| CA (1) | CA2281211C (en) |
Families Citing this family (118)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000078197A (en) * | 1998-09-03 | 2000-03-14 | Toshiba Corp | Communication node and packet transfer method |
| DE69927339T2 (en) * | 1998-11-30 | 2006-06-22 | Matsushita Electric Industrial Co., Ltd., Kadoma | Data transmission method |
| DE19860531C1 (en) * | 1998-12-30 | 2000-08-10 | Univ Muenchen Tech | Method for the transmission of coded digital signals |
| US6514140B1 (en) * | 1999-06-17 | 2003-02-04 | Cias, Inc. | System for machine reading and processing information from gaming chips |
| US7380015B1 (en) * | 1999-09-10 | 2008-05-27 | Kdd Corporation | Apparatus and method for compression-transmitting and decoding picture information and storage medium stored its control programs |
| KR100677070B1 (en) * | 1999-10-02 | 2007-02-01 | 삼성전자주식회사 | Error control method for video bitstream data in wireless multimedia communication and computer readable medium therefor |
| US6614936B1 (en) * | 1999-12-03 | 2003-09-02 | Microsoft Corporation | System and method for robust video coding using progressive fine-granularity scalable (PFGS) coding |
| US7389356B2 (en) * | 1999-12-15 | 2008-06-17 | Microsoft Corporation | Generalized differentiation methods and arrangements for adaptive multimedia communications |
| US7093028B1 (en) * | 1999-12-15 | 2006-08-15 | Microsoft Corporation | User and content aware object-based data stream transmission methods and arrangements |
| US6907073B2 (en) * | 1999-12-20 | 2005-06-14 | Sarnoff Corporation | Tweening-based codec for scaleable encoders and decoders with varying motion computation capability |
| DE10003398A1 (en) * | 2000-01-27 | 2001-08-02 | Alcatel Sa | Process for improving the signal quality of optical signals, transmission system and transmitter |
| US6700933B1 (en) * | 2000-02-15 | 2004-03-02 | Microsoft Corporation | System and method with advance predicted bit-plane coding for progressive fine-granularity scalable (PFGS) video coding |
| DE60020672T2 (en) * | 2000-03-02 | 2005-11-10 | Matsushita Electric Industrial Co., Ltd., Kadoma | Method and apparatus for repeating the video data frames with priority levels |
| US6601217B1 (en) * | 2000-04-13 | 2003-07-29 | Sun Microsystems, Inc. | System and method for error correction in an electronic communication |
| KR100716093B1 (en) * | 2000-07-10 | 2007-05-09 | 가부시키가이샤 히타치세이사쿠쇼 | Multimedia communication terminal |
| JP2004504756A (en) * | 2000-07-17 | 2004-02-12 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Data stream encoding |
| US6978306B2 (en) | 2000-08-10 | 2005-12-20 | Pts Corporation | Multi-tier video delivery network |
| US6718507B1 (en) * | 2000-09-12 | 2004-04-06 | At&T Corp. | System and method for representing compressed information |
| US6728909B1 (en) * | 2000-09-26 | 2004-04-27 | Hewlett-Packard Development Company, L.P. | Data communication with speculative reception of data in a data processing system |
| US6907070B2 (en) * | 2000-12-15 | 2005-06-14 | Microsoft Corporation | Drifting reduction and macroblock-based control in progressive fine granularity scalable video coding |
| US6990151B2 (en) * | 2001-03-05 | 2006-01-24 | Intervideo, Inc. | Systems and methods for enhanced error concealment in a video decoder |
| JP2002271788A (en) * | 2001-03-08 | 2002-09-20 | Monolith Co Ltd | Image coding method and device and image decoding method and device |
| US7366258B2 (en) * | 2001-06-08 | 2008-04-29 | Broadcom Corporation | Chip blanking and processing in SCDMA to mitigate impulse and burst noise and/or distortion |
| US7958532B2 (en) * | 2001-06-18 | 2011-06-07 | At&T Intellectual Property Ii, L.P. | Method of transmitting layered video-coded information |
| US7631242B2 (en) * | 2001-06-22 | 2009-12-08 | Broadcom Corporation | System, method and computer program product for mitigating burst noise in a communications system |
| EP1271955A3 (en) * | 2001-06-26 | 2007-05-02 | Koninklijke Philips Electronics N.V. | Method for packet transmission using packet retransmission requests, and mechanism for controlling transmission of such requests |
| US6745364B2 (en) * | 2001-06-28 | 2004-06-01 | Microsoft Corporation | Negotiated/dynamic error correction for streamed media |
| JP4141952B2 (en) * | 2001-07-19 | 2008-08-27 | トムソン ライセンシング | Method, system, and receiver for improving signal reception |
| US7174384B2 (en) | 2001-07-31 | 2007-02-06 | Dinastech Ipr Limited | Method for delivering large amounts of data with interactivity in an on-demand system |
| US7574728B2 (en) | 2001-07-31 | 2009-08-11 | Dinastech Ipr Limited | System for delivering data over a network |
| US20030041162A1 (en) * | 2001-08-27 | 2003-02-27 | Hochmuth Roland M. | System and method for communicating graphics images over a computer network |
| KR100827147B1 (en) * | 2001-10-19 | 2008-05-02 | 삼성전자주식회사 | Transceiver and method for re-transmission and decording of high speed data in cdma mobile communication system |
| US6801964B1 (en) * | 2001-10-25 | 2004-10-05 | Novell, Inc. | Methods and systems to fast fill media players |
| CN101527684A (en) * | 2001-12-15 | 2009-09-09 | 汤姆森许可公司 | System and method for delivering data streams of multiple data types at different priority levels |
| US7017102B1 (en) * | 2001-12-27 | 2006-03-21 | Network Equipment Technologies, Inc. | Forward Error Correction (FEC) for packetized data networks |
| JP2005515697A (en) * | 2001-12-28 | 2005-05-26 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Uneven error protection using forward error correction based on Reed-Solomon code |
| US7292583B2 (en) | 2002-01-04 | 2007-11-06 | Scientific-Atlanta, Inc. | Receiving streams over asynchronous networks |
| US7155532B2 (en) * | 2002-01-04 | 2006-12-26 | Scientific-Atlanta, Inc. | Transmitting streams over asynchronous networks |
| US7403828B2 (en) * | 2002-01-16 | 2008-07-22 | Aviom, Inc. | System and method for transmitting and distributing audio or video data over a network system having multiple receivers |
| US7061912B1 (en) * | 2002-01-17 | 2006-06-13 | Microtune (San Diego) , Inc. | Method and apparatus of packet loss concealment for CVSD coders |
| KR20040075938A (en) * | 2002-01-18 | 2004-08-30 | 코닌클리케 필립스 일렉트로닉스 엔.브이. | Robust signal coding |
| US7228488B1 (en) | 2002-02-15 | 2007-06-05 | Network Equipment Technologies, Inc. | System and method for secure communication over packet network |
| US7640485B1 (en) | 2002-02-15 | 2009-12-29 | Network Equipment Technologies, Inc. | Non-relay initialization for modems |
| US7545819B1 (en) | 2002-02-15 | 2009-06-09 | Network Equipment Technologies, Inc. | Techniques for asynchronous compensation for secure communications |
| CN1633815A (en) * | 2002-02-18 | 2005-06-29 | 皇家飞利浦电子股份有限公司 | Coding a data stream with unequal error protection |
| US6778254B2 (en) * | 2002-02-18 | 2004-08-17 | Ando Electric Co., Ltd. | Motion picture code evaluator and related systems |
| US20040057465A1 (en) * | 2002-09-24 | 2004-03-25 | Koninklijke Philips Electronics N.V. | Flexible data partitioning and packetization for H.26L for improved packet loss resilience |
| US7177658B2 (en) | 2002-05-06 | 2007-02-13 | Qualcomm, Incorporated | Multi-media broadcast and multicast service (MBMS) in a wireless communications system |
| KR100442473B1 (en) * | 2002-05-30 | 2004-07-30 | 주식회사 클릭티브이 | Digital animation controller via network |
| US6851084B2 (en) * | 2002-06-10 | 2005-02-01 | Harris Corporation | Forward error correction method and system for reliable transmission of real time data over a packet based network |
| MY162363A (en) * | 2002-06-11 | 2017-06-15 | Thomson Licensing Sa | Multimedia server with simple adaptation to dynamic network loss conditions |
| DE10227165A1 (en) | 2002-06-18 | 2004-01-29 | Siemens Ag | Method and arrangement for coding or decoding a sequence of digital data |
| US7047480B2 (en) * | 2002-11-12 | 2006-05-16 | Microsoft Corporation | Matrix multiplication in a Galois field for forward error correction |
| US8352991B2 (en) * | 2002-12-09 | 2013-01-08 | Thomson Licensing | System and method for modifying a video stream based on a client or network environment |
| US6847521B2 (en) * | 2003-01-03 | 2005-01-25 | Dell Products L.P. | PCB as a structural component and internal chassis alignment |
| EP2538695A1 (en) * | 2003-02-21 | 2012-12-26 | NEC Corporation | Image data distribution control method, device, system, and program |
| US7827458B1 (en) | 2003-03-03 | 2010-11-02 | Apple Inc. | Packet loss error recovery |
| US7434145B2 (en) * | 2003-04-02 | 2008-10-07 | Qualcomm Incorporated | Extracting soft information in a block-coherent communication system |
| US7231557B2 (en) * | 2003-04-02 | 2007-06-12 | Qualcomm Incorporated | Methods and apparatus for interleaving in a block-coherent communication system |
| US8196000B2 (en) * | 2003-04-02 | 2012-06-05 | Qualcomm Incorporated | Methods and apparatus for interleaving in a block-coherent communication system |
| CN100461116C (en) * | 2003-04-02 | 2009-02-11 | 高通股份有限公司 | Methods and apparatus for interleaving in a block-coherent communication system |
| US7395346B2 (en) * | 2003-04-22 | 2008-07-01 | Scientific-Atlanta, Inc. | Information frame modifier |
| US8204076B2 (en) * | 2003-05-01 | 2012-06-19 | Genesis Microchip Inc. | Compact packet based multimedia interface |
| US7839860B2 (en) | 2003-05-01 | 2010-11-23 | Genesis Microchip Inc. | Packet based video display interface |
| US8068485B2 (en) | 2003-05-01 | 2011-11-29 | Genesis Microchip Inc. | Multimedia interface |
| US8059673B2 (en) | 2003-05-01 | 2011-11-15 | Genesis Microchip Inc. | Dynamic resource re-allocation in a packet based video display interface |
| KR100698620B1 (en) | 2003-06-16 | 2007-03-21 | 삼성전자주식회사 | Digital transmitter/receiver system having a robust error correction coding/decoding apparatus and a method error correction coding/decoding therof |
| US8804761B2 (en) | 2003-08-21 | 2014-08-12 | Qualcomm Incorporated | Methods for seamless delivery of broadcast and multicast content across cell borders and/or between different transmission schemes and related apparatus |
| US8694869B2 (en) | 2003-08-21 | 2014-04-08 | QUALCIMM Incorporated | Methods for forward error correction coding above a radio link control layer and related apparatus |
| US7318187B2 (en) | 2003-08-21 | 2008-01-08 | Qualcomm Incorporated | Outer coding methods for broadcast/multicast content and related apparatus |
| JP2005079793A (en) * | 2003-08-29 | 2005-03-24 | Nec Corp | Image communication method, image communication system, image transmitter, and image receiver |
| US7634090B2 (en) | 2003-09-26 | 2009-12-15 | Genesis Microchip Inc. | Packet based high definition high-bandwidth digital content protection |
| GB2415873A (en) * | 2004-06-30 | 2006-01-04 | Nokia Corp | Erasure information generation in Forward Error Correction decoding |
| US7925955B2 (en) * | 2005-03-10 | 2011-04-12 | Qualcomm Incorporated | Transmit driver in communication system |
| US8693540B2 (en) * | 2005-03-10 | 2014-04-08 | Qualcomm Incorporated | Method and apparatus of temporal error concealment for P-frame |
| US7886201B2 (en) * | 2005-03-10 | 2011-02-08 | Qualcomm Incorporated | Decoder architecture for optimized error management in streaming multimedia |
| US20070033496A1 (en) * | 2005-07-14 | 2007-02-08 | Hitachi, Ltd. | System and method for adjusting BER/PER to increase network stream-based transmission rates |
| KR100735276B1 (en) * | 2005-08-18 | 2007-07-03 | 삼성전자주식회사 | Method and apparatus for decoding a mpe-fec frame in a dvb-h system |
| US7852853B1 (en) * | 2006-02-07 | 2010-12-14 | Nextel Communications Inc. | System and method for transmitting video information |
| JP4193865B2 (en) * | 2006-04-27 | 2008-12-10 | ソニー株式会社 | Digital signal switching device and switching method thereof |
| US20070297375A1 (en) * | 2006-06-27 | 2007-12-27 | Motorola, Inc. | System and method for data transmission in an ad hoc communication network |
| CN101485204A (en) * | 2006-06-29 | 2009-07-15 | 皇家飞利浦电子股份有限公司 | Method and apparatus for encoding and decoding data with error correction |
| US8306060B2 (en) * | 2006-11-07 | 2012-11-06 | Samsung Electronics Co., Ltd. | System and method for wireless communication of uncompressed video having a composite frame format |
| JP4250654B2 (en) * | 2006-11-17 | 2009-04-08 | 株式会社東芝 | COMMUNICATION DEVICE, COMMUNICATION METHOD, AND COMMUNICATION PROGRAM |
| US7937640B2 (en) * | 2006-12-18 | 2011-05-03 | At&T Intellectual Property I, L.P. | Video over IP network transmission system |
| WO2008087603A2 (en) * | 2007-01-19 | 2008-07-24 | Koninklijke Philips Electronics, N.V. | Method and system of single carrier block transmission with parallel encoding and decoding |
| CN101296055B (en) * | 2007-04-29 | 2013-01-09 | 华为技术有限公司 | Data package dispatching method and device |
| EP1995978B1 (en) * | 2007-05-25 | 2010-07-07 | Psytechnics Ltd | Video quality assessment |
| EP2183882A1 (en) * | 2007-08-15 | 2010-05-12 | Maxlinear, Inc. | Method and apparatus for preserving deinterleaving erasure information of block interleaved coded signal |
| KR20100075466A (en) * | 2007-08-30 | 2010-07-02 | 톰슨 라이센싱 | Methods and systems for providing different data loss protection |
| JP5109787B2 (en) * | 2008-05-02 | 2012-12-26 | 富士通株式会社 | Data transmission system, program and method |
| US8042143B2 (en) * | 2008-09-19 | 2011-10-18 | At&T Intellectual Property I, L.P. | Apparatus and method for distributing media content |
| US8429440B2 (en) | 2009-05-13 | 2013-04-23 | Stmicroelectronics, Inc. | Flat panel display driver method and system |
| US8156238B2 (en) | 2009-05-13 | 2012-04-10 | Stmicroelectronics, Inc. | Wireless multimedia transport method and apparatus |
| CN101902315B (en) * | 2009-06-01 | 2013-04-17 | 华为技术有限公司 | Retransmission method, device and communication system based on forward error correction |
| US8769139B2 (en) * | 2010-01-29 | 2014-07-01 | Clarendon Foundation, Inc. | Efficient streaming server |
| US20110191446A1 (en) * | 2010-01-29 | 2011-08-04 | Clarendon Foundation, Inc. | Storing and streaming media content |
| US20110191447A1 (en) * | 2010-01-29 | 2011-08-04 | Clarendon Foundation, Inc. | Content distribution system |
| US8260833B2 (en) * | 2010-01-29 | 2012-09-04 | Clarendon Foundation, Inc. | Naming scheme for media storage |
| US20110191439A1 (en) * | 2010-01-29 | 2011-08-04 | Clarendon Foundation, Inc. | Media content ingestion |
| EP2567546A4 (en) * | 2010-05-10 | 2014-01-15 | Samsung Electronics Co Ltd | Method and apparatus for transmitting and receiving layered coded video |
| US8671234B2 (en) | 2010-05-27 | 2014-03-11 | Stmicroelectronics, Inc. | Level shifting cable adaptor and chip system for use with dual-mode multi-media device |
| US8406134B2 (en) | 2010-06-25 | 2013-03-26 | At&T Intellectual Property I, L.P. | Scaling content communicated over a network |
| JP4929393B1 (en) | 2010-11-19 | 2012-05-09 | 株式会社東芝 | Reception device, reception method, and reception program |
| EP2595351A1 (en) * | 2011-11-15 | 2013-05-22 | Eaton Industries GmbH | Device for a digital transfer system, digital transfer system and data transfer method |
| US9848217B2 (en) * | 2012-01-20 | 2017-12-19 | Korea Electronics Technology Institute | Method for transmitting and receiving program configuration information for scalable ultra high definition video service in hybrid transmission environment, and method and apparatus for effectively transmitting scalar layer information |
| US10320520B2 (en) * | 2013-11-15 | 2019-06-11 | Hitachi, Ltd. | Communication device, system and method |
| US9380351B2 (en) * | 2014-01-17 | 2016-06-28 | Lg Display Co., Ltd. | Apparatus for transmitting encoded video stream and method for transmitting the same |
| EP3910960A1 (en) * | 2014-08-07 | 2021-11-17 | Sony Group Corporation | Transmission device, transmission method, and reception device |
| GB2536299C (en) * | 2015-03-13 | 2017-07-05 | Gurulogic Microsystems Oy | Method of communicating data packets within data communication systems |
| WO2017164595A1 (en) * | 2016-03-21 | 2017-09-28 | 엘지전자(주) | Broadcast signal transmitting/receiving device and method |
| US10594661B1 (en) * | 2017-06-13 | 2020-03-17 | Parallels International Gmbh | System and method for recovery of data packets transmitted over an unreliable network |
| FR3070566B1 (en) * | 2017-08-30 | 2020-09-04 | Sagemcom Broadband Sas | PROCESS FOR RECOVERING A TARGET FILE OF AN OPERATING SOFTWARE AND DEVICE FOR USE |
| US11606528B2 (en) * | 2018-01-03 | 2023-03-14 | Saturn Licensing Llc | Advanced television systems committee (ATSC) 3.0 latency-free display of content attribute |
| CN112400290B (en) | 2019-05-12 | 2024-04-05 | 阿密蒙有限公司 | Method for transmitting video via User Datagram Protocol (UDP) communication link |
| GB2598701B (en) * | 2020-05-25 | 2023-01-25 | V Nova Int Ltd | Wireless data communication system and method |
| CN112333470B (en) * | 2020-10-27 | 2022-05-27 | 杭州叙简科技股份有限公司 | FEC (forward error correction) system based on video frame |
| CN113709510A (en) * | 2021-08-06 | 2021-11-26 | 联想(北京)有限公司 | High-speed data real-time transmission method and device, equipment and storage medium |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5001561A (en) | 1990-05-02 | 1991-03-19 | At&T Bell Laboratories | Embedded coding system for video signals |
| US5231486A (en) * | 1992-07-27 | 1993-07-27 | General Electric Company | Data separation processing in a dual channel digital high definition television system |
| US5771081A (en) | 1994-02-28 | 1998-06-23 | Korea Telecommunication Authority | Bit system for transmitting digital video data |
| JPH07283740A (en) | 1994-04-05 | 1995-10-27 | Sony Corp | Transmission device, reception device and transmission system |
| US5481312A (en) * | 1994-09-12 | 1996-01-02 | At&T Corp. | Method of and apparatus for the transmission of high and low priority segments of a video bitstream over packet networks |
| US5617541A (en) * | 1994-12-21 | 1997-04-01 | International Computer Science Institute | System for packetizing data encoded corresponding to priority levels where reconstructed data corresponds to fractionalized priority level and received fractionalized packets |
| DE19547707A1 (en) | 1995-12-20 | 1997-07-03 | Thomson Brandt Gmbh | Process, encoder and decoder for the transmission of hierarchical digital signals divided into several parts |
| JPH11127138A (en) * | 1997-10-24 | 1999-05-11 | Sony Corp | Error correction coding method, device therefor, and data transmission method |
| US6317462B1 (en) * | 1998-10-22 | 2001-11-13 | Lucent Technologies Inc. | Method and apparatus for transmitting MPEG video over the internet |
-
1998
- 1998-10-22 US US09/176,982 patent/US6490705B1/en not_active Expired - Lifetime
-
1999
- 1999-08-31 CA CA002281211A patent/CA2281211C/en not_active Expired - Fee Related
- 1999-10-12 EP EP99308040.7A patent/EP0996292B1/en not_active Expired - Lifetime
- 1999-10-21 JP JP29907499A patent/JP3831559B2/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| JP2000134620A (en) | 2000-05-12 |
| CA2281211A1 (en) | 2000-04-22 |
| EP0996292A1 (en) | 2000-04-26 |
| US6490705B1 (en) | 2002-12-03 |
| EP0996292B1 (en) | 2017-05-17 |
| JP3831559B2 (en) | 2006-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2281211C (en) | Method and apparatus for receiving mpeg video over the internet | |
| CA2281353C (en) | Method and apparatus for transmitting mpeg video over the internet | |
| Boyce | Packet loss resilient transmission of MPEG video over the Internet | |
| US7116714B2 (en) | Video coding | |
| KR100495820B1 (en) | Video coding | |
| CA2409499C (en) | Video coding using the sequence numbers of reference pictures for error correction | |
| JP2006518127A (en) | Picture decoding method | |
| Frossard et al. | AMISP: a complete content-based MPEG-2 error-resilient scheme | |
| Golaghazadeh et al. | Checksum-filtered list decoding applied to H. 264 and H. 265 video error correction | |
| Vilei et al. | A novel unbalanced multiple description scheme for video transmission over wlan | |
| Nguyen et al. | Adaptive error protection for Scalable Video Coding extension of H. 264/AVC | |
| Tian et al. | Improved H. 264/AVC video broadcast/multicast | |
| Swann et al. | Error resilient transmission of MPEG-II over noisy wireless ATM networks | |
| Frossard et al. | Content-based MPEG-2 structuring and protection | |
| Aladrovic et al. | An error resilience scheme for layered video coding | |
| Horn | Scalable Internet Video Streaming With Unequal Error Protection | |
| Jiang et al. | Robust video coding over wireless channels using TRIRF inter-frame coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed |