CA2149068A1 - Sound-synchronized video system - Google Patents
Sound-synchronized video systemInfo
- Publication number
- CA2149068A1 CA2149068A1 CA002149068A CA2149068A CA2149068A1 CA 2149068 A1 CA2149068 A1 CA 2149068A1 CA 002149068 A CA002149068 A CA 002149068A CA 2149068 A CA2149068 A CA 2149068A CA 2149068 A1 CA2149068 A1 CA 2149068A1
- Authority
- CA
- Canada
- Prior art keywords
- visemes
- audio signal
- phonemes
- stream
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4341—Demultiplexing of audio and video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
- G10L21/10—Transforming into visible information
- G10L2021/105—Synthesis of the lips movements from speech, e.g. for talking heads
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B2220/00—Record carriers by type
- G11B2220/90—Tape-like record carriers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
Abstract
A stream of unsynchronized audio signal, representing speech and video signal of a speaker, is processed by decoding the signal, memorizing a plurality of visemes corresponding to phonemes in the audio signal, fetching from the plurality visemes corresponding to phonemes in the audio signal, and imparting a synchronism to the video signal and audio signal by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream. According to an embodiment, the fetching step includes fetching visemes of the lip movement. The system is suitable for use in a videophone.
Description
21~9068 SOUND~ ONTZED VIDEO SYSTEM
This is a continuation-in-part of U.S. application of H.H.
Chen et al., Serial No. 08/210,198 filed March 18, 1994, and U.S. application of H.H. Chen et al., Serial No. 08/210,819 filed March 18, 1994, both assigned to the same assignee as this application. The contents of these applications are hereby incorporated herein as if fully recited herein.
This application is also related to the application of T. Chen Serial No. 08/210,529, also assigned to the same assignee as this application.
Field of the In~ention This invention relates to synchronizing facial expressions with speech, and particularly to sound-synchronized two-way video communication such as videotelephony.
Bac~ ou~d of the Invention Videophone, teleconferencing, multimedia, and other visual communication techniques often produce a delay in the video signal relative to voice signal, particularly at low bit rates. The delayed video sequence is visually annoying in real-time telecommunication services because the voice is not synchronized with the mouth movement or facial expression of the speaker. Inserting extra delay in the speech signal permits synchronization. However the delay occurs in both directions and produces undesirable discomfort.
Summary of the Invention According to an aspect of the invention, a stream, of digitized decoded speech and video signals from a speaker, is processed by memorizing a plurality of visemes corresponding to phonemes in the audio signal, fetching visemes corresponding to phonemes in the audio signal, and imparting a synchronism to the decoded video and audio signals by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
According to another aspect of the invention, imparting includes extracting visemes from the instantaneous video signal being decoded and updating the memory of the visemes with the extracted visemes.
These and other features of the invention are pointed out in the claims, objects and advantages of the invention will become evident from the following detailed description when read in light of the accompanying drawings.
.
Brief Description of the Drawin~s Fig. 1 is a block diagram of a system embodying features of the invention.
Fig. 2 is a flow diagram of the operation of components of the system in Fig. 1.
Detailed De~cription of Preferred Embodiments Fig. 1 illustrates a system embodying features of the invention and illustrates N stations STl, ST2 ... STN
communicating digitally with each other through a communication link CLl. The communication link CLl may be a wire link, wireless, microwave, satellite link, or any other type of link. The stations STl to STN may be substantially identical or different.
In the station STl, a transmitter TRl includes a camera, microphone, encoder, and conventional transmitting equipment for transmitting digitized audio and visual data such as for a videophone. The received signal is processed in a receiver generally designated REl.
In the receiver REl, a codec (coder-decoder) CDl decodes the signals to the receiver REl and encodes the signals from the transmitter TRl. The codec CDl also separates the digitized video and audio signals into the digital video and speech components. The audio signal goes directly to an audio-visual (A-V) display DIl.
At the same time, a phoneme recognition module PRl divides the incoming speech components into recognizable phonemes. A lookup table LTl stores a number of visemes.
A viseme is a sequence of one or more facial feature positions corresponding to a phoneme. Visemes perform the basic units of visual articulatory mouth shapes. A viseme contains mouth parameters which specify the mouth opening, height, width, and protrusion. The lookup table LTl maps phonemes into visemes which specify the corresponding mouth shape parameters, such as the positions of lips, jaws, tongue, and teeth.
At the video output of the codec CDl, a feature extraction module FEl, extracts mouth information visemes containing the mouth shape and the mouth location from the decoded and delayed video signal. A mouth deformation module MDl receives inputs from the video signal, the information from the feature extraction module FEl, and the visemes from the lookup table LTl. The mouth deformation module MDl uses the mouth location generated by the feature extraction module FEl to locate the mouth in the current frame. It also replaces the information concerning the mouth shape from the feature extraction module FEl with the mouth shape specified by the viseme from the lookup table LTl. It further performs texture mapping of the mouth image specified in the feature extraction module FEl to the mouth image specified by the lookup table LTl. The output video is applied to the display DIl together with the audio signal and produces lip synchronizati`on.
A memory MEl stores and time stamps mouth information from the feature extraction module FEl for phoneme-to-viseme identification. It then enters the phoneme-to-viseme identification in the lookup table LTl to update the lookup table. In one embodiment a microprocessor, not shown, controls the operation of the modules and members of the receiver REl. The modules and elements FEl, PRl, MEl, MDl, LTl form a correction arrangement COl.
According to one embodiment of the invention, any or all of the stations ST2 to STN differ from the station STl. For example the stations ST2 to STN may be ordinary videophones without the delay correction disclosed herein.
According to another embodiment of the invention, one or all of the station ST2 to STN are identical to and operate identical to the station STl. The Stations STl to STN form a system in which the stations communicate with each other.
According to an embodiment of the invention, the correction arrangement COl is applied as an add-on or retrofit to stations without such correction.
2l49o68 Fig. 2 is a flow chart indicating the operation of the receiver REl and codec CDl. In step 100 of Fig. 2, the codec CDl decodes and separates the video and audio signals.
In step 110, the phoneme recognition module PRl analyzes the phonemes of the speech in the audio signal to recognize the current phoneme in the video signal. In step 114, the lookup table LTl accesses the viseme corresponding to the phoneme from the phoneme recognition module PRl.
In the meantime, during this operation, in step 117, the feature extraction module FEl extracts the viseme in the decoded video signal by comparing the latter with a face model, and stores it in the memory MEl. In step 120 the memory module MEl memorizes the extracted feature and in step 124, it updates the visemes in the lookup table LTl.
In step 127, the mouth deformation module MDl applies the mouth shape extracted from the lookup table LTl to the current video signal. In step 130 the display DIl displays and plays the video and audio signals. In step 134, the arrangement continues at step 100.
The manner of extracting phonemes from speech sequences, the manner of memorizing visemes corresponding to phonemes from a particular speaker or from a universal speaker, the manner of extracting the correct facial features and applying them to a video signal to deform the mouth, etc., are disclosed in the co-pending U.S. patent application Serial No. 08/210,198 filed March 18, 1994 and entitled "Library Based Coding of Video Signals", Serial No.
08/210,819, filed March 18, 1994 and entitled "Audiovisual Dubbing System and Method", both assigned to the same assignee as the present application. Further aspects of the above are also disclosed in U.S. Patent No. 3,743,391, U.S.
Patent No. 4,975,960, and in a paper presented by Shigeo Morishima et al. at the 1989 ICASSP in Glasgow, UK, entitled 21~9068 ~An Intelligent Facial Image Coding Driven By Speech and Phoneme", as well as in the aforementioned U.S. application Serial No. 08/210,529 of T. Chen, filed March 18, 1994.
According to other embodiments of the invention, the techniques and structures described in the aforementioned and the following for various elements and modules such as the phoneme extraction are used herein.
The mouth deformation module MD1 also performs texture mapping. A technique for this purpose is described in a paper by J. Yau and N. Duffy, " A texture mapping approach to 3D facial image synthesis," Comput. Graphics Forum, no. 7, pp. 129-134, 1988. According to an embodiment of the invention, such texture mapping is used as part of this invention.
Further literature showing the extraction of facial shape and formation from a video signal by speech analysis appears in H. Choi et. al. entitled "Analysis and Synthesis of Facial Expressions in Knowledge Based Coding of Facial Image Sequences", at the International Conference on Acoustics Speech Signal Processing, pp. 2737-40 (1991); H.
Chen, T. Chen, B. Haskell, A. Kaplan, S. Keshev, E. Petajan, "Audio-assisted video coding/processing," Contribution MPEG
94/084, March 1994; A.E. Kaplan and S. Keshav, "Talking heads made simple," in Facial Animation Workshop, University of Pennsylvania, Nov. 1993; and A. Lippman, "Semantic bandwidth compression: Speechmaker," in Proc. Picture Coding Symposium, 1981. According to other embodiments of the invention the techniques and means disclosed in the aforementioned and following for various elements and modules are used in the methods and means disclosed herein.
In the phoneme recognition module PR1, an automatic speech recognizer called the BLASR (Bell Labs 2l~9o68 -Automatic Speech Recognition) serves to produce a sequence of phoneme/duration pairs for the input speech. According to an embodiment of the invention, 47 DARPA (Defense Advanced Research Projects Agency) phonemes for the english language and 12 visemes operate in the lookup table, although these could be any number. According to another embodiment of the invention, a larger set of visemes produce finer video rendering. However, the set of 12 visemes suffices for ordinary videophones. A phone group contains phonemes that correspond to similar mouth shapes. According to one embodiment of the invention, speech recognition at the level of individual phonemes is used. According to another embodiment, speech recognition at the level of phone groups, rather than individual phonemes, is used.
Initially, the visemes contain generic mouth shape parameters which are speaker independent. According to an embodiment of the invention, the lookup table LTl updates its viseme content based on the memory MEl which holds speaker dependent facial parameters.
According to one embodiment of the invention, a viseme is obtained by using a face model to synthesize the mouth area. The latter covers the lips, jaw, teeth and cheeks. According to an embodiment of the invention, this is accomplished with a wire frame model as disclosed in K.
Aizawa, H. Harashima, and T. Saito, "Model-based analysis synthesis image coding system for a person's face," Signal Processing: Image Communication, vol. 1, no. 2, Octo. 1989, pp. 139-152. The mouth deformation is controlled by a number of control points around the lips and the normalized positions of these points are given to the visemes. For each deformed surface patch, texture mapping is applied to determine the color of the patch. According to an embodiment of the invention, six control points are used, .
two at the ends of the mouth, and four in the middle of the upper and bottom edges of the lips.
According to one embodiment of the invention, the module PRl performs phoneme recognition on the input audio signal to break down the speaker's utterances into a sequence of phonemes. These phonemes are then mapped into visemes which specify the corresponding mouth shape parameters such as the positions of lips, jaw, tongue, and teeth in the table LTl. A generic viseme table is used at the beginning to start the process. Later the mouth parameters are constantly updated according to the actual mouth shape of the speaker extracted from the delayed video image.
In parallel to audio signal processing, the video input images are analyzed to extract the mouth area for modification. This mouth information is stored in a memory of the table LTl and time stamped for the purpose of phoneme to viseme identification. Such information is used to generate speaker dependent mouth parameters. These position parameters then serve to modify the mouth area in the current frame to produce sound synchronized video.
While the modules and other elements shown as boxes in Fig. 2 may be separable members, according to an embodiment of the invention, any or all the various modules and members of a station are parts of monolithic arrangement such as a chip. According to another embodiment of the invention, the modules and members shown as boxes have their respective functions performed in a microprocessor. As mentioned, various embodiments of the invention employ techniques and means disclosed in the above publications, patents, and applications for the various elements and modules disclosed herein.
The invention synchronizes video and audio signals that had originally been acquired as synchronized signals but had become unsynchronized by processing in videophone, teleconferencing, multimedia, and other visual communication techniques. Delay of the video signal relative to the audio signal, which results in lack of synchronism, can occur during encoding in the transmitter, transmission, and decoding in the receiver.
While embodiments of the invention have been described in detail, it will be evident to those skilled in the art that the invention may be embodied otherwise without departing from its spirit and scope.
This is a continuation-in-part of U.S. application of H.H.
Chen et al., Serial No. 08/210,198 filed March 18, 1994, and U.S. application of H.H. Chen et al., Serial No. 08/210,819 filed March 18, 1994, both assigned to the same assignee as this application. The contents of these applications are hereby incorporated herein as if fully recited herein.
This application is also related to the application of T. Chen Serial No. 08/210,529, also assigned to the same assignee as this application.
Field of the In~ention This invention relates to synchronizing facial expressions with speech, and particularly to sound-synchronized two-way video communication such as videotelephony.
Bac~ ou~d of the Invention Videophone, teleconferencing, multimedia, and other visual communication techniques often produce a delay in the video signal relative to voice signal, particularly at low bit rates. The delayed video sequence is visually annoying in real-time telecommunication services because the voice is not synchronized with the mouth movement or facial expression of the speaker. Inserting extra delay in the speech signal permits synchronization. However the delay occurs in both directions and produces undesirable discomfort.
Summary of the Invention According to an aspect of the invention, a stream, of digitized decoded speech and video signals from a speaker, is processed by memorizing a plurality of visemes corresponding to phonemes in the audio signal, fetching visemes corresponding to phonemes in the audio signal, and imparting a synchronism to the decoded video and audio signals by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
According to another aspect of the invention, imparting includes extracting visemes from the instantaneous video signal being decoded and updating the memory of the visemes with the extracted visemes.
These and other features of the invention are pointed out in the claims, objects and advantages of the invention will become evident from the following detailed description when read in light of the accompanying drawings.
.
Brief Description of the Drawin~s Fig. 1 is a block diagram of a system embodying features of the invention.
Fig. 2 is a flow diagram of the operation of components of the system in Fig. 1.
Detailed De~cription of Preferred Embodiments Fig. 1 illustrates a system embodying features of the invention and illustrates N stations STl, ST2 ... STN
communicating digitally with each other through a communication link CLl. The communication link CLl may be a wire link, wireless, microwave, satellite link, or any other type of link. The stations STl to STN may be substantially identical or different.
In the station STl, a transmitter TRl includes a camera, microphone, encoder, and conventional transmitting equipment for transmitting digitized audio and visual data such as for a videophone. The received signal is processed in a receiver generally designated REl.
In the receiver REl, a codec (coder-decoder) CDl decodes the signals to the receiver REl and encodes the signals from the transmitter TRl. The codec CDl also separates the digitized video and audio signals into the digital video and speech components. The audio signal goes directly to an audio-visual (A-V) display DIl.
At the same time, a phoneme recognition module PRl divides the incoming speech components into recognizable phonemes. A lookup table LTl stores a number of visemes.
A viseme is a sequence of one or more facial feature positions corresponding to a phoneme. Visemes perform the basic units of visual articulatory mouth shapes. A viseme contains mouth parameters which specify the mouth opening, height, width, and protrusion. The lookup table LTl maps phonemes into visemes which specify the corresponding mouth shape parameters, such as the positions of lips, jaws, tongue, and teeth.
At the video output of the codec CDl, a feature extraction module FEl, extracts mouth information visemes containing the mouth shape and the mouth location from the decoded and delayed video signal. A mouth deformation module MDl receives inputs from the video signal, the information from the feature extraction module FEl, and the visemes from the lookup table LTl. The mouth deformation module MDl uses the mouth location generated by the feature extraction module FEl to locate the mouth in the current frame. It also replaces the information concerning the mouth shape from the feature extraction module FEl with the mouth shape specified by the viseme from the lookup table LTl. It further performs texture mapping of the mouth image specified in the feature extraction module FEl to the mouth image specified by the lookup table LTl. The output video is applied to the display DIl together with the audio signal and produces lip synchronizati`on.
A memory MEl stores and time stamps mouth information from the feature extraction module FEl for phoneme-to-viseme identification. It then enters the phoneme-to-viseme identification in the lookup table LTl to update the lookup table. In one embodiment a microprocessor, not shown, controls the operation of the modules and members of the receiver REl. The modules and elements FEl, PRl, MEl, MDl, LTl form a correction arrangement COl.
According to one embodiment of the invention, any or all of the stations ST2 to STN differ from the station STl. For example the stations ST2 to STN may be ordinary videophones without the delay correction disclosed herein.
According to another embodiment of the invention, one or all of the station ST2 to STN are identical to and operate identical to the station STl. The Stations STl to STN form a system in which the stations communicate with each other.
According to an embodiment of the invention, the correction arrangement COl is applied as an add-on or retrofit to stations without such correction.
2l49o68 Fig. 2 is a flow chart indicating the operation of the receiver REl and codec CDl. In step 100 of Fig. 2, the codec CDl decodes and separates the video and audio signals.
In step 110, the phoneme recognition module PRl analyzes the phonemes of the speech in the audio signal to recognize the current phoneme in the video signal. In step 114, the lookup table LTl accesses the viseme corresponding to the phoneme from the phoneme recognition module PRl.
In the meantime, during this operation, in step 117, the feature extraction module FEl extracts the viseme in the decoded video signal by comparing the latter with a face model, and stores it in the memory MEl. In step 120 the memory module MEl memorizes the extracted feature and in step 124, it updates the visemes in the lookup table LTl.
In step 127, the mouth deformation module MDl applies the mouth shape extracted from the lookup table LTl to the current video signal. In step 130 the display DIl displays and plays the video and audio signals. In step 134, the arrangement continues at step 100.
The manner of extracting phonemes from speech sequences, the manner of memorizing visemes corresponding to phonemes from a particular speaker or from a universal speaker, the manner of extracting the correct facial features and applying them to a video signal to deform the mouth, etc., are disclosed in the co-pending U.S. patent application Serial No. 08/210,198 filed March 18, 1994 and entitled "Library Based Coding of Video Signals", Serial No.
08/210,819, filed March 18, 1994 and entitled "Audiovisual Dubbing System and Method", both assigned to the same assignee as the present application. Further aspects of the above are also disclosed in U.S. Patent No. 3,743,391, U.S.
Patent No. 4,975,960, and in a paper presented by Shigeo Morishima et al. at the 1989 ICASSP in Glasgow, UK, entitled 21~9068 ~An Intelligent Facial Image Coding Driven By Speech and Phoneme", as well as in the aforementioned U.S. application Serial No. 08/210,529 of T. Chen, filed March 18, 1994.
According to other embodiments of the invention, the techniques and structures described in the aforementioned and the following for various elements and modules such as the phoneme extraction are used herein.
The mouth deformation module MD1 also performs texture mapping. A technique for this purpose is described in a paper by J. Yau and N. Duffy, " A texture mapping approach to 3D facial image synthesis," Comput. Graphics Forum, no. 7, pp. 129-134, 1988. According to an embodiment of the invention, such texture mapping is used as part of this invention.
Further literature showing the extraction of facial shape and formation from a video signal by speech analysis appears in H. Choi et. al. entitled "Analysis and Synthesis of Facial Expressions in Knowledge Based Coding of Facial Image Sequences", at the International Conference on Acoustics Speech Signal Processing, pp. 2737-40 (1991); H.
Chen, T. Chen, B. Haskell, A. Kaplan, S. Keshev, E. Petajan, "Audio-assisted video coding/processing," Contribution MPEG
94/084, March 1994; A.E. Kaplan and S. Keshav, "Talking heads made simple," in Facial Animation Workshop, University of Pennsylvania, Nov. 1993; and A. Lippman, "Semantic bandwidth compression: Speechmaker," in Proc. Picture Coding Symposium, 1981. According to other embodiments of the invention the techniques and means disclosed in the aforementioned and following for various elements and modules are used in the methods and means disclosed herein.
In the phoneme recognition module PR1, an automatic speech recognizer called the BLASR (Bell Labs 2l~9o68 -Automatic Speech Recognition) serves to produce a sequence of phoneme/duration pairs for the input speech. According to an embodiment of the invention, 47 DARPA (Defense Advanced Research Projects Agency) phonemes for the english language and 12 visemes operate in the lookup table, although these could be any number. According to another embodiment of the invention, a larger set of visemes produce finer video rendering. However, the set of 12 visemes suffices for ordinary videophones. A phone group contains phonemes that correspond to similar mouth shapes. According to one embodiment of the invention, speech recognition at the level of individual phonemes is used. According to another embodiment, speech recognition at the level of phone groups, rather than individual phonemes, is used.
Initially, the visemes contain generic mouth shape parameters which are speaker independent. According to an embodiment of the invention, the lookup table LTl updates its viseme content based on the memory MEl which holds speaker dependent facial parameters.
According to one embodiment of the invention, a viseme is obtained by using a face model to synthesize the mouth area. The latter covers the lips, jaw, teeth and cheeks. According to an embodiment of the invention, this is accomplished with a wire frame model as disclosed in K.
Aizawa, H. Harashima, and T. Saito, "Model-based analysis synthesis image coding system for a person's face," Signal Processing: Image Communication, vol. 1, no. 2, Octo. 1989, pp. 139-152. The mouth deformation is controlled by a number of control points around the lips and the normalized positions of these points are given to the visemes. For each deformed surface patch, texture mapping is applied to determine the color of the patch. According to an embodiment of the invention, six control points are used, .
two at the ends of the mouth, and four in the middle of the upper and bottom edges of the lips.
According to one embodiment of the invention, the module PRl performs phoneme recognition on the input audio signal to break down the speaker's utterances into a sequence of phonemes. These phonemes are then mapped into visemes which specify the corresponding mouth shape parameters such as the positions of lips, jaw, tongue, and teeth in the table LTl. A generic viseme table is used at the beginning to start the process. Later the mouth parameters are constantly updated according to the actual mouth shape of the speaker extracted from the delayed video image.
In parallel to audio signal processing, the video input images are analyzed to extract the mouth area for modification. This mouth information is stored in a memory of the table LTl and time stamped for the purpose of phoneme to viseme identification. Such information is used to generate speaker dependent mouth parameters. These position parameters then serve to modify the mouth area in the current frame to produce sound synchronized video.
While the modules and other elements shown as boxes in Fig. 2 may be separable members, according to an embodiment of the invention, any or all the various modules and members of a station are parts of monolithic arrangement such as a chip. According to another embodiment of the invention, the modules and members shown as boxes have their respective functions performed in a microprocessor. As mentioned, various embodiments of the invention employ techniques and means disclosed in the above publications, patents, and applications for the various elements and modules disclosed herein.
The invention synchronizes video and audio signals that had originally been acquired as synchronized signals but had become unsynchronized by processing in videophone, teleconferencing, multimedia, and other visual communication techniques. Delay of the video signal relative to the audio signal, which results in lack of synchronism, can occur during encoding in the transmitter, transmission, and decoding in the receiver.
While embodiments of the invention have been described in detail, it will be evident to those skilled in the art that the invention may be embodied otherwise without departing from its spirit and scope.
Claims (12)
1. A method of processing a stream of decoded and unsynchronized audio and video signals of a speaker, comprising:
memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;
fetching visemes corresponding to phonemes in the audio signal; and imparting a synchronism to the decoded video and audio signals by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;
fetching visemes corresponding to phonemes in the audio signal; and imparting a synchronism to the decoded video and audio signals by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
2. A method as in claim 1, wherein the imparting step includes extracting visemes from the decoded video signal and updating the memory of the visemes with the extracted visemes.
3. A method as in claim 1, wherein the fetching step includes fetching visemes of the lip movement.
4. A communication method, comprising:
transmitting and receiving streams of synchronized video and audio signals at a multiplicity of stations;
at each of said stations, decoding said signals at a plurality of said multiplicity of stations:
memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;
fetching from the plurality visemes visemes corresponding to phonemes in the audio signal; and imparting a synchronism to the video signal and audio signal by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
transmitting and receiving streams of synchronized video and audio signals at a multiplicity of stations;
at each of said stations, decoding said signals at a plurality of said multiplicity of stations:
memorizing a plurality of visemes corresponding to phonemes in the decoded audio signal;
fetching from the plurality visemes visemes corresponding to phonemes in the audio signal; and imparting a synchronism to the video signal and audio signal by applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
5. A method as in claim 4, wherein the imparting step includes extracting visemes from the decoded video signal and updating the memory of the visemes with the extracted visemes.
6. A method as in claim 4, wherein the fetching step includes fetching visemes of lip movement.
7. An apparatus for processing a stream of decoded audio and video signals of a speaker, comprising:
means for memorizing a plurality of visemes corresponding to phonemes in the audio signal;
means, coupled to said means for memorizing, for imparting a synchronism to the video signal and audio signal by fetching from the plurality visemes visemes corresponding to phonemes in the audio signal and applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
means for memorizing a plurality of visemes corresponding to phonemes in the audio signal;
means, coupled to said means for memorizing, for imparting a synchronism to the video signal and audio signal by fetching from the plurality visemes visemes corresponding to phonemes in the audio signal and applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
8. An apparatus as in claim 7, wherein the decoding means includes means for decoding the audio signal from the stream and decoding the video signal from the stream.
9. An apparatus as in claim 7, wherein the imparting means includes means for fetching visemes of the lip movement.
10. A communication system, comprising:
a plurality of stations each having means for transmitting and receiving video and audio signals;
a communications network linking said stations;
means, at said stations, for decoding the signals;
at a plurality of said stations:
means coupled to said decoding means for memorizing a plurality of visemes corresponding to phonemes in the audio signal;
means, coupled to said respective decoding means at each station, for imparting a synchronism to the video signal and audio signal, by fetching from the plurality visemes visemes corresponding to phonemes in the audio signal and applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
a plurality of stations each having means for transmitting and receiving video and audio signals;
a communications network linking said stations;
means, at said stations, for decoding the signals;
at a plurality of said stations:
means coupled to said decoding means for memorizing a plurality of visemes corresponding to phonemes in the audio signal;
means, coupled to said respective decoding means at each station, for imparting a synchronism to the video signal and audio signal, by fetching from the plurality visemes visemes corresponding to phonemes in the audio signal and applying the fetched visemes to the unsynchronized video signal of the stream in synchronism with corresponding phonemes in the audio signal of the stream.
11. An apparatus as in claim 10, wherein said decoding means includes means for decoding the audio signal from the stream and decoding the video signal from the stream.
12. An apparatus as in claim 10, wherein the imparting means includes means for fetching visemes of lip movement.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US263,271 | 1988-10-27 | ||
US08/263,271 US5608839A (en) | 1994-03-18 | 1994-06-21 | Sound-synchronized video system |
Publications (1)
Publication Number | Publication Date |
---|---|
CA2149068A1 true CA2149068A1 (en) | 1995-12-22 |
Family
ID=23001067
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CA002149068A Abandoned CA2149068A1 (en) | 1994-06-21 | 1995-05-10 | Sound-synchronized video system |
Country Status (4)
Country | Link |
---|---|
US (1) | US5608839A (en) |
EP (1) | EP0689362A3 (en) |
JP (1) | JPH0823530A (en) |
CA (1) | CA2149068A1 (en) |
Families Citing this family (61)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5878396A (en) * | 1993-01-21 | 1999-03-02 | Apple Computer, Inc. | Method and apparatus for synthetic speech in facial animation |
AU5027796A (en) | 1995-03-07 | 1996-09-23 | Interval Research Corporation | System and method for selective recording of information |
US5907351A (en) * | 1995-10-24 | 1999-05-25 | Lucent Technologies Inc. | Method and apparatus for cross-modal predictive coding for talking head sequences |
US5880788A (en) * | 1996-03-25 | 1999-03-09 | Interval Research Corporation | Automated synchronization of video image sequences to new soundtracks |
EP0890168B1 (en) * | 1996-03-26 | 2002-09-04 | BRITISH TELECOMMUNICATIONS public limited company | Image synthesis |
US5943648A (en) * | 1996-04-25 | 1999-08-24 | Lernout & Hauspie Speech Products N.V. | Speech signal distribution system providing supplemental parameter associated data |
US5963217A (en) * | 1996-11-18 | 1999-10-05 | 7Thstreet.Com, Inc. | Network conference system using limited bandwidth to generate locally animated displays |
US5893062A (en) | 1996-12-05 | 1999-04-06 | Interval Research Corporation | Variable rate video playback with synchronized audio |
US6263507B1 (en) * | 1996-12-05 | 2001-07-17 | Interval Research Corporation | Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data |
KR100236974B1 (en) | 1996-12-13 | 2000-02-01 | 정선종 | Sync. system between motion picture and text/voice converter |
US5884267A (en) * | 1997-02-24 | 1999-03-16 | Digital Equipment Corporation | Automated speech alignment for image synthesis |
US6208356B1 (en) * | 1997-03-24 | 2001-03-27 | British Telecommunications Public Limited Company | Image synthesis |
KR100240637B1 (en) * | 1997-05-08 | 2000-01-15 | 정선종 | Syntax for tts input data to synchronize with multimedia |
GB9715516D0 (en) * | 1997-07-22 | 1997-10-01 | Orange Personal Comm Serv Ltd | Data communications |
US7366670B1 (en) * | 1997-08-05 | 2008-04-29 | At&T Corp. | Method and system for aligning natural and synthetic video to speech synthesis |
US6567779B1 (en) * | 1997-08-05 | 2003-05-20 | At&T Corp. | Method and system for aligning natural and synthetic video to speech synthesis |
US7392190B1 (en) * | 1997-11-07 | 2008-06-24 | At&T Corp. | Coarticulation method for audio-visual text-to-speech synthesis |
US6112177A (en) * | 1997-11-07 | 2000-08-29 | At&T Corp. | Coarticulation method for audio-visual text-to-speech synthesis |
US7117155B2 (en) * | 1999-09-07 | 2006-10-03 | At&T Corp. | Coarticulation method for audio-visual text-to-speech synthesis |
JP3688879B2 (en) * | 1998-01-30 | 2005-08-31 | 株式会社東芝 | Image recognition apparatus, image recognition method, and recording medium therefor |
JPH11219446A (en) * | 1998-02-03 | 1999-08-10 | Matsushita Electric Ind Co Ltd | Video/sound reproducing system |
US6449595B1 (en) * | 1998-03-11 | 2002-09-10 | Microsoft Corporation | Face synthesis system and methodology |
US6272231B1 (en) * | 1998-11-06 | 2001-08-07 | Eyematic Interfaces, Inc. | Wavelet-based facial motion capture for avatar animation |
IT1314671B1 (en) * | 1998-10-07 | 2002-12-31 | Cselt Centro Studi Lab Telecom | PROCEDURE AND EQUIPMENT FOR THE ANIMATION OF A SYNTHESIZED HUMAN FACE MODEL DRIVEN BY AN AUDIO SIGNAL. |
JP2003503925A (en) * | 1999-06-24 | 2003-01-28 | コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ | Post-synchronization of information streams |
US6594629B1 (en) * | 1999-08-06 | 2003-07-15 | International Business Machines Corporation | Methods and apparatus for audio-visual speech detection and recognition |
US7630897B2 (en) * | 1999-09-07 | 2009-12-08 | At&T Intellectual Property Ii, L.P. | Coarticulation method for audio-visual text-to-speech synthesis |
US7155735B1 (en) | 1999-10-08 | 2006-12-26 | Vulcan Patents Llc | System and method for the broadcast dissemination of time-ordered data |
US6766299B1 (en) * | 1999-12-20 | 2004-07-20 | Thrillionaire Productions, Inc. | Speech-controlled animation system |
US6526395B1 (en) * | 1999-12-31 | 2003-02-25 | Intel Corporation | Application of personality models and interaction with synthetic characters in a computing system |
US7038683B1 (en) * | 2000-01-28 | 2006-05-02 | Creative Technology Ltd. | Audio driven self-generating objects |
US6757682B1 (en) | 2000-01-28 | 2004-06-29 | Interval Research Corporation | Alerting users to items of current interest |
US6813607B1 (en) * | 2000-01-31 | 2004-11-02 | International Business Machines Corporation | Translingual visual speech synthesis |
US6539354B1 (en) | 2000-03-24 | 2003-03-25 | Fluent Speech Technologies, Inc. | Methods and devices for producing and using synthetic visual speech based on natural coarticulation |
IT1320002B1 (en) * | 2000-03-31 | 2003-11-12 | Cselt Centro Studi Lab Telecom | PROCEDURE FOR THE ANIMATION OF A SYNTHESIZED VOLTOHUMAN MODEL DRIVEN BY AN AUDIO SIGNAL. |
US7149686B1 (en) * | 2000-06-23 | 2006-12-12 | International Business Machines Corporation | System and method for eliminating synchronization errors in electronic audiovisual transmissions and presentations |
US6778252B2 (en) * | 2000-12-22 | 2004-08-17 | Film Language | Film language |
US6614466B2 (en) * | 2001-02-22 | 2003-09-02 | Texas Instruments Incorporated | Telescopic reconstruction of facial features from a speech pattern |
US7069214B2 (en) * | 2001-02-26 | 2006-06-27 | Matsushita Electric Industrial Co., Ltd. | Factorization for generating a library of mouth shapes |
US7058889B2 (en) * | 2001-03-23 | 2006-06-06 | Koninklijke Philips Electronics N.V. | Synchronizing text/visual information with audio playback |
GB2378776A (en) * | 2001-05-22 | 2003-02-19 | Canon Kk | Apparatus and method for managing a multi-modal interface in which the inputs feedback on each other |
CA2393014A1 (en) * | 2001-07-11 | 2003-01-11 | Genlyte Thomas Group Llc | Switch/power drop unit for modular wiring system |
US7085259B2 (en) * | 2001-07-31 | 2006-08-01 | Comverse, Inc. | Animated audio messaging |
US20030058932A1 (en) * | 2001-09-24 | 2003-03-27 | Koninklijke Philips Electronics N.V. | Viseme based video coding |
CN1313979C (en) * | 2002-05-03 | 2007-05-02 | 三星电子株式会社 | Apparatus and method for generating 3-D cartoon |
US7827034B1 (en) * | 2002-11-27 | 2010-11-02 | Totalsynch, Llc | Text-derived speech animation tool |
WO2004100128A1 (en) * | 2003-04-18 | 2004-11-18 | Unisay Sdn. Bhd. | System for generating a timed phomeme and visem list |
GB0324536D0 (en) * | 2003-10-21 | 2003-11-26 | Mitel Networks Corp | Detecting acoustic echoes using microphone arrays |
US20060031879A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of news-related broadcasted or streamed multimedia content |
US20060031916A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of broadcasted or streamed multimedia content |
US20060031885A1 (en) * | 2004-04-30 | 2006-02-09 | Vulcan Inc. | Management and non-linear presentation of music-related broadcasted or streamed multimedia content |
WO2010068151A1 (en) * | 2008-12-08 | 2010-06-17 | Telefonaktiebolaget L M Ericsson (Publ) | Device and method for synchronizing received audio data with video data |
US8751228B2 (en) * | 2010-11-04 | 2014-06-10 | Microsoft Corporation | Minimum converted trajectory error (MCTE) audio-to-video engine |
WO2013086027A1 (en) * | 2011-12-06 | 2013-06-13 | Doug Carson & Associates, Inc. | Audio-video frame synchronization in a multimedia stream |
CN104115160A (en) * | 2011-12-19 | 2014-10-22 | 诺基亚公司 | A method and apparatus for creating and displaying a face sketch avatar |
US8655152B2 (en) | 2012-01-31 | 2014-02-18 | Golden Monkey Entertainment | Method and system of presenting foreign films in a native language |
US10460732B2 (en) * | 2016-03-31 | 2019-10-29 | Tata Consultancy Services Limited | System and method to insert visual subtitles in videos |
WO2019161229A1 (en) | 2018-02-15 | 2019-08-22 | DMAI, Inc. | System and method for reconstructing unoccupied 3d space |
CN112204564A (en) * | 2018-02-15 | 2021-01-08 | 得麦股份有限公司 | System and method for speech understanding via integrated audio and visual based speech recognition |
WO2019161200A1 (en) | 2018-02-15 | 2019-08-22 | DMAI, Inc. | System and method for conversational agent via adaptive caching of dialogue tree |
CN113242361B (en) * | 2021-07-13 | 2021-09-24 | 腾讯科技(深圳)有限公司 | Video processing method and device and computer readable storage medium |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4104625A (en) * | 1977-01-12 | 1978-08-01 | Atari, Inc. | Apparatus for providing facial image animation |
US4260229A (en) * | 1978-01-23 | 1981-04-07 | Bloomstein Richard W | Creating visual images of lip movements |
US4305131A (en) * | 1979-02-05 | 1981-12-08 | Best Robert M | Dialog between TV movies and human viewers |
US4757541A (en) * | 1985-11-05 | 1988-07-12 | Research Triangle Institute | Audio visual speech recognition |
GB8528143D0 (en) * | 1985-11-14 | 1985-12-18 | British Telecomm | Image encoding & synthesis |
JP2644789B2 (en) * | 1987-12-18 | 1997-08-25 | 富士通株式会社 | Image transmission method |
JP2518683B2 (en) * | 1989-03-08 | 1996-07-24 | 国際電信電話株式会社 | Image combining method and apparatus thereof |
JP3023961B2 (en) * | 1989-10-04 | 2000-03-21 | 三菱電機株式会社 | Encoder and decoder |
GB9019829D0 (en) * | 1990-09-11 | 1990-10-24 | British Telecomm | Speech analysis and image synthesis |
US5313522A (en) * | 1991-08-23 | 1994-05-17 | Slager Robert P | Apparatus for generating from an audio signal a moving visual lip image from which a speech content of the signal can be comprehended by a lipreader |
WO1993018505A1 (en) * | 1992-03-02 | 1993-09-16 | The Walt Disney Company | Voice transformation system |
-
1994
- 1994-06-21 US US08/263,271 patent/US5608839A/en not_active Expired - Lifetime
-
1995
- 1995-05-10 CA CA002149068A patent/CA2149068A1/en not_active Abandoned
- 1995-06-14 EP EP95304087A patent/EP0689362A3/en not_active Withdrawn
- 1995-06-21 JP JP7176873A patent/JPH0823530A/en active Pending
Also Published As
Publication number | Publication date |
---|---|
JPH0823530A (en) | 1996-01-23 |
US5608839A (en) | 1997-03-04 |
EP0689362A2 (en) | 1995-12-27 |
EP0689362A3 (en) | 1996-06-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US5608839A (en) | Sound-synchronized video system | |
US6330023B1 (en) | Video signal processing systems and methods utilizing automated speech analysis | |
US5890120A (en) | Matching, synchronization, and superposition on orginal speaking subject images of modified signs from sign language database corresponding to recognized speech segments | |
KR0180543B1 (en) | Image signal encoding and communicating apparatus using means for extracting particular portions of an object image | |
Chen et al. | Audio-visual integration in multimodal communication | |
US20040141093A1 (en) | Post-synchronizing an information stream | |
EP0993197B1 (en) | A method and an apparatus for the animation, driven by an audio signal, of a synthesised model of human face | |
KR100872732B1 (en) | Video communication system using model-based coding and prioritization techniques | |
US5426460A (en) | Virtual multimedia service for mass market connectivity | |
US6014625A (en) | Method and apparatus for producing lip-movement parameters in a three-dimensional-lip-model | |
CN101018314A (en) | The apparatus and method for image communication of mobile communication terminal | |
JPH05153581A (en) | Face picture coding system | |
JP2004533666A (en) | Communications system | |
JP2009266240A (en) | Method and device for adjusting video and speech, computer readable medium, and method for combining data and decoder | |
CA2162199A1 (en) | Acoustic-assisted image processing | |
Chen et al. | Lip synchronization using speech-assisted video processing | |
Chen et al. | Speech-assisted video processing: Interpolation and low-bitrate coding | |
EP0893923A1 (en) | Video communication system | |
KR100229538B1 (en) | Apparatus and method for encoding a facial movement | |
Chou et al. | Speech recognition for image animation and coding | |
Chen et al. | Lip synchronization in talking head video utilizing speech information | |
KR20040037099A (en) | Viseme based video coding | |
JP2644789B2 (en) | Image transmission method | |
Chen et al. | Speech-assisted lip synchronization in audio-visual communications | |
JPH01190187A (en) | Picture transmission system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
EEER | Examination request | ||
FZDE | Discontinued | ||
FZDE | Discontinued |
Effective date: 19990510 |