US20010011218A1 - A system and apparatus for recognizing speech - Google Patents
A system and apparatus for recognizing speech Download PDFInfo
- Publication number
- US20010011218A1 US20010011218A1 US09/804,041 US80404101A US2001011218A1 US 20010011218 A1 US20010011218 A1 US 20010011218A1 US 80404101 A US80404101 A US 80404101A US 2001011218 A1 US2001011218 A1 US 2001011218A1
- Authority
- US
- United States
- Prior art keywords
- active
- speech
- arcs
- states
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/28—Constructional details of speech recognition systems
- G10L15/34—Adaptation of a single recogniser for parallel processing, e.g. by use of multiple processors or cloud computing
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
Definitions
- the present invention relates to speech recognition systems and more particularly to a method and apparatus for recognizing speech using a general purpose shared memory multiprocessor machine.
- Speech recognizers also known as speech-to-text systems or automatic speech recognition (ASR) systems, identify words and produce a textual representation of a received speech signal.
- ASR automatic speech recognition
- typical speech recognizers break down human speech into several distinct layers.
- a phoneme for example, is the smallest unit of speech that differentiates utterances in a given language or dialect. However, a single phoneme may be pronounced differently depending on how it is used in a word or depending on the speaker.
- a context dependent unit is an acoustic realization of a phoneme as manifested in a particular context. These units combine to form words which together combine to form sentences, thereby creating the basic structure of human speech.
- a language model maps these basic speech sounds into sentences.
- a typical speech recognizer includes computer hardware and software which identifies spoken speech signals and evaluates the signal with respect to a language model to obtain a textual representation of what the speaker said.
- One type of speech recognizer is an isolated word recognition system which requires a speaker to pause after each spoken word so that the recognizer can identify each word in isolation. However, the rate at which speech can be inputted and processed in these recognizers is reduced and using such a system is unnatural to the speaker.
- Another type of speech recognizer is a continuous speech recognition system which allows a user to speak normally with no pauses in-between words. A continuous speech system allows a more natural speech flow, but because it is more difficult to distinguish where a particular word ends and where the next word begins, a continuous speech recognition system and the algorithm running on this type of system are complex.
- a language model and a speech signal are inputted into a recognizer.
- a language model consists of, for example, one or more models of context dependent units having probability distributions associated therewith, models that map context dependent units to words, and models that map words to sentences.
- the speech signal is partitioned into a plurality of speech frames which may contain a portion of or a complete phone. Each frame is evaluated with respect to a subset of the context dependent phone models. The results of this process are then used to progress through the higher levels of the language model. This process continues until the recognizer processes all the speech frames in an utterance. Because of the number of calculations, associated complex processing, and the need to run in a real-time environment, existing speech recognizers are limited to isolated word recognition or sacrifice accuracy to obtain real-time performance. In addition, current speech recognizers have models that are hard-coded into the system making speech recognition possible for only limited vocabularies.
- Special-purpose machines allow speech recognizers to achieve real-time or near real-time processing capability.
- Some special-purpose machines have been built that are specially designed to take advantage of parallelism to do speech recognition. An example is described in K. A. Wen and J. F. Wang, “Efficient computing methods for parallel processing: An implementation of the Viterbi algorithm,” Computers Math. Applic., 17 (12) 1989, pages 1511-1521.
- these machines are not suitable for recognition of large-vocabulary continuous speech because they do not have the necessary generality to accommodate these large vocabularies.
- a drawback associated with these special purpose machines is that they are hard-coded with a particular language model and therefore can only be used for a particular recognition task.
- the present invention meets the needs and avoids the disadvantages and drawbacks of existing speech recognition systems by providing a speaker independent continuous speech recognition method for recognizing a variety of speech inputs in real time.
- a signal corresponding to a plurality of speech frames is received.
- a language model is received in a general purpose shared memory machine having a plurality of processors.
- the language model is an implicit description of a graph consisting of a plurality of states and arcs.
- the graph and the speech input are processed in parallel using the plurality of processors.
- FIG. 1 is a general block diagram of a speech recognizer in an embodiment of the present invention.
- FIG. 2 shows a general flowchart illustrating the method in accordance with the present invention.
- FIG. 3 is a portion of an implicit graph used in accordance with the present invention.
- FIG. 4 is a more detailed flowchart illustrating the processing of arcs and states.
- FIG. 5 is sample high-level code for processing a frame in the present invention.
- FIG. 6 is sample high-level code illustrating the likelihood calculation in the present invention.
- FIG. 7 is sample high-level code illustrating the handling of the hash table for multi-threading the composition of the FSM portion of the algorithm.
- the parallel speech recognizer in accordance with the present invention utilizes a sequential recognizer.
- improvements made in sequential speech recognition such as phone modeling, likelihood calculations, grammar representations, etc., can be applied to the parallel recognizer of the present invention.
- the present invention utilizes the two-level Viterbi search algorithm as described in C. -H. Lee and L. R. Rabiner, “A Frame-Synchronous Network Search Algorithm for Connected Word Recognition”, IEEE Transactions on Acoustics, Speech, Signal Processing, Vol. 37, No. 11, November 1989, a copy of which is included as Appendix A and is made a part of this application.
- the two-level Viterbi search algorithm operates at the boundary between the Hidden Markov Model (HMM) layer where signals representing speech frames are matched with HMM's that represent context dependent units as well as an upper layer which represents the mapping of context dependent units to sentences.
- HMM Hidden Markov Model
- the mapping of context dependent units to sentences is done using on-demand composition of Finite State Transducers (FSM) as described in Mohri et al., “Weighted Automata in Text and Speech Processing”, Proceedings of the ECAI 96 Workshop, ECAI, 1996, a copy of which is included as Appendix B and is made a part of this application.
- FSM Finite State Transducers
- FIG. 1 is a general block diagram of a parallel speech recognizer 10 in accordance with the present invention and is used to illustrate the processing relationship between multiple processors 1, 2-N and a shared memory 35 .
- An input speech signal in analog form, is received by an input device 20 .
- the input signal is digitally sampled, for example every 10 milliseconds, which may occur at the input device 20 in machine 30 or by an alternative receiving device (not shown).
- Each sample undergoes spectral analysis and other forms of signal processing known in the art resulting in a parametric representation of the input signal as a frame or vector of real numbers.
- a language model is also inputted to recognizer 10 .
- the language model contains models of the basic speech units and an implicit description of a graph, consisting of states and arcs, that serves to map basic speech units to sentences.
- the recognizer in accordance with the present invention is capable of receiving different language models and is not limited to one particular model as found in prior special-purpose parallel recognizers.
- the language model used can, for example, be made-up of: one or more models of context dependent units which have probability distributions associated therewith; models that map context dependent units to words; and models that map words to sentences.
- a shared memory multiprocessor machine 30 used to parallel process the Viterbi search algorithm includes, at its most basic level, an interface bus 25 , microprocessors 1, 2-N and memory 35 .
- the speech algorithm is housed within multiprocessor machine 30 and run in parallel using processors 1,2-N to produce a representation of the signal received on line 15 .
- processors 1,2-N employed in machine 30 effects the speed and efficiency of the speech recognizer in processing received input signals.
- the output is received by interface device 40 .
- the outputs can be transmitted to a display apparatus, speech understanding tool or further processed depending upon the eventual use of the output.
- the machine 30 is a general purpose shared memory machine having a plurality of processors.
- Machine 30 is considered a general purpose machine in that it does not require hard-wiring or hard-coding for a particular type of language model or algorithm.
- the recognizer is capable of processing increased vocabulary sizes by inputting different language models unlike prior parallel speech recognizers which used hard-wired special purpose machines to parallel process speech algorithms.
- the speech recognition system maps between an input speech waveform, context dependent units, words and sentences to produce a textual representation of the input signal.
- This general process flow is best illustrated in FIG. 2.
- a speech signal at step 100 is inputted to a signal processor at step 110 .
- the signal may be an analog signal in which case the signal processor digitally samples the signal and produces a frame or vector of real numbers.
- a language model is also inputted at step 115 where the model is an implicit description of a graph consisting of a plurality of states and arcs.
- the system is initialized at step 120 and a determination is made, at step 130 , if any speech frames remain to be processed by the system. If no frames remain, the process is complete and the process terminates at step 135 .
- step 140 each frame is processed in parallel in multiprocessor machine 30 .
- step 145 a clean-up step is performed to validate that the processing for a particular frame assigned to a thread is complete and the process returns to step 130 and continues for subsequent frames.
- the recognizer processes input speech frames sequentially, however, the processing associated with each frame is performed in parallel as will be described in detail below.
- each state S 0 and S 1 in the language model has associated therewith a plurality of incoming arcs, illustrated for example by arc a 0 and outgoing arcs illustrated for example by arc a 1 .
- the state S 1 from which arc a 1 originates is referred to as the source state and arc a 1 , which flows from the source state S 1 , is referred to as an outgoing arc.
- the number of states and arcs in the implicit graph has been limited to these few for explanation purposes only.
- a speech frame at step 146 is mapped to the input language model having a plurality of states and arcs.
- the active arc list is empty and the active state list contains only the start state of the graph.
- Each thread in the multi-processor configuration is assigned a subset of the active state set. This allocation of states determines the structure of the parallel algorithm.
- Each thread will process approximately N/P states where N is the number of active states and P corresponds to the number of processors.
- Each state is assigned to a single thread which is determined by taking the state number mod P.
- the active arcs originating from an active state are assigned to the same thread as the associated state. In this manner, a particular thread processes an active state subset as well as its associated active arc subset to take advantage of the multiprocessor parallel configuration. This technique enhances data locality.
- the active arc set is updated based on the active states from the graph.
- Each thread performs the likelihood calculation for each arc assigned to that particular thread as depicted at step 160 .
- Each thread computes the minimum cost for its active arc subset at step 160 and participates in the computation of the global minimum cost at step 165 . For example, this is done through the use of a vector that stores the maximum likelihood for each thread as well as using a sequential loop to compute the final minimum cost value.
- FIG. 5 illustrates sample high level code for processing a frame.
- the calculation for determining the minimum cost at step 160 is best explained with reference to FIG. 3.
- the process calculates the likelihood costs of the active arcs, for example in FIG. 3 arcs a 0 and a 1 .
- the state costs associated with states S 1 and S 3 and the likelihood costs associated with arcs a 1 and a 3 have already been calculated.
- a cost associated with arc a 1 is determined by adding the likelihood cost of arc a 1 and the state cost for state S 1 which is a source state for arc a 1 .
- the state cost for state S 2 is calculated by determining the minimum of the costs of the incoming arcs a 1 and a 3 associated with state S 2 .
- the local minimum cost for the thread is the minimum cost over all the states reached by arcs in the active arc subset for that thread.
- the procedure for calculating likelihoods avoids recomputing the likelihood of a frame matching a particular context dependent unit by remembering the calculations that it has performed in the past.
- This technique which is known in the art as “memo-ization” or “caching,” reduces the cost of computing multiple likelihoods. It is implemented using a bit vector that indicates whether a particular likelihood has been calculated and a result vector that holds previously computed likelihoods.
- the present invention takes advantage of a property of the computation, namely that the calculation of a particular likelihood will always produce the same value, and a property that many shared memory machines employ, namely that writes from a single thread are seen in order by other threads.
- FIG. 6 illustrates an example of high-level code for performing the likelihood calculation in accordance with the present invention.
- the arcs with costs that are not within the range of the minimum cost determined in step 165 of FIG. 4 plus a predetermined threshold value, which is an input to the recognizer, are pruned at step 170 .
- Each thread goes through the active arcs assigned to it pruning the arcs if their associated costs fall outside the computed range.
- the new active states are determined at step 175 using the results from step 170 .
- An arc is completed if the likelihood calculation for the most recent frame determines that there was a match with the underlying context dependent unit and the arc's cost is within the computed range.
- Step 175 adds the destination states of completed arcs to the active state set.
- the FSM layer is queried at step 180 to determine the transitions out of newly active states using on-demand composition of the FSMs.
- the thread that determines that a particular state becomes active may not be the thread that is assigned to that next state.
- This computation is performed by first storing the states which an arc designates as active.
- This storage data structure is in the form of a two dimensional array wherein each element in the array contains a linked list.
- a state “S” is added to the linked list at location [T, S mod P] in the array by Thread T, if that thread identifies the state as newly active.
- a thread, “T” queries the FSM layer for the states in the lists at locations [1 . . . P, T] of the array and adds them to its active state subset.
- the multi-threading of the FSM library is centered on the routines for on-demand composition of automata. Two or more automata are combined to produce a composed automaton, whose states correspond to tuples with a tuple containing one state from each of the input automata.
- These routines make use of a hash table which maps from tuples of states to state numbers in the composed automaton.
- different threads need to update the hash table simultaneously which requires careful synchronization to avoid data contention. Locking access to the hash table as a whole is an inadequate solution, as too much time would be spent waiting for the lock.
- FIG. 7 illustrates high-level code for handling the hash table for multi-threading the composition of the FSMs portion of the algorithm.
- Table 1 illustrates the average run time over 300 sentences for the 20,000 word Advanced Projects Research Agency (ARPA) North American Business News (NAB) task. Number of Processors Sequential 1 2 4 8 12 16 Average Run 35.1 33.7 20.4 12.3 8.4 7.8 7.6 Time Increase Speed 1.0 1.0 1.7 2.8 4.2 4.5 4.6 Over Sequential Relative to Real- 3.9 3.7 2.3 1.4 0.9 0.9 0.8 Time
- ARPA Advanced Projects Research Agency
- NAB North American Business News
- the column labeled Sequential contains the run times using a sequential recognizer on one processor of the Power Challenge XL. Columns labeled 1, 2, 4, 8, 12 and 16 denote the results from using the indicated number of processors.
- the run time for the parallel recognizer in accordance with the present invention using 8 processors provides real-time performance. As can be seen from Table 1, the recognition speed drops off as more processors are used which is due, in part, by synchronization at locks on shared data structures and at barriers between phases of the Viterbi algorithm. The response time improvements with respect to previous sequential algorithms are achieved based on the parallelization of the Viterbi search, likelihood calculations and the on-demand FSM composition.
- the speech recognition system in accordance with the present invention uses a general purpose shared memory multiprocessor machine to perform continuous parallel speech recognition.
- the system receives a language model as an input thereby accommodating larger vocabularies and complex speech patterns while using the same underlying algorithm.
Abstract
Description
- The present invention relates to speech recognition systems and more particularly to a method and apparatus for recognizing speech using a general purpose shared memory multiprocessor machine.
- Speech recognizers, also known as speech-to-text systems or automatic speech recognition (ASR) systems, identify words and produce a textual representation of a received speech signal. In order to accomplish this, typical speech recognizers break down human speech into several distinct layers. A phoneme, for example, is the smallest unit of speech that differentiates utterances in a given language or dialect. However, a single phoneme may be pronounced differently depending on how it is used in a word or depending on the speaker. A context dependent unit is an acoustic realization of a phoneme as manifested in a particular context. These units combine to form words which together combine to form sentences, thereby creating the basic structure of human speech. A language model maps these basic speech sounds into sentences.
- A typical speech recognizer includes computer hardware and software which identifies spoken speech signals and evaluates the signal with respect to a language model to obtain a textual representation of what the speaker said. One type of speech recognizer is an isolated word recognition system which requires a speaker to pause after each spoken word so that the recognizer can identify each word in isolation. However, the rate at which speech can be inputted and processed in these recognizers is reduced and using such a system is unnatural to the speaker. Another type of speech recognizer is a continuous speech recognition system which allows a user to speak normally with no pauses in-between words. A continuous speech system allows a more natural speech flow, but because it is more difficult to distinguish where a particular word ends and where the next word begins, a continuous speech recognition system and the algorithm running on this type of system are complex.
- A language model and a speech signal are inputted into a recognizer. A language model consists of, for example, one or more models of context dependent units having probability distributions associated therewith, models that map context dependent units to words, and models that map words to sentences. The speech signal is partitioned into a plurality of speech frames which may contain a portion of or a complete phone. Each frame is evaluated with respect to a subset of the context dependent phone models. The results of this process are then used to progress through the higher levels of the language model. This process continues until the recognizer processes all the speech frames in an utterance. Because of the number of calculations, associated complex processing, and the need to run in a real-time environment, existing speech recognizers are limited to isolated word recognition or sacrifice accuracy to obtain real-time performance. In addition, current speech recognizers have models that are hard-coded into the system making speech recognition possible for only limited vocabularies.
- Special-purpose machines allow speech recognizers to achieve real-time or near real-time processing capability. Some special-purpose machines have been built that are specially designed to take advantage of parallelism to do speech recognition. An example is described in K. A. Wen and J. F. Wang, “Efficient computing methods for parallel processing: An implementation of the Viterbi algorithm,” Computers Math. Applic., 17 (12) 1989, pages 1511-1521. However, these machines are not suitable for recognition of large-vocabulary continuous speech because they do not have the necessary generality to accommodate these large vocabularies. A drawback associated with these special purpose machines is that they are hard-coded with a particular language model and therefore can only be used for a particular recognition task. Another disadvantage with these systems is that they are designed only for isolated word recognition and are not suitable for continuous speech recognition. Moreover, none of these systems has the flexibility for receiving a language model as an input that is composed of a number of layers which are combined on-the-fly or implicitly during recognition. Therefore, none of these special-purpose machines can be used for general-purpose recognition of large-vocabulary continuous speech. In addition, special-purpose machines are prohibitively expensive, and are usually limited to development by large corporations making accessibility to the general public virtually impossible.
- With the advancements in commercially available multi-processor systems, there is an opportunity to develop a continuous speech recognition system that uses a general purpose shared memory multiprocessor machine to perform continuous parallel speech recognition. There is also a need for a parallel speech recognizer that is capable of receiving a language model as an input so that much larger vocabularies as well as complex speech patterns can use the same underlying programming algorithm used for standard speech recognition tasks without requiring hard coding of a particular model.
- The present invention meets the needs and avoids the disadvantages and drawbacks of existing speech recognition systems by providing a speaker independent continuous speech recognition method for recognizing a variety of speech inputs in real time. A signal corresponding to a plurality of speech frames is received. A language model is received in a general purpose shared memory machine having a plurality of processors. The language model is an implicit description of a graph consisting of a plurality of states and arcs. The graph and the speech input are processed in parallel using the plurality of processors.
- FIG. 1 is a general block diagram of a speech recognizer in an embodiment of the present invention.
- FIG. 2 shows a general flowchart illustrating the method in accordance with the present invention.
- FIG. 3 is a portion of an implicit graph used in accordance with the present invention.
- FIG. 4 is a more detailed flowchart illustrating the processing of arcs and states.
- FIG. 5 is sample high-level code for processing a frame in the present invention.
- FIG. 6 is sample high-level code illustrating the likelihood calculation in the present invention.
- FIG. 7 is sample high-level code illustrating the handling of the hash table for multi-threading the composition of the FSM portion of the algorithm.
- The parallel speech recognizer in accordance with the present invention utilizes a sequential recognizer. By using a sequential recognizer, improvements made in sequential speech recognition, such as phone modeling, likelihood calculations, grammar representations, etc., can be applied to the parallel recognizer of the present invention. Accordingly, the present invention utilizes the two-level Viterbi search algorithm as described in C. -H. Lee and L. R. Rabiner, “A Frame-Synchronous Network Search Algorithm for Connected Word Recognition”, IEEE Transactions on Acoustics, Speech, Signal Processing, Vol. 37, No. 11, November 1989, a copy of which is included as Appendix A and is made a part of this application. The two-level Viterbi search algorithm operates at the boundary between the Hidden Markov Model (HMM) layer where signals representing speech frames are matched with HMM's that represent context dependent units as well as an upper layer which represents the mapping of context dependent units to sentences. The mapping of context dependent units to sentences is done using on-demand composition of Finite State Transducers (FSM) as described in Mohri et al., “Weighted Automata in Text and Speech Processing”, Proceedings of the ECAI 96 Workshop, ECAI, 1996, a copy of which is included as Appendix B and is made a part of this application.
- FIG. 1 is a general block diagram of a
parallel speech recognizer 10 in accordance with the present invention and is used to illustrate the processing relationship betweenmultiple processors 1, 2-N and a sharedmemory 35. An input speech signal, in analog form, is received by aninput device 20. The input signal is digitally sampled, for example every 10 milliseconds, which may occur at theinput device 20 inmachine 30 or by an alternative receiving device (not shown). Each sample undergoes spectral analysis and other forms of signal processing known in the art resulting in a parametric representation of the input signal as a frame or vector of real numbers. A language model is also inputted to recognizer 10. The language model contains models of the basic speech units and an implicit description of a graph, consisting of states and arcs, that serves to map basic speech units to sentences. The recognizer in accordance with the present invention is capable of receiving different language models and is not limited to one particular model as found in prior special-purpose parallel recognizers. The language model used can, for example, be made-up of: one or more models of context dependent units which have probability distributions associated therewith; models that map context dependent units to words; and models that map words to sentences. - A shared
memory multiprocessor machine 30, used to parallel process the Viterbi search algorithm includes, at its most basic level, aninterface bus 25,microprocessors 1, 2-N andmemory 35. The speech algorithm is housed withinmultiprocessor machine 30 and run in parallel usingprocessors 1,2-N to produce a representation of the signal received on line 15. As will be clear from the description and processing results described below, the number ofmicroprocessors 1, 2-N, employed inmachine 30 effects the speed and efficiency of the speech recognizer in processing received input signals. Once the input signals have been processed by the recognizer, the output is received byinterface device 40. The outputs can be transmitted to a display apparatus, speech understanding tool or further processed depending upon the eventual use of the output. Themachine 30, in accordance with the present invention, is a general purpose shared memory machine having a plurality of processors.Machine 30 is considered a general purpose machine in that it does not require hard-wiring or hard-coding for a particular type of language model or algorithm. In this manner, the recognizer is capable of processing increased vocabulary sizes by inputting different language models unlike prior parallel speech recognizers which used hard-wired special purpose machines to parallel process speech algorithms. - The speech recognition system according to the present invention maps between an input speech waveform, context dependent units, words and sentences to produce a textual representation of the input signal. This general process flow is best illustrated in FIG. 2. A speech signal at
step 100 is inputted to a signal processor atstep 110. The signal may be an analog signal in which case the signal processor digitally samples the signal and produces a frame or vector of real numbers. A language model is also inputted atstep 115 where the model is an implicit description of a graph consisting of a plurality of states and arcs. The system is initialized atstep 120 and a determination is made, atstep 130, if any speech frames remain to be processed by the system. If no frames remain, the process is complete and the process terminates atstep 135. If there are remaining speech frames to be processed by the recognizer, the process continues to step 140 where each frame is processed in parallel inmultiprocessor machine 30. Atstep 145, a clean-up step is performed to validate that the processing for a particular frame assigned to a thread is complete and the process returns to step 130 and continues for subsequent frames. In this manner, the recognizer processes input speech frames sequentially, however, the processing associated with each frame is performed in parallel as will be described in detail below. - Turning briefly to FIG. 3, which illustrates an example of a portion of an implicit graph used in the present invention, each state S0 and S1 in the language model has associated therewith a plurality of incoming arcs, illustrated for example by arc a0 and outgoing arcs illustrated for example by arc a1. The state S1 from which arc a1 originates is referred to as the source state and arc a1, which flows from the source state S1, is referred to as an outgoing arc. The number of states and arcs in the implicit graph has been limited to these few for explanation purposes only.
- In FIG. 4, a speech frame at
step 146 is mapped to the input language model having a plurality of states and arcs. Initially, the active arc list is empty and the active state list contains only the start state of the graph. Each thread in the multi-processor configuration is assigned a subset of the active state set. This allocation of states determines the structure of the parallel algorithm. Each thread will process approximately N/P states where N is the number of active states and P corresponds to the number of processors. Each state is assigned to a single thread which is determined by taking the state number mod P. The active arcs originating from an active state are assigned to the same thread as the associated state. In this manner, a particular thread processes an active state subset as well as its associated active arc subset to take advantage of the multiprocessor parallel configuration. This technique enhances data locality. - At
step 150, the active arc set is updated based on the active states from the graph. Each thread performs the likelihood calculation for each arc assigned to that particular thread as depicted atstep 160. Each thread computes the minimum cost for its active arc subset atstep 160 and participates in the computation of the global minimum cost atstep 165. For example, this is done through the use of a vector that stores the maximum likelihood for each thread as well as using a sequential loop to compute the final minimum cost value. FIG. 5 illustrates sample high level code for processing a frame. - The calculation for determining the minimum cost at
step 160 is best explained with reference to FIG. 3. As previously stated, the process calculates the likelihood costs of the active arcs, for example in FIG. 3 arcs a0 and a1. The state costs associated with states S1 and S3 and the likelihood costs associated with arcs a1 and a3 have already been calculated. A cost associated with arc a1 is determined by adding the likelihood cost of arc a1 and the state cost for state S1 which is a source state for arc a1. The state cost for state S2 is calculated by determining the minimum of the costs of the incoming arcs a1 and a3 associated with state S2. The local minimum cost for the thread is the minimum cost over all the states reached by arcs in the active arc subset for that thread. - The procedure for calculating likelihoods avoids recomputing the likelihood of a frame matching a particular context dependent unit by remembering the calculations that it has performed in the past. This technique, which is known in the art as “memo-ization” or “caching,” reduces the cost of computing multiple likelihoods. It is implemented using a bit vector that indicates whether a particular likelihood has been calculated and a result vector that holds previously computed likelihoods. To multi-thread the likelihood calculation, the present invention takes advantage of a property of the computation, namely that the calculation of a particular likelihood will always produce the same value, and a property that many shared memory machines employ, namely that writes from a single thread are seen in order by other threads. Together these properties allow the present algorithm to avoid using any synchronization for the memorization vectors even though there is technically the potential for interference between two computations of the same likelihood. The first property guarantees that even if multiple threads try to compute the same likelihood concurrently, they are guaranteed to write the same value into the result vector. The second property, combined with a careful ordering, writes to the vectors (in particular, writing the result into the result vector before setting the bit in the bit vector) guarantees that if a computation finds a one in the bit vector, then it is guaranteed to find the correct likelihood in the result vector. FIG. 6 illustrates an example of high-level code for performing the likelihood calculation in accordance with the present invention.
- The arcs with costs that are not within the range of the minimum cost determined in
step 165 of FIG. 4 plus a predetermined threshold value, which is an input to the recognizer, are pruned atstep 170. Each thread goes through the active arcs assigned to it pruning the arcs if their associated costs fall outside the computed range. The new active states are determined atstep 175 using the results fromstep 170. An arc is completed if the likelihood calculation for the most recent frame determines that there was a match with the underlying context dependent unit and the arc's cost is within the computed range. Step 175 adds the destination states of completed arcs to the active state set. In addition, the FSM layer is queried atstep 180 to determine the transitions out of newly active states using on-demand composition of the FSMs. - Because the active arc calculations are assigned to a particular thread based on the state from which they originate, the thread that determines that a particular state becomes active may not be the thread that is assigned to that next state. This computation is performed by first storing the states which an arc designates as active. This storage data structure is in the form of a two dimensional array wherein each element in the array contains a linked list. A state “S” is added to the linked list at location [T, S mod P] in the array by Thread T, if that thread identifies the state as newly active. Once this data structure is built, a thread, “T”, queries the FSM layer for the states in the lists at locations [1 . . . P, T] of the array and adds them to its active state subset.
- The multi-threading of the FSM library is centered on the routines for on-demand composition of automata. Two or more automata are combined to produce a composed automaton, whose states correspond to tuples with a tuple containing one state from each of the input automata. These routines make use of a hash table which maps from tuples of states to state numbers in the composed automaton. However, different threads need to update the hash table simultaneously which requires careful synchronization to avoid data contention. Locking access to the hash table as a whole is an inadequate solution, as too much time would be spent waiting for the lock. Instead, the present invention uses one lock to manage a small collection of hash buckets which increases contention slightly, but decreases substantially the number of locks required in comparison to a one-lock per bucket implementation. Reordering the code to minimize the amount of time any thread holds a bucket lock further reduces contention of the hash table. FIG. 7 illustrates high-level code for handling the hash table for multi-threading the composition of the FSMs portion of the algorithm.
- The following results were achieved using the algorithm of the present invention on a Silicon Graphics Power Challenge XL multiprocessor, however the principles of the invention can be implemented on any shared memory machine having a plurality of microprocessors. Table 1 illustrates the average run time over 300 sentences for the 20,000 word Advanced Projects Research Agency (ARPA) North American Business News (NAB) task.
Number of Processors Sequential 1 2 4 8 12 16 Average Run 35.1 33.7 20.4 12.3 8.4 7.8 7.6 Time Increase Speed 1.0 1.0 1.7 2.8 4.2 4.5 4.6 Over Sequential Relative to Real- 3.9 3.7 2.3 1.4 0.9 0.9 0.8 Time - The column labeled Sequential contains the run times using a sequential recognizer on one processor of the Power Challenge XL. Columns labeled 1, 2, 4, 8, 12 and 16 denote the results from using the indicated number of processors. The run time for the parallel recognizer in accordance with the present invention using 8 processors provides real-time performance. As can be seen from Table 1, the recognition speed drops off as more processors are used which is due, in part, by synchronization at locks on shared data structures and at barriers between phases of the Viterbi algorithm. The response time improvements with respect to previous sequential algorithms are achieved based on the parallelization of the Viterbi search, likelihood calculations and the on-demand FSM composition.
- The speech recognition system in accordance with the present invention uses a general purpose shared memory multiprocessor machine to perform continuous parallel speech recognition. The system receives a language model as an input thereby accommodating larger vocabularies and complex speech patterns while using the same underlying algorithm.
Claims (30)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/804,041 US6374212B2 (en) | 1997-09-30 | 2001-03-13 | System and apparatus for recognizing speech |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/941,379 US6249761B1 (en) | 1997-09-30 | 1997-09-30 | Assigning and processing states and arcs of a speech recognition model in parallel processors |
US09/804,041 US6374212B2 (en) | 1997-09-30 | 2001-03-13 | System and apparatus for recognizing speech |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US08/941,379 Continuation US6249761B1 (en) | 1997-09-30 | 1997-09-30 | Assigning and processing states and arcs of a speech recognition model in parallel processors |
Publications (2)
Publication Number | Publication Date |
---|---|
US20010011218A1 true US20010011218A1 (en) | 2001-08-02 |
US6374212B2 US6374212B2 (en) | 2002-04-16 |
Family
ID=25476375
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US08/941,379 Expired - Lifetime US6249761B1 (en) | 1997-09-30 | 1997-09-30 | Assigning and processing states and arcs of a speech recognition model in parallel processors |
US09/804,041 Expired - Lifetime US6374212B2 (en) | 1997-09-30 | 2001-03-13 | System and apparatus for recognizing speech |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US08/941,379 Expired - Lifetime US6249761B1 (en) | 1997-09-30 | 1997-09-30 | Assigning and processing states and arcs of a speech recognition model in parallel processors |
Country Status (1)
Country | Link |
---|---|
US (2) | US6249761B1 (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020188450A1 (en) * | 2001-04-26 | 2002-12-12 | Siemens Aktiengesellschaft | Method and system for defining a sequence of sound modules for synthesis of a speech signal in a tonal language |
EP1361736A1 (en) * | 2002-05-08 | 2003-11-12 | Sap Ag | Method for recognition of speech information |
US20050251390A1 (en) * | 2002-02-04 | 2005-11-10 | Mark Catchpole | Speech recognition circuit using parallel processors |
US20080059190A1 (en) * | 2006-08-22 | 2008-03-06 | Microsoft Corporation | Speech unit selection using HMM acoustic models |
US20080059184A1 (en) * | 2006-08-22 | 2008-03-06 | Microsoft Corporation | Calculating cost measures between HMM acoustic models |
US20100114577A1 (en) * | 2006-06-27 | 2010-05-06 | Deutsche Telekom Ag | Method and device for the natural-language recognition of a vocal expression |
US20120245940A1 (en) * | 2009-12-08 | 2012-09-27 | Nuance Communications, Inc. | Guest Speaker Robust Adapted Speech Recognition |
US20130185430A1 (en) * | 2012-01-18 | 2013-07-18 | LineRate Systems, Inc. | Multi-level hash tables for socket lookups |
US20140223310A1 (en) * | 2013-02-01 | 2014-08-07 | Nuance Communications, Inc. | Correction Menu Enrichment with Alternate Choices and Generation of Choice Lists in Multi-Pass Recognition Systems |
Families Citing this family (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6249761B1 (en) * | 1997-09-30 | 2001-06-19 | At&T Corp. | Assigning and processing states and arcs of a speech recognition model in parallel processors |
EP1285434A1 (en) * | 2000-05-23 | 2003-02-26 | Thomson Licensing S.A. | Dynamic language models for speech recognition |
US20030009334A1 (en) * | 2001-07-03 | 2003-01-09 | International Business Machines Corporation | Speech processing board for high volume speech processing applications |
CA2359544A1 (en) * | 2001-10-22 | 2003-04-22 | Dspfactory Ltd. | Low-resource real-time speech recognition system using an oversampled filterbank |
US6990445B2 (en) * | 2001-12-17 | 2006-01-24 | Xl8 Systems, Inc. | System and method for speech recognition and transcription |
US20030115169A1 (en) * | 2001-12-17 | 2003-06-19 | Hongzhuan Ye | System and method for management of transcribed documents |
US7243060B2 (en) * | 2002-04-02 | 2007-07-10 | University Of Washington | Single channel sound separation |
US6879954B2 (en) * | 2002-04-22 | 2005-04-12 | Matsushita Electric Industrial Co., Ltd. | Pattern matching for large vocabulary speech recognition systems |
WO2003105126A1 (en) | 2002-06-05 | 2003-12-18 | Vas International, Inc. | Biometric identification system |
US20050187772A1 (en) * | 2004-02-25 | 2005-08-25 | Fuji Xerox Co., Ltd. | Systems and methods for synthesizing speech using discourse function level prosodic features |
JP5103907B2 (en) * | 2005-01-17 | 2012-12-19 | 日本電気株式会社 | Speech recognition system, speech recognition method, and speech recognition program |
US8200495B2 (en) | 2005-02-04 | 2012-06-12 | Vocollect, Inc. | Methods and systems for considering information about an expected response when performing speech recognition |
US7895039B2 (en) * | 2005-02-04 | 2011-02-22 | Vocollect, Inc. | Methods and systems for optimizing model adaptation for a speech recognition system |
US7865362B2 (en) | 2005-02-04 | 2011-01-04 | Vocollect, Inc. | Method and system for considering information about an expected response when performing speech recognition |
US7949533B2 (en) * | 2005-02-04 | 2011-05-24 | Vococollect, Inc. | Methods and systems for assessing and improving the performance of a speech recognition system |
US7827032B2 (en) | 2005-02-04 | 2010-11-02 | Vocollect, Inc. | Methods and systems for adapting a model for a speech recognition system |
JP4702936B2 (en) * | 2005-06-28 | 2011-06-15 | キヤノン株式会社 | Information processing apparatus, control method, and program |
ATE449403T1 (en) * | 2005-12-12 | 2009-12-15 | Gregory John Gadbois | MULTI-VOICE SPEECH RECOGNITION |
US20070156615A1 (en) * | 2005-12-29 | 2007-07-05 | Ali Davar | Method for training a classifier |
US8914290B2 (en) | 2011-05-20 | 2014-12-16 | Vocollect, Inc. | Systems and methods for dynamically improving user intelligibility of synthesized speech in a work environment |
EP2893435B1 (en) * | 2012-09-07 | 2019-05-08 | Carnegie Mellon University | Methods for hybrid gpu/cpu data processing |
US9594744B2 (en) * | 2012-11-28 | 2017-03-14 | Google Inc. | Speech transcription including written text |
US9978395B2 (en) | 2013-03-15 | 2018-05-22 | Vocollect, Inc. | Method and system for mitigating delay in receiving audio stream during production of sound from audio stream |
US10089989B2 (en) | 2015-12-07 | 2018-10-02 | Semiconductor Components Industries, Llc | Method and apparatus for a low power voice trigger device |
US10714121B2 (en) | 2016-07-27 | 2020-07-14 | Vocollect, Inc. | Distinguishing user speech from background speech in speech-dense environments |
WO2021033889A1 (en) | 2019-08-20 | 2021-02-25 | Samsung Electronics Co., Ltd. | Electronic device and method for controlling the electronic device |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4567606A (en) * | 1982-11-03 | 1986-01-28 | International Telephone And Telegraph Corporation | Data processing apparatus and method for use in speech recognition |
JPH01503786A (en) * | 1987-06-25 | 1989-12-21 | イビイ イスティテュト ビオキミコ イタリアノ ジョバンニ ロレンツィニ ソチエタ ペル アツィオニ | Prostaglandin derivatives, their production methods and pharmaceutical compositions containing them |
US4914590A (en) * | 1988-05-18 | 1990-04-03 | Emhart Industries, Inc. | Natural language understanding system |
US5216748A (en) * | 1988-11-30 | 1993-06-01 | Bull, S.A. | Integrated dynamic programming circuit |
DE69314353T2 (en) * | 1992-06-15 | 1998-03-05 | British Telecomm | SERVICE FACILITIES |
US5452459A (en) * | 1993-01-08 | 1995-09-19 | Digital Equipment Corporation | Method and apparatus for allocating server access in a distributed computing environment |
US5459798A (en) * | 1993-03-19 | 1995-10-17 | Intel Corporation | System and method of pattern recognition employing a multiprocessing pipelined apparatus with private pattern memory |
US5870735A (en) * | 1996-05-01 | 1999-02-09 | International Business Machines Corporation | Method and system for generating a decision-tree classifier in parallel in a multi-processor system |
US5806032A (en) * | 1996-06-14 | 1998-09-08 | Lucent Technologies Inc. | Compilation of weighted finite-state transducers from decision trees |
US6249761B1 (en) * | 1997-09-30 | 2001-06-19 | At&T Corp. | Assigning and processing states and arcs of a speech recognition model in parallel processors |
-
1997
- 1997-09-30 US US08/941,379 patent/US6249761B1/en not_active Expired - Lifetime
-
2001
- 2001-03-13 US US09/804,041 patent/US6374212B2/en not_active Expired - Lifetime
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7162424B2 (en) | 2001-04-26 | 2007-01-09 | Siemens Aktiengesellschaft | Method and system for defining a sequence of sound modules for synthesis of a speech signal in a tonal language |
US20020188450A1 (en) * | 2001-04-26 | 2002-12-12 | Siemens Aktiengesellschaft | Method and system for defining a sequence of sound modules for synthesis of a speech signal in a tonal language |
US7587319B2 (en) * | 2002-02-04 | 2009-09-08 | Zentian Limited | Speech recognition circuit using parallel processors |
US20050251390A1 (en) * | 2002-02-04 | 2005-11-10 | Mark Catchpole | Speech recognition circuit using parallel processors |
US10971140B2 (en) | 2002-02-04 | 2021-04-06 | Zentian Limited | Speech recognition circuit using parallel processors |
US8768696B2 (en) | 2002-02-04 | 2014-07-01 | Zentian Limited | Speech recognition circuit using parallel processors |
US20090326941A1 (en) * | 2002-02-04 | 2009-12-31 | Mark Catchpole | Speech recognition circuit using parallel processors |
US8036890B2 (en) | 2002-02-04 | 2011-10-11 | Zentian Limited | Speech recognition circuit using parallel processors |
US10217460B2 (en) | 2002-02-04 | 2019-02-26 | Zentian Limited. | Speech recognition circuit using parallel processors |
US9536516B2 (en) | 2002-02-04 | 2017-01-03 | Zentian Limited | Speech recognition circuit using parallel processors |
EP1361736A1 (en) * | 2002-05-08 | 2003-11-12 | Sap Ag | Method for recognition of speech information |
US20100114577A1 (en) * | 2006-06-27 | 2010-05-06 | Deutsche Telekom Ag | Method and device for the natural-language recognition of a vocal expression |
US9208787B2 (en) * | 2006-06-27 | 2015-12-08 | Deutsche Telekom Ag | Method and device for the natural-language recognition of a vocal expression |
US20080059184A1 (en) * | 2006-08-22 | 2008-03-06 | Microsoft Corporation | Calculating cost measures between HMM acoustic models |
US8234116B2 (en) * | 2006-08-22 | 2012-07-31 | Microsoft Corporation | Calculating cost measures between HMM acoustic models |
US20080059190A1 (en) * | 2006-08-22 | 2008-03-06 | Microsoft Corporation | Speech unit selection using HMM acoustic models |
US9478216B2 (en) * | 2009-12-08 | 2016-10-25 | Nuance Communications, Inc. | Guest speaker robust adapted speech recognition |
US20120245940A1 (en) * | 2009-12-08 | 2012-09-27 | Nuance Communications, Inc. | Guest Speaker Robust Adapted Speech Recognition |
US20130185430A1 (en) * | 2012-01-18 | 2013-07-18 | LineRate Systems, Inc. | Multi-level hash tables for socket lookups |
US20140223310A1 (en) * | 2013-02-01 | 2014-08-07 | Nuance Communications, Inc. | Correction Menu Enrichment with Alternate Choices and Generation of Choice Lists in Multi-Pass Recognition Systems |
US9858038B2 (en) * | 2013-02-01 | 2018-01-02 | Nuance Communications, Inc. | Correction menu enrichment with alternate choices and generation of choice lists in multi-pass recognition systems |
Also Published As
Publication number | Publication date |
---|---|
US6374212B2 (en) | 2002-04-16 |
US6249761B1 (en) | 2001-06-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6249761B1 (en) | Assigning and processing states and arcs of a speech recognition model in parallel processors | |
Digalakis et al. | Genones: Generalized mixture tying in continuous hidden Markov model-based speech recognizers | |
Soong et al. | A Tree. Trellis based fast search for finding the n best sentence hypotheses in continuous speech recognition | |
US4819271A (en) | Constructing Markov model word baseforms from multiple utterances by concatenating model sequences for word segments | |
US5787396A (en) | Speech recognition method | |
JP2543148B2 (en) | A device for learning a voice recognition device | |
US6574597B1 (en) | Fully expanded context-dependent networks for speech recognition | |
JP3049259B2 (en) | Voice recognition method | |
EP0903728A2 (en) | Block algorithm for pattern recognition | |
KR101970041B1 (en) | Methods for Hybrid GPU/CPU Data Processing | |
JP4141495B2 (en) | Method and apparatus for speech recognition using optimized partial probability mixture sharing | |
US6178401B1 (en) | Method for reducing search complexity in a speech recognition system | |
US20050159952A1 (en) | Pattern matching for large vocabulary speech recognition with packed distribution and localized trellis access | |
EP0664535A2 (en) | Large vocabulary connected speech recognition system and method of language representation using evolutional grammar to represent context free grammars | |
EP0314908A2 (en) | Automatic determination of labels and markov word models in a speech recognition system | |
Ljolje et al. | Efficient general lattice generation and rescoring | |
KR101120765B1 (en) | Method of speech recognition using multimodal variational inference with switching state space models | |
EP0573553A1 (en) | Method for recognizing speech using linguistically-motivated hidden markov models | |
JPS62231995A (en) | Decision of probability value | |
EP0903730B1 (en) | Search and rescoring method for a speech recognition system | |
US20120245919A1 (en) | Probabilistic Representation of Acoustic Segments | |
US11705116B2 (en) | Language and grammar model adaptation using model weight data | |
Schwartz et al. | Efficient, high-performance algorithms for n-best search | |
Phillips et al. | Parallel speech recognition | |
US5764851A (en) | Fast speech recognition method for mandarin words |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
FPAY | Fee payment |
Year of fee payment: 12 |
|
AS | Assignment |
Owner name: AT&T PROPERTIES, LLC, NEVADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T CORP.;REEL/FRAME:040955/0226 Effective date: 20160204 Owner name: AT&T INTELLECTUAL PROPERTY II, L.P., GEORGIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T PROPERTIES, LLC;REEL/FRAME:040955/0257 Effective date: 20160204 |
|
AS | Assignment |
Owner name: NUANCE COMMUNICATIONS, INC., MASSACHUSETTS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AT&T INTELLECTUAL PROPERTY II, L.P.;REEL/FRAME:041498/0316 Effective date: 20161214 |