US6873993B2 - Indexing method and apparatus - Google Patents

Indexing method and apparatus Download PDF

Info

Publication number
US6873993B2
US6873993B2 US09/863,424 US86342401A US6873993B2 US 6873993 B2 US6873993 B2 US 6873993B2 US 86342401 A US86342401 A US 86342401A US 6873993 B2 US6873993 B2 US 6873993B2
Authority
US
United States
Prior art keywords
sub
query
word
key
entry
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US09/863,424
Other versions
US20020052870A1 (en
Inventor
Jason Peter Andrew Charlesworth
Philip Neil Garner
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Assigned to CANON KABUSHIKI KAISHA reassignment CANON KABUSHIKI KAISHA ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHARLESWORTH, JASON PETER ANDREW, GARNER, PHILIP NEIL
Publication of US20020052870A1 publication Critical patent/US20020052870A1/en
Application granted granted Critical
Publication of US6873993B2 publication Critical patent/US6873993B2/en
Adjusted expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • G06F16/3331Query processing
    • G06F16/334Query execution
    • G06F16/3343Query execution using phonetics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/45Clustering; Classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/61Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/63Querying
    • G06F16/632Query formulation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/60Information retrieval; Database structures therefor; File system structures therefor of audio data
    • G06F16/68Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/683Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/685Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using automatically derived transcript of audio data, e.g. lyrics
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99943Generating database or data structure, e.g. via user interface

Definitions

  • the present invention relates to an apparatus and method for indexing sequences of sub-word units, such as sequences of phonemes or the like.
  • the invention can be used to identify regions of a database for search in response to a user's input query.
  • the input query may be a voiced or typed query.
  • Databases of information are well known and suffer from the problem of how to locate and retrieve the desired information from the database quickly and efficiently.
  • Existing database search tools allow the user to search the database using typed key words. Whilst this is quick and efficient, this type of searching is not suitable for various kinds of databases, such as video or audio databases.
  • the technique proposed for matching the sequences of phonemes firstly defines a set of features in the query, each feature being taken as an overlapping fixed size fragment from the phoneme string. It then identifies the frequency of occurrence of the features in both the query and the annotation and then finally determines a measure of the similarity between the query and the annotation using a cosine measure of these frequencies of occurrences.
  • indexing provides a one-to-many mapping between the index (sometimes referred to as the key) and the data in the database.
  • this indexing is simple, but for phonemes this raises a number of difficulties. Firstly, because there are only a small number of phonemes (approximately 43 in the English language) means that a naive mapping using a single phoneme as a key is not sufficiently discriminating, since any given phoneme will occur several thousand times in the database. Secondly, because of the relatively poor recognition rate of phonemes (60% to 70%) means that any one-to-many mapping will make it difficult to retrieve data where the query phoneme or annotation phoneme was misrecognised, inserted or omitted. Finally, performing any statistical retrieval methods becomes computationally unfeasible.
  • the present invention aims to provide an efficient sub-word indexing technique which can be used in a retrieval system to identify areas of a database for searching.
  • FIG. 1 is a schematic block diagram illustrating a user terminal which allows the user to retrieve information from an input typed or voice query;
  • FIG. 2 is a schematic diagram of phoneme and word lattice annotation data which is generated from a voiced input by the user for annotating a document;
  • FIG. 3 a diagrammatically illustrates the block nature of an annotation stored in the annotation database which forms part of the user terminal shown in FIG. 1 ;
  • FIG. 3 b is a schematic diagram illustrating a sequence of annotation phonemes which is included in one of the blocks of the annotation shown in FIG. 3 a;
  • FIG. 3 c schematically illustrates a sequence of phoneme clusters for the phoneme sequence shown in FIG. 3 b and illustrates how these phoneme clusters can be grouped to form a number of overlapping phoneme cluster N-grams;
  • FIG. 4 is a flowchart illustrating the main processing steps involved in creating a phoneme index
  • FIG. 5 illustrates an example of a phoneme index which is generated during the processing of the steps shown in FIG. 4 ;
  • FIG. 6 is a flowchart illustrating the processing steps involved in performing a phoneme search of an annotation database
  • FIG. 7 a schematically illustrates a sequence of phonemes representing an input query
  • FIG. 7 b schematically illustrates the way in which the sequence of phonemes shown in FIG. 7 a can be divided into a number of overlapping phoneme N-grams;
  • FIG. 7 c illustrates a number of overlapping phoneme cluster N-grams derived from the phoneme N-grams shown in FIG. 7 b;
  • FIG. 8 is a flowchart illustrating the main processing steps involved in using the phoneme index to identify locations of the annotation for phoneme matching
  • FIG. 9 a is a flowchart illustrating part of the process steps involved in determining the different phoneme clusters
  • FIG. 9 b is a flowchart illustrating the remaining process steps involved in determining the different phoneme clusters
  • FIG. 10 is a schematic block diagram illustrating the form of an alternative user terminal which is operable to retrieve a data file from a database located within a remote server in response to an input voice query;
  • FIG. 11 illustrates another user terminal which allows a user to retrieve data from a database located within a remote server in response to an input voice query.
  • Embodiments of the present invention can be implemented using dedicated hardware circuits, but the embodiment to be described is implemented in computer software or code, which is run in conjunction with processing hardware such as a personal computer, work station, photocopier, facsimile machine, personal digital assistant (PDA), web browser or the like.
  • processing hardware such as a personal computer, work station, photocopier, facsimile machine, personal digital assistant (PDA), web browser or the like.
  • FIG. 1 is a block diagram illustrating the form of a user terminal 59 which is used, in this embodiment, to retrieve documents from a document database 29 in response to a voice or typed query input by the user 39 .
  • the “document” may be text documents, audio files, video files, photographs, mixtures of these etc.
  • the user terminal 59 may be, for example, a personal computer, a hand-held device or the like.
  • the user terminal 59 comprises the document database 29 , an annotation database 31 comprising a descriptive annotation of each of the documents in the document database 29 , a phoneme matcher 33 , a phoneme index 35 , a word matcher 37 , a word index 38 , a combiner unit 40 , an automatic speech recognition unit 51 , a phonetic transcription unit 75 , a keyboard 3 , a microphone 7 and a display 57 .
  • the user inputs either a voice query via the microphone 7 or a typed query via the keyboard 3 and the query is processed either by the automatic speech recognition unit 51 or the phonetic transcription unit 75 to generate corresponding phoneme and word data.
  • the phoneme data is input to the phoneme matcher 33 which is operable to perform a phoneme search in the annotation database 31 with reference to a phoneme index 35 .
  • the word data is input to the word matcher 37 which is operable to search the annotation database 31 with reference to the word index 38 .
  • the results of the phoneme and word search of the annotation database are then input to the combiner unit 40 which uses these results to retrieve a ranked list of documents 41 from the document database 29 which are output to the display unit 57 for display to the user 39 .
  • the annotation data for each document comprises a combined phoneme (or phoneme-like) and word lattice.
  • FIG. 2 illustrates the form of the phoneme and word lattice annotation data generated for the spoken annotation “picture of the Taj Mahal”.
  • the phoneme and word lattice is an acyclic directed graph with a single entry point and a single exit point. It represents different parses of the user's input. It is not simply a sequence of words with alternatives, since each word does not have to be replaced by a single alternative, one word can be substituted for two or more words or phonemes, and the whole structure can form a substitution for one or more words or phonemes. Therefore, the density of the data within the phoneme and word lattice annotation data essentially remains linear throughout the annotation data, rather than growing exponentially as in the case of a system which generates the N-best word lists for the annotation input.
  • annotation data for each document (d) is stored in the annotation database 31 and has the following general form:
  • Block(i) i 0, 1, 2, . . .
  • annotation data is word annotation data, phoneme annotation data or if it is mixed is provided since the annotation data may include just word data, just phoneme data or both word and phoneme data.
  • annotation data is divided into blocks (B) of nodes (n) in order to allow the search to jump into the middle of the annotation data.
  • the header therefore includes a time index which associates the location of the blocks of annotation data within the memory to a given time offset between the time of start and the time corresponding to the beginning of the block.
  • the header also includes data defining the word set used (i.e. the dictionary), the phoneme set used and their probabilities and the language to which the vocabulary pertains.
  • the header may also include details of the automatic speech recognition system or the phonetic transcription system used to generate the annotation data and any appropriate settings thereof which were used during the generation of the annotation data.
  • the blocks of annotation data then follow the header and identify, for each node in the block, the time offset of the node from the start of the block, the phoneme links which connect that node to other nodes by phonemes and word links which connect that node to other nodes by words.
  • Each phoneme link and word link identifies the phoneme or word which is associated with the link. They also identify the offset to the current node. For example, if node n 50 is linked to node n 55 by a phoneme link, then the offset to node n 50 is 5. As those skilled in the art will appreciate, using an offset indication like this allows the division of the continuous annotation data into separate blocks.
  • an automatic speech recognition unit outputs weightings indicative of the confidence of the speech recognition unit's output
  • these weightings or confidence scores would also be included within the data structure.
  • a confidence score would be provided for each node which is indicative of the confidence of arriving at the node and each of the phoneme and word links would include a transition score depending upon the weighting given to the corresponding phoneme or word. These weightings would then be used to control the search and retrieval of the data files by discarding those matches which have a low confidence score.
  • a word indexing scheme and a phoneme indexing scheme is used in order to identify portions in the annotation database 31 against which a direct comparison with the input query is made.
  • the word index 38 and the way that it is used is well known to those skilled in the art and will not be described further.
  • the phoneme index 35 is generated and subsequently used to identify portions of the annotation database 31 for comparison with the input query will now be described in more detail.
  • the query comprises the sequence of phonemes /sh//iy//p/ and the annotation (A) comprises the sequence of phonemes /s//eh//p/.
  • the sequence of three query phonemes is used as an index into the annotation (in the form of a trigram)
  • the phonemes are clustered into confusable sets, and the phonemes in the query and in the annotation are classified into their respective sets, then there is a better chance that there will be a match between the query and the annotation if they both sound alike.
  • C 1 ⁇ / s / ⁇ / z / ⁇ / sh / ⁇ / zh / ⁇
  • C 2 ⁇ / t / ⁇ / k / ⁇ / g / ⁇ / b / ⁇ / p / ⁇ ⁇ ⁇
  • C 5 ⁇ / eh / ⁇ / ih / ⁇ / iy / ⁇ ⁇
  • a hash indexing technique which tries to create a unique mapping between a key and an entry in a list.
  • trigrams of the above phoneme clusters are used as the key to the index. The way that the phoneme index 35 is created and used will now be described in more detail.
  • FIG. 3 a schematically illustrates the block form of the annotation data which is stored in the database 31 for one of the documents (d) stored in the document database 29 .
  • the annotation data comprises successive blocks of data B 0 d to B M-1 d .
  • the annotation data within each block includes the nodes within the block and the phoneme and word links which are associated with the nodes.
  • the annotation data for document d includes a canonical sequence of phonemes i.e. one with no alternatives.
  • FIG. 3 b illustrates the canonical sequence of phonemes within the i th block for the annotation data for document d.
  • block B d i comprises the canonical sequence of phonemes a 0 di to a Ndi di which extend between nodes n 0 di to n Ndi di .
  • FIG. 3 c illustrates the overlapping “cluster trigrams” 101 generated for the sequence of annotation phonemes in block B d i shown in FIG. 3 b.
  • the cluster in which each of the annotation phonemes in block B d i belongs is determined.
  • cluster trigrams are determined from overlapping groups of three cluster identifications for the sequence of annotation phonemes.
  • the first cluster trigram determined is C(a di 0 ) C(a di 1 ) C(a di 2 ) then cluster trigram C(a di 1 ) C(a di 2 ) C(a di 3 ) etc.
  • the trigrams which bridge adjacent blocks For example, in this embodiment, it would be necessary to consider the following cluster trigrams: C(a di-1 N(di-1)-1 ) C(a di-1 Ndi-1 ) C(a di 0 ) and cluster trigram C(a di-1 Ndi-1 ) C(a di 0 ) C(a di 1 ).
  • each entry is addressed by an index (IDX) which takes a value between zero and S- 1 and each entry includes a data field to store the key (KEY) associated with the entry and a data field to store the pointers which identify the relevant locations within the annotation database where the phoneme data associated with the KEY can be found.
  • IDX index
  • each entry is empty.
  • the size of the table depends on the number of different phoneme clusters and the number of cluster identifications in the key. In this case, three cluster identifications (trigrams) are provided in each key. If there are ten clusters, then the number of different possible keys is 3 10 .
  • S should, theoretically be made approximately equal to 3 10 .
  • the size of the index can be set to have some initial size and data can be added to the index until more than a predetermined percentage of the entries are full. At this point, a bigger table can be created and the data from the old table copied over to the new table. This process can be repeated until there is no more data to be added to the index. Although this means that some memory is wasted, this is insignificant compared to the memory required for the pointers which will be stored in the table.
  • step s 1 the system calculates the value of a function (f(KEY)) which is dependent upon the key, i.e. a function of the current cluster trigram, which value is used as the index (IDX) into the table A.
  • f(KEY) defines a mapping between the cluster trigram and an entry in the table A and always yields a number between zero and S- 1 .
  • the function used is: [C[1]K c C[2]K c C[3]K c ]mod S (1)
  • K c is the number of phoneme clusters and C[1] is the number of the cluster to which the first annotation phoneme in the trigram belongs, C[2] is the number of the cluster to which the second annotation phoneme in the trigram belongs and C[3] is the number of the cluster to which the third annotation phoneme belongs.
  • step s 3 the system checks the corresponding entry in the table, A, and determines whether or not the key stored in that entry (KEY) is the same as the key for the current phoneme cluster trigram (key) or is the null key (indicating that this entry is empty). If in step s 3 the system determines that the key stored in the entry (KEY) matches the current key (key) or the null key, then the processing proceeds to step s 5 where a pointer is added to that entry of the table, A, which points to the node associated with the first phoneme in the phoneme trigram associated with the current input key.
  • the data which would be added to the table would be a pointer which points to the node n 0 di since annotation phoneme a 0 di is associated with node n 0 di .
  • the system also changes the key for the entry (KEY) to the current key (key). The processing of the current cluster trigram then ends and a similar processing is performed on the next cluster trigram.
  • step s 3 the processing determines that the IDX th entry in the table, A, has already been assigned to a different cluster trigram, then the processing proceeds to step s 7 where the system tries another entry in the table by changing the value of IDX in some predetermined way. In this embodiment, this is achieved by calculating:
  • step s 7 the processing returns to step s 3 .
  • FIG. 5 illustrates the form of a phoneme index that is generated by the above processing.
  • each entry in the table includes the index number (IDX) of the entry, the key (KEY) associated with the entry and (if the key is not the null key) one or more pointers pointing to nodes in the annotation database 31 .
  • these pointers have the form n[p,q,r] where p is the annotation for document p, q is the q th block of nodes within that annotation data and r is the r th node within that block.
  • FIG. 6 illustrates the processing steps performed by the phoneme matcher 33 on this phoneme data.
  • the phoneme matcher 33 converts the received query phoneme data into overlapping phoneme trigrams.
  • FIG. 7 a illustrates a sequence of query phonemes q 0 to q 5 representative of phoneme data received by the phoneme matcher 33 and
  • FIG. 7 b illustrates how this sequence of query phonemes is converted into five overlapping trigrams of query phonemes 103 .
  • each of the query phoneme trigrams is converted into phoneme cluster trigrams 105 , as illustrated in FIG. 7 c , by classifying each of the query phonemes in the phoneme trigram into one of the above classes or clusters.
  • the processing then proceeds to step s 15 where the phoneme matcher 33 uses each of the cluster trigrams generated for the input query to address the phoneme index 35 in order to identify relevant locations in the annotation database 31 .
  • FIG. 8 illustrates the processing steps used by the phoneme matcher 33 in carrying out step s 15 .
  • the phoneme matcher 33 calculates the index (IDX) for the entry in the table, A, by inserting the current query cluster trigram into the function defined above in equation (1).
  • the processing then proceeds to step s 153 where the phoneme matcher 33 checks the corresponding entry in the table, A, and determines whether or not the key stored in that entry (KEY) is the null key (indicating that this entry is empty). If it is, then the processing proceeds to step s 155 where the phoneme matcher 33 determines that there is no corresponding annotation in the annotation database and outputs an appropriate output to the combiner unit 40 . The processing then ends.
  • step s 153 the phoneme matcher 33 determines that the key stored in the entry (KEY) is not equal to the null key
  • the processing proceeds to step s 157 where the phoneme matcher 33 determines whether or not the key stored in the entry (KEY) is the same as the key for the current query cluster trigram (key). If it is then the processing proceeds to step s 159 where the phoneme matcher 33 retrieves the pointers from that entry. The processing then ends.
  • step s 157 the phoneme matcher 33 determines, in step s 157 , that the key for the entry (KEY) does not equal the key for the current query cluster trigram (key), then the processing proceeds to step s 161 where the phoneme matcher tries another entry in the table by changing the value of the index (IDX) using equation (2) given above and then returning to step s 153 .
  • the phoneme matcher 33 retrieves the pointers from the index or determines that there is no data stored for the current query cluster trigram, the phoneme matcher 33 then performs a similar processing for the next query cluster trigram until all the query cluster trigrams have been processed in this way. The processing then proceeds to step s 17 shown in FIG. 6 , where the phoneme matcher 33 uses the pointers identified in step s 15 to identify regions within the annotation database 31 which will be matched with the actual phoneme data received by the phoneme matcher 33 . In this embodiment, these regions are identified by comparing the pointers retrieved in step s 159 for successive query cluster trigrams and looking for pointers which point to portions in the annotation database 31 which are next to each other.
  • the phoneme matcher 33 will identify node 40 of the 32 nd block of annotation data for the 3 rd document as being a region of the annotation database for further processing in step s 19 .
  • the pointers stored in the phoneme index 35 for the key c 5 c 3 c 6 includes a pointer to node n[ 3 , 32 , 40 ] and the pointers stored in the phoneme index 35 for the key c 3 c 6 c 1 includes a pointer to node n[ 3 , 32 , 41 ], which is immediately after node n[ 3 , 32 , 40 ] and is therefore consistent with a portion of an annotation having successive cluster trigrams c 5 c 3 c 6 and then c 3 c 6 c 1 which may match with the input query.
  • the phoneme matcher 33 After the phoneme matcher 33 has identified the regions in step s 17 , it performs a phoneme comparison between the received query phoneme data and the phoneme data stored in the annotation database 31 at the regions identified in step s 17 .
  • This phoneme comparison can be performed by comparing M-grams of the query with similar M-grams of the annotation (as described in the applicant's earlier UK application GB 9905201.1, the content of which is incorporated herein by reference) or by performing a dynamic programming comparison between the sequence of query phonemes and the sequence of annotation phonemes (using, for example, one of the techniques described in the applicant's earlier UK application GB 9925574.7, the content of which is incorporated herein by reference).
  • the results of these phoneme comparisons are then output to the combiner unit 40 where the results are combined with the output from the word matcher 37 in order to retrieve and rank the appropriate documents from the document database 29 .
  • the decoding methods m 1 and m 2 need to be distinguished since one of the decodings may come from a text-to-phoneme converter within the phonetic transcription unit 75 whilst the other may come from the automatic speech recognition unit 51 and these two different decoding techniques will suffer from different types of confusion.
  • Each of these probabilities in equation (3) above can be determined in advance during a training routine by applying known speech to the automatic speech recognition unit 51 and known text into the phonetic transcription unit 75 and by monitoring the output from the recognition unit 51 and the phonetic transcription unit 75 respectively. The way that such a training routine would be performed will be well known to those skilled in the art and will not, therefore, be described further here.
  • K c phoneme clusters or classes there are K c phoneme clusters or classes and for each cluster there is a many-to-one mapping between decoded phonemes and the clusters. This mapping will depend on the decoding method employed. For example, an /s/ decoded by the automatic speech recognition unit 51 may be in cluster c 4 , but an /s/ decoded by the phonetic transcription unit 75 may be in cluster c 2 . Therefore, for any particular cluster (K i ) there will be a probability that two decodings from the same source are classed into that cluster which is determined by summing the probability given in equation (3) above for all possible combinations of decodings, p 1 and p 2 , which belong to that cluster, i.e.
  • the task of defining the clusters aims, therefore, to maximise P(all correctly classified), subject to the constraints that each phoneme (via a particular decoding method) is in one and only one cluster.
  • a Monte Carlo algorithm is used in order to determine phoneme classifications which maximise this probability.
  • FIG. 9 a illustrates the steps involved in this Monte Carlo algorithm.
  • step s 200 the number of phoneme clusters that will be used is determined. As those skilled in the art will appreciate, if there are too few clusters then there will be insufficient discrimination and if there are too many clusters then the data may not be retrievable. In this embodiment, in order to provide classifications which are sufficiently discriminative, ten clusters are defined. Once the number of clusters has been determined, the system randomly assigns, in step s 201 , phonemes to these clusters and stores this as a current configuration. The processing then proceeds to step s 203 where the system determines the probability that the phonemes are correctly classified in the clusters for the current configuration, i.e. the system calculates the probability given in equation (5) above.
  • step s 205 the system randomly selects a phoneme and a target cluster to which the selected phoneme may be moved.
  • step s 207 the system calculates what the probability given in equation (5) would be if the selected phoneme is moved into the target cluster.
  • step s 209 the system compares this new probability calculated in step s 207 with the probability for the current configuration which was calculated in step s 203 . If the new probability is higher than the probability for the current configuration, then the processing passes to step s 211 where the system moves the selected phoneme to the target cluster to replace the current configuration.
  • step s 213 the system determines whether or not the probability calculated for the new configuration is better than the “best ever” probability. If it is, then the processing proceeds to step s 215 where the system stores the current configuration as the best ever configuration that it has encountered. Otherwise step s 215 is skipped and the processing proceeds to step s 219 where the system determines whether or not convergence has been reached.
  • step s 217 the system proceeds to step s 219 where it determines whether or not convergence has been reached. If it has, then the processing ends and the phoneme clusters are stored. If convergence has not been reached, then the processing proceeds to step s 221 where a random value (RV) between zero and one is chosen and then compared with a threshold (Th) in step s 223 . If the random value, RV, is less than the threshold then the processing returns to step s 205 above and the procedure is repeated. On the other hand, if the random value, RV is not less than the threshold, then the processing proceeds to step s 225 where the best ever configuration is copied into the current configuration and then the processing again returns to step s 205 .
  • RV random value
  • the phoneme clusters for the text to phoneme transcription unit 75 are predominantly the same as those for the automatic speech recognition 51 , with the exception of the “uh” phone which is in cluster c 1 while it is in cluster c 9 for the clusters of the automatic speech recognition unit 51 .
  • the clusters given above are given by way of example only. The precise clusters that are used will depend on the matching method used to compare the phonemes in the clusters.
  • FIG. 10 illustrates an embodiment in which the document database 29 and the search engine 53 are located in a remote server 60 and in which the user terminal 59 accesses the database 29 via the network interface unit 67 and 69 and a data network 68 (such as the Internet).
  • both the documents and the annotations are stored in the database 29 .
  • the user terminal 59 can only receive voice queries from the microphone 7 . These queries are converted into phoneme and word data by the automatic speech recognition unit 51 .
  • This data is then passed to the control unit 55 which controls the transmission of data over the data network 68 to the search engine 53 located within the remote server 60 .
  • the search engine 53 uses the phoneme index to carry out a search in the database 29 in a similar manner to the way in which the search was performed in the above embodiment.
  • the results of the search are then transmitted back from the search engine 53 to the control unit 55 via the data network 68 .
  • the control unit 55 then considers the search results received back from the network and displays appropriate data on the display 57 for viewing by the user 39 .
  • the automatic speech recognition unit 51 in the remote server 60 .
  • FIG. 11 Such an embodiment is shown in FIG. 11 .
  • the input voice query from the user is passed via input line 61 to a speech encoding unit 73 which is operable to encode the speech for efficient transfer through the data network 68 .
  • the encoded data is then passed to the control unit 55 which transmits the data over the network 68 to remote server 60 , where it is processed by the automatic speech recognition unit 51 .
  • the speech recognition unit is operable to only generate phoneme data which is then passed to the search engine for use in searching the database 29 using the phoneme index 35 .
  • the search results generated by the search engine are then passed, via the network interface 69 and the network 68 , back to the user terminal 59 .
  • the search results received back from the remote server are then passed via the network interface unit 67 to the control unit 55 which analyses the results and generates and displays appropriate data on the display 57 for viewing by the user 39 .
  • a user terminal 59 may be provided which only allows typed inputs from the user and which has the search engine and the database located in the remote server.
  • the phonetic transcription unit 75 may be located in the remote server 60 as well.
  • annotation database and the document database were separate. As those skilled in the art will appreciate, in some embodiments, the annotation database and the document database may form a single database. Additionally, the annotations in the annotation database may form the data to be retrieved.
  • the annotation database was searched using both phonemes and words.
  • the phoneme index described above and its use in searching the annotation database may be used in a system which does not search using words as well.
  • a phoneme index was described.
  • the above technique can be used for features other than phonemes, such as phones, syllables or katakana (Japanese alphabet) or any other sub-word unit of speech.
  • This indexing technique could also be used in other applications, such as the indexing of DNA sequences and the like.
  • hash indexing technique has been described.
  • other indexing techniques can be used in order to identify portions of the database for carrying out a detailed phoneme search using phoneme data generated from the input query.
  • the pointers stored in the index were of the form n[p,q,r], where p is the annotation for document p, q is the q th block of nodes within that annotation and r is the r th node within that block.
  • p is the annotation for document p
  • q is the q th block of nodes within that annotation
  • r is the r th node within that block.
  • other types of pointers could be used.
  • the count of the node since the start of the lattice or the time and rank (where more than one node have the same time) of the relevant nodes could be used.
  • each of the query cluster trigrams were applied to the index and the appropriate entries found. These entries were then compared in order to identify portions of the phoneme lattice for further searching.
  • the processing of the query cluster trigrams may be performed incrementally, i.e. identify all the entries for the first query trigram, then obtain those for the second query trigram and retain those which are close in time to those of the first trigram etc.
  • the system when identifying regions where successive cluster trigrams occur close in time, the system compared the node numbers in the pointers which are retrieved. However, in some embodiments, it is better to compare the time offsets stored for the nodes. This is because in some applications, the lattice will have many branches and two nodes which are very close in time could have completely different node numbers.
  • the aim of this part of the system is to identify cluster trigrams which are in chronological order within the annotation and which occur within a limited period of time of each other. In this way, if there is an error in a trigram, it will only result in two to three trigram misses. Consequently, the time leeway should be comparable to about four or five phonemes which is about 0.2 to 0.3 seconds.

Abstract

An indexing apparatus and method are described for use in identifying portions of data in a database for comparison with a query. In an embodiment, the index includes a key which comprises a sequence of phoneme classifications derived from the input query by classifying each of the phonemes in the input query with a number of phoneme classes, with the phonemes in each class being defined as those that are confusable with the other phonemes in the same class.

Description

The present invention relates to an apparatus and method for indexing sequences of sub-word units, such as sequences of phonemes or the like. The invention can be used to identify regions of a database for search in response to a user's input query. The input query may be a voiced or typed query.
Databases of information are well known and suffer from the problem of how to locate and retrieve the desired information from the database quickly and efficiently. Existing database search tools allow the user to search the database using typed key words. Whilst this is quick and efficient, this type of searching is not suitable for various kinds of databases, such as video or audio databases.
A recent proposal has been made to annotate such video and audio databases with a phonetic transcription of the speech content of the audio and video files, with subsequent retrieval being achieved by comparing a phonetic transcription of the user's input query with the phoneme annotation data in the database. The technique proposed for matching the sequences of phonemes firstly defines a set of features in the query, each feature being taken as an overlapping fixed size fragment from the phoneme string. It then identifies the frequency of occurrence of the features in both the query and the annotation and then finally determines a measure of the similarity between the query and the annotation using a cosine measure of these frequencies of occurrences.
However, as those skilled in the art will appreciate, if the database is large, then this retrieval method becomes unfeasibly long. An indexing method is therefore required.
As is well known, indexing provides a one-to-many mapping between the index (sometimes referred to as the key) and the data in the database. Where the database comprises words, this indexing is simple, but for phonemes this raises a number of difficulties. Firstly, because there are only a small number of phonemes (approximately 43 in the English language) means that a naive mapping using a single phoneme as a key is not sufficiently discriminating, since any given phoneme will occur several thousand times in the database. Secondly, because of the relatively poor recognition rate of phonemes (60% to 70%) means that any one-to-many mapping will make it difficult to retrieve data where the query phoneme or annotation phoneme was misrecognised, inserted or omitted. Finally, performing any statistical retrieval methods becomes computationally unfeasible.
The present invention aims to provide an efficient sub-word indexing technique which can be used in a retrieval system to identify areas of a database for searching.
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which:
FIG. 1 is a schematic block diagram illustrating a user terminal which allows the user to retrieve information from an input typed or voice query;
FIG. 2 is a schematic diagram of phoneme and word lattice annotation data which is generated from a voiced input by the user for annotating a document;
FIG. 3 a diagrammatically illustrates the block nature of an annotation stored in the annotation database which forms part of the user terminal shown in FIG. 1;
FIG. 3 b is a schematic diagram illustrating a sequence of annotation phonemes which is included in one of the blocks of the annotation shown in FIG. 3 a;
FIG. 3 c schematically illustrates a sequence of phoneme clusters for the phoneme sequence shown in FIG. 3 b and illustrates how these phoneme clusters can be grouped to form a number of overlapping phoneme cluster N-grams;
FIG. 4 is a flowchart illustrating the main processing steps involved in creating a phoneme index;
FIG. 5 illustrates an example of a phoneme index which is generated during the processing of the steps shown in FIG. 4;
FIG. 6 is a flowchart illustrating the processing steps involved in performing a phoneme search of an annotation database;
FIG. 7 a schematically illustrates a sequence of phonemes representing an input query;
FIG. 7 b schematically illustrates the way in which the sequence of phonemes shown in FIG. 7 a can be divided into a number of overlapping phoneme N-grams;
FIG. 7 c illustrates a number of overlapping phoneme cluster N-grams derived from the phoneme N-grams shown in FIG. 7 b;
FIG. 8 is a flowchart illustrating the main processing steps involved in using the phoneme index to identify locations of the annotation for phoneme matching;
FIG. 9 a is a flowchart illustrating part of the process steps involved in determining the different phoneme clusters;
FIG. 9 b is a flowchart illustrating the remaining process steps involved in determining the different phoneme clusters;
FIG. 10 is a schematic block diagram illustrating the form of an alternative user terminal which is operable to retrieve a data file from a database located within a remote server in response to an input voice query; and
FIG. 11 illustrates another user terminal which allows a user to retrieve data from a database located within a remote server in response to an input voice query.
Embodiments of the present invention can be implemented using dedicated hardware circuits, but the embodiment to be described is implemented in computer software or code, which is run in conjunction with processing hardware such as a personal computer, work station, photocopier, facsimile machine, personal digital assistant (PDA), web browser or the like.
Data File Retrieval
FIG. 1 is a block diagram illustrating the form of a user terminal 59 which is used, in this embodiment, to retrieve documents from a document database 29 in response to a voice or typed query input by the user 39. The “document” may be text documents, audio files, video files, photographs, mixtures of these etc. The user terminal 59 may be, for example, a personal computer, a hand-held device or the like. As shown, the user terminal 59 comprises the document database 29, an annotation database 31 comprising a descriptive annotation of each of the documents in the document database 29, a phoneme matcher 33, a phoneme index 35, a word matcher 37, a word index 38, a combiner unit 40, an automatic speech recognition unit 51, a phonetic transcription unit 75, a keyboard 3, a microphone 7 and a display 57.
In operation, the user inputs either a voice query via the microphone 7 or a typed query via the keyboard 3 and the query is processed either by the automatic speech recognition unit 51 or the phonetic transcription unit 75 to generate corresponding phoneme and word data. The phoneme data is input to the phoneme matcher 33 which is operable to perform a phoneme search in the annotation database 31 with reference to a phoneme index 35. Similarly, the word data is input to the word matcher 37 which is operable to search the annotation database 31 with reference to the word index 38. The results of the phoneme and word search of the annotation database are then input to the combiner unit 40 which uses these results to retrieve a ranked list of documents 41 from the document database 29 which are output to the display unit 57 for display to the user 39.
In this embodiment, the annotation data for each document comprises a combined phoneme (or phoneme-like) and word lattice. FIG. 2 illustrates the form of the phoneme and word lattice annotation data generated for the spoken annotation “picture of the Taj Mahal”. As shown, the phoneme and word lattice is an acyclic directed graph with a single entry point and a single exit point. It represents different parses of the user's input. It is not simply a sequence of words with alternatives, since each word does not have to be replaced by a single alternative, one word can be substituted for two or more words or phonemes, and the whole structure can form a substitution for one or more words or phonemes. Therefore, the density of the data within the phoneme and word lattice annotation data essentially remains linear throughout the annotation data, rather than growing exponentially as in the case of a system which generates the N-best word lists for the annotation input.
In this embodiment, the annotation data for each document (d) is stored in the annotation database 31 and has the following general form:
HEADER
    • flag if word if phoneme if mixed
    • time index associating the location of blocks of annotation data within memory to a given time point.
    • word set used (i.e. the dictionary)
    • phoneme set used
    • the language to which the vocabulary pertains
    • phoneme probability data
Block(i) i=0, 1, 2, . . .
    • node nj j=0, 1, 2, . . .
      • time offset of node from start of block
      • phoneme links (k) k=0, 1, 2 . . . offset to node nj=nk−nj (nk is node to which link K extends) or if nk is in block(i+1) offset to node nj=nk+Nb−nj (where Nb is the number of nodes in block(i)) phoneme associated with link (k)
      • word links (l) l=0, 1, 2, . . . offset to node nj=ni−nj (nj is node to which link l extends) or if nk is in block(i+1) offset to node nj=nk+Nb−nj(where Nb is the number of nodes in block(i)) word associated with link (l)
The flag identifying if the annotation data is word annotation data, phoneme annotation data or if it is mixed is provided since the annotation data may include just word data, just phoneme data or both word and phoneme data.
In this embodiment the annotation data is divided into blocks (B) of nodes (n) in order to allow the search to jump into the middle of the annotation data. The header therefore includes a time index which associates the location of the blocks of annotation data within the memory to a given time offset between the time of start and the time corresponding to the beginning of the block.
The header also includes data defining the word set used (i.e. the dictionary), the phoneme set used and their probabilities and the language to which the vocabulary pertains. The header may also include details of the automatic speech recognition system or the phonetic transcription system used to generate the annotation data and any appropriate settings thereof which were used during the generation of the annotation data.
The blocks of annotation data then follow the header and identify, for each node in the block, the time offset of the node from the start of the block, the phoneme links which connect that node to other nodes by phonemes and word links which connect that node to other nodes by words. Each phoneme link and word link identifies the phoneme or word which is associated with the link. They also identify the offset to the current node. For example, if node n50 is linked to node n55 by a phoneme link, then the offset to node n50 is 5. As those skilled in the art will appreciate, using an offset indication like this allows the division of the continuous annotation data into separate blocks.
In an embodiment where an automatic speech recognition unit outputs weightings indicative of the confidence of the speech recognition unit's output, these weightings or confidence scores would also be included within the data structure. In particular, a confidence score would be provided for each node which is indicative of the confidence of arriving at the node and each of the phoneme and word links would include a transition score depending upon the weighting given to the corresponding phoneme or word. These weightings would then be used to control the search and retrieval of the data files by discarding those matches which have a low confidence score.
In order to provide an efficient retrieval method, a word indexing scheme and a phoneme indexing scheme is used in order to identify portions in the annotation database 31 against which a direct comparison with the input query is made. The word index 38 and the way that it is used is well known to those skilled in the art and will not be described further. However, the way in which the phoneme index 35 is generated and subsequently used to identify portions of the annotation database 31 for comparison with the input query will now be described in more detail.
As mentioned above, the use of a single phoneme as the key for a phoneme index will not provide sufficient discrimination, since each phoneme will occur several thousand times in the annotation database 31. Further, since current automatic speech recognition systems have a relatively poor phoneme recognition rate (60 to 70%), indexing using the phonemes directly will make it difficult to retrieve the data where the query phoneme or the annotation phoneme was misrecognised. Since the automatic speech recognition system tends to produce decoding errors for similar sounding phonemes, such as /s/ and /z/ and not highly dissimilar phonemes, such as /z/ and /g/, the error rate of indexing can be greatly reduced by indexing on confusable clusters of phonemes rather than individual phonemes.
Considering, for example, the situation where a query and an annotation exist for the word “sheep” and the query (Q) comprises the sequence of phonemes /sh//iy//p/ and the annotation (A) comprises the sequence of phonemes /s//eh//p/. If the sequence of three query phonemes is used as an index into the annotation (in the form of a trigram), then it will not be possible to retrieve the data since the sequence of query phonemes does not match the sequence of annotation phonemes. However, if the phonemes are clustered into confusable sets, and the phonemes in the query and in the annotation are classified into their respective sets, then there is a better chance that there will be a match between the query and the annotation if they both sound alike. For example, if the following phoneme classifications are defined: C 1 = { / s / / z / / sh / / zh / } C 2 = { / t / / k / / g / / b / / p / } C 5 = { / eh / / ih / / iy / }
and the query phonemes and the annotation phonemes in the above illustration are classified into these classes, then this will result in the following cluster trigrams:
C(Q)={C1 C5 C2}
C(A)={C1 C5 C2}
Therefore, matching using the clustered query and annotation will now work since C(Q)=C(A) and the data can be retrieved.
In this embodiment, a hash indexing technique is used which tries to create a unique mapping between a key and an entry in a list. In this embodiment trigrams of the above phoneme clusters are used as the key to the index. The way that the phoneme index 35 is created and used will now be described in more detail.
In order to create the phoneme index 35 the phoneme annotation data stored in the annotation database 31 is converted into phoneme cluster trigrams. The way that this is achieved is illustrated in FIG. 3. In particular, FIG. 3 a schematically illustrates the block form of the annotation data which is stored in the database 31 for one of the documents (d) stored in the document database 29. As shown, the annotation data comprises successive blocks of data B0 d to BM-1 d. As mentioned above, the annotation data within each block includes the nodes within the block and the phoneme and word links which are associated with the nodes. In order to illustrate the indexing method used in this embodiment, the remaining description will assume that the annotation data for document d includes a canonical sequence of phonemes i.e. one with no alternatives. FIG. 3 b illustrates the canonical sequence of phonemes within the ith block for the annotation data for document d. As shown, block Bd i comprises the canonical sequence of phonemes a0 di to aNdi di which extend between nodes n0 di to nNdi di. FIG. 3 c illustrates the overlapping “cluster trigrams” 101 generated for the sequence of annotation phonemes in block Bd i shown in FIG. 3 b. As shown in FIG. 3 c, the cluster in which each of the annotation phonemes in block Bd i belongs is determined. Then cluster trigrams are determined from overlapping groups of three cluster identifications for the sequence of annotation phonemes. In particular, the first cluster trigram determined is C(adi 0) C(adi 1) C(adi 2) then cluster trigram C(adi 1) C(adi 2) C(adi 3) etc. Although not shown in FIG. 3 c, it is also necessary to consider the trigrams which bridge adjacent blocks. For example, in this embodiment, it would be necessary to consider the following cluster trigrams: C(adi-1 N(di-1)-1) C(adi-1 Ndi-1) C(adi 0) and cluster trigram C(adi-1 Ndi-1) C(adi 0) C(adi 1).
To create the index, a large table or array, A, having S entries is created. In this embodiment, each entry is addressed by an index (IDX) which takes a value between zero and S-1 and each entry includes a data field to store the key (KEY) associated with the entry and a data field to store the pointers which identify the relevant locations within the annotation database where the phoneme data associated with the KEY can be found. Initially each of the entries is empty. The size of the table depends on the number of different phoneme clusters and the number of cluster identifications in the key. In this case, three cluster identifications (trigrams) are provided in each key. If there are ten clusters, then the number of different possible keys is 310. Therefore, in this case, S should, theoretically be made approximately equal to 310. However, in practice, some of the possible keys are unlikely to occur. Therefore, the size of the index can be set to have some initial size and data can be added to the index until more than a predetermined percentage of the entries are full. At this point, a bigger table can be created and the data from the old table copied over to the new table. This process can be repeated until there is no more data to be added to the index. Although this means that some memory is wasted, this is insignificant compared to the memory required for the pointers which will be stored in the table.
The way that data is added to the table will now be explained with reference to FIG. 4. As shown, in step s1, the system calculates the value of a function (f(KEY)) which is dependent upon the key, i.e. a function of the current cluster trigram, which value is used as the index (IDX) into the table A. In particular, the function f(KEY) defines a mapping between the cluster trigram and an entry in the table A and always yields a number between zero and S-1. In this embodiment, the function used is:
[C[1]KcC[2]KcC[3]Kc]mod S  (1)
where Kc is the number of phoneme clusters and C[1] is the number of the cluster to which the first annotation phoneme in the trigram belongs, C[2] is the number of the cluster to which the second annotation phoneme in the trigram belongs and C[3] is the number of the cluster to which the third annotation phoneme belongs. For example, for the illustration above where C(A)={c1c5c2}, C[1]=1, C[2]=5 and C[3]=2.
Once IDX has been determined in step s1, the processing proceeds to step s3 where the system checks the corresponding entry in the table, A, and determines whether or not the key stored in that entry (KEY) is the same as the key for the current phoneme cluster trigram (key) or is the null key (indicating that this entry is empty). If in step s3 the system determines that the key stored in the entry (KEY) matches the current key (key) or the null key, then the processing proceeds to step s5 where a pointer is added to that entry of the table, A, which points to the node associated with the first phoneme in the phoneme trigram associated with the current input key. For example, for the key c(a0 di) c(a1 di) c(a2 di), the data which would be added to the table would be a pointer which points to the node n0 di since annotation phoneme a0 di is associated with node n0 di. If the key stored in the entry is currently the null key, then in step s5, the system also changes the key for the entry (KEY) to the current key (key). The processing of the current cluster trigram then ends and a similar processing is performed on the next cluster trigram. If at step s3, the processing determines that the IDXth entry in the table, A, has already been assigned to a different cluster trigram, then the processing proceeds to step s7 where the system tries another entry in the table by changing the value of IDX in some predetermined way. In this embodiment, this is achieved by calculating:
IDX=(IDX+V)mod S  (2)
where V is some fixed number which is not a factor of S (other than 1). The reason that V should not be a factor of S is that this ensures that all entries in the table are tried. For example, if S=10 and V=2 then this technique would simply keep trying either just the odd or just the even entries of table A. In order to avoid this problem, S should preferably be prime. After step s7, the processing returns to step s3.
Once all the cluster trigrams in all the blocks of each annotation have been processed in this way, the table, A, is stored as the phoneme index 35. FIG. 5 illustrates the form of a phoneme index that is generated by the above processing. As can be seen from FIG. 5, each entry in the table includes the index number (IDX) of the entry, the key (KEY) associated with the entry and (if the key is not the null key) one or more pointers pointing to nodes in the annotation database 31. As shown, in this embodiment, these pointers have the form n[p,q,r] where p is the annotation for document p, q is the qth block of nodes within that annotation data and r is the rth node within that block.
The way that the phoneme matcher 33 uses the phoneme index 35 in response to an input query in order to identify portions of the annotation database 31 for matching with the input query will now be described with reference to FIGS. 6 to 8.
When a user inputs a query the phoneme data generated either by the automatic speech recognition unit 51 or the phonetic transcription unit 75 (depending upon whether the input query was received through the microphone or through the keyboard) is input to the phoneme matcher 33. FIG. 6 illustrates the processing steps performed by the phoneme matcher 33 on this phoneme data. As shown, in step s11, the phoneme matcher 33 converts the received query phoneme data into overlapping phoneme trigrams. FIG. 7 a illustrates a sequence of query phonemes q0 to q5 representative of phoneme data received by the phoneme matcher 33 and FIG. 7 b illustrates how this sequence of query phonemes is converted into five overlapping trigrams of query phonemes 103. The processing then proceeds to step s13 where each of the query phoneme trigrams is converted into phoneme cluster trigrams 105, as illustrated in FIG. 7 c, by classifying each of the query phonemes in the phoneme trigram into one of the above classes or clusters. The processing then proceeds to step s15 where the phoneme matcher 33 uses each of the cluster trigrams generated for the input query to address the phoneme index 35 in order to identify relevant locations in the annotation database 31.
FIG. 8 illustrates the processing steps used by the phoneme matcher 33 in carrying out step s15. As shown, in step s151, the phoneme matcher 33 calculates the index (IDX) for the entry in the table, A, by inserting the current query cluster trigram into the function defined above in equation (1). The processing then proceeds to step s153 where the phoneme matcher 33 checks the corresponding entry in the table, A, and determines whether or not the key stored in that entry (KEY) is the null key (indicating that this entry is empty). If it is, then the processing proceeds to step s155 where the phoneme matcher 33 determines that there is no corresponding annotation in the annotation database and outputs an appropriate output to the combiner unit 40. The processing then ends.
If at step s153 the phoneme matcher 33 determines that the key stored in the entry (KEY) is not equal to the null key, the processing proceeds to step s157 where the phoneme matcher 33 determines whether or not the key stored in the entry (KEY) is the same as the key for the current query cluster trigram (key). If it is then the processing proceeds to step s159 where the phoneme matcher 33 retrieves the pointers from that entry. The processing then ends. If, however, the phoneme matcher 33 determines, in step s157, that the key for the entry (KEY) does not equal the key for the current query cluster trigram (key), then the processing proceeds to step s161 where the phoneme matcher tries another entry in the table by changing the value of the index (IDX) using equation (2) given above and then returning to step s153.
Once the phoneme matcher 33 retrieves the pointers from the index or determines that there is no data stored for the current query cluster trigram, the phoneme matcher 33 then performs a similar processing for the next query cluster trigram until all the query cluster trigrams have been processed in this way. The processing then proceeds to step s17 shown in FIG. 6, where the phoneme matcher 33 uses the pointers identified in step s15 to identify regions within the annotation database 31 which will be matched with the actual phoneme data received by the phoneme matcher 33. In this embodiment, these regions are identified by comparing the pointers retrieved in step s159 for successive query cluster trigrams and looking for pointers which point to portions in the annotation database 31 which are next to each other. For example, referring to the phoneme index illustrated in FIG. 5, if the nth query cluster trigram is c5c3c6 and the n+1th query cluster trigram is c3c6c1, then the phoneme matcher 33 will identify node 40 of the 32nd block of annotation data for the 3rd document as being a region of the annotation database for further processing in step s19. This is because the pointers stored in the phoneme index 35 for the key c5c3c6 includes a pointer to node n[3,32,40] and the pointers stored in the phoneme index 35 for the key c3c6c1 includes a pointer to node n[3,32,41], which is immediately after node n[3,32,40] and is therefore consistent with a portion of an annotation having successive cluster trigrams c5c3c6 and then c3c6c1 which may match with the input query.
After the phoneme matcher 33 has identified the regions in step s17, it performs a phoneme comparison between the received query phoneme data and the phoneme data stored in the annotation database 31 at the regions identified in step s17. This phoneme comparison can be performed by comparing M-grams of the query with similar M-grams of the annotation (as described in the applicant's earlier UK application GB 9905201.1, the content of which is incorporated herein by reference) or by performing a dynamic programming comparison between the sequence of query phonemes and the sequence of annotation phonemes (using, for example, one of the techniques described in the applicant's earlier UK application GB 9925574.7, the content of which is incorporated herein by reference). The results of these phoneme comparisons are then output to the combiner unit 40 where the results are combined with the output from the word matcher 37 in order to retrieve and rank the appropriate documents from the document database 29.
In the above description, it has been assumed that the phonemes have been classified into a number of sets of confusable phonemes. The way that these phoneme clusters are determined in this embodiment will now be described. If two phoneme decodings have been made, once during the annotation phase and once during the query phase, then the probability of the two decodings, p1 and p2, coming from the same source is given by:
P(p 1 ,p 2 |m 1 ,m 2)=Σx P(p 1 |x,m 1)P(p 2 |x,m 2)P(x)  (3)
where P(p|x,m) is the probability of decoding phoneme x as phoneme p when decoding method m is used and P(x) is the probability of phoneme x occurring. The decoding methods m1 and m2 need to be distinguished since one of the decodings may come from a text-to-phoneme converter within the phonetic transcription unit 75 whilst the other may come from the automatic speech recognition unit 51 and these two different decoding techniques will suffer from different types of confusion. Each of these probabilities in equation (3) above can be determined in advance during a training routine by applying known speech to the automatic speech recognition unit 51 and known text into the phonetic transcription unit 75 and by monitoring the output from the recognition unit 51 and the phonetic transcription unit 75 respectively. The way that such a training routine would be performed will be well known to those skilled in the art and will not, therefore, be described further here.
If it is assumed that there are Kc phoneme clusters or classes and for each cluster there is a many-to-one mapping between decoded phonemes and the clusters. This mapping will depend on the decoding method employed. For example, an /s/ decoded by the automatic speech recognition unit 51 may be in cluster c4, but an /s/ decoded by the phonetic transcription unit 75 may be in cluster c2. Therefore, for any particular cluster (Ki) there will be a probability that two decodings from the same source are classed into that cluster which is determined by summing the probability given in equation (3) above for all possible combinations of decodings, p1 and p2, which belong to that cluster, i.e. by calculating: P ( Assigned to same cluster K i ) = p 1 K i p 2 K i P ( p 1 , p 2 , m 1 , m 2 ) ( 4 )
The probability that all decodings are correctly classified is therefore given by: P ( All correctly classified ) = i = 1 K c P ( Assigned to same cluster K i ) ( 5 )
The task of defining the clusters aims, therefore, to maximise P(all correctly classified), subject to the constraints that each phoneme (via a particular decoding method) is in one and only one cluster. In this embodiment, a Monte Carlo algorithm is used in order to determine phoneme classifications which maximise this probability.
FIG. 9 a illustrates the steps involved in this Monte Carlo algorithm. Initially, in step s200, the number of phoneme clusters that will be used is determined. As those skilled in the art will appreciate, if there are too few clusters then there will be insufficient discrimination and if there are too many clusters then the data may not be retrievable. In this embodiment, in order to provide classifications which are sufficiently discriminative, ten clusters are defined. Once the number of clusters has been determined, the system randomly assigns, in step s201, phonemes to these clusters and stores this as a current configuration. The processing then proceeds to step s203 where the system determines the probability that the phonemes are correctly classified in the clusters for the current configuration, i.e. the system calculates the probability given in equation (5) above.
The processing then proceeds to step s205 where the system randomly selects a phoneme and a target cluster to which the selected phoneme may be moved. Then, in step s207, the system calculates what the probability given in equation (5) would be if the selected phoneme is moved into the target cluster. Then in step s209, the system compares this new probability calculated in step s207 with the probability for the current configuration which was calculated in step s203. If the new probability is higher than the probability for the current configuration, then the processing passes to step s211 where the system moves the selected phoneme to the target cluster to replace the current configuration. The processing then proceeds to step s213 where the system determines whether or not the probability calculated for the new configuration is better than the “best ever” probability. If it is, then the processing proceeds to step s215 where the system stores the current configuration as the best ever configuration that it has encountered. Otherwise step s215 is skipped and the processing proceeds to step s219 where the system determines whether or not convergence has been reached.
If at step s209, the system determines that the new probability is not higher than the probability for the current configuration, then the processing proceeds to step s217 where the system moves the selected phoneme to the target cluster to replace the current configuration with a probability dependent upon the difference between the new probability and the probability for the current configuration. For example, a difference probability can be defined as: d = 1 2 - λ ( s 1 - s 2 ) ( 6 )
where s1 is the probability determined for the current configuration and s2 is the probability for the proposed new configuration and λ is a parameter which can be tuned to get best performance. Therefore, when the two probabilities are the same d=½ and when the probability for the new configuration is massively worse than the probability for the current configuration d will be approximately equal to zero. Then if a random number generator is used which randomly picks a number between zero and one, then the proposed configuration will replace the current configuration if d>r, otherwise the proposed configuration is discarded.
After step s217, the system proceeds to step s219 where it determines whether or not convergence has been reached. If it has, then the processing ends and the phoneme clusters are stored. If convergence has not been reached, then the processing proceeds to step s221 where a random value (RV) between zero and one is chosen and then compared with a threshold (Th) in step s223. If the random value, RV, is less than the threshold then the processing returns to step s205 above and the procedure is repeated. On the other hand, if the random value, RV is not less than the threshold, then the processing proceeds to step s225 where the best ever configuration is copied into the current configuration and then the processing again returns to step s205. However, by setting the threshold Th to be close to one ensures that the best ever configuration is only copied into the current configuration very occasionally. As those skilled in the art will appreciate the processing of steps s221 to s225 is provided to try and ensure that the system does not remain stuck in a local minimum.
The inventors have found that performing this clustering algorithm for the English phoneme set with ten clusters and for a given user, when the decodings come from the automatic speech recognition unit 51, gives the following phoneme clusters:
c1={aa ae ah aw eh ey uw}
c2={ao l oy r w}
c3={d dh t th}
c4={ax ea er hh oh ow ua}
c5={m sil}
c6={b f p v}
c7={s z}
c8={ch g jh k sh zh}
c9={n ng}
c10={ay ia ih iy uh y}
and when the decodings come from the phonetic transcription unit 75, gives the following phoneme clusters:
c1={aa ae ah aw eh ey uh uw}
c2={ao l oy r w}
c3={d dh t th}
c4={ax ea er hh oh ow ua}
c5={m sil}
c6={b f p v}
c7={s z}
c8={ch g jh k sh zh}
c9={n ng}
c10={ay ia ih iy y}
As those skilled in the art will appreciate, the phoneme clusters for the text to phoneme transcription unit 75 are predominantly the same as those for the automatic speech recognition 51, with the exception of the “uh” phone which is in cluster c1 while it is in cluster c9 for the clusters of the automatic speech recognition unit 51. As those skilled in the art will appreciate, the clusters given above are given by way of example only. The precise clusters that are used will depend on the matching method used to compare the phonemes in the clusters.
Alternative Embodiments
In the above embodiment, the document database 29, the annotation database 31 and the speech recognition unit 51 were all located within the user terminal 59. As those skilled in the art will appreciate, this is not essential. FIG. 10 illustrates an embodiment in which the document database 29 and the search engine 53 are located in a remote server 60 and in which the user terminal 59 accesses the database 29 via the network interface unit 67 and 69 and a data network 68 (such as the Internet). In this embodiment, both the documents and the annotations are stored in the database 29. In this embodiment, the user terminal 59 can only receive voice queries from the microphone 7. These queries are converted into phoneme and word data by the automatic speech recognition unit 51. This data is then passed to the control unit 55 which controls the transmission of data over the data network 68 to the search engine 53 located within the remote server 60. The search engine 53 then uses the phoneme index to carry out a search in the database 29 in a similar manner to the way in which the search was performed in the above embodiment. The results of the search are then transmitted back from the search engine 53 to the control unit 55 via the data network 68. The control unit 55 then considers the search results received back from the network and displays appropriate data on the display 57 for viewing by the user 39.
In addition to locating the database 29 and the search engine 53 in the remote server 60, it is also possible to locate the automatic speech recognition unit 51 in the remote server 60. Such an embodiment is shown in FIG. 11. As shown, in this embodiment, the input voice query from the user is passed via input line 61 to a speech encoding unit 73 which is operable to encode the speech for efficient transfer through the data network 68. The encoded data is then passed to the control unit 55 which transmits the data over the network 68 to remote server 60, where it is processed by the automatic speech recognition unit 51. In this embodiment, the speech recognition unit is operable to only generate phoneme data which is then passed to the search engine for use in searching the database 29 using the phoneme index 35. The search results generated by the search engine are then passed, via the network interface 69 and the network 68, back to the user terminal 59. The search results received back from the remote server are then passed via the network interface unit 67 to the control unit 55 which analyses the results and generates and displays appropriate data on the display 57 for viewing by the user 39.
In a similar manner, a user terminal 59 may be provided which only allows typed inputs from the user and which has the search engine and the database located in the remote server. In such an embodiment, the phonetic transcription unit 75 may be located in the remote server 60 as well.
In the above embodiment, the annotation database and the document database were separate. As those skilled in the art will appreciate, in some embodiments, the annotation database and the document database may form a single database. Additionally, the annotations in the annotation database may form the data to be retrieved.
In the above embodiment, the annotation database was searched using both phonemes and words. As those skilled in the art will appreciate, the phoneme index described above and its use in searching the annotation database may be used in a system which does not search using words as well.
In the above embodiment, a phoneme index was described. As those skilled in the art will appreciate, the above technique can be used for features other than phonemes, such as phones, syllables or katakana (Japanese alphabet) or any other sub-word unit of speech. This indexing technique could also be used in other applications, such as the indexing of DNA sequences and the like.
In the above embodiment, a hash indexing technique has been described. As those skilled in the art will appreciate, other indexing techniques can be used in order to identify portions of the database for carrying out a detailed phoneme search using phoneme data generated from the input query.
In the above embodiment, the function defined in equation (1) above was used to define the mapping between the cluster trigram and the entry in the table. However, with this technique, if C[1] or C[2] or C[3] equals zero then this will yield zero. Instead, the function used could be:
(C[3]+Kc(C[2]+KcC[1]))mod S  (7)
or, for a general N-gram of length n: ( i = 1 n K c ( n - i ) · C [ i ] ) mod S ( 8 )
In the above embodiment, the pointers stored in the index were of the form n[p,q,r], where p is the annotation for document p, q is the qth block of nodes within that annotation and r is the rth node within that block. As those skilled in the art will appreciate, other types of pointers could be used. For example, the count of the node since the start of the lattice or the time and rank (where more than one node have the same time) of the relevant nodes could be used.
In the above embodiment, when the query is being applied to the index, each of the query cluster trigrams were applied to the index and the appropriate entries found. These entries were then compared in order to identify portions of the phoneme lattice for further searching. Alternatively, the processing of the query cluster trigrams may be performed incrementally, i.e. identify all the entries for the first query trigram, then obtain those for the second query trigram and retain those which are close in time to those of the first trigram etc.
In the above embodiment, when identifying regions where successive cluster trigrams occur close in time, the system compared the node numbers in the pointers which are retrieved. However, in some embodiments, it is better to compare the time offsets stored for the nodes. This is because in some applications, the lattice will have many branches and two nodes which are very close in time could have completely different node numbers. The aim of this part of the system is to identify cluster trigrams which are in chronological order within the annotation and which occur within a limited period of time of each other. In this way, if there is an error in a trigram, it will only result in two to three trigram misses. Consequently, the time leeway should be comparable to about four or five phonemes which is about 0.2 to 0.3 seconds.

Claims (48)

1. An apparatus for identifying one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of sub-word units, said apparatus comprising:
a memory for storing data defining a plurality of sub-word unit classes, each class comprising sub-word units that are confusable with other sub-word units in the same class;
a memory for storing an index having a plurality of entries, each entry having an associated identifier for identifying the entry and each entry comprising:
a key associated with the entry and which is related to the identifier for the entry in a predetermined manner; and
a number of pointers which point to portions of data in the database which correspond to the key associated with the entry,
wherein each key comprises a sequence of sub-word unit classifications which is derived from a corresponding sequence of sub-word units appearing in the database by classifying each of the sub-word units in the sequence into one of the plurality of sub-word unit classes;
means for classifying each of the sub-word units in the input query into one of the plurality of sub-word unit classes and for defining one or more sub-sequences of query sub-word unit classifications;
means for determining a corresponding identifier for an entry in the index for each of the one or more sub-sequences of query sub-word unit classifications;
means for comparing the key associated with each of the determined identifiers determined by said determining means with the corresponding sub-sequence of query sub-word unit classifications; and
means for retrieving one or more pointers from the index in accordance with the output of said comparing means, which one or more pointers identify the one or more portions of data in the database for comparison with the input query.
2. An apparatus according to claim 1, wherein the sub-word units are phonemes.
3. An apparatus according to claim 1, wherein at least ten sub-word unit classes are defined in advance.
4. An apparatus according to claim 1, wherein each key is related to the corresponding identifier by a predetermined mathematical function.
5. An apparatus according to claim 4, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 W [ C [ i ] K c ] ) Mod S ,
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and W is the number of sub-word unit classifications in each key.
6. An apparatus according to claim 4, wherein said determining means is operable to determine a corresponding identifier for each subsequence of query sub-word unit classifications using the predetermined mathematical function.
7. An apparatus according to claim 4, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 n K c ( n - i ) · C [ i ] ) Mod S
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and n is the number of sub-word unit classifications in each key.
8. An apparatus according to claim 1, wherein said determining means is operable to identifier a new identifier for another entry in the index for a subsequence of query sub-word unit classifications if said comparing means determines that the key for the identifier is not the same as the subsequence of query sub-word unit classifications.
9. An apparatus according to claim 8, wherein said determining means is operable to determine a new identifier using the following equation:

IDX=[IDX+V]Mod S,
where IDX is the identifier, S is the number of entries in the index, and V is a predetermined number.
10. An apparatus according to claim 1, wherein the key for one or more of the entries is a null key indicating that there are no pointers stored in the index for that entry.
11. An apparatus according to claim 1, wherein the input query is a spoken query and said apparatus further comprises a speech recognition system for processing the spoken query and for outputting the sequence of sub-word units.
12. An apparatus for searching a database in response to a query input by a user, the database comprising a plurality of sequences of sub-word units and the query comprising at least one sequence of sub-word units, said apparatus comprising:
an apparatus according to any of claims 1 to 11 for identifying one or more portions of data in the database for comparison with the input query; and
means for comparing the one a more sequences of query sub-word units with the identified one or more portions of data in the database.
13. An apparatus according to claim 12, wherein said means for comparing the input query with the portions of data in the database uses a dynamic programming comparison technique.
14. An apparatus according to claim 12, further comprising means for retrieving one or more data files in accordance with the results of said comparing means.
15. An apparatus according to claim 1, wherein the input query is a typed query and said apparatus further comprises means for converting the typed query into the sequence of sub-word units.
16. An apparatus for identifying one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of features, said apparatus comprising:
a memory for storing data defining a plurality of feature classes, each class comprising features that are confusable with other features in the same class;
a memory for storing an index having a plurality of entries, each entry having an associated identifier for identifying the entry and each entry comprising:
a key associated with the entry and which is related to the identifier for the entry in a predetermined manner; and
a number of pointers which point to portions of data in the database which correspond to the key associated with the entry,
wherein each key comprises a sequence of feature classifications which is derived from a corresponding sequence of features appearing in the database by classifying each of the features in the sequence into one of the plurality of feature classes;
means for classifying each of the features in the input query into one of the plurality of feature classes and for defining one or more sub-sequences of query feature classifications;
means for determining a corresponding identifier for an entry in the index for each of the one or more sub-sequences of query feature classifications;
means for comparing the key associated with each of the determined identifiers determined by said determining means with the corresponding sub-sequence of query feature classifications; and
means for retrieving one or more pointers from the index in accordance with the output of said comparing means, which one or more pointers identify the one or more portions of data in the database for comparison with the input query.
17. A method of identifying one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of sub-word units, the method comprising the steps of:
storing data defining a plurality of sub-word unit classes, each class comprising sub-word units that are confusable with other sub-word units in the same class;
storing an index having a plurality of entries, each entry having an associated identifier for identifying the entry,
a key associated with the entry and which is related to the identifier for the entry in a predetermined manner, and
a number of pointers which point to portions of data in the database which correspond to the key associated with the entry.
wherein each key comprises a sequence of sub-word unit classifications which is derived from a corresponding sequence of sub-word units appearing in the database by classifying each of the sub-word units in the sequence into one of the plurality of sub-word unit classes;
classifying each of the sub-word units in the input query into one of the plurality of sub-word unit classes and for defining one or more sub-sequences of query sub-word unit classifications;
determining a corresponding identifier for an entry in the index for each of the one or more sub-sequences of query sub-word unit classifications;
comparing the key associated with each of the determined identifiers determined in said determining step with the corresponding sub-sequence of query sub-word unit classifications; and
retrieving one or more pointers from the index in accordance with the output of said comparing step, which one or more pointers identify the one or more portions of data in the database for comparison with the input query.
18. A method according to claim 17, wherein the sub-word units are phonemes.
19. A method according to claim 17, wherein at least ten sub-word unit classes are defined in advance.
20. A method according to claim 17, wherein each key is related to the corresponding identifier by a predetermined mathematical function.
21. A method according to claim 20, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 W [ C [ i ] K c ] ) Mod S
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and W is the number of sub-word unit classifications in each key.
22. A method according to claim 20, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 n K c ( n - i ) · C [ i ] ) Mod S
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and n is the number of sub-word unit classifications in each key.
23. A method according to claim 17, wherein said determining step identifies a new identifier for another entry in the index for a subsequence of query sub-word unit classifications if said comparing step determines that the key for the identifier is not the same as the subsequence of query sub-word unit classifications.
24. A method according to claim 23, wherein said determining step determines a new identifier using the following equation:

IDX=[IDX+V] Mod S,
where IDX is the identifier, S is the number of entries in the index, and V is a predetermined number.
25. A method according to claim 24, wherein said determining step determines a corresponding identifier for each subsequence of query sub-word unit classifications using the predetermined mathematical function.
26. A method according to claim 17, wherein the input query is a typed query and wherein the method further comprises a step of converting the typed query into the sequence of sub-word units.
27. A method according to claim 17, wherein the input query is a spoken query and wherein the method further comprises a step of using a speech recognition system to process the spoken query to generate the sequence of sub-word units.
28. A method of searching a database in response to a query input by a user, the database comprising a plurality of sequences of sub-word units and the query comprising at least one sequence of sub-word units, said method comprising:
method steps of claim 17 for identifying one or more portions of data in the database for comparison with the input query; and
a step of comparing the one or more sequences of query sub-word units with the identified one or more portions of data in the database.
29. A method according to claim 28, wherein said comparing step uses a dynamic programming comparison technique to compare the input query with the portions of data.
30. A method according to claim 28, further comprising a step of retrieving one or more data files in dependence upon the results of said comparing step.
31. A method according to claim 17, wherein the key for one or more of the entries is a null key indicating that there are no pointers stored in the index for that entry.
32. An apparatus for identifying one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of sub-word units, the apparatus comprising:
a first memory operable to store data defining a plurality of sub-word unit classes, each class comprising sub-word units that are confusable with other sub-word units in the same class;
a second memory operable to more an index having a plurality of entries, each entry having an associated identifier for identifying the entry and each entry comprising:
a key associated with the entry and which is related to the identifier for the entry in a predetermined manner; and
a number of pointers which point to portions of data in the database which correspond to the key for the entry;
wherein each key comprises a sequence of sub-word unit classifications which is derived from a corresponding sequence of sub-word units appearing in the database by classifying each of the sub-word units in the sequence into one of the plurality of sub-word unit classes;
a classifier operable to classify each of the sub-word units in the input query into one of the plurality of sub-word unit classes and to define one or more sub-sequences of query sub-word unit classifications;
a determiner operable to determine a corresponding identifier for an entry in the index for each of the one or more sub-sequences of query sub-word unit classifications;
a comparator operable to compare the key associated with each of the determined identifiers determined by said determiner with the corresponding sub-sequence of query sub-word unit classifications; and
a retriever operable to retrieve one or more pointers from the index in accordance with the output of said comparator, which one or more pointers identify the one or more portions of data in the database for comparison with the input query.
33. An apparatus according to claim 32 wherein the sub-word units are phonemes.
34. An apparatus according to claim 32 wherein at least ten sub-word unit classes are defined in advance.
35. An apparatus according to claim 32 wherein each key is related to the corresponding identifier by a predetermined mathematical function.
36. An apparatus according to claim 35, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 W [ C [ i ] K c ] ) Mod S
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and W is the number of sub-word unit classifications in each key.
37. An apparatus according to claim 35, wherein each key is related to the corresponding identifier by the following equation: ( i = 1 n K c ( n - i ) · C [ i ] ) Mod S
where Kc is the number of sub-word unit classes, S is the number of entries in the index, C is the number of the sub-word class to which the ith sub-word unit in the sequence of sub-word units corresponding to the key belongs, and n is the number of sub-word unit classifications in each key.
38. An apparatus according to claim 35, wherein said determiner is operable to determine a corresponding identifier for each subsequence of query sub-word unit classifications using the predetermined mathematical function.
39. An apparatus according to claim 32, wherein said determiner is operable to identify a new identifier for another entry in the index for a subsequence of query sub-word unit classifications if said comparator determines that the key for the identifier is not the same as the subsequence of query sub-word unit classifications.
40. An apparatus according to claim 39, wherein said determiner is operable to determine a new identifier using the following equation:

IDX=[IDX+V]Mod S,
where IDX is the identifier, S is the number of entries in the index, and V is a predetermined number.
41. An apparatus according to claim 32, wherein the key for one or more of the entries is a null key indicating that there are no pointers stored in the index for that entry.
42. An apparatus according to claim 32, wherein the input query is a typed query and said apparatus further comprises a converter operable to convert the typed query into the sequence of sub-word units.
43. An apparatus according to claim 32, wherein the input query is a spoken query and said apparatus further comprises a speech recognition system operable to process the spoken query and to output the sequence of sub-word units.
44. An apparatus for searching a database in response to a query input by a user, the database comprising a plurality of sequences of sub-word units and the query comprising at least one sequence of sub-word units, said apparatus comprising:
an apparatus according to claim 32 operable to identify one or more portions of data in the database for comparison with the input query; and
a sequence comparator operable to compare the one or more sequences of query sub-word units with the identified one or more portions of data in the database.
45. An apparatus according to claim 44, wherein said sequence comparator is operable to compare the input query with the portions of data in the database using a dynamic programming comparison technique.
46. An apparatus according to claim 44, further comprising a data file retriever operable to retrieve one or more data files in accordance with the results of said sequence comparator.
47. An apparatus for identifying one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of features, said apparatus comprising:
a first memory operable to store data defining a plurality of feature classes, each class comprising features that are confusable with other features in the same class;
a second memory operable to store an index having a plurality of entries, each entry having an associated identifier for identifying the entry and each entry comprising:
a key associated with the entry and which is related to the identifier for to entry in a predetermined manner; and
a number of pointers which point to portions of data in the database which correspond to the key for the entry,
wherein each key comprises a sequence of feature classifications which is derived from a corresponding sequence of features appearing in the database by classifying each of the features in the sequence into one of the plurality of feature classes;
a classifier operable to classify each of the features in the input query into one of the plurality of feature classes and to define one or more sub-sequences of query feature classifications;
a determiner operable to determine a corresponding identifier for an entry in said index for each of said one or more sub-sequences of query feature classifications;
a comparator operable to compare the key associated with each of the determined identifiers determined by said determiner with the corresponding sub-sequence of query feature classifications; and
a retriever operable to retrieve one or more pointers from the index in accordance with the output of said comparator, which one or more pointers identify the one or more portions of data in the database for comparison with the input query.
48. A storage medium storing computer readable program code for executing a method of controlling a processor to identify one or more portions of data in a database for comparison with a query input by a user, the query and the portions of data each comprising a sequence of sub-word units, said program code comprising:
code for storing data defining a plurality of sub-word unit classes, each class comprising sub-word units that are confusable with other sub-word units in the same class;
code for storing an index having a plurality of entries, each entry having an associated identifier for identifying the entry and each entry comprising:
a key associated with the entry and which is related to the identifier for the entry in a predetermined manner, and
a number of pointers which point to portions of data in the database which correspond to the key for the entry,
wherein each key comprises a sequence of sub-word unit classifications which is derived from a corresponding sequence of sub-word units appearing in the database by classifying each of the sub-word units in the sequence into one of the plurality of sub-word unit classes;
code for classifying each of the sub-word units in the input query into one of the plurality of sub-word unit classes and defining one or more sub-sequences of query sub-word unit classifications;
code for determining a corresponding identifier for an entry in the index for each of the one or more sub-sequences of query sub-word unit classifications;
code for comparing the key associated with each of the determined identifiers determined by said determining code with the corresponding sub-sequence of query sub-word unit classifications; and
code for retrieving one or more pointers from the index in accordance with the output by said comparing code, which one or more pointers identify the one or more portion of data in the database for comparison with the input query.
US09/863,424 2000-06-21 2001-05-24 Indexing method and apparatus Expired - Fee Related US6873993B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GBGB0015233.0A GB0015233D0 (en) 2000-06-21 2000-06-21 Indexing method and apparatus
GB0015233.0 2000-06-21

Publications (2)

Publication Number Publication Date
US20020052870A1 US20020052870A1 (en) 2002-05-02
US6873993B2 true US6873993B2 (en) 2005-03-29

Family

ID=9894129

Family Applications (1)

Application Number Title Priority Date Filing Date
US09/863,424 Expired - Fee Related US6873993B2 (en) 2000-06-21 2001-05-24 Indexing method and apparatus

Country Status (4)

Country Link
US (1) US6873993B2 (en)
EP (1) EP1168199A3 (en)
JP (1) JP2002063199A (en)
GB (1) GB0015233D0 (en)

Cited By (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030204399A1 (en) * 2002-04-25 2003-10-30 Wolf Peter P. Key word and key phrase based speech recognizer for information retrieval systems
US20050210389A1 (en) * 2004-03-17 2005-09-22 Targit A/S Hyper related OLAP
US20060106843A1 (en) * 2004-11-17 2006-05-18 Targit A/S Database track history
US20060111890A1 (en) * 2004-11-24 2006-05-25 Microsoft Corporation Controlled manipulation of characters
US20060206324A1 (en) * 2005-02-05 2006-09-14 Aurix Limited Methods and apparatus relating to searching of spoken audio data
US20060265222A1 (en) * 2005-05-20 2006-11-23 Microsoft Corporation Method and apparatus for indexing speech
US20060282408A1 (en) * 2003-09-30 2006-12-14 Wisely David R Search system and method via proxy server
US20070033003A1 (en) * 2003-07-23 2007-02-08 Nexidia Inc. Spoken word spotting queries
US20070106685A1 (en) * 2005-11-09 2007-05-10 Podzinger Corp. Method and apparatus for updating speech recognition databases and reindexing audio and video content using the same
US20070106760A1 (en) * 2005-11-09 2007-05-10 Bbnt Solutions Llc Methods and apparatus for dynamic presentation of advertising, factual, and informational content using enhanced metadata in search-driven media applications
US20070106509A1 (en) * 2005-11-08 2007-05-10 Microsoft Corporation Indexing and searching speech with text meta-data
US20070106512A1 (en) * 2005-11-09 2007-05-10 Microsoft Corporation Speech index pruning
US20070106693A1 (en) * 2005-11-09 2007-05-10 Bbnt Solutions Llc Methods and apparatus for providing virtual media channels based on media search
US20070112837A1 (en) * 2005-11-09 2007-05-17 Bbnt Solutions Llc Method and apparatus for timed tagging of media content
US20070118873A1 (en) * 2005-11-09 2007-05-24 Bbnt Solutions Llc Methods and apparatus for merging media content
US20070143110A1 (en) * 2005-12-15 2007-06-21 Microsoft Corporation Time-anchored posterior indexing of speech
US20070143282A1 (en) * 2005-03-31 2007-06-21 Betz Jonathan T Anchor text summarization for corroboration
US20070150800A1 (en) * 2005-05-31 2007-06-28 Betz Jonathan T Unsupervised extraction of facts
US20070174262A1 (en) * 2003-05-15 2007-07-26 Morten Middelfart Presentation of data using meta-morphing
US20070250320A1 (en) * 2006-04-25 2007-10-25 General Motors Corporation Dynamic clustering of nametags in an automated speech recognition system
WO2007134293A2 (en) * 2006-05-12 2007-11-22 Nexidia, Inc. Wordspotting system
US7310600B1 (en) * 1999-10-28 2007-12-18 Canon Kabushiki Kaisha Language recognition using a similarity measure
US7313521B1 (en) 2000-03-04 2007-12-25 Georgia Tech Research Corporation Phonetic searching
US20080019620A1 (en) * 2006-07-21 2008-01-24 Gerald Vierheilig Lubricating unit
US20080071542A1 (en) * 2006-09-19 2008-03-20 Ke Yu Methods, systems, and products for indexing content
US20080104149A1 (en) * 2006-11-01 2008-05-01 Ephraim Meriwether Vishniac Managing Storage of Individually Accessible Data Units
US20080162125A1 (en) * 2006-12-28 2008-07-03 Motorola, Inc. Method and apparatus for language independent voice indexing and searching
US20080162472A1 (en) * 2006-12-28 2008-07-03 Motorola, Inc. Method and apparatus for voice searching in a mobile communication device
US20080270138A1 (en) * 2007-04-30 2008-10-30 Knight Michael J Audio content search engine
US20080301539A1 (en) * 2007-04-30 2008-12-04 Targit A/S Computer-implemented method and a computer system and a computer readable medium for creating videos, podcasts or slide presentations from a business intelligence application
US20090030894A1 (en) * 2007-07-23 2009-01-29 International Business Machines Corporation Spoken Document Retrieval using Multiple Speech Transcription Indices
US20090187845A1 (en) * 2006-05-16 2009-07-23 Targit A/S Method of preparing an intelligent dashboard for data monitoring
US20090216740A1 (en) * 2008-02-25 2009-08-27 Bhiksha Ramakrishnan Method for Indexing for Retrieving Documents Using Particles
US20090287486A1 (en) * 2008-05-14 2009-11-19 At&T Intellectual Property, Lp Methods and Apparatus to Generate a Speech Recognition Library
US20090288118A1 (en) * 2008-05-14 2009-11-19 At&T Intellectual Property, Lp Methods and Apparatus to Generate Relevance Rankings for Use by a Program Selector of a Media Presentation System
US20100153366A1 (en) * 2008-12-15 2010-06-17 Motorola, Inc. Assigning an indexing weight to a search term
US20100169274A1 (en) * 2006-11-01 2010-07-01 Ab Initio Technology Llc Managing storage of individually accessible data units
US20100179811A1 (en) * 2009-01-13 2010-07-15 Crim Identifying keyword occurrences in audio data
US7783628B2 (en) 2003-05-15 2010-08-24 Targit A/S Method and user interface for making a presentation of data using meta-morphing
US20100280827A1 (en) * 2009-04-30 2010-11-04 Microsoft Corporation Noise robust speech classifier ensemble
US20100324900A1 (en) * 2009-06-19 2010-12-23 Ronen Faifkov Searching in Audio Speech
US7911482B1 (en) 2006-01-06 2011-03-22 Videomining Corporation Method and system for efficient annotation of object trajectories in image sequences
US20110106792A1 (en) * 2009-11-05 2011-05-05 I2 Limited System and method for word matching and indexing
US7949674B2 (en) 2006-07-17 2011-05-24 Targit A/S Integration of documents with OLAP using search
US20110218802A1 (en) * 2010-03-08 2011-09-08 Shlomi Hai Bouganim Continuous Speech Recognition
WO2011112187A1 (en) * 2010-03-10 2011-09-15 Ab Initio Technology Llc Managing storage of individually accessible data units
US20110224982A1 (en) * 2010-03-12 2011-09-15 c/o Microsoft Corporation Automatic speech recognition based upon information retrieval methods
US8312022B2 (en) 2008-03-21 2012-11-13 Ramp Holdings, Inc. Search engine optimization
US8347202B1 (en) 2007-03-14 2013-01-01 Google Inc. Determining geographic locations for place names in a fact repository
US20130179430A1 (en) * 2012-01-09 2013-07-11 Wajam Internet Technologies Inc. Proxy server for application on mobile device
US8589165B1 (en) * 2007-09-20 2013-11-19 United Services Automobile Association (Usaa) Free text matching system and method
US8650175B2 (en) 2005-03-31 2014-02-11 Google Inc. User interface for facts query engine with snippets from information sources that include query terms and answer terms
US8682913B1 (en) 2005-03-31 2014-03-25 Google Inc. Corroborating facts extracted from multiple sources
US8719260B2 (en) 2005-05-31 2014-05-06 Google Inc. Identifying the unifying subject of a set of facts
US8751498B2 (en) 2006-10-20 2014-06-10 Google Inc. Finding and disambiguating references to entities on web pages
US8812435B1 (en) 2007-11-16 2014-08-19 Google Inc. Learning objects and facts from documents
US8831946B2 (en) 2007-07-23 2014-09-09 Nuance Communications, Inc. Method and system of indexing speech data
US8996470B1 (en) 2005-05-31 2015-03-31 Google Inc. System for ensuring the internal consistency of a fact repository
US20150255060A1 (en) * 2014-03-05 2015-09-10 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US20150255059A1 (en) * 2014-03-05 2015-09-10 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US9508345B1 (en) 2013-09-24 2016-11-29 Knowles Electronics, Llc Continuous voice sensing
US20170126738A1 (en) * 2013-03-14 2017-05-04 Daniel Shawcross Wilkerson Hard Object: Lightweight Hardware Enforcement of Encapsulation, Unforgeability, and Transactionality
US9811570B2 (en) 2011-07-08 2017-11-07 Ab Initio Technology Llc Managing storage of data for range-based searching
US9934166B2 (en) 2010-12-10 2018-04-03 Daniel Shawcross Wilkerson Hard object: constraining control flow and providing lightweight kernel crossings
US20190051296A1 (en) * 2017-08-09 2019-02-14 Lenovo (Singapore) Pte. Ltd. Performing action on active media content
US20200134010A1 (en) * 2018-10-26 2020-04-30 International Business Machines Corporation Correction of misspellings in qa system
US11392631B2 (en) * 2014-07-29 2022-07-19 Groupon, Inc. System and method for programmatic generation of attribute descriptors

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7062482B1 (en) * 2001-02-22 2006-06-13 Drugstore. Com Techniques for phonetic searching
US20050086705A1 (en) * 2003-08-26 2005-04-21 Jarman Matthew T. Method and apparatus for controlling play of an audio signal
US20050144003A1 (en) * 2003-12-08 2005-06-30 Nokia Corporation Multi-lingual speech synthesis
US20060036438A1 (en) * 2004-07-13 2006-02-16 Microsoft Corporation Efficient multimodal method to provide input to a computing device
US7725318B2 (en) * 2004-07-30 2010-05-25 Nice Systems Inc. System and method for improving the accuracy of audio searching
JP4587165B2 (en) * 2004-08-27 2010-11-24 キヤノン株式会社 Information processing apparatus and control method thereof
US8942985B2 (en) * 2004-11-16 2015-01-27 Microsoft Corporation Centralized method and system for clarifying voice commands
US8924212B1 (en) 2005-08-26 2014-12-30 At&T Intellectual Property Ii, L.P. System and method for robust access and entry to large structured data using voice form-filling
US8108452B2 (en) * 2006-01-12 2012-01-31 Yahoo! Inc. Keyword based audio comparison
US9275129B2 (en) * 2006-01-23 2016-03-01 Symantec Corporation Methods and systems to efficiently find similar and near-duplicate emails and files
US7925975B2 (en) 2006-03-10 2011-04-12 Microsoft Corporation Searching for commands to execute in applications
AU2014202186B2 (en) * 2006-11-01 2016-03-03 Ab Initio Technology Llc Managing storage of individually accessible data units
US8150170B2 (en) * 2008-05-30 2012-04-03 Microsoft Corporation Statistical approach to large-scale image annotation
US8301447B2 (en) * 2008-10-10 2012-10-30 Avaya Inc. Associating source information with phonetic indices
US20130179386A1 (en) * 2009-04-09 2013-07-11 Sigram Schindler Innovation expert system, ies, and its ptr data structure, ptr-ds
CN102023995B (en) * 2009-09-22 2013-01-30 株式会社理光 Speech retrieval apparatus and speech retrieval method
US9400790B2 (en) * 2009-12-09 2016-07-26 At&T Intellectual Property I, L.P. Methods and systems for customized content services with unified messaging systems
JP5436307B2 (en) * 2010-03-31 2014-03-05 Kddi株式会社 Similar document search device
US8719257B2 (en) 2011-02-16 2014-05-06 Symantec Corporation Methods and systems for automatically generating semantic/concept searches
US9311914B2 (en) * 2012-09-03 2016-04-12 Nice-Systems Ltd Method and apparatus for enhanced phonetic indexing and search
US10795916B2 (en) * 2016-03-15 2020-10-06 Facebook, Inc. Temporal clustering of social networking content
US11443734B2 (en) * 2019-08-26 2022-09-13 Nice Ltd. System and method for combining phonetic and automatic speech recognition search
JPWO2022013994A1 (en) * 2020-07-16 2022-01-20

Citations (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4227176A (en) 1978-04-27 1980-10-07 Dialog Systems, Inc. Continuous speech recognition method
US4736429A (en) 1983-06-07 1988-04-05 Matsushita Electric Industrial Co., Ltd. Apparatus for speech recognition
US4903305A (en) 1986-05-12 1990-02-20 Dragon Systems, Inc. Method for representing word models for use in speech recognition
US4975959A (en) 1983-11-08 1990-12-04 Texas Instruments Incorporated Speaker independent speech recognition process
US4980918A (en) 1985-05-09 1990-12-25 International Business Machines Corporation Speech recognition system with efficient storage and rapid assembly of phonological graphs
US4985924A (en) 1987-12-24 1991-01-15 Kabushiki Kaisha Toshiba Speech recognition apparatus
US5075896A (en) 1989-10-25 1991-12-24 Xerox Corporation Character and phoneme recognition based on probability clustering
US5131043A (en) 1983-09-05 1992-07-14 Matsushita Electric Industrial Co., Ltd. Method of and apparatus for speech recognition wherein decisions are made based on phonemes
US5136655A (en) 1990-03-26 1992-08-04 Hewlett-Pacard Company Method and apparatus for indexing and retrieving audio-video data
US5202952A (en) * 1990-06-22 1993-04-13 Dragon Systems, Inc. Large-vocabulary continuous speech prefiltering and processing system
EP0597798A1 (en) 1992-11-13 1994-05-18 International Business Machines Corporation Method and system for utilizing audible search patterns within a multimedia presentation
US5333275A (en) 1992-06-23 1994-07-26 Wheatley Barbara J System and method for time aligning speech
US5390278A (en) 1991-10-08 1995-02-14 Bell Canada Phoneme based speech recognition
EP0689153A2 (en) 1994-05-30 1995-12-27 Texas Instruments Incorporated Character recognition
US5500920A (en) 1993-09-23 1996-03-19 Xerox Corporation Semantic co-occurrence filtering for speech recognition and signal transcription applications
US5577249A (en) 1992-07-31 1996-11-19 International Business Machines Corporation Method for finding a reference token sequence in an original token string within a database of token strings using appended non-contiguous substrings
GB2302199A (en) 1996-09-24 1997-01-08 Allvoice Computing Plc Text processing
US5594641A (en) 1992-07-20 1997-01-14 Xerox Corporation Finite-state transduction of related word forms for text indexing and retrieval
US5638425A (en) 1992-12-17 1997-06-10 Bell Atlantic Network Services, Inc. Automated directory assistance system using word recognition and phoneme processing method
US5640487A (en) 1993-02-26 1997-06-17 International Business Machines Corporation Building scalable n-gram language models using maximum likelihood maximum entropy n-gram models
US5649060A (en) 1993-10-18 1997-07-15 International Business Machines Corporation Automatic indexing and aligning of audio and text using speech recognition
EP0789349A2 (en) 1996-02-09 1997-08-13 Canon Kabushiki Kaisha Pattern matching method and apparatus and telephone system
US5675706A (en) * 1995-03-31 1997-10-07 Lucent Technologies Inc. Vocabulary independent discriminative utterance verification for non-keyword rejection in subword based speech recognition
US5680605A (en) * 1995-02-07 1997-10-21 Torres; Robert J. Method and apparatus for searching a large volume of data with a pointer-based device in a data processing system
US5684925A (en) 1995-09-08 1997-11-04 Matsushita Electric Industrial Co., Ltd. Speech representation by feature-based word prototypes comprising phoneme targets having reliable high similarity
US5708759A (en) 1996-11-19 1998-01-13 Kemeny; Emanuel S. Speech recognition using phoneme waveform parameters
US5721939A (en) 1995-08-03 1998-02-24 Xerox Corporation Method and apparatus for tokenizing text
US5729741A (en) 1995-04-10 1998-03-17 Golden Enterprises, Inc. System for storage and retrieval of diverse types of information obtained from different media sources which includes video, audio, and text transcriptions
US5737723A (en) 1994-08-29 1998-04-07 Lucent Technologies Inc. Confusable word detection in speech recognition
US5737489A (en) 1995-09-15 1998-04-07 Lucent Technologies Inc. Discriminative utterance verification for connected digits recognition
US5752227A (en) 1994-05-10 1998-05-12 Telia Ab Method and arrangement for speech to text conversion
EP0849723A2 (en) 1996-12-20 1998-06-24 ATR Interpreting Telecommunications Research Laboratories Speech recognition apparatus equipped with means for removing erroneous candidate of speech recognition
US5781884A (en) 1995-03-24 1998-07-14 Lucent Technologies, Inc. Grapheme-to-phoneme conversion of digit strings using weighted finite state transducers to apply grammar to powers of a number basis
US5787414A (en) 1993-06-03 1998-07-28 Kabushiki Kaisha Toshiba Data retrieval system using secondary information of primary data to be retrieved as retrieval key
US5799267A (en) 1994-07-22 1998-08-25 Siegel; Steven H. Phonic engine
WO1998047084A1 (en) 1997-04-17 1998-10-22 Sharp Kabushiki Kaisha A method and system for object-based video description and linking
US5835667A (en) 1994-10-14 1998-11-10 Carnegie Mellon University Method and apparatus for creating a searchable digital video library and a system and method of using such a library
US5852822A (en) * 1996-12-09 1998-12-22 Oracle Corporation Index-only tables with nested group keys
WO1999005681A1 (en) 1997-07-23 1999-02-04 Siemens Aktiengesellschaft Process for storing search parameters of an image sequence and access to an image stream in said image sequence
US5870740A (en) 1996-09-30 1999-02-09 Apple Computer, Inc. System and method for improving the ranking of information retrieval results for short queries
US5873061A (en) 1995-05-03 1999-02-16 U.S. Philips Corporation Method for constructing a model of a new word for addition to a word model database of a speech recognition system
US5907821A (en) 1995-11-06 1999-05-25 Hitachi, Ltd. Method of computer-based automatic extraction of translation pairs of words from a bilingual text
GB2331816A (en) 1997-10-16 1999-06-02 Imarket Inc Searching a database using a phonetically encoded inverted index
US5983177A (en) 1997-12-18 1999-11-09 Nortel Networks Corporation Method and apparatus for obtaining transcriptions from multiple training utterances
US5999902A (en) 1995-03-07 1999-12-07 British Telecommunications Public Limited Company Speech recognition incorporating a priori probability weighting factors
US6023536A (en) 1995-07-03 2000-02-08 Fujitsu Limited Character string correction system and method using error pattern
US6061679A (en) 1997-11-25 2000-05-09 International Business Machines Corporation Creating and searching a data structure ordered by ranges of key masks associated with the data structure
WO2000031723A1 (en) 1998-11-25 2000-06-02 Sony Electronics, Inc. Method and apparatus for very large vocabulary isolated word recognition in a parameter sharing speech recognition system
WO2000054168A2 (en) 1999-03-05 2000-09-14 Canon Kabushiki Kaisha Database annotation and retrieval
GB2349260A (en) 1999-04-23 2000-10-25 Canon Kk Training apparatus
US6172675B1 (en) 1996-12-05 2001-01-09 Interval Research Corporation Indirect manipulation of data using temporally related data, with particular application to manipulation of audio or audiovisual data
US6182039B1 (en) * 1998-03-24 2001-01-30 Matsushita Electric Industrial Co., Ltd. Method and apparatus using probabilistic language model based on confusable sets for speech recognition
US6192337B1 (en) 1998-08-14 2001-02-20 International Business Machines Corporation Apparatus and methods for rejecting confusible words during training associated with a speech recognition system
US6236964B1 (en) 1990-02-01 2001-05-22 Canon Kabushiki Kaisha Speech recognition apparatus and method for matching inputted speech and a word generated from stored referenced phoneme data
US6243680B1 (en) 1998-06-15 2001-06-05 Nortel Networks Limited Method and apparatus for obtaining a transcription of phrases through text and spoken utterances
US6272242B1 (en) 1994-07-15 2001-08-07 Ricoh Company, Ltd. Character recognition method and apparatus which groups similar character patterns
US6314400B1 (en) 1998-09-16 2001-11-06 U.S. Philips Corporation Method of estimating probabilities of occurrence of speech vocabulary elements
US6321226B1 (en) 1998-06-30 2001-11-20 Microsoft Corporation Flexible keyboard searching
US6389395B1 (en) 1994-11-01 2002-05-14 British Telecommunications Public Limited Company System and method for generating a phonetic baseform for a word and using the generated baseform for speech recognition
US6487532B1 (en) * 1997-09-24 2002-11-26 Scansoft, Inc. Apparatus and method for distinguishing similar-sounding utterances speech recognition
US6490563B2 (en) 1998-08-17 2002-12-03 Microsoft Corporation Proofreading with text to speech feedback
US6535850B1 (en) 2000-03-09 2003-03-18 Conexant Systems, Inc. Smart training and smart scoring in SD speech recognition system with user defined vocabulary
US6567816B1 (en) * 2000-03-07 2003-05-20 Paramesh Sampatrai Desai Method, system, and program for extracting data from database records using dynamic code
US6567778B1 (en) 1995-12-21 2003-05-20 Nuance Communications Natural language speech recognition using slot semantic confidence scores related to their word recognition confidence scores
US6662180B1 (en) 1999-05-12 2003-12-09 Matsushita Electric Industrial Co., Ltd. Method for searching in large databases of automatically recognized text

Patent Citations (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4227176A (en) 1978-04-27 1980-10-07 Dialog Systems, Inc. Continuous speech recognition method
US4736429A (en) 1983-06-07 1988-04-05 Matsushita Electric Industrial Co., Ltd. Apparatus for speech recognition
US5131043A (en) 1983-09-05 1992-07-14 Matsushita Electric Industrial Co., Ltd. Method of and apparatus for speech recognition wherein decisions are made based on phonemes
US4975959A (en) 1983-11-08 1990-12-04 Texas Instruments Incorporated Speaker independent speech recognition process
US4980918A (en) 1985-05-09 1990-12-25 International Business Machines Corporation Speech recognition system with efficient storage and rapid assembly of phonological graphs
US4903305A (en) 1986-05-12 1990-02-20 Dragon Systems, Inc. Method for representing word models for use in speech recognition
US4985924A (en) 1987-12-24 1991-01-15 Kabushiki Kaisha Toshiba Speech recognition apparatus
US5075896A (en) 1989-10-25 1991-12-24 Xerox Corporation Character and phoneme recognition based on probability clustering
US6236964B1 (en) 1990-02-01 2001-05-22 Canon Kabushiki Kaisha Speech recognition apparatus and method for matching inputted speech and a word generated from stored referenced phoneme data
US5136655A (en) 1990-03-26 1992-08-04 Hewlett-Pacard Company Method and apparatus for indexing and retrieving audio-video data
US5202952A (en) * 1990-06-22 1993-04-13 Dragon Systems, Inc. Large-vocabulary continuous speech prefiltering and processing system
US5390278A (en) 1991-10-08 1995-02-14 Bell Canada Phoneme based speech recognition
US5333275A (en) 1992-06-23 1994-07-26 Wheatley Barbara J System and method for time aligning speech
US5594641A (en) 1992-07-20 1997-01-14 Xerox Corporation Finite-state transduction of related word forms for text indexing and retrieval
US5577249A (en) 1992-07-31 1996-11-19 International Business Machines Corporation Method for finding a reference token sequence in an original token string within a database of token strings using appended non-contiguous substrings
EP0597798A1 (en) 1992-11-13 1994-05-18 International Business Machines Corporation Method and system for utilizing audible search patterns within a multimedia presentation
US5638425A (en) 1992-12-17 1997-06-10 Bell Atlantic Network Services, Inc. Automated directory assistance system using word recognition and phoneme processing method
US5640487A (en) 1993-02-26 1997-06-17 International Business Machines Corporation Building scalable n-gram language models using maximum likelihood maximum entropy n-gram models
US5787414A (en) 1993-06-03 1998-07-28 Kabushiki Kaisha Toshiba Data retrieval system using secondary information of primary data to be retrieved as retrieval key
US5500920A (en) 1993-09-23 1996-03-19 Xerox Corporation Semantic co-occurrence filtering for speech recognition and signal transcription applications
US5649060A (en) 1993-10-18 1997-07-15 International Business Machines Corporation Automatic indexing and aligning of audio and text using speech recognition
EP0649144B1 (en) 1993-10-18 2000-01-05 International Business Machines Corporation Automatic indexing of audio using speech recognition
US5752227A (en) 1994-05-10 1998-05-12 Telia Ab Method and arrangement for speech to text conversion
EP0689153A2 (en) 1994-05-30 1995-12-27 Texas Instruments Incorporated Character recognition
US6272242B1 (en) 1994-07-15 2001-08-07 Ricoh Company, Ltd. Character recognition method and apparatus which groups similar character patterns
US5799267A (en) 1994-07-22 1998-08-25 Siegel; Steven H. Phonic engine
US5737723A (en) 1994-08-29 1998-04-07 Lucent Technologies Inc. Confusable word detection in speech recognition
US5835667A (en) 1994-10-14 1998-11-10 Carnegie Mellon University Method and apparatus for creating a searchable digital video library and a system and method of using such a library
US6389395B1 (en) 1994-11-01 2002-05-14 British Telecommunications Public Limited Company System and method for generating a phonetic baseform for a word and using the generated baseform for speech recognition
US5680605A (en) * 1995-02-07 1997-10-21 Torres; Robert J. Method and apparatus for searching a large volume of data with a pointer-based device in a data processing system
US5999902A (en) 1995-03-07 1999-12-07 British Telecommunications Public Limited Company Speech recognition incorporating a priori probability weighting factors
US5781884A (en) 1995-03-24 1998-07-14 Lucent Technologies, Inc. Grapheme-to-phoneme conversion of digit strings using weighted finite state transducers to apply grammar to powers of a number basis
US5675706A (en) * 1995-03-31 1997-10-07 Lucent Technologies Inc. Vocabulary independent discriminative utterance verification for non-keyword rejection in subword based speech recognition
US5729741A (en) 1995-04-10 1998-03-17 Golden Enterprises, Inc. System for storage and retrieval of diverse types of information obtained from different media sources which includes video, audio, and text transcriptions
US5873061A (en) 1995-05-03 1999-02-16 U.S. Philips Corporation Method for constructing a model of a new word for addition to a word model database of a speech recognition system
US6023536A (en) 1995-07-03 2000-02-08 Fujitsu Limited Character string correction system and method using error pattern
US5721939A (en) 1995-08-03 1998-02-24 Xerox Corporation Method and apparatus for tokenizing text
US5684925A (en) 1995-09-08 1997-11-04 Matsushita Electric Industrial Co., Ltd. Speech representation by feature-based word prototypes comprising phoneme targets having reliable high similarity
US5737489A (en) 1995-09-15 1998-04-07 Lucent Technologies Inc. Discriminative utterance verification for connected digits recognition
US5907821A (en) 1995-11-06 1999-05-25 Hitachi, Ltd. Method of computer-based automatic extraction of translation pairs of words from a bilingual text
US6567778B1 (en) 1995-12-21 2003-05-20 Nuance Communications Natural language speech recognition using slot semantic confidence scores related to their word recognition confidence scores
EP0789349A2 (en) 1996-02-09 1997-08-13 Canon Kabushiki Kaisha Pattern matching method and apparatus and telephone system
GB2302199A (en) 1996-09-24 1997-01-08 Allvoice Computing Plc Text processing
US5870740A (en) 1996-09-30 1999-02-09 Apple Computer, Inc. System and method for improving the ranking of information retrieval results for short queries
US5708759A (en) 1996-11-19 1998-01-13 Kemeny; Emanuel S. Speech recognition using phoneme waveform parameters
US6172675B1 (en) 1996-12-05 2001-01-09 Interval Research Corporation Indirect manipulation of data using temporally related data, with particular application to manipulation of audio or audiovisual data
US5852822A (en) * 1996-12-09 1998-12-22 Oracle Corporation Index-only tables with nested group keys
EP0849723A2 (en) 1996-12-20 1998-06-24 ATR Interpreting Telecommunications Research Laboratories Speech recognition apparatus equipped with means for removing erroneous candidate of speech recognition
WO1998047084A1 (en) 1997-04-17 1998-10-22 Sharp Kabushiki Kaisha A method and system for object-based video description and linking
WO1999005681A1 (en) 1997-07-23 1999-02-04 Siemens Aktiengesellschaft Process for storing search parameters of an image sequence and access to an image stream in said image sequence
US6487532B1 (en) * 1997-09-24 2002-11-26 Scansoft, Inc. Apparatus and method for distinguishing similar-sounding utterances speech recognition
GB2331816A (en) 1997-10-16 1999-06-02 Imarket Inc Searching a database using a phonetically encoded inverted index
US6026398A (en) 1997-10-16 2000-02-15 Imarket, Incorporated System and methods for searching and matching databases
US6061679A (en) 1997-11-25 2000-05-09 International Business Machines Corporation Creating and searching a data structure ordered by ranges of key masks associated with the data structure
US5983177A (en) 1997-12-18 1999-11-09 Nortel Networks Corporation Method and apparatus for obtaining transcriptions from multiple training utterances
US6182039B1 (en) * 1998-03-24 2001-01-30 Matsushita Electric Industrial Co., Ltd. Method and apparatus using probabilistic language model based on confusable sets for speech recognition
US6243680B1 (en) 1998-06-15 2001-06-05 Nortel Networks Limited Method and apparatus for obtaining a transcription of phrases through text and spoken utterances
US6321226B1 (en) 1998-06-30 2001-11-20 Microsoft Corporation Flexible keyboard searching
US6192337B1 (en) 1998-08-14 2001-02-20 International Business Machines Corporation Apparatus and methods for rejecting confusible words during training associated with a speech recognition system
US6490563B2 (en) 1998-08-17 2002-12-03 Microsoft Corporation Proofreading with text to speech feedback
US6314400B1 (en) 1998-09-16 2001-11-06 U.S. Philips Corporation Method of estimating probabilities of occurrence of speech vocabulary elements
WO2000031723A1 (en) 1998-11-25 2000-06-02 Sony Electronics, Inc. Method and apparatus for very large vocabulary isolated word recognition in a parameter sharing speech recognition system
WO2000054168A2 (en) 1999-03-05 2000-09-14 Canon Kabushiki Kaisha Database annotation and retrieval
GB2349260A (en) 1999-04-23 2000-10-25 Canon Kk Training apparatus
US6662180B1 (en) 1999-05-12 2003-12-09 Matsushita Electric Industrial Co., Ltd. Method for searching in large databases of automatically recognized text
US6567816B1 (en) * 2000-03-07 2003-05-20 Paramesh Sampatrai Desai Method, system, and program for extracting data from database records using dynamic code
US6535850B1 (en) 2000-03-09 2003-03-18 Conexant Systems, Inc. Smart training and smart scoring in SD speech recognition system with user defined vocabulary

Non-Patent Citations (36)

* Cited by examiner, † Cited by third party
Title
Besling S: "A statistical approach to multilingual phonetic transcription," Phillips Journal of Research, NL, Elsevier, Amsterdam, vol. 49, No. 4, 1995, pp. 367-379, XP004000261, ISSN: 0165-5817.
C. Berge, "Graphs & Hypergraphs", 1976, North Holland Mathematical Library, Amsterdam, pg. 175, XP-002192893.
C. Gagnoulet et al., "Mairievox: A Voice-Activated Information System", Speech Communication, 10 (1991), pp. 23-31.
D. Sankoff et al., "Time Warps, String Edits, and Macromolecules: The Theory And Practice of Sequence Comparison", CSLI Publications (1999) ISBN: 1-57586-217-4, pp. 1-44, 211-214, 311-321 and 359-362.
D.A. James, et al., "A Fast Lattice-Based Approach to Vocabulary Independent Wordspotting", 1994 IEEE Int. Conf. On Acoustics, Speech and Sig. Proc., 1994 ICASSP-94, Apr. 19-22, 1994, vol. 1, pp. I/377-I/380.
E. Wold, et al., "Content-Based Classification Search, and Retrieval of Audio", IEEE Multimedia, vol. 3, No. 3, 1996, pp. 27-36.
F. Schiel et al., "The Partitur Format at BAS", In Proc. of the First Int'l. Conference on Language Resources and Evaluation, Granada, Spain, 1998 7 pages.
G. Micca et al: Three Dimensional DP for Phonetic Lattice Matching, International Conference on Digital Signal Processing, Fierence, Italy, 1987, pp. 547-551.
Gerber, C. "A Genral Approach To Speech Recognition", Proceedings of the Final Workshop on Multimedia Information Retrieval (Miro '95), Sep. 18, 1995, pgs. 1-12, Glasgow, Scotland, XP-02232856.
Hsin-Min Wang, "Retrieval of Mandarin Spoken Documents Based On Syllable Lattice Matching", Pattern Recognition Letters, 21, pp. 615-624, Jun. 2000.
IBM, "Template Averaging For Adapting A Dynamic Time Warping Speech", IBM Technical Disclosure Bulletin, IBM Corp., NY, US, vol. 32, No. 11, Apr. 1, 1990, pp. 422-426, XP000097761, ISBN, 0018-8689.
J. Markowitz, "Using Speech Recognition", Prentice Hall, NJ, 1996, pp. 220-221.
J. Skilling, "Classic Maximum Entropy", in J. Skilling (ed.), Maximum Entropy And Bayesian Methods, pp. 45-52 (1989) by Kluwer Academic Publishers.
J. Zobel et al. "Phonetic String Matching, Lessons From Information Retrieval", Sigir Forum, Assn. for Computing Machinery, New York, US, 1996, pp. 166-172, XP00087022.
J.T. Foote et al., "Unconstrained keyword spotting using phone lattices with application to spoken document retrieval"; Computer Speech and Language (1997) 11, pp. 207-224.
Jerry H. Wright et al., "Statistical Models for Topic Identification Using Phoneme Substrings", IEEE, 1996, 0-7803-3192-3/96, pp. 307-310.
Jokinin P et al: "A Comparison of Approximate String Matching Algorithms", Software Practice & Experience, GB, John Wiley & Sons Ltd., Chicester, vol. 26, No. 12, 1 Dec. 1996, pp. 1439-1458, XP000655612, ISSN: 0038-0644.
Kai-Fu Lee, "Automatic Speech Recognition, The Development of the SPHINX System" Kluwer Academic Publishers, pp. 28-29 (1989).
Kenny Ng and Victor W. Zue: "Phonetic Recognition for Spoken Document Retrieval", Spoken Language Systems Group, MIT Laboratory for Computer Science, MA 02139, USA 4 pages, No Date.
Kenny Ng and Victor W. Zue: "Subword Unit Representations for Spoken Document Retrieval", Spoken Language Systems Group, MIT Laboratory for Computer Science, MA 02139, USA No Date 4 pages.
Kenny Ng: "Survey of Approaches to Information Retrieval of Speech Messages", Spoken Language Systems Group, Laboratory for Computer Science, Massachusetts Institute of Technology, Feb. 16, 1996, pp. 1-34.
L.R. Bahl et al., "A Method for the Construction of Acoustic Markov Models for Words", Oct. 1993, IEEE, Tansactions on Speech and Audio Processing, vol. 1, Issue 4, pp. 443-452.
M J Witbrock et al: "Using Words and Phonetic Strings for Efficient Information Retrieval from Imperfectly Transcribed Spoken Documents", School of Computer Sciences, Carnegie Mellon University, XP-002146047 1997 4 pages.
Martin Wechsler et al., "Spoken Document Retrieval Based on Phoneme Recognition", Dissertation submitted to the Swiss Federal Institute of Technology (ETH) Zurich, 1998, DISS.ETH No. 12879, pp. i-x and 1-121.
N. Jain et al., "Creating Speaker-Specific Phonetic Templates With A Speaker-Independent Phonetic Recognizer, Implications For Voice Dialing", 1996 IEEE Int'l. Conf. On Acoustics, Speech & Signal Processing Conf. Proceedings, vol. 2, pp. 881-884.
Okawa et al. "Automatic training of phoneme dictionary based on mutual information criterion", IEEE 1994, pp. 241-244.* *
P. Gelin et al., "Keyword Spotting For Video Soundtrack Indexing", 1996, IEEE Int. Conf. On Acoustics, Speech & Sig. Prc., 1996, ICASSP-96 Conf. Proceedings, vol. 1, pp. 299-302, May 7-10, 1996.
P. Schmid et al., "Automatically Generated Word Pronunciations From Phoneme Classifier Output", Statistical Signal and Array Processing, Proc. of the Int'l Conf. on Acoustics, Speech and Signal Processing (ICASSP), IEEE, vol. 4, Apr. 27, 1993, pp. 223-226.
R Haeb-Umbach et al: "Automatic Transcription of Unknown Words in a Speech Recognition System", 0-7803-2431-5/95, 1995, IEEE, pp. 840-843.
Rahim "A neural tree network for phoneme classification with experiments on the TIMIT database", IEEE 1992, pp. 345-348.* *
S. Srinivasan et al., "Phonetic Confusion Matrix Based Spoken Document Retrieval", Proc. of the 23rd Annual Int'l. ACM SIGIR Conf on Research & Development in Information Retrival Jul. 24-28, 2000, pp. 81-87.
Srinivasan et al. "Phonetic confusion matrix based spoken document retrieval", ACM 2000, pp. 81-87.* *
Steve Cassidy et al., "EMU: An Enhanced Hierarchical Speech Data Management System", Proc. of the 6th Australian Speech Science & Technology Conf., Adelaide, 1996, pp. 381-386.
Steven Bird, et al., "Towards A Formal Framework For Linguistic Annotations", Linguistic Data Consortium, Univ. of PA., Dec., 1998.
Steven Bird, et al., A Formal Framework for Linguistic Annotation, Technical Report, MS-CIS-99-01, Dept. of Computer & Information Science, Mar., 1999, Linguistic Data Consortium, University of PA, pp. 1-48.
Yutaka Kobayashi et al: "Matching Algorithms between a Phonetic Lattice and Two Types of Templates-Lattice and Graph", Dept. of Computer Science, Kyoto Institute of Technology, IEEE, 1985, CH2118-/8/85/000-1597, pp. 1597-1600.

Cited By (122)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7310600B1 (en) * 1999-10-28 2007-12-18 Canon Kabushiki Kaisha Language recognition using a similarity measure
US20090083033A1 (en) * 2000-03-04 2009-03-26 Georgia Tech Research Corporation Phonetic Searching
US7475065B1 (en) 2000-03-04 2009-01-06 Georgia Tech Research Corporation Phonetic searching
US7406415B1 (en) 2000-03-04 2008-07-29 Georgia Tech Research Corporation Phonetic searching
US7769587B2 (en) 2000-03-04 2010-08-03 Georgia Tech Research Corporation Phonetic searching
US7313521B1 (en) 2000-03-04 2007-12-25 Georgia Tech Research Corporation Phonetic searching
US20030204399A1 (en) * 2002-04-25 2003-10-30 Wolf Peter P. Key word and key phrase based speech recognizer for information retrieval systems
US20070174262A1 (en) * 2003-05-15 2007-07-26 Morten Middelfart Presentation of data using meta-morphing
US7779018B2 (en) * 2003-05-15 2010-08-17 Targit A/S Presentation of data using meta-morphing
US7783628B2 (en) 2003-05-15 2010-08-24 Targit A/S Method and user interface for making a presentation of data using meta-morphing
US20070033003A1 (en) * 2003-07-23 2007-02-08 Nexidia Inc. Spoken word spotting queries
US7904296B2 (en) * 2003-07-23 2011-03-08 Nexidia Inc. Spoken word spotting queries
US20060282408A1 (en) * 2003-09-30 2006-12-14 Wisely David R Search system and method via proxy server
US8468444B2 (en) 2004-03-17 2013-06-18 Targit A/S Hyper related OLAP
US20050210389A1 (en) * 2004-03-17 2005-09-22 Targit A/S Hyper related OLAP
US20060106843A1 (en) * 2004-11-17 2006-05-18 Targit A/S Database track history
US7774295B2 (en) 2004-11-17 2010-08-10 Targit A/S Database track history
US8082145B2 (en) 2004-11-24 2011-12-20 Microsoft Corporation Character manipulation
US7778821B2 (en) 2004-11-24 2010-08-17 Microsoft Corporation Controlled manipulation of characters
US20100265257A1 (en) * 2004-11-24 2010-10-21 Microsoft Corporation Character manipulation
US20060111890A1 (en) * 2004-11-24 2006-05-25 Microsoft Corporation Controlled manipulation of characters
US8694317B2 (en) * 2005-02-05 2014-04-08 Aurix Limited Methods and apparatus relating to searching of spoken audio data
US20060206324A1 (en) * 2005-02-05 2006-09-14 Aurix Limited Methods and apparatus relating to searching of spoken audio data
US8682913B1 (en) 2005-03-31 2014-03-25 Google Inc. Corroborating facts extracted from multiple sources
US8650175B2 (en) 2005-03-31 2014-02-11 Google Inc. User interface for facts query engine with snippets from information sources that include query terms and answer terms
US20070143282A1 (en) * 2005-03-31 2007-06-21 Betz Jonathan T Anchor text summarization for corroboration
US9208229B2 (en) * 2005-03-31 2015-12-08 Google Inc. Anchor text summarization for corroboration
US20060265222A1 (en) * 2005-05-20 2006-11-23 Microsoft Corporation Method and apparatus for indexing speech
US7634407B2 (en) 2005-05-20 2009-12-15 Microsoft Corporation Method and apparatus for indexing speech
US8825471B2 (en) 2005-05-31 2014-09-02 Google Inc. Unsupervised extraction of facts
US9558186B2 (en) 2005-05-31 2017-01-31 Google Inc. Unsupervised extraction of facts
US8996470B1 (en) 2005-05-31 2015-03-31 Google Inc. System for ensuring the internal consistency of a fact repository
US8719260B2 (en) 2005-05-31 2014-05-06 Google Inc. Identifying the unifying subject of a set of facts
US20070150800A1 (en) * 2005-05-31 2007-06-28 Betz Jonathan T Unsupervised extraction of facts
US20070106509A1 (en) * 2005-11-08 2007-05-10 Microsoft Corporation Indexing and searching speech with text meta-data
US7809568B2 (en) * 2005-11-08 2010-10-05 Microsoft Corporation Indexing and searching speech with text meta-data
US7801910B2 (en) * 2005-11-09 2010-09-21 Ramp Holdings, Inc. Method and apparatus for timed tagging of media content
US7831428B2 (en) * 2005-11-09 2010-11-09 Microsoft Corporation Speech index pruning
US20090222442A1 (en) * 2005-11-09 2009-09-03 Henry Houh User-directed navigation of multimedia search results
US20070106685A1 (en) * 2005-11-09 2007-05-10 Podzinger Corp. Method and apparatus for updating speech recognition databases and reindexing audio and video content using the same
CN101305362B (en) * 2005-11-09 2012-04-18 微软公司 Speech index pruning
US20070106760A1 (en) * 2005-11-09 2007-05-10 Bbnt Solutions Llc Methods and apparatus for dynamic presentation of advertising, factual, and informational content using enhanced metadata in search-driven media applications
US9697231B2 (en) 2005-11-09 2017-07-04 Cxense Asa Methods and apparatus for providing virtual media channels based on media search
US9697230B2 (en) 2005-11-09 2017-07-04 Cxense Asa Methods and apparatus for dynamic presentation of advertising, factual, and informational content using enhanced metadata in search-driven media applications
US20070106512A1 (en) * 2005-11-09 2007-05-10 Microsoft Corporation Speech index pruning
US20070106693A1 (en) * 2005-11-09 2007-05-10 Bbnt Solutions Llc Methods and apparatus for providing virtual media channels based on media search
US20070112837A1 (en) * 2005-11-09 2007-05-17 Bbnt Solutions Llc Method and apparatus for timed tagging of media content
US20070118873A1 (en) * 2005-11-09 2007-05-24 Bbnt Solutions Llc Methods and apparatus for merging media content
US20070143110A1 (en) * 2005-12-15 2007-06-21 Microsoft Corporation Time-anchored posterior indexing of speech
US7831425B2 (en) 2005-12-15 2010-11-09 Microsoft Corporation Time-anchored posterior indexing of speech
US7911482B1 (en) 2006-01-06 2011-03-22 Videomining Corporation Method and system for efficient annotation of object trajectories in image sequences
US9245526B2 (en) * 2006-04-25 2016-01-26 General Motors Llc Dynamic clustering of nametags in an automated speech recognition system
US20070250320A1 (en) * 2006-04-25 2007-10-25 General Motors Corporation Dynamic clustering of nametags in an automated speech recognition system
WO2007134293A2 (en) * 2006-05-12 2007-11-22 Nexidia, Inc. Wordspotting system
US7640161B2 (en) * 2006-05-12 2009-12-29 Nexidia Inc. Wordspotting system
US20070271241A1 (en) * 2006-05-12 2007-11-22 Morris Robert W Wordspotting system
WO2007134293A3 (en) * 2006-05-12 2009-04-09 Nexidia Inc Wordspotting system
US20090187845A1 (en) * 2006-05-16 2009-07-23 Targit A/S Method of preparing an intelligent dashboard for data monitoring
US7949674B2 (en) 2006-07-17 2011-05-24 Targit A/S Integration of documents with OLAP using search
US20080019620A1 (en) * 2006-07-21 2008-01-24 Gerald Vierheilig Lubricating unit
US8694318B2 (en) * 2006-09-19 2014-04-08 At&T Intellectual Property I, L. P. Methods, systems, and products for indexing content
US20080071542A1 (en) * 2006-09-19 2008-03-20 Ke Yu Methods, systems, and products for indexing content
US8751498B2 (en) 2006-10-20 2014-06-10 Google Inc. Finding and disambiguating references to entities on web pages
US9760570B2 (en) 2006-10-20 2017-09-12 Google Inc. Finding and disambiguating references to entities on web pages
US8489553B2 (en) 2006-11-01 2013-07-16 Ab Initio Technology Llc Managing storage of individually accessible data units
US20110196844A1 (en) * 2006-11-01 2011-08-11 Ab Initio Technology Llc Managing storage of individually accessible data units
US8639674B2 (en) 2006-11-01 2014-01-28 Ab Initio Technology Llc Managing storage of individually accessible data units
US20080104149A1 (en) * 2006-11-01 2008-05-01 Ephraim Meriwether Vishniac Managing Storage of Individually Accessible Data Units
US7885932B2 (en) 2006-11-01 2011-02-08 Ab Initio Technology Llc Managing storage of individually accessible data units
US8214331B2 (en) 2006-11-01 2012-07-03 Ab Initio Technology Llc Managing storage of individually accessible data units
US8949189B2 (en) 2006-11-01 2015-02-03 Ab Initio Technology Llc Managing storage of individually accessible data units
US8229902B2 (en) 2006-11-01 2012-07-24 Ab Initio Technology Llc Managing storage of individually accessible data units
US20100169274A1 (en) * 2006-11-01 2010-07-01 Ab Initio Technology Llc Managing storage of individually accessible data units
US20080162125A1 (en) * 2006-12-28 2008-07-03 Motorola, Inc. Method and apparatus for language independent voice indexing and searching
US20080162472A1 (en) * 2006-12-28 2008-07-03 Motorola, Inc. Method and apparatus for voice searching in a mobile communication device
US8347202B1 (en) 2007-03-14 2013-01-01 Google Inc. Determining geographic locations for place names in a fact repository
US9892132B2 (en) 2007-03-14 2018-02-13 Google Llc Determining geographic locations for place names in a fact repository
US7983915B2 (en) * 2007-04-30 2011-07-19 Sonic Foundry, Inc. Audio content search engine
US20080301539A1 (en) * 2007-04-30 2008-12-04 Targit A/S Computer-implemented method and a computer system and a computer readable medium for creating videos, podcasts or slide presentations from a business intelligence application
US20080270138A1 (en) * 2007-04-30 2008-10-30 Knight Michael J Audio content search engine
US20090030894A1 (en) * 2007-07-23 2009-01-29 International Business Machines Corporation Spoken Document Retrieval using Multiple Speech Transcription Indices
US9405823B2 (en) * 2007-07-23 2016-08-02 Nuance Communications, Inc. Spoken document retrieval using multiple speech transcription indices
US8831946B2 (en) 2007-07-23 2014-09-09 Nuance Communications, Inc. Method and system of indexing speech data
US8589165B1 (en) * 2007-09-20 2013-11-19 United Services Automobile Association (Usaa) Free text matching system and method
US8812435B1 (en) 2007-11-16 2014-08-19 Google Inc. Learning objects and facts from documents
US8229921B2 (en) * 2008-02-25 2012-07-24 Mitsubishi Electric Research Laboratories, Inc. Method for indexing for retrieving documents using particles
US20090216740A1 (en) * 2008-02-25 2009-08-27 Bhiksha Ramakrishnan Method for Indexing for Retrieving Documents Using Particles
US8312022B2 (en) 2008-03-21 2012-11-13 Ramp Holdings, Inc. Search engine optimization
US20090288118A1 (en) * 2008-05-14 2009-11-19 At&T Intellectual Property, Lp Methods and Apparatus to Generate Relevance Rankings for Use by a Program Selector of a Media Presentation System
US20090287486A1 (en) * 2008-05-14 2009-11-19 At&T Intellectual Property, Lp Methods and Apparatus to Generate a Speech Recognition Library
US9277287B2 (en) 2008-05-14 2016-03-01 At&T Intellectual Property I, L.P. Methods and apparatus to generate relevance rankings for use by a program selector of a media presentation system
US9497511B2 (en) 2008-05-14 2016-11-15 At&T Intellectual Property I, L.P. Methods and apparatus to generate relevance rankings for use by a program selector of a media presentation system
US9077933B2 (en) 2008-05-14 2015-07-07 At&T Intellectual Property I, L.P. Methods and apparatus to generate relevance rankings for use by a program selector of a media presentation system
US9202460B2 (en) 2008-05-14 2015-12-01 At&T Intellectual Property I, Lp Methods and apparatus to generate a speech recognition library
US20100153366A1 (en) * 2008-12-15 2010-06-17 Motorola, Inc. Assigning an indexing weight to a search term
US20100179811A1 (en) * 2009-01-13 2010-07-15 Crim Identifying keyword occurrences in audio data
US8423363B2 (en) * 2009-01-13 2013-04-16 CRIM (Centre de Recherche Informatique de Montréal) Identifying keyword occurrences in audio data
US8412525B2 (en) * 2009-04-30 2013-04-02 Microsoft Corporation Noise robust speech classifier ensemble
US20100280827A1 (en) * 2009-04-30 2010-11-04 Microsoft Corporation Noise robust speech classifier ensemble
US8321218B2 (en) 2009-06-19 2012-11-27 L.N.T.S. Linguistech Solutions Ltd Searching in audio speech
US20100324900A1 (en) * 2009-06-19 2010-12-23 Ronen Faifkov Searching in Audio Speech
US20110106792A1 (en) * 2009-11-05 2011-05-05 I2 Limited System and method for word matching and indexing
US20110218802A1 (en) * 2010-03-08 2011-09-08 Shlomi Hai Bouganim Continuous Speech Recognition
EP3550441A1 (en) * 2010-03-10 2019-10-09 AB Initio Technology LLC Managing storage of individually accessible data units
WO2011112187A1 (en) * 2010-03-10 2011-09-15 Ab Initio Technology Llc Managing storage of individually accessible data units
US20110224982A1 (en) * 2010-03-12 2011-09-15 c/o Microsoft Corporation Automatic speech recognition based upon information retrieval methods
US9934166B2 (en) 2010-12-10 2018-04-03 Daniel Shawcross Wilkerson Hard object: constraining control flow and providing lightweight kernel crossings
US9811570B2 (en) 2011-07-08 2017-11-07 Ab Initio Technology Llc Managing storage of data for range-based searching
US20130179430A1 (en) * 2012-01-09 2013-07-11 Wajam Internet Technologies Inc. Proxy server for application on mobile device
US8620898B2 (en) * 2012-01-09 2013-12-31 Wajam Internet Technologies Inc. Proxy server for application on mobile device
US20170126738A1 (en) * 2013-03-14 2017-05-04 Daniel Shawcross Wilkerson Hard Object: Lightweight Hardware Enforcement of Encapsulation, Unforgeability, and Transactionality
US9935975B2 (en) * 2013-03-14 2018-04-03 Daniel Shawcross Wilkerson Hard object: lightweight hardware enforcement of encapsulation, unforgeability, and transactionality
US9508345B1 (en) 2013-09-24 2016-11-29 Knowles Electronics, Llc Continuous voice sensing
US9437187B2 (en) * 2014-03-05 2016-09-06 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US20150255059A1 (en) * 2014-03-05 2015-09-10 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US9431007B2 (en) * 2014-03-05 2016-08-30 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US20150255060A1 (en) * 2014-03-05 2015-09-10 Casio Computer Co., Ltd. Voice search device, voice search method, and non-transitory recording medium
US11392631B2 (en) * 2014-07-29 2022-07-19 Groupon, Inc. System and method for programmatic generation of attribute descriptors
US20190051296A1 (en) * 2017-08-09 2019-02-14 Lenovo (Singapore) Pte. Ltd. Performing action on active media content
US10964316B2 (en) * 2017-08-09 2021-03-30 Lenovo (Singapore) Pte. Ltd. Performing action on active media content
US20200134010A1 (en) * 2018-10-26 2020-04-30 International Business Machines Corporation Correction of misspellings in qa system
US10803242B2 (en) * 2018-10-26 2020-10-13 International Business Machines Corporation Correction of misspellings in QA system

Also Published As

Publication number Publication date
GB0015233D0 (en) 2000-08-16
JP2002063199A (en) 2002-02-28
US20020052870A1 (en) 2002-05-02
EP1168199A3 (en) 2006-04-19
EP1168199A2 (en) 2002-01-02

Similar Documents

Publication Publication Date Title
US6873993B2 (en) Indexing method and apparatus
US7310600B1 (en) Language recognition using a similarity measure
US7212968B1 (en) Pattern matching method and apparatus
US6882970B1 (en) Language recognition using sequence frequency
US7054812B2 (en) Database annotation and retrieval
US7590605B2 (en) Lattice matching
EP1949260B1 (en) Speech index pruning
US7257533B2 (en) Database searching and retrieval using phoneme and word lattice
US7542966B2 (en) Method and system for retrieving documents with spoken queries
US7831425B2 (en) Time-anchored posterior indexing of speech
KR101255405B1 (en) Indexing and searching speech with text meta-data
US7634407B2 (en) Method and apparatus for indexing speech
JP5440177B2 (en) Word category estimation device, word category estimation method, speech recognition device, speech recognition method, program, and recording medium
US20070179784A1 (en) Dynamic match lattice spotting for indexing speech content
Bulyko et al. Subword speech recognition for detection of unseen words.
GB2355836A (en) Pattern matching
Seide et al. Word-lattice based spoken-document indexing with standard text indexers
GB2355837A (en) Pattern matching
GB2355835A (en) Pattern matching
KR20020052196A (en) Pattern matching method and apparatus

Legal Events

Date Code Title Description
AS Assignment

Owner name: CANON KABUSHIKI KAISHA, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHARLESWORTH, JASON PETER ANDREW;GARNER, PHILIP NEIL;REEL/FRAME:012324/0787

Effective date: 20011108

CC Certificate of correction
FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20170329