WO2002019147A1 - Method and apparatus for digital media management, retrieval, and collaboration - Google Patents

Method and apparatus for digital media management, retrieval, and collaboration Download PDF

Info

Publication number
WO2002019147A1
WO2002019147A1 PCT/US2001/026841 US0126841W WO0219147A1 WO 2002019147 A1 WO2002019147 A1 WO 2002019147A1 US 0126841 W US0126841 W US 0126841W WO 0219147 A1 WO0219147 A1 WO 0219147A1
Authority
WO
WIPO (PCT)
Prior art keywords
digital media
search
user
selected digital
receiving
Prior art date
Application number
PCT/US2001/026841
Other languages
French (fr)
Inventor
Sharon Flank
Ruth Sperer
David Ian Forbes
Ed Klein
Randy St. Jean
Donna Romer
James Rothey
Robert Griffin
Keith Simonsen
Gerald Ehlers
Original Assignee
Emotion, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Emotion, Inc. filed Critical Emotion, Inc.
Priority to AU2001288469A priority Critical patent/AU2001288469A1/en
Publication of WO2002019147A1 publication Critical patent/WO2002019147A1/en
Priority to US10/063,411 priority patent/US6941294B2/en
Priority to US10/063,410 priority patent/US7177879B2/en
Priority to US10/063,412 priority patent/US6895407B2/en
Priority to US10/063,409 priority patent/US6922691B2/en
Priority to US10/063,413 priority patent/US6944611B2/en
Priority to US10/063,414 priority patent/US7418444B2/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5846Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99932Access augmentation or optimizing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99934Query formulation, input preparation, or translation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99935Query augmenting and refining, e.g. inexact access
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99944Object-oriented database structure
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99944Object-oriented database structure
    • Y10S707/99945Object-oriented database structure processing
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99948Application of database or data structure, e.g. distributed, multimedia, or image

Definitions

  • thumbnails are browseables, or small representations of the actual images, videos, or other media files in the system.
  • a browseable is created by optimizing an image or video frame for online browsing, so a browseable has lower resolution and smaller dimensions than the original file. It is commonplace, however, to find that the resolution and dimensions are not well suited to the company. It would be desirable to have a management system that conveniently permits a system administrator to customize the software in this respect.
  • NLP Natural language processing
  • Clustering is well known, for example in U.S. Pat. No. 5,317,507, 5,758,257, 5,675,819, 5,778,362, and 5,875,446. See also Buckley, Chris, J. Walz, M. Mitra and C. Cardie, "Using Clustering and Super Concepts within SMART: TREC 6" (http://trec.nist.gov/pubs/trec6/t6_piOceedings.html); Zamir, Oren, O. Etzioni, Madani, and
  • the evaluation of information retrieval systems became an essential part of the field in the early '90s, and was strongly advanced by the TREC evaluations designed at NIST beginning in 1993.
  • the TREC evaluation contains different tracks, but the tracks all share the following common features:
  • the evaluation is done using strict test conditions that contain a set of queries, a collection of documents, and relevance judgements.
  • the evaluations use evaluation scores such as precision and recall that supposedly predict real users' satisfaction from a system
  • the software according to the invention incorporates a glossary management tool that makes it easy for each client to customize terminology to the needs of a particular business.
  • a glossary manager a company can customize a number of feature names in the system to provide a more familiar context for their users.
  • a system administrator can also customize the manner in which "thumbnail" or "preview” images are presented.
  • the system performs clustering on search queries, and searches media records multi-moda-ly, using two or more approaches such as image searching and text searching.
  • An administrator can tune search parameters.
  • Two or more streams of metadata may be aligned and correlated with a media file, facilitating later searching.
  • the system evaluates itself. It folds popularity information into rankings of search results.
  • Fig. 1 shows a typical rights management editor interaction
  • Fig. 2 shows a typical customization interaction screen
  • Fig. 3 shows a screen permitting customers to adjust search parameters
  • Fig. 4 shows a sample of a face-matching system according to the invention
  • Fig. 5 illustrates a viewer according to the invention with text annotation box.
  • the inventive software manages media files, or assets, making it possible to store and retrieve them, to collaborate with colleagues to edit and work with them, and to deliver them, either for further processing or in finished form for use.
  • the workflow of the inventive software is designed to provide flexible choices among features that support a creative user who works with rich media files.
  • a user logs on to the software (an optional step)
  • she can immediately see if any system-internal messages are waiting in her system email inbox, either sent from other users collaborating with her on media files, or sent automatically by the inventive system to notify her of various occurrences that may pertain to her work, such as a project update.
  • the user can read her messages, reply to a message, or compose a new message to send to another system user.
  • a composed message may consist of text only, or it may have a system-internal file attached.
  • the user may wish to work on a project, such as an advertising campaign. She enters the project work area, and creates a new project. This creates a virtual workspace in the inventive system that is dedicated to holding media files and all their related files, including edited versions, colleague comments, official approvals, and so on. She decides which users will be allowed to share access to the project space. She may upload files that she has already created into the project work area. From there, she can send a media file off to a colleague for comments, editing, or approval.
  • a project such as an advertising campaign. She enters the project work area, and creates a new project. This creates a virtual workspace in the inventive system that is dedicated to holding media files and all their related files, including edited versions, colleague comments, official approvals, and so on. She decides which users will be allowed to share access to the project space. She may upload files that she has already created into the project work area. From there, she can send a media file off to a colleague for comments, editing, or approval.
  • she can search for the file in the file storage area of the inventive syste She enters search terms, generally either by typing or by speaking into an input device, and finds an appropriate media file. She then saves the file to her new project. She may decide, instead, to have the full, high-resolution version of that file sent to her for a different project.
  • delivery options for high-resolution files include on-line delivery via ftp, http, or special network; via CD delivery, or even via mail delivery of the file in analog format. Ordering and file delivery in the inventive system are subject to several conditions, which the software verifies and tracks.
  • Rights management If a user determines that she needs to use a rights-restricted media file, she must engage in a dialogue with the inventive software about her planned use of the file. Companies can customize sets of questions that the software presents to users, depending on which file a user wants to order. The user may have to answer whether she intends to publish the file, where it will be published, to what size audience, at what location, on what date. The answers deteimine whether the file can be made available, and can also determine the price. For example, if she is showing ten seconds of a movie for a one-time training seminar in Canada for an audience of fifty people, her cost may be far less than making those same ten seconds of movie available on the home page of a major entertainment Web site. And making the entire movie available on the Web may perhaps not be permitted at any price.
  • the rights management process controls every aspect of the order.
  • a company may allow its employees to use its media files for personal use, marketing use, or in an internal organization project. Or they may allow all three uses in different situations. They can track the user's use of the files, they can designate the licensing type for the files and make sure a rights agreement is included, and they can enforce legal or company policy restrictions that may apply.
  • the rights management process helps clients establish all of these elements so they can apply them appropriately to their stored media files.
  • the invention described here makes it easier for individual clients to customize a rights management setup so that the software will make the right pricing and availability decisions for each projected use.
  • a question-answering paradigm an administrator or a manager decides what questions should be presented to each potential user of a file, and what set of multiple choice answers to supply.
  • the software maps the questions and answers to a rights and usage package, and, if applicable, each package is assigned pricing.
  • an administrator may determine that a file's availability depends on:
  • the questions are grouped appropriately for a company's particular file collections, and are assigned to either groups of files or individual files. Questions and their answers may serve as fundamental criteria for whether a file is made available; they may serve as modifiers to an available file by raising or lowering pricing depending on usage; or they may simply be informational questions that do not play a role in decision-making, but allow companies to track file use and user needs.
  • the rights management process When ordering files, users answer questions that ask for information about how they are planning to use the files they order. The questions a user answers are the usage questions a company designates for each possible use. This set of usage questions is called a rights package. Creating a rights package is the first step in the rights management process.
  • a rights package, or set of questions, is added to other rights packages, to make up a rights package group. This allows a company to make more than one rights package available for a group of files.
  • a rights package group is added to a particular license type. The license type describes the possible ways a file can legally be used.
  • a rights agreement is added to define details clearly about the user's rights. In the final step, the rights agreement is associated with a particular license type through a control table. The control table also contains pricing and approval information for each rights package.
  • the core piece of this process is the creation and grouping of questions that cover a company's business and legal interests for their stored media file usage. Part of this process is completed with what may be termed a rights package question role editor. A typical editor interaction is shown in Fig. 1. Glossary Customization Tool and Usability Customization Tool.
  • the software according to the invention incorporates a glossary management tool that makes it easy for each client to customize te ⁇ riinology to the needs of a particular business. With this tool, termed a glossary manager, a company can customize a number of feature names in the system to provide a more fa-xiiliar context for their users.
  • the system also includes a usability management tool that customizes the software interface "look and feel" to more closely reflect a company's corporate culture, image, and to reflect the system context within the company.
  • a part of the system termed a usability manager allows a company to change system background colors, fonts, and the embedded logo. This tool also makes localization into a different language simpler.
  • a typical customization interaction screen is shown in Fig. 2.
  • User-configurable browseable generation In a typical asset management system, users browse through media file collections and view thumbnail images of files to decide which files they want to work with. These thumbnails are browseables, or small representations of the actual images, videos, or other media files in the system A browseable is created by optimizing an image or video frame for online browsing, so a browseable has lower resolution and smaller dimensions than the original file.
  • the browseable (the proxy of the actual media file) is what the user sees in the system according to the invention, and browseable size is generally set by a system administrator for use throughout the system.
  • the system permits generating these lower-resolution proxy files in a way that can be configured by the customer.
  • Natural language processing techniques In the system according to the invention, the natural language processing component is layered.
  • the system architecture supports a conventional search algorithm, which operates as a plug-and-play support application.
  • the algorithm is then used in a word expansion process.
  • the word expansion is applied to both a user's search query, and to the stored files' captions, that is, descriptive metadata identifying individual files.
  • the query undergoes several types of NLP processing (detailed below), and the result of each element in the processing adds new query components (for example, synonyms) and/or weights (which emphasize or de-emphasize words depending on their value to the query).
  • Tl e resulting, or processed, query contains expanded terms and weighting information that can be passed to any search engine. This makes it possible to use multi-search agent (or "federated search") applications as well.
  • a user search query Once a user query is received, it is divided into individual tokens, which may consist of single words or multiwords.
  • Multiwords are words the search engine recognizes as one unit, instead of as separate words. With this feature, when a user searches for sea lions, the system does not bring up lions by the ocean.)
  • a variation of conventional pattern matching is used. If a single word is recognized as matching a word that is part of a stored multiword, the processor decides whether to treat the single word as part of a multiword, based on the contents of the stored pattern and the input pattern.
  • Stored patterns include not just literal words, but also syntactic categories (such as adjective or non- verb), semantic categories (such as nationality or government entity), or exact matches. If the input matches the stored pattern information, then it is interpreted as a multiword rather than as independent words.
  • Processing determines whether the resulting word is a function word (closed-class) or content word (open-class).
  • the processor ignores function words.
  • For content words the related concepts for each sense of the word are retrieved from the semantic net. If the root word is unknown, the word is treated as a keyword, requiring an exact match.
  • the system then matches the expanded, weighted search query with stored media files in order to return search results to the user.
  • Media files are identified by the metadata associated with them, e.g. in a caption.
  • Text-based files can be identified by metadata associated with them, and also by their content.
  • the software provides a workflow to add captions to each file.
  • the files are then uploaded into the database, and the file captions are expanded and the information stored for later file searching.
  • the processor looks up each word in tlie caption or text. Words that may be related in the semantic net are located based on stored links, and the looked-up word, along with any related words, are all displayed as the "expansion" of that word.
  • the processor detei ines whether the current word or phrase corresponds to a proper name, a location, or something else. If it corresponds to a name, a name expansion process is invoked that displays the name and related names such as nicknames and other variants, based on a linked name file. If the current word or phrase corresponds to a location, a location expansion process is invoked that accesses a gazetteer and displays tlie location and related locations based on linked location information found in tlie gazetteer and supporting files. If the current word or phrase is neither a name nor a location, it is expanded using the semantic net links and weights associated with those links. Strongly related concepts are given high weights, while more remotely related concepts receive lower weights, making them less exact matches. Thus, for a query on car, documents or metadata containing car and automobile are listed highest, followed by those with sedan, coupe, and convertible, and then by more remotely related concepts such as transmission, hood, and trunk.
  • the expanded word or phrase is stored in an index database, where it is available for use in searching as described below. Processing then returns to expand the next word or phrase in the text or caption.
  • the system according to the invention handles certain words and phrases individually. Multiwords are matched as a whole unit, and names and locations are identified and looked up in the separate name and location files. Next, noun phrases and other syntactic units are identified.
  • a part-of-speech tagger uses linguistic and statistical information to tag the parts of speech for sections of the user query. Only words that match by part of speech are considered to match, and if two or more parts of speech are possible for a particular word, it is tagged with both. After tagging, word affixes (i.e. suffixes) are stripped from query words to obtain a word root, using conventional inflectional morphology. If a word in a query is not known, affixes are stripped from the word one by one until a known word is found.
  • An intermediate query is then formulated to match against the file index database. Texts or captions that match queries are then returned, ranked, and displayed to the user, with those that match best being displayed at the top of the list.
  • the searching is implemented by first building a B-tree of ID lists, one for each concept in the text database.
  • the ID lists have an entry for each object whose text contains a reference to a given concept.
  • An entry consists of an object ID and a weight.
  • the object ID provides a unique identifier and is a positive integer assigned when the object is indexed.
  • the weight reflects the relevance of the concept to the object's text, and is also a positive integer.
  • the object ID and a weight are inserted into the ID list of every concept that is in any way relevant to the text.
  • the ID lists of every concept in the query are retrieved and combined as specified by the query. Since ID lists contain IDs with weights in sorted order, determining existence and relevance of a match is simultaneous and fast, using only a small number of processor instructions for each concept-object pair.
  • Search Technologies The system allows users to search for media files with many different types of search queries. For example, users may submit search queries by speaking them, typing them, copying them, or drawing the search queries.
  • Files are characterized in several ways. First, they have an identifier, generally similar to a filename, which is unique within the system and makes it possible to link up all the objects related to a file. These can include the actual high-resolution asset, lower-resolution thumbnails or other proxies for browsing, and information about the file, or metadata. Searching can be performed on the file identifier, or it can be performed on the metadata. In the case of metadata searching, it is desirable to offer search alternatives that go beyond the exact matching process involved in a standard keyword search.
  • Some systems use controlled vocabulary searching as an optimization of keyword searching.
  • Keyword searches simply match exactly on any word in the user's search query that appears in the search target.
  • the search target is the metadata describing a media file.
  • the set of potential keywords is quite large (as large as the vocabulary of English, or whatever language(s) are being used). If there are no limitations on the search vocabulary that can be employed, a user can enter a search for puma and fail to find any files captioned as mountain lion or cougar, even though they all refer to the same thing. Controlled vocabulary is an attempt to address this problem, albeit at considerable cost. In a controlled vocabulary retrieval system, cataloguers all agree to use the same terms.
  • system software incorporates additional advanced technology for locating stored files. Rather than limiting se--rching to a controlled vocabulary, the system software includes natural language search, which allows cataloguers and users to employ any words in English (or whatever natural language the retrieval system is using).
  • a semantic network of concepts additional linguistic techniques, including: phrase matching derivational morphology, in lieu of ste-ximing part of speech tagging name recognition location recognition User-Tunable Search Parameters.
  • the system according to the invention provides a screen for customers to adjust search parameters, to reflect their company use of stored media file collections. This is shown in Fig. 3. While the parameters may themselves be well-known in a searching system, what is emphasized here is that the user (or, more likely, an administrator) can be granted access to such fundamental decisions about search as:
  • system according to the invention can be carried out on an internet, meaning an IP-based network, and in particular may be carried out on the Internet, meaning the global IP-based network.
  • Multimodal Search Currently, search methods focus on textual input.
  • the current invention incorporates new search techniques, and combines them in novel ways.
  • Image search is becoming useful in commercial applications.
  • user search input is provided in a new way. Users may wish to select an existing image as example input, so that a search consists of "Give me more images like this.” Perhaps even more useful is the ability to select part of an image, analogous with "Give me more like this part.”
  • identifying the part may be done in either of two exemplary ways:
  • Touch screen user touches the screen to identify the portion of the image that feeds into the search.
  • search modalities can be combined. This novel approach to search is particularly applicable to multimedia. Examples of combined, or multimodal, searches, include:
  • Vocabulary Management Ideally, the semantic net of concepts is quite large and attempts to incorporate every word or term in English (or other language being used for cataloguing and searching). No matter how large the semantic net may be, there will be a periodic need to expand or edit it. New words appear in English periodically, and, although many may be slang and therefore not particularly important in a business context, some will be real new words and will be important enough to include. For example, rollerblading and in-line skating are relatively new terms in English, and depicting those actions is useful in advertising. So the terms need to be added to the semantic net. Semantic net/vocabulary maintenance is generally a manual process, particularly where the user has an existing media library with a thesaurus and vocabulary management process. Such maintenance can also be performed automatically.
  • a user maintaining a semantic net would track search queries in a query log. From the query log, he would determine wliich words are actually novel and are candidates to be added to the system vocabulary, by expanding the query log using morphology, and possibly a spell checker and name identifier. The remaining terms that were not matched are the basis of a list for adding terms to the vocabulary.
  • Tools to manage vocabulary include: • A morphological analyzer. This tool strips off any endings and morphological alterations in a query to find the stem, and checks to see if the stem is in the current vocabulary. If the stem is not, the user doing the maintenance might try:
  • a spell checker This tool uses the conventional algorithms to see if the supposedly new word is actually a misspelling of a known word. If it is not a misspelling, the user might try:
  • a name identifier This tool checks to see if the supposedly new word is in a name configuration, in that it follows a known first name in the query. If it does, it is added to a candidate name database. If it is not, it is proposed as a possible new word to be added to the system's vocabulary.
  • Nideo and audio files can be timecoded, or marked such that the software in which they run can locate a specific frame (for videos) or measure (for audio) at any time.
  • the system according to the invention permits searching time-based media, including video and audio files, by combining two search elements.
  • the first is a standard search, including but not limited to natural language search.
  • the second is a time indicator, such as a SMPTE (Society of Motion Picture and Television Engineers) standard timecode.
  • Face recognition is an additional technology that can be used in searching. Face recognition is a subset of the more general technology of object recognition, and indeed techniques described here may extend to additional technologies as well.
  • the current state of the art in face recognition technology makes it possible to take a manually created, labeled library of faces, and match faces from a video to that library.
  • a user might work with a news video and use a face recognition program to label Nelson Mandela in it.
  • the output of the face recognition program would be a time-coded segment, with start and stop times, of when Nelson Mandela was on camera, with the label "Nelson Mandela" attached to the time codes.
  • face recognition currently does not achieve 100% precision or recall, it can still be useful.
  • one known system offers a contract rights management capability for films that demands time-coded segments with names attached, and assumes that users will create those manually, so that the correct contract restrictions for each film segment can be attached to the right time codes. Given a small library of the actors in a film, it would be possible to do a fast, automated match-up of time codes and actors, even with imperfect face recognition technology. Selecting the correct actor from forty publicity shots would be much simpler than selecting from among thousands of faces.
  • the system according to the invention carries out the automated creation of the face library.
  • Required elements include time-coded metadata (for example, the voice recognition transcript of a video), and the ability to find the names of people in text. Each time a face and a person's name appear at the same time code, that occurrence is a potential new entry for the face library.
  • a user may run the face-to-name matcher for thousands of hours and sift out the recurring matches as the most likely. In this way, a reference library of faces is created, and new material can be catalogued automatically.
  • the software according to the invention approaches this by using alignment techniques to match up two or more streams of metadata.
  • a broadcast news program may contain closed captioning for the hearing-impaired. It may also contain a separate description of the news footage, probably created manually by the news department.
  • the system according to the invention uses alignment to match the description, which is not time-coded, with the closed captioning, which is time-coded. This process allows the system to add time codes to the non-time-coded stream.
  • the software uses that new, derived stream (i.e. the description with newly added time codes), and searches for proper names within it.
  • face recognition algorithms on the video stream the software finds faces. The system tries to match up the faces with the proper names that describe who they are.
  • This matched set provides us with a rough cut of a face (or object) reference library. This is exemplified in Fig. 4.
  • Face recognition can also be employed to manage the library or archive of media files.
  • Media libraries are assembled over time, often from disparate sources, and may contain multiple copies of a single media file, either with the same metadata or with different metadata.
  • Duplicate detection is therefore an important element of library and archive management, and face recognition (and, more generally, image recognition) can be leveraged to provide that capability. More broadly, for video, scene detection technology can assist in the process of identifying duplicates so that they can be purged from the library.
  • Clustering involves combining user search queries in such a way that the searches can be analyzed usefully to provide answers to business questions.
  • Clustering has received considerable attention in document information retrieval (IR) and more recently, in video IR as a means of refining retrieval results based on user preferences or profiles, and to characterize the marketplace.
  • IR document information retrieval
  • the prior art contains many examples of clustering applied in information retrieval systems, but they all apply to search results returned to users rather than search queries submitted by users.
  • This system characterizes the information needs of groups (and subgroups) of users, with respect to a collection of media files (e.g. images, videos, sound clips, text, multimedia objects).
  • media files e.g. images, videos, sound clips, text, multimedia objects.
  • This system applies clustering technology to user- submitted search queries, and to the files retrieved in search results. It also includes:
  • novel clustering techniques include using a semantic expansion (such as the WordNet hierarchical thesaurus) and phrase identification (such as noun phrase, and name and location identification) as the basis for the clustering.
  • a semantic expansion such as the WordNet hierarchical thesaurus
  • phrase identification such as noun phrase, and name and location identification
  • each query Before user queries can be analyzed, they must be expanded to a "common denominator". To expand the user search queries, we use natural language techniques. Specifically, we treat each query as if it were metadata within our system, as described in the NLP section, above. Each query is expanded through the application of a semantic net, so that it contains synonyms and related terms. This expansion provides a set of features to which we can apply standard clustering technology.
  • the data set can be selected according to various criteria, including:
  • a first step is to select the data set on which clustering is to be performed.
  • clustering can be performed on queries or on assets to be retrieved (documents, images, video, audio, mixed media).
  • a sample query set may include short queries, as is standard on Web searches, long queries, as seen in TREC (U.S. government- sponsored text retrieval conferences), or as produced by QBE (query by example), in which an asset or set of assets are themselves inserted into a query.
  • a second step is to perform analysis on the queries using, for example, linguistic methods, such as:
  • Morphology or stemming removed tense and plural markers and other affixes to find the word root.
  • a third step is to expand the queries. Ideally, this step includes expansion using a thesaurus or semantic net of synonyms, super-terms, and other relationships.
  • a fourth step is, for each of the terms in each expanded query, assign a weight based on how close that term is to the original query. The exact weighting will vary by application, but the basic understanding is that more closely matching terms are weighted close to 100.
  • a fifth step is to create a vector for each expanded query.
  • a sixth step is to a pply a statistical clustering algorithm in order to group similar queries together.
  • a 1-ierarchical or linear clustering strategy may be used, depending on whether the desired clusters are hierarchical or not. Clustering may allow overlap, which means that a query may appear in more than one cluster.
  • a seventh step is to apply the clustering algorithm until the stopping condition is met, e.g. the desired number of clusters is obtained, or a combination of cluster number and desired distinctiveness of clusters is reached.
  • An eighth step relates to the clusters.
  • Clusters are most useful to a human observer if they bear useful names that reflect the content.
  • implicit popularity rankings may be derived from one user or set of users and applied to a different user or set of users. For example, if User A places a media file in a projects folder or shopping cart, the information on her activity can also be used to re-rank search results for User B, who is in some way similar to user A. In another example, if users with ".edu" e-mail addresses buy certain things, it makes sense to re-rank search results to favor those things when showing results to other ".edu" users. In the system according to the invention, if registered users who work for advertising agencies have placed certain items in their shopping carts, other advertising agency employees can have their search results re-ranked to favor those items. The opposite can be true as well: the same system can be used to disfavor certain items because they have been sold too many times to advertising agencies, for example.
  • the retrieval system does not answer any of the TREC tracks — a media file is described by a short paragraph or by keywords. Even though it is possible to develop a unique test collection for purposes of evaluation, it will not necessarily predict the performance of new systems, or even for existing ones.
  • 'Performance as used in this inventive system is a score on a scale of user satisfaction with the respect to search results rankings.
  • User satisfaction is determined by tracking the user actions together with the system's data. For example, for a very good performance, the user would submit a search query and purchase the media file that was ranked first. Possible indications of poor performance might include a user issuing several consecutive queries, getting thousands of results, and te ⁇ ninating the session without further examining any of the media files.
  • the software application according to the invention includes a viewer for annotations, or mark-ups, of full motion digital still images, video frames, or other indexed data structures.
  • the viewer preferably accessible on a personal computer with a Web browser, serves as a helper application for the underlying application of the system according to the invention.
  • the viewer allows users to view and mark up (comment on) media on a separate, see-through, lightweight layer.
  • the viewer supports annotations that are created and rendered so that they appear overlaid on the related media.
  • Position, size, frame number, and other attributes of annotations are saved in a separate annotation file, and when played back the annotations appear at the saved location and with the saved attributes.
  • creating and editing of annotations is be done in WYSIWYG (what you see is what you get) fashion. This is illustrated in Fig. 5.
  • the annotations may consist of text, graphics, and digitized audio, and are stored separately from the original file information.
  • An indexing scheme relates individual annotations to individual video frames. On full motion video playback, the annotations are displayed on the originally annotated frames.
  • One central part of the system according to die invention is the use of a downloadable, view-only application, which enables users who do not have the full power to create annotations nonetheless to be able to view annotations created by others.
  • files are stored centrally.
  • the viewer provides security, e.g. SSL, server authentication, and proxy authentication.
  • SSL Secure Sockets Layer
  • proxy authentication e.g. HTTPS, HTTPS, Secure Sockets Layer
  • the underlying application according to the invention powers the full lifecycle of digital media from content collaboration, deep storage, search and retrieval using a natural language search and the ability effectively to move bandwidth-intensive media over the Web.
  • the software is preferably Web-based.
  • the viewer is a standalone, platform-native "helper" application that is noimally launched by a Web browser, from a specific link on a Web page that is part of the Web server of the system according to the invention.
  • the viewer has the ability to upload annotation data back to the Web server. Except for the two actions of launching and uploading, the viewer does not require any network connection with the Web server or the Web browser.
  • the current viewer in the system according to the invention on a PC or Macintosh platform allows the user to view and annotate various video, still image, audio, and text formats, including MPEG, QuickTime, or RealMedia videos, bitmap, jpeg, and tiff images, MP3, .wav, .aiff, .au, and .snd audio files, and .pdf document files.
  • the viewer currently has nine modes in which it can operate, all of which would be relevant to the mobile user. They are:
  • the current (Mac and Windows) viewers can be launched in any of these nine modes, generally by receiving an XML annotation file.
  • Attributes common to all annotation types are: position, size, frame number (video only), and page number (multipage document only). All of the graphical attributes such as colors and fonts are modifiable using platform-native dialog boxes. These attributes can be set for new annotations as well as previously created ones.
  • each annotation is recorded with the timecode of the specific frame on which it was created.
  • the viewer stops playback at each frame where an annotation exists.
  • the viewer also contains a drop-down list of all annotated frames (by timecode) so that individual annotated frames can be viewed quickly. Controls are also provided for stepping by a single frame in either direction, as well as a control displaying the timecode of the current frame where the viewer is paused.
  • the PDA (or handheld or wearable) version of the viewer builds on the same general architecture.
  • the server generates a simple XML annotations file, sends it to the client (i.e. the viewer via the Web browser), and handles the returned HTTP messages from the viewer.
  • the following annotation types are desirably supported on a PDA version:
  • the system according to the invention supports the full life cycle of media; allowing organizations to expedite the process of searching, collaborating and distributing of digital assets, such as imagery, video files, pictures and text documents.
  • Users desirably have the freedom to collaborate anywhere at anytime using die annotation viewer according to the invention on a PDA.
  • Users are preferably able to annotate video and still images with a virtual pen, sticky note, text and even voice using a wireless modem
  • Annotations are preferably capable of being added, edited, moved, or deleted.
  • a PDA-based digital media management solution takes digital delivery and collaboration to the next level by eliminating the need of a traditional PC and Ethernet connection.
  • users no longer need to carry laptops to stay up to date on work involving video or still images.
  • the viewer supports both HTTP authentication and Proxy authentication.
  • the viewer displays a login dialog where the user can enter a user ID and password to be used for authentication. Since the viewer is activated and closed frequently, it should not prompt the user to log in each time it connects to the server.
  • the viewer instead provides a mechanism for saving the user ID and password on the local workstation (in encrypted form) and passing this information to the server without any prompting. It is desirable that the viewer support SSL encryption for all of its HTTP requests.

Abstract

The software according to the invention incorporates a glossary management tool (fig. 2) that makes it easy for each client to customize terminology (fig. 2) to the needs of a particular business. With this tool, termed a glossary manager (fig. 2), a company can customize a number of feature names (fig. 2) in the system to provide a more familiar context (fig. 2) for their users. A system administrator can also customize the manner in which 'thumbnail' or 'preview' images (fig. 2) are presented. The system performs clustering on search queries, and searches media records multi-modally, using two or more approaches such as image searching and text searching. An administrator can tune search parameters. Two or more streams of metadata may be aligned and correlated with a media file, facilitating later searching. The system evaluates itself. It folds popularity information into rankings of search results.

Description

Method and Apparatus for Digital Media Management, Retrieval, and Collaboration
Background
Creative designers in various fields work with materials from many different sources when they develop an advertisement, film, brochure, or other finished product. These materials may have been created originally by the creative designer, but for a different project, and stored in a media library or archive for later reuse. They may have been created by colleagues, and stored in an area that allows different individuals to collaborate on materials. Or they may have been created by professional artists for licensing or sale. These materials are often called rich media files, or assets. Rich media files may include, but are not limited to, video, photography, graphics, audio, mixed media files, logos, presentations, and text. These media files can exist in the widest possible range of formats. It is a ponderous task to manage such assets. Annotation mechanisms for such assets include the system described in U.S. Pat. No. 5,493,677, assigned to the same assignee as the assignee of the present invention. Systems to manage such assets include that described in U.S. Pat. No. 6,012,068.
It would be desirable to have a management system for digital media which stre--rr-lines the task of accounting for rights to use such media, including copyright rights. Systems relating to rights management include those described in U.S. Pat. Nos. 4,337,483, 5,201,047,
5,260,999, 5,263,158, 5,319,705, 5,438,508, 5,629,980, 5,765,152, and 5,553,143.
Users in different businesses may use different terminology to refer to the various media management functions. For example, some may use the term library, while others use archive. Some may use project workspace, while others use share or collaboration tool. In many systems, changing terminology requires tedious progra-ximing effort which risks introducing errors into the software. It would be desirable to have a management system that conveniently permits non-technical users to customize such terminology on a per-system basis. U.S. Pat. No. 5,850,561 describes a glossary construction tool for creating glossary from text. Different companies using the software according to the invention may have different corporate culture, image, and system context within the company. It would be desirable to have a management system that conveniently permits non-technical users to customize the software on a per-system basis with respect to such corporate concerns.
In a typical asset management system, users browse through media file collections and view thumbnail images of files to decide which files they want to work with. These thumbnails are browseables, or small representations of the actual images, videos, or other media files in the system. A browseable is created by optimizing an image or video frame for online browsing, so a browseable has lower resolution and smaller dimensions than the original file. It is commonplace, however, to find that the resolution and dimensions are not well suited to the company. It would be desirable to have a management system that conveniently permits a system administrator to customize the software in this respect.
Natural language processing (NLP) techniques are well known, including their use in information retrieval applications (Strzalkowski, 1993), (Strzalkowski, Perez Carballo and Marinescu, 1995), (Evans and Zhai, 1996). Past systems have attempted to improve upon vocabulary management techniques, for example as described in U.S. Pat. Nos. 5,251,316 and 6,125,236. Past approaches for searching multimedia include U.S. Pat. Nos. 6,243,713 and 5,794,249.
Clustering is well known, for example in U.S. Pat. No. 5,317,507, 5,758,257, 5,675,819, 5,778,362, and 5,875,446. See also Buckley, Chris, J. Walz, M. Mitra and C. Cardie, "Using Clustering and Super Concepts within SMART: TREC 6" (http://trec.nist.gov/pubs/trec6/t6_piOceedings.html); Zamir, Oren, O. Etzioni, Madani, and
Karp, KDD-97, "Fast And Intuitive Clustering Of Web Documents;" and Koller, Daphne, and Mehran Sahami, ML-97, "Hierarchically Classifying Documents Using Very Few Words." Rankings relating to relevance are discussed in U.S. Pat. No. 5,642,502.
The evaluation of information retrieval systems became an essential part of the field in the early '90s, and was strongly advanced by the TREC evaluations designed at NIST beginning in 1993. The TREC evaluation contains different tracks, but the tracks all share the following common features:
They are designed to provide a comparative evaluation between different systems, usually provided by different participants.
The evaluation is done using strict test conditions that contain a set of queries, a collection of documents, and relevance judgements.
The evaluations use evaluation scores such as precision and recall that supposedly predict real users' satisfaction from a system
While these evaluations are indeed helpful in comparing the performance of different IR systems, they do not provide constant feedback on the performance of a live IR syste The base performance of an TR system could be at first evaluated using a standard measurement such as the one above, but as more media files are added to a system and users submit queries in an uncontrolled manner, it is hard to predict or estimate the performance of the system. In addition, if the system does not fall into the initial TREC evaluation tracks, it is necessary to develop an independent test case — a very costly task. It is desirable to have a better self- evaluation system for such a digital asset manager.
Known annotation-related systems are discussed in U.S. Pat. No. 5,600,775, 6,006,241, and 5,938,724.
Summary of the invention
The software according to the invention incorporates a glossary management tool that makes it easy for each client to customize terminology to the needs of a particular business. With this tool, termed a glossary manager, a company can customize a number of feature names in the system to provide a more familiar context for their users. A system administrator can also customize the manner in which "thumbnail" or "preview" images are presented. The system performs clustering on search queries, and searches media records multi-moda-ly, using two or more approaches such as image searching and text searching. An administrator can tune search parameters. Two or more streams of metadata may be aligned and correlated with a media file, facilitating later searching. The system evaluates itself. It folds popularity information into rankings of search results.
Description of the drawing
The invention will be described with respect to a drawing in several figures, of which:
Fig. 1 shows a typical rights management editor interaction;
Fig. 2 shows a typical customization interaction screen;
Fig. 3 shows a screen permitting customers to adjust search parameters;
Fig. 4 shows a sample of a face-matching system according to the invention;
Fig. 5 illustrates a viewer according to the invention with text annotation box.
Detailed description
The inventive software manages media files, or assets, making it possible to store and retrieve them, to collaborate with colleagues to edit and work with them, and to deliver them, either for further processing or in finished form for use.
The workflow of the inventive software is designed to provide flexible choices among features that support a creative user who works with rich media files. As soon as a user logs on to the software (an optional step), she can immediately see if any system-internal messages are waiting in her system email inbox, either sent from other users collaborating with her on media files, or sent automatically by the inventive system to notify her of various occurrences that may pertain to her work, such as a project update. The user can read her messages, reply to a message, or compose a new message to send to another system user. A composed message may consist of text only, or it may have a system-internal file attached.
The user may wish to work on a project, such as an advertising campaign. She enters the project work area, and creates a new project. This creates a virtual workspace in the inventive system that is dedicated to holding media files and all their related files, including edited versions, colleague comments, official approvals, and so on. She decides which users will be allowed to share access to the project space. She may upload files that she has already created into the project work area. From there, she can send a media file off to a colleague for comments, editing, or approval.
If she decides to rework a previously created file for use in the new ad campaign, she can search for the file in the file storage area of the inventive syste She enters search terms, generally either by typing or by speaking into an input device, and finds an appropriate media file. She then saves the file to her new project. She may decide, instead, to have the full, high-resolution version of that file sent to her for a different project. In the inventive system, delivery options for high-resolution files include on-line delivery via ftp, http, or special network; via CD delivery, or even via mail delivery of the file in analog format. Ordering and file delivery in the inventive system are subject to several conditions, which the software verifies and tracks. These conditions include whether the user has the proper rights and permissions, not only to view the file but also to use or publish it. If use is subject to a fee, the fee must be paid, whether by billing, credit card, or other means. If usage requires personal approval, the request is routed to the right person and approval is granted and tracked, all within the inventive software syste
Rights management. If a user determines that she needs to use a rights-restricted media file, she must engage in a dialogue with the inventive software about her planned use of the file. Companies can customize sets of questions that the software presents to users, depending on which file a user wants to order. The user may have to answer whether she intends to publish the file, where it will be published, to what size audience, at what location, on what date. The answers deteimine whether the file can be made available, and can also determine the price. For example, if she is showing ten seconds of a movie for a one-time training seminar in Canada for an audience of fifty people, her cost may be far less than making those same ten seconds of movie available on the home page of a major entertainment Web site. And making the entire movie available on the Web may perhaps not be permitted at any price.
When users order files in the software according to the invention, the rights management process controls every aspect of the order. A company may allow its employees to use its media files for personal use, marketing use, or in an internal organization project. Or they may allow all three uses in different situations. They can track the user's use of the files, they can designate the licensing type for the files and make sure a rights agreement is included, and they can enforce legal or company policy restrictions that may apply. The rights management process helps clients establish all of these elements so they can apply them appropriately to their stored media files. The invention described here makes it easier for individual clients to customize a rights management setup so that the software will make the right pricing and availability decisions for each projected use. Central to the system is a question-answering paradigm: an administrator or a manager decides what questions should be presented to each potential user of a file, and what set of multiple choice answers to supply. The software maps the questions and answers to a rights and usage package, and, if applicable, each package is assigned pricing.
For example, an administrator may determine that a file's availability depends on:
What department is giving the presentation?
Who is the audience?
Will the image be included in any handouts?
Will the image be displayed on an overhead? When is the presentation date?
Does the work contain images of any sculpture, painting, drawing, or other art object that may be copyrighted?
The questions, in turn, are grouped appropriately for a company's particular file collections, and are assigned to either groups of files or individual files. Questions and their answers may serve as fundamental criteria for whether a file is made available; they may serve as modifiers to an available file by raising or lowering pricing depending on usage; or they may simply be informational questions that do not play a role in decision-making, but allow companies to track file use and user needs.
The rights management process. When ordering files, users answer questions that ask for information about how they are planning to use the files they order. The questions a user answers are the usage questions a company designates for each possible use. This set of usage questions is called a rights package. Creating a rights package is the first step in the rights management process.
A rights package, or set of questions, is added to other rights packages, to make up a rights package group. This allows a company to make more than one rights package available for a group of files. A rights package group is added to a particular license type. The license type describes the possible ways a file can legally be used. A rights agreement is added to define details clearly about the user's rights. In the final step, the rights agreement is associated with a particular license type through a control table. The control table also contains pricing and approval information for each rights package.
The core piece of this process is the creation and grouping of questions that cover a company's business and legal interests for their stored media file usage. Part of this process is completed with what may be termed a rights package question role editor. A typical editor interaction is shown in Fig. 1. Glossary Customization Tool and Usability Customization Tool. The software according to the invention incorporates a glossary management tool that makes it easy for each client to customize teπriinology to the needs of a particular business. With this tool, termed a glossary manager, a company can customize a number of feature names in the system to provide a more fa-xiiliar context for their users.
The system also includes a usability management tool that customizes the software interface "look and feel" to more closely reflect a company's corporate culture, image, and to reflect the system context within the company. A part of the system termed a usability manager allows a company to change system background colors, fonts, and the embedded logo. This tool also makes localization into a different language simpler. A typical customization interaction screen is shown in Fig. 2.
User-configurable browseable generation. In a typical asset management system, users browse through media file collections and view thumbnail images of files to decide which files they want to work with. These thumbnails are browseables, or small representations of the actual images, videos, or other media files in the system A browseable is created by optimizing an image or video frame for online browsing, so a browseable has lower resolution and smaller dimensions than the original file. The browseable (the proxy of the actual media file) is what the user sees in the system according to the invention, and browseable size is generally set by a system administrator for use throughout the system. Importantly, the system permits generating these lower-resolution proxy files in a way that can be configured by the customer.
Natural language processing techniques. In the system according to the invention, the natural language processing component is layered. The system architecture supports a conventional search algorithm, which operates as a plug-and-play support application. The algorithm is then used in a word expansion process. The word expansion is applied to both a user's search query, and to the stored files' captions, that is, descriptive metadata identifying individual files. When a user submits a search query, the query undergoes several types of NLP processing (detailed below), and the result of each element in the processing adds new query components (for example, synonyms) and/or weights (which emphasize or de-emphasize words depending on their value to the query). Tl e resulting, or processed, query contains expanded terms and weighting information that can be passed to any search engine. This makes it possible to use multi-search agent (or "federated search") applications as well.
To expand a user search query, once a user query is received, it is divided into individual tokens, which may consist of single words or multiwords. (Multiwords are words the search engine recognizes as one unit, instead of as separate words. With this feature, when a user searches for sea lions, the system does not bring up lions by the ocean.) For this process, a variation of conventional pattern matching is used. If a single word is recognized as matching a word that is part of a stored multiword, the processor decides whether to treat the single word as part of a multiword, based on the contents of the stored pattern and the input pattern. Stored patterns include not just literal words, but also syntactic categories (such as adjective or non- verb), semantic categories (such as nationality or government entity), or exact matches. If the input matches the stored pattern information, then it is interpreted as a multiword rather than as independent words.
Processing then determines whether the resulting word is a function word (closed-class) or content word (open-class). The processor ignores function words. For content words, the related concepts for each sense of the word are retrieved from the semantic net. If the root word is unknown, the word is treated as a keyword, requiring an exact match.
The system then matches the expanded, weighted search query with stored media files in order to return search results to the user. Media files are identified by the metadata associated with them, e.g. in a caption. Text-based files can be identified by metadata associated with them, and also by their content. When users import media files into the system, the software provides a workflow to add captions to each file. The files are then uploaded into the database, and the file captions are expanded and the information stored for later file searching. To expand the file captions, the processor looks up each word in tlie caption or text. Words that may be related in the semantic net are located based on stored links, and the looked-up word, along with any related words, are all displayed as the "expansion" of that word. The processor detei ines whether the current word or phrase corresponds to a proper name, a location, or something else. If it corresponds to a name, a name expansion process is invoked that displays the name and related names such as nicknames and other variants, based on a linked name file. If the current word or phrase corresponds to a location, a location expansion process is invoked that accesses a gazetteer and displays tlie location and related locations based on linked location information found in tlie gazetteer and supporting files. If the current word or phrase is neither a name nor a location, it is expanded using the semantic net links and weights associated with those links. Strongly related concepts are given high weights, while more remotely related concepts receive lower weights, making them less exact matches. Thus, for a query on car, documents or metadata containing car and automobile are listed highest, followed by those with sedan, coupe, and convertible, and then by more remotely related concepts such as transmission, hood, and trunk.
Once the expansion is complete, the expanded word or phrase is stored in an index database, where it is available for use in searching as described below. Processing then returns to expand the next word or phrase in the text or caption.
The system according to the invention handles certain words and phrases individually. Multiwords are matched as a whole unit, and names and locations are identified and looked up in the separate name and location files. Next, noun phrases and other syntactic units are identified. A part-of-speech tagger uses linguistic and statistical information to tag the parts of speech for sections of the user query. Only words that match by part of speech are considered to match, and if two or more parts of speech are possible for a particular word, it is tagged with both. After tagging, word affixes (i.e. suffixes) are stripped from query words to obtain a word root, using conventional inflectional morphology. If a word in a query is not known, affixes are stripped from the word one by one until a known word is found.
An intermediate query is then formulated to match against the file index database. Texts or captions that match queries are then returned, ranked, and displayed to the user, with those that match best being displayed at the top of the list. In an exemplary system, the searching is implemented by first building a B-tree of ID lists, one for each concept in the text database. The ID lists have an entry for each object whose text contains a reference to a given concept. An entry consists of an object ID and a weight. The object ID provides a unique identifier and is a positive integer assigned when the object is indexed. The weight reflects the relevance of the concept to the object's text, and is also a positive integer.
To add an object to an existing index, the object ID and a weight are inserted into the ID list of every concept that is in any way relevant to the text. For searching, the ID lists of every concept in the query are retrieved and combined as specified by the query. Since ID lists contain IDs with weights in sorted order, determining existence and relevance of a match is simultaneous and fast, using only a small number of processor instructions for each concept-object pair.
Search Technologies. The system allows users to search for media files with many different types of search queries. For example, users may submit search queries by speaking them, typing them, copying them, or drawing the
The process of locating a particular file in a large archive is a special area for innovation within the inventive software. Files are characterized in several ways. First, they have an identifier, generally similar to a filename, which is unique within the system and makes it possible to link up all the objects related to a file. These can include the actual high-resolution asset, lower-resolution thumbnails or other proxies for browsing, and information about the file, or metadata. Searching can be performed on the file identifier, or it can be performed on the metadata. In the case of metadata searching, it is desirable to offer search alternatives that go beyond the exact matching process involved in a standard keyword search.
Some systems use controlled vocabulary searching as an optimization of keyword searching.
Keyword searches simply match exactly on any word in the user's search query that appears in the search target. (In the system according to the invention, the search target is the metadata describing a media file.) The set of potential keywords is quite large (as large as the vocabulary of English, or whatever language(s) are being used). If there are no limitations on the search vocabulary that can be employed, a user can enter a search for puma and fail to find any files captioned as mountain lion or cougar, even though they all refer to the same thing. Controlled vocabulary is an attempt to address this problem, albeit at considerable cost. In a controlled vocabulary retrieval system, cataloguers all agree to use the same terms. In practical terms, this implies that, when cataloguing, they must check their controlled vocabulary lists and be sure not to deviate. Sometimes tools can be built to aid in this process, depending on the size of the controlled vocabulary. Similarly, tools can also be provided to searchers to control their search requests. However, controlled vocabulary systems do not scale beyond a few thousand terms, since it is impractical to look up every word in English for every search. For broader retrieval systems, for faster cataloguing, and for simpler searching, a different approach is superior.
In addition to standard keyword and Boolean searching, the system software incorporates additional advanced technology for locating stored files. Rather than limiting se--rching to a controlled vocabulary, the system software includes natural language search, which allows cataloguers and users to employ any words in English (or whatever natural language the retrieval system is using).
Natural language search incorporates:
a semantic network of concepts additional linguistic techniques, including: phrase matching derivational morphology, in lieu of ste-ximing part of speech tagging name recognition location recognition User-Tunable Search Parameters. The system according to the invention provides a screen for customers to adjust search parameters, to reflect their company use of stored media file collections. This is shown in Fig. 3. While the parameters may themselves be well-known in a searching system, what is emphasized here is that the user (or, more likely, an administrator) can be granted access to such fundamental decisions about search as:
(a) how good a match has to be before it is displayed to the user, e.g. 50%, and
(b) how "creative" the search should be, i.e. how much should the search terms be expanded to include more distant synonyms and related terms.
It should be borne in mind that the system according to the invention can be carried out on an internet, meaning an IP-based network, and in particular may be carried out on the Internet, meaning the global IP-based network.
Multimodal Search. Currently, search methods focus on textual input. The current invention incorporates new search techniques, and combines them in novel ways.
Image search is becoming useful in commercial applications. In the system according to the invention, user search input is provided in a new way. Users may wish to select an existing image as example input, so that a search consists of "Give me more images like this." Perhaps even more useful is the ability to select part of an image, analogous with "Give me more like this part." In the system according to the invention, identifying the part may be done in either of two exemplary ways:
1. Touch screen: user touches the screen to identify the portion of the image that feeds into the search.
2. Markup, using pen or other screen drawing metaphor, including through the system media viewer, which is described in more detail below. In addition, search modalities can be combined. This novel approach to search is particularly applicable to multimedia. Examples of combined, or multimodal, searches, include:
- voice and text - touch screen and text
- drawing and voice
- drawing and touch screen
and so on.
Vocabulary Management. Ideally, the semantic net of concepts is quite large and attempts to incorporate every word or term in English (or other language being used for cataloguing and searching). No matter how large the semantic net may be, there will be a periodic need to expand or edit it. New words appear in English periodically, and, although many may be slang and therefore not particularly important in a business context, some will be real new words and will be important enough to include. For example, rollerblading and in-line skating are relatively new terms in English, and depicting those actions is useful in advertising. So the terms need to be added to the semantic net. Semantic net/vocabulary maintenance is generally a manual process, particularly where the user has an existing media library with a thesaurus and vocabulary management process. Such maintenance can also be performed automatically.
To maintain a vocabulary for an information retrieval application that accepts user queries in natural language, a user maintaining a semantic net would track search queries in a query log. From the query log, he would determine wliich words are actually novel and are candidates to be added to the system vocabulary, by expanding the query log using morphology, and possibly a spell checker and name identifier. The remaining terms that were not matched are the basis of a list for adding terms to the vocabulary.
Tools to manage vocabulary include: A morphological analyzer. This tool strips off any endings and morphological alterations in a query to find the stem, and checks to see if the stem is in the current vocabulary. If the stem is not, the user doing the maintenance might try:
A spell checker. This tool uses the conventional algorithms to see if the supposedly new word is actually a misspelling of a known word. If it is not a misspelling, the user might try:
A name identifier. This tool checks to see if the supposedly new word is in a name configuration, in that it follows a known first name in the query. If it does, it is added to a candidate name database. If it is not, it is proposed as a possible new word to be added to the system's vocabulary.
Searching Audio/Video by Timecode Correlation with Search Criteria. Nideo and audio files can be timecoded, or marked such that the software in which they run can locate a specific frame (for videos) or measure (for audio) at any time. Importantly, the system according to the invention permits searching time-based media, including video and audio files, by combining two search elements. The first is a standard search, including but not limited to natural language search. The second is a time indicator, such as a SMPTE (Society of Motion Picture and Television Engineers) standard timecode. Face recognition is an additional technology that can be used in searching. Face recognition is a subset of the more general technology of object recognition, and indeed techniques described here may extend to additional technologies as well. The current state of the art in face recognition technology makes it possible to take a manually created, labeled library of faces, and match faces from a video to that library. For example, a user might work with a news video and use a face recognition program to label Nelson Mandela in it. The output of the face recognition program would be a time-coded segment, with start and stop times, of when Nelson Mandela was on camera, with the label "Nelson Mandela" attached to the time codes. While face recognition currently does not achieve 100% precision or recall, it can still be useful. For example, one known system offers a contract rights management capability for films that demands time-coded segments with names attached, and assumes that users will create those manually, so that the correct contract restrictions for each film segment can be attached to the right time codes. Given a small library of the actors in a film, it would be possible to do a fast, automated match-up of time codes and actors, even with imperfect face recognition technology. Selecting the correct actor from forty publicity shots would be much simpler than selecting from among thousands of faces.
Iπ-portantly, the system according to the invention carries out the automated creation of the face library. Required elements include time-coded metadata (for example, the voice recognition transcript of a video), and the ability to find the names of people in text. Each time a face and a person's name appear at the same time code, that occurrence is a potential new entry for the face library. A user may run the face-to-name matcher for thousands of hours and sift out the recurring matches as the most likely. In this way, a reference library of faces is created, and new material can be catalogued automatically.
The software according to the invention approaches this by using alignment techniques to match up two or more streams of metadata. For example, a broadcast news program may contain closed captioning for the hearing-impaired. It may also contain a separate description of the news footage, probably created manually by the news department. The system according to the invention uses alignment to match the description, which is not time-coded, with the closed captioning, which is time-coded. This process allows the system to add time codes to the non-time-coded stream. The software then uses that new, derived stream (i.e. the description with newly added time codes), and searches for proper names within it. At the same time, using face recognition algorithms on the video stream, the software finds faces. The system tries to match up the faces with the proper names that describe who they are.
This matched set provides us with a rough cut of a face (or object) reference library. This is exemplified in Fig. 4.
Face recognition can also be employed to manage the library or archive of media files. Media libraries are assembled over time, often from disparate sources, and may contain multiple copies of a single media file, either with the same metadata or with different metadata. Duplicate detection is therefore an important element of library and archive management, and face recognition (and, more generally, image recognition) can be leveraged to provide that capability. More broadly, for video, scene detection technology can assist in the process of identifying duplicates so that they can be purged from the library.
Clustering and Other Ways to Determine Stored File Usage. Clustering involves combining user search queries in such a way that the searches can be analyzed usefully to provide answers to business questions. Clustering has received considerable attention in document information retrieval (IR) and more recently, in video IR as a means of refining retrieval results based on user preferences or profiles, and to characterize the marketplace. The prior art contains many examples of clustering applied in information retrieval systems, but they all apply to search results returned to users rather than search queries submitted by users. In the system according to the invention, we cluster search queries by topic. We then use that information to adjust the collections of stored files so that the file collections will better meet users' needs.
This system characterizes the information needs of groups (and subgroups) of users, with respect to a collection of media files (e.g. images, videos, sound clips, text, multimedia objects). Some common groupings include:
search queries that brought back no files
search queries that brought back no files the user was interested in
search queries that lead to expressions of user interest or sales
This system applies clustering technology to user- submitted search queries, and to the files retrieved in search results. It also includes:
machine learning as applied to the above. • characterizing the information needs over time, by user type, or by other factors.
• methods for reporting file collection needs to interested parties (for example, media suppliers). The system informs a supplier that pictures of earthquakes are selling briskly, or that users keep looking for videos of dance performances but cannot find any.
• methods for adjusting file collections based on the results of the clustering analysis, above.
novel clustering techniques. These include using a semantic expansion (such as the WordNet hierarchical thesaurus) and phrase identification (such as noun phrase, and name and location identification) as the basis for the clustering.
Before user queries can be analyzed, they must be expanded to a "common denominator". To expand the user search queries, we use natural language techniques. Specifically, we treat each query as if it were metadata within our system, as described in the NLP section, above. Each query is expanded through the application of a semantic net, so that it contains synonyms and related terms. This expansion provides a set of features to which we can apply standard clustering technology.
Obtaining valuable information on user preferences for stored files begins with deciding what information a client wants to understand. The data set can be selected according to various criteria, including:
queries from a particular subset of users (e.g. registered users, users by industry, new users)
queries that lead to success (sale or other indication)
queries that lead to failure A first step is to select the data set on which clustering is to be performed. In an information retrieval (IR) context, clustering can be performed on queries or on assets to be retrieved (documents, images, video, audio, mixed media). A sample query set may include short queries, as is standard on Web searches, long queries, as seen in TREC (U.S. government- sponsored text retrieval conferences), or as produced by QBE (query by example), in which an asset or set of assets are themselves inserted into a query.
A second step is to perform analysis on the queries using, for example, linguistic methods, such as:
Tokenization: determine word/token boundaries. In English, tokenization mostly coincides with spaces between words, with certain complications (Alzheimer's = 1 token vs. she's = 2 tokens).
Morphology or stemming: removed tense and plural markers and other affixes to find the word root.
Identify names, locations, noun phrases: using a pattern matcher or other methodology, determine words were groupings for special handling. For example, for names, match certain kind of variance; for locations, match subset; for noun phrases, we to complete and head-only matches higher than modifier home-only matches.
A third step is to expand the queries. Ideally, this step includes expansion using a thesaurus or semantic net of synonyms, super-terms, and other relationships.
A fourth step is, for each of the terms in each expanded query, assign a weight based on how close that term is to the original query. The exact weighting will vary by application, but the basic understanding is that more closely matching terms are weighted close to 100.
A fifth step is to create a vector for each expanded query. In order to apply a statistical clustering algorithm, we arrange the vectors into a matrix. A sixth step is to a pply a statistical clustering algorithm in order to group similar queries together. A 1-ierarchical or linear clustering strategy may be used, depending on whether the desired clusters are hierarchical or not. Clustering may allow overlap, which means that a query may appear in more than one cluster.
A seventh step is to apply the clustering algorithm until the stopping condition is met, e.g. the desired number of clusters is obtained, or a combination of cluster number and desired distinctiveness of clusters is reached.
An eighth step relates to the clusters. Clusters are most useful to a human observer if they bear useful names that reflect the content. Use a semantic net hierarchy, combined with term frequency in a reference corpus, to identify the lowest-level term in the hierarchy that subsumes all the queries.
Folding Popularity Into Rankings. Many information retrieval applications currently incorporate relevance feedback into their judgements of how to rank search results returned to a user. In all cases, however, the past systems utilize explicit user feedback, not implicit feedback. That is, they rank files by requiring a user to indicate what items he is interested in, once a set of items is returned by a search. Importantly, in the system according to the invention, the system discerns implicit popularity rahkings based on a ranked set of user actions. The system then then uses those rankings to re-rank subsequent search results. The user actions from which popularity may be determined include, but are not limited to:
whether a file is placed in a projects folder or other work space
whether a file is placed in a shopping cart
whether a file is purchased
In addition, implicit popularity rankings may be derived from one user or set of users and applied to a different user or set of users. For example, if User A places a media file in a projects folder or shopping cart, the information on her activity can also be used to re-rank search results for User B, who is in some way similar to user A. In another example, if users with ".edu" e-mail addresses buy certain things, it makes sense to re-rank search results to favor those things when showing results to other ".edu" users. In the system according to the invention, if registered users who work for advertising agencies have placed certain items in their shopping carts, other advertising agency employees can have their search results re-ranked to favor those items. The opposite can be true as well: the same system can be used to disfavor certain items because they have been sold too many times to advertising agencies, for example.
Retrieval system self-evaluation. The retrieval system according to the invention does not answer any of the TREC tracks — a media file is described by a short paragraph or by keywords. Even though it is possible to develop a unique test collection for purposes of evaluation, it will not necessarily predict the performance of new systems, or even for existing ones.
What is now described is a method for ongoing evaluation of IR system performance based on search results combined with user feedback. This approach enables the system to alert a human system manager about observed degradation in the system performance. As the performance evaluation system becomes more knowledgeable through methods of machine learning, the system is desirablyl able to change its own parameters in order to improve its performance.
'Performance" as used in this inventive system is a score on a scale of user satisfaction with the respect to search results rankings. User satisfaction is determined by tracking the user actions together with the system's data. For example, for a very good performance, the user would submit a search query and purchase the media file that was ranked first. Possible indications of poor performance might include a user issuing several consecutive queries, getting thousands of results, and teπninating the session without further examining any of the media files. Viewer. The software application according to the invention includes a viewer for annotations, or mark-ups, of full motion digital still images, video frames, or other indexed data structures. The viewer, preferably accessible on a personal computer with a Web browser, serves as a helper application for the underlying application of the system according to the invention. The viewer allows users to view and mark up (comment on) media on a separate, see-through, lightweight layer. The viewer supports annotations that are created and rendered so that they appear overlaid on the related media. Position, size, frame number, and other attributes of annotations are saved in a separate annotation file, and when played back the annotations appear at the saved location and with the saved attributes. Preferably, to the greatest extent possible, creating and editing of annotations is be done in WYSIWYG (what you see is what you get) fashion. This is illustrated in Fig. 5.
The annotations may consist of text, graphics, and digitized audio, and are stored separately from the original file information. An indexing scheme relates individual annotations to individual video frames. On full motion video playback, the annotations are displayed on the originally annotated frames. One central part of the system according to die invention is the use of a downloadable, view-only application, which enables users who do not have the full power to create annotations nonetheless to be able to view annotations created by others.
In an exemplary embodiment, files are stored centrally. Preferably the viewer provides security, e.g. SSL, server authentication, and proxy authentication. The underlying application according to the invention powers the full lifecycle of digital media from content collaboration, deep storage, search and retrieval using a natural language search and the ability effectively to move bandwidth-intensive media over the Web. The software is preferably Web-based.
The viewer is a standalone, platform-native "helper" application that is noimally launched by a Web browser, from a specific link on a Web page that is part of the Web server of the system according to the invention. The viewer has the ability to upload annotation data back to the Web server. Except for the two actions of launching and uploading, the viewer does not require any network connection with the Web server or the Web browser. The current viewer in the system according to the invention on a PC or Macintosh platform allows the user to view and annotate various video, still image, audio, and text formats, including MPEG, QuickTime, or RealMedia videos, bitmap, jpeg, and tiff images, MP3, .wav, .aiff, .au, and .snd audio files, and .pdf document files. The viewer currently has nine modes in which it can operate, all of which would be relevant to the mobile user. They are:
annotate a media file select a keyframe select storyboard frames (and optionally a keyframe) set the download (cache) directory send a message and, optionally, attached files to another user download a media file upload media files to a project upload media files to a public collection select a playlist
The current (Mac and Windows) viewers can be launched in any of these nine modes, generally by receiving an XML annotation file.
Attributes common to all annotation types are: position, size, frame number (video only), and page number (multipage document only). All of the graphical attributes such as colors and fonts are modifiable using platform-native dialog boxes. These attributes can be set for new annotations as well as previously created ones.
For video, each annotation is recorded with the timecode of the specific frame on which it was created. When movies with annotations are played back, the viewer stops playback at each frame where an annotation exists. The viewer also contains a drop-down list of all annotated frames (by timecode) so that individual annotated frames can be viewed quickly. Controls are also provided for stepping by a single frame in either direction, as well as a control displaying the timecode of the current frame where the viewer is paused. The PDA (or handheld or wearable) version of the viewer builds on the same general architecture. The server generates a simple XML annotations file, sends it to the client (i.e. the viewer via the Web browser), and handles the returned HTTP messages from the viewer. The following annotation types are desirably supported on a PDA version:
Type Attributes
Freehand drawing (pen) Line widthColor
Transparent text Font name, Font size, Font color
Sticky note Font name, Font size, Font color, Background color Sound Audio data
In this way, the system according to the invention supports the full life cycle of media; allowing organizations to expedite the process of searching, collaborating and distributing of digital assets, such as imagery, video files, pictures and text documents. Users desirably have the freedom to collaborate anywhere at anytime using die annotation viewer according to the invention on a PDA. Users are preferably able to annotate video and still images with a virtual pen, sticky note, text and even voice using a wireless modem Annotations are preferably capable of being added, edited, moved, or deleted.
A PDA-based digital media management solution takes digital delivery and collaboration to the next level by eliminating the need of a traditional PC and Ethernet connection. In addition, users no longer need to carry laptops to stay up to date on work involving video or still images.
Preferably the viewer supports both HTTP authentication and Proxy authentication. As an option, the viewer displays a login dialog where the user can enter a user ID and password to be used for authentication. Since the viewer is activated and closed frequently, it should not prompt the user to log in each time it connects to the server. The viewer instead provides a mechanism for saving the user ID and password on the local workstation (in encrypted form) and passing this information to the server without any prompting. It is desirable that the viewer support SSL encryption for all of its HTTP requests.
25

Claims

We claim:
1. A method for use with first and second metadata streams with respect to a video stream, the first metadata stream time-coded with respect to the video stream and the second metadata stream not time-coded with respect to the video stream, the method comprising the steps of:
aligning the second metadata stream with the first metadata stream;
adding time codes to the second metadata stream, based on the alignment;
searching for proper names within the second metadata stream;
finding faces within the video stream;
matching faces with proper names; and
placing the matched faces and proper names into a reference library.
2. The method of claim 1 wherein the video stream is news footage, wherein the first metadata stream is closed captioning for the hearing impaired, and wherein the second metadata stream is a separate description of the news footage.
3. A method for use with a system storing myriad digital media files, the method comprising the steps of:
generating myriad first thumbnail images, each first thumbnail image corresponding to one of the digital media files, each first thumbnail image having lower resolution and smaller dimensions then the corresponding digital media file, each first thumbnail image having first dimensions;
displaying a subset of the first thumbnail images, the subset comprising more than one and less than all of the myriad first thumbnail images, the subset created as a result of input from a first user;
receiving a configuration command from a second user, the configuration co-mnand indicative of second dimensions differing from the first dimensions;
generating myriad second thumbnail images, each second thumbnail image corresponding to one of the digital media files, each second thumbnail image having lower resolution and smaller dimensions then the corresponding digital media file, each second thumbnail image having second dimensions;
displaying a subset of the second thumbnail images, the subset comprising more than one and less than all of the myriad second thumbnail images, the subset created as a result of input from a third user.
4. The method of claim 3 wherein the first and third users are the same user.
5. The method of claim 3 wherein the first and third users are different users.
6. A method for use with a system storing myriad digital media files, the method comprising the steps of:
generating myriad first thumbnail images, each first thumbnail image corresponding to one of the digital media files, each first thumbnail image having lower resolution and smaller dimensions then the corresponding digital media file, each first thumbnail image having first dimensions;
displaying a subset of the first thumbnail images, the subset comprising more than one and less than all of the myriad first thumbnail images, the subset created as a result of input from a first user; receiving a configuration command from a second user, the configuration command indicative of second resolution differing from the first resolution;
generating myriad second thumbnail images, each second thumbnail image corresponding to one of the digital media files, each second thumbnail image having lower resolution and smaller dimensions then the corresponding digital media file, each second thumbnail image having second resolution;
displaying a subset of the second thumbnail images, the subset comprising more than one and less than all of the myriad second thumbnail images, the subset created as a result of input from a third user.
7. The method of claim 6 wherein the first and third users are the same user.
8. The method of claim 6 wherein the first and third users are different users.
9. A method for use with a system storing digital media records, the system comprising a vocabulary file of words keyed to the digital media records, the method comprising the steps of:
receiving a search query by computer from a user, the search query including words;
logging the search query by computer, yielding a query log;
processing the query log to identify words in the query log which are not words already included in the vocabulary file and which are not words which are variants of words already included in the vocabulary file, the processing performed by computer, said words defined as "not-found words;"
adding the not-found words to the vocabulary file, and keying the not-found words to the digital media records.
10. The method of claim 9 wherein the vocabulary file further comprises a thesaurus and the step of keying the not-found words to the digital media records comprises adding the not- found words to the thesaurus.
11. The method of claim 9 wherein the adding and keying steps are performed manually.
12. Tlie method of claim 9 wherein the adding and keying steps are performed automatically.
13. The method of claim 9 wherein the processing step comprises morphological analysis.
14. The method of claim 9 wherein the processing step comprises spell checking.
15. The method of claim 9 wherein the processing step comprises name identification.
16. A method for use with a system storing digital media records, the system comprising a data file defining rights management questions and multiple-choice answers thereto, the method comprising the steps of:
receiving, from a first user, a selection of rights management questions and respective multiple-choice answers for the data file;
receiving, from a second user non-identical to the first user, a query regarding a first digital media record;
presenting, to the second user, a first rights management question among the selection of rights management questions;
receiving, from the second user, a multiple-choice answer from among the respective multiple-choice answers with respect to the first rights management question, and storing same;
presenting, to the second user, a second rights management question among the selection of rights management questions;
receiving, from the second user, a multiple-choice answer from among the respective multiple-choice answers with respect to the second rights management question, and storing same;
presenting, to the second user, a third rights management question among the selection of rights management questions;
receiving, from the second user, a multiple-choice answer from among the respective multiple-choice answers with respect to the third rights management question, and storing same;
deriving pricing information for the second user with respect to the first digital media record from the stored multiple-choice answers;
receiving, from a third user non-identical to the first user and non-identical to the second user, a query regarding a second digital media record;
presenting, to the third user, a fourth rights management question among the selection of rights management questions;
receiving, from the third user, a multiple-choice answer from among the respective multiple- choice answers with respect to the fourth rights management question, and storing same;
presenting, to the third user, a fifth rights management question among the selection of rights management questions; receiving, from the third user, a multiple-choice answer from among the respective multiple- choice answers with respect to the fifth rights management question, and storing same;
presenting, to the third user, a sixth rights management question among the selection of rights management questions;
receiving, from the third user, a multiple-choice answer from among the respective multiple- choice answers with respect to the sixth rights management question, and storing same;
deriving pricing information for tlie third user with respect to the second digital media record from the stored multiple-choice answers.
17. The method of claim 16 wherein:
the first and fourth rights management questions are the same;
the second and fifth rights management questions are the same; and
the third and sixth rights management questions are the same.
18. A method for use with a system storing digital media records, the system comprising a search engine the operation of which is defined by a plurality of stored parameters, the stored parameters defining, for the search engine, an extent to which the search engine expands search terms, the system coπimunicatively coupled to an internet and providing a hypertext transfer protocol server interface to the internet, the method comprising the steps of:
receiving from a first user, via a hypertext transfer protocol client session, a change to one of the parameters;
storing the change to the one of the stored parameters; receiving from a second user, via a hypertext transfer protocol client session, a search request;
passing the search request to the search engine;
perfoπr-ing, by the search engine, a search determined by the search request and determined by the stored parameters, and deriving search results therefrom;
reporting the search results to the second user.
19. A method for use with a system storing digital media records, the system comprising a search engine searching said digital media records, the method comprising the steps of:
displaying all or part of a first digital media record to a user;
receiving, from the user, first information indicative of a selection of less than all of the displayed portion of the first digital media record;
receiving, from the user, second information indicative of text;
performing, by the search engine, a search of the digital media records with respect to the first information and the second information, and deriving search results therefrom; and
displaying the search results to the user.
20. The method of claim 19 wherein the step of receiving first information comprises the user touching a screen to identify the portion of an image displayed on the screen.
21. The method of claim 19 wherein the step of receiving first information comprises the user drawing on a pad to identify the portion of an image displayed on the screen.
22. The method of claim 19 wherein the step of receiving second information comprises the user typing at a keyboard.
23. The method of claim 19 wherein the step of receiving second information comprises receiving voice information and performing speech recognition upon the voice information.
24. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving search requests from users;
logging the search requests;
expanding the search requests;
applying a statistical clustering algorithm, thereby grouping similar search requests together;
identifying, using a semantic net hierarchy, a lowest-level term in the hierarchy that subsumes all of the queries in a grouping of search requests;
communicating the identified term to a user.
25. The method of claim 24 wherein the expanding is performed using a thesaurus.
26. Tlie method of claim 24 wherein the expanding is performed using a semantic net comprising synonyms and super-terms.
27. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving search requests from users; perfoiming, by the search engine, searches based upon the search requests, yielding respective search results, each search result defining selected digital media records, the digital media records each having associated metadata;
logging the search results and the metadata associated with digital media records selected therein;
expanding the metadata, defining expanded query metadata results;
applying a statistical clustering algorithm, thereby grouping similar expanded query metadata results together;
identifying, using a semantic net hierarchy, a lowest-level term in the hierarchy that subsumes aU of the expanded query metadata results in the grouping of expanded query metadata results;
communicating the identified term to a user.
28. The method of claim 27 wherein the expanding is performed using a thesaurus.
29. The method of claim 27 wherein the expanding is performed using a semantic net comprising synonyms and super-terms.
30. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
performing, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records; logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users selecting particular ones of the selected digital media records;
receiving a second search request from a second user;
performing, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
rar- ng the second selected digital media records downward for at least one second selected digital media record that was selected by at least one first user.
31. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
perforating, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users selecting particular ones of the selected digital media records;
receiving a second search request from a second user;
performing, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
ranking the second selected digital media records downward for at least one second selected digital media record that was selected by at least a predeterrnined number of first users.
32. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
performing, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, ormation indicative of subsequent actions by the first users selecting particular ones of the selected digital media records;
receiving a second search request from a second user;
perfoiming, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
ranking the second selected digital media records downward for at least one second selected digital media record that was selected at least a predetermined number of times by first users.
33. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of: receiving first search requests from a plurality of first users;
perforating, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
displaying said first selected digital media records presented in an order, said order defining a first-presented first selected digital media record;
logging the event of a first user purchasing said first-presented first selected digital media record;
reporting, to a second user, the frequency of occurrence of the logged events.
34. A method for use with a system storing digital media records for purchase and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
perfoiming, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
displaying said first selected digital media records presented in an order, said order defining a first-presented first selected digital media record;
logging the event of a first user issuing several consecutive search requests, and receiving respective first selected digital media requests greater in number than a predetermined threshold for said several consecutive search requests; reporting, to a second user, the frequency of occurrence of the logged events.
35. The method of claim 34 wherein the logged events further comprise the first user failing purchase any of the stored digital media records.
36. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving search requests from users;
performing, by the search engine, searches based upon the search requests, yielding respective search results, each search result defining selected digital media records or being empty;
logging the search requests for which the search result is empty;
expanding the logged search requests;
applying a statistical clustering algorithm to the logged search requests, thereby grouping similar search requests together;
communicating a group of search requests to a user.
37. The method of claim 36 wherein the expanding is performed using a thesaurus.
38. The method of claim 36 wherein the expanding is performed using a semantic net comprising synonyms and super-terms.
39. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the system permitting user expressions of interest in particular stored digital media records, the method comprising the steps of: receiving search requests from users;
perfoiming, by the search engine, searches based upon the search requests, yielding respective search results, each search result defining selected digital media records;
logging the search requests for which a user has expressed interest in a selected digital media record;
expanding the logged search requests;
applying a statistical clustering algorithm to the logged search requests, thereby grouping similar search requests together;
communicating a group of search requests to a user.
40. The method of claim 39 wherein the expanding is performed using a thesaurus.
41. The method of claim 39 wherein the expanding is performed using a semantic net comprising synonyms and super-terms.
42. The method of claim 39 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online shopping cart.
43. The method of claim 39 wherein a user expressing interest in a selected digital media record comprises the user purchasing rights to use the selected digital media record.
44. The method of claim 39 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online projects folder or other work space.
45. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
performing, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users expressing interest in particular ones of the selected digital media records;
receiving a second search request from a second user;
performing, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
ranking the second selected digital media records upward for at least one second selected digital media record that was for which at least one first user expressed interest.
46. The method of claim 45 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online shopping cart.
47. The method of claim 45 wherein a user expressing interest in a selected digital media record comprises the user purchasing rights to use the selected digital media record.
48. The method of claim 45 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online projects folder or other work space.
49. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
performing, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users expressing interest in particular ones of the selected digital media records;
receiving a second search request from a second user;
performing, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
ranking the second selected digital media records upward for at least one second selected digital media record that was for which at least a predeteimined number of first users expressed interest.
50. The method of claim 49 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online shopping cart.
51. The method of claim 49 wherein a user expressing interest in a selected digital media record comprises the user purchasing rights to use the selected digital media record.
52. The method of claim 49 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online projects folder or other work space.
53. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
performing, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users expressing interest in particular ones of the selected digital media records;
receiving a second search request from a second user;
performing, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records;
r--nking the second selected digital media records upward for at least one second selected digital media record for which first users expressed interest at least a predetermined number of times.
54. The method of claim 53 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online shopping cart.
55. The method of claim 53 wherein a user expressing interest in a selected digital media record comprises the user purchasing rights to use the selected digital media record.
56. The method of claim 53 wherein a user expressing interest in a selected digital media record comprises the user placing the selected digital media record into an online projects folder or other work space.
57. A method for use with a system storing digital media records and comprising a search engine searching said stored digital media records, the method comprising the steps of:
receiving first search requests from a plurality of first users;
perfoiming, by the search engine, searches based upon the first search requests, yielding respective first search results, each first search result defining first selected digital media records;
logging the first search results;
receiving, from the first users, information indicative of subsequent actions by the first users expressing interest in particular ones of the selected digital media records;
receiving a second search request from a second user;
perfoiming, by the search engine, a search based upon the second search request, yielding respective second search results, the second search results defining second selected digital media records; ranking the second selected digital media records upward for at least one second selected digital media record for which expression of interest satisfies a predetermined threshold, wherein expression of interest comprises a weighted function of two or more of the following events: the user placing the selected digital media record into an online shopping cart, the user purchasing rights to use the selected digital media record, and the user placing the selected digital media record into an online projects folder or other work space.
58. A method for use with a metadata stream, with respect to a video stream, the metadata stream time-coded with respect to the video stream, the method comprising the steps, performed by a computer, of:
searching for proper names within the metadata stream;
finding faces within the video stream;
matcliing faces with proper names; and
placing the matched faces and proper names into a reference library.
59. The method of claim 58 wherein the video stream is news footage, wherein the metadata stream is closed captioning for the hearing impaired.
60. A method for use with a metadata stream with respect to at least one video stream, the metadata stream time-coded with respect to the at least one video stream, the method comprising the steps, performed by a computer, of:
searching for proper names within the metadata stream;
finding faces within the at least one video stream;
matching faces with proper names; placing the matched faces and proper names into a reference library; and
for a selected proper name, generating a list of locations within the at least one video stream associated with that selected proper name.
61. The method of claim 60 wherein the video stream is news footage, wherein the metadata stream is closed captioning for the hearing impaired.
62. The method of claim 60 further comprising the step of communicating the list of locations to a rights management system.
63. A method for use with a system for managing digital media files, the method comprising the steps of:
using face recognition to recognize faces portrayed in the digital media files, yielding metadata with respect to the digital media files indicative of the recognized faces;
analyzing the metadata indicative of the recognized faces to detect duplicate files among the digital media files; and
displaying the duplicate files for a user.
64. A method for use with a system for managing digital media files, the method comprising the steps of:
using image recognition to recognize images portrayed in the digital media files, yielding metadata with respect to the digital media files indicative of the recognized images;
analyzing the metadata indicative of the recognized images to detect duplicate files among the digital media files; and displaying the duplicate files for a user.
65. A method for use with a system for managing digital media files, the method comprising the steps of:
using scene detection to detect scenes portrayed in the digital media files, yielding metadata with respect to the digital media files indicative of the detected scenes;
analyzing the metadata indicative of the detecting scenes to detect duplicate files among the digital media files; and
displaying the duplicate files for a user.
PCT/US2001/026841 2000-08-28 2001-08-28 Method and apparatus for digital media management, retrieval, and collaboration WO2002019147A1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
AU2001288469A AU2001288469A1 (en) 2000-08-28 2001-08-28 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,411 US6941294B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,410 US7177879B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,412 US6895407B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,409 US6922691B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,413 US6944611B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration
US10/063,414 US7418444B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US22883700P 2000-08-28 2000-08-28
US60/228,837 2000-08-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US10/063,409 Continuation US6922691B2 (en) 2000-08-28 2002-04-21 Method and apparatus for digital media management, retrieval, and collaboration

Publications (1)

Publication Number Publication Date
WO2002019147A1 true WO2002019147A1 (en) 2002-03-07

Family

ID=22858744

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/026841 WO2002019147A1 (en) 2000-08-28 2001-08-28 Method and apparatus for digital media management, retrieval, and collaboration

Country Status (3)

Country Link
US (5) US7418444B2 (en)
AU (1) AU2001288469A1 (en)
WO (1) WO2002019147A1 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6895407B2 (en) 2000-08-28 2005-05-17 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6941294B2 (en) 2000-08-28 2005-09-06 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
GB2424091A (en) * 2005-03-11 2006-09-13 Alamy Ltd Ranking of images in the results of a search
US7240049B2 (en) * 2003-11-12 2007-07-03 Yahoo! Inc. Systems and methods for search query processing using trend analysis
US7428533B2 (en) 2004-12-06 2008-09-23 Yahoo! Inc. Automatic generation of taxonomies for categorizing queries and search query processing using taxonomies
US7620628B2 (en) 2004-12-06 2009-11-17 Yahoo! Inc. Search processing with automatic categorization of queries
US7933765B2 (en) 2007-01-25 2011-04-26 Corbis Corporation Cross-lingual information retrieval
US7937403B2 (en) 2006-10-30 2011-05-03 Yahoo! Inc. Time-based analysis of related keyword searching
US7953504B2 (en) 2004-05-14 2011-05-31 Synaptics Incorporated Method and apparatus for selecting an audio track based upon audio excerpts
US8073828B2 (en) 2007-06-14 2011-12-06 Curbis Corporation Licensed rights clearance and tracking for digital assets
US8219494B1 (en) 2007-08-16 2012-07-10 Corbis Corporation End-to-end licensing of digital media assets
US8341195B1 (en) 2007-10-04 2012-12-25 Corbis Corporation Platform for managing media assets for multi-model licensing over multi-level pricing and asset grouping

Families Citing this family (295)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100331827B1 (en) * 1999-08-24 2002-04-09 구자홍 Method for realizing reservation guidance/reservation conviction screen of digital TV
JP2002202975A (en) * 2000-11-02 2002-07-19 Canon Inc Data retrieval device and method
US7027987B1 (en) * 2001-02-07 2006-04-11 Google Inc. Voice interface for a search engine
US7231381B2 (en) * 2001-03-13 2007-06-12 Microsoft Corporation Media content search engine incorporating text content and user log mining
US6748398B2 (en) * 2001-03-30 2004-06-08 Microsoft Corporation Relevance maximizing, iteration minimizing, relevance-feedback, content-based image retrieval (CBIR)
GB2375192B (en) * 2001-04-27 2003-04-16 Premier Systems Technology Ltd Search engine systems
US7149732B2 (en) * 2001-10-12 2006-12-12 Microsoft Corporation Clustering web queries
US7130864B2 (en) * 2001-10-31 2006-10-31 Hewlett-Packard Development Company, L.P. Method and system for accessing a collection of images in a database
US7283992B2 (en) * 2001-11-30 2007-10-16 Microsoft Corporation Media agent to suggest contextually related media content
US7231343B1 (en) * 2001-12-20 2007-06-12 Ianywhere Solutions, Inc. Synonyms mechanism for natural language systems
US6795903B2 (en) * 2002-01-17 2004-09-21 Thomas Licensing S.A. System and method for searching for duplicate data
US7567953B2 (en) 2002-03-01 2009-07-28 Business Objects Americas System and method for retrieving and organizing information from disparate computer network information sources
US20030182139A1 (en) * 2002-03-22 2003-09-25 Microsoft Corporation Storage, retrieval, and display of contextual art with digital media files
WO2003083620A2 (en) * 2002-03-29 2003-10-09 Digital Generation Systems, Inc. System for cataloging of audio video and print media on a server
US20070245247A1 (en) * 2002-05-14 2007-10-18 Kaleidescape, Inc. Grid-like guided user interface for video selection and display
CN1628452B (en) * 2002-05-17 2010-09-01 株式会社Ntt都科摩 De-fragmentation of transmission sequences
US20030236750A1 (en) * 2002-06-19 2003-12-25 Suermondt Henri Jacques Customized information access
US20040006628A1 (en) * 2002-07-03 2004-01-08 Scott Shepard Systems and methods for providing real-time alerting
US7801838B2 (en) * 2002-07-03 2010-09-21 Ramp Holdings, Inc. Multimedia recognition system comprising a plurality of indexers configured to receive and analyze multimedia data based on training data and user augmentation relating to one or more of a plurality of generated documents
US20060095450A1 (en) * 2002-09-27 2006-05-04 Millikan Thomas N Use of a metadata presort file to sort compressed audio files
US20040204939A1 (en) * 2002-10-17 2004-10-14 Daben Liu Systems and methods for speaker change detection
CA2409114A1 (en) * 2002-10-22 2004-04-22 N-Liter Inc. Method for information retrieval
US7120626B2 (en) * 2002-11-15 2006-10-10 Koninklijke Philips Electronics N.V. Content retrieval based on semantic association
KR100511785B1 (en) * 2002-12-20 2005-08-31 한국전자통신연구원 A System and A Method for Authoring Multimedia Content Description Metadata
US7082572B2 (en) 2002-12-30 2006-07-25 The Board Of Trustees Of The Leland Stanford Junior University Methods and apparatus for interactive map-based analysis of digital video content
US7823058B2 (en) * 2002-12-30 2010-10-26 The Board Of Trustees Of The Leland Stanford Junior University Methods and apparatus for interactive point-of-view authoring of digital video content
US8307273B2 (en) * 2002-12-30 2012-11-06 The Board Of Trustees Of The Leland Stanford Junior University Methods and apparatus for interactive network sharing of digital video content
US7167874B2 (en) * 2003-01-22 2007-01-23 International Business Machines Corporation System and method for command line administration of project spaces using XML objects
US7287025B2 (en) * 2003-02-12 2007-10-23 Microsoft Corporation Systems and methods for query expansion
US7823077B2 (en) 2003-03-24 2010-10-26 Microsoft Corporation System and method for user modification of metadata in a shell browser
US7421438B2 (en) 2004-04-29 2008-09-02 Microsoft Corporation Metadata editing control
US7712034B2 (en) 2003-03-24 2010-05-04 Microsoft Corporation System and method for shell browser
US7769794B2 (en) 2003-03-24 2010-08-03 Microsoft Corporation User interface for a file system shell
US7240292B2 (en) 2003-04-17 2007-07-03 Microsoft Corporation Virtual address bar user interface control
US7627552B2 (en) 2003-03-27 2009-12-01 Microsoft Corporation System and method for filtering and organizing items based on common elements
US7925682B2 (en) 2003-03-27 2011-04-12 Microsoft Corporation System and method utilizing virtual folders
US7650575B2 (en) 2003-03-27 2010-01-19 Microsoft Corporation Rich drag drop user interface
US20040196306A1 (en) * 2003-04-04 2004-10-07 Microsoft Corporation Method and system for flagging digital ink note content
JP2004336343A (en) * 2003-05-07 2004-11-25 Canon Inc Image processing system
KR20060020624A (en) * 2003-05-12 2006-03-06 코닌클리케 필립스 일렉트로닉스 엔.브이. Method of searching for media objects
US7702668B2 (en) * 2003-06-16 2010-04-20 Microsoft Corporation Asset composition
US20050010859A1 (en) * 2003-07-09 2005-01-13 Mcdonough Carol P. System for processing documents and associated ancillary information
US7096230B2 (en) * 2003-08-01 2006-08-22 Sap Aktiengesellschaft Computer-implemented method and system to support in developing a process specification for a collaborative process
EP1503311A1 (en) * 2003-08-01 2005-02-02 Sap Ag Computer-implemented method and system to support in developing a process specification for a collaborative process
US20050038861A1 (en) * 2003-08-14 2005-02-17 Scott Lynn Method and system for dynamically generating electronic communications
US7756750B2 (en) 2003-09-02 2010-07-13 Vinimaya, Inc. Method and system for providing online procurement between a buyer and suppliers over a network
US7505964B2 (en) 2003-09-12 2009-03-17 Google Inc. Methods and systems for improving a search ranking using related queries
US8024335B2 (en) 2004-05-03 2011-09-20 Microsoft Corporation System and method for dynamically generating a selectable search extension
US8055907B2 (en) * 2003-10-24 2011-11-08 Microsoft Corporation Programming interface for a computer platform
FI20031928A0 (en) * 2003-12-31 2003-12-31 Nokia Corp Method of combining media documents in a terminal of a cellular network and terminal of a cellular network
US7657846B2 (en) 2004-04-23 2010-02-02 Microsoft Corporation System and method for displaying stack icons
US7694236B2 (en) 2004-04-23 2010-04-06 Microsoft Corporation Stack icons representing multiple objects
EP1591916B1 (en) * 2004-04-26 2013-11-06 Sap Ag Method, computer program and device for deleting data sets contained in a table system
US8707209B2 (en) 2004-04-29 2014-04-22 Microsoft Corporation Save preview representation of files being created
US20050246324A1 (en) * 2004-04-30 2005-11-03 Nokia Inc. System and associated device, method, and computer program product for performing metadata-based searches
KR100561435B1 (en) * 2004-05-07 2006-03-17 삼성전자주식회사 Method and apparatus for synchronizing between metadata and storing media thereof
JP4429081B2 (en) * 2004-06-01 2010-03-10 キヤノン株式会社 Information processing apparatus and information processing method
BE1016079A6 (en) * 2004-06-17 2006-02-07 Vartec Nv METHOD FOR INDEXING AND RECOVERING DOCUMENTS, COMPUTER PROGRAM THAT IS APPLIED AND INFORMATION CARRIER PROVIDED WITH THE ABOVE COMPUTER PROGRAM.
US7100135B2 (en) * 2004-06-18 2006-08-29 Intel Corporation Method and system to evaluate signal line spacing
US20070016559A1 (en) * 2005-07-14 2007-01-18 Yahoo! Inc. User entertainment and engagement enhancements to search system
US8249929B2 (en) * 2004-08-11 2012-08-21 Adknowledge, Inc. Method and system for generating and distributing electronic communications for maximum revenue
US8429190B2 (en) 2004-08-11 2013-04-23 Adknowledge, Inc. Method and system for generating and distributing electronic communications
US8370323B2 (en) 2004-08-31 2013-02-05 Intel Corporation Providing information services related to multimodal inputs
US20070005490A1 (en) * 2004-08-31 2007-01-04 Gopalakrishnan Kumar C Methods and System for Distributed E-commerce
US7873911B2 (en) * 2004-08-31 2011-01-18 Gopalakrishnan Kumar C Methods for providing information services related to visual imagery
US20060230073A1 (en) * 2004-08-31 2006-10-12 Gopalakrishnan Kumar C Information Services for Real World Augmentation
US20060047704A1 (en) * 2004-08-31 2006-03-02 Kumar Chitra Gopalakrishnan Method and system for providing information services relevant to visual imagery
US7363305B2 (en) * 2004-09-02 2008-04-22 Microsoft Corporation Centralized terminology and glossary management
US7617232B2 (en) * 2004-09-02 2009-11-10 Microsoft Corporation Centralized terminology and glossary development
EP1640879A1 (en) * 2004-09-28 2006-03-29 Swisscom Mobile AG Procedure and means for the query of recording-data in a database
US8051096B1 (en) * 2004-09-30 2011-11-01 Google Inc. Methods and systems for augmenting a token lexicon
US8443279B1 (en) * 2004-10-13 2013-05-14 Stryker Corporation Voice-responsive annotation of video generated by an endoscopic camera
US7925669B2 (en) * 2004-10-13 2011-04-12 Sony Corporation Method and apparatus for audio/video attribute and relationship storage and retrieval for efficient composition
US20060100977A1 (en) * 2004-10-26 2006-05-11 Ya-Ling Hsu System and method for using embedded supplemental information
US7542969B1 (en) * 2004-11-03 2009-06-02 Microsoft Corporation Domain knowledge-assisted information processing
US7716209B1 (en) 2004-11-03 2010-05-11 Microsoft Corporation Automated advertisement publisher identification and selection
US8335753B2 (en) * 2004-11-03 2012-12-18 Microsoft Corporation Domain knowledge-assisted information processing
US7912827B2 (en) * 2004-12-02 2011-03-22 At&T Intellectual Property Ii, L.P. System and method for searching text-based media content
US8538970B1 (en) * 2004-12-30 2013-09-17 Google Inc. Personalizing search results
US7660416B1 (en) * 2005-01-11 2010-02-09 Sample Digital Holdings Llc System and method for media content collaboration throughout a media production process
US20080126418A1 (en) * 2005-02-11 2008-05-29 Koninklijke Philips Electronics, N.V. Method and Apparatus to Store Content and a Query on a Device
US20060200461A1 (en) * 2005-03-01 2006-09-07 Lucas Marshall D Process for identifying weighted contextural relationships between unrelated documents
US7958085B1 (en) * 2005-03-07 2011-06-07 Adobe Systems Incorporated Managing media-content licenses, including option formation
US20060218187A1 (en) * 2005-03-25 2006-09-28 Microsoft Corporation Methods, systems, and computer-readable media for generating an ordered list of one or more media items
US20080120312A1 (en) * 2005-04-07 2008-05-22 Iofy Corporation System and Method for Creating a New Title that Incorporates a Preexisting Title
US8732175B2 (en) 2005-04-21 2014-05-20 Yahoo! Inc. Interestingness ranking of media objects
US10210159B2 (en) * 2005-04-21 2019-02-19 Oath Inc. Media object metadata association and ranking
US8195646B2 (en) 2005-04-22 2012-06-05 Microsoft Corporation Systems, methods, and user interfaces for storing, searching, navigating, and retrieving electronic information
US20060253421A1 (en) * 2005-05-06 2006-11-09 Fang Chen Method and product for searching title metadata based on user preferences
US20060294128A1 (en) * 2005-05-21 2006-12-28 Kula Media Group Enhanced methods for media processing and distribution
US7305128B2 (en) * 2005-05-27 2007-12-04 Mavs Lab, Inc. Anchor person detection for television news segmentation based on audiovisual features
US10445359B2 (en) * 2005-06-07 2019-10-15 Getty Images, Inc. Method and system for classifying media content
US20070112839A1 (en) * 2005-06-07 2007-05-17 Anna Bjarnestam Method and system for expansion of structured keyword vocabulary
US20060287994A1 (en) * 2005-06-15 2006-12-21 George David A Method and apparatus for creating searches in peer-to-peer networks
JP4678246B2 (en) * 2005-06-20 2011-04-27 船井電機株式会社 Program search support system
US7665028B2 (en) 2005-07-13 2010-02-16 Microsoft Corporation Rich drag drop user interface
US8121895B2 (en) 2005-07-21 2012-02-21 Adknowledge, Inc. Method and system for delivering electronic communications
US8156114B2 (en) * 2005-08-26 2012-04-10 At&T Intellectual Property Ii, L.P. System and method for searching and analyzing media content
US8677377B2 (en) 2005-09-08 2014-03-18 Apple Inc. Method and apparatus for building an intelligent automated assistant
US8176101B2 (en) * 2006-02-07 2012-05-08 Google Inc. Collaborative rejection of media for physical establishments
US20070073708A1 (en) * 2005-09-28 2007-03-29 Smith Adam D Generation of topical subjects from alert search terms
JP4542013B2 (en) * 2005-09-29 2010-09-08 富士フイルム株式会社 Print order system, program, program storage medium, and print order server
US20070078883A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Using location tags to render tagged portions of media files
US7412534B2 (en) * 2005-09-30 2008-08-12 Yahoo! Inc. Subscription control panel
US20070078876A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Generating a stream of media data containing portions of media files using location tags
US20070078712A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Systems for inserting advertisements into a podcast
US20070079321A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Picture tagging
US20070078714A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Automatically matching advertisements to media files
US20070078897A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Filemarking pre-existing media files using location tags
US8108378B2 (en) * 2005-09-30 2012-01-31 Yahoo! Inc. Podcast search engine
US20070078896A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Identifying portions within media files with location tags
US20070077921A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Pushing podcasts to mobile devices
US20070078898A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Server-based system and method for retrieving tagged portions of media files
US20070078832A1 (en) * 2005-09-30 2007-04-05 Yahoo! Inc. Method and system for using smart tags and a recommendation engine using smart tags
US20070106627A1 (en) * 2005-10-05 2007-05-10 Mohit Srivastava Social discovery systems and methods
US20070083674A1 (en) * 2005-10-07 2007-04-12 Sony Ericsson Mobile Communications Ab Method and apparatus for republishing electronic content from a wireless communication device
US8209222B2 (en) 2005-10-12 2012-06-26 Adknowledge, Inc. Method and system for encrypting data delivered over a network
US20070168487A1 (en) * 2005-12-02 2007-07-19 Dufour Remi Computer system, method and software for managing, evaluating, classifying and storing on a server a set of productivity and games software files under server client control for delivery to third parties
US20070162876A1 (en) * 2005-12-29 2007-07-12 Quirk Timothy B Contextual album browsing
JP2007214664A (en) * 2006-02-07 2007-08-23 Funai Electric Co Ltd Hard disk recorder
WO2007105202A2 (en) * 2006-03-10 2007-09-20 Avraham Shpigel Automatic reusable definitions identification (rdi) method
US7668869B2 (en) * 2006-04-03 2010-02-23 Digitalsmiths Corporation Media access system
US7822318B2 (en) * 2006-04-26 2010-10-26 Sony Ericsson Mobile Communications Ab Smart random media object playback
US8856105B2 (en) * 2006-04-28 2014-10-07 Hewlett-Packard Development Company, L.P. Dynamic data navigation
US8041156B2 (en) * 2006-05-31 2011-10-18 Agfa Inc Single-frame and multi-frame image data conversion system and method
US8122341B1 (en) 2006-06-17 2012-02-21 Google Inc. Sharing geographical information between users
US20080004947A1 (en) * 2006-06-28 2008-01-03 Microsoft Corporation Online keyword buying, advertisement and marketing
US9318108B2 (en) 2010-01-18 2016-04-19 Apple Inc. Intelligent automated assistant
US8599251B2 (en) * 2006-09-14 2013-12-03 Olympus Imaging Corp. Camera
US8527469B2 (en) * 2006-10-13 2013-09-03 Sony Corporation System and method for automatic detection of duplicate digital photos
US7634467B2 (en) * 2006-10-31 2009-12-15 Microsoft Corporation Implicit, specialized search of business objects using unstructured text
US8661029B1 (en) 2006-11-02 2014-02-25 Google Inc. Modifying search result ranking based on implicit user feedback
US8108371B2 (en) * 2006-11-30 2012-01-31 Microsoft Corporation Web engine search preview
US20090157631A1 (en) * 2006-12-14 2009-06-18 Jason Coleman Database search enhancements
US8938463B1 (en) 2007-03-12 2015-01-20 Google Inc. Modifying search result ranking based on implicit user feedback and a model of presentation bias
US8694374B1 (en) 2007-03-14 2014-04-08 Google Inc. Detecting click spam
US9092510B1 (en) 2007-04-30 2015-07-28 Google Inc. Modifying search result ranking based on a temporal element of user feedback
US7983915B2 (en) * 2007-04-30 2011-07-19 Sonic Foundry, Inc. Audio content search engine
US20080270344A1 (en) * 2007-04-30 2008-10-30 Yurick Steven J Rich media content search engine
US20080270110A1 (en) * 2007-04-30 2008-10-30 Yurick Steven J Automatic speech recognition with textual content input
US8117185B2 (en) * 2007-06-26 2012-02-14 Intertrust Technologies Corporation Media discovery and playlist generation
US20090012934A1 (en) * 2007-07-03 2009-01-08 Corbis Corporation Searching for rights limited media
KR20090002810A (en) * 2007-07-04 2009-01-09 삼성전자주식회사 Method for storing the broadcast on a data broadcast and a imaging apparatus having the same
US8781996B2 (en) 2007-07-12 2014-07-15 At&T Intellectual Property Ii, L.P. Systems, methods and computer program products for searching within movies (SWiM)
US8694511B1 (en) 2007-08-20 2014-04-08 Google Inc. Modifying search result ranking based on populations
US20110004671A1 (en) * 2007-09-07 2011-01-06 Ryan Steelberg System and Method for Secure Delivery of Creatives
US9886814B2 (en) * 2007-09-07 2018-02-06 Veritone, Inc. System and method for secure sharing of creatives
US20100131085A1 (en) * 2007-09-07 2010-05-27 Ryan Steelberg System and method for on-demand delivery of audio content for use with entertainment creatives
US9959700B2 (en) * 2007-09-07 2018-05-01 Veritone, Inc. System and method for secured delivery of creatives
US20090094113A1 (en) * 2007-09-07 2009-04-09 Digitalsmiths Corporation Systems and Methods For Using Video Metadata to Associate Advertisements Therewith
US8086620B2 (en) 2007-09-12 2011-12-27 Ebay Inc. Inference of query relationships
US10296936B1 (en) 2007-09-26 2019-05-21 Videomining Corporation Method and system for measuring effectiveness of a marketing campaign on digital signage
US8364020B2 (en) * 2007-09-28 2013-01-29 Motorola Mobility Llc Solution for capturing and presenting user-created textual annotations synchronously while playing a video recording
US7870130B2 (en) * 2007-10-05 2011-01-11 International Business Machines Corporation Techniques for identifying a matching search term in an image of an electronic document
US8380045B2 (en) * 2007-10-09 2013-02-19 Matthew G. BERRY Systems and methods for robust video signature with area augmented matching
US8909655B1 (en) 2007-10-11 2014-12-09 Google Inc. Time based ranking
US8862582B2 (en) * 2007-11-15 2014-10-14 At&T Intellectual Property I, L.P. System and method of organizing images
US8170280B2 (en) * 2007-12-03 2012-05-01 Digital Smiths, Inc. Integrated systems and methods for video-based object modeling, recognition, and tracking
JP4524702B2 (en) * 2008-02-08 2010-08-18 コニカミノルタビジネステクノロジーズ株式会社 Data management apparatus, search condition information management method, and computer program
US8311344B2 (en) * 2008-02-15 2012-11-13 Digitalsmiths, Inc. Systems and methods for semantically classifying shots in video
US8239889B2 (en) 2008-03-10 2012-08-07 Hulu, LLC Method and apparatus for collecting viewer survey data and for providing compensation for same
US8578408B2 (en) * 2008-03-10 2013-11-05 Hulu, LLC Method and apparatus for providing directed advertising based on user preferences
US9690786B2 (en) * 2008-03-17 2017-06-27 Tivo Solutions Inc. Systems and methods for dynamically creating hyperlinks associated with relevant multimedia content
US8996376B2 (en) 2008-04-05 2015-03-31 Apple Inc. Intelligent text-to-speech conversion
US9305026B2 (en) 2008-04-30 2016-04-05 Beyondvia Technologies Systems and methods for digital images utilizing an infinite cylinder user interface
US8769423B2 (en) 2008-04-30 2014-07-01 Beyondvia Technologies Content distribution systems and methods using digital images
US9310980B2 (en) 2012-08-21 2016-04-12 Beyondvia Technologies Systems and methods for performance comparisons utilizing an infinite cylinder user interface
US8074176B2 (en) * 2008-04-30 2011-12-06 Beyondvia Technologies, Llc Electronic communications dialog using sequenced digital images stored in an image dictionary
US8806353B2 (en) * 2008-04-30 2014-08-12 Beyondvia Technologies Mobile electronic communications dialog using sequenced digital images
US10304126B2 (en) 2008-04-30 2019-05-28 Beyondvia Technologies Visual communication systems and methods designing and building entire experiences
US8311390B2 (en) * 2008-05-14 2012-11-13 Digitalsmiths, Inc. Systems and methods for identifying pre-inserted and/or potential advertisement breaks in a video sequence
US8001101B2 (en) * 2008-06-23 2011-08-16 Microsoft Corporation Presenting instant answers to internet queries
US20100042610A1 (en) * 2008-08-15 2010-02-18 Microsoft Corporation Rank documents based on popularity of key metadata
JP5418051B2 (en) * 2008-09-08 2014-02-19 株式会社リコー Workflow management system
US20100125496A1 (en) * 2008-11-19 2010-05-20 Wong Victor K System and method for automated generation of advertising
US8396865B1 (en) 2008-12-10 2013-03-12 Google Inc. Sharing search engine relevance data between corpora
US20100162303A1 (en) * 2008-12-23 2010-06-24 Cassanova Jeffrey P System and method for selecting an object in a video data stream
US7962487B2 (en) * 2008-12-29 2011-06-14 Microsoft Corporation Ranking oriented query clustering and applications
US8108393B2 (en) 2009-01-09 2012-01-31 Hulu Llc Method and apparatus for searching media program databases
US9569770B1 (en) 2009-01-13 2017-02-14 Amazon Technologies, Inc. Generating constructed phrases
US9607324B1 (en) 2009-01-23 2017-03-28 Zakta, LLC Topical trust network
US10007729B1 (en) 2009-01-23 2018-06-26 Zakta, LLC Collaboratively finding, organizing and/or accessing information
US10191982B1 (en) 2009-01-23 2019-01-29 Zakata, LLC Topical search portal
US8527537B2 (en) 2009-02-19 2013-09-03 Hulu, LLC Method and apparatus for providing community-based metadata
US8782709B2 (en) * 2009-02-19 2014-07-15 Hulu, LLC Method and apparatus for providing a program guide having search parameter aware thumbnails
US20100235354A1 (en) * 2009-03-12 2010-09-16 International Business Machines Corporation Collaborative search engine system
US9009146B1 (en) 2009-04-08 2015-04-14 Google Inc. Ranking search results based on similar queries
US20100303440A1 (en) * 2009-05-27 2010-12-02 Hulu Llc Method and apparatus for simultaneously playing a media program and an arbitrarily chosen seek preview frame
US10241752B2 (en) 2011-09-30 2019-03-26 Apple Inc. Interface for a virtual digital assistant
US10241644B2 (en) 2011-06-03 2019-03-26 Apple Inc. Actionable reminder entries
US8799253B2 (en) * 2009-06-26 2014-08-05 Microsoft Corporation Presenting an assembled sequence of preview videos
US9431006B2 (en) 2009-07-02 2016-08-30 Apple Inc. Methods and apparatuses for automatic speech recognition
US8447760B1 (en) 2009-07-20 2013-05-21 Google Inc. Generating a related set of documents for an initial set of documents
US9298700B1 (en) * 2009-07-28 2016-03-29 Amazon Technologies, Inc. Determining similar phrases
US8855460B2 (en) 2009-07-31 2014-10-07 Citizenglobal, Inc. Systems and methods for delivering and exporting edited media in multiple formats
US10007712B1 (en) 2009-08-20 2018-06-26 Amazon Technologies, Inc. Enforcing user-specified rules
EP2341450A1 (en) 2009-08-21 2011-07-06 Mikko Kalervo Väänänen Method and means for data searching and language translation
US8498974B1 (en) 2009-08-31 2013-07-30 Google Inc. Refining search results
US8972391B1 (en) 2009-10-02 2015-03-03 Google Inc. Recent interest based relevance scoring
US8659662B2 (en) * 2009-10-14 2014-02-25 Harris Corporation Surveillance system with target based scrolling and related methods
US8587651B2 (en) * 2009-10-14 2013-11-19 Harris Corporation Surveillance system for transcoding surveillance image files while retaining image acquisition time metadata and associated methods
US8477188B2 (en) * 2009-10-14 2013-07-02 Harris Corporation Surveillance system for transcoding surveillance image files while retaining geospatial metadata and associated methods
US8874555B1 (en) 2009-11-20 2014-10-28 Google Inc. Modifying scoring data based on historical changes
US20110125764A1 (en) * 2009-11-26 2011-05-26 International Business Machines Corporation Method and system for improved query expansion in faceted search
US8332530B2 (en) * 2009-12-10 2012-12-11 Hulu Llc User interface including concurrent display of video program, histogram, and transcript
US8806341B2 (en) * 2009-12-10 2014-08-12 Hulu, LLC Method and apparatus for navigating a media program via a histogram of popular segments
US8577887B2 (en) * 2009-12-16 2013-11-05 Hewlett-Packard Development Company, L.P. Content grouping systems and methods
US8244707B2 (en) * 2010-01-15 2012-08-14 Hulu Llc Method and apparatus for providing supplemental video content for third party websites
US9256899B2 (en) * 2010-01-15 2016-02-09 Dell Products, L.P. System and method for separation of software purchase from fulfillment
US10387927B2 (en) 2010-01-15 2019-08-20 Dell Products L.P. System and method for entitling digital assets
US8301596B2 (en) 2010-01-15 2012-10-30 Hulu Llc Method and apparatus for providing supplemental video content for third party websites
US9235399B2 (en) * 2010-01-15 2016-01-12 Dell Products L.P. System and method for manufacturing and personalizing computing devices
US8615514B1 (en) 2010-02-03 2013-12-24 Google Inc. Evaluating website properties by partitioning user feedback
US8682667B2 (en) 2010-02-25 2014-03-25 Apple Inc. User profiling for selecting user specific voice input processing information
US8799658B1 (en) 2010-03-02 2014-08-05 Amazon Technologies, Inc. Sharing media items with pass phrases
US8924379B1 (en) 2010-03-05 2014-12-30 Google Inc. Temporal-based score adjustments
US8959093B1 (en) 2010-03-15 2015-02-17 Google Inc. Ranking search results based on anchors
US8170783B2 (en) 2010-03-16 2012-05-01 Dell Products L.P. System and method for handling software activation in entitlement
US20110279224A1 (en) * 2010-05-14 2011-11-17 Koreafirstec Co., Ltd. Remote control method and apparatus using smartphone
US9623119B1 (en) 2010-06-29 2017-04-18 Google Inc. Accentuating search results
US8832083B1 (en) 2010-07-23 2014-09-09 Google Inc. Combining user feedback
US8806340B2 (en) * 2010-09-01 2014-08-12 Hulu, LLC Method and apparatus for embedding media programs having custom user selectable thumbnails
JP6042337B2 (en) 2010-09-27 2016-12-14 フル・エルエルシー Method and apparatus for providing user information to improve advertising experience
US10068266B2 (en) 2010-12-02 2018-09-04 Vinimaya Inc. Methods and systems to maintain, check, report, and audit contract and historical pricing in electronic procurement
US9002867B1 (en) 2010-12-30 2015-04-07 Google Inc. Modifying ranking data based on document changes
US20130067346A1 (en) * 2011-09-09 2013-03-14 Microsoft Corporation Content User Experience
IL216118A0 (en) 2011-11-03 2012-02-29 Google Inc Customer support solution recommendation system
CN102622454B (en) * 2012-04-23 2014-05-28 杭州电子科技大学 Video website-oriented Internet video search method based on text analysis
US20130305385A1 (en) * 2012-05-09 2013-11-14 Cloud Cover Music Streaming audio playback service and methodology
US9721563B2 (en) 2012-06-08 2017-08-01 Apple Inc. Name recognition system
US8949240B2 (en) * 2012-07-03 2015-02-03 General Instrument Corporation System for correlating metadata
US8661049B2 (en) 2012-07-09 2014-02-25 ZenDesk, Inc. Weight-based stemming for improving search quality
US9779219B2 (en) 2012-08-09 2017-10-03 Dell Products L.P. Method and system for late binding of option features associated with a device using at least in part license and unique ID information
US9547647B2 (en) 2012-09-19 2017-01-17 Apple Inc. Voice-based media searching
EP2946279B1 (en) 2013-01-15 2019-10-16 Viki, Inc. System and method for captioning media
US20140229393A1 (en) * 2013-02-11 2014-08-14 James E. Malackowski Ip content discovery platform
US9292537B1 (en) 2013-02-23 2016-03-22 Bryant Christopher Lee Autocompletion of filename based on text in a file to be saved
US9183499B1 (en) 2013-04-19 2015-11-10 Google Inc. Evaluating quality based on neighbor features
US9818401B2 (en) * 2013-05-30 2017-11-14 Promptu Systems Corporation Systems and methods for adaptive proper name entity recognition and understanding
US10170114B2 (en) 2013-05-30 2019-01-01 Promptu Systems Corporation Systems and methods for adaptive proper name entity recognition and understanding
US9449599B2 (en) * 2013-05-30 2016-09-20 Promptu Systems Corporation Systems and methods for adaptive proper name entity recognition and understanding
WO2014197334A2 (en) 2013-06-07 2014-12-11 Apple Inc. System and method for user-specified pronunciation of words for speech synthesis and recognition
US9769546B2 (en) 2013-08-01 2017-09-19 Hulu, LLC Preview image processing using a bundle of preview images
US9311300B2 (en) 2013-09-13 2016-04-12 International Business Machines Corporation Using natural language processing (NLP) to create subject matter synonyms from definitions
US10175850B2 (en) 2013-11-23 2019-01-08 Sharp Laboratories Of America Search inquiry method using contextual annotation
US9990433B2 (en) 2014-05-23 2018-06-05 Samsung Electronics Co., Ltd. Method for searching and device thereof
US11314826B2 (en) 2014-05-23 2022-04-26 Samsung Electronics Co., Ltd. Method for searching and device thereof
US9430463B2 (en) 2014-05-30 2016-08-30 Apple Inc. Exemplar-based natural language processing
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9959305B2 (en) * 2014-07-08 2018-05-01 Microsoft Technology Licensing, Llc Annotating structured data for search
US10146748B1 (en) * 2014-09-10 2018-12-04 Google Llc Embedding location information in a media collaboration using natural language processing
US9668121B2 (en) 2014-09-30 2017-05-30 Apple Inc. Social reminders
US10685042B2 (en) * 2014-12-22 2020-06-16 Amazon Technologies, Inc. Identifying join relationships based on transactional access patterns
US10120905B2 (en) 2014-12-22 2018-11-06 Amazon Technologies, Inc. Efficient determination of join paths via cardinality estimation
KR102306538B1 (en) 2015-01-20 2021-09-29 삼성전자주식회사 Apparatus and method for editing content
US10567477B2 (en) 2015-03-08 2020-02-18 Apple Inc. Virtual assistant continuity
JP6704985B2 (en) * 2015-04-05 2020-06-03 デジタル・アセット・ホールディングス・エルエルシー Digital asset brokerage electronic payment platform
US9578173B2 (en) 2015-06-05 2017-02-21 Apple Inc. Virtual assistant aided communication with 3rd party service in a communication session
US20160378747A1 (en) * 2015-06-29 2016-12-29 Apple Inc. Virtual assistant for media playback
US10747498B2 (en) 2015-09-08 2020-08-18 Apple Inc. Zero latency digital assistant
US10671428B2 (en) 2015-09-08 2020-06-02 Apple Inc. Distributed personal assistant
US11010550B2 (en) 2015-09-29 2021-05-18 Apple Inc. Unified language modeling framework for word prediction, auto-completion and auto-correction
US10366158B2 (en) 2015-09-29 2019-07-30 Apple Inc. Efficient word encoding for recurrent neural network language models
US10691473B2 (en) 2015-11-06 2020-06-23 Apple Inc. Intelligent automated assistant in a messaging environment
US10049668B2 (en) 2015-12-02 2018-08-14 Apple Inc. Applying neural network language models to weighted finite state transducers for automatic speech recognition
US10223066B2 (en) 2015-12-23 2019-03-05 Apple Inc. Proactive assistance based on dialog communication between devices
US11386141B1 (en) * 2016-01-25 2022-07-12 Kelline ASBJORNSEN Multimedia management system (MMS)
US10446143B2 (en) 2016-03-14 2019-10-15 Apple Inc. Identification of voice inputs providing credentials
US9934775B2 (en) 2016-05-26 2018-04-03 Apple Inc. Unit-selection text-to-speech synthesis based on predicted concatenation parameters
US9972304B2 (en) 2016-06-03 2018-05-15 Apple Inc. Privacy preserving distributed evaluation framework for embedded personalized systems
US10249300B2 (en) 2016-06-06 2019-04-02 Apple Inc. Intelligent list reading
US10049663B2 (en) 2016-06-08 2018-08-14 Apple, Inc. Intelligent automated assistant for media exploration
DK179309B1 (en) 2016-06-09 2018-04-23 Apple Inc Intelligent automated assistant in a home environment
US10509862B2 (en) 2016-06-10 2019-12-17 Apple Inc. Dynamic phrase expansion of language input
US10490187B2 (en) 2016-06-10 2019-11-26 Apple Inc. Digital assistant providing automated status report
US10067938B2 (en) 2016-06-10 2018-09-04 Apple Inc. Multilingual word prediction
US10192552B2 (en) 2016-06-10 2019-01-29 Apple Inc. Digital assistant providing whispered speech
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
DK179343B1 (en) 2016-06-11 2018-05-14 Apple Inc Intelligent task discovery
DK179415B1 (en) 2016-06-11 2018-06-14 Apple Inc Intelligent device arbitration and control
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
DK179049B1 (en) 2016-06-11 2017-09-18 Apple Inc Data driven natural language event detection and classification
AU2017326987B2 (en) * 2016-09-19 2022-08-04 Promptu Systems Corporation Systems and methods for adaptive proper name entity recognition and understanding
US10043516B2 (en) 2016-09-23 2018-08-07 Apple Inc. Intelligent automated assistant
US10593346B2 (en) 2016-12-22 2020-03-17 Apple Inc. Rank-reduced token representation for automatic speech recognition
DK201770439A1 (en) 2017-05-11 2018-12-13 Apple Inc. Offline personal assistant
DK179745B1 (en) 2017-05-12 2019-05-01 Apple Inc. SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT
DK179496B1 (en) 2017-05-12 2019-01-15 Apple Inc. USER-SPECIFIC Acoustic Models
DK201770432A1 (en) 2017-05-15 2018-12-21 Apple Inc. Hierarchical belief states for digital assistants
DK201770431A1 (en) 2017-05-15 2018-12-20 Apple Inc. Optimizing dialogue policy decisions for digital assistants using implicit feedback
DK179560B1 (en) 2017-05-16 2019-02-18 Apple Inc. Far-field extension for digital assistant services
US10643178B1 (en) 2017-06-16 2020-05-05 Coupa Software Incorporated Asynchronous real-time procurement system
US11036996B2 (en) * 2019-07-02 2021-06-15 Baidu Usa Llc Method and apparatus for determining (raw) video materials for news
US11514340B2 (en) * 2019-11-08 2022-11-29 International Business Machines Corporation Machine learning for technical tool selection
US11615074B2 (en) 2020-10-01 2023-03-28 Bank Of America Corporation System and methods for intelligent path selection of enhanced distributed processors

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5251316A (en) * 1991-06-28 1993-10-05 Digital Equipment Corporation Method and apparatus for integrating a dynamic lexicon into a full-text information retrieval system
US5752244A (en) * 1996-07-15 1998-05-12 Andersen Consulting Llp Computerized multimedia asset management system
US5765152A (en) * 1995-10-13 1998-06-09 Trustees Of Dartmouth College System and method for managing copyrighted electronic media
US5822432A (en) * 1996-01-17 1998-10-13 The Dice Company Method for human-assisted random key generation and application for digital watermark system
US5903892A (en) * 1996-05-24 1999-05-11 Magnifi, Inc. Indexing of media content on a network
US5963205A (en) * 1995-05-26 1999-10-05 Iconovex Corporation Automatic index creation for a word processor
US5983237A (en) * 1996-03-29 1999-11-09 Virage, Inc. Visual dictionary
US6072904A (en) * 1997-12-31 2000-06-06 Philips Electronics North America Corp. Fast image retrieval using multi-scale edge representation of images
US6243713B1 (en) * 1998-08-24 2001-06-05 Excalibur Technologies Corp. Multimedia document retrieval by application of multimedia queries to a unified index of multimedia data for a plurality of multimedia data types

Family Cites Families (75)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2448824A1 (en) * 1979-02-06 1980-09-05 Telediffusion Fse VIDEOTEX SYSTEM PROVIDED WITH INFORMATION ACCESS CONTROL MEANS
US5241671C1 (en) * 1989-10-26 2002-07-02 Encyclopaedia Britannica Educa Multimedia search system using a plurality of entry path means which indicate interrelatedness of information
US5201047A (en) 1989-12-21 1993-04-06 International Business Machines Corporation Attribute-based classification and retrieval system
US5263158A (en) 1990-02-15 1993-11-16 International Business Machines Corporation Method and system for variable authority level user access control in a distributed data processing system having multiple resource manager
US5317507A (en) * 1990-11-07 1994-05-31 Gallant Stephen I Method for document retrieval and for word sense disambiguation using neural networks
US5325298A (en) * 1990-11-07 1994-06-28 Hnc, Inc. Methods for generating or revising context vectors for a plurality of word stems
US5438508A (en) * 1991-06-28 1995-08-01 Digital Equipment Corporation License document interchange format for license management system
US5260999A (en) * 1991-06-28 1993-11-09 Digital Equipment Corporation Filters in license management system
US5442778A (en) * 1991-11-12 1995-08-15 Xerox Corporation Scatter-gather: a cluster-based method and apparatus for browsing large document collections
US5331417A (en) * 1992-09-15 1994-07-19 Digital Pictures, Inc. System and method of displaying a plurality of digital video images
US6947959B1 (en) * 1992-10-01 2005-09-20 Quark, Inc. Digital media asset management system and process
US5319705A (en) 1992-10-21 1994-06-07 International Business Machines Corporation Method and system for multimedia access control enablement
EP0622930A3 (en) * 1993-03-19 1996-06-05 At & T Global Inf Solution Application sharing for computer collaboration system.
US5553143A (en) 1994-02-04 1996-09-03 Novell, Inc. Method and apparatus for electronic licensing
US5493677A (en) * 1994-06-08 1996-02-20 Systems Research & Applications Corporation Generation, archiving, and retrieval of digital images with evoked suggestion-set captions and natural language interface
US5682487A (en) 1994-06-10 1997-10-28 Bay Networks, Inc. Method and apparatus providing resizable views
US5675819A (en) * 1994-06-16 1997-10-07 Xerox Corporation Document information retrieval using global word co-occurrence patterns
US5706497A (en) * 1994-08-15 1998-01-06 Nec Research Institute, Inc. Document retrieval using fuzzy-logic inference
US5600775A (en) * 1994-08-26 1997-02-04 Emotion, Inc. Method and apparatus for annotating full motion video and other indexed data structures
US5850561A (en) * 1994-09-23 1998-12-15 Lucent Technologies Inc. Glossary construction tool
US5532839A (en) 1994-10-07 1996-07-02 Xerox Corporation Simplified document handler job recovery system with reduced memory duplicate scanned image detection
US5629980A (en) * 1994-11-23 1997-05-13 Xerox Corporation System for controlling the distribution and use of digital works
US5758257A (en) * 1994-11-29 1998-05-26 Herz; Frederick System and method for scheduling broadcast of and access to video programs and other data using customer profiles
US5642502A (en) * 1994-12-06 1997-06-24 University Of Central Florida Method and system for searching for relevant documents from a text database collection, using statistical ranking, relevancy feedback and small pieces of text
WO1997008604A2 (en) * 1995-08-16 1997-03-06 Syracuse University Multilingual document retrieval system and method using semantic vector matching
US5721902A (en) * 1995-09-15 1998-02-24 Infonautics Corporation Restricted expansion of query terms using part of speech tagging
US6125236A (en) * 1995-12-05 2000-09-26 Intel Corporation Method and apparatus for providing user control of multimedia parameters
US5794249A (en) * 1995-12-21 1998-08-11 Hewlett-Packard Company Audio/video retrieval system that uses keyword indexing of digital recordings to display a list of the recorded text files, keywords and time stamps associated with the system
US5987459A (en) 1996-03-15 1999-11-16 Regents Of The University Of Minnesota Image and document management system for content-based retrieval
US5991876A (en) * 1996-04-01 1999-11-23 Copyright Clearance Center, Inc. Electronic rights management and authorization system
US5778362A (en) * 1996-06-21 1998-07-07 Kdl Technologies Limted Method and system for revealing information structures in collections of data items
US5864845A (en) 1996-06-28 1999-01-26 Siemens Corporate Research, Inc. Facilitating world wide web searches utilizing a multiple search engine query clustering fusion strategy
US5832495A (en) * 1996-07-08 1998-11-03 Survivors Of The Shoah Visual History Foundation Method and apparatus for cataloguing multimedia data
US5832499A (en) * 1996-07-10 1998-11-03 Survivors Of The Shoah Visual History Foundation Digital library system
US5813014A (en) * 1996-07-10 1998-09-22 Survivors Of The Shoah Visual History Foundation Method and apparatus for management of multimedia assets
US6545687B2 (en) * 1997-01-09 2003-04-08 Canon Kabushiki Kaisha Thumbnail manipulation using fast and aspect ratio zooming, compressing and scaling
US6006241A (en) 1997-03-14 1999-12-21 Microsoft Corporation Production of a video stream with synchronized annotations over a computer network
US5875446A (en) * 1997-02-24 1999-02-23 International Business Machines Corporation System and method for hierarchically grouping and ranking a set of objects in a query context based on one or more relationships
US5920861A (en) * 1997-02-25 1999-07-06 Intertrust Technologies Corp. Techniques for defining using and manipulating rights management data structures
KR100438693B1 (en) * 1997-06-04 2005-08-17 삼성전자주식회사 Voice and video multiple transmission system
US6012068A (en) 1997-06-30 2000-01-04 International Business Machines Corporation Media manager for access to multiple media types
US6484149B1 (en) * 1997-10-10 2002-11-19 Microsoft Corporation Systems and methods for viewing product information, and methods for generating web pages
US6546405B2 (en) 1997-10-23 2003-04-08 Microsoft Corporation Annotating temporally-dimensioned multimedia content
JPH11203359A (en) * 1998-01-14 1999-07-30 Fuji Photo Film Co Ltd Network photo service system
US6385596B1 (en) * 1998-02-06 2002-05-07 Liquid Audio, Inc. Secure online music distribution system
US6834130B1 (en) 1998-02-18 2004-12-21 Minolta Co., Ltd. Image retrieval system for retrieving a plurality of images which are recorded in a recording medium, and a method thereof
US6349373B2 (en) * 1998-02-20 2002-02-19 Eastman Kodak Company Digital image management system having method for managing images according to image groups
US6038333A (en) 1998-03-16 2000-03-14 Hewlett-Packard Company Person identifier and management system
US6415282B1 (en) * 1998-04-22 2002-07-02 Nec Usa, Inc. Method and apparatus for query refinement
US6578073B1 (en) 1998-05-13 2003-06-10 Hewlett-Packard Development Company, L.P. Accelerated content delivery over a network using reduced size objects
US6226618B1 (en) * 1998-08-13 2001-05-01 International Business Machines Corporation Electronic content delivery system
JP3915267B2 (en) * 1998-09-07 2007-05-16 富士ゼロックス株式会社 Document search apparatus and document search method
US6430357B1 (en) * 1998-09-22 2002-08-06 Ati International Srl Text data extraction system for interleaved video data streams
US6725227B1 (en) * 1998-10-02 2004-04-20 Nec Corporation Advanced web bookmark database system
US6523028B1 (en) * 1998-12-03 2003-02-18 Lockhead Martin Corporation Method and system for universal querying of distributed databases
US6424358B1 (en) * 1998-12-03 2002-07-23 Lockheed Martin Corporation Method and system for importing database information
US6321220B1 (en) * 1998-12-07 2001-11-20 Altavista Company Method and apparatus for preventing topic drift in queries in hyperlinked environments
US6920610B1 (en) * 1999-03-02 2005-07-19 Microsoft Corporation Method and system for browsing a low-resolution image
US6484161B1 (en) * 1999-03-31 2002-11-19 Verizon Laboratories Inc. Method and system for performing online data queries in a distributed computer system
JP2000312334A (en) 1999-04-27 2000-11-07 Canon Inc Image storage device
US6404441B1 (en) * 1999-07-16 2002-06-11 Jet Software, Inc. System for creating media presentations of computer software application programs
US6734886B1 (en) * 1999-12-21 2004-05-11 Personalpath Systems, Inc. Method of customizing a browsing experience on a world-wide-web site
JP2001186334A (en) 1999-12-27 2001-07-06 Canon Inc Device, system and method for picture processing, and storage medium
JP2001251616A (en) * 2000-03-02 2001-09-14 Media Glue Corp Method and device for converting multiplexed sound/ moving picture compressing-coded signal, and medium recorded with conversion program
EP1187476A4 (en) * 2000-04-10 2005-08-10 Sony Corp Asset management system and asset management method
US6711561B1 (en) * 2000-05-02 2004-03-23 Iphrase.Com, Inc. Prose feedback in information access system
US6636848B1 (en) * 2000-05-31 2003-10-21 International Business Machines Corporation Information search using knowledge agents
AU7593601A (en) * 2000-07-14 2002-01-30 Atabok Inc Controlling and managing digital assets
US6944340B1 (en) 2000-08-07 2005-09-13 Canon Kabushiki Kaisha Method and apparatus for efficient determination of recognition parameters
AU2001288469A1 (en) 2000-08-28 2002-03-13 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6941294B2 (en) * 2000-08-28 2005-09-06 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6581055B1 (en) * 2000-09-11 2003-06-17 Oracle International Corporation Query optimization with switch predicates
US6735583B1 (en) * 2000-11-01 2004-05-11 Getty Images, Inc. Method and system for classifying and locating media content
US20040205333A1 (en) * 2003-04-14 2004-10-14 Telefonaktiebolaget Lm Ericsson (Publ) Method and system for digital rights management
US8732175B2 (en) * 2005-04-21 2014-05-20 Yahoo! Inc. Interestingness ranking of media objects

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5251316A (en) * 1991-06-28 1993-10-05 Digital Equipment Corporation Method and apparatus for integrating a dynamic lexicon into a full-text information retrieval system
US5963205A (en) * 1995-05-26 1999-10-05 Iconovex Corporation Automatic index creation for a word processor
US5765152A (en) * 1995-10-13 1998-06-09 Trustees Of Dartmouth College System and method for managing copyrighted electronic media
US5822432A (en) * 1996-01-17 1998-10-13 The Dice Company Method for human-assisted random key generation and application for digital watermark system
US5983237A (en) * 1996-03-29 1999-11-09 Virage, Inc. Visual dictionary
US5903892A (en) * 1996-05-24 1999-05-11 Magnifi, Inc. Indexing of media content on a network
US5752244A (en) * 1996-07-15 1998-05-12 Andersen Consulting Llp Computerized multimedia asset management system
US6072904A (en) * 1997-12-31 2000-06-06 Philips Electronics North America Corp. Fast image retrieval using multi-scale edge representation of images
US6243713B1 (en) * 1998-08-24 2001-06-05 Excalibur Technologies Corp. Multimedia document retrieval by application of multimedia queries to a unified index of multimedia data for a plurality of multimedia data types

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6922691B2 (en) 2000-08-28 2005-07-26 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6941294B2 (en) 2000-08-28 2005-09-06 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6944611B2 (en) 2000-08-28 2005-09-13 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US6895407B2 (en) 2000-08-28 2005-05-17 Emotion, Inc. Method and apparatus for digital media management, retrieval, and collaboration
US7177879B2 (en) 2000-08-28 2007-02-13 Corbis Digital Asset Management Services, Llc Method and apparatus for digital media management, retrieval, and collaboration
US7418444B2 (en) 2000-08-28 2008-08-26 Corbis Corporation Method and apparatus for digital media management, retrieval, and collaboration
US7240049B2 (en) * 2003-11-12 2007-07-03 Yahoo! Inc. Systems and methods for search query processing using trend analysis
CN100401292C (en) * 2003-11-12 2008-07-09 雅虎公司 Systems and methods for search query processing using trend analysis
US7562076B2 (en) 2003-11-12 2009-07-14 Yahoo! Inc. Systems and methods for search query processing using trend analysis
US7953504B2 (en) 2004-05-14 2011-05-31 Synaptics Incorporated Method and apparatus for selecting an audio track based upon audio excerpts
US7428533B2 (en) 2004-12-06 2008-09-23 Yahoo! Inc. Automatic generation of taxonomies for categorizing queries and search query processing using taxonomies
US7620628B2 (en) 2004-12-06 2009-11-17 Yahoo! Inc. Search processing with automatic categorization of queries
GB2424091A (en) * 2005-03-11 2006-09-13 Alamy Ltd Ranking of images in the results of a search
US7937403B2 (en) 2006-10-30 2011-05-03 Yahoo! Inc. Time-based analysis of related keyword searching
US7933765B2 (en) 2007-01-25 2011-04-26 Corbis Corporation Cross-lingual information retrieval
US8073828B2 (en) 2007-06-14 2011-12-06 Curbis Corporation Licensed rights clearance and tracking for digital assets
US8219494B1 (en) 2007-08-16 2012-07-10 Corbis Corporation End-to-end licensing of digital media assets
US8341195B1 (en) 2007-10-04 2012-12-25 Corbis Corporation Platform for managing media assets for multi-model licensing over multi-level pricing and asset grouping

Also Published As

Publication number Publication date
US6944611B2 (en) 2005-09-13
US7177879B2 (en) 2007-02-13
US6895407B2 (en) 2005-05-17
US7418444B2 (en) 2008-08-26
AU2001288469A1 (en) 2002-03-13
US20020188621A1 (en) 2002-12-12
US20030004968A1 (en) 2003-01-02
US20020194198A1 (en) 2002-12-19
US6922691B2 (en) 2005-07-26
US20020194200A1 (en) 2002-12-19
US20020194197A1 (en) 2002-12-19

Similar Documents

Publication Publication Date Title
US6944611B2 (en) Method and apparatus for digital media management, retrieval, and collaboration
US6941294B2 (en) Method and apparatus for digital media management, retrieval, and collaboration
US7437351B2 (en) Method for searching media
US20100305959A1 (en) System and method for providing a media content exchange
US7912827B2 (en) System and method for searching text-based media content
US5895464A (en) Computer program product and a method for using natural language for the description, search and retrieval of multi-media objects
US20070124282A1 (en) Video data directory
US20040017390A1 (en) Self instructional authoring software tool for creation of a multi-media presentation
US20070050406A1 (en) System and method for searching and analyzing media content
US20070106794A1 (en) Method and system for access to electronic version of a physical work based on user ownership of the physical work
US20030011630A1 (en) Self instructional authoring software tool for the creation of a multi-media resume
Christel et al. Techniques for the creation and exploration of digital video libraries
Christel Automated Metadata in Multimedia Information Systems
Bolettieri et al. Automatic metadata extraction and indexing for reusing e-learning multimedia objects
Lian Innovative Internet video consuming based on media analysis techniques
Kim Toward video semantic search based on a structured folksonomy
US9015159B1 (en) Method for searching media
Over I Modifier Irl
US20080281822A1 (en) Interactive System For Building, Organising, and Sharing One's Own Databank of Works of Literature in one or More Languages
Liu et al. Semantic extraction and semantics-based annotation and retrieval for video databases
Geisler et al. Crowdsourcing the indexing of film and television media
Di Bono et al. WP9: A review of data and metadata standards and techniques for representation of multimedia content
La Barre et al. Film retrieval on the web: sharing, naming, access and discovery
USRE45952E1 (en) Method for searching media
Brophy et al. Synchronised Object Retrieval: a feasibility study into enhanced information retrieval in multimedia environments using synchronisation protocols

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PH PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

WWE Wipo information: entry into national phase

Ref document number: 10063411

Country of ref document: US

Ref document number: 10063410

Country of ref document: US

Ref document number: 10063414

Country of ref document: US

Ref document number: 10063413

Country of ref document: US

Ref document number: 10063412

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP