US20110125747A1 - Data classification based on point-of-view dependency - Google Patents
Data classification based on point-of-view dependency Download PDFInfo
- Publication number
- US20110125747A1 US20110125747A1 US12/822,991 US82299110A US2011125747A1 US 20110125747 A1 US20110125747 A1 US 20110125747A1 US 82299110 A US82299110 A US 82299110A US 2011125747 A1 US2011125747 A1 US 2011125747A1
- Authority
- US
- United States
- Prior art keywords
- data
- point
- view
- feature
- input item
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 32
- 239000013598 vector Substances 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims description 10
- 230000008520 organization Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 6
- 238000012706 support-vector machine Methods 0.000 description 11
- 230000007935 neutral effect Effects 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 4
- 238000012552 review Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
Definitions
- the present invention relates to automated data classification in general and data classifiers of documents based on content in particular.
- Data classification systems are useful in many applications.
- One application is in filtering data, as might be done as part of a search over a corpus of data. While many data structures might be used with a data classification system, a typical example is a corpus containing many, many data items organized as units such as records or documents. While a document is used as an example of a data item, it should be understood that statements might be equally applicable to data items that are not normally referred to as documents.
- a data classification system might be used to a filter documents from a large corpus to flag or otherwise identify relevant documents distinctly from less relevant documents.
- a company or an analyst might want to review news items from a large corpus of news items, but only those news items that relate to a particular company or set of companies. They could use a data classification system to flag news items that relate to the companies of interest and provide those relevant documents for further processing, such as manual review.
- a data classification system classifies documents as being “in” or “not in” a particular class, or classifies documents as being in one or more of two or more classes.
- a class might be “all documents containing phrase P” and the simple data classification system classifies each document as either being in the class or not being in the class (binary decision).
- the class might be “all documents mentioning phrase P or its synonyms” or the class might be “all documents apparently relating to topic T”.
- a conventional data classification system might first convert documents into enumerated features though a process of feature generation.
- One way that this can be done is to tokenize text into a distinct dictionary of features with associated enumerated values.
- Advanced techniques may pre-process text with grammatical knowledge to enrich tokens in a way to aid in a classification task (e.g., part-of-speech POS tagging, negation prefixing, etc.).
- “Stop” words (“a”, “the”, “but”, “and”, etc.) are often removed to improve efficiency.
- the data classification system can then perform feature selection, selecting a subset of features that either enhance, or at least minimize loss of, the information content of the document.
- feature selection is primarily performed for efficiency reasons, as many machine learning algorithms display non-linear efficiency with respect to the number of distinct features.
- the selected features can be weighted (which can also be thought of as a “soft” feature selection, where some features are selected strongly and other features are selected weakly), to enhance a machine learning algorithm.
- An example of feature weighting is the use of Inverse Document Frequency (IDF), wherein terms get more weight if they occur more frequently than their general average in a wider corpus and less weight if they appear less frequently than their general average.
- IDF Inverse Document Frequency
- Training might involve providing the data classification system with a corpus and classifications for each document in the training corpus.
- some of the documents in the training corpus are tagged as being examples of members of the class while the others are tagged as being counterexamples.
- the data classification system then operates a training process wherein discriminating patterns are preferably discovered in the training corpus between the examples and the counterexamples.
- Techniques for pattern discrimination have been studied in considerable detail. Examples of machine learning classification techniques include, but are not limited to, Na ⁇ ve Bayes, Support Vector Machines, Maximum Entropy, and k-nearest neighbor. Others might be found in use or in literature on the topic.
- More complex data classification systems have been developed. For example, instead of simply classifying an input document as being an example of a member of the class or a counterexample (a binary classification), the input document might be classified into one of more than just two possibilities (M-ary classification into M classes). For example, when evaluating news stories, a simple data classification system might just make a binary decision as to whether a particular news story refers to topic T or not, while a more complex data classification system might define each class as relating to a particular topic and would classify the input document into one or more of two or more classes.
- Data classification systems might make hard decisions as to how to classify a given input document. Some data classification systems might make soft decisions, wherein an input document is not necessarily classified into a class with absolute certainty, but it is tagged with one or more value(s) indicating the degree(s) to which the document would be associated with each of one or more classes.
- One aspect of the present invention provides a method of data classification, wherein an input item is classified by associating the input item with one or more classes from a set of one or more classes in a data classification system, including identifying relevant features in an input item to form a feature vector for the input item, receiving at the data classification system an indication of a point-of-view, adjusting the feature vector according to the point-of-view indication or modifying a pattern discriminator (e.g., trainer and classifier) to inline-process feature vectors depending on the provided point-of-view (e.g., SVM custom kernels), and classifying the input item into the set of classes according to the point-of-view.
- a pattern discriminator e.g., trainer and classifier
- the point-of-view data can be introduced either as a pre-process step prior to passing it off to the pattern discriminator, or can be incorporated directly into the pattern discriminator if it is applicable (e.g., custom kernels in a support vector machine could be enhanced with point-of-view data).
- the pattern discriminator can detect arbitrary patterns given a similarly prepared dataset during both training and subsequent classification of unclassified documents.
- Some advantages of such a system include improved accuracy within a given classification problem, as it focuses the pattern discrimination engine on the correct context given a point-of-view to operate from. Another advantage is improvements over applications of a given trained model to new points-of-view not incorporated in the original training. This can be the result of methodologies focusing on the features related to the point-of-view while having the effect of abstracting the point-of-view itself.
- Another aspect of the present invention provides a data classification system, wherein the system includes at least one input item, at least one feature vector, and at least one data classifier defined by point-of-view dependency, wherein the system uses feature weighting in order to rate and classify input items.
- the data classifier classifies one or more data sets based upon patterns observed during a training process with one or more training data sets.
- the data classification system may rely on a mathematical engine, such as a support vector machine, to engage in feature weighting.
- an input item may include, but is not limited to, a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.), a collection of documents with some structural organization, an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams.
- a word processing document e.g., a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.)
- a collection of documents with some structural organization e.g., an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams.
- a data classification system is provided with access to one or more of the items of the corpus and based upon the analysis of the one or more items, the data classification system arrives at a determination about each of the one or more items, where an example determination is whether or not a given item belongs into a particular class.
- the data classification system is “trained” using examples, wherein the data classification system is provided with several example items and an indication, for each of the example items, of the classification for those example items.
- the invention further encompasses data classifiers that classify received data sets based upon specific patterns. These patterns are observed during a training process with training data sets.
- FIG. 1 illustrates a classifier with point-of-view dependencies according to aspects of the present invention.
- FIG. 2 illustrates a data classification system according to aspects of the present invention.
- FIG. 3 illustrates a trainer in a data classification system according to aspects of the present invention.
- FIG. 4 illustrates a classifier in a data classification system according to aspects of the present invention.
- FIG. 5 illustrates a system wizard usable for accepting user input of a point of view for a training session.
- FIG. 6 illustrates a system wizard usable for accepting user input on variances of the predicted sentiment for an automatically selected set of articles.
- an input item may include, but is not limited to, a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.), a collection of documents with some structural organization, an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams.
- a word processing document e.g., a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.)
- a collection of documents with some structural organization e.g., an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams.
- relevant feature refers to a uniquely identifiable attribute that could affect the detection of patterns within a corpus.
- Relevant features might be domain specific, for example, in the case of English text classification, a relevant feature might be the presence of a unique word within a document, regardless of position.
- a “feature vector”, as used herein, refers to a list of features describing an instance, wherein a feature is the specification of an attribute and its value. For example, in the case of English text classification, the attribute of a feature might be a unique word within a document and the value of the feature might be the number of occurrences of the unique word within the document.
- classifier refers to a system, apparatus or code for mapping from unlabeled instances to discrete classes. Classifiers may use a mapping form (e.g., decision tree) and an interpretation procedure, including rules for how to handle unknowns. Some classifiers might also provide probability estimates or scores. These scores can be evaluated to yield a discrete decision.
- mapping form e.g., decision tree
- interpretation procedure e.g., rules for how to handle unknowns.
- Some classifiers might also provide probability estimates or scores. These scores can be evaluated to yield a discrete decision.
- trainer refers to a system, apparatus or code for examining a set of known labeled instances to detect implicit patterns and create models. Classifiers can then apply these models to future unlabelled instances to generate discrete classes (see “classifier”).
- point-of-view and “POV” are interchangeably used herein and refer to a variable frame of reference when examining or processing a current document. The same instance may be placed in different classes given different “point-of-views”.
- FIG. 1 is a block diagram of a classifier with point-of-view dependencies according to embodiments of the present invention.
- input documents can be classified into classes for a given point of view (point-of-view dependency).
- point-of-view dependency In many cases, a corpus might be divided into classes one way for one point-of-view and would be divided into those classes differently for a different point-of-view.
- classifier 40 might classify an input document 15 into class 301 or 302 depending on a given point of view 201 or 202 . This allows for improved data assessment over a conventional data classification system that might always classify a document into an example or a counterexample (or one or more of a plurality of classes in M-ary classification).
- An example illustrating the instant system might be the collection of documents regarding a lawsuit.
- the documents will likely contain references to a court (e.g., company A) and several side references to other companies (e.g., companies B-D). From the point-of-view of company A, the documents should be classified as belonging to a lawsuit class that company A may track and analyze daily. From the point of view of companies B-D, the very same documents would not be considered lawsuit documents (i.e., classified under a lawsuit class) because, from their perspective, the documents are not about a lawsuit concerning companies B-D.
- an analyst might be searching a news report corpus for articles about layoffs and the data classification system might classify incoming articles as being about layoffs, or not about layoffs.
- the data classification system might classify incoming articles as being about layoffs, or not about layoffs.
- a traditional data classification system would only recognize that the article is about layoffs regardless of whether it concerns Company A or Company B.
- this distinction is easily made. As such, the system is trained against the difference and would be able to, when given the Company A point-of-view, correctly classify as Layoffs and when given the Company B point-of-view, correctly classify as NOT-Layoffs within the same document.
- FIG. 2 is a high-level block diagram illustrating a data classification system 10 according to the present invention.
- System 10 accepts a training corpus, such as training corpus 12 , for training the data classification system.
- the training corpus 12 contains labeled documents.
- some of the documents in the training corpus are tagged as being examples of members of the class while the others are tagged as being counterexamples.
- the data classification system 10 then operates a trainer 20 wherein discriminating patterns are discovered in the training corpus 12 between the examples and counterexamples to generate a model 30 .
- Examples of pattern discrimination techniques include, but are not limited to, Na ⁇ ve Bayes, Support Vector Machines, Maximum Entropy, and k-nearest neighbor.
- system 10 When a new input document 15 is presented, system 10 operates a classifier 40 , which classifies document 15 using model 30 and generates a predicted class 50 . For example, in a simple binary classification system, document 15 is classified as either being in the class or not being in the class.
- Point of view (POV) 60 includes context sensitive information to enable trainer 20 to discriminate a single classification between multiple points-of-view.
- POV 70 is an input to classifier 40 to enable it to generate a single classification between multiple points-of-view.
- FIG. 3 shows trainer 20 in greater detail.
- Trainer 20 accepts a training corpus 12 as its input and outputs a model 30 with discerned patterns in training corpus 12 .
- trainer 20 is shown with a feature generator 1 , feature selector 2 , feature weighter 3 , and a pattern discriminator 4 .
- Feature generator 1 generates, from input corpus 12 , features to be considered for discrimination. With each document in training corpus 12 distilled to a set of enumerated features, trainer 20 can then operate feature selector 2 to select a subset of features that either enhance, or at least minimize loss of, the information content of the document. Feature selection is primarily performed for efficiency reasons, as many pattern discrimination techniques display non-linear efficiency with respect to the number of distinct features.
- Feature weighter 3 then weights the selected features to enhance a pattern discrimination algorithm.
- a pattern discriminator 4 is run to discriminate patterns within the training corpus, where the pattern discriminator is optionally provided with a custom kernel when the pattern discriminator supports it.
- Point-of-view information 60 can be introduced in various components of trainer 20 to discriminate a single classification between multiple points-of-view.
- FIG. 4 shows classifier 40 in greater detail. Classifier 40 accepts an input document 15 as its input and outputs a predicted class 50 for input document 15 . In the figure, classifier 40 is shown with a feature generator 5 , a feature selector 6 , a feature weighter 7 , and a model applier 8 . Feature generator 5 generates features from input document 15 to be considered for discrimination.
- classifier 40 can then operate a feature selector 6 to select a subset of features that either enhance, or at least minimize loss of, the information content of the document. Feature weighter 7 then weights the selected features. Finally, a model applier 8 applies model 30 to input document 15 to predict the class of input document 15 , where a custom kernel is optionally provided to model 30 . Predicted class 50 is produced optionally with confidence values for input document 15 . Point-of-view information 70 can be introduced in various components of classifier 40 to enable it to generate a single classification between multiple points-of-view.
- POV information uses include, but are not limited to, custom feature generation, feature selection, feature weighting and custom kernel generation.
- Custom feature generation based on POV could generate additional features not normally generated in traditional classification systems where the additional features may be indicative of a relationship between a given POV and a conventional feature.
- Feature selection might be based on POV, wherein features that appear unrelated are stripped out from the vector.
- Feature weighting might also be based on POV, wherein features are weighted based on relationship (i.e., the value associated with the attribute is modified in cases where the pattern discrimination engine supports it).
- a custom kernel might be created when a pattern discriminator supports it (e.g., Support Vector Machine (SVM)).
- SVM Support Vector Machine
- the custom kernel can apply POV weighting of features dynamically during training and classification.
- a novel data classification system might employ POV sentiment ratings. These ratings capture a person's or organization's point-of-view on any sentiment (e.g., article sentiment) using a positive, neutral, and negative (3-point) scale.
- documents can be classified as positive, neutral or negative documents, with respect to a particular POV. This can be used to provide an automated point-of-view sentiment classification service.
- the data classification system performs ternary classification.
- gradations of positive and negative are possible, yielding the more than three classes to choose from, e.g., “strongly positive”, “positive”, “slightly positive”, neutral, etc.
- the data classification system is trained with a training corpus wherein each document in the corpus is labeled with its class.
- the data classification system can then create three models: a positive model, a neutral model, and a negative model. This process can be expanded to classification into more than three classes.
- the ratings extend a particular point-of-view to all articles for a specific subject such as a company, competitor, or the like.
- the user can gain business insights from enhanced sentiment reports as well as sentiment report filters in all other reports.
- sentiment report filters in all other reports.
- the user may gain important business insights by using the systems described herein. For example, a user employed by company X may need to know who the top authors are that are currently writing negative articles (i.e., negative from company X's perspective) about a competitive lawsuit, wherein the user is particularly interested in all articles written during the last week (i.e., top authors/negative articles/lawsuit company topic/seven days rating).
- the user may need to find out how editorial coverage opinion is changing with respect to company X's handling of a specific crisis (i.e., sentiment over time/crisis company topic rating).
- the user may need to investigate what types of publications contain positive articles about company X's recent product launch (top publications/positive articles/product launch company topic rating).
- Sentiment ratings can be automatically applied as articles enter the system, i.e., without human intervention. More specifically, the sentiment ratings might work through a system wizard that captures the point-of-view for a subject while a person reviews and validates ratings such as ratings on articles.
- FIG. 5 illustrates a step of a system wizard, which asks an user to choose a point of view for a training session. For example, the user is asked to choose among Microsoft, Sun Microsystems, Hewlett-Packard, Dell, Gateway, Apple, and Sony Electronics as the point of view for the training session.
- FIG. 6 shows another step of the system wizard, which asks an user to confirm, correct or ignore the predicted sentiment for an automatically selected set of articles. For example, the user is asked to review positive predictions for a set of articles for Microsoft. In a practical setting, new or revised ratings can be applied overnight to all articles for a subject within the system. New articles receive ratings as they enter the system.
- the data classification systems and methods of the instant invention employ event-based machine learning, including an advanced patterns recognition engine, point-of-view capture algorithms, pre-population with a large corpus of rated events, and closed-loop learning for continued point-of-view learning.
- event-based machine learning including an advanced patterns recognition engine, point-of-view capture algorithms, pre-population with a large corpus of rated events, and closed-loop learning for continued point-of-view learning.
- the wizard can be run multiple times (i.e., trained) which improves rating consistency.
- the wizard can be trained any time the user desires to tune sentiment ratings and a new point-of-view may be applied to an entire user account history in batch mode. Moreover, manual ratings and individual article manual overrides can be incorporated into new ratings going forward. Under specific circumstances, it may be necessary to rerun the wizard, particularly when there is a dramatic change in the article profiles. For example, if company X is confronted with a new crisis or company X changes from being a private company to being a public company, the wizard may have to be rerun and thus retrained.
- the data classification system can be used to predict classification of an unlabeled instance. For example, in a ternary classification system, each of the three models is applied in order to predict classification, and a confidence number is returned from each model's classification. Since the three models may disagree (two or more models could claim that the instance is in that model's class), a weighting scheme is applied amongst the three models to break disagreement and produce the single predicted class.
- one implementation was 77%-97% accurate depending on the scenario. This was generated by training on 2 ⁇ 3 of a labeled corpus and testing classification against the remaining 1 ⁇ 3 of a labeled corpus.
- in-corpus accuracy dropped significantly (10-30%)
- cross-corpus accuracy application of trained model to a new corpus in different domain
- fell to statistically insignificant levels i.e., the results were no more accurate than random guesses.
- SVM Support Vector Machine
- SVMs are capable of operating efficiently on large feature spaces, which reduces the need to modify feature vectors for efficiency reasons.
- SVMs support the concept of “weighting” feature vectors which was initially used.
- Equation 1 had the effect of giving more weight to features closer to the point-of-view, with more weight given for a proximity forward of the POV and less weight given for a proximity prior to the POV. Using that equation, documents were distinguishable on the basis of POV.
- adding context sensitive information to the feature vector enables the mathematical engine to discriminate a single classification between different POVs.
- This information can be added in any of the components in FIG. 3 and FIG. 4 . In this example, it is done within “feature weighter” however it is equally applicable to all components.
Abstract
Data classification is used to classified input items by associating the input items with one or more classes from a set of one or more classes in a data classification system, including identifying relevant features in an input item to form a feature vector for the input item, receiving at the data classification system an indication of a point-of-view, adjusting the feature vector according to the point-of-view indication or modifying a pattern discriminator (e.g., trainer and classifier) to inline-process feature vectors depending on the provided point-of-view (e.g., SVM custom kernels), and classifying the input item into the set of classes according to the point-of-view. The point-of-view data can be introduced either as a pre-process step prior to passing it off to the pattern discrimination algorithm, or can be incorporated directly into the pattern discrimination algorithm if applicable. The pattern discrimination algorithms can detect arbitrary patterns given a similarly prepared dataset during both training and subsequent classification of unclassified documents.
Description
- This application is a Continuation of U.S. application Ser. No. 10/931,291, filed Aug. 30, 2004, entitled DATA CLASSIFICATION BASED ON POINT-OF-VIEW DEPENDENCY,” (Attorney Docket No. 021389-000410US) now allowed, which claims priority from co-pending U.S. Provisional Patent Application No. 60/499,196 filed Aug. 28, 2003 entitled DATA CLASSIFICATION BASED ON POINT-OF-VIEW DEPENDENCY, all of which are hereby incorporated by reference, as if set forth in full in this document, for all purposes.
- The present invention relates to automated data classification in general and data classifiers of documents based on content in particular.
- Data classification systems are useful in many applications. One application is in filtering data, as might be done as part of a search over a corpus of data. While many data structures might be used with a data classification system, a typical example is a corpus containing many, many data items organized as units such as records or documents. While a document is used as an example of a data item, it should be understood that statements might be equally applicable to data items that are not normally referred to as documents.
- A data classification system might be used to a filter documents from a large corpus to flag or otherwise identify relevant documents distinctly from less relevant documents. As an example, a company or an analyst might want to review news items from a large corpus of news items, but only those news items that relate to a particular company or set of companies. They could use a data classification system to flag news items that relate to the companies of interest and provide those relevant documents for further processing, such as manual review.
- In the general case, a data classification system classifies documents as being “in” or “not in” a particular class, or classifies documents as being in one or more of two or more classes. In an extremely simple data classification system, a class might be “all documents containing phrase P” and the simple data classification system classifies each document as either being in the class or not being in the class (binary decision). In other simple, but slightly more involved data classification systems, the class might be “all documents mentioning phrase P or its synonyms” or the class might be “all documents apparently relating to topic T”.
- A conventional data classification system might first convert documents into enumerated features though a process of feature generation. One way that this can be done is to tokenize text into a distinct dictionary of features with associated enumerated values. Advanced techniques may pre-process text with grammatical knowledge to enrich tokens in a way to aid in a classification task (e.g., part-of-speech POS tagging, negation prefixing, etc.). “Stop” words (“a”, “the”, “but”, “and”, etc.) are often removed to improve efficiency. With each document distilled to a set of enumerated features, the data classification system can then perform feature selection, selecting a subset of features that either enhance, or at least minimize loss of, the information content of the document. Arguably, feature selection is primarily performed for efficiency reasons, as many machine learning algorithms display non-linear efficiency with respect to the number of distinct features.
- The selected features can be weighted (which can also be thought of as a “soft” feature selection, where some features are selected strongly and other features are selected weakly), to enhance a machine learning algorithm. An example of feature weighting is the use of Inverse Document Frequency (IDF), wherein terms get more weight if they occur more frequently than their general average in a wider corpus and less weight if they appear less frequently than their general average.
- The above processes can be done on documents in a training corpus as well as documents in the corpus that are to be classified. Training might involve providing the data classification system with a corpus and classifications for each document in the training corpus. Thus, for a simple binary classification process, some of the documents in the training corpus are tagged as being examples of members of the class while the others are tagged as being counterexamples.
- The data classification system then operates a training process wherein discriminating patterns are preferably discovered in the training corpus between the examples and the counterexamples. Techniques for pattern discrimination have been studied in considerable detail. Examples of machine learning classification techniques include, but are not limited to, Naïve Bayes, Support Vector Machines, Maximum Entropy, and k-nearest neighbor. Others might be found in use or in literature on the topic.
- More complex data classification systems have been developed. For example, instead of simply classifying an input document as being an example of a member of the class or a counterexample (a binary classification), the input document might be classified into one of more than just two possibilities (M-ary classification into M classes). For example, when evaluating news stories, a simple data classification system might just make a binary decision as to whether a particular news story refers to topic T or not, while a more complex data classification system might define each class as relating to a particular topic and would classify the input document into one or more of two or more classes.
- Data classification systems might make hard decisions as to how to classify a given input document. Some data classification systems might make soft decisions, wherein an input document is not necessarily classified into a class with absolute certainty, but it is tagged with one or more value(s) indicating the degree(s) to which the document would be associated with each of one or more classes.
- One problem with existing data classification systems is that real world examples might be more involved and items would be classified differently depending on other considerations. Hence, there is a considerable need in the art for a more sophisticated classification system capable of classifying items based on multiple inputs into multidimensional categories.
- One aspect of the present invention provides a method of data classification, wherein an input item is classified by associating the input item with one or more classes from a set of one or more classes in a data classification system, including identifying relevant features in an input item to form a feature vector for the input item, receiving at the data classification system an indication of a point-of-view, adjusting the feature vector according to the point-of-view indication or modifying a pattern discriminator (e.g., trainer and classifier) to inline-process feature vectors depending on the provided point-of-view (e.g., SVM custom kernels), and classifying the input item into the set of classes according to the point-of-view. The point-of-view data can be introduced either as a pre-process step prior to passing it off to the pattern discriminator, or can be incorporated directly into the pattern discriminator if it is applicable (e.g., custom kernels in a support vector machine could be enhanced with point-of-view data). The pattern discriminator can detect arbitrary patterns given a similarly prepared dataset during both training and subsequent classification of unclassified documents.
- Some advantages of such a system include improved accuracy within a given classification problem, as it focuses the pattern discrimination engine on the correct context given a point-of-view to operate from. Another advantage is improvements over applications of a given trained model to new points-of-view not incorporated in the original training. This can be the result of methodologies focusing on the features related to the point-of-view while having the effect of abstracting the point-of-view itself.
- Another aspect of the present invention provides a data classification system, wherein the system includes at least one input item, at least one feature vector, and at least one data classifier defined by point-of-view dependency, wherein the system uses feature weighting in order to rate and classify input items. The data classifier classifies one or more data sets based upon patterns observed during a training process with one or more training data sets. In addition, the data classification system may rely on a mathematical engine, such as a support vector machine, to engage in feature weighting.
- As used herein, each item to be classified is described as a document, but it should be understood that items that are classified are not limited to items that are considered documents in all contexts. For example, an input item may include, but is not limited to, a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.), a collection of documents with some structural organization, an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams. A data classification system is provided with access to one or more of the items of the corpus and based upon the analysis of the one or more items, the data classification system arrives at a determination about each of the one or more items, where an example determination is whether or not a given item belongs into a particular class. In some cases, the data classification system is “trained” using examples, wherein the data classification system is provided with several example items and an indication, for each of the example items, of the classification for those example items.
- The invention further encompasses data classifiers that classify received data sets based upon specific patterns. These patterns are observed during a training process with training data sets.
- A further understanding of the nature and advantages of the inventions herein may be realized by reference to the remaining portions of the specification and the attached drawings.
- The present invention is best understood when read in conjunction with the accompanying figures which serve to illustrate the preferred embodiments. It is understood, however, that the invention is not limited to the specific embodiments disclosed in the figures.
-
FIG. 1 illustrates a classifier with point-of-view dependencies according to aspects of the present invention. -
FIG. 2 illustrates a data classification system according to aspects of the present invention. -
FIG. 3 illustrates a trainer in a data classification system according to aspects of the present invention. -
FIG. 4 illustrates a classifier in a data classification system according to aspects of the present invention. -
FIG. 5 illustrates a system wizard usable for accepting user input of a point of view for a training session. -
FIG. 6 illustrates a system wizard usable for accepting user input on variances of the predicted sentiment for an automatically selected set of articles. - The following definitions are set forth to illustrate and define the meaning and scope of the various terms used herein.
- The terms “input item” and “document” are interchangeably used herein and refer to any item that can be used in conjunction with the present classification method. For example, an input item may include, but is not limited to, a word processing document, a file of a particular format (e.g., ASCII file, XML file, UTF-8 file, etc.), a collection of documents with some structural organization, an image, text, a combination of images and text, media, spreadsheet data, a collection of bytes, or other organizations of data or data streams.
- The term “relevant feature” refers to a uniquely identifiable attribute that could affect the detection of patterns within a corpus. Relevant features might be domain specific, for example, in the case of English text classification, a relevant feature might be the presence of a unique word within a document, regardless of position.
- A “feature vector”, as used herein, refers to a list of features describing an instance, wherein a feature is the specification of an attribute and its value. For example, in the case of English text classification, the attribute of a feature might be a unique word within a document and the value of the feature might be the number of occurrences of the unique word within the document.
- The term “classifier” refers to a system, apparatus or code for mapping from unlabeled instances to discrete classes. Classifiers may use a mapping form (e.g., decision tree) and an interpretation procedure, including rules for how to handle unknowns. Some classifiers might also provide probability estimates or scores. These scores can be evaluated to yield a discrete decision.
- The term “trainer” refers to a system, apparatus or code for examining a set of known labeled instances to detect implicit patterns and create models. Classifiers can then apply these models to future unlabelled instances to generate discrete classes (see “classifier”).
- The terms “point-of-view” and “POV” are interchangeably used herein and refer to a variable frame of reference when examining or processing a current document. The same instance may be placed in different classes given different “point-of-views”.
-
FIG. 1 is a block diagram of a classifier with point-of-view dependencies according to embodiments of the present invention. Using the novel data classification systems and methods described herein, input documents can be classified into classes for a given point of view (point-of-view dependency). In many cases, a corpus might be divided into classes one way for one point-of-view and would be divided into those classes differently for a different point-of-view. As shown inFIG. 1 ,classifier 40 might classify aninput document 15 intoclass view - An example illustrating the instant system might be the collection of documents regarding a lawsuit. The documents will likely contain references to a defendant (e.g., company A) and several side references to other companies (e.g., companies B-D). From the point-of-view of company A, the documents should be classified as belonging to a lawsuit class that company A may track and analyze daily. From the point of view of companies B-D, the very same documents would not be considered lawsuit documents (i.e., classified under a lawsuit class) because, from their perspective, the documents are not about a lawsuit concerning companies B-D.
- In another example, an analyst might be searching a news report corpus for articles about layoffs and the data classification system might classify incoming articles as being about layoffs, or not about layoffs. Given a document with two threads of discussion, one regarding layoffs at Company A and one regarding a merger at Company B, a traditional data classification system would only recognize that the article is about layoffs regardless of whether it concerns Company A or Company B. However, in embodiments of a novel data classification system as described herein, this distinction is easily made. As such, the system is trained against the difference and would be able to, when given the Company A point-of-view, correctly classify as Layoffs and when given the Company B point-of-view, correctly classify as NOT-Layoffs within the same document. Hence, the classification of “sentiment” or “favorability” applies perfectly herein, i.e., given a certain document, it may easily be defined as “favorable” for one company and defined as “unfavorable” for another company.
-
FIG. 2 is a high-level block diagram illustrating adata classification system 10 according to the present invention.System 10 accepts a training corpus, such astraining corpus 12, for training the data classification system. In one embodiment, thetraining corpus 12 contains labeled documents. For a simple binary data classification system in this embodiment, some of the documents in the training corpus are tagged as being examples of members of the class while the others are tagged as being counterexamples. Thedata classification system 10 then operates atrainer 20 wherein discriminating patterns are discovered in thetraining corpus 12 between the examples and counterexamples to generate amodel 30. Examples of pattern discrimination techniques include, but are not limited to, Naïve Bayes, Support Vector Machines, Maximum Entropy, and k-nearest neighbor. - When a
new input document 15 is presented,system 10 operates aclassifier 40, which classifiesdocument 15 usingmodel 30 and generates a predictedclass 50. For example, in a simple binary classification system,document 15 is classified as either being in the class or not being in the class. - Point of view (POV) 60 includes context sensitive information to enable
trainer 20 to discriminate a single classification between multiple points-of-view. Similarly,POV 70 is an input toclassifier 40 to enable it to generate a single classification between multiple points-of-view. -
FIG. 3 showstrainer 20 in greater detail.Trainer 20 accepts atraining corpus 12 as its input and outputs amodel 30 with discerned patterns intraining corpus 12. In the figure,trainer 20 is shown with afeature generator 1,feature selector 2, feature weighter 3, and apattern discriminator 4.Feature generator 1 generates, frominput corpus 12, features to be considered for discrimination. With each document intraining corpus 12 distilled to a set of enumerated features,trainer 20 can then operatefeature selector 2 to select a subset of features that either enhance, or at least minimize loss of, the information content of the document. Feature selection is primarily performed for efficiency reasons, as many pattern discrimination techniques display non-linear efficiency with respect to the number of distinct features. Feature weighter 3 then weights the selected features to enhance a pattern discrimination algorithm. Finally, apattern discriminator 4 is run to discriminate patterns within the training corpus, where the pattern discriminator is optionally provided with a custom kernel when the pattern discriminator supports it. Point-of-view information 60 can be introduced in various components oftrainer 20 to discriminate a single classification between multiple points-of-view.
FIG. 4 showsclassifier 40 in greater detail.Classifier 40 accepts aninput document 15 as its input and outputs a predictedclass 50 forinput document 15. In the figure,classifier 40 is shown with afeature generator 5, a feature selector 6, a feature weighter 7, and amodel applier 8.Feature generator 5 generates features frominput document 15 to be considered for discrimination. Afterinput document 15 is distilled into a set of enumerated features,classifier 40 can then operate a feature selector 6 to select a subset of features that either enhance, or at least minimize loss of, the information content of the document. Feature weighter 7 then weights the selected features. Finally, amodel applier 8 appliesmodel 30 to inputdocument 15 to predict the class ofinput document 15, where a custom kernel is optionally provided tomodel 30.Predicted class 50 is produced optionally with confidence values forinput document 15. Point-of-view information 70 can be introduced in various components ofclassifier 40 to enable it to generate a single classification between multiple points-of-view. - POV information uses include, but are not limited to, custom feature generation, feature selection, feature weighting and custom kernel generation. Custom feature generation based on POV could generate additional features not normally generated in traditional classification systems where the additional features may be indicative of a relationship between a given POV and a conventional feature. Feature selection might be based on POV, wherein features that appear unrelated are stripped out from the vector. Feature weighting might also be based on POV, wherein features are weighted based on relationship (i.e., the value associated with the attribute is modified in cases where the pattern discrimination engine supports it). Similarly, a custom kernel might be created when a pattern discriminator supports it (e.g., Support Vector Machine (SVM)). The custom kernel can apply POV weighting of features dynamically during training and classification.
- The following specific examples are intended to illustrate embodiments of data classification systems according to aspects of the invention and should not be construed as limiting the scope of the claims.
- (i) Point-of-View (POV) Sentiment Ratings
- A novel data classification system might employ POV sentiment ratings. These ratings capture a person's or organization's point-of-view on any sentiment (e.g., article sentiment) using a positive, neutral, and negative (3-point) scale. In this system, documents can be classified as positive, neutral or negative documents, with respect to a particular POV. This can be used to provide an automated point-of-view sentiment classification service. Where a document must be classified as positive, neutral or negative, the data classification system performs ternary classification. In other variations, gradations of positive and negative are possible, yielding the more than three classes to choose from, e.g., “strongly positive”, “positive”, “slightly positive”, neutral, etc. In order to achieve the ternary classification, the data classification system is trained with a training corpus wherein each document in the corpus is labeled with its class. The data classification system can then create three models: a positive model, a neutral model, and a negative model. This process can be expanded to classification into more than three classes.
- The ratings extend a particular point-of-view to all articles for a specific subject such as a company, competitor, or the like. The user can gain business insights from enhanced sentiment reports as well as sentiment report filters in all other reports. There are many examples of how the user may gain important business insights by using the systems described herein. For example, a user employed by company X may need to know who the top authors are that are currently writing negative articles (i.e., negative from company X's perspective) about a competitive lawsuit, wherein the user is particularly interested in all articles written during the last week (i.e., top authors/negative articles/lawsuit company topic/seven days rating). Alternatively, the user may need to find out how editorial coverage opinion is changing with respect to company X's handling of a specific crisis (i.e., sentiment over time/crisis company topic rating). Yet in another scenario, the user may need to investigate what types of publications contain positive articles about company X's recent product launch (top publications/positive articles/product launch company topic rating).
- Sentiment ratings can be automatically applied as articles enter the system, i.e., without human intervention. More specifically, the sentiment ratings might work through a system wizard that captures the point-of-view for a subject while a person reviews and validates ratings such as ratings on articles.
FIG. 5 illustrates a step of a system wizard, which asks an user to choose a point of view for a training session. For example, the user is asked to choose among Microsoft, Sun Microsystems, Hewlett-Packard, Dell, Gateway, Apple, and Sony Electronics as the point of view for the training session.FIG. 6 shows another step of the system wizard, which asks an user to confirm, correct or ignore the predicted sentiment for an automatically selected set of articles. For example, the user is asked to review positive predictions for a set of articles for Microsoft. In a practical setting, new or revised ratings can be applied overnight to all articles for a subject within the system. New articles receive ratings as they enter the system. - In one embodiment, the data classification systems and methods of the instant invention employ event-based machine learning, including an advanced patterns recognition engine, point-of-view capture algorithms, pre-population with a large corpus of rated events, and closed-loop learning for continued point-of-view learning. Generally, the more the system is taught the more the system knows. Hence, the wizard can be run multiple times (i.e., trained) which improves rating consistency.
- The wizard can be trained any time the user desires to tune sentiment ratings and a new point-of-view may be applied to an entire user account history in batch mode. Moreover, manual ratings and individual article manual overrides can be incorporated into new ratings going forward. Under specific circumstances, it may be necessary to rerun the wizard, particularly when there is a dramatic change in the article profiles. For example, if company X is confronted with a new crisis or company X changes from being a private company to being a public company, the wizard may have to be rerun and thus retrained.
- After training, the data classification system can be used to predict classification of an unlabeled instance. For example, in a ternary classification system, each of the three models is applied in order to predict classification, and a confidence number is returned from each model's classification. Since the three models may disagree (two or more models could claim that the instance is in that model's class), a weighting scheme is applied amongst the three models to break disagreement and produce the single predicted class.
- In tests, one implementation was 77%-97% accurate depending on the scenario. This was generated by training on ⅔ of a labeled corpus and testing classification against the remaining ⅓ of a labeled corpus. When a conventional process stack was applied against the same problem, in-corpus accuracy dropped significantly (10-30%), and cross-corpus accuracy (application of trained model to a new corpus in different domain) fell to statistically insignificant levels (i.e., the results were no more accurate than random guesses).
- (ii) SVM Pattern Discrimination
- In one example, a Support Vector Machine (SVM) pattern discrimination algorithm was chosen for classification. SVMs are capable of operating efficiently on large feature spaces, which reduces the need to modify feature vectors for efficiency reasons. In addition, SVMs support the concept of “weighting” feature vectors which was initially used.
- In the feature weighting scheme that was examined, one can apply a relatively unsophisticated algorithm of weighting. Given a bag of aliases representing a simplistic Point-of-View (e.g., “IBM”, “International Business Machines”, “Big-Blue”), one can weight all features against sentence-level proximity to an alias within the bag. In the test-case, a feature was weighted according to the number of sentences away from the nearest alias, using the formula shown in
Equation 1, where FSP is feature-to-sentence proximity going forward from the alias and BSP is feature-to-sentence proximity going backward from the alias. -
FeatureWeight=Max(0.95FSP,0.80BSP) (Equ. 1) - Using the formula of
Equation 1 had the effect of giving more weight to features closer to the point-of-view, with more weight given for a proximity forward of the POV and less weight given for a proximity prior to the POV. Using that equation, documents were distinguishable on the basis of POV. - Thus, adding context sensitive information to the feature vector enables the mathematical engine to discriminate a single classification between different POVs. This information can be added in any of the components in
FIG. 3 andFIG. 4 . In this example, it is done within “feature weighter” however it is equally applicable to all components. - Various modifications and variations of the present invention will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Although the invention has been described in connection with specific preferred embodiments, it should be understood that the invention as claimed should not be unduly limited to to such specific embodiments. Indeed, various modifications of the described modes for carrying out the invention which are obvious to those skilled in the art are intended to be within the scope of the claims.
Claims (11)
1. A method of classification, wherein an input item is classified by associating the input item with one or more classes from a set of classes in a data classification system, said method comprising the steps of:
receiving the input item to be classified;
identifying relevant features in the input item to form a feature vector for the input item;
receiving an indication of a point of view at the data classification system;
adjusting the feature vector or modifying a pattern discriminator according to the point-of-view indication; and
classifying the input item into the set of classes according to the point-of-view.
2. The method of claim 1 , wherein the step of adjusting the feature vector comprises generating custom features.
3. The method of claim 1 , wherein the step of adjusting the feature vector comprises selecting a subset of features.
4. The method of claim 1 , wherein the step of modifying a pattern discrimination algorithm comprises generating a custom kernel.
5. The method of claim 1 , wherein the step of adjusting the feature vector comprises weighting features.
6. The method of claim 5 , wherein weighting features uses proximity weighting.
7. The method of claim 6 , wherein proximity weighting calculates weight of a feature as the maximum of 0.95 raised to the power of FSP and 0.80 raised to the power of BSP, wherein FSP is the number of sentences going forward from a nearest alias to the feature and BSP is the number of sentences going backward from a nearest alias to the feature, wherein an alias is a representation of a point-of-view.
8. The method of claim 1 , wherein the input item is selected from the group consisting of a word processing document, an ASCII file, an XML file, a UTF-8 file, a collection of documents with some structural organization, an image, a text, a combination of images and text, media, spreadsheet data, a collection of bytes, an organization of data and a data stream.
9. A data classification system comprising at least one input item, at least one feature vector, and at least one data classifier defined by point-of-view dependency, wherein the data classification system is configured to perform one or more of feature generation, feature selection, feature weighting, and custom kernel generation in order to rate and classify the input item.
10. The data classification system of claim 9 , wherein the input item is selected from the group consisting of a word processing document, an ASCII file, an XML file, a UTF-8 file, a collection of documents with some structural organization, an image, a text, a combination of images and text, media, spreadsheet data, a collection of bytes, an organization of data and a data stream.
11. The data classification system of claim 9 , wherein the data classifier classifies one or more data sets based upon patterns observed during a training process with one or more training data sets.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/822,991 US20110125747A1 (en) | 2003-08-28 | 2010-06-24 | Data classification based on point-of-view dependency |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US49919603P | 2003-08-28 | 2003-08-28 | |
US10/931,291 US7769759B1 (en) | 2003-08-28 | 2004-08-30 | Data classification based on point-of-view dependency |
US12/822,991 US20110125747A1 (en) | 2003-08-28 | 2010-06-24 | Data classification based on point-of-view dependency |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/931,291 Continuation US7769759B1 (en) | 2003-08-28 | 2004-08-30 | Data classification based on point-of-view dependency |
Publications (1)
Publication Number | Publication Date |
---|---|
US20110125747A1 true US20110125747A1 (en) | 2011-05-26 |
Family
ID=42358919
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/931,291 Expired - Fee Related US7769759B1 (en) | 2003-08-28 | 2004-08-30 | Data classification based on point-of-view dependency |
US12/822,991 Abandoned US20110125747A1 (en) | 2003-08-28 | 2010-06-24 | Data classification based on point-of-view dependency |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/931,291 Expired - Fee Related US7769759B1 (en) | 2003-08-28 | 2004-08-30 | Data classification based on point-of-view dependency |
Country Status (1)
Country | Link |
---|---|
US (2) | US7769759B1 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120053926A1 (en) * | 2010-08-31 | 2012-03-01 | Red Hat, Inc. | Interactive input method |
DE102013101871A1 (en) * | 2013-02-26 | 2014-08-28 | PSYWARE GmbH | Word-based speech analysis and speech analysis facility |
CN104636489A (en) * | 2015-02-27 | 2015-05-20 | 百度在线网络技术(北京)有限公司 | Processing method and processing device for description attribute data |
US20170154182A1 (en) * | 2015-11-30 | 2017-06-01 | International Business Machines Corporation | System, method and apparatus for usable code-level statistical analysis with applications in malware detection |
US20180225280A1 (en) * | 2017-02-03 | 2018-08-09 | Benedict R. Dugan | Systems and methods for improved text classification |
CN108391446A (en) * | 2017-06-20 | 2018-08-10 | 埃森哲环球解决方案有限公司 | Based on machine learning algorithm automatically extracting to the training corpus for data sorter |
CN108694202A (en) * | 2017-04-10 | 2018-10-23 | 上海交通大学 | Configurable Spam Filtering System based on sorting algorithm and filter method |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7769759B1 (en) * | 2003-08-28 | 2010-08-03 | Biz360, Inc. | Data classification based on point-of-view dependency |
US7925602B2 (en) * | 2007-12-07 | 2011-04-12 | Microsoft Corporation | Maximum entropy model classfier that uses gaussian mean values |
US8788497B2 (en) * | 2008-09-15 | 2014-07-22 | Microsoft Corporation | Automated criterion-based grouping and presenting |
US20100076923A1 (en) * | 2008-09-25 | 2010-03-25 | Microsoft Corporation | Online multi-label active annotation of data files |
US20110251977A1 (en) * | 2010-04-13 | 2011-10-13 | Michal Cialowicz | Ad Hoc Document Parsing |
JP5567049B2 (en) * | 2012-02-29 | 2014-08-06 | 株式会社Ubic | Document sorting system, document sorting method, and document sorting program |
JP5827208B2 (en) * | 2012-11-30 | 2015-12-02 | 株式会社Ubic | Document management system, document management method, and document management program |
US9483730B2 (en) | 2012-12-07 | 2016-11-01 | At&T Intellectual Property I, L.P. | Hybrid review synthesis |
US10102481B2 (en) | 2015-03-16 | 2018-10-16 | Conduent Business Services, Llc | Hybrid active learning for non-stationary streaming data with asynchronous labeling |
FI20165240A (en) * | 2016-03-22 | 2017-09-23 | Utopia Analytics Oy | PROCEDURES, SYSTEMS AND RESOURCES FOR MODERATING CONTENTS |
US11416518B2 (en) | 2018-10-30 | 2022-08-16 | Optum, Inc. | Machine learning for machine-assisted data classification |
US10796203B2 (en) * | 2018-11-30 | 2020-10-06 | International Business Machines Corporation | Out-of-sample generating few-shot classification networks |
US11157475B1 (en) | 2019-04-26 | 2021-10-26 | Bank Of America Corporation | Generating machine learning models for understanding sentence context |
US11783005B2 (en) | 2019-04-26 | 2023-10-10 | Bank Of America Corporation | Classifying and mapping sentences using machine learning |
US11423231B2 (en) | 2019-08-27 | 2022-08-23 | Bank Of America Corporation | Removing outliers from training data for machine learning |
US11449559B2 (en) | 2019-08-27 | 2022-09-20 | Bank Of America Corporation | Identifying similar sentences for machine learning |
US11526804B2 (en) | 2019-08-27 | 2022-12-13 | Bank Of America Corporation | Machine learning model training for reviewing documents |
US11556711B2 (en) | 2019-08-27 | 2023-01-17 | Bank Of America Corporation | Analyzing documents using machine learning |
CN111949763B (en) * | 2020-07-15 | 2024-03-22 | 有米科技股份有限公司 | Single-item category determining method and device based on text material |
Citations (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5737591A (en) * | 1996-05-23 | 1998-04-07 | Microsoft Corporation | Database view generation system |
US5832470A (en) * | 1994-09-30 | 1998-11-03 | Hitachi, Ltd. | Method and apparatus for classifying document information |
US6199034B1 (en) * | 1995-05-31 | 2001-03-06 | Oracle Corporation | Methods and apparatus for determining theme for discourse |
US6243723B1 (en) * | 1997-05-21 | 2001-06-05 | Nec Corporation | Document classification apparatus |
US20010044759A1 (en) * | 2000-05-17 | 2001-11-22 | Matsushita Electric Industrial Co., Ltd. | Information recommendation apparatus and information recommendation system |
US20020059303A1 (en) * | 2000-10-27 | 2002-05-16 | Yoshihiro Ohmori | Multimedia data management system |
US6421463B1 (en) * | 1998-04-01 | 2002-07-16 | Massachusetts Institute Of Technology | Trainable system to search for objects in images |
US6446061B1 (en) * | 1998-07-31 | 2002-09-03 | International Business Machines Corporation | Taxonomy generation for document collections |
US20020124187A1 (en) * | 2000-09-28 | 2002-09-05 | Recourse Technologies, Inc. | System and method for analyzing protocol streams for a security-related event |
US20020199194A1 (en) * | 1999-12-21 | 2002-12-26 | Kamal Ali | Intelligent system and methods of recommending media content items based on user preferences |
US6505195B1 (en) * | 1999-06-03 | 2003-01-07 | Nec Corporation | Classification of retrievable documents according to types of attribute elements |
US20030037151A1 (en) * | 2001-08-16 | 2003-02-20 | Andras Montvay | Method of handling utilization conflicts in digital networks |
US20030101163A1 (en) * | 2000-08-22 | 2003-05-29 | Microsoft Corporation | Method and system for searching for words and phrases in active and stored ink word documents |
US20030115193A1 (en) * | 2001-12-13 | 2003-06-19 | Fujitsu Limited | Information searching method of profile information, program, recording medium, and apparatus |
US20030225763A1 (en) * | 2002-04-15 | 2003-12-04 | Microsoft Corporation | Self-improving system and method for classifying pages on the world wide web |
US20040025178A1 (en) * | 1998-07-23 | 2004-02-05 | Gordon Donald F. | Interactive user interface |
US20040049534A1 (en) * | 2002-09-09 | 2004-03-11 | Opinionlab, Inc. | Receiving and reporting page-specific user feedback concerning one or more particular web pages of a website |
US20040059740A1 (en) * | 2002-09-19 | 2004-03-25 | Noriko Hanakawa | Document management method |
US20040083270A1 (en) * | 2002-10-23 | 2004-04-29 | David Heckerman | Method and system for identifying junk e-mail |
US20050004857A1 (en) * | 2003-04-17 | 2005-01-06 | Portfolio Search, Inc. | Methods for evaluating the financial strength of a holding in comparison to other holdings |
US20050108001A1 (en) * | 2001-11-15 | 2005-05-19 | Aarskog Brit H. | Method and apparatus for textual exploration discovery |
US20050262039A1 (en) * | 2004-05-20 | 2005-11-24 | International Business Machines Corporation | Method and system for analyzing unstructured text in data warehouse |
US20090216693A1 (en) * | 1999-04-28 | 2009-08-27 | Pal Rujan | Classification method and apparatus |
US7769759B1 (en) * | 2003-08-28 | 2010-08-03 | Biz360, Inc. | Data classification based on point-of-view dependency |
-
2004
- 2004-08-30 US US10/931,291 patent/US7769759B1/en not_active Expired - Fee Related
-
2010
- 2010-06-24 US US12/822,991 patent/US20110125747A1/en not_active Abandoned
Patent Citations (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5832470A (en) * | 1994-09-30 | 1998-11-03 | Hitachi, Ltd. | Method and apparatus for classifying document information |
US6199034B1 (en) * | 1995-05-31 | 2001-03-06 | Oracle Corporation | Methods and apparatus for determining theme for discourse |
US5737591A (en) * | 1996-05-23 | 1998-04-07 | Microsoft Corporation | Database view generation system |
US6243723B1 (en) * | 1997-05-21 | 2001-06-05 | Nec Corporation | Document classification apparatus |
US6421463B1 (en) * | 1998-04-01 | 2002-07-16 | Massachusetts Institute Of Technology | Trainable system to search for objects in images |
US20040025178A1 (en) * | 1998-07-23 | 2004-02-05 | Gordon Donald F. | Interactive user interface |
US6446061B1 (en) * | 1998-07-31 | 2002-09-03 | International Business Machines Corporation | Taxonomy generation for document collections |
US20090216693A1 (en) * | 1999-04-28 | 2009-08-27 | Pal Rujan | Classification method and apparatus |
US6505195B1 (en) * | 1999-06-03 | 2003-01-07 | Nec Corporation | Classification of retrievable documents according to types of attribute elements |
US20020199194A1 (en) * | 1999-12-21 | 2002-12-26 | Kamal Ali | Intelligent system and methods of recommending media content items based on user preferences |
US20010044759A1 (en) * | 2000-05-17 | 2001-11-22 | Matsushita Electric Industrial Co., Ltd. | Information recommendation apparatus and information recommendation system |
US20030101163A1 (en) * | 2000-08-22 | 2003-05-29 | Microsoft Corporation | Method and system for searching for words and phrases in active and stored ink word documents |
US20020124187A1 (en) * | 2000-09-28 | 2002-09-05 | Recourse Technologies, Inc. | System and method for analyzing protocol streams for a security-related event |
US20020059303A1 (en) * | 2000-10-27 | 2002-05-16 | Yoshihiro Ohmori | Multimedia data management system |
US20030037151A1 (en) * | 2001-08-16 | 2003-02-20 | Andras Montvay | Method of handling utilization conflicts in digital networks |
US20050108001A1 (en) * | 2001-11-15 | 2005-05-19 | Aarskog Brit H. | Method and apparatus for textual exploration discovery |
US20030115193A1 (en) * | 2001-12-13 | 2003-06-19 | Fujitsu Limited | Information searching method of profile information, program, recording medium, and apparatus |
US20030225763A1 (en) * | 2002-04-15 | 2003-12-04 | Microsoft Corporation | Self-improving system and method for classifying pages on the world wide web |
US20040049534A1 (en) * | 2002-09-09 | 2004-03-11 | Opinionlab, Inc. | Receiving and reporting page-specific user feedback concerning one or more particular web pages of a website |
US20040059740A1 (en) * | 2002-09-19 | 2004-03-25 | Noriko Hanakawa | Document management method |
US20040083270A1 (en) * | 2002-10-23 | 2004-04-29 | David Heckerman | Method and system for identifying junk e-mail |
US20050004857A1 (en) * | 2003-04-17 | 2005-01-06 | Portfolio Search, Inc. | Methods for evaluating the financial strength of a holding in comparison to other holdings |
US7769759B1 (en) * | 2003-08-28 | 2010-08-03 | Biz360, Inc. | Data classification based on point-of-view dependency |
US20050262039A1 (en) * | 2004-05-20 | 2005-11-24 | International Business Machines Corporation | Method and system for analyzing unstructured text in data warehouse |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8838453B2 (en) * | 2010-08-31 | 2014-09-16 | Red Hat, Inc. | Interactive input method |
US20120053926A1 (en) * | 2010-08-31 | 2012-03-01 | Red Hat, Inc. | Interactive input method |
US9805740B2 (en) * | 2013-02-26 | 2017-10-31 | PSYWARE GmbH | Language analysis based on word-selection, and language analysis apparatus |
DE102013101871A1 (en) * | 2013-02-26 | 2014-08-28 | PSYWARE GmbH | Word-based speech analysis and speech analysis facility |
US20160005421A1 (en) * | 2013-02-26 | 2016-01-07 | PSYWARE GmbH | Language analysis based on word-selection, and language analysis apparatus |
CN104636489A (en) * | 2015-02-27 | 2015-05-20 | 百度在线网络技术(北京)有限公司 | Processing method and processing device for description attribute data |
US20170154182A1 (en) * | 2015-11-30 | 2017-06-01 | International Business Machines Corporation | System, method and apparatus for usable code-level statistical analysis with applications in malware detection |
US10754947B2 (en) * | 2015-11-30 | 2020-08-25 | International Business Machines Corporation | System, method and apparatus for usable code-level statistical analysis with applications in malware detection |
US10846401B2 (en) * | 2015-11-30 | 2020-11-24 | International Business Machines Corporation | System, method and apparatus for usable code-level statistical analysis with applications in malware detection |
US20180225280A1 (en) * | 2017-02-03 | 2018-08-09 | Benedict R. Dugan | Systems and methods for improved text classification |
US10740563B2 (en) * | 2017-02-03 | 2020-08-11 | Benedict R. Dugan | System and methods for text classification |
CN108694202A (en) * | 2017-04-10 | 2018-10-23 | 上海交通大学 | Configurable Spam Filtering System based on sorting algorithm and filter method |
CN108391446A (en) * | 2017-06-20 | 2018-08-10 | 埃森哲环球解决方案有限公司 | Based on machine learning algorithm automatically extracting to the training corpus for data sorter |
US20180365322A1 (en) * | 2017-06-20 | 2018-12-20 | Accenture Global Solutions Limited | Automatic extraction of a training corpus for a data classifier based on machine learning algorithms |
US11409779B2 (en) * | 2017-06-20 | 2022-08-09 | Accenture Global Solutions Limited | Automatic extraction of a training corpus for a data classifier based on machine learning algorithms |
Also Published As
Publication number | Publication date |
---|---|
US7769759B1 (en) | 2010-08-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20110125747A1 (en) | Data classification based on point-of-view dependency | |
Muslea et al. | Active learning with multiple views | |
Antonie et al. | Text document categorization by term association | |
US7971150B2 (en) | Document categorisation system | |
Liu et al. | Improving text classification using local latent semantic indexing | |
US7689531B1 (en) | Automatic charset detection using support vector machines with charset grouping | |
JP4034602B2 (en) | Data classification device, active learning method of data classification device, and active learning program | |
US7711673B1 (en) | Automatic charset detection using SIM algorithm with charset grouping | |
CN104750844A (en) | Method and device for generating text characteristic vectors based on TF-IGM, method and device for classifying texts | |
WO2008137368A1 (en) | Web page analysis using multiple graphs | |
Haralambous et al. | Text classification using association rules, dependency pruning and hyperonymization | |
JP2008203933A (en) | Category creation method and apparatus and document classification method and apparatus | |
Silva et al. | On text-based mining with active learning and background knowledge using svm | |
Rawat et al. | Pnpood: Out-of-distribution detection for text classification via plug andplay data augmentation | |
Chiang et al. | The Chinese text categorization system with association rule and category priority | |
Kim et al. | Examining variations of prominent features in genre classification | |
JP2010272004A (en) | Discriminating apparatus, discrimination method, and computer program | |
Wakaki et al. | Rough set-aided feature selection for automatic web-page classification | |
US20110295781A1 (en) | Apparatus and Method for Improved Classifier Training | |
Read et al. | Efficient multi-label classification for evolving data streams | |
Davoudi et al. | A semantic-based feature extraction method using categorical clustering for Persian document classification | |
Shubha et al. | An efficient machine Learning Bayes Sentiment Classification method based on review comments | |
Altınçay et al. | Ternary encoding based feature extraction for binary text classification | |
Gomez et al. | Using biased discriminant analysis for email filtering | |
Ghanbari et al. | Incremental RotBoost algorithm: An application for spam filtering |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |