WO2008145031A1

WO2008145031A1 - Method and system for judging of the inportance of article, and sliding window

Info

Publication number: WO2008145031A1
Application number: PCT/CN2008/070600
Authority: WO
Inventors: Liang Dong; Rongfang Shao
Original assignee: Tencent Technology (Shenzhen) Company Limited
Priority date: 2007-05-31
Filing date: 2008-03-27
Publication date: 2008-12-04
Also published as: CN101071419A; CN100520767C

Abstract

A method for judging of the importance of an article, including: a sliding window start sliding take the article jumping-off point as a starting point, the sliding window not collects the slid words again, when the words of the sliding window collected reach a preset number, the total number of the slid words is recorded, reset the starting point of slide, and continue the sliding process from the reset the starting point of slide; the largest number of value is obtained from the number of the sliding window recorded, and the importance of the article is judged based on the largest number of value. At the same time, the invention also relates to a system for judging of the importance of the article, and a sliding window.

Description

Method and system for judging the importance of an article and sliding window

Technical field

The present invention relates to the field of network retrieval, and in particular to a method and system for determining the importance of an article and a sliding window. Background of the invention

Retrieving relevant articles using keywords on the network is one of the important ways to share network resources. Due to the rich network resources, a search keyword often corresponds to a large number of articles, which requires the network system to judge the importance of each article, so that the relatively important articles are displayed in front of the search results, which will be relatively unimportant. The article is displayed at the back, allowing users to read more important articles first, saving users time.

The current common practice is to judge the importance of the vocabulary in the article. If an article has a rich vocabulary, it means that it has something to say, and it belongs to an important article; on the other hand, if the article is only a few words repeated in the whole article or in part, the vocabulary is poor, indicating that it has nothing to say, and it is an unimportant article. The prior art is based on the method of word frequency statistics to judge the importance of an article.

Referring to Figure 1, a flow chart of a method for judging the importance of an article on the network is described below. The specific steps are as follows.

Step S101: Separate adjacent words in the article sentence by using a space.

E.g:

Before the word segmentation: I watched a TV series today, and I was touched by a boy in the drama, ......

After the word segmentation: I watched a TV series today and was touched by a boy in the drama. Step S102: Count the word frequency of each word, that is, the number of times the word appears in the article. E.g:

Today, 5 times; see, 35 times; out, 100 times; TV series, 10 times .....

Step S103: Calculate and determine whether the number of occurrences of the words satisfies a preset condition. If not, the article is considered to be relatively important; if satisfied, the article is considered to be relatively unimportant.

The preset conditions can be:

1) The total number of words is less than 5;

2) The word frequency of the most single word is greater than 30% of the total word frequency, or the most frequently occurring 5% word frequency is greater than 50% of the total word frequency, or the most frequent 20% word frequency is greater than 80% of the total word frequency;

3) The average word frequency is more than 5.

For example, if the average word frequency of the above word "article" exceeds 5, the article is considered to be relatively unimportant. The above method judges the importance of the article by counting the word frequency of each word in the article, but the word frequency reflects the global characteristics of the article and does not reflect the local characteristics of the article. For example: If an overall vocabulary is rich, but the local vocabulary is poor, the existing judgment method is easy to misjudge the article as an important article. Therefore, the existing judgment methods cannot effectively judge articles such as the overall vocabulary but the local vocabulary is poor, which brings inconvenience to users. Summary of the invention

The embodiment of the invention provides a method for judging the importance of an article, which can more effectively identify an article and is convenient for the user to use.

The embodiment of the invention further provides a system for judging the importance of an article, which can more effectively identify the article and is convenient for the user to use.

The embodiment of the invention further provides a sliding window for traversing the article on the network, and can effectively obtain related parameters of the vocabulary richness in the article.

Embodiments of the present invention relate to a method for determining the importance of an article, including: sliding a window from The initial sliding start point of the article starts to slide, and the sliding window does not repeatedly collect the slipped words. When the words collected by the sliding window reach a preset number, the total number of words that have been slipped is recorded, and the sliding is reset. a starting point, and continuing the sliding process from the resetting starting point of the sliding;

The maximum number of values is obtained in the number of the sliding window records, and the importance of the article is determined according to the obtained maximum number of values.

The embodiment of the invention further relates to a system for determining the importance of an article, comprising: a sliding window, a maximum value obtaining unit and an importance determining unit, wherein the sliding window comprises a word collecting unit, a word recording unit and a starting unit:

The starting unit is configured to control the sliding window to start sliding from an initial sliding starting point of the article;

The word collecting unit is configured to perform non-repeating collection of the slipped words, and when the collected words reach a preset quantity, send startup information to the activation unit and the word recording unit; Resetting the sliding starting point after starting the information, and starting the sliding window to continue the sliding process from the reset starting point of the sliding;

The term recording unit is configured to record, after receiving the startup information, the total number of words that the sliding window has slid;

The maximum value obtaining unit is configured to obtain a maximum quantity value in the quantity recorded by the word recording unit;

The importance judging unit is configured to judge the importance of the article according to the obtained maximum quantity value.

The embodiment of the invention further relates to a sliding window, the sliding window comprising a word collecting unit, a word recording unit and a starting unit:

The starting unit is configured to control the sliding window to start sliding from an initial sliding starting point of the article; The word collecting unit is configured to perform non-repeating collection of the slipped words, and when the collected words reach a preset quantity, send startup information to the activation unit and the word recording unit; Resetting the sliding starting point after starting the information, and starting the sliding window to continue the sliding process from the reset starting point of the sliding;

The term recording unit is configured to record, after receiving the activation information, the number of words that the sliding window has slipped. BRIEF DESCRIPTION OF THE DRAWINGS

1 is a flow chart of a method for judging the importance of an article on a network;

2 is a flowchart of a method for determining the importance of an article according to a first embodiment of the present invention; FIG. 3 is a flowchart of a method for determining the importance of an article according to a second embodiment of the present invention; FIG. 5 is a schematic diagram of a system for determining the importance of an article according to a fourth embodiment of the present invention; FIG. 6 is a schematic diagram of a system for determining the importance of an article according to a fifth embodiment of the present invention; A schematic structural diagram of a sliding window provided by a sixth embodiment of the present invention. Mode for carrying out the invention

The above described objects, features, and advantages of the embodiments of the present invention will be more apparent from the embodiments of the invention.

The embodiment of the present invention uses a preset sliding window to start sliding from the initial sliding starting point of the article, and the sliding window does not repeatedly collect the slipped words. When the words collected by the sliding window reach a preset number, the number of slipped words is recorded. , reset the starting point, continue to slide, the sliding window repeats the process until the entire article is slid; the largest number of records is obtained in the number of sliding window records, and the importance of the article is determined according to the size of the number.

Wherein, the initial sliding starting point can be the starting point of the article, the ending point of the article, or the advance The set article is scheduled to slide. Moreover, the article can be either a web article stored on the network or a local article that is offline with the network.

Preferably, after determining the importance of the article, the importance of the article is further output to the network or locally.

Referring to FIG. 2, a flow chart of a method for determining the importance of an article according to a first embodiment of the present invention is provided. The specific steps are as follows.

Step S201: Start sliding with the preset sliding window starting from the beginning of the article.

A sliding window is provided, which includes a left border, a right border, and a database, and the database does not repeatedly store the words between the left and right borders. The database can contain up to a preset number of words. For example: The preset number is preferably 6.

Use this sliding window to start with the first word at the beginning of the article. When sliding, the left edge of the sliding window does not move, and the right border slides to the right.

Step S202: The sliding window does not repeatedly collect the slipped words.

The sliding window does not repeatedly collect the slipped words during the sliding process, that is, the collected words do not overlap each other. The sliding window stores the collected words in the database.

Step S203: When the words collected by the sliding window reach a preset number, the number of words slipped is recorded.

As the sliding window slides, the amount of words collected by the sliding window increases continuously, and when the amount of words collected by the sliding window reaches a preset number, the total amount of words that the sliding window slides from the starting point is recorded.

Step S204, reset the starting point, and continue to slide until the entire article is slid.

Slide the window to reset the starting point, empty the collected words, and continue to slide. The starting point for resetting may be the last word collected by the sliding window, or the next word or previous word of the word, or the first few words of the word. When the words collected by the sliding window reach the maximum again, the sliding window records the total number of words that have slipped again, and clears the collected words, resets the starting point again, and continues to slide until the entire article is slid.

When the words collected by the sliding window reach the preset number, the right border stops moving, and the left border moves to the right to the word position as the starting point.

Step S205: Acquire the largest number of values in the number of records in the sliding window.

After the sliding window slides through the entire article, the number of slipped words per record is obtained for the sliding window, and the largest one is extracted from the above number.

Step S206: Determine the importance of the article according to the magnitude of the quantity value.

The quantity value obtained is used as the basis for judging the importance of the article. If the value is large, the importance of the article is relatively low; if the value is small, the article is relatively important.

If the article's overall vocabulary is rich but the local vocabulary is poor, there are a lot of repeated words. In the embodiment of the present invention, when the sliding window slides over the portion, the slipped window is not repeatedly collected due to the sliding window. Under the condition that the number of collected words is fixed, the number of words sliding through the sliding window is relatively larger than other parts. The number of slipped words recorded in the slide is relatively large, which will become the basis for judging the importance of the article. In this way, the embodiment of the present invention adopts a correlation value that reflects the most inferior part of the article vocabulary, and serves as a basis for judging the importance of the article, and can effectively judge the article with rich overall vocabulary but partial vocabulary, which is convenient for the user to use.

In the above embodiment, the technical scheme of starting the sliding from the beginning of the article as a starting point is described in detail. In fact, the embodiment of the present invention can also start from the beginning of the article and start to slide toward the beginning of the article. The embodiment of the present invention does not limit the manner of the sliding starting point.

In the embodiment of the present invention, adjacent words in the sentence sentence are spaced apart by a space before the sliding window starts to slide, so that the sliding window can recognize the words in the sliding. And the embodiment of the present invention can also judge the importance of the article by setting a predetermined value.

Referring to FIG. 3, a method flow for determining the importance of an article according to a second embodiment of the present invention is provided. Figure, the specific steps are as follows.

Step S301: Space adjacent words in the article by spaces.

For example, the beginning of the article is: "The flowers are also beautiful, the grass is flourishing, the scenery here is good", using spaces to separate the adjacent words: "Flowers are also beautiful grass is also flourishing here is good scenery."

Step S302: Start sliding by using a preset sliding window starting from the beginning of the article.

The sliding window includes a left border "[", a right border "]" and a database, and the database stores the words between the left and right borders without repeating. The database can contain up to a preset number of words. For example: The preset number can be 6.

For example, the position of the sliding window at this time is: "[] The flower is also beautiful, the grass is also flourishing, the scenery is good." When starting to slide, the right border of the sliding window "] " starts to move right.

Step S303: The sliding window does not repeatedly collect the slipped words.

For example, the position of the sliding window is: "[Flower] Also beautiful grass is flourishing here is good scenery", when the word "flower" is judged not to overlap with the words collected in the database, the word "flower" is collected; the position of the sliding window is moved. For: "When the flowers are beautiful, the grass is good," when the word "also" is repeated with the words "also" in the database, the word "also" is no longer collected.

Step S304: When the words collected by the sliding window reach a preset number, the number of words slipped is recorded.

For example, when the position of the sliding window moves: "[Flower is also beautiful and grass is flourishing here] When the scenery is good", the number of words collected in the sliding window is 6, and when the preset value is reached, the number of words that the sliding window has slipped is recorded, that is, 7 words such as "flower, also, beautiful, grass, also, lush, here", record the value 7.

Step S305, setting a word collected at the last sliding window as a starting point, and clearing the sliding The words collected in the window continue to slide until the entire article is slid.

For example, starting with the word "here", the left border of the sliding window moves to the right border, and the right border continues to slide to the right. The position of the sliding window is: "Flower is also beautiful and grass is also flourishing here [] Good scenery."

When the words collected by the sliding window reach the maximum again, the sliding window records the total number of words that have slipped again, and clears the collected words, and slides again until it slides over the entire article.

Step S306: Obtaining the largest number of values in the number of records in the sliding window.

For example, the number of first records is 7, the second is 8, the third is 12..., and when compared, 12 is the largest, then 12 is the maximum number of words.

Step S307: Compare the quantity value with a predetermined value, if less than, determine that the article is an important article.

For example, a predetermined value of 16, comparing 12 to 16 and 12 < 16, this article is a relatively important article.

In this embodiment, the article is judged to be an important article or a non-important article by means of a predetermined value, and the importance of the article can be directly judged accurately, which is convenient and practical.

In the embodiment of the present invention, the number of words slipped can be recorded by calculating the length of the sliding window, and the retrieved articles can be sorted according to the maximum number of acquired words, so that the articles are arranged in order of importance. The length of the sliding window is the amount of words contained between the left and right borders of the sliding window.

Referring to FIG. 4, a flow chart of a method for determining the importance of an article according to a third embodiment of the present invention is provided. The specific steps are as follows.

In step S401, adjacent words in the article are separated by spaces. For example, there is a paragraph in the middle of the article: "I am so happy today, I am so happy, I am so happy, I am very happy." I used spaces to separate the adjacent words: "I am so happy today, I am so happy, I am very happy."

Step S402: Start sliding by using a preset sliding window starting from the beginning of the article.

For example, the position of the sliding window at this time is: "[] Good day, happy, ah, happy, really happy, very happy." When you start to slide, the right edge of the sliding window "" " starts to move to the right.

At this point, the length of the sliding window is 0, and the word collected by the sliding window is 0.

Step S403: The sliding window does not repeatedly collect the slipped words, and records the number of words that the sliding window has slipped.

For example, the position of the sliding window is: "[Today] I am so happy, I am so happy, I am very happy." When judging the word "Today" does not overlap with the words collected in the database, collect the words "Today"; The position is: "[I am so happy today] I am happy, I am very happy." When the judgment word "good" is repeated with the word "good" in the database, the word "good" is no longer collected.

At this point, the length of the sliding window is 5 and the sliding window collects 4 words.

Step S404: When the words collected by the sliding window reach a preset number, the number of words slipped is recorded.

For example, the position of the sliding window is: "[Good day, happy, happy, really happy"], at this time, the sliding window has a length of 10, and the sliding window collects the word 6.

Step S405, setting a word collected at the last sliding window as a starting point, and clearing the sliding The words collected in the window continue to slide until the entire article is slid.

For example, the left border of the sliding window moves to the right border, and the right border continues to slide to the right. When the words collected by the sliding window reach the maximum again, the sliding window records the total number of words that have slipped again, and clears the collected words. Words, slide again again, until you slide over the entire article.

Step S406: Obtaining the largest number of values in the number of records in the sliding window.

For example, the length of the sliding window recorded for the first time is 10, the second time is 11, the third time is 18..., and when 18 is the largest, 18 is used as the maximum number of words.

Step S407, the quantity value is used as the weight of the article importance, and the importance degree of the article is determined according to the weight value.

For example, 18 is used as the weight of the article's importance, compared with the weights of other articles, and sorted by weight.

In this embodiment, by sorting by weight, the frontmost of the most important articles can be arranged, and the rest are arranged in order of importance, which is very convenient for the user to use. Moreover, the embodiment does not need to set relevant values for judging whether the article is important, and can more objectively reflect the importance of the article.

Based on the above method for judging the importance of an article, the embodiment of the present invention provides a system for judging the importance of an article, which can effectively identify an article with a rich overall vocabulary but a lack of local vocabulary, and is convenient for the user to use.

5 is a schematic diagram of a system for determining the importance of an article according to a fourth embodiment of the present invention, including a sliding window 51, a maximum value obtaining unit 52, and an importance determining unit 53. The sliding window 51 includes a word collecting unit 511 and a word recording unit. 512, and a starting unit 513.

The activation unit 513 controls the sliding window 51 to start sliding from the initial sliding start point of the article. Wherein, the initial sliding starting point can be the starting point of the article, the ending point of the article, or the advance The set article is scheduled to slide.

The word collecting unit 511 does not repeatedly collect the slipped words, and transmits the activation information to the activation unit 513 and the word recording unit 512 when the collected words reach the preset number. The startup unit 513 resets the starting point after receiving the startup information, and starts the sliding window 51 until the entire article is slid. The preset number may preferably be 6.

The word recording unit 512 records the number of sliding windows 51 sliding over the words. The term recording unit 512 can record the total number of words that the sliding window has slid after receiving the activation information. The vocabulary recording unit 512 can also continuously record the total number of words that the sliding window has slid during the sliding process, and end the recording of the slid after receiving the activation information, and record the total number of words that have finally slipped.

The maximum value acquisition unit 52 acquires the largest value among the number recorded by the word recording unit 512, and transmits it to the importance judging unit 53.

The importance judging unit 53 judges the importance of the article based on the maximum number value.

In order to facilitate the word collecting unit 511 and the word recording unit 512 to collect and record words, the embodiment of the present invention can break up the words in the article by the word breaking unit.

In the above embodiment, the technical solution in which the sliding window 51 starts to slide from the beginning of the article is described in detail. In fact, the sliding window 51 can also start from the beginning of the article and start to slide toward the beginning of the article. The embodiment of the present invention does not limit the manner in which the sliding starting point is agreed.

Referring to FIG. 6, a schematic diagram of a system for determining the importance of an article according to a fifth embodiment of the present invention includes a sliding window 51, a maximum value obtaining unit 52, an importance determining unit 53, and a word breaking unit 54, and the sliding window 51 includes word collection. The unit 511, the word recording unit 512, and the activation unit 513.

The word breaking unit 54 spaces adjacent words in the article with spaces.

The sliding window 51, the maximum value obtaining unit 52, and the importance determining unit 53 are in this embodiment. The functions and functions in the examples are the same as those in the embodiment shown in FIG. 5 and will not be described again.

Preferably, the system may further comprise an importance level output unit. The importance level output unit is connected to the importance judgment unit 53. The importance level output unit is configured to output the importance of the article to the network or locally after the importance judging unit 53 judges the importance of the article.

In the embodiment of the present invention, the entire article is traversed through a sliding window, and the relevant parameters of the vocabulary richness in the article can be effectively obtained.

Referring to FIG. 7, a schematic structural diagram of a sliding window 51 according to a sixth embodiment of the present invention includes a word collecting unit 511, a word recording unit 512, and a starting unit 513.

The start unit 512 controls the sliding window 51 to start sliding at the beginning of the article or at the end of the article.

The word collecting unit 512 performs non-repeating collection of the slipped words, and when the collected words reach the preset number, sends the activation information to the initiating unit 513 and the word recording unit 512; the last word collected by the initiating unit 513 by the word collecting unit 511 As a starting point, restart the sliding window 51 until it slides through the entire article. The preset number is preferably 6.

The word recording unit 512 records the number of sliding windows 51 sliding over the words.

The sliding window 51 further includes a left boundary and a right boundary. When sliding, the right boundary moves rightward from the starting point. When the words collected by the word collecting unit 512 reach a preset number, the right boundary stops moving, and the left boundary moves to the right, until the left and right borders Contains only one word.

In summary, the embodiment of the present invention uses a preset sliding window to start sliding at the beginning of the article or the end of the article, and the sliding window does not repeatedly collect the slipped words, and the words collected by the sliding window reach a preset number. When recording, the number of words slipped over, reset the starting point, continue to slide, the sliding window repeats the process until the entire article is slid; the largest number of records is obtained in the number of sliding window records, and the article is judged according to the size of the number The importance of it. If the article's overall vocabulary is rich but the local vocabulary is poor, there are a lot of word duplication. The phenomenon. In the embodiment of the present invention, when the sliding window slides over the portion, the slipped window is not repeatedly collected due to the sliding window. Under the condition that the number of collected words is fixed, the number of words sliding through the sliding window is relatively larger than other parts. The number of slipped words recorded in the slide is relatively large, which will become the basis for judging the importance of the article. Thus, the embodiment of the present invention adopts a correlation value that reflects the most inferior part of the article vocabulary as a basis for judging the importance of the article.

Compared with the method for counting the word frequency in the article in the prior art, the embodiment of the present invention can effectively judge the article with rich overall vocabulary but partial vocabulary, which is convenient for the user to use.

The method, system and sliding window for determining the importance of an article provided by the embodiment of the present invention are described in detail above. The foregoing description explains the principles and embodiments of the embodiments of the present invention by using specific examples. The description of the above embodiments is only for facilitating understanding of the method and the core idea of the embodiments of the present invention. Meanwhile, it is generally known to those skilled in the art. The present invention is not limited to the embodiments of the present invention. The details of the present invention are not limited to the embodiments of the present invention.

Claims

Claim

A method for determining the importance of an article, comprising:

The sliding window starts to slide from the initial sliding starting point of the article, and the sliding window does not repeatedly collect the slipped words, and when the words collected by the sliding window reach a preset number, the total number of words that have been slipped is recorded, Resetting the slide start point and continuing the above sliding process from the reset start point of the slide;

2. The method of claim 1, further comprising: spacing adjacent words in the article with spaces.

3. The method of claim 1, wherein the sliding window does not repeatedly collect the slipped words comprises:

The sliding window determines whether each of the slipped words is repeated with the collected words, and if not, the words are collected, and if repeated, the words are not collected.

The method of claim 1, wherein the resetting the sliding start point comprises:

Setting a word collected lastly by the sliding window as a sliding starting point, or setting a next word of the last collected word as a sliding starting point, or setting a last word of the last collected word as a sliding starting point;

Empty the words collected by the sliding window.

The method according to any one of claims 1 to 4, wherein the determining the importance of the article based on the maximum number of values comprises:

The obtained maximum quantity value is compared with a predetermined value, such as less than the predetermined value, to determine that the article is an important article.

The method according to any one of claims 1 to 4, wherein said The maximum number of values to determine the importance of the article includes:

The maximum quantity value is used as the weight of the article importance, and the importance of the article is determined according to the weight value.

The method according to any one of claims 1 to 4, characterized in that the initial slip starting point is a starting point of the article, an ending point of the article, or a predetermined predetermined sliding point of the article.

The method according to any one of claims 1 to 4, characterized in that the article is a web article or a local article that is offline with the network.

The method according to any one of claims 8 to 8, wherein the method further comprises: after determining the importance of the document:

The importance of exporting the article to the network or locally.

The method according to any one of claims 1 to 4, wherein in the method, the continuing the sliding process from the resetting start point of the sliding comprises:

The above sliding process is continued from the resetting start point of the slide until the entire article is slid.

11. A system for determining the importance of an article, comprising: a sliding window, a maximum value obtaining unit, and an importance determining unit, the sliding window comprising a word collecting unit, a word recording unit, and a starting unit:

The word recording unit is configured to record that the sliding window has been slid after receiving the startup information The total number of words used;

12. The system of claim 11 wherein the system further comprises a word breaking unit for spacing adjacent words in the article with spaces.

13. The system of claim 11 or 12, wherein the system further comprises an importance level output unit;

The importance level output unit is configured to output the importance degree of the article to the network or the locality after the importance judging unit judges the importance of the article.

14. A sliding window, wherein the sliding window comprises a word collecting unit, a word recording unit and a starting unit:

The term recording unit is configured to record, after receiving the activation information, the number of words that the sliding window has slipped.

The sliding window according to claim 14, wherein the sliding window further comprises a left boundary and a right boundary, and when sliding, the right boundary is shifted right from the starting point, when the words collected by the word collecting unit When the preset number is reached, the right border stops moving, and the left border moves to the right until a word is included between the left and right borders.

The sliding window according to claim 14 or 15, wherein the word collecting unit is configured to continue the sliding process from the reset starting point of the sliding until the entire article is slid.