WO2002097667A2

WO2002097667A2 - Visual and interactive wrapper generation, automated information extraction from web pages, and translation into xml

Info

Publication number: WO2002097667A2
Application number: PCT/IB2002/003036
Authority: WO
Inventors: Robert Baumgartner; Sergio Flesca; Georg Gottlob; Marcus Herzog
Original assignee: Lixto Software Gmbh
Priority date: 2001-05-31
Filing date: 2002-05-28
Publication date: 2002-12-05
Also published as: US7581170B2; US20050022115A1; EP1430420A2; WO2002097667A8

Abstract

A method and a system for information extraction from Web pages formatted with markup languages such as HTML [8]. A method and system for interactively and visually describing information patterns of interest based on visualized sample Web pages [5,6,16-29]. A method and data structure for representing and storing these patterns [1]. A method and system for extracting information corresponding to a set of previously defined patterns from Web pages [2], and a method for transforming the extracted data into XML is described. Each pattern is defined via the (interactive) specification of one or more filters. Two or more filters for the same pattern contribute disjunctively to the pattern definition [3], that is, an actual pattern describes the set of all targets specified by any of its filters. A method and for extracting relevant elements from Web pages by interpreting and executing a previously defined wrapper program of the above form on an input Web page [9-14] and producing as output the extracted elements represented in a suitable data structure. A method and system for automatically translating said output into XML format by exploiting the hierarchical structure of the patterns and by using pattern names as XML tags is described.

Description

VISUAL A I ND INTERACTIVE WRAPPER GENERATION, ' AUTOMATED

INFORMATION EXTRACTION FROM WEB PAGES, AND TRANSLATION INTO

\ XML

I. INTRODUCTION

A. RELATED APPLICATIONS

[01] The present application claims priority from the copending U.S. Provisional

Applicaion Serial No. 60/294,213, having the same title filed May 31, 2001

13. BACKGROUND

FIELD OF INVENTION

[02] This disc osure teaches techniques related in general to the field of information processing. Mors particularly, the teachings relate to methods, systems and computer- prograni product's for information extraction' from Web pages, to the construction of wrappers (i.e. extraction programs), based on example Web pages, and to the transformation of "relevant" parts of HTML documents into XML.

BASIC CONCEPTS, TERMINOLOGY, AND INTRODUCTION

[03] The World Wide Web (abbreviated as Web) is the world's largest data repository and information retrieval system. In this environment, client machines effect transactions to Web servers ising thej Hypertext Transfer Protocol (HTTP), which is an application protocol usually providiηg user access to files formatted in a standard page description language known as Hypen:ext Markup Language (HTML). HTML provides basic document formatting (and some logic markup) and allows the developer to specify "links" to other servers and documents. In the Internet paradigm, a network location reference to a server or to a specific Web resource at a server (for example a Web page), is identified by a so-called Uniform Resource Locator (URL) having a well-defined syntax for describing such a network location. The use of an (HTML-compatible) browser (e.g. Netscape Navigator, Microsoft Internet Explorer, Amaya or Opera) at a client machine involves the specification of a link by the means of an URL. The client then makes a request to the server (also referred to as "Web site") i entified by the link and receives in return an HTML document or some other object of a known file ty je. Simple and less sophisticated browsers can easily be written in a short time and wit little effort in (object-oriented) programming languages such as Java, where powerful program libraries are available that already contain modules or classes providing the

[07]

[08]

[14]

[18] I These approaches are discussed in detail. a) Wrapper progra Im ' ming ' languages and environments

[31] Note that there are Web change monitoring and notification tools that have much better information extraction capabilities because they rely on a separate and independent supervised wrapper generator. An example is the continual query system OpenCQ (L.Liu, C.Pu, and Wei Tang, "Continual Queries for Internet Scale Event-Driven Information

! istructurqd and Semistructured Data from Text Documents", Proceedings of the [33]

learning 'structured data and translation into XML are given. This works very nicely on plain

that some explicit description (or tagging) of the relevant data is already contained in the sample document (in form of names of section headers) or at least unambiguously determined i I by simp e attribi.tes of headers (such as their font size), which is very often not the case. There is no possibility of locating desired extraction items by more complex contextual conditions. Moreover, no user-defined complex patterns can be created that are not already determined by basic formatting issues and paragraph headings. Accordingly, the constructed wrappers are not robust, except for sets of input documents having exactly the same formatting structure. No translation into XML is offered.

A I nother e !'arly approach of supervised wrapper generation was developed in "Wrapper

Generation for Web Accessible Data Sources" by J.R. Gruser, L. Raschid, M.E. Nidal, and L.Brigh: at the Proceedings of CoopIS 1998. A translation to a semistructured data format can be generated.

The folk wing systems are more advanced and can be considered the prior art most related to our owp invention. J ₍

The XWRAP system by Ling Liu, Carlton Pu and Wei Han (L. Liu, C Pu, and W. Han

"XWRi⁵ -P: An XJML-enabled Wrapper Construction System for Web Information Sources",

Proceedings of th i e 16th Inte Irnational Confer i elnce onl. Data Engineering San Diego CA,

February 28 - March 3,2000, LEEE Computer Society Press, pp. 611-621, 2000). See also the

will facilitate the wrapper code generation. After we get all the extraction rules, XWrap can compile these rules into a runable program."

[44] The main! drawbacks of this system arc:

I

• The limited expressive power of its pattern definition mechanism. First, the system lacks sufficiently powerful semi-automatic generalization mechanisms that allow a j' user to specify several similar patterns at once while marking only a single pattern of he desired type. Secondly, the system lacks sufficient visual facilities for imposing inherent (internal) or contextual (external) conditions to an extraction pattern, e.g.

' extraction pattern X should appear after a recognized instance of the extraction ttern Y but not before an instance of pattern Z", and so on. Thirdly, the division into he two levels of description "region''! and "token" and the automatic hierarchical

I I . . ' i structure extractor severely limit the ways to define extraction patterns. For example,

d h ll b h i i paths generated by the wizard and adding further HEL language constructs. The expressive power of the set₍ of queries that can be visually generated (without hand coding) is extremely limited. This implies that a user of the W4F system is required to have both expertise of the HEL language and expertise of HTML. This, in turn, means that, notwithstanding the visual

that whi e Lixto 3 discussed in detail, it is only an example implementation and should not be construed, to restrict the scope of the claims in any way,

[55] Further, computer-program products including computer-readable media with instructions that implement the systems and methods disclosed, completely or partially, are also con ;emplated as being within the overall scope of the disclosed teachings. It should be noted that the media could be anything including but not limited to RAMs, ROMs, hard disks CD s, tapes , floppy disks, Internet downloads, etc. In short, any medium that can fix all or a sublet of the¹ instructions, even for a transient period of time, is considered as a computer- readable media for the purposes of this disclosed teaching. Further the type of computer that can imp ement is also not restricted to any particular type, but includes personal computers, workstations, mainframes and the like. Also, it can be implemented on a stand-alone compute: r or in a distributed .fashion across al network, including but not limited to, the Internet.

II RIIEF DESCRIPTION OF THE DRAWINGS

[56] Architecture Overview: A diagr riam depicting the overview of the Lixto architecture.

[57] ' Architecture Overview: Using the extractor as stand-alone program. [58] Pattern- Filter-Diagram: Logical structure of a sample pattern illustrating the general ttern structure,

[59] Packa ge Architecture of preferred embodiment: Package structure of actual implemesntation.

[60] 5. Lixto Screenshots: Vectorized screenshots showing Lixto at work. [61] . Lixto Screenshots: Vectorized screenshots showing Lixto at work (part 2). [62] . Table of em ipiricatl results: Evaluation of L ' i •xto w.r.t. different sample web sites. [63] 8. Example of an HTML tree: Illustiation of a possible way to parse an HTML document.

[64] 9. Pattern EER: The entity relationship diagram of pattern and filters. [65] 0. Filtej EER: The entity relationship diagram of filters and their constituents. [66] 1. Rule Evaluation: Algorithm of evaluating an Elog rule. [67] 2. subsequence evaluation: Algorithm of evaluating the "subsequence" predicate. [68] 3. before evaluation: Algorithm of evaluating the "before" predicate for tree filters. [69] 4. before string evaluation: Algorithm for evaluating the "before" predicate for string filters.

target pi .ttern instance is allowed to appear. For example, a "before" condition might

[114] The XML Translation Builder, another interactive module of the visual builder (Fig.l), is responsible for supporting a wrapper designer during the generation of the so-called XML translation program. One key issue is that, by default, pattern names that are chosen by the designer during the pattern design process are taken as output XML tags and that the

I I hierarc y of ex xacted pattern instances (which is always a proper tree-like hierarchy) determines the structure of the output XML document. Thus, in case no specific action is taken I y the de signer, a standard translation of the 'extracted pattern instances into XML will be perf srmed without any need of further interaction. However, Lixto also offers a wrapper designer the po ssibility to modify the standard XML translation in the three following ways:

The designer can rename some patte 3rrns with the effect that a new name instead of the actual pattern name appears as tag name in the XML translation.

The designer can suppress some patterns from the translation. In this case, instances of non-suppressed patterns that are children of a suppressed pattern instance /will appear in the XML translation as children of the closest non-suppressed ancestor of I. For example, assume a designer wants to construct a wrapper that extracts all records from the third table of some Web page. She may first construct a pattern table that precisely identifies the third table. Then she may define a pattern record using a filter whose parent-p attern is the table pattern (with the effect that only records from the document's third table are identified as instances of the record pattern). In this case the pattern hierarchy is of the form document <- table <- record, where each arrow symbolizes a reference to a parent pattern jWhile the pattern table has an essential

(but in sense auxiliary) role in the i definition of extraction items, the designer may decide to suppress it in the XML output. Then the XML output document will display a hierarchy of the form document < record , where each record pattern instance is a child of the document instance.

For each pattern the designer can choose the set of attributes that should be carried over to the output XML document. For example, font, color, or positional attributes of HTML items can either be carried over to the XML output or can be suppressed in case thely are not of interest.

[115] The desijred modalities of the XML translation (as described above) are determined during the wra per design process by a very simple and very user-friendly graphical interface and are stored, in form of a so called "XML translation scheme" that encodes the mappir g between extraction patterns and tine XML] schema in a suitable form.

pattern, building and program construct on phases, and as a stand-alone program (Fig.2)

and las r childre are tables, The sequence is jtoo general to match the desired target only, [162]

[163]

[164]

[165] More specifically, each filter refers to a parent pattern from which it has to extract the desired information and specifies how to retrieve this information from a parent pattern inst nce. Essen ially a filter specifies the type of the desired information and how to distinguish the desired information from other similar information, i.e. it specifies some additional conditions that the desired information must fulfill. All the conditions in a filter are interpreted conjunctively in the sense that a portion of a Web page satisfies a filter if and only if it satisfies all the conditions expressed in these filter. For a sample pattern see Fig. 3.

[166] Fig. 9 shows an extended entity-relationship diagram (EER) describing a data structure for representing and storing the definitiori of information patterns. For an explanation of various notations used in entity relationship diagrams refer to Bernhard

ThaJUnem, " Ent.ty Relationship Modeling ^foundations of Database Technology", Springer, ISBN 2540654^' '04. We recall that there are alternative ways to describe this data structure, such as class diagrams or logical represent, ion.

[167] A pattern [901] can be a document pattern [903] or a source pattern [904], and is characterized by a unique name [902].

[168] A document pattern represents a set of similarly structured Web pages, and is defined

[916] by one or more document filters [915]. Document filters define a way to retrieve documents from the text (in particular, from given links) of other documents. Essentially, evaluating a document filter (which takes as parent pattern a string pattern instance or a constant string representing the URL) the corresponding documents are fetched and their doc ment tree constructed. With document filters, patterns can be reused on further pages, and L recursive behavior can be exhibited.

[169] A source pattern describes parts or e ements of (HTML) documents. It can be a tree pattern [907] oi a string pattern [906]. Tree patterns define the information of interest using the tree representation of (HTML) documents, whereas string patterns represent (HTML) doc ments or parts of those as plain text disregarding formatting tags.

[170] A free p ttern is defined by one or more tree filters [912] (relation [909]), whereas a string p attern is] defined by one or more string filters [911] (relation [908]). Filters [910] can be free, string or document filters; string filters can be distinguished in text filters [913] and attribu e filters [914] - both are described below in more detail. Tree and string filters extract information (relation [905]) from instances of a pattern (either a source or a document pattern I. An attribute filter can define a string pattern only and works on tree and document pattern s only. A text filter can define a string pattem only and can refer to all kind of patterns

[175] xamplte of yahoo 'auction pages: Ϋjahoo Auction pages are (at least were at the time these li ιes were written) structured as follows - each item is described in one table row, however, , there A e two different tables and headings, namely "Featured Auctions" and "All auctions '. One can define "record" by referring to two different parent patterns, "record" contains two filters, where one refers to "tablefeatured" (1515) and the other refers to

I

"tableal I" (1514) as parent pattern.

[176] liters [ 001] are mainly characterized by the way they identify information being exfracte]d. There are three different kind of filters:

Tree filters [1003] extract free regions from other tree regions. They are defined by 1020] a 'tree region extraction definition [1015]. Pattern instances identified by a free region e:ι traction definition can be further filtered out by imposing additional tree onditiors. Tree\region extraction definitions specify the characteristics of the simple nr gener tl tree regions being extracted, hi particular they specify how to identify the

;he roo i t elements off these trees and, if the desired free regi •ons are not subtrees (in the spirit o I f our ab ^!ove d 1efinition), the cha !racteri Istics of the first and last child that belongs ro the region.

String filters [1002], which are further subdivided into attribute filters and text filters:

Attribute filters [1004] extract information from attribute values. They must id Ientify ' an a 1ttribute designator whose information has to be extracted and impose some further conditions (as explained later). For this reason, attribute filters are defined by [1021] attribute extraction definitions [1016]. An attribute filter extracts the values of one kind of attribute designator (or optionally, of more than one kind, e.g. defined via regular expressions).

Text filters [1005] always extract a substring of the parent string pattern instance, but they are also defined by [1017] a string extraction definitions [1014] that can further restrict the characteristics of the substring being extracted. Al string extraction 'definition is essentially the specification of a language that defines all the substrings that can be extracted.

[177] e give three easy examples of when to use which kind of filter based upon the exampl page used above whose HTML tree is depicted in Fig.8:

defihitijon of chjid patterns is not clearly defrned). [267] Patterns (and their filters) are restricted in their use of parent patterns in the following

I manner In case of a free pattern (all filters are tree exfraction rales), all parent patterns ofthe filters contained in this pattern need to be tr 3e patterns or document patterns. In case of a string pattern, all parent patterns ofthe filters contained in this pattern can either be string patterns or tree patterns. In case of a document pattern, all parent patterns ofthe filters contain d in the document pattern have to be string patterns.

[268] ' Hn case o f a homogeneous pattern, i.e. all filters refer to the same parent pattern, the notion ώf "pareiit pattern" can be associated with a pattern rather than with its filters. Im fact,

I I i in a more restrictive embodiment ofthe disclosed invention, where only homogeneous patterns: are allowed, the parent pattern is always specified together with a pattern and not for its filters

[269] As for standard datalog rales, a pattern acts like a disjunction of rale bodies: to be extracted, i.e. to be an instance of this pattern, a target needs to be in the solution set of at least one rale. Adding rales usually matches more targets, while adding constraints in the rule bodies removes unwanted matches.

[270] The extracted targets of a pattern can be minimized. Urn our preferred embodiment we chose t< consider only thos_je targets not contained m any other target of tine same pattern instance. Minimization can be carried out even with recursive patterns. However, this type of minimisation is restricted tp instances of the same parent-pattern instance. Hence, in example [1521,11522] of Fig.15, where a nested tables are to be extracted, minimization does not cut off the interior tables, because levery table has a different table as parent pattern.

[271] As those skilled in the art can easily recognize, instead of minimization various other alternative ways' of simplifying or restricting the legal output pattern instances can be used. It is conceivable that in certain contexts some other methods may be more appropriate than the describ d minimization. Lixto is open for incorporating such other methods in alternative embodiments.

[272] Note that the reason why a recursive ('approach was taken in the second example of

Fig.15 s precisely the use o I f free path mini 1m 1ization I . In fact, starting from the document root, the free] path ". * table", due to minimization, matches only those tables which occur at the outermbst level of any hierarchy of nested tables. To get all tables one can either disallow tree patri m: nimizatipn (which is an option in the] GUI; then instead ofthe star another sign is used for this general descendant] navigation), or much better, use recursion to distinguish the various hierarchy levels, hn the first solution, all extracted tables at any nesting level are direct

be usefμl. In a final step the evaluator removes the instances whose position does not appear m any φf the specified intervals [1104], and returns] [1105] the resulting list of pattern instances. ' '

[300] When evaluating a pattern, the union of all exfracted instances of all its rules is considered and all non-minimal targets, i.e. all pattern instances which derive from the same pare int fristance a !s am instance which is entirely contained inside this instance are dropped.

This m: mimization step is performed by default, but can be optionally omitted if the designer wishes so. d) Evaluating tree extraction predicates and tree conditions

[301] The evaluation ofthe tree extraction predicates is mainly concerned with the computation of he list of elements matched by an element path definition. Essentially, this computation ban be split into two parts: finding the elements reachable from a given start elemem t (the root ofthe pa Trent pattern tree-region) following a certain tree path, and the validation of ₍these elements w.r.t. the specified attribute conditions

[302] Finding elements: The following functions compute the list of .elements matched by an incompletely specified tree path. They use the function children(x) that returns the list of the children ofthe element x ordered with respect of their position in the document tree and the function findDescendants(e, t) defined below that returns the list ofthe descendants of an element x of name t, again ordered with respect to their position in the document free. The

[] identify lists and a notation like [x|C] identifies a list having head x and tail C, whereas the function concat returns the concatenation of two lists. Essentially, the function matchelements(e,p) seeks for the paths in the document tree rooted in the element e that fulfill the element path definition. It starts by finding the elements reachable from e whose type is the san e as the type contained in the first part ofthe free path/?. This process is iterated with the rem aiming part ofthe path/?.

function mat helements (e,p) : List of matched elements INPUT

Ata. element e (i . e . , a node of 'a document parse-tree)

Ab. incompletely specified tree path p

OUTPUT]

List of matched elements

BEGIN

[e list of children of x */

List of matched elements

[308]

[309]

[310]

R := concat (R, stringpattern (s,x,y) ) ; return R

[311] The test 'substrg(s,x,y) in L(spd)" checks whether a string is in the language defined by a regular expression (see e.g., J. E. Hopcroft and J. Ulhman: Introduction to Automata Theory, Languages, and Computation, Addison- Wesley, ISBN: 0201441241) or to verify that such a string is in the relation defined by a concept predicate (or a mixture of both). Thus the set ofthe valid (ground) substitutions for an atom subtext (s,spd,X) is trivially computed by calling the funct ion match ubtext(s, spd). In pur preferred embodiment, only those matches are considered which are not already part o: 'another one, using left-to-right evaluation.

[312] The computation o the above function can be speeded up with the same methods as used for finite s ate automata (see again the book of Hopcroft and Ullman).

[313] As for the predicate subtext, the evaluation of before and after predicates for string sources uses the! function matchsubtext. A flow diagram for the evaluation ofthe before predicate is reported in Figl 14. The evaluation ofthe after predicate is completely analogous, as will be understood by those skilled in the art, and is not reported here.

[314] The evaluator starts [1401] by computing all the substrings contained in the parent pattern instance 5 matching the string path definition spd [1402]. Each of these substrings is

I in an iteration ofthe loop [1404,1403,1406,1408,1407], where it is verified that it is in the desired di stance interval [1403,1406], and if this is the case, the string source (i.e. the string t gether with its position) is inserted into the] result list [1408].

1

[315] As for elements, the atoms notbefore(s,x,spd,d) and notafter(s,x,spd,d) are evaluated using the corresponding atoms before(s,x,spd,0,d,Y_lD) and after(s,x,spd,0,d,Y,D). The evaluation of a ontains(s,spd,X) atom derives straightforwardly from the matchsubtext j functio n. In our preferred embodiment we cpnsidei "minimal" regular expressions only, i.e. regular expressions not contained in each other, starting evaluation from the left to the right with a 'greedy" operator interpretation (like Perl treats regular expressions).

[316] The evaluation of concept predicate straightforwardly follows from this definition.

Indeed syntactic, concept predicate can be evaluated by simply testing the membership of a string to the language defined by a regular expressions, whereas semantic predicates are directly represented as sets of ground atoms. f) Translation into XML

[317] The XM translator offers the possibility to map exfracted instances ofthe pattern instance base imp XML elements. A simple correspondence holds: Pattern name = EloglDB [318]

[319]

//next line

appear eore t e esre target pattern .

Java¹ Sv ing browser with navigation capabi ities is used) [1602]. This sample page is

shou 'ldL be addedI' [1607]. Ot¹herwise, the pattern already exactly identifies the desired

(1) Constructing a Tree Filter (Fig. 17)

express that the content of an element is a date and is before 10th of March 2001 (keep

di i ih b

must oc ur after the desired pattern instance. Note that such pattern references do not

designer! the relevant groups and the designer selects which groups to consider and

for free conditions) to opt for one ofthe following two modes: The standard mode is to refer

case, srijice we are merely interested in writing a wrapper for a single Web page. However,

time to maintain! compatibility with a servlet frontend. (3) UI Package I

[497] If the consistency condition is not fulfilled, a warning signal is given or a warning email is sent to an administrator and/or wrapper execution is stopped. Consider a wrapper designer, who once created a wrapper and uses it since then for repeated exfraction; after three months, the stri cture ofthe page changes significantly and he could lose some information without even knowing about it. The wrapper program hence can give some warning message that sopne structural conditions are no longer fulfilled, and this page does not classify as a suited piage for this wrapper. It is useful that, such consistency conditions can be specified by the wrapper designer herself, because she knows best, which requirements to pose as conditions for s able extraction.

[498] As just sijaid, one way to create consistency conditions is to mark that some patterns need to have at l reast one injstance to make th ibb wrapper consistent (in this case, labeling a checkbox for such a pattern is sufficient). Such patterns are constructed in the same way as ordinary ones. Tlrey do not necessarily need to exfract information.

(10) Possible Example Scenarios

[499] One possible scenario uses Lixto wrappers to wrap sites of TV broadcast companies and provide an iiser interface to query the combined XML companions of TV program HTML pages of various₍ channels.

[500] ] Another 'scenario is to provide support in programming a video recorder. Instead of typing in a VPS number one can choose to type in the name of an interesting broadcast. The VR returns mor information about this from a wrapped database and the option to program one of these broadcasts.

(11) Informationi Pipes System

[501] In one off our embodiments, the extraction job server is embedded into a user- personalizable infoπmation process flow which accesses the XML output in order to query it and deliver parts of it to the user in case of changes, and a merger to map several XML outputs of different wrappers into a common scheme. Lixto wrappers can be embedded into a persona lizable riser information pipe. There information is processed from various sources, wrapped, merged, transformed and finally delivered.

[502] Lixto is a tool for Web infoπmation extraction, information labelling and franslation to XML. Its ideal cpmpanion is the InfoPipes system, which provides a tool for Information integration and ransfornration, and multi-platform information delivery.

[503] InfoPipes takes care of navigating through passwords, HTML forms, etc. to provide

Lixto an HTML page to wrap. InfoPipes is capable [of integrating various XML companions, transforming them, and querying them by providing easy-to-use graphical interfaces for

Claims

WHAT IS CLAIMED IS

1. A wrapper generation system comprising: a networlc includinglat least one example document and at least one production document; and j

I a visual builder that is adapted to interactively generate a wrapper program by letting a user visually and interactively declare at least one desired property of example-elements to be extracted from trie example document thereby creating user declarations; a program evaluator adapted to execute a wrapper program over the production document and toiexfract desired production elements from the production document and to translate: the production elements into XML yielding an XML companion ofthe production document.

2. The vjrapper generation system of Claim 1> wherein the visual builder further includes: on pattern builder adapted to provide a visual interface for a user to specify at least cjme deshied pattern whose instances are to be extracted from the document.

3. The wrapper generation system of Claim 2, wherein the visual builder further includes: ail XML translation builder adapted to interactively generate an XML translation scheme based onluser specifications that specifies how to franslate at least one pattern into

XML using XM franslation rules.

4. The wrapper g ;eneration system of Claim 1, wherein the program evaluator further includes:

I an extractor that exfracts data from the production document and provides an XML document

5. The wrapper g neration system of Claim 2; further including: i j XML translator that performs actual mapping from a set of pattern instances to an

XML document,

6. The wrapper generation system of Claim 1, wherein the example document is received in a

I browser window ι

7. The wrapper generation system of Claim 1, wherein a user controls wrapper generation, selects examples from the example document and adds further user specifications.

colgroub, caption, td, p, br, 'div, blockquote, body, head, hi, h2, h3, h4, h5, h6, dl, dd, dt, ol, ul, li, dir, menu, form, input, select, option, address, center, pre, xmp, nobr, wbr, hr, img, b, i, font-size, font-color, underline, blink, a (anchors), href, tt, big, sup, sub, cite, code, strong, em, sarrip, area, map, script, and CSS styles applied to the content and structure.

25. The (wrapper generation system of Claim 13, wherein said generalized location descriptor is obtai ed from a corresponding location descriptor by syntactic generalization operations.

26. The wrapper 'generation system of Claim 25, wherein said generalized location descriptor is ob_jtaiiied bbyy iinlsertin g zero or more wildcards, by substituting wildcards for zero or more elements and by eliminating zero or more el Iem ! ents I from said location descriptor.

27. The wrapper generation system of Claim 13, wherein the example-document is tree- structured and wherein each location descriptor, cal ed plain tree path, created for an example- element occurring in an example-document corresponds to the sequence of element-types on the path in the parsing tree of said example-document from the root to said example-element, and where the corresponding generalized location descriptor, called incompletely specified tree patli, is obtained from said location descriptor by inserting zero or more wildcards, by substituting wildcards for zero or more elements and by eliminating zero or more elements from said location descriptor.

28. The wrapper generation system of Claim 13, wherein at least one production-document is tree-structured and wherein each location descriptor], called plain tree path, created for an example ■ -elemen occurring in an example-document corresponds to the sequence of element- types on the pat in the parsing tree of said example-document from the root to said example- element. and whe re the corresponding generalized location descriptor, called incompletely specified tree pat :hh, is obtained from said location descriptor by inserting zero or more wildcards. , by substituting wildcards for zero or more elements and by eliminating zero or more elements from said location descriptor.

29. The wrapper generation system of Claim 13, wherein at least one production-document is inHTMI format) (and wherein each location descriptor, called plain tree path, created for an example-elemeni occurring in an example-document corresponds to the sequence of HTML tags on the path in the parsing tree of said example-document from the root to said example-

5. The wrapper I

1. The ^; [method [of Claim 57 wherein at least one exfraction pattern is organized according

elimimatjng zero or more elements from said location descriptor.

94. The method f Claim 57, wherein the pattern description generated for each of said patterns consists jjof a set of rules formulated in a logic programming language and where said wrapper consists of a logic program containing said rules.

95. The method of Claim 94 wherein the variables and terms of said logic program range over

I elemen occurπiή; g in the documents to which said wrapper is applied,

I

96. The method of Claim 94 wherein the variables and terms of said logic program range over element and strin .gs occurring in the documents to which said wrapper is applied.

97. The method f Claim 94 wherein the variables arid terms of said logic program range over elements, element-lists, and] strings occurring! in the documents to which said wrapper is applied,

98. Th method bf Claim 94 wherein the variables and terms of said logic program range over elements, el bment-lists, strings and attribute values occurring in the documents to which said wrapper is ap plied.

99. The method of Claim 94 wherein said documents are free-structured and where the variables of said logic program range over nodes ofthe parsing frees of documents to which said wrapper is ajpplied,

100. the method of Claim 94 wherein said documents are free-structured and where the variables of said ogic program range over nodes ofthe parsing trees and over strings occurrm g in the documents to which said wrapper is applied.

101. The method¹ of Claim 94 wherein said documents are free-structured and where the variables of said logic program range over nodes and sequences of nodes ofthe parsing frees of documents to which said wrapper is applied.

102. The method of Claim 94 wherein said documents are free-structured and where the variables of said ogic program range over nodes ofthe parsing free of documents to which said wrabper is applied, over sequences of such nodes, and over strings occurring in documents to which said wrapper is applied. ]

wherein the contains conditions impose one or more restrictions on some subelement ofthe pattern to be defined and the notcontains conditions require that instances ofthe pattern to be defined do not contain any subelement that satisfies specified restrictions.

I I

125. Th method of Claim 123, wherein range intervals can be specified with each rule of said extended logic program, and wherein only those facts are computed by a rule that match the ranges of said intervals in the ordering induced by document order.