WO2002069249A1 - A method for matching spatial patterns - Google Patents

A method for matching spatial patterns Download PDF

Info

Publication number
WO2002069249A1
WO2002069249A1 PCT/US2002/005857 US0205857W WO02069249A1 WO 2002069249 A1 WO2002069249 A1 WO 2002069249A1 US 0205857 W US0205857 W US 0205857W WO 02069249 A1 WO02069249 A1 WO 02069249A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature points
matching
template
subset
triangle
Prior art date
Application number
PCT/US2002/005857
Other languages
French (fr)
Inventor
Shannon R. Campbell
Original Assignee
Campbell Shannon R
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Campbell Shannon R filed Critical Campbell Shannon R
Publication of WO2002069249A1 publication Critical patent/WO2002069249A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/757Matching configurations of points or features
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30176Document

Definitions

  • TITLE A method for matching spatial patterns
  • This invention relates to the fields of visual pattern recognition, object recognition, image registration, and the matching of spatial patterns in two or more dimensions.
  • a basic description of template matching is as follows.
  • a "square' template is a binary array: the edges of the square are given a value of and the background and central portion of the square are given a value of '0'.
  • the edges of an input object are found and compared to the "square' template. If the position, shape, orientation, and scale of the input object are very near to that of the "square' template, then a correlation of the input object with the "square' template yields a high value indicating a match. If the input object is a square, but is at a slightly different position, orientation, or scale, template matching fails. How to match identical patterns that are transformed in position, rotation, and scale is a currently unsolved problem in the field of pattern recognition.
  • Fourier components are able to represent single closed contours efficiently using the coefficients of periodic functions.
  • a measure for the distance between the two vectors containing those coefficients is used to determine a match.
  • This approach is rotation invariant, but not scale invariant. Furthermore, it does not represent complex two dimensional objects readily.
  • Structural descriptions are interesting because they have the potential to perform position, rotation, and scale invariant pattern recognition.
  • Structural descriptions consist of graphs in which the nodes indicate some feature of the object and the connections between nodes indicate spatial relationships such as up, down, right, left, top, bottom, and middle.
  • S. E. Palmer in his book “Vision Science, Photons to Phenomenology", MIT Press, Cambridge Mass., 1999, pp. 394, presents a structural graph for the letter "A.
  • the graph for this object contains 12 nodes and 14 edges. This is a surprisingly complex graph for such a simple object.
  • the relationship between the complexity of the object and the size of the resulting graph is unknown.
  • one can construct different graphs for this object To date, there is no consistent and automatic method of generating these structural graphs.
  • My method is based on matching of polyhedra and their appropriate adjacent and neighboring polyhedra in a tessellation of feature points.
  • the tessellation be the Delaunay triangulation in two dimensions and let us consider image data.
  • Feature points consist of the position of the feature and labels describing that feature.
  • the labels can be "corner', "line termination', or any other description of an image feature.
  • the method then performs a search for matching triangles between the Delaunay triangulation of input feature points and the Delaunay triangulation of template feature points stored in memory.
  • Triangles match if there is sufficient similarity in the angles and labels associated with each node, or feature point, in the triangle.
  • Matching a single triangle represents the significant event of matching three node labels and their spatial relationships.
  • a positive match between the angles and nodes of one triangle and its three adjacent triangles results in a total of twelve matching angles and six node labels.
  • This is a highly discriminating and therefore useful method for performing spatial pattern recognition. Additional neighboring triangles can be examined for increased confidence of match. This method is capable of performing spatial pattern matching independent of transformations in position, rotation, and scale. Most other pattern recognition techniques do not have these properties.
  • a high probability of match can be achieved using a small set of adjacent and neighboring triangles and is capable of yielding a positive match even if the object is significantly occluded.
  • the present invention is useful in character recognition as for: kanji, Cyrillic, Arabic, Chinese, Korean, Japanese, and others.
  • Other applications include: Matching of stellar patterns for telescope aiming, matching of patterns for target recognition, and matching of patterns for robot navigation.
  • Biometric applications include matching of biological data for identification of individuals, such as matching of facial features, matching retinal features, matching iris features, and matching fingerprint features.
  • Recognition of manmade objects such as tools for robotic applications.
  • the present invention can be used for highly accurate matching of features to determine if a bill is counterfeit or not.
  • the present invention can be used to determine differences between similarly sized coins.
  • Use in recognition of textures Recognition of manmade objects such as cars, forks, machined pieces.
  • Applications in industry arise in precise measurements of the spatial relationships of parts and pieces of an item for quality control.
  • Use in recognition of three dimensional objects such as proteins.
  • Medical applications such as matching shapes of cells, identification of cells in urine, blood, or other tissue samples.
  • the feature points may lie in dimensions of time and frequency, and recognizing features in these dimensions is necessary.
  • This algorithm can be used in spatial-temporal recognition such as sign language, gesture recognition, recognition of specific motion forms including but not limited to tennis or golf swings, recognition of behavioral motion patterns in humans, rats, mice, fish, and other animals.
  • Applications exist in recognizing patterns in data.
  • Data mining applications exist such as finding relationships in patterns in how individuals buy groceries.
  • Stereoscopic matching, image registration, and morphing from one image to another are other applications.
  • This algorithm should also find applications in image compression.
  • Known objects may be compressed to a list of features, their positions and angular relationships instead of a large number of pixel color values. There are many ways to create tessellations.
  • tessellation the Voronoi tesselation, the Delaunay triangulation, and the Delaunay triangulation with constraints, such as minumim weight triangulation, angular balanced triangulation, area balanced triangulation.
  • constraints such as minumim weight triangulation, angular balanced triangulation, area balanced triangulation.
  • tessellation are grouped under the generic labels of unstructured grid generation, mesh generation, triangulations based on quadtrees, or polyhedralization based on octrees, quadrangulation, hierarchical adaptive tesselation, adaptive polygonal tesselation, and Steiner tetrahedralization. Many of these algorithms have equivalents in three and higher dimensions.
  • Figure 1 is a chart displaying the numeral five
  • Figure 2 is the chart of Fig. 1 showing feature points using asterisks
  • Figure 3 is a chart showing a Delaunay triangulation of the feature points of Fig. 2 where each triangle is identified by a number so that adjacency relationships are able to be easily denote.
  • Figure 4 is a chart showing a triangle from Fig. 3 where each node of the triangle is labeled and is uniquely associated with an angle;
  • Figure 5 is a chart showing the triangle from Fig. 4 and its three adjacent triangles
  • Figure 6 is a chart showing a probability curve of matching of all three angle labels in the correct cyclic order as the number of labels increases;
  • Figure 7 is a chart showing the Delaunay triangulation for the mirror image of the feature points of the numeral five shown in Fig. 2;
  • Figure 8 is a chart showing the numeral two with asterisks representing locations of feature points; and Figure 9 is a chart showing the numeral three with asterisks representing locations of feature points.
  • My method for matching spatial patterns is based on the matching of polyhedra and their appropriate adjacent and neighboring polyhedra between tessellations of the input feature points and the template feature points.
  • the method is independent of the tessellation used or the dimension of the space.
  • I will explain the invention using the Delaunay triangulation in two dimensions and the feature points will be based on salient features of two dimensional imagery.
  • I use the Delaunay triangulation because of its relative speed of computation and because it tends to connect points that are near to one another. See M. Sambridge, J. Braun, and H. McQueen, "Geophysical parameterization and interpolation of irregular data using natural neighbors", Geophysical Journal International, Vol. 122, pp. 837-857, 1995, for an introduction to the Delaunay triangulation, its applications, and references to various algorithms for computing the triangulation.
  • Figure 1 displays a simple object, the numeral five.
  • Figure 2 displays the numeral five and the locations of feature points for this object as asterisks.
  • the feature points are the salient features of line terminations, corners, and locations of high curvature. These specific "salient features" are used for demonstration purposes only. This patent does not deal with how to create feature points from imagery or other data, but only how to match their spatial relationships. The specific features such as corners, line terminations, etc., are used as labels for the feature points.
  • Figure 3 displays the Delaunay triangulation of the feature points shown in Figure 2.
  • Figure 4 displays a close up view of Triangle 2 from Figure 3.
  • the nodes are labeled, as are their associated angular values and the side opposite each node. If a side of a triangle is shared by another triangle, then those triangles are adjacent.
  • Figure 5 displays Triangle 2 and its adjacent triangles.
  • Figure 5 demonstrates that adjacency relationships are defined geometrically and that each node has an associated angle and adjacent triangle.
  • the phrase neighboring triangles is used to refer to those triangles that are near to a triangle, but not adjacent to it. For example, in Figure 3, Triangle 5 and Triangle 6 are neighbors of Triangle 2.
  • the first step is to begin with a triangle in the list of triangles of the input, and begin looking for a similar triangle in the list of triangles from the template.
  • Triangles are similar if there is a sufficient similarity in their angles and if there is sufficient similarity in their node labels.
  • the next step is to examine the appropriate adjacent triangles. For example, if Triangle 2 of Figure 3 has a match with another triangle in a template stored in memory, the next step is to examine the appropriate adjacent triangles in both the input and the template.
  • the triangles opposite the node associated with 89 degrees are compared, then the triangles opposite the nodes associated with 61 degrees are compared, and then the triangles opposite the nodes associated with 30 degrees are compared. If all three of the appropriate adjacent triangles match their appropriate counterparts triangles in the template, then six nodes, their labels, and their spatial relationships (twelve angles) match. Depending on the conditions used for matching individual triangles, there is the possibility that matching one triangle and its three appropriate adjacent triangles is exceptionally good evidence for a positive match between the input and the template.
  • each node can contain a label that describes some form of visual information.
  • this label is a line termination, a corner and the angle it subtends, or a label specifying the radius of curvature for a salient portion of a curve.
  • the features of each node need not be restricted to low-level visual features such as corners, curvature, or line terminations.
  • the visual features can be more complex, such as an eye, a nose, or an eyebrow.
  • a search based on matching node labels is more efficient than searching for matching angles.
  • node labels are the only distinguishing information in this example.
  • the "3' is differentiated from the "2' by having one " junction, three line terminations, and two corners.
  • the "2' has no "T' junction, two line terminations, and four corners. Because node labels are based on visual information at each node, the algorithm can differentiate between objects that have identical feature point locations.
  • a handwritten numeral two can have significantly different appearances and salient feature points than the numeral two shown in Figure 6.
  • My matching method is position, rotation, and scale invariant because the angles and labels associated with each node are used to determine similarity of triangles.
  • the angle between nodes does not change if the locations of the nodes are translated, rotated, or rescaled. If these locations undergo an affine or nonlinear transformation, then it is possible for the underlying triangulation to change and thereby affect the accuracy of the match.
  • the Delaunay triangulation is not scale invariant because the resultant triangles increase in size if the separation between the points increases in size. Since the method uses the similarity in angles and nodes, and this is not dependent on the size of the triangle, my matching method is scale invariant.
  • my method for performing spatial recognition is powerful because it is based on local information. If a dozen appropriate adjacent and neighboring triangles of an object are matched with a template in memory, and each triangle has a probability of 0.0203 of being matched randomly, then there is exceptional evidence of a positive match, even if there are a thousand triangles in the tessellations being compared. This indicates that considerable occlusion or deformation exists.
  • my algorithm does not necessarily depend on the Delaunay triangulation to perform matches of triangles.
  • Other tessellation algorithms can be used to generate polygons based on the location of feature points. Matching an individual polygon and its appropriate adjacent and neighboring polygons is a conceptually identical to my method of matching spatial relationships. Details of the implementation, search, and decision making process will be different, but the basic idea underlying the algorithm is identical to that presented here.
  • This algorithm can also use tessellation algorithms in three and higher dimensions to create polyhedra that are searched to perform spatial pattern matching in those dimensions.
  • my algorithm can be used to match feature points in parameter space.
  • a word can be represented by feature points whose locations are given in units of time, amplitude, and frequency.
  • the tessellation of feature points can be done regardless of the units of the dimensions, thus this algorithm can be used for matching spatial structures in parameter space.

Abstract

A method for matching the spatial relationships between an input set of feature points and a template set of feature points, where a feature point consists of a location in space and a label describing the feature at that location. A tessellation over the feature point locations is performed (Fig.5). Next, a search identifies polyhedra (Fig.5, triangle 2) that have similar contents, the contents being the angles and labels associated with feature points of the polyhedra (Fig.5, nodes 1,2 and 3). Once a match is found, then appropriate adjacent and neighbouring polyhedra (Fig.5, triangles 1, 3 and 4) are examined. Because this matching method is based on similarities in node angles and labels, it is independent of transformations in position, rotation and scale if the angles in the polyhedra created by the tessellation are position, rotation, and scale invariant.

Description

TITLE: A method for matching spatial patterns
CROSS-REFERENCE TO RELATED APPLICATIONS: U.S. application serial number 09/788,852, filed 2/21/2001, now pending.
BACKGROUND OF THE INVENTION
RELATED ART
This invention relates to the fields of visual pattern recognition, object recognition, image registration, and the matching of spatial patterns in two or more dimensions.
References Cited
U.S. 4,783,829 Nov., 1988 Al ira et al 382/199
U.S. 4,891,762 Jan., 1990 Chotiros 701/223
U.S. 6,181,806 Jan., 2001 Kado et al 382/100
S. E. Palmer, "Vision Science, Photons to Phenomenology", MIT Press, Cambridge Mass., 1999. S. Ullman, "High-level Vision: Object recognition and visual cognition", MIT Press, Cambridge Mass., 1996. H. Bunke and B.T. Messmer, "Recent advances in graph matching", Int. J. Part. Recog. Art. Intell., Vol. 11, No. 1, pp. 169-203, Feb. 1997. J. R. Ullmann, "An Algorithm for Subgraph Isomorphism", J. Assoc. Comput. Mach., Vol. 23, No. 1, pp. 31-42, 1976.
L. Shapiro and R. M. Haralick, "A metric for comparing relational descriptions", IEEE Part. Anal. Mach. Int. Vol. 7, pp. 90-94, 1985.
J. E. Hummel and I. Biederman, "Dynamic binding in a neural network for shape recognition", Psych. Rev., Vol. 99, pp. 480-517, 1992.
N. Ahuja, "Dot pattern processing using Voronoi Neighbourhoods", IEEE Part. Anal. Mach. Int. Vol. 4, pp. 336-343, 1982. Y.-W. Chiang and R.-C. Wang, "Seal identification using the Delaunay tessellation", Proc. Nat. Sci. Council, Rep. of China, Part A: Physical Sci. and Eng., Vol. 22, No. 6, pp. 751-757, Nov 1998.
G. Weber, L. Knipping, and H. Alt, "An application of point pattern matching in astronautics", J. Symb. Comp., Vol. 17, No. 4, pp. 321-340, 1994.
M. Sambridge, J. Braun, and H. McQueen, "Geophysical parameterization and interpolation of irregular data using natural neighbors", Geophysical Journal International, Vol. 122, pp. 837-857, 1995.
Much work has been done in object recognition. S. E. Palmer's, ."Vision Science, Photons to Phenomenology", MIT Press, Cambridge Mass., 1999, and S. Ullman's "High-level Vision: Object recognition and visual cognition", MIT Press, Cambridge Mass., 1996, are works that contain more specific references to some of the basic topics on pattern recognition mentioned in this section.
Some useful pattern recognition algorithms are based on template matching. A basic description of template matching is as follows. A "square' template is a binary array: the edges of the square are given a value of and the background and central portion of the square are given a value of '0'. The edges of an input object are found and compared to the "square' template. If the position, shape, orientation, and scale of the input object are very near to that of the "square' template, then a correlation of the input object with the "square' template yields a high value indicating a match. If the input object is a square, but is at a slightly different position, orientation, or scale, template matching fails. How to match identical patterns that are transformed in position, rotation, and scale is a currently unsolved problem in the field of pattern recognition.
Other object recognition algorithms have been based on analysis of the Fourier spectra of objects. These algorithms are rotation invariant, but are not scale invariant. A specific problem with this approach is that often the entire image is used and objects are not segmented prior to analysis. Thus information about multiple objects is sometimes included in the Fourier spectra. Furthermore, if half of the object is occluded, this method of shape matching fails because large scale features, or low frequency components, will be different from those stored in the matching template.
One can also approximate the shape of an object using Fourier components. Fourier components are able to represent single closed contours efficiently using the coefficients of periodic functions. A measure for the distance between the two vectors containing those coefficients is used to determine a match. This approach is rotation invariant, but not scale invariant. Furthermore, it does not represent complex two dimensional objects readily.
Another approach is structural descriptions. Structural descriptions are interesting because they have the potential to perform position, rotation, and scale invariant pattern recognition. Structural descriptions consist of graphs in which the nodes indicate some feature of the object and the connections between nodes indicate spatial relationships such as up, down, right, left, top, bottom, and middle. For example, S. E. Palmer, in his book "Vision Science, Photons to Phenomenology", MIT Press, Cambridge Mass., 1999, pp. 394, presents a structural graph for the letter "A. The graph for this object contains 12 nodes and 14 edges. This is a surprisingly complex graph for such a simple object. The relationship between the complexity of the object and the size of the resulting graph is unknown. Furthermore, one can construct different graphs for this object. To date, there is no consistent and automatic method of generating these structural graphs.
Another complication of structural graphs is that matching graphs is itself a complex problem. For examples of various approaches to the graph matching problem in the computer vision domain, see H. Bunke and B.T. Messmer, "Recent advances in graph matching", Int. J. Part. Recog. Art. Intel!, Vol. 11, No. 1, pp. 169-203, Feb. 1997, J. R. Ullmann, "An Algorithm for Subgraph Isomorphism" J Assoc. Comput. Mach., Vol. 23, No. 1, pp. 31-42, 1976, and L. Shapiro and R.M. Haralick, "A metric for comparing relational descriptions", IEEE Part. Anal. Mach. Int. vol. 7, pp. 90-94, 1985. All of these works contain algorithms that are complex and time consuming for objects with more than a few edges and nodes.
Other researchers have tried to encode spatial relationships with neural networks. J. E. Hummel and I. Biederman, "Dynamic binding in a neural network for shape recognition", Psych. Rev., Vol. 99, pp. 480-517, 1992, use neural networks to encode spatial relationships between geons in the RBC (recognition by components) theory of object recognition. These neural networks encode features such as location, orientation, and scale. While this system works for simple objects, it is unclear how the system performs on multiple objects that contain many features. This is because the system has limited capability in representing different locations, orientations, and scale. Therefore the system may fail when making fine distinctions between objects with large numbers of features.
Several other researchers use a rigid mathematical procedure (e.g. Voronoi tessellation, Delaunay triangulation, Gabriel graphs, and minimal spanning trees) to create graphs from feature points and then perform graph matching. N. Ahuja, in his paper, "Dot pattern processing using Voronoi Neighbourhoods", IEEE Patt. Anal. Mach. Int. Vol. 4, pp. 336-343, 1982, suggests a relaxation labeling method to perform matching of dot patterns using Voronoi tessellation, a method both complex and time consuming for large objects. In Y.-W. Chiang and R.-C. Wang, "Seal identification using the Delaunay tessellation", Proc. Nat. Sci. Council, Rep. of China, Part A: Physical Sci. and Eng., Vol. 22, No. 6, pp. 751-757, Nov 1998, the authors match histograms of the areas of resulting Delaunay triangles to recognize Chinese Seals. In G. Weber, L. Knipping, and H. Alt, "An application of point pattern matching in astronautics", J. Symb. Comp., Vol. 17, No. 4, pp. 321-340, 1994, the authors use a Delaunay triangulation between stars and then look for matches in the slope and length of edges between pairs of stars, thus their algorithm is not rotation or scale invariant. A similar matching method is proposed in N.P. Chotiros' U.S. patent no. 4,891,762 entitled "Method and apparatus for tracking, mapping and recognition of spatial patterns". Chotiros' algorithm matches only lengths of edges in a Delaunay triangulation and thus achieves invariant matching with respect to rotation, but not with respect to scale. In U.S. patent no. 6,181,806 issued to Kado et al, entitled "Apparatus for identifying a person using facial features", a method is described for recognizing faces based on matching the brightness of patches of an object with those stored in memory. The authors use triangles as the patches, but they do not use angle information, or adjacency relationships to perform spatial matching of a triangular network. Their algorithm is not scale or rotation invariant.
In Akira et al, U.S. patent no. 4,783,829 entitled "Pattern Recognition Apparatus", the authors propose a method which matches polygons that are similar in area, thus creating an algorithm that is not scale invariant.
BRIEF SUMMARY OF THE INVENTION
My method is based on matching of polyhedra and their appropriate adjacent and neighboring polyhedra in a tessellation of feature points. For concreteness, let the tessellation be the Delaunay triangulation in two dimensions and let us consider image data. Feature points consist of the position of the feature and labels describing that feature. The labels can be "corner', "line termination', or any other description of an image feature. The method then performs a search for matching triangles between the Delaunay triangulation of input feature points and the Delaunay triangulation of template feature points stored in memory. Triangles match if there is sufficient similarity in the angles and labels associated with each node, or feature point, in the triangle. Matching a single triangle represents the significant event of matching three node labels and their spatial relationships. A positive match between the angles and nodes of one triangle and its three adjacent triangles results in a total of twelve matching angles and six node labels. This is a highly discriminating and therefore useful method for performing spatial pattern recognition. Additional neighboring triangles can be examined for increased confidence of match. This method is capable of performing spatial pattern matching independent of transformations in position, rotation, and scale. Most other pattern recognition techniques do not have these properties. Lastly, a high probability of match can be achieved using a small set of adjacent and neighboring triangles and is capable of yielding a positive match even if the object is significantly occluded.
The present invention is useful in character recognition as for: kanji, Cyrillic, Arabic, Chinese, Korean, Japanese, and others. Other applications include: Matching of stellar patterns for telescope aiming, matching of patterns for target recognition, and matching of patterns for robot navigation. Biometric applications include matching of biological data for identification of individuals, such as matching of facial features, matching retinal features, matching iris features, and matching fingerprint features. Recognition of manmade objects such as tools for robotic applications. Recognition of spatial features in three and higher dimensions for datasets in these dimensions. Matching and recognition of spatial relationships of MRI data, CT data, ultrasound data, range data. Matching of scan patterns placed on consumer goods. Matching of features from surfaces embedded in three dimensions, for example, the features of a face are located on a surface in three dimensions. Matching spatial relationships found in words. Matching of electrophoresis gels for DNA or other biologically related data. The present invention can be used for highly accurate matching of features to determine if a bill is counterfeit or not. The present invention can be used to determine differences between similarly sized coins. Use in recognition of textures. Recognition of manmade objects such as cars, forks, machined pieces. Applications in industry arise in precise measurements of the spatial relationships of parts and pieces of an item for quality control. Use in recognition of three dimensional objects such as proteins. Medical applications such as matching shapes of cells, identification of cells in urine, blood, or other tissue samples. Use in identification of spatial relationships between body parts in MRI data. This algorithm has applications for matching patterns in non-spatial dimensions. For example, in voice recognition, the feature points may lie in dimensions of time and frequency, and recognizing features in these dimensions is necessary. This algorithm can be used in spatial-temporal recognition such as sign language, gesture recognition, recognition of specific motion forms including but not limited to tennis or golf swings, recognition of behavioral motion patterns in humans, rats, mice, fish, and other animals. Applications exist in recognizing patterns in data. Data mining applications exist such as finding relationships in patterns in how individuals buy groceries. Stereoscopic matching, image registration, and morphing from one image to another are other applications. This algorithm should also find applications in image compression. Known objects may be compressed to a list of features, their positions and angular relationships instead of a large number of pixel color values. There are many ways to create tessellations. For example, we note the several common forms of tessellation: the Voronoi tesselation, the Delaunay triangulation, and the Delaunay triangulation with constraints, such as minumim weight triangulation, angular balanced triangulation, area balanced triangulation. Also many forms of tessellation are grouped under the generic labels of unstructured grid generation, mesh generation, triangulations based on quadtrees, or polyhedralization based on octrees, quadrangulation, hierarchical adaptive tesselation, adaptive polygonal tesselation, and Steiner tetrahedralization. Many of these algorithms have equivalents in three and higher dimensions.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
Figure 1 is a chart displaying the numeral five;
Figure 2 is the chart of Fig. 1 showing feature points using asterisks;
Figure 3 is a chart showing a Delaunay triangulation of the feature points of Fig. 2 where each triangle is identified by a number so that adjacency relationships are able to be easily denote.
Figure 4 is a chart showing a triangle from Fig. 3 where each node of the triangle is labeled and is uniquely associated with an angle;
Figure 5 is a chart showing the triangle from Fig. 4 and its three adjacent triangles;
Figure 6 is a chart showing a probability curve of matching of all three angle labels in the correct cyclic order as the number of labels increases;
Figure 7 is a chart showing the Delaunay triangulation for the mirror image of the feature points of the numeral five shown in Fig. 2;
Figure 8 is a chart showing the numeral two with asterisks representing locations of feature points; and Figure 9 is a chart showing the numeral three with asterisks representing locations of feature points. DETAILED DESCRIPTION OF THE INVENTION
My method for matching spatial patterns is based on the matching of polyhedra and their appropriate adjacent and neighboring polyhedra between tessellations of the input feature points and the template feature points. The method is independent of the tessellation used or the dimension of the space. For the purposes of the detailed description of the invention, I will explain the invention using the Delaunay triangulation in two dimensions and the feature points will be based on salient features of two dimensional imagery. I use the Delaunay triangulation because of its relative speed of computation and because it tends to connect points that are near to one another. See M. Sambridge, J. Braun, and H. McQueen, "Geophysical parameterization and interpolation of irregular data using natural neighbors", Geophysical Journal International, Vol. 122, pp. 837-857, 1995, for an introduction to the Delaunay triangulation, its applications, and references to various algorithms for computing the triangulation.
Figure 1 displays a simple object, the numeral five. Figure 2 displays the numeral five and the locations of feature points for this object as asterisks. The feature points are the salient features of line terminations, corners, and locations of high curvature. These specific "salient features" are used for demonstration purposes only. This patent does not deal with how to create feature points from imagery or other data, but only how to match their spatial relationships. The specific features such as corners, line terminations, etc., are used as labels for the feature points. Figure 3 displays the Delaunay triangulation of the feature points shown in Figure 2.
From the Delaunay triangulation one can compute a list of adjacent triangles and the three angles of each triangle. Table 1 displays a list of the angles and the adjacent triangles for each triangle in Fig. 3.
Table 1
Adjacent Triangles angles Triangle 1 0 0 2 30 31 119
Triangle 2 3 1 4 89 61 30
Triangle 3 0 2 5 45 88 46
Triangle 4 2 0 6 34 85 61
Triangle 5 3 6 0 38 45 97
Triangle 6 5 4 0 56 53 71
The sum of the angles for each triangle in Table 1 is not always 180 degrees because the angles have been rounded to the nearest integer. A "0' in the "Adjacent Triangles' columns of Table 1 indicates that, that side of the triangle has no neighbor. For the
Delaunay triangulation shown in Figure 3, only Triangle 2 has three adjacent triangles.
Figure 4 displays a close up view of Triangle 2 from Figure 3. In Figure 4 the nodes are labeled, as are their associated angular values and the side opposite each node. If a side of a triangle is shared by another triangle, then those triangles are adjacent. Figure 5 displays Triangle 2 and its adjacent triangles. Figure 5 demonstrates that adjacency relationships are defined geometrically and that each node has an associated angle and adjacent triangle. The phrase neighboring triangles is used to refer to those triangles that are near to a triangle, but not adjacent to it. For example, in Figure 3, Triangle 5 and Triangle 6 are neighbors of Triangle 2.
To match an input object with a template in memory, the first step is to begin with a triangle in the list of triangles of the input, and begin looking for a similar triangle in the list of triangles from the template. Triangles are similar if there is a sufficient similarity in their angles and if there is sufficient similarity in their node labels. Once a single triangle match has been found, the next step is to examine the appropriate adjacent triangles. For example, if Triangle 2 of Figure 3 has a match with another triangle in a template stored in memory, the next step is to examine the appropriate adjacent triangles in both the input and the template. Thus the triangles opposite the node associated with 89 degrees are compared, then the triangles opposite the nodes associated with 61 degrees are compared, and then the triangles opposite the nodes associated with 30 degrees are compared. If all three of the appropriate adjacent triangles match their appropriate counterparts triangles in the template, then six nodes, their labels, and their spatial relationships (twelve angles) match. Depending on the conditions used for matching individual triangles, there is the possibility that matching one triangle and its three appropriate adjacent triangles is exceptionally good evidence for a positive match between the input and the template.
I use the phrase "exceptionally good evidence' for a positive match based on the following reasoning. Consider the case of matching random sets of points on a plane and there are no labels to associate with these points. In this case, my method can only perform matching by comparing the angles between triangles. Also assume that the locations of the points are not measured with great accuracy and therefore the angles created by the triangulation are not expected to have perfect matches with those angles stored in a template. Let the angles be denoted by 15 different angle labels. Each angle label represents a range of angles from 0-12 degrees, 12-24 degrees, 24-36 degrees, ... , 144-156 degrees, 156-168 degrees, 168-180 degrees. One estimates that matching two angles between two random triangles occurs with a probability of approximately 3/152, or 0.013333. The reasoning behind this estimate follows. Matching one angle label out of 15 occurs with probability of 1/15. Matching two labels occurs with probability 1/152. The third angle does not need to be matched because only two angles are independent in a triangle. The factor of three arises from the fact that to obtain rotational invariance, the three cyclic permutations of the angles must be examined. This estimate is only approximately correct because the distribution of angles in a Delaunay triangulation is not uniform. Figure 6 displays numerical results indicating how the probability of matching all three angle labels depends on the number of angle labels used. For 15 labels, the probability of matching the three angle labels of two randomly created triangles is approximately 0.0203. The probability of matching any four triangles, regardless of their spatial relationships, between two random patterns that have 30 triangles each is (302)*(0.03024) or 1.5 x 10-4. Matching one triangle and its three appropriate adjacent triangles occurs with an even smaller probability. Thus matching a small number of adjacent triangles is an extremely rare event, even when one allows for inaccuracy in the angle matches and does not consider additional node labels that would further increase this algorithms ability to discriminate spatial patterns. Figure 7 displays the Delaunay triangulation for the points representing the reflection of the feature points of the numeral five shown in Figure 2. Note that this mirror image of the numeral five results in a triangulation in which the angles of each triangle are the same and the adjacency relationships are identical. The difference is that the angles of the triangles have been non-cyclically permuted. One can use this difference to recognize objects that are the mirror reflection of templates stored in memory.
Matching triangles becomes a more accurate method of performing object recognition when one also considers that each node can contain a label that describes some form of visual information. For the case of the numeral five, this label is a line termination, a corner and the angle it subtends, or a label specifying the radius of curvature for a salient portion of a curve. The features of each node need not be restricted to low-level visual features such as corners, curvature, or line terminations. The visual features can be more complex, such as an eye, a nose, or an eyebrow. Depending on the uniqueness of the node labels, one can imagine cases in which a search based on matching node labels is more efficient than searching for matching angles.
As an example of the importance of node labels, consider the following example. In Figure 8 and Figure 9 the numerals two and three are drawn as they might appear on an LCD display. Both numerals have feature points at the same locations and all feature points are corners, " junctions, or line terminations. Because the feature points are at the same locations, the resulting Delaunay triangulation is identical, thus node labels are the only distinguishing information in this example. The "3' is differentiated from the "2' by having one " junction, three line terminations, and two corners. The "2' has no "T' junction, two line terminations, and four corners. Because node labels are based on visual information at each node, the algorithm can differentiate between objects that have identical feature point locations.
Note that a handwritten numeral two can have significantly different appearances and salient feature points than the numeral two shown in Figure 6. For practical applications, it may be necessary to have several different triangulation templates to match the wide variety of handwritten twos that exist.
My matching method is position, rotation, and scale invariant because the angles and labels associated with each node are used to determine similarity of triangles. The angle between nodes does not change if the locations of the nodes are translated, rotated, or rescaled. If these locations undergo an affine or nonlinear transformation, then it is possible for the underlying triangulation to change and thereby affect the accuracy of the match. Note that the Delaunay triangulation is not scale invariant because the resultant triangles increase in size if the separation between the points increases in size. Since the method uses the similarity in angles and nodes, and this is not dependent on the size of the triangle, my matching method is scale invariant.
Further, note that my method for performing spatial recognition is powerful because it is based on local information. If a dozen appropriate adjacent and neighboring triangles of an object are matched with a template in memory, and each triangle has a probability of 0.0203 of being matched randomly, then there is exceptional evidence of a positive match, even if there are a thousand triangles in the tessellations being compared. This indicates that considerable occlusion or deformation exists.
My method allows for a simple and generic matching of spatial relationships between image features that has not been described in the literature of pattern recognition, nor in patent databases.
The applications of my algorithm are numerous. The most obvious application is object recognition. Other applications include registration of two images. This is useful in stereoscopic matching, satellite image registration, and the registration of medical images.
Furthermore, my algorithm does not necessarily depend on the Delaunay triangulation to perform matches of triangles. Other tessellation algorithms can be used to generate polygons based on the location of feature points. Matching an individual polygon and its appropriate adjacent and neighboring polygons is a conceptually identical to my method of matching spatial relationships. Details of the implementation, search, and decision making process will be different, but the basic idea underlying the algorithm is identical to that presented here. This algorithm can also use tessellation algorithms in three and higher dimensions to create polyhedra that are searched to perform spatial pattern matching in those dimensions.
Lastly, my algorithm can be used to match feature points in parameter space. For example, in the auditory domain a word can be represented by feature points whose locations are given in units of time, amplitude, and frequency. The tessellation of feature points can be done regardless of the units of the dimensions, thus this algorithm can be used for matching spatial structures in parameter space.
While the invention has been described with reference to at least one preferred embodiment, it is to be clearly understood by those skilled in the art that the invention is not limited thereto.

Claims

CLAIMSWhat is claimed is:
1. A pattern matching method comprising the steps of: a) selecting a template having template feature points and an object having object feature points; b) selecting a set of obj ect feature points ; c) selecting a subset of object feature points from the set of object feature points; d) selecting a set of template feature points; e) selecting a subset of template feature points from the set of template feature points; f) defining a match criterion between the object and template feature points; g) defining a subset match criterion between the subset of object feature points and the subset of template feature points; h) matching the subsets of object and template feature points according to the subset match criterion; i) selecting, a further subset of object feature points adjacent to the matched subset of obj ect feature points; j) selecting, a further subset of template feature points adjacent to the matched subset of template feature points; k) matching the further subsets of object and template feature points according to the subset match criterion; 1) repeating steps (i-k) until the set of object and template feature points are one of:
I. fully matched in accordance with the match criterion, and
II. found to be unmatchable; m) selecting a further template and repeating steps (b-1) if the sets are not matchable.
2. The method of claim 1 wherein, in steps c, e, i, and j the selection of feature points is from a polyhedron based on a tessellation of said feature points.
3. The method of claim 2 wherein the polyhedron is a triangle from a Delaunay triangulation.
4. The method of claim 1 wherein the set of feature points to be matched are taken from a set of recognizable image features comprising at least one of: corners, line terminations, line crossings, red points, eyes, wheels, nostrils, and mouths.
PCT/US2002/005857 2001-02-21 2002-02-21 A method for matching spatial patterns WO2002069249A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/788,852 US6542638B2 (en) 2001-02-21 2001-02-21 Method for matching spatial patterns
US09/788,852 2001-02-21

Publications (1)

Publication Number Publication Date
WO2002069249A1 true WO2002069249A1 (en) 2002-09-06

Family

ID=25145773

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2002/005857 WO2002069249A1 (en) 2001-02-21 2002-02-21 A method for matching spatial patterns

Country Status (2)

Country Link
US (1) US6542638B2 (en)
WO (1) WO2002069249A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609948A (en) * 2012-02-10 2012-07-25 浙江理工大学 Manipulation detection method for copy-paste distorted photo digital photos
CN103903249A (en) * 2012-12-27 2014-07-02 纽海信息技术(上海)有限公司 Image matching system and method
CN104408726A (en) * 2014-12-02 2015-03-11 上海许书华电子科技有限公司 Triangle-based image contour matching algorithm

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1502098B1 (en) * 2002-05-03 2008-09-17 ViaLogy, LLC Method for characterizing microarray output data
JP3952293B2 (en) * 2003-01-06 2007-08-01 ソニー株式会社 Fingerprint verification apparatus and method, recording medium, and program
US20060030985A1 (en) * 2003-10-24 2006-02-09 Active Recognition Technologies Inc., Vehicle recognition using multiple metrics
US20060052943A1 (en) * 2004-07-28 2006-03-09 Karthik Ramani Architectures, queries, data stores, and interfaces for proteins and drug molecules
GB0417346D0 (en) * 2004-08-04 2004-09-08 Forensic Eye Ltd Image comparison system and method
US7787651B2 (en) * 2005-11-11 2010-08-31 Intel Corporation Triangular method for hypotheses filtration in a cognitive control framework
TWI391850B (en) * 2009-03-31 2013-04-01 Au Optronics Corp Method of determining input pattern and computer readable storage medium
EP2617012B1 (en) 2010-09-16 2015-06-17 Mor Research Applications Ltd. Method and system for analyzing images
JP5787644B2 (en) * 2011-07-01 2015-09-30 キヤノン株式会社 Image processing apparatus and image processing apparatus control method
CN103093450B (en) * 2011-10-31 2017-03-08 鸿富锦精密工业(深圳)有限公司 product local alignment method and system
US9299157B2 (en) 2012-10-31 2016-03-29 Environmental Systems Research Institute (ESRI) Scale-invariant superpixel region edges
KR102072656B1 (en) 2013-07-16 2020-02-03 삼성전자 주식회사 Tessellation device including cache, method thereof, and system including the tessellation device
JP2017039189A (en) * 2015-08-20 2017-02-23 株式会社東芝 Layout detector and picking apparatus
US10136103B2 (en) * 2015-11-23 2018-11-20 Lexmark International, Inc. Identifying consumer products in images
US10460458B1 (en) * 2017-09-14 2019-10-29 United States Of America As Represented By The Secretary Of The Air Force Method for registration of partially-overlapped aerial imagery using a reduced search space methodology with hybrid similarity measures
JP7076974B2 (en) * 2017-10-03 2022-05-30 キヤノン株式会社 Image processing equipment, control methods and programs

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5659633A (en) * 1991-11-29 1997-08-19 Advanced Recognition Technologies, Inc. Character recognition method utilizing compass directions and torsion points as features

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4783829A (en) 1983-02-23 1988-11-08 Hitachi, Ltd. Pattern recognition apparatus
US4891762A (en) 1988-02-09 1990-01-02 Chotiros Nicholas P Method and apparatus for tracking, mapping and recognition of spatial patterns
USRE36656E (en) * 1991-05-21 2000-04-11 International Business Machines Corporation Generalized shape autocorrelation for shape acquistion and recognition
GB9201856D0 (en) * 1992-01-29 1992-03-18 British Telecomm Method of forming a template
KR0158038B1 (en) 1993-03-29 1998-12-15 모리시다 요오이찌 Apparatus for identifying person
US6393159B1 (en) * 1998-06-29 2002-05-21 The Regents Of The University Of California Multiscale characterization and analysis of shapes
US6384826B1 (en) * 1998-08-14 2002-05-07 Xerox Corporation Method, apparatus and computer medium for surface reconstruction by Voronoi filtering

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5659633A (en) * 1991-11-29 1997-08-19 Advanced Recognition Technologies, Inc. Character recognition method utilizing compass directions and torsion points as features

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHOO C.Y. ET AL.: "Feature-based recognition of triangulated arbitrary surfaces", PROCEEDINGS OF RENSSELAER'S SECOND INTERNATIONAL CONFERENCE ON COMPUTER INTEGRATED MANUFACTURING, May 1990 (1990-05-01), pages 105 - 112, XP010016949 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102609948A (en) * 2012-02-10 2012-07-25 浙江理工大学 Manipulation detection method for copy-paste distorted photo digital photos
CN103903249A (en) * 2012-12-27 2014-07-02 纽海信息技术(上海)有限公司 Image matching system and method
CN103903249B (en) * 2012-12-27 2017-10-13 北京京东尚科信息技术有限公司 Image matching system and method
CN104408726A (en) * 2014-12-02 2015-03-11 上海许书华电子科技有限公司 Triangle-based image contour matching algorithm
CN104408726B (en) * 2014-12-02 2019-05-17 上海许书华电子科技有限公司 A kind of image outline matching process based on triangle

Also Published As

Publication number Publication date
US20020154819A1 (en) 2002-10-24
US6542638B2 (en) 2003-04-01

Similar Documents

Publication Publication Date Title
WO2002069249A1 (en) A method for matching spatial patterns
Soltanpour et al. A survey of local feature methods for 3D face recognition
Andreopoulos et al. 50 years of object recognition: Directions forward
Sung Learning and example selection for object and pattern detection
Ullman High-level vision: Object recognition and visual cognition
Grigorescu et al. Distance sets for shape filters and shape recognition
Ullman An approach to object recognition: Aligning pictorial descriptions
Chowdhary 3D object recognition system based on local shape descriptors and depth data analysis
Lim et al. Planar arrangement of high-dimensional biomedical data sets by isomap coordinates
Mokhtarian et al. Robust automatic selection of optimal views in multi-view free-form object recognition
Cupec et al. Object recognition based on convex hull alignment
Tang et al. Principal curvature measures estimation and application to 3D face recognition
Bronstein et al. Feature-based methods in 3D shape analysis
CN111680571A (en) LDA (latent Dirichlet Allocation) dimensionality reduction-based multi-feature fusion plant leaf identification method
Liang et al. Mesh-based scale-invariant feature transform-like method for three-dimensional face recognition under expressions and missing data
Zubair Machine Learning Based Biomedical Image Analysis and Feature Extraction Methods
Song et al. Shape-based 3D model retrieval
Murase et al. Three‐dimensional object recognition from appearance—Parametric eigenspace method
Pears et al. 3d face recognition
Huttenlocher et al. Recognizing rigid objects by aligning them with an image
Fisher et al. Integrating iconic and structured matching
Swets The self-organizing hierarchical optimal subspace learning and inference framework for view-based object recognition and image retrieval
Stein Structural indexing for object recognition
Tang et al. Invariant object recognition using a neural template classifier
Forsyth et al. Shape models and object recognition

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP