Gesture-controlled interfaces for self-service machines and other applications

(12) United States Patent

Cohen et al.

US006950534B2

(io) Patent No.: US 6,950,534 B2 (45) Date of Patent: Sep. 27,2005

(54) GESTURE-CONTROLLED INTERFACES FOR SELF-SERVICE MACHINES AND OTHER APPLICATIONS

(75) Inventors: Charles J. Cohen, Ann Arbor, MI (US); Glenn Beach, Brooklyn, MI (US); Brook Cavell, Ann Arbor, MI (US); Gene Foulk, Ann Arbor, MI (US); Charles J. Jacobus, Ann Arbor, MI (US); Jay Obermark, Ann Arbor, MI (US); George Paul, Bedford, NH (US)

(73) Assignee: Cybernet Systems Corporation, Ann

Arbor, MI (US)

( * ) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

(21) Appl. No.: 10/759,459

(22) Filed: Jan. 16, 2004

(65) Prior Publication Data

US 2004/0161132 Al Aug. 19, 2004

Related U.S. Application Data

(63) Continuation-in-part ol application No. 09/371,460, filed on

Aug. 10, 1999, now Pat. No. 6,681,031. (60) Provisional application No. 60/096,126, filed on Aug. 10,

1998.

(51) Int. C I. G06K 9 00

(56) References Cited

U.S. PATENT DOCUMENTS

5,047,952 A 9/1991 Kramer et al 364/513.5

5,423,554 A 6/1995 Davis 273/437

5,454,043 A * 9/1995 Freeman 382/168

5,481,454 A 1/1996 Inoue et al 364/419

(Continued)

OTHER PUBLICATIONS

U.S. Air force Instruction, "Aircraft Cockpit and Formation Flight Signals," May 1994.

(Continued)

Primary Examiner—Kanjibrai Patel

Assistant Examiner—Abolfazl Tabatabai

(74) Attorney, Agent, or Firm—Gifford, Krass, Groh,

Sprinkle, Anderson & Citkowski, PC

(57) ABSTRACT

A gesture recognition interface for use in controlling selfservice machines and other devices is disclosed. A gesture is defined as motions and kinematic poses generated by humans, animals, or machines. Specific body features are tracked, and static and motion gestures are interpreted. Motion gestures are defined as a family of parametrically delimited oscillatory motions, modeled as a linear-inparameters dynamic system with added geometric constraints to allow for real-time recognition using a small amount of memory and processing time. A linear least squares method is preferably used to determine the parameters which represent each gesture. Feature position measure is used in conjunction with a bank of predictor bins seeded with the gesture parameters, and the system determines which bin best fits the observed motion. Recognizing static pose gestures is preferably performed by localizing the body/object from the rest of the image, describing that object, and identifying that description. The disclosure details methods for gesture recognition, as well as the overall architecture for using gesture recognition to control of devices, including self-service machines.

2 Claims, 19 Drawing Sheets

12

12.

U.S. PATENT DOCUMENTS

5,544,050 A 8/1996 Abe et al 364/419

5,563,988 A 10/1996 Maes et al 395/121

5,570,301 A 10/1996 Barrus 364/559

5,581,276 A * 12/1996 Cipolla et al 345/156

5,594,469 A 1/1997 Freeman 345/158

5,612,719 A 3/1997 Beernink et al 345/173

5,652,849 A 7/1997 Conway et al 395/327

5,659.764 A 8/1997 Sakiyama et al 395/753

5,668,573 A 9/1997 Favot et al 345/156

5,670,987 A 9/1997 Doi et al 345/156

5,696,892 A * 12/1997 Redmann et al 345/582

5,699,441 A 12/1997 Sagawa et al 382/100

5,710,833 A 1/1998 Moghaddam et al 382/228

5,714,698 A 2/1998 Tokioka et al 73/865.4

5,732,227 A 3/1998 Kuzunuki et al 395/333

5,757,360 A 5/1998 Nitta et al 345/156

5,759,044 A 6/1998 Redmond 434/307 R

5,767,842 A 6/1998 Korth 345/168

5,798,758 A 8/1998 Harada et al 345/339

5,801,704 A 9/1998 Oohara et al 345/358

5,813,406 A 9/1998 Kramer et al 128/782

5,828,779 A 10/1998 Maggioni 382/165

5,864,808 A 1/1999 Ando et al 704/251

5,864,848 A 1/1999 Horvitz et al 707/6

5,875,257 A 2/1999 Marrin et al 382/107

5,880,411 A 3/1999 Gillespie et al 178/18.01

5,887,069 A 3/1999 Sakou et al 382/100

5,889,236 A 3/1999 Gillespie et al 178/18.01

5,889,523 A 3/1999 Wilcox et al 345/357

5,898,434 A 4/1999 Small et al 345/348

5,901,246 A 5/1999 Hoffberg et al 382/209

5,903,229 A 5/1999 Kishi 341/20

5,907,328 A 5/1999 Brash II et al 345/358

5,907,852 A 5/1999 Yamada 707/541

5,917,490 A 6/1999 Kuzunuki et al 345/351

5,990,865 A * 11/1999 Gard 345/156

6,035,053 A 3/2000 Yoshioka et al 345/351

6,061,055 A * 5/2000 Marks 382/276

6,137,908 A 10/2000 Rhee 382/187

6,144,366 A * 11/2000 Numazaki et al 345/156

6,219,032 Bl * 4/2001 Rosenberg et al 345/157

6,272,231 Bl 8/2001 Maurer et al 382/103

6,301,370 Bl 10/2001 Steffens et al 382/103

6,335,977 Bl 1/2002 Kage 382/107

6,456,728 Bl * 9/2002 Doi et al 382/103

6,804,396 B2 * 10/2004 Higaki et al 382/181

OTHER PUBLICATIONS

U.S. Army Field Manual No. 21-60, Washington, D.C., Sep. 30, 1987.

Arnold, V.L., "Ordinary Differential Equations," MIT Press, 1978.

Cohen, C, "Dynamical System Representation, Generation and Recognition of Basic Oscillatory Motion Gestures and Applications for the Control of Actuated Mechanisms," Ph.D. Dissertation, Univ. of Michigan, 1996.

Frank, D., "HUD Expands Kiosk Program," Federal Computer Week, Mar. 8, 1999.

Hager, G., Chang, W, Morse, A.; "Robot Feedback Control Based on Stereo Vision: Towards Calibration-Free HandEye Coordination," IEEE Int. Conf. Robotics and Automation, San Diego, CA, May 1994.

Hauptmann, A., "Speech and Gestures for Graphic Image Manipulation," Computer Human Interaction 1989 Proa, pp. 241-245, May 1989.

Hirsch, M. Smale, S., "Differential Equations, Dynamical Systems and Linear Algebra," Academic Press, Orlando, FL, 1974.

Kanade, T, "Computer Recognition of Human Faces,"
Birkhauser Verlag, Basel and Stuttgart, 1977.
Karon, P., "Beating an Electronic Pathway to Government
with Online Kiosks," Los Angeles Times, Aug. 25, 1996.
Link-Belt Construction Equipment Co., "Operating Safety:
Crames & Excavators," 1987.

Turk, M., Pentland, A, "Eigenfaces for Recognition," Journal of Cognitive Neuroscience, 3, 1, 71-86, 1991. Narendra, K. Baiakrishnan, J. "Improving Transient Response to Adaptive. Control Systems Using Multiple Models and Switching," IEEE Trans, on Automatic Control, 39:1861-1866, Sep. 1994.

Rizzi, A., Whitecomb, L., Koditschek, D.; "Distributed Real-Time Control of a Spatial Robot Juggler," IEEE Computer, 25(5) May 1992.

Wolf, C, Morrel-Samuels, P., "The use of hand-drawn gesetures for text editing," Int. Journ. of Man-Machine Studies, vol. 27, pp. 91-102, 1987.

Wolf, C, Rhyne, J., "ATaxonomic Approach to Understanding Direct Manipulation," Jour, of the Human Factors Society 31th Annual Meeting, pp. 576-580. Yuille, A., "Deformable Templates for Face Recognition," Journ. of Cognitive Neuroscience, 3, 1, 59-70, 1991. C. Cohen, G. Beach, G. Paul, J. Obermark, G. Foulk, "Issues of Controlling Public Kiosks and other Self Service Machines using Gesture Recognition," Oct. 1998. L. Conway, C. Cohen, "Video Mirroring and Iconic Gestures: Enhancing Basic Videophones to Provide Visual Coaching and Visual Control", no date available. C. Cohen, L. Conway, D. Koditschek, G. Roston, "Dynamic System Representation of Basic and Non-Linear in Parameters Oscillatory Motion Gestures," Oct. 1997. C. Cohen, L. Conway, D. Koditschek, "Dynamic System Representation, Generation, and Recognition of Basic Oscillatory Motion Gestures," Oct. 1996.

C. Cohen, G. Beach, B. Cavell, G. Foulk, J. Obermark, G. Paul, "The Control of Self Service Machines Using Gesture Recognition," (Aug. 1999).

* cited by examiner