WO2005003948A1

WO2005003948A1 - Control system and control method

Info

Publication number: WO2005003948A1
Application number: PCT/JP2004/006643
Authority: WO
Inventors: Shigeru Enomoto; Yoshinori Washizu; Ryuji Yamamoto; Munetaka Tsuda; Tomonori Shimomura; Junichi Rekimoto
Original assignee: Sony Computer Entertainment Inc.; Sony Corporation
Priority date: 2003-07-08
Filing date: 2004-05-18
Publication date: 2005-01-13
Also published as: JP4723799B2; JP2005031799A

Abstract

A user interface is provided which is excellent in operability. In a control system, there is provided, on the inner side of a display screen (34) of a display device (32), a noncontact input device (26) that determines a shape or action of an input part including at least a part of the body of a user or at least a part of an object operated by the user or determines a distance to the input part. An analyzing part analyzes the shape or action of the input part or the distance thereto determined by the noncontact input device (26) to determine an user's instruction. A control part executes a function corresponding to the user's instruction determined by the analyzing part.

Description

Specification

Control system and control method

Technical field

The present invention relates to control technology for electronic devices and the like, and particularly relates to the shape or motion of an input part including a part of a user's body, an object operated by the user, or a distance to the input part. The present invention relates to a control system and a control method having a user interface capable of inputting a user's instruction. Background art

[0002] With the development of IT, a large number of electronic devices having advanced functions have appeared. As the functions of electronic devices become more sophisticated, the content of instructions for users to execute functions on electronic devices is diversifying, and a user interface (UI) that is easy to operate and has excellent operability is needed. Sex is increasing. For example, in the case of personal computers (PCs), a character-based user interface that previously allows the user to enter command strings using a keyboard has been mainly used. Is displayed on a display screen, and a user operates a pointer on the display screen with a pointing device such as a mouse to input an instruction.

Disclosure of the invention

Problems to be solved by the invention

[0003] With the advent of the GUI, the user has been relieved of the trouble of memorizing and inputting a command character string for causing a PC to execute a desired function. However, despite the widespread use of PCs by the masses, some users, especially children and the elderly, are not accustomed to operating a mouse and feel a sense of resistance. Others are calling for a UI that allows them to enter instructions with a feeling closer to the real world. In the future, it is expected that electronic devices such as PCs will further penetrate into daily life and be used by more diverse user layers, so the development of UIs that are more intuitive, easy to understand, and easy to operate will be developed. desired.

[0004] The present invention has been made in view of such a situation, and its object is to provide an excellent operability. In providing a user interface.

Means for solving the problem

[0005] One embodiment of the present invention relates to a control system. The control system includes a detection unit that detects a shape or an operation of one or more input parts including at least a part of a user's body or at least a part of an object operated by the user, or a distance to the input part. The analysis unit that determines the user's instruction by praying the shape or motion of the input part detected by the input unit or the distance to the input part, and the function corresponding to the user's instruction determined by the analysis unit And a control unit to execute.

[0006] The detection unit may be an imaging device capable of acquiring distance information to an input part. The detection unit may be an input device that includes a plurality of electrodes and detects a change in capacitance between the electrodes due to approach of an input site.

[0007] It is to be noted that any combination of the above-described components and any conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as embodiments of the present invention.

The invention's effect

[0008] According to the present invention, it is possible to provide a user interface with excellent operability.

BEST MODE FOR CARRYING OUT THE INVENTION

[0009] (First embodiment)

FIG. 1 shows a configuration of a control system 10 according to the first embodiment. The control system 10 displays an input device 20 for inputting an instruction from a user, a control device 40 for controlling an operation of an application or the like according to the instruction input from the input device 20, and an image output from the control device 40. And a display device 30 to be used. The control system 10 of the present embodiment includes a user interface (hereinafter, referred to as a “user interface”) that allows a user to input an instruction by an action (gesture) using a part of the body such as a finger, hand, foot, or head. A “gestural interface”). This enables the user to handle objects displayed on the display screen in the same way as in the real world, and to input instructions to applications, resulting in an intuitively superior force and operability. A user interface can be realized. In the present embodiment, the input device 20 is an input device including at least a part of a user's body. Has the function of a detection unit that detects the shape or movement of the part, or the distance to the input part

FIG. 2 shows a configuration example of the input device 20 and the display device 30 according to the embodiment. In the example shown in FIG. 2, as the display device 30, an arbitrary display device 32 such as a liquid crystal display device, a plasma display device, or a cathode ray tube (Cathord Ray Tube: CRT) display device is used. An imaging device 22 having a distance measuring function, which is provided integrally with or separately from the imaging device 32, is used. The control device 40 analyzes the operation of the user photographed by the imaging device 22 by image processing, determines a gesture performed by the user, and acquires a user instruction. According to this configuration example, a wide range of the user's body can be imaged and its operation can be determined, so that the user can give an instruction by a gesture using a hand or foot that can be moved with just a finger. it can. This method is suitable when the user makes a gesture at a position some distance from the imaging device 22. In the case of adopting a digital interface of a system that does not depend on the distance to the user, an imaging device having no distance measuring function may be used as the input device 20, as described later. In order to provide a gesture interface that allows a user to handle an object on the display screen with a finger or the like, it is preferable to use the imaging device 22 having a distance measurement function.

FIG. 3 shows another configuration example of the input device 20 and the display device 30 according to the embodiment.

In the example shown in FIG. 3, a projector 36 that projects an image on a screen 38 is used as the display device 30, and an imaging device 22 having a distance measuring function is used as the input device 20. In the example of FIG. 3, an image is projected on a transparent or translucent screen 38 made of glass or the like provided in front of the user by a projector 36 provided on the upper rear side of the user, and the image displayed on the screen 38 is displayed. A user making a gesture toward the camera is photographed by the imaging device 22 provided on the opposite side of the screen 38. According to this configuration example, it is possible to acquire the motion of a wide range of the body of the user and to perform the motion while directly touching the image projected on the user force screen 38 or in the vicinity of the image. The ability to give the user an operational feeling as if he / she is directly handling the projected object with his / her hand. Further, since the imaging device 22 can be arranged at a position distant from the screen 38, even if the user makes a gesture near the screen 38, the user's body part The distance to the position can be detected with high accuracy.

FIG. 4 shows another configuration example of the input device 20 and the display device 30 according to the embodiment. In the example shown in FIG. 4, an arbitrary display device 32 such as a liquid crystal display device, a plasma display device, or a CRT display device is used as the display device 30, and a display screen 34 of the display device 32 is used as the input device 20. The touch panel 24 provided inside is used. Alternatively, an image may be projected on the surface of the touch panel 24 by a projector. The touch panel 24 can be any type of touch panel, such as a resistance pressure-sensitive method or an infrared detection method. According to this configuration example, the user can input an instruction while directly touching an object or the like displayed on the display screen 34.

FIG. 5 shows still another configuration example of the input device 20 and the display device 30 according to the embodiment. In the example shown in FIG. 5, an arbitrary display device 32 such as a liquid crystal display device, a plasma display device, and a CRT display device is used as the display device 30, and the display screen 34 of the display device 32 is used as the input device 20. A non-contact type input device 26 provided inside is used. Alternatively, a configuration in which an image is projected on the surface of the non-contact input device 26 by a projector may be employed. The non-contact input device 26 is an input device capable of detecting the shape of an object, such as a fingertip of a user, and the distance to the object when the object approaches, for example, Japanese Patent Application Laid-Open No. 2002-342033. The input device disclosed in US Pat. The non-contact type input device disclosed in Japanese Patent Application Laid-Open No. 2002-342033 is provided with a plurality of linear electrodes arranged vertically and horizontally, and when a conductive object such as a fingertip of a user approaches the electrodes. At the same time, it detects the change in capacitance according to the degree of approach, and obtains three-dimensional position information of an object near the input device. According to this configuration example, the non-contact input device 26 can be provided close to the display screen 34, and it is possible to accurately detect the user's movement and the shape of the body part near the display screen 34. Therefore, the user can input an instruction with an input part such as a finger near the image while viewing the image displayed on the display screen 34. With the non-contact input device 26, only the part of the user's body that is in the vicinity of the input device is detected, eliminating the need to extract a specific part to analyze the motion and simplifying the processing. can do. In addition, since the shape and distance can be detected simply by approaching, it is possible to input instructions without touching the display screen 34. Therefore, even a user who feels resistance to directly touching the display screen 34 can be used comfortably. Furthermore, since the user who does not need to press the display screen 34 strongly can detect the approach before touching the display screen 34, it is possible to provide a user interface with good responsiveness and excellent operational feeling.

[0014] Which of the plurality of configuration examples described above is adopted may be determined according to the environment of the place where the control system 10 is installed, the type of application or content to be controlled, and the like. Les ,. For example, in an application in which a plurality of users share and use one display device 30, or in an application in which a content such as a movie is played, the user needs to keep a relatively large distance from the display device 30. Since it is assumed that an instruction is input, the configuration example shown in FIG. 2 or FIG. 3 may be adopted. Further, in the case of an application that one user uses personally, for example, when editing data such as images and documents, it is assumed that the distance between the user and the display device 30 is relatively short. Therefore, the configuration example shown in FIG. 4 or FIG. 5 may be adopted. When constructing the control system 10 using a plurality of configuration examples in combination, it is necessary to select which type of the gesture interface is to be adopted for each application or content, and to switch and use the input device 20 as appropriate. You may.

FIG. 6 shows an internal configuration of the control device 40. In terms of hardware, this configuration can be realized by any computer CPU, memory, or other LSI, and in software, the power realized by programs loaded into the memory, etc. The functional blocks are drawn. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. The control device 40 includes an acquisition unit 42 that acquires an input signal detected by the input device 20, an analysis unit 44 that analyzes a user operation from the input signal acquired by the acquisition unit 42, and determines a user instruction, And a control unit 46 for executing a function corresponding to the user's instruction determined by the analysis unit 44.

When the input device 20 is an imaging device with a distance measuring function, the analysis unit 44 acquires an image having distance information captured by the imaging device, and determines a user operation by image processing. The analysis unit 44 uses a shape recognition technique to detect a part of the user's body, such as the head, eyes, hands, fingers, The motion of the user may be determined by extracting a foot or the like and analyzing the motion of the extracted body part. When the input device 20 is a touch panel, the analysis unit 44 may determine the user's operation by analyzing the shape or time change of the input signal. When the input device 20 is a non-contact type input device, the analysis unit 44 may determine the user's motion by analyzing the shape and distance of the input signal and their time change.

FIG. 7 shows hardware components of the control device 40. The control device 40 includes a CPU 120, an input interface 122, a display interface 124, a memory 130, a node disk 132, and a drive device 128. These components are electrically connected by a signal transmission path such as a bus 126.

The input interface 122 acquires an input signal detected by the input device 20. The display interface 124 outputs an image signal to be displayed on the display device 30. The hard disk 132 is a large-capacity magnetic storage device, and stores various files. The recording medium 140 records a program for causing the CPU 120 to realize the functions of the control device 40 described above. When the recording medium 140 is inserted into the drive device 128, the program is read into the memory 130 or the hard disk 132, and the CPU 120 performs the control processing of the present embodiment by the read program. The recording medium 140 is a computer-readable medium such as a CD-ROM, a DVD, and an FD.

Here, in the power example described above in which the program is recorded on the recording medium 140, the program may be transmitted from an external server regardless of whether it is wireless or wired. In the hardware configuration shown in FIG. 7, the program is stored in the hard disk 132 in advance, not only when the program is realized from the outside but also when the computer realizes the control function of the present embodiment. It is well understood by those skilled in the art that this is the case.

The function of the control unit 46 may be realized by an operating system (オペレーティング S) executed by a CPU or the like of a computer, an input / output control application, or the like. For example, in the case of an OS that employs a GUI such as a window system, the user's instruction is notified to the application of the topmost window near the location where the input part such as the user's finger has approached, and the Application responds to user instructions The associated function may be executed. If there is no window at that position, the OS or I / O control application may execute the function associated with the instruction.

[0021] 1. Example of instruction input by multiple points

In a conventional user interface using a pointing device, a force is used to input an instruction with a single pointer. According to the artificial interface provided by the control system 10 according to the present embodiment, the user can use fingers, hands, and feet. An instruction can be given by using a plurality of input units using, for example.

FIG. 8 and FIG. 9 are diagrams for explaining an example of inputting an instruction by the movement of a plurality of fingers of the user. With the thumb and index finger closed as shown in FIG. 8, the user brings his / her finger close to the icon 200 displayed on the screen of the display device 30, and opens the thumb and index finger as shown in FIG. Is performed. At this time, the acquisition unit 42 sends the input signal detected by the input device 20 to the analysis unit 44, and the analysis unit 44 analyzes the movement of the user's finger and determines that the user has issued an instruction to open the finger. I do.

If the input device 20 is a camera with a distance measurement function, the analysis unit 44 extracts the user's hand by using a technique such as shape recognition and tracks the movement of the finger, and performs an operation of opening the finger by the user. Determine what you have done. When the input device 20 is a touch panel or a non-contact type input device, the analysis unit 44 divides a set of input parts approaching or touching coordinates near the icon 200 on the display screen into two, and It is determined that the user has performed the operation of opening the finger when the user has moved away from the camera. Only when a group of input parts is divided into two parts When two input parts move in a direction away from each other, it may be determined that the user has performed the operation of opening the finger.

When the user performs the operation of opening the finger, the control unit 46 executes a function associated with the operation of opening the finger. For example, the control unit 46 may execute a function of activating an application associated with the icon 200 displayed near the user's finger. If the icon is associated with a file, an application that can handle the file may be started and the function to open the file may be executed. In this way, the "open finger" operation is associated with functions such as "open application" and "open file". By doing so, the user's operation and the application's operation are intuitively matched, so that a more intuitive user interface with a higher level of affinity and immediate affinity can be realized.

[0025] The operation of opening a finger may be associated with functions such as "start", "determine", and "determine". In addition, each application may implement a function corresponding to the operation of opening the finger. For example, in an image processing application, when a user performs an operation of opening a finger on an image, a function of enlarging the portion or extending the portion in the direction in which the finger is opened may be executed.

[0026] Conversely, as shown in FIG. 9, the user moves his / her finger close to the icon 200 displayed on the screen of the display device 30 with the thumb and the index finger open, and as shown in FIG. When the thumb and the forefinger are closed, the analyzing unit 44 determines that the user has performed the finger closing operation on the icon 200, and the control unit 46 performs the function associated with the finger closing operation. You can do it. For example, the control unit 46 may execute a function of terminating the application associated with the icon 200 or the window displayed near the user's finger, or may execute the function of terminating the icon 200 or the file associated with the window. You may perform the function to close the. Further, in an image processing application, when a user performs an operation of closing a finger on an image, a function of reducing that part or reducing the part in a direction in which the finger is closed may be executed.

[0027] In the above example, the operation of opening or closing a finger has been described. In addition, the control unit 46 receives an instruction using a plurality of input sites including a user's finger, hand, and the like, and performs the operation. The associated function may be executed. For example, in a book browsing application, when a user performs an operation of pinching a corner of a displayed book page, the user may memorize the page and execute a “pinch bookmark” function. . In a role-playing game, when the user performs an operation of holding an item by placing a finger near the displayed item, a function of “picking up” the item may be executed.

[0028] 2. Example of instruction input using shape

FIG. 10 is a diagram for explaining an example of inputting an instruction based on the shape of the user's hand. As shown in FIG. 9, it is assumed that the user places his palm on the display screen with his / her hand open. If the input device 20 is a camera with a distance measurement function, the analysis unit 44 The shape of the hand is determined by extracting the hand of the user by the technique. When the input device 20 is a touch panel or a non-contact type input device, the analysis unit 44 extracts a feature point from a shape of an object approaching or touching the display screen by using a known technique. Alternatively, the shape of the object is determined by performing evaluation using a predetermined evaluation formula. A general hand shape may be stored in a database, and the shape of the hand may be determined by extracting a matching shape from the database. The shape may be determined based on the area of the detected object. For example, when the user places his / her hand on the display screen, the area is the largest when the hand is open and the smallest when the user holds the hand. Utilizing this fact, the shape of the hand may be determined. When the user places his / her palm on the screen with his / her hand open, the control unit 46 performs a function corresponding to the shape, for example, an application corresponding to the window 210 which is displayed at the highest position in the position where the palm is placed. Perform the function to "quit". By associating the actions of “put your palm” with functions such as “stop” and “end”, it is possible to realize a user interface that is intuitive and easy to operate.

FIG. 11 is a diagram illustrating an example of mapping an object to a hand shape. As shown in Fig. 11, the user moves a finger from the position of the object 220 on the display screen to the left hand with the other hand, here the right hand, and creates a specific shape with the other hand, here the left hand. Suppose you went. The method of determining the shape of the hand is the same as in the example described above. At this time, the control unit 46 executes a function of storing a file corresponding to the object 220 moved by the finger of the right hand in a storage location of the hard disk 132 corresponding to the shape of the left hand. The storage location corresponding to the shape of the left hand may be a directory or a folder in the file system, or may be a virtual folder. You can also associate the file with the shape of your hand. It is not only necessary to associate one storage location with one shape. For example, the storage location may be associated with each finger of an open hand.

FIG. 12 shows an example of a table for mapping an object to a hand shape. The table 230 is held in the memory 130 or the hard disk 132 of the control device 40. The table 230 has a shape column 232 and a storage location column 234. The shape column 232 holds an image representing a hand shape, parameters, and the like. Recognize hand shapes as images When recognizing the image, the file name of the image file may be stored.When recognizing the shape of the hand as a parameter using feature points or evaluation formulas, the parameter or the file in which the parameter is stored You can keep the file name and so on. The storage location column 234 holds the storage location of the object. The hand shape and the storage location corresponding to the shape may be registered in advance, or if the hand shape is an unregistered shape when the user performs the operation shown in FIG. 11, The controller 46 may register the shape and storage location of the hand in the table 230. At this time, the user may specify the storage location, or an appropriate storage location may be automatically assigned. When associating the file itself with the hand shape, the storage location field 234 holds the file name of the file.

FIG. 13 is a view for explaining an example of extracting an object stored by the operation shown in FIG. As shown in Fig. 13, the user creates the shape of the hand corresponding to the storage location of the object to be retrieved with one hand, here the left hand, and moves away from the position near the left hand with the other hand, here the right hand. Perform the operation of moving your finger. At this time, the control unit 46 specifies the object stored in the storage location corresponding to the shape of the left hand, and displays it on the display screen. The control unit 46 may execute a function of opening a file corresponding to the object.

[0032] 3. Example of instruction input according to distance

In the configuration example using the imaging device 22 with the distance measuring function shown in FIGS. 2 and 3 and the configuration example using the non-contact type input device 26 shown in FIG. The distance to can be detected. Hereinafter, an example of the instruction input according to the distance will be described.

FIG. 14 is a diagram for explaining an example of an instruction input according to a distance. In the example shown in FIG. 14, the control unit 46 displays a moving image in which a fish is swimming on the display device. Here, when the user approaches the display screen and the distance from the display screen to the user's hand falls below a predetermined threshold, the control unit 46 sets the moving image in which the fish escapes from the vicinity of the user's hand. Switch the display to the image. As a result, it is possible to give a sense of realism as if a real living fish were swimming on the display screen. Although related to Example 4 below, the speed at which the user's hand approaches is calculated, and when the speed is fast, the threshold for switching the display is increased, and the hand is at a far position. The fish may start to escape from time to time, and when the speed is slow, the threshold value may be reduced so that the fish does not escape until the hand comes close. When the distance between the user's hand and the display screen is equal to or greater than a predetermined value, the control unit 46 displays a moving image in which the fish is freely swimming, and when the distance is smaller than the predetermined value, the fish is displayed. May display a moving image of swimming while avoiding the position of the user's hand. Since the non-contact input device can accurately detect distance information near the display screen, in this example, it is more preferable to use the non-contact input device as the input device 20.

When an imaging device is used as the input device 20, if the user is too far from the imaging device, there is a possibility that the user's instruction cannot be determined accurately due to the relationship between the distance measurement function and the accuracy of the image processing. In addition, when a non-contact input device is used as the input device 20, unless a part such as a user's hand approaches the input device 20, the capacitance does not change and cannot be detected. As described above, since there is an upper limit value of the distance at which the user can input an instruction according to the characteristics of the input device 20, the above-described example of a moving image in which a fish swims indicates whether or not the instruction can be input. It may be used as a means. For example, if the position of the user's hand is too far away from the input device and cannot input an instruction, a moving image of the fish swimming freely is displayed, and input is not possible. Is presented to the user, and when the user approaches the input device and reaches a distance at which an instruction can be input, a moving image in which a fish swims around the user's hand to avoid the fish is displayed, and input can be performed. May be presented to the user. The control unit 46 previously holds a distance threshold value for determining whether or not an instruction input to the input device 20 is possible.

4. Example of Instruction Input According to Movement Amount, Speed, and Acceleration

FIG. 15 and FIG. 16 are diagrams for explaining the function of moving the object displayed on the screen. In the examples of FIG. 15 and FIG. 16, the user tries to move the object displayed on the display device 30 by shifting it with a finger. At this time, the control unit 46 detects the movement amount, speed, and acceleration of a part such as a user's finger, and determines the movement amount of the object accordingly. For example, the weight of each object, that is, the virtual energy value required to move the object is set in advance, and the energy value and the amount of movement when the user moves his or her finger to move the object are set. , Speed, or acceleration The moving state of the object is controlled based on the degree. As shown in FIG. 15, the control unit 46 moves the user's finger quickly for a light object, as shown in FIG. Does not move the object if the finger is quickly moved. At first, the heavy object starts moving by slowly moving the finger of the user, and moves gradually following the movement of the finger. This makes it possible to simulate a moving state simulating actual static friction and dynamic frictional resistance, and to give the user an operational feeling reflecting the weight of the object. The technology described in this example can be used in a user interface using a conventional pointing device.

The control unit 46 may determine the magnitude of the force applied to the object by the user and control the moving state of the object according to the magnitude of the force. When an imaging device or a non-contact type input device is used as the input device 20, the magnitude of the force may be determined based on the distance between the input device 20 and the user's hand. For example, the closer the user's hand is to the display screen, the more force may be applied to the object. When a touch panel is used as the input device 20, the magnitude of the force may be determined based on the magnitude of the pressing force. When a conventional pointing device is used, the magnitude of the force may be determined based on the degree of pressing of a pressure-sensitive mouse or the like.

FIG. 17 is a diagram for explaining a paper handling function. In FIG. 17, the control unit 46 displays an image of a book opened on the display device 30. When the user places his finger on the page and slowly moves his finger, the control unit 46 turns the page gradually following the movement of the user's finger, and turns the page when it exceeds a predetermined position. Displays a moving image on the next page. When the user moves his finger quickly, the control unit 46 displays an image in which the paper cannot follow the movement of the finger, but turns halfway and returns. As a result, it is possible to give the user a sense of presence as if he / she is actually reading a book. The control unit 46 may hold in advance a threshold value of the speed used to determine whether the paper can follow the movement of the user. As another example of handling paper, when the user's finger moves too quickly, the control unit 46 indicates that the paper makes a wrinkle or performs an operation such as pinching or tearing the paper. At this time, the paper may be made uneven, or the paper may be displayed as torn. [0038] As another example of the instruction input according to the moving amount, the speed, and the acceleration, when the user moves an object with a finger or the like and passes over another object, the other object is changed according to the moving speed. A predetermined process may be performed on the object. For example, the function of opening the file corresponding to the moving object by the application corresponding to the passed object may be executed.

[0039] 5. Example of instruction input for 3D object

Conventionally, when displaying a three-dimensional object on a display screen and accepting operations such as moving and rotating the three-dimensional object from the user, it is difficult to input three-dimensional instructions because the movement of the pointer is two-dimensional. There was a problem. In the configuration example using the imaging device 22 with the distance measuring function shown in FIGS. 2 and 3, and the configuration example using the non-contact type input device 26 shown in FIG. Since it can be obtained in a three-dimensional manner, it is possible to provide a gesture interface suitable for handling a three-dimensional space.

FIG. 18 is a diagram for describing an example of handling a three-dimensional object. When the user performs a gesture of grasping the object 240 and turning it in the depth direction of the paper near the three-dimensional object 240 displayed on the display device 30 by the control unit 46, the control unit 46 performs the same operation as the direction in which the user turns his hand. Rotate the 3D object 240 in the direction. As described above, according to the gesture interface of the present embodiment, it is possible to give an operational feeling as if a 3D object is actually handled by hand, and to provide a user interface for handling a 3D space. Operability can be dramatically improved.

(Second Embodiment)

In the second embodiment, a technique for inputting an instruction based on the shape or motion of an object operated by the user, which is not part of the body of the user, or the distance to the object will be described. The overall configuration of the control system 10 of the present embodiment and the internal configurations of the input device 20, the display device 30, and the control device 40 are the same as those of the first embodiment. In the present embodiment, a configuration example using the non-contact input device 26 shown in FIG. 5 will be mainly described, but the same applies to a case where another configuration example is used. In the case of the non-contact input device 26 that detects a change in capacitance, a conductive object is placed on the display screen, and when the user touches the object, the capacitance changes and the shape of the object is detected. You. Utilizing this, conductive objects A user interface can be constructed by associating predetermined functions with shapes and movements of the user.

FIG. 19 is a diagram for explaining an example in which an instruction is input based on the shape and movement of an object.

In the example shown in FIG. 19, a window 250 of a music reproduction application controlled by the control unit 46 is displayed on the display device 30. At this time, it is assumed that the user places the volume control unit 260 having a shape corresponding to the volume control function on the display screen and moves the slider 262 left and right. The analysis unit 44 analyzes the shape of the input signal, detects that the input signal is the volume control unit 260, analyzes the movement of the knob 262, and transmits it to the control unit 46. The control unit 46 controls the volume in the music playback application according to the amount of movement of the knob 262.

[0043] Since the non-contact input device that detects a change in capacitance detects only a conductive object, a specific shape is drawn by a conductive wire on the bottom surface of the insulating object, and the shape and a predetermined function are determined. It may be associated.

The present invention has been described based on the embodiments. This embodiment is an exemplification, and it is understood by those skilled in the art that various modifications can be made to the combination of each component and each processing process, and that such modifications are also within the scope of the present invention. By the way.

In the embodiment, an example has been described in which control system 10 corresponds to an electronic device such as a personal computer. In another example, a non-contact input device 26 may be used as the input device 20, and the display device 30 and the input device 20 may be provided on the tabletop of the table so that a game or the like can be enjoyed thereon. Further, the display device 30 and the input device 20 may be provided on a floor of a passage or the like to display a footprint of a user walking or to navigate a destination of the user by using an image or light.

Industrial applicability

The present invention can be applied to a user interface for controlling an electronic device or the like.

Brief Description of Drawings

FIG. 1 is a diagram showing a configuration of a control system according to an embodiment.

FIG. 2 is a diagram showing a configuration example of an input device and a display device according to an embodiment.

FIG. 3 is a diagram showing another configuration example of the input device and the display device according to the embodiment. FIG. 4 is a diagram showing still another configuration example of the input device and the display device according to the embodiment.

FIG. 5 is a diagram showing still another configuration example of the input device and the display device according to the embodiment.

FIG. 6 is a diagram showing an internal configuration of a control device.

FIG. 7 is a diagram showing hardware components of the control device.

FIG. 8 is a diagram for explaining an example of inputting an instruction by movement of a plurality of fingers of a user.

FIG. 9 is a diagram for explaining an example in which an instruction is input by movement of a plurality of fingers of a user.

FIG. 10 is a diagram for explaining an example of inputting an instruction according to the shape of a user's hand.

FIG. 11 is a diagram illustrating an example of mapping an object to a hand shape.

FIG. 12 is a diagram showing an example of a table for mapping objects to hand shapes.

13 is a diagram for explaining an example of retrieving an object stored by the operation shown in FIG.

FIG. 14 is a diagram for explaining an example of an instruction input according to a distance.

FIG. 15 is a diagram for explaining a function of moving an object displayed on a screen.

FIG. 16 is a diagram for explaining a function of moving an object displayed on a screen.

FIG. 17 is a diagram for explaining a function for handling paper.

FIG. 18 is a diagram for describing an example of handling a three-dimensional object.

FIG. 19 is a diagram for explaining an example of inputting an instruction based on the shape and movement of an object.

Explanation of symbols

10 "Control system, 20" Input device, 22 Image sensor, 24 Touch panel, 26 Non-contact type input device, 30 Display device, 40 Control device, 42 ··· Acquisition unit, 44 ··· Analysis unit, 46 ··· Control unit.

Claims

The scope of the claims

[1] a detection unit that detects an operation of a plurality of input parts including at least a part of a user's body or at least a part of an object operated by the user;

An analysis unit that analyzes an operation of the input part detected by the detection unit and determines a user instruction;

A control unit that executes a function corresponding to a user's instruction determined by the analysis unit.

2. The control system according to claim 1, wherein, when the user performs an operation of opening or closing a finger, the control unit executes a function associated with the operation.

[3] a display unit for presenting an image to a user is further provided;

When the user performs an operation of opening or closing a finger in the vicinity of the object displayed on the display unit, the control unit is associated with the operation of opening or closing the finger in an application associated with the object. 3. The control system according to claim 2, wherein the control system executes the function.

[4] a detection unit that detects a shape or an operation of an input part including at least a part of a user's body or at least a part of an object operated by the user, or a distance to the input part; An analysis unit that analyzes the shape or motion of the input part and the distance to the input part to determine a user instruction,

[5] The control unit associates the shape of the input part with the storage location of the data, and when the data storage request is received from a user, the control part corresponds to the shape of the input part detected by the detection unit. The control system according to claim 1, wherein the data is stored in a storage location.

[6] The control unit reads the storage location force data corresponding to the shape of the input part detected by the detection unit when a data read request is received from a user. 5. The control system according to 5.

[7] Further comprising a display unit for presenting an image to a user,

When the distance to the input part is equal to or more than a predetermined threshold value, the control unit displays an image indicating the fact on the display unit, and the distance to the input part is a predetermined threshold value. 5. The control system according to claim 4, wherein when the value falls below the threshold, the image is switched to an image indicating the fact.

[8] The control system according to claim 7, wherein the threshold and the value are determined based on accuracy when the analysis unit analyzes information on the input part detected by the detection unit.

[9] The control method according to claim 1, wherein the control unit calculates a movement amount, a speed, or an acceleration of the input part, and controls the function based on at least one of them. Control system.

[10] A display unit for presenting an image to a user is further provided.

The control unit controls the user to move the object displayed on the display unit.

The moving amount of the object is determined according to the moving amount, speed, or acceleration of the input portion when the input portion is moved close to the object. The control system as described.

[11] The control unit sets a parameter representing the weight of the object displayed on the display unit, and determines the amount of movement of the object by further considering the parameter. The control system according to claim 10, wherein

[12] The control system according to any one of [1] to [11], wherein the detection unit is an imaging device capable of acquiring distance information to the input part.

13. The input device according to claim 1, wherein the detection unit is an input device that includes a plurality of electrodes and detects a change in capacitance between the electrodes due to the approach of the input site. The control system as described.

[14] detecting a shape or an action of an input part including at least a part of a user's body or at least a part of an object operated by the user, or a distance to the input part;

Analyzing the detected shape or motion of the input part or the distance to the input part to determine a user's instruction; Performing a function corresponding to the determined user's instruction;

A control method comprising:

[15] Information detected from a detection unit that detects a shape or an operation of an input part including at least a part of a user's body or at least a part of an object operated by the user, or a distance to the input part, A function of acquiring and analyzing the information to determine a user instruction; and a function of executing a function corresponding to the determined user instruction;

Computer program for causing a computer to realize the following.

[16] Information detected from a detection unit that detects a shape or an operation of an input part including at least a part of a user's body or at least a part of an object operated by the user, or a distance to the input part, A function of acquiring and analyzing the information to determine a user instruction; and a function of executing a function corresponding to the determined user instruction.

A computer-readable recording medium on which a program for causing a computer to realize the above is recorded.