US20100167820A1

US20100167820A1 - Human interface device

Info

Publication number: US20100167820A1
Application number: US12/649,277
Authority: US
Inventors: Houssam Barakat; Bradford R. Lilly; Krishna Shenai
Original assignee: Individual
Current assignee: University of Toledo
Priority date: 2008-12-29
Filing date: 2009-12-29
Publication date: 2010-07-01

Abstract

A human interface device with an inherent built-in feedback mechanism for use by a user to remotely interface with a computer-simulated environment is disclosed herein. The human interface device comprises at least one sensor configured to sense a condition within the action of the computer-simulated environment and also operable to generate a communication concerning the sensed condition. At least one micro-controller is positioned within the human interface device and configured to receive the communication concerning the sensed condition from the at least one sensor. The at least one micro-controller is further configured to generate communication in reaction to the communication from the sensor. At least one actuator is configured to receive the communication from the at least one micro-controller and provide a sensory experience in reaction to the communication.

Description

BACKGROUND OF THE INVENTION

This invention relates in general to human interaction with computer-simulated environments. More specifically, this invention relates to multimodal devices having an inherent built-in feedback mechanism that can be used to remotely interface human interaction with computer-simulated environments, such as generated by computer-based gaming and virtual reality systems.
Gaming and virtual reality systems allow a user or a number of users to interact with a computer-simulated environment. A typical system includes a computer that establishes the computer-simulated environment. The computer-simulated environment can be “virtual reality” or based on a real environment such as, for example, simulations for pilot or combat training. The computer-simulated environment can also be for gaming or based on an imagined environment such as, for example, imaginary interplanetary worlds. Most computer-simulated environments are primarily visual experiences, displayed either on a computer screen or through special or stereoscopic displays, but some computer-simulated environments established by computers include additional sensory experiences, such as sound through speakers or headphones or vibration through user input devices such as controllers. Users can typically interact with such computer-simulated environments through the use of standard input devices such as a keyboard and a mouse, or through multimodal devices such as, for example, controllers, wired gloves, joysticks or steering wheels.

SUMMARY OF THE INVENTION

According to this invention, there is provided a human interface device with an inherent built-in feedback mechanism for use by a user to remotely interface with a computer-simulated environment. The human interface device comprises at least one sensor configured to sense a condition within the action of the computer-simulated environment and also operable to generate a communication concerning the sensed condition. At least one micro-controller is positioned within the human interface device and configured to receive the communication concerning the sensed condition from the at least one sensor. The at least one micro-controller is further configured to generate communication in reaction to the communication from the sensor. At least one actuator is configured to receive the communication from the at least one micro-controller and provide a sensory experience in reaction to the communication.
According to this invention, there is also provided a method of providing a sensory experience to a user using the human interface device described above. The method comprises the steps of sensing a condition within the action of the gaming or virtual reality system established by the computer, generating a communication concerning the sensed condition, sending the communication to a human interface device for processing, generating a communication within the human interface device in reaction to the received communication, sending the human interface device communication to at least one actuator; and providing a sensory experience in reaction to the received human interface device communication.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of the major components of a conventional gaming system.

FIG. 2 is a schematic view of the major components of a first embodiment of the invention.

FIG. 3 is a side view, in elevation, of a first embodiment of the human interface device of FIG. 2.

FIG. 4 is a side view, in perspective, of a second embodiment of a human interface device.

FIG. 5 is a side view, in perspective, of a third embodiment of a human interface device.

FIG. 6 is a schematic view of the components of the human interface device shown in FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

For purposes of this patent application, the terms “gaming system” and “virtual reality system” will be used interchangeably and are defined to include any system, structure(s), or device(s) incorporating a technology which allows a user to interact with a computer-simulated environment.
Referring now to the drawings, there is illustrated in FIG. 1 a gaming system, indicated generally at 10, that is conventional in the art. A gaming system is one example of system operable to establish a computer-simulated environment and the invention is not limited to gaming systems. The basic structure and mode of operation of the gaming system 10 are well known in the art, and only those portions of the gaming system 10 that are necessary for a complete understanding of the invention will be described. Some examples of commercially available gaming systems 10 include Wii™ marketed by Nintendo, Playstation® marketed by Sony Corporation, and Xbox® marketed by Microsoft Corporation.
As shown in FIG. 1, a typical gaming system 10 includes at least one human interface device (HID) 12, a plurality of sensors 14, at least one host device or system or device 16, and a display (not shown).
The HID 12 is an input device used to affect the gaming system 10 or govern the movement or actions of an entity within the computer-simulated game/environment. While the HID 12 is typically connected via wires to the host system 16, the HID 12 can also be operated in a wireless mode. As will be explained in more detail below, the HID 12 can comprise many different physical forms.
As shown in FIG. 1, the HID 12 typically includes, among other things, a micro-controller 18 and at least one actuator 20. As will be explained in more detail below, the micro-controller 18 is configured for processing of information and communication of information to a variety of other devices. The actuator 20 is configured to induce a sensory-based reaction by the user of the HID 12. Examples of actuators 20 include vibration inducing or motion causing motors, heat causing elements, sound causing elements and motion causing solenoids. As shown in FIG. 1, the actuator 20 is electrically connected to the micro-controller 18 and configured to respond to signals from the micro-controller 18.
The sensors 14 are configured to provide input of the status of game play or game conditions. The sensors 14 can sense a condition within the action of the gaming system in that the sensors 14 can sense some physical condition associated with the user, such as activity of the user, the physical condition of the user, or the physical environment in which the user is participating in the computer-simulated environment. For example, one example of a sensor 12 is an accelerometer, which can measure the magnitude, direction and force of movement of the user. Other types of sensors 14 can be used to provide input regarding user-initiated actions, such as pulling a trigger and movement of a virtual character. Still other sensors 14 can be used to provide input regarding other conditions, such as for example the heart rate of the user, the temperature and humidity of the gaming environment, and the mental consciousness of the user. While the plurality of sensors 14 illustrated in FIG. 1 is shown as being positioned exterior to the HID 12, it should be understood that the plurality of sensors 14 can be positioned in various locations, including internal to the HID 12. While the embodiment shown in FIG. 1 illustrates a quantity of four sensors 14, it should be understood than more or less than four sensors can be used.
The host system 16 typically includes, among other things, a host central processing unit (CPU) 22 and is configured to control the overall functions of the gaming system. Examples of overall system functions include loading of games software, start up, and shut down. The host CPU 22 typically processes input information from a variety of sources and controls the play of the gaming system.
In operation, the sensors 14 sense a condition potentially affecting the play of the gaming system 10 and generate a communication concerning the sensed condition. The sensors 14 can communicate with the micro-controller 18 positioned within the HID 12 or the host CPU 22. The micro-controller 18 can be configured to receive the communications generated by the individual sensors 14. The communication of the various sensors 14 with the micro-controller 18 is shown in FIG. 1 as communications C1 a and the communication of the various sensors 14 with the host CPU 22 is shown as communications C1 b. The micro-controller 18 processes the information and communicates the information from the sensors 14 to the host CPU 22 positioned within the host system 16 as shown by communication C2. The host CPU 22 processes the sensor information and determines an appropriate action and/or response. The action and/or response is communicated from the host CPU 22 to the micro-controller 18 positioned within the HID 12 by communication C3. The micro-controller 18 receives the communication C3 and determines the appropriate actuator 20 response. The micro-controller 18 then communicates with the appropriate actuator 20 as shown by communication C4. The actuator 20 can be configured to receive the communication C4 and provide a sensory experience in reaction to the communication. For example, the sensory experience can be a feeling of vibration or recoil. The communication C4 instructs the actuator 20 to initiate and perform an action, such as generate vibrations for a specified period of time. This cycle is repeated as additional communications C1 a and C1 b are received from the sensors 14.
As further shown in FIG. 1, additional communication of a general nature, indicated as CG, occurs between the host CPU 22 and the micro-controller 18 positioned within the HID 12. The communication CG can include typical game play instructions.
Referring now to FIG. 2, there is illustrated an improved gaming system, indicated generally at 40, in accordance with a first embodiment of the invention. In the illustrated embodiment as shown in FIG. 2, the gaming system 40 includes at least one HID 42, a plurality of sensors 44, and at least one host device or system 46.
As shown in FIG. 2, the host system 46 can be substantially similar to the host system 16 illustrated in FIG. 1. Similarly, the plurality of sensors 44 shown in FIG. 2 can be substantially similar to the plurality of sensors 14 illustrated in FIG. 1.
The HID 42 shown in FIG. 2 includes, among other things, a micro-controller 50 and at least one actuator 48. The actuator 48 shown in FIG. 2 can be substantially similar to the actuator 20 illustrated in FIG. 1. The micro-controller 50 varies from the micro-controller 18 illustrated in FIG. 1 in that it performs the additional processing of the sensor information C1 a without communicating the sensor information C1 a with the host system 46. Accordingly, the HID 42 may include different components and different electronic circuitry than the HID 12 illustrated in FIG. 1.
Generally, the improved gaming system 40 varies from the traditional gaming system 10 shown in FIG. 1 in several ways. First, the HID 42 does not communicate with the host system 46 as to the processing of the sensor information C1 a. Rather, the HID 42 performs the steps of receiving the sensor information C1 a, internally processing the sensor information C1 a, determining the appropriate actuator response and communicating with the appropriate actuator without communicating the sensor information C1 a to the host system 46. The micro-controller 50 is thus capable of independently determining the response of the actuator 48 that is appropriate or desirable to the condition that is sensed by any of the sensors 44. Second, while the actuator 48 shown in FIG. 2 can be substantially similar to the actuator 20 shown in FIG. 1, the actuator 48 is configured to respond solely to sensor information C1 a processed by the micro-controller 50. Accordingly, the actuator 48 does not require communication from the host system 46 for operation.
Referring now to FIGS. 3-5, the human interface devices 42 can have various physical embodiments. As shown in FIG. 3, a first physical embodiment of one example of a HID 146 is illustrated. In this embodiment, the HID 146 is in the form of a firearm. The firearm HID 146 is configured to interact with the gaming system or virtual reality system such that the firearm HID 146 simulates the firing of a weapon. Accordingly, in an effort to make the gaming system as realistic as possible, the firearm HID 146 has all of the typical firearm components including a stock 160, a muzzle 161, a magazine 162, a trigger 163, a receiver 164, a sight 165 and a butt end 166. In the illustrated embodiment, the firearm HID 146 also includes a recoil mechanism 170, a sensor switch 171, a mode switch 172, a power supply 173, a transmitter/receiver 174, and a motion control 175. While not illustrated in FIG. 3, the firearm HID 146 also includes the micro-controller 50 shown in FIG. 2 and associated circuitry.
As illustrated in FIG. 3 and with reference to FIG. 2, the recoil mechanism 170 is the same device as the actuator 48 shown in FIG. 2. The recoil mechanism 170 is configured to provide a realistic “backward kick” or force resulting from the act of firing the firearm HID 146. In the illustrated embodiment, the recoil mechanism 170 is a solenoid. However, recoil mechanism can be other suitable actuator devices. In operation, as the trigger 163 is pulled, a sensor 144, illustrated as positioned internal to the firearm HID 146, senses the motion of the trigger 163 and sends information to the micro-controller 50 as shown in FIG. 2. The micro-controller receives the sensor information, internally processes the sensor information and communicates with the recoil mechanism 170, thereby producing the recoil motion. Accordingly, the recoil mechanism 170 produces an immediate, real time action in the firearm HID 146 in response to sensor input without the interaction with the host system 46. While the actuator shown in FIG. 3 is a recoil mechanism 170, the actuator can be other suitable devices, and non-limiting examples are provided below.
The sensor switch 171 is configured to enable or disable communication from the sensor 144 to the micro-controller 50. In the illustrated embodiment, the sensor switch 171 is an on/off rocker switch which is conventional in the art. However, the sensor switch 171 can be other suitable devices, such as for example, a slide switch, capable of enabling or disabling the communication from the sensor 144 to the micro-controller 50.
The mode switch 172 is configured to provide the firearm HID 146 with various modes of game play. The mode switch 172 includes settings for single shot, semi-automatic and full automatic firing rate. In operation, as each mode is selected, the sensor 144 and recoil mechanism 170 react accordingly. In other embodiments, the mode switch 172 can be configured to provide various modes of game play in accordance with the nature of the virtual reality scenario. For example, in a scenario in which the virtual reality game involves fishing, the HID 42 can be configured as a fishing pole and the mode switch 172 could be configured to provide the amount of bait used.
Referring again to FIGS. 2 and 3, the power supply 173 is configured to provide sufficient power to the firearm HID 146 such that the firearm HID 146 is independent of the host system 46 for power. As shown in FIG. 3, the power supply 173 is a rechargeable battery pack. However, the power supply 173 can be other suitable devices sufficient to provide power to the firearm HID 146. In other embodiments, the power supply 173 can be connected to a conventional electrical outlet (not shown).
The transmitter/receiver 174 is configured to transmit and receive information to and from the host system 46. The transmitted and received information can include general communications CG, as shown in FIG. 2, as well as other necessary information. In the illustrated embodiment, the transmitter/receiver 174 operates on radio frequencies, although other methods of operation are possible.
The motion control 175 is an input device used to affect the gaming system 46 or govern the movement or actions of an entity within the computer-simulated game. As one example, the motion control 175 can be used to move an entity within the game forward, backward or sideways. In the illustrated embodiment, the motion control 175 is knob having 360° of available motion. However, the motion control 175 can be any mechanism or device suitable to govern the movement or actions of an entity within the computer-simulated game.
While not illustrated in FIG. 3, it should be understood that the firearm HID 146 can project images onto the display of the gaming system. The images can include any suitable visual affect, such as for example a cross-hair and bullet tracers.
As shown in FIG. 4, a second physical embodiment of a HID 246 is illustrated. In this embodiment, the HID 246 is in the form of a joystick. Non-limiting examples include a joystick 246 that can be used in gaming systems involving the control of helicopters or antique aircraft. The HID 246 illustrated in FIG. 4 includes the sensor, micro-controller, and feedback mechanism and these components operate in the same manner as described above.
As shown in FIG. 5, a third physical embodiment of a HID 346 is illustrated. In this embodiment, the HID 346 is in the form of a steering wheel. Non-limiting examples where the steering wheel 346 can be used in gaming systems involving the control of race cars or boats. The HID 346 illustrated in FIG. 5 includes all of the components and operates in the same manner as described above.
While the illustrated embodiments of the human interface devices include a firearm, joystick, and steering wheel, it should be understood the human interface devices can represent any control mechanism, such as, for example, a fishing pole, a tennis racket, or a surf board suitable to affect the gaming system 10 or govern the movement or actions of an entity within the computer-simulated game.
One example of suitable internal components and the circuitry within the HID 42 are shown in FIG. 6. In the illustrated embodiment, the HID 42 includes the micro-controller 50, feedback mechanism (or actuator) 48, a receiver 80, a transmitter 82, voltage conversion circuitry 84, a power supply 86, memory 88, a time device 89, a variable signal generator 90, various user switches, 92 a and 92 b, and sensor output 94.
The HID 42 is activated through a switch, such as for example user switch 92 a. Activating the HID 42 enables the micro-controller 50. The micro-controller 50 awaits a user event, a sensor event or sensor input. All sensor outputs 94 and inputs from user switches 92 a and 92 b interact directly with and only with the micro-controller 50.
After receiving input, the micro-controller 50 communicates with the host system 46. The communications from the micro-controller 50 to the host system 46 are transmitted via industry standard communication protocols and devices, such as for example the receiver 80 and the transmitter 82. The HID 42 does not require custom protocols, additional or special hardware, or custom or special drivers in addition to the standard protocols, hardware and drivers currently residing on the host system 46.
The micro-controller 50 can interpret certain sensor outputs 94 and certain inputs from user switches 92 a and 92 b as input that require user feedback through the feedback mechanism 48. In those situations, the micro-controller 50 enables the power supply 86 to supply power to the feedback mechanism 48. The micro-controller 50 further directs the feedback mechanism 48 to provide a level of feedback as limited according to the user setting 92 a. As described above, the feedback mechanism 48 can apply the feedback to the user in any desired form. In one embodiment, the user input 92 a can be channeled to the micro-controller 50, causing the micro-controller 50 to enable the variable signal generator 90. The signal generator 90 communicates a signal to a power amplifier which amplifies the signal so that the signal may vary the state of the feedback mechanism 48 accordingly. The user's settings 92 a and 92 b have an effect on the type of feedback the user receives. In one embodiment, the user can adjust the variable signal generator 90 through a user switch 92 b to get a different form of signal wave and, accordingly, a different form of feedback. While the illustrated embodiment shown in FIG. 6 provides one example of suitable internal components and the circuitry within the HID 42, it should be appreciated that the HID 42 could include different internal components and different circuitry that operate and function in a different manner.
The improved HID 42 provides many benefits over a traditional HID 12. First, because the sensor information processing is performed within the HID 42 and communication with the host system 46 is not necessary for processing the sensor information, the HID 42 is compatible with any gaming or virtual reality system without software or hardware changes to the host system 46. For example, a common HID can be used with different host systems through adaptors. Second, because the HID 42 is independent of the host system 46, any number of HIDs 42 can be used simultaneously with the host system 46. Third, the HID 42 is customizable to communicate with various types of sensors and provide various types of sensory feedback. Fourth, the HID can have a wide variety of physical embodiments. Other advantages are also apparent from a reading of the specification and claims and from a study of the Figures.
It is also noted that a sensor can communicate directly with the feedback mechanism, bypassing the micro-controller. For example, a photo-sensor could trigger the feedback mechanism when the photo-sensor is exposed to light. Alternatively, a pressure sensor button can cause feedback to occur upon being pushed. In such an arrangement, the sensor and feedback mechanism can be connected by circuitry.
The principle and mode of operation of this invention have been described in its preferred embodiments. However, it should be noted that this invention may be practiced otherwise than as specifically illustrated and described without departing from its scope.

Claims

1. A human interface device for use by a user to interact with a computer-simulated environment comprising:

at least one sensor configured to sense a condition within the computer-simulated environment and generate a first communication concerning the sensed condition;

at least one micro-controller configured to receive the first communication concerning the sensed condition from the at least one sensor, the at least one micro-controller further configured to generate a second communication in reaction to the first communication from the sensor; and

at least one actuator configured to receive the second communication from the at least one micro-controller and provide a sensory experience in reaction to the second communication, wherein the at least one micro-controller is operable to independently determine the response of the at least one actuator appropriate to the condition sensed by the at least one sensor.

2. The human interface device of claim 1 wherein the at least one actuator is configured to respond solely to sensor information processed by the at least one micro-controller.

3. A gaming system comprising:

a host device having a central processing unit configured to control overall functions of the gaming system; and

a human interface device for use by a user of the gaming system and comprising:

at least one sensor configured to sense a condition within the gaming system and generate a first communication concerning the sensed condition;

at least one micro-controller positioned within the human interface device and configured to receive the first communication concerning the sensed condition from the at least one sensor, the at least one micro-controller further configured to independently process the first communication and generate a second communication in reaction to the first communication from the sensor; and

at least one actuator configured to receive the second communication from the at least one micro-controller and provide a sensory experience in reaction to the second communication.

4. The gaming system of claim 3 wherein the at least one controller does not require communication from the host system for operation.

5. The gaming system of claim 3 wherein the at least one controller does not communicate with the host system as to the processing of the first communication.

6. A method of operating a human interface device to remotely interface with a computer, the method comprising the steps of:

sensing a condition within the action of a virtual reality system;

generating a first communication concerning the sensed condition;

sending the first communication to a human interface device for processing;

generating a second communication solely within the human interface device in reaction to the received communication;

sending the second communication to at least one actuator positioned with the human interface device; and

providing a sensory experience with the at least one actuator in reaction to the received human interface device communication.