WO2003103761A1 - Wearable interface control unit - Google Patents

Wearable interface control unit Download PDF

Info

Publication number
WO2003103761A1
WO2003103761A1 PCT/US2002/028880 US0228880W WO03103761A1 WO 2003103761 A1 WO2003103761 A1 WO 2003103761A1 US 0228880 W US0228880 W US 0228880W WO 03103761 A1 WO03103761 A1 WO 03103761A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
control unit
communication
operator
embedded processor
Prior art date
Application number
PCT/US2002/028880
Other languages
French (fr)
Inventor
M. Vice Jack
R. Tracey Michael
E. Lathan Corinna
Original Assignee
Anthrotronix, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anthrotronix, Inc. filed Critical Anthrotronix, Inc.
Priority to AU2002336480A priority Critical patent/AU2002336480A1/en
Publication of WO2003103761A1 publication Critical patent/WO2003103761A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/163—Wearable computers, e.g. on a belt
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
    • A61M2205/00—General characteristics of the apparatus
    • A61M2205/35—Communication
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61M—DEVICES FOR INTRODUCING MEDIA INTO, OR ONTO, THE BODY; DEVICES FOR TRANSDUCING BODY MEDIA OR FOR TAKING MEDIA FROM THE BODY; DEVICES FOR PRODUCING OR ENDING SLEEP OR STUPOR
    • A61M5/00—Devices for bringing media into the body in a subcutaneous, intra-vascular or intramuscular way; Accessories therefor, e.g. filling or cleaning devices, arm-rests
    • A61M5/14—Infusion devices, e.g. infusing by gravity; Blood infusion; Accessories therefor
    • A61M5/142—Pressure infusion, e.g. using pumps
    • A61M5/14244—Pressure infusion, e.g. using pumps adapted to be carried by the patient, e.g. portable on the body
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications

Definitions

  • This invention relates to a computing apparatus for communicating with remote devices. More specifically, the computing device is adapted to be worn on a body part of an operator to enable remote communication between an operator and a remote device.
  • Portable computing apparatus such as laptop computers and personal digital apparatus, are commonly used for remote computing needs and communication with computer systems and networks.
  • a person utilizing such apparatus can enter data into the apparatus as long as the apparatus has an input device and source of power.
  • portable computing apparatus also contain communication electronics, such as a modem, which enable the operator to send and receive data to and from the apparatus and other computer systems or networks.
  • Most modems require the operator to physically connect their apparatus to a telecommunication link.
  • recently modems capable of transmitting and receiving data from remote apparatus through a wireless connection have become common. Accordingly, portable computing apparatus, which enable operators to remotely communicate with other devices and transmit data to and receive data from other devices, is common in the art.
  • laptop computers enable people to do computing from a relatively compact personal computer and transmit data through a connection to a network or other computer system.
  • personal digital assistants with communications hardware enable users to do remote computing on a more limited basis and to transmit files to remote apparatus through a communications connection to a computer network.
  • neither the laptop nor the personal digital assistant is designed to be worn on a body part of the user to enable the user to conduct their physical activity without physically holding the apparatus.
  • laptops, personal digital assistants, and similar computing apparatus are not generally designed to enable wireless communication with another remote apparatus other than computer apparatus or enable bi-directional communication with such apparatus. Accordingly, what is desired is an embedded processor, which can be worn on a body part of the user that enables remote wireless communication with a remote apparatus.
  • This invention comprises an apparatus and system to enable remote communication between an input device in communication with an embedded processor and a remote apparatus.
  • a control apparatus to enable wireless communication with a remote apparatus.
  • the control apparatus includes an embedded processor with a visual display, an input device, a power supply, an analog-digital converter, communication electronics, and a wireless communication apparatus associated with the embedded processor.
  • the input device may be an analog input device, a digital input device, or a combination of such input devices.
  • the analog-digital converter digitizes the data received from the input devices.
  • the communication electronics communicates data between the analog-digital converter and the embedded processor.
  • the remote apparatus includes wireless communication electronics to enable communication with the embedded processor of the control unit. Communication data between the remote apparatus and the embedded processor may include; audio, visual, tactile, and other sensory data.
  • the components of the control apparatus are mounted in a protective case, and the protective case may be mounted in a sleeve that may be adjustably mounted to the operator's body.
  • the sleeve may include sensors secured to different locations to enable remote communication between the operator and the remote apparatus.
  • the control apparatus may include a glove tethered thereto, wherein the glove may include sensors mounted at locations within the glove. For example, sensors may be mounted at the fingertip portions, along the length of the finger, the back of the hand and along the wrist portion. Accordingly, the control unit provides a remote wireless communication tool between the operator and a remote apparatus.
  • a second aspect of the invention is an article comprising a computer-readable signal bearing medium for communicating data between a control unit and a remote apparatus.
  • the article includes means in the medium for communicating video data for display on a control unit, means in the medium for communicating command data to a remote apparatus, and means in the medium for communicating sensory data from said remote apparatus to said control unit.
  • the video data communication means preferably receives and processes video data from said remote apparatus. Processing of the video data preferably includes decompressing received video data for display to an operator.
  • the command data communication means preferably receives and communicates command data from the control unit to the remote apparatus.
  • the sensory data communication means preferably includes transmitting sensory data from the remote apparatus to the control unit.
  • FIG. 1 is a perspective view of the control unit according to the preferred embodiment of this invention, and is suggested for printing on the first page of the issued patent.
  • FIG. 2 is a perspective view of the modular components of the control unit.
  • FIG. 3 is a perspective view of the modular components of the control unit together with a sectional view of the case of the control unit.
  • FIG. 4 is a top view of a sleeve for holding the control unit.
  • FIG. 5 is a perspective view of the sleeve for holding the control unit.
  • FIG. 6 is a top view of the control unit, a vibro-tactile glove, and the instrumented glove.
  • FIG. 7 is a flow chart illustrating flow of video data from a remote apparatus to the control unit.
  • FIG. 8 is a flow chart illustrating flow of command data from the control unit to the remote apparatus.
  • FIG. 9 is a flow chart illustrating flow of telemetry data from the remote apparatus to the control unit.
  • FIG. 10 is an illustration of a graphical user interface.
  • FIG. 11 is a flow chart illustrating flow of sensory data from the remote apparatus to the control unit.
  • An operator control unit having an embedded processor to communicate with an apparatus remote to the control unit is designed in a manner that enables access of the hands of the operator for operation of controls, rather than for holding the apparatus.
  • the operator control unit is designed to be worn on a body part of the operator so that the operator may use his/her hands or other means to operate the unit or other apparatus.
  • the control unit is designed to be used in challenging climate conditions, wherein the environmental elements do not affect operation of the unit.
  • Hardware Fig. 1 is a perspective view of the operator control unit 10.
  • the unit includes a case 20 adapted to house the internal components of the control unit 10.
  • the internal components include a miniaturized personal computer 15 having an embedded processor (not shown) and a visual display 22.
  • the embedded processor computes data received from a remote apparatus or from an operator actuated input device and communicates the processed data to the operator through display on the visual display 22 or an alternative communication tool.
  • the embedded processor of the control unit 10 includes a wireless communication apparatus (not shown) to enable communication between the embedded processor and an apparatus remote from the control unit 10.
  • the wireless communication apparatus may be configured for internet protocol communication so that the unit can utilize this protocol for communication with a remote apparatus.
  • the wireless communication apparatus may be utilized for software communication in the form of serial and other data.
  • the embedded processor and associated visual display 22 are housed within an interior section of the case 20.
  • a top wall 24 of the case includes a window 26 for viewing the visual display 22.
  • the case 20 is designed to hold a plurality of input devices 30, 32, 34, 36 and 38.
  • the input devices 30-38 may be in the form of an analog input apparatus, a digital input apparatus, or a selection of ⁇ such input devices.
  • the input devices 30-38 extend outside the walls of the case 20 and are accessible to the operator.
  • the input devices 30-38 enable an operator of the control unit to communicate commands from the operator to the remote apparatus.
  • the visual display 22 may include sensory controls that enable the operator to use the visual display 22 as an input device to communicate with the remote apparatus through the control unit 10.
  • the operator control unit 10 includes a case 20 to house the embedded processor and associated visual display 22, as well as input devices 30-38 for communicating with the embedded processor.
  • the input devices 30-38 are mounted in one section 40 of the case 20.
  • Each of the input devices 30-38 are mapped to a set of corresponding logical states in the remote apparatus or in the control unit.
  • a logical state may correspond to action of one or more motors on the remote apparatus.
  • One of the input devices may be a proportional input device, such as a joystick 30, as shown in Fig. 1.
  • the joystick 30 is enclosed within a neoprene boot 50 to protect the components of the joystick 30 from dust and moisture.
  • the boot 50 also functions as a seal between the input device 30 and the case 20.
  • a distal end 52 of the joystick 30 extends from a surface of the case 20 and may be actuated by the operator.
  • a proximal end 54 of the joystick 30 is connected to a circuit board 56 that remains within an interior section of the case 20.
  • the circuit board 56 is an analog circuit board that provides a 0 to 5 volt signal per axis of rotation. As the joystick 30 is actuated in the X-Y plane, a signal is produced that corresponds to the degree of actuation.
  • the signal is in the form of a voltage output that preferably ranges from 0 to 5 volts, but may be calibrated for a lesser or greater output. Variations in movement of the joystick 30 in the X-Y plane provide different output voltages to the associated circuit board 56. Actuation of the joystick 30 may be communicated to a respective logical state or motor of the remote apparatus, thus controlling direction, velocity, illumination, or any apparatus that is adapted to receive a variable input.
  • the signal from the circuit board 56 associated with the proportional input device 30 is processed by an analog to digital converter to digitize the data into a computer readable format, followed by streaming of the processed data to a communication port of the embedded processor. Accordingly, the joystick is an input device that provides a proportional signal to communicate with a transmission apparatus.
  • the operator control unit 10 may include additional input devices in the form of switches 32, 34, 36 and 38.
  • the switches may be in the form of analog input devices or binary input devices.
  • the switches 32-38 are shown in Fig. 1 as being mounted through an exterior wall of the case 20 of the control unit 10. Prior to operation of the control unit, each of the input devices 30-38 are mapped and calibrated to a logical state on the control unit and/or logical states on a remote apparatus.
  • Each switch 32-38 includes a distal end 32a-38a, respectively, and a proximal end 32b-38b, respectively, extending from an interior section of the case 20 to an area exterior to the case 20.
  • the proximal ends 32b-38b of the switches 32-38 are mounted on a circuit board 60 that is stored within an interior section of the case 20.
  • Switches 36 and 38 are shown as toggle switches, and switches 32 and 34 are shown as push button switches.
  • input from the switches 32-38 are communicated to the associated circuit board 60 which provides data received from the input device to the analog-digital converter (not shown) to digitize the data into a computer readable format. Accordingly, the proportional input device 30 and the switches 32-38 enable communication between the operator and the remote apparatus through the embedded processor.
  • the input devices 30-38 shown in Fig. 1 are merely exemplary illustrations of specific input devices that may be employed within the case 20. Any variety of switches or alternative input devices that are capable of communicating data signals to an analog-digital converter may be substituted in their place.
  • the distal ends 32a-38a of the switches 32-38 may be actuated by the operator.
  • the toggle switches 36 and 38 may be multidirectional input devices. By moving the toggle switch 36 or 38 a signal is communicated to the associated circuit board 60. Similarly, by pushing the button of the switch 32 or 34, a signal is communicated to the associated circuit board 60.
  • Actuation of the switches 32, 34, 36 and/or 38 conveys communication from the operator to the circuit board 60 associated with the actuated input device which forwards the data to an analog-digital converter.
  • Wired communication electronics are integrated into the analog-digital converter to digitize the data into a computer readable format and to communicate data received from the input devices 30- 38 to the embedded processor of the miniaturized personal computer 15.
  • Actuation of any of the switches 32, 34, 36 and/or 38 may communicate data to the embedded processor in the form of switching modes of operation, switching proximity sensors, controlling software, and navigation within the software and/or operating system. Accordingly, actuation of the input devices enables an operator of the unit to communicate a variety of signals to the embedded processor and should not be limited by the above-noted modes.
  • FIG. 2 is a drawing showing how some of these components may be modular and detached from the case 20.
  • the input devices 30- 38 are mounted on a secondary controller 90.
  • the secondary controller 90 is connected to the case 20 of the control unit 10 by means of a tether 92.
  • the circuit boards 56 and 60 (not shown) that receive the input from the joystick 30 and input devices 32-38, respectively, may be housed in the secondary controller 90.
  • Data received from the input devices 30-38 is forwarded to the analog-digital converter (not shown) to digitize the data into a computer readable format.
  • the analog-digital converter may be housed within the control unit 20, or it may be housed in a separate unit tethered to the control unit.
  • the power supply (not shown) for the control unit 10 may be housed within the case 20, or it may be housed in a separate unit tethered to the control unit 10.
  • the secondary controller 90 may be held in the hand of the operator, or it may include a fastener that enables the secondary controller 90 to be mounted to an article or a secondary location.
  • a variety of fastening means may be employed, including hook and loop, clips, an adhesive, and combinations of fastening means.
  • the input devices 30-38 of the control unit 10 may be mounted to a secondary controller 90 to enable the input devices to be placed in comfortable proximity to the operator of the unit.
  • Fig. 3 is an illustration of the control unit 10 showing the miniaturized personal computer 15 housed within the case 20.
  • the tether 92 which extends from the secondary controller to the case 20 is shown connected to the miniaturized personal computer 15.
  • this view shows a wireless communication apparatus 96 connected to the miniaturized personal computer 15 and housed within the case 20.
  • the control unit 10 may be designed to house a wireless communication apparatus 96 as well as receive a tether 92 to enable communication from a remote input device to the miniaturized personal computer 15.
  • the operator control unit 10 is designed to withstand harsh environmental conditions and to remain in proper operating condition when exposed to natural elements such as dust and moisture.
  • the case 20 is comprised of several materials, including Delron, Aluminum and Lexan. Delron is a material that mitigates radio-frequency interference with the associated wireless communication electronics, and enhances shock resistance of the embedded processor and associated visual display 22.
  • the Lexan material is used for the transparent display cover for the visual display 22. Proper operation of the control unit 10 requires the operator to be able to properly view the visual display 22 without interference. As such, the visual display 22 should include a transparent covering while providing resistance to external elements.
  • Aluminum is also used in the composition of the case 20 to hold the Lexan display cover in a proper position.
  • the operator control unit may be made from other materials including Aluminum, Aluminum Alloy, polymers, and other impact resistant materials. Accordingly, the materials that comprise the composition of the case contribute to the functionality of the unit.
  • the input devices are designed to withstand harsh environmental conditions and to prevent dust, dirt and moisture from affecting communication with the embedded processor.
  • the toggle switches 36 and 38 are separated by a silicone rubber membrane 76 and 78, respectively, to prevent moisture and dust from entering the case through any apertures associated with the switches 36 and 38.
  • the membrane 76, 78 may be comprised of an alternative material that provides protection of the interior section of the case 20 and associated circuit board 60 from damage due to dust, moisture and environmental weather conditions.
  • the push button switches 32 and 34 are shown with a grommet 72 and 74, respectively, sealing the opening between the button and the case.
  • the grommet 72, 74 protects the associated circuit board 60 within the case from damage from environmental elements such as dust and moisture.
  • the grommet 72, 74 is made of a polymer material that can withstand dust and moisture.
  • the grommet 72, 74 may be comprised of an alternative material that provides protection of the button 32, 34 and associated circuit board 60 from damage due to dust, moisture and environmental weather conditions.
  • the operator control unit 10 is a portable unit that may be held in the hands of an operator or may be secured to a limb or a garment of the operator. In order to enable an operator to perform multiple tasks simultaneously, the control unit 10 may be placed in a sleeve 200 as shown in Fig. 4.
  • the sleeve 200 holds the hardware components of the control unit 10, while enabling the unit to be attached to a body part of the operator.
  • Straps 210, 220 and 230 attach the sleeve to an object or limb of the operator.
  • Each strap 210, 220 and 230 includes a proximal end 212, 222 and 232, respectively, and a distal end 214, 224 and 234, respectively.
  • each of the straps is remote from the sleeve.
  • Attachment components may be in the form of hook and loop fasteners mounted on the proximal ends 212, 222 and 232 and distal ends 214, 224 and 234 of each strap 210, 220 and 230.
  • This enables the sleeve to be adjustable for fitting on to different size body parts, as well different size bodies of operators.
  • the sleeve 200 illustrated in Fig. 4 is one formative of the design and configuration of the sleeve. For example, if the sleeve 200 is mounted to a forearm of the operator, each of the straps 210, 220 and 230 are wrapped around the circumference of the forearm.
  • each strap has hook receiving means and the distal ends 214, 224 and 234 of each strap have hook means.
  • This enables the distal end of each strap to secure to the proximal end of each strap.
  • mechanical fastening means such as hook and loop fasteners
  • the sleeve 200 is a component that holds the control unit 10 and enables an operator to attach the control unit to a part of his/her body to enable hands-free operation of the unit.
  • the sleeve shown in Fig. 4 includes a first pocket 240 adapted to receive the control unit 20.
  • the sleeve 200 may include a second pocket 250 adapted to receive a power supply.
  • the power supply stored in the second pocket 250 is a replaceable power source 252 .
  • the power source may be in the form of a battery pack.
  • the second pocket 250 may be attached to the sleeve 200 by a hook and loop fastener associated with strap 220. At such time as the operator needs to access the power source 252 to provide a replacement to the power of the control unit 10, the operator may open the second pocket 250, remove the power source 252, and provide a replacement power source 252 in the pocket 250.
  • the sleeve 200 may also include a third pocket 260 to hold an analog to digital converter 262.
  • Fig. 5 is a perspective view of the sleeve 200 illustrating the side of the sleeve 200 with a second pocket 250 adapted to receive the power source 252 and a third pocket 260 adapted to receive the analog-digital converter 262.
  • the sleeve 200 includes an aperture 270 that fits over the visual display 22, an aperture 280 to enable access to the joystick 30, and a set of apertures 292, 294, 296 and 298 to enable access to the switches 32, 34, 36 and 38.
  • the power source 252 and analog- digital converter 262 may be mounted within an interior section of the case 20 of the control unit 10, or at an alternative location in proximity to the embedded processor.
  • the sleeve 200 may include several pockets, an aperture 270 and apertures 292-298 to enable viewing data conveyed to the operator through the visual display and to enable access to the input devices of the control unit 10.
  • a vibro-tactile feedback apparatus may be used in communication with the control unit 10 as an additional or alternative feedback device.
  • One formative of the vibro-tactile feedback device 400 is shown in Fig. 6 in a form of a glove 410 to enable the operator to use their hand and fingers to communicate with the remote apparatus and to manipulate the controls of the unit 10.
  • the glove 410 includes five independent sleeves 412, 414, 416, 418 and 420 for five fingers of the hand of the operator.
  • Each of the sleeves 412-420 has a sensor 422, 424, 426, 428 and 430, respectively, that is sewn into the respective sleeve.
  • a fingertip portion of each sleeve may include an additional sensor 432, 434, 436, 438 and 440.
  • the glove 410 may also include an accelerometer 450 on the back of the hand portion of the glove.
  • the accelerometer 450 may be a dual-axis accelerometer or a tri-axis accelerometer.
  • a dual-axis accelerometer transmits the orientation of the hand in a two coordinate plane and communicates the orientation and action of the hand to the remote apparatus.
  • a tri-axis accelerometer records and communicates the orientation of the hand in a three coordinate plane.
  • the sleeve sensors 422-430 are bend sensors which are activated upon bending a finger, and the fingertip sensors 432-440 are pressure sensors which are activated when pressure is applied to the sensor.
  • Each of the sensors 422-430 and 432-440 may be analog or binary sensors.
  • the sensors are wired to the analog-digital converter of the control unit 10.
  • One or more of the fingertip sensors 432-440 are activated by exerting sufficient pressure to exceed a set threshold; the resulting sensor data is sent to a circuit board in the control unit 10 in communication with the activated sensor. Sensor data from the circuit board is forwarded to the analog-digital converter to place the data into a computer readable format and forward the data through a communication stream to the embedded processor.
  • the data received by the embedded processor is forwarded to a wireless communication apparatus associated with the embedded processor, which sends the data to the remote apparatus.
  • Various logical states of the remote apparatus are mapped to the actuated sensor.
  • bending of a finger in communication with one or more bend sensors 422-430 in a sleeve communicates data to the remote apparatus for control of various logical state on the remote apparatus.
  • the sensors located on the finger portions and hand portion of the glove may be used to communicate with the remote apparatus by controlling specific logical states of the remote apparatus which are mapped to the sensor of the instrumented glove.
  • the control unit 10 may include a sleeve 460 with input devices such as buttons, switches, and other sensors to communicate with the remote apparatus.
  • One form of the sleeve 460 is shown in Fig. 6 with vibro-tactile motors 462-474 embedded therein.
  • the sleeve 460 is shown tethered to the control unit 10.
  • the remote apparatus communicates with the operator, it sends data through wireless communication electronics to a-corresponding wireless communication electronics associated with the embedded processor of the control unit 10. The data are received and processed by the embedded processor and forwarded to one or more of the vibro-tactile motors 462-474 which are mapped to the logical state of the remote apparatus.
  • the communication can be in the form of feedback associated with a command sent from the operator or movement of the remote apparatus.
  • the remote apparatus may include proximity and sonar sensors to detect physical objects around the remote apparatus.
  • the remote apparatus can communicate contact between a particular sensor and an obstacle. For example, if a specific proximity sensor is actuated, the remote apparatus will send feedback to a motor on the sleeve mapped to the sensor.
  • the motor may vibrate or otherwise be actuated to communicate to the operator actuation of the associated sensor on the remote apparatus.
  • the vibro-tactile motors 462-474 may be connected as an array of motors, such that specific pre-programmed actions are transmitted from the remote apparatus to the array which causes specific vibration patterns to be produced in the array.
  • the operator can actuate one or more sensors 476-486 on the sleeve 460 to control a logical state on the remote apparatus. These sensors could include electromagnetic, force, strain, and acceleration activated sensors, as well as other types of sensors. Activation of one or more of the sensors 476-486 sends a data signal through the control unit 10 to the remote apparatus by means of the same process that signals from the instrumented glove are connected. Accordingly, the sensors and vibro-tactile motors of the sleeve enable the operator to transmit and receive data to and from a remote apparatus while providing full use and mobility of their hands.
  • the vibro-tactile sleeve 460 of Fig. 6 may be independent from, and tethered to the control unit 10, or the sleeve 460 may be tethered to the control unit 400.
  • Fig. 6 illustrates one embodiment where the control unit 10 is mounted in one location and the sleeve 460 is tethered to the control unit 10.
  • the vibro-tactile sleeve 460 may be designed with the control unit 10 embedded therein. This would provide the operator an alternative tool to receive data from a remote apparatus.
  • the sleeve and glove apparatus are another form of remote communication between the remote apparatus and the operator.
  • both the sleeve 200, 460 are preferably comprised of a durable, lightweight and flexible material.
  • the control unit is adapted to function in harsh environments, and the sleeves 200, 460 must be able to withstand harsh environmental conditions while protecting the components of the control unit 10.
  • the sleeves 200, 460 must be adaptable to be fitted onto different body parts of different operators, and must also be lightweight, as it is preferable to mitigate the weight of the components of the control unit 20.
  • the sleeves are made of a Courdura fabric.
  • alternative materials that meet the requirements of durability, flexibility and weight may be used for the sleeves. Accordingly, the material of the sleeves 200, 460 is an important factor in preserving the operation and handling of the control unit 10.
  • the hardware components of the control unit 10, 405 may be used to visually convey data from a remote apparatus to an operator of the control unit. Visual data are displayed to the operator on the visual display 22.
  • the embedded processor receives video data and displays the video data on the visual display 22.
  • Fig. 7 is a flow chart 500 illustrating the steps involved in receiving a video stream and for displaying the video stream on the visual display in a manner that is understandable by the operator.
  • the video stream that is received by the embedded processor is in a compressed format.
  • the embedded processor receives compressed video data 502 from a remote apparatus through the wireless communication apparatus.
  • the video data are then decompressed into pixel data to a known memory location 504.
  • the decompressed pixel data are copied from the known memory location to the visual display for display in an interpretable format to the operator of the unit 506. Accordingly, video data for communication to the operator are processed from a compressed format to a decompressed format and displayed to the operator of the unit.
  • control unit 10, 405 is also adapted to convey commands from the operator of the unit to a remote apparatus.
  • actuation of the joystick 30 and/or switches 32-38 is a form of communication.
  • actuation of a sensor on the sleeve or a sensor in the glove generates command data.
  • the data associated with actuation of a sensor or an alternative input device are processed and forwarded to a remote apparatus for actuation of the apparatus.
  • Fig. 8 is a flow chart 520 illustrating the steps involved in processing command data and communicating with a remote apparatus. Actuation data received from an input device or sensor are conveyed to a circuit board connected to the input device 524.
  • the actuation data may be in analog or digital format.
  • the actuation data are sent from the circuit board to an analog- digital converter 526 for conversion to a computer readable format, i.e. digitized.
  • the digitized data are placed into a digital communication stream 528 and then sent to a communication port of the embedded processor 530.
  • the digital communication stream may be in RS232, USB or an alternative format depending upon the format utilized by the analog- digital converter.
  • the digitized data are parsed 532 to determine the coordinate positions of the proportional data as well as the values associated with the data inputted through the input devices and/or sensors.
  • the parsed data are computed 534 to determine the commands of the operator, and the computed data are forwarded to the remote apparatus 536 through wireless communication electronics. Accordingly, the data from the input devices are processed by the hardware of the control unit and forwarded from the control unit to the remote apparatus with which the unit is communicating.
  • the control unit includes a process for displaying telemetry data from a remote apparatus to an operator of the unit through the visual display. Telemetry data may include roll, pitch and yaw data for the position of a remote apparatus, global positioning satellite data, compass, radar, sonar, and other sensory data that convey any form of telemetry to communicate to an operator remote from the apparatus. Fig.
  • FIG. 9 is a flow chart 550 -illustrating the steps involved in processing data for display to an operator through the visual display.
  • Data are sent from the remote apparatus to the wireless communication apparatus in the control unit 554.
  • the data are sent to the communication apparatus of the control unit in digital format.
  • the data received by the communication apparatus are forwarded to the embedded processor of the control unit 556.
  • the data are processed for display on the visual display 558.
  • graphical data are computed based upon telemetry data of the remote apparatus.
  • Graphical data are computed in the form of pixel data. Appropriate pixel data such as roll, pitch and yaw are computed and stored in a known memory location 560.
  • the raw pixel data values are copied from the known memory location to the visual display to convey the data to the control unit 562 in a visual format. Accordingly, the process of receiving and conveying telemetry data includes computing and processing position data of a remote apparatus and displaying said data on an operator of the control unit.
  • Telemetry data is not limited to visual display. Rather, the telemetry data may also be in the form of a vibro-tactile display as shown in Fig. 6.
  • Fig. 1 1 is a flow chart 700 illustrating the steps involved in processing data for display to an operator through the vibro- tactile display.
  • Data are sent from the remote apparatus to the communication apparatus in the control unit 704 via wireless communication electronics.
  • the data are sent to the communication apparatus of the control unit in digital format. Thereafter, the data received by the communication apparatus are forwarded to the embedded processor of the control unit 706.
  • the data are processed and motor values are recomputed based upon telemetry data 708.
  • the new motor values are sent to the vibro- tactile display 710 to convey telemetry data to the control unit in a sensory format.
  • the process of receiving and conveying telemetry data includes computing and processing position data of a remote apparatus and displaying the data to an operator of the control unit through the vibro-tactile display embedded in the sleeve of the control unit.
  • Fig. 10 is one form of a graphical user interface 600 for conveying the received data to the operator in an intuitive manner.
  • the wire frame 610 is a three dimensional visual representation of the remote apparatus.
  • the position of the wire frame is indicative of the position of the remote apparatus. For example, as the data values of the roll, pitch and yaw values of the remote apparatus change, the position of the wire frame will rotate to reflect the roll, pitch and yaw data values.
  • the graphical user interface 600 may include a compass 620 to communicate the geographic orientation of the remote apparatus relative to the control unit.
  • a significant portion of the graphical user interface is programmed to include a display for video data 630.
  • the graphical user interface 600 may include bar graphs 640, 650, 660 and 670 to convey additional telemetry data to the operator. As shown in Fig. 10, there are four bar graphs 640, 650, 660 and 670. The bar graphs may be used to display voltage, velocity, temperature, and communication data from the remote apparatus to the operator of the control unit. Each of these measurements conveys important factors to the operator for maintaining proper operation of the remote apparatus.
  • the graphical user interface 600 may be programmed to receive and display additional data.
  • the remote apparatus and control unit may be programmed to communicate with a satellite to convey position data at 680. Accordingly, the graphical user interface 600 is an intuitive display for conveying video and telemetry data received by the embedded processor to the operator of the control unit.
  • Prior art teleoperation of remote apparatus require hardware and software which allow high-bandwidth information exchange, including video, audio and other forms of sensory data. Since teleoperation equipment is commonly used in a field environment, it is important that the components of the equipment be designed to withstand operation while exposed to dirt, dust, moisture and high impact conditions.
  • the hardware components of the control unit including peripheral devices, are fully ruggedized and are capable of withstanding extreme conditions, including dust, moisture and dirt.
  • the input device be modular to enable the operator to place the input devices in a comfortable operating location. Similarly, in certain operating conditions it is necessary for the operator to have both hands free to perform other tasks.
  • Wearable computing devices such as a sleeve and/or glove enables the teleoperation equipment to be integrated into the operator's clothing or to be mounted on the body without impacting upon the vision of the operator or protruding from the body of the operator.
  • the sleeve is adjustable to fit operators of different sizes. It allows the operator's hands to be free during operation and to enable the operator to perform various tasks while using the control unit.
  • control unit may be designed to communicate with a variety of remote apparatus.
  • the remote apparatus may be in an electronic or mechanical form with logical states mapped to corresponding input devices and motors of the control unit.
  • the control unit may also be used to download topographical data or other geographical data or to provide live-video feedback from visual sensors on a remote apparatus.
  • the visual display may be in the form of a liquid crystal display, or an alternative display medium that enables viewing by the operator while maintaining the integrity of the control unit.
  • the wireless communication electronics may be in the form of wireless communication electronics connected to the embedded processor, or an alternative communication electronics that enables wireless communication of data between the embedded processor and a corresponding wireless communication apparatus on a remote apparatus.
  • the scope of the invention should not be limited to the input devices described herein. Alternative input devices that enable communication of data between the control unit and the remote apparatus may be employed. Accordingly, the scope of protection of this invention is limited only by the following claims and their equivalents.

Abstract

A control unit (10) in wireless communication with a remote apparatus.The control unit (10) is adapted to be placed on a limb of the operator to enable the operator to use one or both hands to operate the unit. The control unit (10) includes input devices (30-38) mapped to logical states of the remote apparatus. Actuation of the input devices (30-38) is communicated to a logical states on the remote apparatus actuated by the input device. In addition, the control unit (10) includes communication means for receiving input from sensors attached to the remote apparatus. Input received by the control unit (10) may be displayed to the operator on a visual (22) or tactile display (400) in communication with the control unit. The input devices (30-38) from the control (10) and the remote apparatus communicate data in the form of audio, visual, tactile and other forms of sensory data.

Description

Wearable Interface Control Unit
GOVERNMENT INTEREST
The U.S. Government has a paid-up license in this invention and the right in limited circumstances to require the patent owner to license others on reasonable terms as provided for by the term of contract number F04701-01-C-0018 awarded by the Department of Defense.
CROSS REFERENCE TO RELATED APPLICATION(S)
This a Continuation-in-Part utility patent application of Application Serial No. 10/164,736, filed June 7, 2000, entitled "Wearable Interface Control Unit", now pending.
BACKGROUND OF THE INVENTION
Technical Field
This invention relates to a computing apparatus for communicating with remote devices. More specifically, the computing device is adapted to be worn on a body part of an operator to enable remote communication between an operator and a remote device.
Description Of The Prior Art
Portable computing apparatus, such as laptop computers and personal digital apparatus, are commonly used for remote computing needs and communication with computer systems and networks. A person utilizing such apparatus can enter data into the apparatus as long as the apparatus has an input device and source of power.
Many portable computing apparatus also contain communication electronics, such as a modem, which enable the operator to send and receive data to and from the apparatus and other computer systems or networks. Most modems require the operator to physically connect their apparatus to a telecommunication link. However, recently modems capable of transmitting and receiving data from remote apparatus through a wireless connection have become common. Accordingly, portable computing apparatus, which enable operators to remotely communicate with other devices and transmit data to and receive data from other devices, is common in the art.
The are several apparatus that enable remote communication. For example, laptop computers enable people to do computing from a relatively compact personal computer and transmit data through a connection to a network or other computer system. Similarly, personal digital assistants with communications hardware enable users to do remote computing on a more limited basis and to transmit files to remote apparatus through a communications connection to a computer network. However, neither the laptop nor the personal digital assistant is designed to be worn on a body part of the user to enable the user to conduct their physical activity without physically holding the apparatus. In addition, laptops, personal digital assistants, and similar computing apparatus are not generally designed to enable wireless communication with another remote apparatus other than computer apparatus or enable bi-directional communication with such apparatus. Accordingly, what is desired is an embedded processor, which can be worn on a body part of the user that enables remote wireless communication with a remote apparatus.
SUMMARY OF THE INVENTION
This invention comprises an apparatus and system to enable remote communication between an input device in communication with an embedded processor and a remote apparatus.
In a first aspect of the invention, a control apparatus is provided to enable wireless communication with a remote apparatus. The control apparatus includes an embedded processor with a visual display, an input device, a power supply, an analog-digital converter, communication electronics, and a wireless communication apparatus associated with the embedded processor. The input device may be an analog input device, a digital input device, or a combination of such input devices. The analog-digital converter digitizes the data received from the input devices. The communication electronics communicates data between the analog-digital converter and the embedded processor. The remote apparatus includes wireless communication electronics to enable communication with the embedded processor of the control unit. Communication data between the remote apparatus and the embedded processor may include; audio, visual, tactile, and other sensory data. The components of the control apparatus are mounted in a protective case, and the protective case may be mounted in a sleeve that may be adjustably mounted to the operator's body. The sleeve may include sensors secured to different locations to enable remote communication between the operator and the remote apparatus. Similarly, the control apparatus may include a glove tethered thereto, wherein the glove may include sensors mounted at locations within the glove. For example, sensors may be mounted at the fingertip portions, along the length of the finger, the back of the hand and along the wrist portion. Accordingly, the control unit provides a remote wireless communication tool between the operator and a remote apparatus.
A second aspect of the invention is an article comprising a computer-readable signal bearing medium for communicating data between a control unit and a remote apparatus. The article includes means in the medium for communicating video data for display on a control unit, means in the medium for communicating command data to a remote apparatus, and means in the medium for communicating sensory data from said remote apparatus to said control unit. The video data communication means preferably receives and processes video data from said remote apparatus. Processing of the video data preferably includes decompressing received video data for display to an operator. The command data communication means preferably receives and communicates command data from the control unit to the remote apparatus. The sensory data communication means preferably includes transmitting sensory data from the remote apparatus to the control unit.
Other features and advantages of this invention will become apparent from the following detailed description of the presently preferred embodiment of the invention, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a perspective view of the control unit according to the preferred embodiment of this invention, and is suggested for printing on the first page of the issued patent.
FIG. 2 is a perspective view of the modular components of the control unit.
FIG. 3 is a perspective view of the modular components of the control unit together with a sectional view of the case of the control unit.
FIG. 4 is a top view of a sleeve for holding the control unit.
FIG. 5 is a perspective view of the sleeve for holding the control unit.
FIG. 6 is a top view of the control unit, a vibro-tactile glove, and the instrumented glove.
FIG. 7 is a flow chart illustrating flow of video data from a remote apparatus to the control unit.
FIG. 8 is a flow chart illustrating flow of command data from the control unit to the remote apparatus.
FIG. 9 is a flow chart illustrating flow of telemetry data from the remote apparatus to the control unit.
FIG. 10 is an illustration of a graphical user interface.
FIG. 11 is a flow chart illustrating flow of sensory data from the remote apparatus to the control unit.
DESCRIPTION OF THE PREFERRED EMBODIMENT
Overview
An operator control unit having an embedded processor to communicate with an apparatus remote to the control unit is designed in a manner that enables access of the hands of the operator for operation of controls, rather than for holding the apparatus. The operator control unit is designed to be worn on a body part of the operator so that the operator may use his/her hands or other means to operate the unit or other apparatus. In addition, the control unit is designed to be used in challenging climate conditions, wherein the environmental elements do not affect operation of the unit.
Technical Background
Hardware Fig. 1 is a perspective view of the operator control unit 10. The unit includes a case 20 adapted to house the internal components of the control unit 10. The internal components include a miniaturized personal computer 15 having an embedded processor (not shown) and a visual display 22. The embedded processor computes data received from a remote apparatus or from an operator actuated input device and communicates the processed data to the operator through display on the visual display 22 or an alternative communication tool. The embedded processor of the control unit 10 includes a wireless communication apparatus (not shown) to enable communication between the embedded processor and an apparatus remote from the control unit 10. The wireless communication apparatus may be configured for internet protocol communication so that the unit can utilize this protocol for communication with a remote apparatus. In addition, the wireless communication apparatus may be utilized for software communication in the form of serial and other data. The embedded processor and associated visual display 22 are housed within an interior section of the case 20. A top wall 24 of the case includes a window 26 for viewing the visual display 22. In addition to the embedded processor and associated visual display 22, the case 20 is designed to hold a plurality of input devices 30, 32, 34, 36 and 38. The input devices 30-38 may be in the form of an analog input apparatus, a digital input apparatus, or a selection of ■ such input devices. The input devices 30-38 extend outside the walls of the case 20 and are accessible to the operator. The input devices 30-38 enable an operator of the control unit to communicate commands from the operator to the remote apparatus. In addition, the visual display 22 may include sensory controls that enable the operator to use the visual display 22 as an input device to communicate with the remote apparatus through the control unit 10. Accordingly, the operator control unit 10 includes a case 20 to house the embedded processor and associated visual display 22, as well as input devices 30-38 for communicating with the embedded processor. The input devices 30-38 are mounted in one section 40 of the case 20. Each of the input devices 30-38 are mapped to a set of corresponding logical states in the remote apparatus or in the control unit. A logical state may correspond to action of one or more motors on the remote apparatus. One of the input devices may be a proportional input device, such as a joystick 30, as shown in Fig. 1. The joystick 30 is enclosed within a neoprene boot 50 to protect the components of the joystick 30 from dust and moisture. Other materials may be used to insulate the joystick 30 from dust, moisture, electro-magnetic interference, and any other conditions that would affect the communication and operation of the joystick 30. The boot 50 also functions as a seal between the input device 30 and the case 20. A distal end 52 of the joystick 30 extends from a surface of the case 20 and may be actuated by the operator. Similarly, a proximal end 54 of the joystick 30 is connected to a circuit board 56 that remains within an interior section of the case 20. The circuit board 56 is an analog circuit board that provides a 0 to 5 volt signal per axis of rotation. As the joystick 30 is actuated in the X-Y plane, a signal is produced that corresponds to the degree of actuation. The signal is in the form of a voltage output that preferably ranges from 0 to 5 volts, but may be calibrated for a lesser or greater output. Variations in movement of the joystick 30 in the X-Y plane provide different output voltages to the associated circuit board 56. Actuation of the joystick 30 may be communicated to a respective logical state or motor of the remote apparatus, thus controlling direction, velocity, illumination, or any apparatus that is adapted to receive a variable input. The signal from the circuit board 56 associated with the proportional input device 30 is processed by an analog to digital converter to digitize the data into a computer readable format, followed by streaming of the processed data to a communication port of the embedded processor. Accordingly, the joystick is an input device that provides a proportional signal to communicate with a transmission apparatus.
In addition to the proportional input device 30, the operator control unit 10 may include additional input devices in the form of switches 32, 34, 36 and 38. The switches may be in the form of analog input devices or binary input devices. The switches 32-38 are shown in Fig. 1 as being mounted through an exterior wall of the case 20 of the control unit 10. Prior to operation of the control unit, each of the input devices 30-38 are mapped and calibrated to a logical state on the control unit and/or logical states on a remote apparatus. Each switch 32-38 includes a distal end 32a-38a, respectively, and a proximal end 32b-38b, respectively, extending from an interior section of the case 20 to an area exterior to the case 20. The proximal ends 32b-38b of the switches 32-38, respectively, are mounted on a circuit board 60 that is stored within an interior section of the case 20. Switches 36 and 38 are shown as toggle switches, and switches 32 and 34 are shown as push button switches. Similar to the proportional input device 30, input from the switches 32-38 are communicated to the associated circuit board 60 which provides data received from the input device to the analog-digital converter (not shown) to digitize the data into a computer readable format. Accordingly, the proportional input device 30 and the switches 32-38 enable communication between the operator and the remote apparatus through the embedded processor.
The input devices 30-38 shown in Fig. 1 are merely exemplary illustrations of specific input devices that may be employed within the case 20. Any variety of switches or alternative input devices that are capable of communicating data signals to an analog-digital converter may be substituted in their place. The distal ends 32a-38a of the switches 32-38 may be actuated by the operator. For example, the toggle switches 36 and 38 may be multidirectional input devices. By moving the toggle switch 36 or 38 a signal is communicated to the associated circuit board 60. Similarly, by pushing the button of the switch 32 or 34, a signal is communicated to the associated circuit board 60. Actuation of the switches 32, 34, 36 and/or 38 conveys communication from the operator to the circuit board 60 associated with the actuated input device which forwards the data to an analog-digital converter. Wired communication electronics are integrated into the analog-digital converter to digitize the data into a computer readable format and to communicate data received from the input devices 30- 38 to the embedded processor of the miniaturized personal computer 15. Actuation of any of the switches 32, 34, 36 and/or 38 -may communicate data to the embedded processor in the form of switching modes of operation, switching proximity sensors, controlling software, and navigation within the software and/or operating system. Accordingly, actuation of the input devices enables an operator of the unit to communicate a variety of signals to the embedded processor and should not be limited by the above-noted modes.
Each of the input devices 30-38 of the control unit 10 are detachable from the case 20 for independent use. Fig. 2 is a drawing showing how some of these components may be modular and detached from the case 20. In the modular format shown, the input devices 30- 38 are mounted on a secondary controller 90. The secondary controller 90 is connected to the case 20 of the control unit 10 by means of a tether 92. The circuit boards 56 and 60 (not shown) that receive the input from the joystick 30 and input devices 32-38, respectively, may be housed in the secondary controller 90. Data received from the input devices 30-38 is forwarded to the analog-digital converter (not shown) to digitize the data into a computer readable format. The analog-digital converter may be housed within the control unit 20, or it may be housed in a separate unit tethered to the control unit. Similarly, the power supply (not shown) for the control unit 10 may be housed within the case 20, or it may be housed in a separate unit tethered to the control unit 10. During operation, the secondary controller 90 may be held in the hand of the operator, or it may include a fastener that enables the secondary controller 90 to be mounted to an article or a secondary location. A variety of fastening means may be employed, including hook and loop, clips, an adhesive, and combinations of fastening means. Accordingly, the input devices 30-38 of the control unit 10 may be mounted to a secondary controller 90 to enable the input devices to be placed in comfortable proximity to the operator of the unit.
Fig. 3 is an illustration of the control unit 10 showing the miniaturized personal computer 15 housed within the case 20. The tether 92 which extends from the secondary controller to the case 20 is shown connected to the miniaturized personal computer 15. In addition, this view shows a wireless communication apparatus 96 connected to the miniaturized personal computer 15 and housed within the case 20. Accordingly, the control unit 10 may be designed to house a wireless communication apparatus 96 as well as receive a tether 92 to enable communication from a remote input device to the miniaturized personal computer 15.
The operator control unit 10 is designed to withstand harsh environmental conditions and to remain in proper operating condition when exposed to natural elements such as dust and moisture. The case 20 is comprised of several materials, including Delron, Aluminum and Lexan. Delron is a material that mitigates radio-frequency interference with the associated wireless communication electronics, and enhances shock resistance of the embedded processor and associated visual display 22. The Lexan material is used for the transparent display cover for the visual display 22. Proper operation of the control unit 10 requires the operator to be able to properly view the visual display 22 without interference. As such, the visual display 22 should include a transparent covering while providing resistance to external elements. Aluminum is also used in the composition of the case 20 to hold the Lexan display cover in a proper position. The operator control unit may be made from other materials including Aluminum, Aluminum Alloy, polymers, and other impact resistant materials. Accordingly, the materials that comprise the composition of the case contribute to the functionality of the unit.
Similar to the control unit as a whole, the input devices are designed to withstand harsh environmental conditions and to prevent dust, dirt and moisture from affecting communication with the embedded processor. In a preferred embodiment, the toggle switches 36 and 38 are separated by a silicone rubber membrane 76 and 78, respectively, to prevent moisture and dust from entering the case through any apertures associated with the switches 36 and 38. However, the membrane 76, 78 may be comprised of an alternative material that provides protection of the interior section of the case 20 and associated circuit board 60 from damage due to dust, moisture and environmental weather conditions. Similarly, the push button switches 32 and 34 are shown with a grommet 72 and 74, respectively, sealing the opening between the button and the case. The grommet 72, 74 protects the associated circuit board 60 within the case from damage from environmental elements such as dust and moisture. In a preferred embodiment, the grommet 72, 74 is made of a polymer material that can withstand dust and moisture. However, the grommet 72, 74 may be comprised of an alternative material that provides protection of the button 32, 34 and associated circuit board 60 from damage due to dust, moisture and environmental weather conditions.
The operator control unit 10 is a portable unit that may be held in the hands of an operator or may be secured to a limb or a garment of the operator. In order to enable an operator to perform multiple tasks simultaneously, the control unit 10 may be placed in a sleeve 200 as shown in Fig. 4. The sleeve 200 holds the hardware components of the control unit 10, while enabling the unit to be attached to a body part of the operator. Straps 210, 220 and 230 attach the sleeve to an object or limb of the operator. Each strap 210, 220 and 230 includes a proximal end 212, 222 and 232, respectively, and a distal end 214, 224 and 234, respectively. The distal end of each of the straps is remote from the sleeve. Attachment components may be in the form of hook and loop fasteners mounted on the proximal ends 212, 222 and 232 and distal ends 214, 224 and 234 of each strap 210, 220 and 230. This enables the sleeve to be adjustable for fitting on to different size body parts, as well different size bodies of operators. The sleeve 200 illustrated in Fig. 4 is one formative of the design and configuration of the sleeve. For example, if the sleeve 200 is mounted to a forearm of the operator, each of the straps 210, 220 and 230 are wrapped around the circumference of the forearm. The proximal ends 212, 222 and 232 of each strap have hook receiving means and the distal ends 214, 224 and 234 of each strap have hook means. This enables the distal end of each strap to secure to the proximal end of each strap. In addition, the use of mechanical fastening means, such as hook and loop fasteners, enables an operator to adjust the sizing of the sleeve depending upon the bulkiness of the clothing of the operator. Alternatively, other types of securing mechanisms, such as buttons, hooks, eyelets, snaps, and a plurality of mechanical fastening means may be used for the sleeve. Accordingly, the sleeve 200 is a component that holds the control unit 10 and enables an operator to attach the control unit to a part of his/her body to enable hands-free operation of the unit.
The sleeve shown in Fig. 4 includes a first pocket 240 adapted to receive the control unit 20. In addition, the sleeve 200 may include a second pocket 250 adapted to receive a power supply. The power supply stored in the second pocket 250 is a replaceable power source 252 . For example, the power source may be in the form of a battery pack. The second pocket 250 may be attached to the sleeve 200 by a hook and loop fastener associated with strap 220. At such time as the operator needs to access the power source 252 to provide a replacement to the power of the control unit 10, the operator may open the second pocket 250, remove the power source 252, and provide a replacement power source 252 in the pocket 250. The sleeve 200 may also include a third pocket 260 to hold an analog to digital converter 262. Fig. 5 is a perspective view of the sleeve 200 illustrating the side of the sleeve 200 with a second pocket 250 adapted to receive the power source 252 and a third pocket 260 adapted to receive the analog-digital converter 262. In addition to the pockets 240, 250 and 260, the sleeve 200 includes an aperture 270 that fits over the visual display 22, an aperture 280 to enable access to the joystick 30, and a set of apertures 292, 294, 296 and 298 to enable access to the switches 32, 34, 36 and 38. Alternatively, the power source 252 and analog- digital converter 262 may be mounted within an interior section of the case 20 of the control unit 10, or at an alternative location in proximity to the embedded processor. Accordingly, the sleeve 200 may include several pockets, an aperture 270 and apertures 292-298 to enable viewing data conveyed to the operator through the visual display and to enable access to the input devices of the control unit 10.
In addition to the control unit 10 shown in Fig. 1, a vibro-tactile feedback apparatus may be used in communication with the control unit 10 as an additional or alternative feedback device. One formative of the vibro-tactile feedback device 400 is shown in Fig. 6 in a form of a glove 410 to enable the operator to use their hand and fingers to communicate with the remote apparatus and to manipulate the controls of the unit 10. The glove 410 includes five independent sleeves 412, 414, 416, 418 and 420 for five fingers of the hand of the operator. Each of the sleeves 412-420 has a sensor 422, 424, 426, 428 and 430, respectively, that is sewn into the respective sleeve. In addition, a fingertip portion of each sleeve may include an additional sensor 432, 434, 436, 438 and 440. The glove 410 may also include an accelerometer 450 on the back of the hand portion of the glove. The accelerometer 450 may be a dual-axis accelerometer or a tri-axis accelerometer. A dual-axis accelerometer transmits the orientation of the hand in a two coordinate plane and communicates the orientation and action of the hand to the remote apparatus. Similarly, a tri-axis accelerometer records and communicates the orientation of the hand in a three coordinate plane. In one form of the glove, the sleeve sensors 422-430 are bend sensors which are activated upon bending a finger, and the fingertip sensors 432-440 are pressure sensors which are activated when pressure is applied to the sensor. Each of the sensors 422-430 and 432-440 may be analog or binary sensors. The sensors are wired to the analog-digital converter of the control unit 10. One or more of the fingertip sensors 432-440 are activated by exerting sufficient pressure to exceed a set threshold; the resulting sensor data is sent to a circuit board in the control unit 10 in communication with the activated sensor. Sensor data from the circuit board is forwarded to the analog-digital converter to place the data into a computer readable format and forward the data through a communication stream to the embedded processor. The data received by the embedded processor is forwarded to a wireless communication apparatus associated with the embedded processor, which sends the data to the remote apparatus. Various logical states of the remote apparatus are mapped to the actuated sensor. Similarly, bending of a finger in communication with one or more bend sensors 422-430 in a sleeve communicates data to the remote apparatus for control of various logical state on the remote apparatus. Accordingly, the sensors located on the finger portions and hand portion of the glove may be used to communicate with the remote apparatus by controlling specific logical states of the remote apparatus which are mapped to the sensor of the instrumented glove.
In addition to communicating actions to the remote apparatus from the glove, the control unit 10 may include a sleeve 460 with input devices such as buttons, switches, and other sensors to communicate with the remote apparatus. One form of the sleeve 460 is shown in Fig. 6 with vibro-tactile motors 462-474 embedded therein. The sleeve 460 is shown tethered to the control unit 10. When the remote apparatus communicates with the operator, it sends data through wireless communication electronics to a-corresponding wireless communication electronics associated with the embedded processor of the control unit 10. The data are received and processed by the embedded processor and forwarded to one or more of the vibro-tactile motors 462-474 which are mapped to the logical state of the remote apparatus. The actuated motor of the sleeve 460 -will vibrate or be otherwise actuated to communicate a phase of a logical state and/or actuation of a specific sensor or a group of sensors to the operator. The communication can be in the form of feedback associated with a command sent from the operator or movement of the remote apparatus. In addition, the remote apparatus may include proximity and sonar sensors to detect physical objects around the remote apparatus. The remote apparatus can communicate contact between a particular sensor and an obstacle. For example, if a specific proximity sensor is actuated, the remote apparatus will send feedback to a motor on the sleeve mapped to the sensor. The motor may vibrate or otherwise be actuated to communicate to the operator actuation of the associated sensor on the remote apparatus. In a further embodiment, the vibro-tactile motors 462-474 may be connected as an array of motors, such that specific pre-programmed actions are transmitted from the remote apparatus to the array which causes specific vibration patterns to be produced in the array. In addition, the operator can actuate one or more sensors 476-486 on the sleeve 460 to control a logical state on the remote apparatus. These sensors could include electromagnetic, force, strain, and acceleration activated sensors, as well as other types of sensors. Activation of one or more of the sensors 476-486 sends a data signal through the control unit 10 to the remote apparatus by means of the same process that signals from the instrumented glove are connected. Accordingly, the sensors and vibro-tactile motors of the sleeve enable the operator to transmit and receive data to and from a remote apparatus while providing full use and mobility of their hands.
The vibro-tactile sleeve 460 of Fig. 6 may be independent from, and tethered to the control unit 10, or the sleeve 460 may be tethered to the control unit 400. Fig. 6 illustrates one embodiment where the control unit 10 is mounted in one location and the sleeve 460 is tethered to the control unit 10. However, in a further embodiment, the vibro-tactile sleeve 460 may be designed with the control unit 10 embedded therein. This would provide the operator an alternative tool to receive data from a remote apparatus. Accordingly, the sleeve and glove apparatus are another form of remote communication between the remote apparatus and the operator.
In addition, both the sleeve 200, 460 are preferably comprised of a durable, lightweight and flexible material. The control unit is adapted to function in harsh environments, and the sleeves 200, 460 must be able to withstand harsh environmental conditions while protecting the components of the control unit 10. Additionally, the sleeves 200, 460 must be adaptable to be fitted onto different body parts of different operators, and must also be lightweight, as it is preferable to mitigate the weight of the components of the control unit 20. In a preferred embodiment, the sleeves are made of a Courdura fabric. However, alternative materials that meet the requirements of durability, flexibility and weight may be used for the sleeves. Accordingly, the material of the sleeves 200, 460 is an important factor in preserving the operation and handling of the control unit 10.
Software
The hardware components of the control unit 10, 405 may be used to visually convey data from a remote apparatus to an operator of the control unit. Visual data are displayed to the operator on the visual display 22. The embedded processor receives video data and displays the video data on the visual display 22. Fig. 7 is a flow chart 500 illustrating the steps involved in receiving a video stream and for displaying the video stream on the visual display in a manner that is understandable by the operator. The video stream that is received by the embedded processor is in a compressed format. The embedded processor receives compressed video data 502 from a remote apparatus through the wireless communication apparatus. The video data are then decompressed into pixel data to a known memory location 504. Thereafter, the decompressed pixel data are copied from the known memory location to the visual display for display in an interpretable format to the operator of the unit 506. Accordingly, video data for communication to the operator are processed from a compressed format to a decompressed format and displayed to the operator of the unit.
In addition to video data, the control unit 10, 405 is also adapted to convey commands from the operator of the unit to a remote apparatus. For example, actuation of the joystick 30 and/or switches 32-38 is a form of communication. Similarly, actuation of a sensor on the sleeve or a sensor in the glove generates command data. The data associated with actuation of a sensor or an alternative input device are processed and forwarded to a remote apparatus for actuation of the apparatus. Fig. 8 is a flow chart 520 illustrating the steps involved in processing command data and communicating with a remote apparatus. Actuation data received from an input device or sensor are conveyed to a circuit board connected to the input device 524. The actuation data may be in analog or digital format. Regardless of the format of the actuation data, the actuation data are sent from the circuit board to an analog- digital converter 526 for conversion to a computer readable format, i.e. digitized. The digitized data are placed into a digital communication stream 528 and then sent to a communication port of the embedded processor 530. The digital communication stream may be in RS232, USB or an alternative format depending upon the format utilized by the analog- digital converter. Following the step of sending the digitized data to the communication port, the digitized data are parsed 532 to determine the coordinate positions of the proportional data as well as the values associated with the data inputted through the input devices and/or sensors. The parsed data are computed 534 to determine the commands of the operator, and the computed data are forwarded to the remote apparatus 536 through wireless communication electronics. Accordingly, the data from the input devices are processed by the hardware of the control unit and forwarded from the control unit to the remote apparatus with which the unit is communicating. In addition to processing command and video data, the control unit includes a process for displaying telemetry data from a remote apparatus to an operator of the unit through the visual display. Telemetry data may include roll, pitch and yaw data for the position of a remote apparatus, global positioning satellite data, compass, radar, sonar, and other sensory data that convey any form of telemetry to communicate to an operator remote from the apparatus. Fig. 9 is a flow chart 550 -illustrating the steps involved in processing data for display to an operator through the visual display. Data are sent from the remote apparatus to the wireless communication apparatus in the control unit 554. The data are sent to the communication apparatus of the control unit in digital format. Thereafter, the data received by the communication apparatus are forwarded to the embedded processor of the control unit 556. In order to convey the data to the operator, the data are processed for display on the visual display 558. For example, graphical data are computed based upon telemetry data of the remote apparatus. Graphical data are computed in the form of pixel data. Appropriate pixel data such as roll, pitch and yaw are computed and stored in a known memory location 560. The raw pixel data values are copied from the known memory location to the visual display to convey the data to the control unit 562 in a visual format. Accordingly, the process of receiving and conveying telemetry data includes computing and processing position data of a remote apparatus and displaying said data on an operator of the control unit.
Telemetry data is not limited to visual display. Rather, the telemetry data may also be in the form of a vibro-tactile display as shown in Fig. 6. Fig. 1 1 is a flow chart 700 illustrating the steps involved in processing data for display to an operator through the vibro- tactile display. Data are sent from the remote apparatus to the communication apparatus in the control unit 704 via wireless communication electronics. The data are sent to the communication apparatus of the control unit in digital format. Thereafter, the data received by the communication apparatus are forwarded to the embedded processor of the control unit 706. In order to convey the data to the operator, the data are processed and motor values are recomputed based upon telemetry data 708. The new motor values are sent to the vibro- tactile display 710 to convey telemetry data to the control unit in a sensory format. Accordingly, the process of receiving and conveying telemetry data includes computing and processing position data of a remote apparatus and displaying the data to an operator of the control unit through the vibro-tactile display embedded in the sleeve of the control unit.
Once telemetry and video data are received and processed by the control unit, they are displayed to the operator through the visual display. Fig. 10 is one form of a graphical user interface 600 for conveying the received data to the operator in an intuitive manner. The wire frame 610 is a three dimensional visual representation of the remote apparatus. The position of the wire frame is indicative of the position of the remote apparatus. For example, as the data values of the roll, pitch and yaw values of the remote apparatus change, the position of the wire frame will rotate to reflect the roll, pitch and yaw data values. Similarly, the graphical user interface 600 may include a compass 620 to communicate the geographic orientation of the remote apparatus relative to the control unit. A significant portion of the graphical user interface is programmed to include a display for video data 630. All received video data that have been processed according to the steps outlined in Fig. 9 are displayed to the operator on the visual display. In addition to the video feed 630, wire frame 610 and compass 620, the graphical user interface 600 may include bar graphs 640, 650, 660 and 670 to convey additional telemetry data to the operator. As shown in Fig. 10, there are four bar graphs 640, 650, 660 and 670. The bar graphs may be used to display voltage, velocity, temperature, and communication data from the remote apparatus to the operator of the control unit. Each of these measurements conveys important factors to the operator for maintaining proper operation of the remote apparatus. In addition to the graphical inputs shown in Fig. 10, the graphical user interface 600 may be programmed to receive and display additional data. For example, the remote apparatus and control unit may be programmed to communicate with a satellite to convey position data at 680. Accordingly, the graphical user interface 600 is an intuitive display for conveying video and telemetry data received by the embedded processor to the operator of the control unit.
Advantages Over The Prior Art
Prior art teleoperation of remote apparatus require hardware and software which allow high-bandwidth information exchange, including video, audio and other forms of sensory data. Since teleoperation equipment is commonly used in a field environment, it is important that the components of the equipment be designed to withstand operation while exposed to dirt, dust, moisture and high impact conditions. The hardware components of the control unit, including peripheral devices, are fully ruggedized and are capable of withstanding extreme conditions, including dust, moisture and dirt. In addition, in a field environment it is important that the input device be modular to enable the operator to place the input devices in a comfortable operating location. Similarly, in certain operating conditions it is necessary for the operator to have both hands free to perform other tasks. Wearable computing devices such as a sleeve and/or glove enables the teleoperation equipment to be integrated into the operator's clothing or to be mounted on the body without impacting upon the vision of the operator or protruding from the body of the operator. In addition, the sleeve is adjustable to fit operators of different sizes. It allows the operator's hands to be free during operation and to enable the operator to perform various tasks while using the control unit.
Alternative Embodiments
It will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. In particular, the control unit may be designed to communicate with a variety of remote apparatus. For example, the remote apparatus may be in an electronic or mechanical form with logical states mapped to corresponding input devices and motors of the control unit. The control unit may also be used to download topographical data or other geographical data or to provide live-video feedback from visual sensors on a remote apparatus. In addition, the visual display may be in the form of a liquid crystal display, or an alternative display medium that enables viewing by the operator while maintaining the integrity of the control unit. Similarly, the wireless communication electronics may be in the form of wireless communication electronics connected to the embedded processor, or an alternative communication electronics that enables wireless communication of data between the embedded processor and a corresponding wireless communication apparatus on a remote apparatus. In addition, the scope of the invention should not be limited to the input devices described herein. Alternative input devices that enable communication of data between the control unit and the remote apparatus may be employed. Accordingly, the scope of protection of this invention is limited only by the following claims and their equivalents.

Claims

CLAIMSWe claim:
1. An operator control apparatus, comprising: an embedded processor with a visual display; an input device; a power supply; an analog-digital converter to digitize data received from said input device; communication electronics to communicate data between said analog-digital converter and said embedded processor; and a wireless communication apparatus in communication with the embedded processor to enable communication between said embedded processor and a remote apparatus.
2. The apparatus of claim 1, wherein said input device is selected from the group consisting of: an analog input device, a digital input device, and combinations thereof.
3. The apparatus of claim 2, wherein said analog input device is adapted to communicate a proportional signal to said remote apparatus.
4. The apparatus of claim 1, wherein said remote apparatus includes wireless communication apparatus to communicate with the embedded processor.
5. The apparatus of claim 1, wherein said communication electronics is integrated into said analog-digital converter.
6. The apparatus of claim 1, wherein said communication electronics is selected from the group consisting of serial, parallel, and a system bus.
7. The apparatus of claim 1, wherein said remote apparatus communicates with said embedded processor in data selected from the group consisting of: audio, visual, tactile, other sensory data, and combinations thereof.
8. The apparatus of claim 1, further comprising electronic circuitry to communicate data received from said input device to said analog-digital converter.
9. The apparatus of claim 1, wherein said visual display includes sensory controls to communicate operator commands to said embedded processor.
10. The apparatus of claim 1, wherein said communication electronics are configured for internet protocol communication.
11. The apparatus of claim 1, wherein said power supply is replaceable.
12. The apparatus of claim 1, further comprising a protective case to enclose components of said control unit.
13. The apparatus of claim 12, wherein said components are selected from the group consisting of: said embedded processor, said visual display, said communication electronics, said circuitry, said input device, said analog-digital converter, said power supply, and combinations thereof.
14. The apparatus of claim 12, wherein said protective case is comprised of an impact resistant material selected from the group consisting of: Aluminum, Aluminum Alloy, polymers, other impact resistant material, and combinations thereof.
15. The apparatus of claim 12, further comprising a sleeve adapted to receive said protective case.
16. The apparatus of claim 15, wherein said sleeve is adjustably mountable on a body part of an operator.
17. The apparatus of claim 1, wherein each component of said apparatus is modular.
18. The apparatus of claim 15, further comprising a sensor secured to said sleeve.
19. The apparatus of claim 15, further comprising a vibro-tactile motor secured to said sleeve.
20. The apparatus of claim 18, wherein said sleeve secured sensor is selected from the group consisting of: electromagnetic, acceleration, strain, pressure, other forms of biometric sensors, and combinations thereof.
21. The apparatus of claim 1 , further comprising a glove tethered to said embedded processor through said analog-digital converter and is adapted to receive a hand of an operator.
22. The apparatus of claim 21, wherein said glove comprises sensors and actuators to communicate with said remote apparatus.
23. The apparatus of claim 21, further comprising a pressure sensor mounted in a fingertip position of said glove.
24. The apparatus of claim 21 , further comprising a bend sensor mounted in a finger section of said glove.
25. The apparatus of claim 21, further comprising an accelerometer mounted to said glove.
26. The apparatus of claim 25, wherein said accelerometer is adapted to communicate variations in position and orientation of said hand to said remote apparatus.
27. The apparatus of claim 1, wherein said input device can be used to communicate with said visual display, a cursor in communication with said visual display, a graphical user interface, and combinations thereof.
28. An article comprising: a computer-readable signal-bearing medium; means in the medium for communicating video data for display on a control unit; means in the medium for communicating command data from said control unit to a remote apparatus; and means in the medium for communicating sensory data from said remote apparatus to said control unit.
29. The article of claim 28, wherein the medium is selected from the group consisting of: a recordable data storage medium, and a modulated carrier signal.
30. The article of claim 28, wherein said video data communication means receives and processes video data from said remote apparatus.
31. The article of claim 28, wherein said received video data is decompressed for display on a visual display of said control unit.
32. The article of claim 28, wherein said command data communication means receives data from actuators in communication with said control unit and communicates said command data from said control unit to said remote apparatus.
33. The article of claim 32, further comprising converting said received command data to a computer readable format.
34. The article of claim 32, wherein said received data is in a format selected from the group consisting of analog data and digital data.
35. The article of claim 28, wherein said sensory data communication means is adapted to transmit sensor data from said remote apparatus to said control unit through a wireless communication apparatus.
36. The article of claim 35, further comprising displaying said received data to an operator through a graphical user interface.
37. The article of claim 28, wherein said sensory data is selected from the group consisting of: visual data, vibro-tactile data, and combinations thereof.
38. An operator control apparatus, comprising: an embedded processor with a visual display; an input device mapped to a logical state; a power supply; an analog-digital converter to digitize data received from said input device; communication electronics to communicate data between said analog-digital converter and said embedded processor; a wireless communication apparatus in communication with the embedded processor to enable communication between said embedded processor and a wireless communication apparatus in communication with an embedded processor of a remote apparatus, wherein actuation of said input device is adapted to be communicated to said logical state.
PCT/US2002/028880 2002-06-07 2002-09-10 Wearable interface control unit WO2003103761A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2002336480A AU2002336480A1 (en) 2002-06-07 2002-09-10 Wearable interface control unit

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16473602A 2002-06-07 2002-06-07
US10/164,736 2002-06-07

Publications (1)

Publication Number Publication Date
WO2003103761A1 true WO2003103761A1 (en) 2003-12-18

Family

ID=29732035

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2002/028880 WO2003103761A1 (en) 2002-06-07 2002-09-10 Wearable interface control unit

Country Status (2)

Country Link
AU (1) AU2002336480A1 (en)
WO (1) WO2003103761A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220406212A1 (en) * 2021-06-19 2022-12-22 Danny Baldwin Virtual Reality Vehicle Operation Simulation

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6088017A (en) * 1995-11-30 2000-07-11 Virtual Technologies, Inc. Tactile feedback man-machine interface device
US6358187B1 (en) * 1999-05-19 2002-03-19 Carl M. Smith Apparatus for monitoring and displaying exertion data

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6088017A (en) * 1995-11-30 2000-07-11 Virtual Technologies, Inc. Tactile feedback man-machine interface device
US6358187B1 (en) * 1999-05-19 2002-03-19 Carl M. Smith Apparatus for monitoring and displaying exertion data

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220406212A1 (en) * 2021-06-19 2022-12-22 Danny Baldwin Virtual Reality Vehicle Operation Simulation
US11854434B2 (en) * 2021-06-19 2023-12-26 Danny Baldwin Virtual reality vehicle operation simulation

Also Published As

Publication number Publication date
AU2002336480A1 (en) 2003-12-22

Similar Documents

Publication Publication Date Title
US6239784B1 (en) Exo-skeletal haptic computer human/computer interface device
US9278453B2 (en) Biosleeve human-machine interface
US20020163495A1 (en) Multi-functional ergonomic interface
US9984541B2 (en) Modular exoskeletal force feedback controller
AU2011223942B2 (en) Intuitive multiple degrees of freedom portable control device
US8125448B2 (en) Wearable computer pointing device
US6850224B2 (en) Wearable ergonomic computer mouse
US6515669B1 (en) Operation input device applied to three-dimensional input device
USRE40698E1 (en) Hand-held trackball computer pointing device
US6554252B2 (en) Device and method for wireless lifting assist devices
US8605036B1 (en) Finger control and data entry device
US20060248478A1 (en) Sensing input actions
JPH0916316A (en) Portable transaction terminal for execution of optical inputand key input of data without using keyboard and hand-driven scanner
WO2016044251A1 (en) Method and System for Joint Position Measurement
US20090128485A1 (en) Pointing input device
WO2003103761A1 (en) Wearable interface control unit
KR20170053854A (en) Wearable motion measurement remote controller as glove
JPH0619618A (en) Input device
JP2001125728A (en) Operation input device
Tsagarakis et al. “BabyGlove”: a device to study hand motion control development in infants
CN215701677U (en) Wearable arm exoskeleton based on flexible sensor
WO2023127078A1 (en) Operation device and operation system
US20240085981A1 (en) Multimodal spacesuit smart glove
CN211987033U (en) Device for simulating and controlling steering wheel
US20230400922A1 (en) Handheld controller

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LU MC NL PT SE SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP