US20070190506A1

US20070190506A1 - Online interactive multimedia system and the transmission method thereof

Info

Publication number: US20070190506A1
Application number: US11/580,302
Authority: US
Inventors: Sheng-Wen Jeng; Zhi-Wei Lian; Shun-Nan Liou; Ming-Jye Tsai
Original assignee: Industrial Technology Research Institute ITRI
Current assignee: Industrial Technology Research Institute ITRI
Priority date: 2005-12-26
Filing date: 2006-10-13
Publication date: 2007-08-16
Also published as: TW200726252A; TWI341692B

Abstract

The present invention relates to an online interactive multimedia system and the transmission method, that the interactive system is adapted to be applied by users of a plurality of remote ends for enabling users at his/her remote end to interact with each other in an online fashion, and each remote end comprises: a at least a micro inertial sensing module, a micro processing device, an audio/video device, a network device, and a displaying device; wherein each micro inertial sensing module is capable of detecting and measuring motions of a user operating the remote end so as to generate gesture signals accordingly; the micro processing device is used for receiving/transmitting and processing audio/video as well as the gesture signals from/to other remote ends in an online fashion, while using the processed signals to control a software arranged in the online interactive multimedia system for enabling the displaying device to display images accordingly; the audio/video device is used for providing real-time audio/video service between the remote end and another remote end connected thereto through a network; and the network device, for connecting the remote end to a network system for enabling the remote end to transmit/receive audio/video and the gesture signals to/from another. remote ends.

Description

FIELD OF THE INVENTION

The present invention relates to an online interactive multirmedia system and the transmission method thereof, and more particularly, to an online video conferencing means capable of enabling remote users to interact with each other, not only by video and audio, but also by motions and gestures; that is, a user of the interactive system not only can perform video conference with other remote users in an online fashion, but also can further-have gesture interaction with them, so that the interactive system can be employed as a platform for interactive sport game, an online fitness training program, or a remote rehabilitation program.

BACKGROUND OF THE INVENTION

As the lifestyle and diet of modern people tends to change in the footsteps of time, more and more high-calorie foods are consumed while just activities of daily living are performed. This has resulted in an increasing number of persons who are obese. This is a problem even amongst children. It is known that a good exercise can help you burn off calories and increase your metabolic rate. Therefore, there are all kinds of fitness center and aerobic exercise clubs available for obese people. However, most of the exercise programs provided by such facilities either are performed at a specific location where trainees can receive instructions in group from a few trainers or instructors, or are capable of providing exercise videos to be purchased by trainees for enabling each trainee to perform the exercise by oneself at home while following the instructions of the videos. Although the exercise video is capable of enabling a person to perform exercise at any time and place, it can not motivate the trainee to develop an exercise routine that can become part of one's everyday life since it is usually boring and robotic, and can not enable the trainee to obtain instructions from trainer instantly for enabling the same to perform the exercise correctly.
One of such system is the electronic health monitoring system disclosed in U.S. Pat. No. 6,144,837, entitled “Method and apparatus for interactively monitoring a physiological condition and for interactively providing health-related information”, as shown in FIG. 1. In the operating of the health monitoring system 10, a multimedia device 15 is used to play an optic disc 17 containing programs of relating training/testing while interacting with a user therewith, and the same time that the physiological conditions of the user can be detected and converted into relating physiological signals by the electrodes 11, 12, 13 whereas the generated physiological signals are transmitted to be displayed on the monitor 16 by the interface device 14, and consequently, the physiological condition of the user is revealed. Although the aforesaid system can assist a user to exercise a training or testing by oneself at home, it is still short of interaction as it can not enable any two users, each at one's home, to interact with each other, and as the programs contained in the optic disc 17 for providing training/test that can be used in the system 10 are all designed to be practice following a specific schedule and can not be changed. Therefore, it is not practical. In addition, it is not economical since once the programs contained in the optic disc are done by the user, additional spending will be necessary to purchase disc containing new exercise programs.
Other examples of such systems are the two interactive teaching systems disclosed in TW Pat. No. 92119274 and the TW Pat. No. 91111827, which are online systems capable of utilizing multimedia devices and Internet to enable interaction between teachers and students, and vice versa. The operation modes of the aforesaid teaching systems are similar to that of the popular MSN messenger. Although the interaction of the aforesaid teaching system is enhanced, the interaction is restricted to audio and video only, and the users of the systems still can not interact with each other by gesture movements.

SUMMARY OF THE INVENTION

In view of the disadvantages of prior art, the primary object of the present invention is to provide an online interactive multimedia system and the transmission method thereof, capable of enabling remote users to interact with each other, not only by video and audio, but also by motions and gestures; that is, a user of the interactive system not only can perform video conference with other remote users in an online fashion, but also can further have gesture interaction with them, so that the interactive system can be employed as a platform for interactive sport game, an online fitness training program, or a remote rehabilitation program.
To achieve the above object, the present invention provides an online interactive multimedia system, adapted to be applied by users of a plurality of remote ends for enabling users at his/her remote end to interact with each other, each remote end comprising:

- at least a micro inertial sensing module, each capable of detecting and measuring motions of a user operating a remote end so as to generate a first gesture signals accordingly;
- a micro processing device, for receiving/transmitting the first gesture signal from/to other remote ends for forming a second gesture signal thereon, while processing the first and the second gesture signals thereof to generate a processing signal accordingly; and
- a displaying device, connected to the micro processing device for enabling the same to receive the processing signal.

Preferably, the micro inertial sensing module is primarily comprised of at least a micro inertial sensor, a wireless transceiver, and peripheral circuits.
Preferably, the micro inertial sensor can be a device selected from the group consisting of an accelerometer and a gyroscope.
Preferably, the micro inertial sensing module can be fitted to any part of a user's body.
Preferably, the micro inertial sensing module can be arranged inside a package.
Preferably, the package can be formed into a bat, a table tennis racket, and a golf club.
Preferably, the first and the second gesture signals are transmitted by a means of network transmission, whereas the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network.
To achieve the above object, the present invention further provides an online interactive multimedia system, adapted to be applied by users of a plurality of remote ends for enabling users at his/her remote end to interact with each other, each remote end comprising:

- at least a micro inertial sensing module, each capable of detecting and measuring motions of a user operating a remote end so as to generate a first gesture signals accordingly;
- an audio/video device, for fetching a first audio/video data from the remote end;
- a micro processing device, for receiving/transmitting the first gesture signal and the first audio/video data from/to other remote ends for forming a second gesture signal and a second audio/video data thereon, while the processing the first and the second gesture signals as well as the first and the second audio/video data to generate a processing signal accordingly; and
- a displaying device, connected to the micro processing device for enabling the same to receive the processing signal.

Preferably, the audio/video device includes camcorders, microphones, speakers, and so on.
Preferably, the audio signal received by the micro processing device is outputted from the audio/video device.
Moreover, to achieve, the above object, the present invention provides a transmission method of an online interactive multimedia, adapted to be applied by users, of a plurality of remote ends of an online interactive multimedia system for enabling users at his/her remote end to interact with each other, the method comprising steps of:

- detecting a gesture of a user operating a remote end for generating a first gesture signal accordingly;
- receiving/transmitting the first gesture signal from/to other remote ends for forming a second gesture signal thereon;
- processing the first and the second gesture signals to generate a processing signal accordingly; and
- receiving the processing signal while displaying corresponding images and motions with respect to the received processing signal.

In a preferred aspect, the transmission method further comprises steps of:

- fetching a first audio/video data form a remote end;
- receiving/transmitting the first audio/video data from/to other remote ends for forming a second gesture signal thereon;
  - processing the first and the second audio/video data to generate an audio/video processing signal accordingly; and
  - receiving the, audio/video processing signal while displaying corresponding audios and videos with respect to the received audio/video processing signal.

Preferably, as the processing signal is comprised of a first signal, acquired from the processing of the first gesture signal, and a second signal, acquired from the processing of the second gesture signal, the displaying of the corresponding images and motions with respect to the received processing signal further comprise steps of:

- displaying images and motions corresponding to the remote end with respect to the first signal; and
- displaying images and motions corresponding to the other remote ends with respect to the second signal.

Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an electronic health monitoring system disclosed in U.S. Pat. No. 6,144,837.
FIG. 2 is a flow chart illustrating a preferred embodiment of the invention.
FIG. 3 shows an online interactive multimedia system according to a preferred embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

For your esteemed members of reviewing committee to further understand and recognize the fulfilled functions and structural characteristics of the invention, several preferable embodiments cooperating with detailed description are presented as the follows.
Please refer to FIG. 2, which is a flow chart illustrating a preferred embodiment, of the invention. In FIG. 2, the transmission method of an online interactive multimedia, adapted to be applied by users of a plurality of remote ends of an online interactive multimedia system for enabling users at his/her remote end to interact with each other, starts at step 30. At step 30, information required for the transmission method is acquired before the flow proceeds to, step 31, which include two sub-steps 301, 302. At sub-step 301, motions of a user operating a remote end is detected and measured so as to generate a first gesture signals accordingly; and at sub-step 302, a first audio/video data of the aforesaid remote end is fetched.
At step 31, an operation is performed at the remote end for receiving the first gesture signal and the first audio/video data and transmitting the same to another remote end for causing a second gesture signal and a second audio/video data to be formed on the another remote end; and then the flow proceeds to step 32. At step 32, the first gesture signal, the first audio/video data, the second gesture signal and the second audio/video data are processed, whereas the process of step 32 further comprises steps of: processing the first and the second gesture signal to form a motion processing signal; and processing the first and the second audio/video data to form an audio/video processing signal; and then the flow proceeds to step 33. Moreover, in a preferred aspect, the signal transmission of the aforesaid flow chart is enabled by a means of network transmission, whereas the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network.
At step 33, the motion processing signal and the audio/video processing signal are received and used for generating corresponding images, sounds and motions with respect to the above processing, which further comprises steps of: displaying images and motions corresponding to the remote end with respect to a first signal, acquired from the processing of the first gesture signal and the first audio/video data; and displaying images and motions corresponding to the other remote ends with respect to a second signal, acquired from the processing of the second gesture signal and the second audio/video data. Thus, as users at the plural remote ends of the invention are enabled to communicate with each other through a means of network transmission, the display at a remote end not only is enabled to display images corresponding to the motions of the user operating the same, but also it is enabled to display images corresponding to the motions of the other users operating other remote end. For instance, a plurality of sub-frames can be presented on a display for enabling the same to display images relating to users at other remote ends.
Please refer to FIG. 3, which shows an online interactive multimedia system according to a preferred embodiment of the invention. In the FIG. 3, the two users A, B respectively operating two remote ends 20, 30 of the online interactive multimedia system are connected through a network system 40 for enabling signals to be transmitted to and from between the two remote ends 20, 30. In this preferred embodiment, more than one micro inertial sensing modules 21 are fitted to the user A operating the remote end 20 while more than one micro inertial sensing modules 31 are fitted to the, user B operating the remote end 30, whereas each of the micro inertial sensing modules 21, 31 can be enabled to be fitted to any part of the users A, B in respective, such as hand, waist, hip, leg, or even head, or neck. That is, it is preferred to attach to the movable parts of a human body. Each of the micro inertial sensing modules 21, 31 is capable of detecting and measuring motions of the users A, B in respective so as to generate gesture signals accordingly while transmitting the gesture signals by wireless transceiver arranged respectively in each of the micro inertial sensing modules 21, 31. Preferably, each of the micro inertial sensing modules 31, 31 is primarily comprised of at least a micro inertial sensor, whereas each micro inertial sensor can be an individual accelerometer, and an individual gyroscope, an assembly of a plural accelerometers, an assembly of a plural gyroscopes, or the combinations thereof. The aforesaid accelerometers or gyroscopes should function with corresponding peripheral circuits so as to perform an operation of acceleration calculation and signal processing, that is known to those skilled in the art and is voided to be described further herein. As the gesture signals are generated by the detection of each of the micro inertial sensing modules 21, 31 upon the movement of the users A, B in respective, the gesture signals are respectively sent to the micro processing devices 23, 33 to be evaluated and interpreted. Any one of the two micro processing devices 22, 33 is capable of receiving/transmitting and processing audio/video signals as well as the gesture signals from/to other remote ends in an online fashion, while using the processed signals to control an interactive software of the online interactive multimedia system for enabling the displaying devices 24, 34 to display images accordingly. It is noted that the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network whereas the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network. As seen in FIG. 3, each displaying devices 24, 34 has a sub-displaying frame, which can be used to display a second image, such as a close-up of the head of the user A, or a still image, etc. Moreover, as each of the micro inertial sensing modules 21, 31 is detecting respectively the motions of the two users A, B, two audio/video devices are used for providing a real-time audio/video service between the two remote ends 20, 30 according to the online fashion, that each of the two audio/ video devices 20, 30 includes audio/video receivers selected form the group consisting of camcorders, microphones, speakers, and the like.
From the above description, it is concluded that a transmission method of an online interactive multimedia, adapted to be applied by users of a plurality of remote ends of the online interactive multimedia system is provided, which comprises steps of:

- (a) detecting gestures of users A, B, respectively operating remote ends 20, 30 for generating-gesture signals accordingly by the use of micro inertial sensing modules 21, 31;
- (b) receiving/transmitting and processing audio/video signals as well as the gesture signal from/to other remote end in an online fashion, while using the processed signals to control a software for respectively enabling displaying devices 24, 34 to display images accordingly by the use of micro processing devices 23, 33;
- (c) providing a real-time audio/video service between the two remote ends 20, 30 according to the online fashion by the use of audio/ video devices 22, 32 in respective;
- (d) connecting the two remote ends 20, 30 to a network system 40 for enabling the two remote ends 20, 30 to transmit/receive audio/video and the gesture signals to/from each other; and
- (e) displaying the audio/video signals and the gesture signal received from the network system 40 on the displaying devices 24, 34 in respective.

Furthermore, instead of arranging each of the micro inertial sensing modules 21, 31 at any location of the two user A, B, each micro inertial sensing modules 21, 31 can be arranged inside a package, whereas the package is formed into a shape of a batting means selected from the group consisting of a bat, a table tennis racket, a golf club, and the like. Such that, by cooperating with matching software and displaying device, a batting movement can be simulated using the package. For instance, If the user A operating the remote end 20 is holding a gold-club-shaped package containing a micro inertial sensing module 21, the swing of the user A can be detected by the micro inertial sensing module 21 and then the result of the detection is transmitted to the micro processing device 23 while the audio/video device 22 is also transmitting the sounds and images of the swing to the micro processing device 23; and then the micro processing device 23 is enabled to transmit all the signals received thereby to another remote end 30 through the network system 40 where the micro processing device 33 will process all those signals to be displayed on the display device 34. Similarly, the above description is suitable to the user B operating the remote end 30 for display the motions of the user B on the displaying device 24 of the remote end 20 operated by the user A. Assuming that user A operating the remote end 20 is a student and user B operating the remote end 30 is a coach, the coach is able to see the swing of the student on his/her displaying device 34 in real time so that he/she can specifically correct the gesture, strength and trajectory of the awing instantly. Assuming that user A operating the remote end 20 is a patient and user B operating the remote end 30 is a doctor, the doctor is able to see the rehabilitation exercise performed by the patient on his/her displaying device 34 in real time so that he/she can specifically direct the exercise instantly. In addition, online game can be enabled by the user of the network server of the network system 40, that is, the two users A, B, are able to select an on line ball game on the network system so as to play the game against each other. It is noted that the amount of user is not limited by the two users A, B, as described in the preferred embodiment shown in FIG. 3, that it can be as many as specified by the online software containing in the online interactive multimedia system.
To sum up, the present invention provides an online interactive multimedia system and the transmission method thereof, and more particularly, to an online video conferencing means capable of enabling remote users to interact with each other, not only by video and audio, but also by motions and gestures; that is, a user of the interactive system not only can perform video conference with other remote users in an online fashion, but also can further have gesture interaction with them.
While the preferred embodiment of the invention has been set forth for the purpose of disclosure, modifications of the disclosed embodiment of the invention as well as other embodiments thereof may occur to those-skilled in the art. Accordingly, the appended claims are intended to cover all embodiments which do not depart from the spirit and scope of the invention.

Claims

1. An online interactive multimedia system, adapted to be applied by users of a plurality of remote ends for enabling users at his/her remote end to interact with each other, each remote end comprising:

at least a micro inertial sensing module, each capable of detecting and measuring motions of a user operating a remote end so as to generate a first gesture signals accordingly;

a micro processing device, for receiving/transmitting the first gesture signal from/to other remote ends for forming a second gesture signal thereon, while processing the first and the second gesture signals thereof to generate a processing signal accordingly; and

a displaying device, connected to the micro processing device for enabling the same to receive the processing signal.

2. The system of claim 1, wherein the micro inertial sensing module is primarily comprised of at least a micro inertial sensor, a wireless transceiver, and peripheral circuits.

3. The system of claim 1, wherein the micro inertial sensing module is comprised of at least an accelerometer.

4. The system of claim 1, wherein the micro inertial sensing module is comprised of at least a gyroscope.

5. The system of claim 1, wherein the micro inertial sensing module is enabled to be fitted to any part of a user's body.

6. The system of claim 1, wherein the micro inertial sensing module can be arranged inside a package.

7. The system of claim 6, wherein the package is formed into a shape of a batting means selected from the group consisting of a bat, a table tennis racket, a golf club, and the like.

8. The system of claim 1, wherein the micro processing device is able to control the software according to the online fashion.

9. The system of claim 1, wherein the first and the second gesture signals are transmitted by a means of network transmission, whereas the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network.

10. An online interactive multimedia system, adapted to be applied by users of a plurality of remote ends for enabling users at his/her remote end to interact with each other, each remote end comprising:

an audio/video device, for fetching a first audio/video data from the remote end;

a micro processing device, for receiving/transmitting the first gesture signal and the first audio/video data from/to other remote ends for forming a second gesture signal and a second audio/video data thereon, while the processing the first and the second gesture signals as well as the first and the second audio/video data to generate a processing signal accordingly; and

11. The system of claim 10, wherein the micro inertial sensing module is primarily comprised of at least a micro inertial sensor, a wireless transceiver, and peripheral circuits.

12. The system of claim 10, wherein the micro inertial sensing module is comprised of at least an accelerometer.

13. The system of claim 10, wherein the micro inertial sensing module is comprised of at least a gyroscope.

14. The system of claim 10, wherein the micro inertial sensing module is enabled to be fitted to any part of a user's body.

15. The system of claim 10, wherein the micro inertial sensing module can be arranged inside a package.

16. The system of claim 15, wherein the package is formed into a shape of a batting means selected from the group consisting of a bat, a table tennis racket, a golf club, and the like.

17. The system of claim 10, wherein the audio/video device includes audio/video receivers selected form the group consisting of camcorders, microphones, speakers, and the like.

18. The system of claim 10, wherein the first and the second audio/video data received by the micro processing device are outputted from the audio/video device.

19. The system of claim 10, wherein the micro processing device is able to control the software according to the online fashion.

20. The system of claim 10, wherein the first and the second gesture signals are transmitted by a means of network transmission, whereas the means of network transmission is a means selected from the group consisting of an ADSL network, a cable local area network, a Bluetooth communication network, a 3G mobile network and a GPRS mobile network.

21. A transmission method of an online interactive multimedia, adapted to be applied by users of a plurality of remote ends of the online interactive multimedia system for enabling users at his/her remote end to interact with each other, comprising steps of:

detecting a gesture of a user operating a remote end for generating a first gesture signal accordingly;

receiving/transmitting the first gesture signal from/to other remote ends for forming a second gesture signal thereon;

processing the first and the second gesture signals to generate a processing signal accordingly; and

receiving the processing signal while displaying corresponding images and motions with respect to the received processing signal.

22. The method of claim 21, further comprising steps of:

fetching a first audio/video data form a remote end;

receiving/transmitting the first audio/video data from/to other remote ends for forming a second gesture signal thereon;

processing the first and the second audio/video data to generate an audio/video processing signal accordingly; and

receiving the audio/video processing signal while displaying corresponding audios and videos with respect to the received audio/video processing signal.

23. The method of claim 21, wherein the processing signal is comprised of a first signal, acquired from the processing of the first gesture signal, and a second signal, acquired from the processing of the second gesture signal.

24. The method of claim 23, wherein the displaying of the corresponding images and motions with respect to the received processing signal further comprise steps of:

displaying images and motions corresponding to the remote end with respect to the first signal; and

displaying images and motions corresponding to the other remote ends with respect to the second signal.