WO2012007034A1 - Sending and receiving information - Google Patents

Sending and receiving information Download PDF

Info

Publication number
WO2012007034A1
WO2012007034A1 PCT/EP2010/060081 EP2010060081W WO2012007034A1 WO 2012007034 A1 WO2012007034 A1 WO 2012007034A1 EP 2010060081 W EP2010060081 W EP 2010060081W WO 2012007034 A1 WO2012007034 A1 WO 2012007034A1
Authority
WO
WIPO (PCT)
Prior art keywords
region
sub
display
causing
user
Prior art date
Application number
PCT/EP2010/060081
Other languages
French (fr)
Inventor
Qifen Yan
Antti Oskari Salo
Teppo Tapani Jokinen
Jarkko Tapio SAUNAMÄKI
Piers Andrew
Lotta Maria Partanen
Zoran Radivojevic
Original Assignee
Nokia Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Corporation filed Critical Nokia Corporation
Priority to PCT/EP2010/060081 priority Critical patent/WO2012007034A1/en
Publication of WO2012007034A1 publication Critical patent/WO2012007034A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066Session management
    • H04L65/1083In-session procedures
    • H04L65/1094Inter-user-equipment sessions transfer or sharing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066Session management
    • H04L65/1096Supplementary features, e.g. call forwarding or call holding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/75Indicating network or usage conditions on the user display
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/141Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/142Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
    • H04N7/144Constructional details of the terminal equipment, e.g. arrangements of the camera and the display camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/141Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/147Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/15Conference systems
    • H04N7/157Conference systems defining a virtual conference space and using avatars or agents

Definitions

  • This invention relates to sending and receiving information.
  • Modern smartphones such as the N97 from Nokia Corporation are configured for sending and receiving video information so as to allow users at different locations to communicate face-to-face in real time.
  • a first aspect of the invention provides a method comprising causing a first device to send information to and receive information from a second device, causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, causing the first device to define at least one sub-region of said region of the display, each of the at least one sub- region displaying a part of said graphical information and causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
  • a second aspect of the invention provides an apparatus comprising at least one processor and at least one memory having stored therein computer program code, wherein the computer program code is configured, when executed by the at least one processor using the at least one memory, to cause the apparatus to perform a method comprising causing a first device to send information to and receive information from a second device, causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information, and causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
  • a third aspect of the invention provides an apparatus comprising means for causing a first device to send information to and receive information from a second device, means for causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, means for causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information and means for causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
  • a fourth aspect of the invention provides a non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to cause a first device to send information to and receive information from a second device, cause the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, cause the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information and cause the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
  • Figure 1 shows a device according to aspects of the invention
  • FIG. 2 and Figure 3 illustrate some aspects of the hardware and software
  • Figure 4 is a block diagram illustrating a configuration of communication software of the device of Figure 1.
  • Figure 5 shows the device of Figure 1 in communication with a remote device.
  • Figure 6 to 9 show display outputs of the device of Figure 1 and the remote device during a video call;
  • Figure 10 is a flow chart illustrating operation of the Figure 1 device.
  • Figure 11 is a block diagram illustrating an alternative configuration of the device communication software.
  • the device 100 embodies aspects of the invention and is not limiting on the scope of the invention.
  • the device 100 has a touch sensitive display, or touchscreen 102 for displaying content and for receiving tactile user inputs comprising one or more touch inputs.
  • the touchscreen 102 may be a resistive or a capacitive touchscreen, an optical touch touchscreen, an infra-red touchscreen or a surface acoustic wave touchscreen.
  • the device may also comprise one or more physical keys 104 for receiving inputs from a user.
  • the device 100 also has a front-facing video camera 106 configured for electronic motion picture acquisition for use in video calling and other applications.
  • the device 100 may be a mobile computer, mobile phone, PDA, internet tablet, smartphone or other portable communications device.
  • FIG. 2 is a block diagram illustrating some aspects of the hardware and software configuration of the device 100.
  • the device 100 includes a controller 200 comprising one or more processors.
  • the controller 200 communicates with other hardware components via a system bus 201.
  • Each other hardware component is connected to the system bus 201 either directly or via an interface.
  • the hardware components include the touchscreen 102, video camera 106 and also a speaker 220 and microphone 230.
  • the device 100 also contains a transceiver 210 connected to the system bus 201 via an interface for communicating over a wireless link, such as a GSM, CDMA, UMTS, LTE, WiMax or Wi-Fi link.
  • the device 100 includes both volatile memory 212 and non-volatile memory 214.
  • the volatile memory 212 is RAM, DRAM or any other type of volatile memory.
  • the volatile memory 212 is used by the controller 200 for the temporary storage of data, for instance when controlling the operation of a hardware or software component or moving data between components.
  • the non-volatile memory 214 is ROM, flash memory, a hard disk or any other type of non- volatile memory.
  • the non- volatile memory 214 stores computer readable instructions used in operating the device 100 as well as storing content and personal data such as images, videos, music and contacts.
  • the controller 200 operates under the control of computer-readable instructions stored in the non-volatile memory 214, which include an operating system 216 and additional software, middleware or firmware modules, and software applications.
  • the operating system 216 is the Maemo operating system, developed by Nokia
  • the touch sensitive display 102 comprises a display part 202 and a tactile interface part 204.
  • the tactile interface part 204 may be a multi-touch device configured to simultaneously detect multiple touch contacts, or may alternatively be a single-touch device.
  • the controller 200 receives inputs from the tactile interface 204 and controls the display shown on the display part 202.
  • the device 100 is shown as comprising a user interface 300.
  • This incorporates the touchscreen 102 and physical keys 104, but also includes outputs, particularly in the form of text and graphics provided on the display part 202.
  • the user interface 300 incorporates the user interface software 218, which is configured to operate along with user interface hardware, including the touchscreen 102 and physical keys 104.
  • the user interface software 218 may be separate from the operating system 216, in which case it interacts closely with the operating system as well as the applications. Alternatively, the user interface software 218 may be integrated with the operating system.
  • software applications include communication software 240 configured for controlling one or more of the hardware components to effect communication between a user of the device and user(s) of remote device(s). It will be appreciated that the communication software 240 receives user inputs by way of the user interface 300 and also interacts with the user interface in producing device outputs.
  • Figure 4 shows a communication software configuration according to an exemplary embodiment of the invention. As shown, the communication software 240 includes a video telephony software application 410 configured for controlling the display part 202, video camera 106, speaker 220, microphone 230, transceiver 210 and other hardware components to effect video communication with users of remote devices.
  • the communication software 240 also includes video processing software 420 for processing video data generated by the video camera 106 during a video call mediated by the video telephony software 410.
  • the video processing software 420 includes face recognition software 422 for identifying faces and facial features displayed on the display part 202.
  • the video processing software 420 also includes region identification software 424 for identifying regions of the display part in which faces or facial features such as ears, noses or eyes are displayed. The region
  • identification process may be said to be dynamic because as the video image on the display changes, for example when a call participant moves their head position or the device 100, the region identification software 424 updates the identified region accordingly to account for the change.
  • Dynamically identifying regions in which a face or facial feature is displayed is known per se to those skilled in the art. Relevant references include, inter alia:
  • the video processing software 420 further comprises video modifying software 426 for modifying the video image, for example by cropping the video image or by enlarging or reducing some areas in size, or by modifying the displayed video in other ways.
  • the video processing software 420 may further comprise gesture recognition software 430 for processing video data to identify human gestures, for example facial or hand gestures.
  • gesture recognition is known per se and described for example in US5454043 and US2008005703, and will not be described in more detail here.
  • the communication software 240 also includes tactile input processing software 440 comprising touch gesture recognition software 450 for processing tactile user inputs received by the touchscreen 102 to identify touch gestures.
  • tactile input processing software 440 comprising touch gesture recognition software 450 for processing tactile user inputs received by the touchscreen 102 to identify touch gestures.
  • touch gestures may be associated with different operations for execution on the device 100. Some gestures may involve movements such as a "pinch" gesture at a specific location, for example.
  • the gesture recognition software 430 and/ or the touch gesture recognition software 450 may have a behaviour recognition engine which can be "trained” to identify user- defined gestures by an iterative process whereby a user repeats the gesture until it can be unambiguously identified by the device 100.
  • the communication software 240 to mediate communication between a user of the device 100 and a user of a remote device 110 will now be described.
  • the remote device 110 will be described as identical to the device 100 and like reference numerals will be used to identify like components.
  • the devices may include the same communication software 240, the invention is not limited to communication between identical or even generally similar devices and the remote device 110 may be of a completely different type to the device 100 and may have substantially different features and configuration.
  • Figures 5 to 9 illustrate operations of the device 100 and the remote device 110 during a communication session between the devices 100, 110. It will be appreciated that in the operations of Figures 5 to 9, the controllers 200 of the devices 100, 110 perform the stated functions and actions, including controlling the display of content on the display parts 202, under control of instructions that form part of the communication software 240 and the user interface software 218 running on the operating systems 216 of the devices 100, 110.
  • Figure 5 shows the first and second devices 100, 110 connected via the wireless link by way of the transceivers 210.
  • the first device 100 and the second devices are exchanging information under control of the communication software 240 running on the devices 100, 110, thereby effecting a video call communication between Jane, a user of the first device, and Frank, a user of the second device.
  • the display part 202 of Jane's device 100 displays graphical information comprising video of Frank.
  • the video is derived from video information captured by the video camera 106 of Frank's device 110 and transmitted to Jane's device 100 over the wireless link.
  • the video data received from Frank's device 110 may be processed at Jane's device 100 and/or Frank's device and modified before it is displayed.
  • the region identification software 424 of Jane's device 100 may dynamically identify a region of the video in which Frank's face is displayed and the video modification software 426 may then crop the video so that Frank's face is appropriately and centrally framed by the display part 202.
  • Figure 6 shows the output 600 of the display part 202 of Jane's device 100 at one particular instant during the video call and illustrates a video 610 of Frank being displayed on the display part 202.
  • the display part 600 also displays graphical information in the form of video 620 of Jane, which is derived from video information captured by the video camera 106 of Jane's device 100. It will be appreciated that this video information may be processed and modified before it is displayed, in a similar way to that described above in relation to the video 610 of Frank.
  • Figure 7 shows the corresponding output on Frank's device 110. As shown, the display of Frank's device 110 has a similar layout to that of Jane's device but in this case the main video image 710 is of Jane and the secondary video image 720 is of Frank.
  • the devices 100, 110 may exchange signals and process the video images so as to ensure that the video image 710 of Jane on Frank's display is the same as the corresponding video image 620 on Jane's display and so that the video image 610 of Frank on Jane's display is the same as the corresponding video image 720 on Frank's display.
  • the video call allows Jane to interact with Frank by way of audio and visual communication, for example by speaking and by way of gestures.
  • Jane can also interact with Frank by way of tactile user input comprising touch contact with the region of the display part 206 where Frank's video image 610 is displayed.
  • the additional "touch" option presents an intuitive and fun option for enhancing the communication.
  • verbal communication is not possible or not wanted, the "touch" interaction allows discrete, fast and silent communication.
  • the communication software 206 is configured to execute different operations depending on the location of the touch interaction. As illustrated in the example of Figure 6, when Jane touches Frank's ears, the communication software 206 disables the audio stream from Jane's device 100 to Frank's device 110 so that Jane's voice is muted to Frank.
  • the location of Frank's ears and other facial features are determined by the region identification software 424.
  • the region identification software 424 dynamically identifies different regions of the display part 206 where Frank's facial features are displayed. For example, the region identification software 424 identifies regions where Frank's ears are displayed and regions were Frank's eyes are displayed. These display regions can be referred to as sub-regions since they are sub-regions of the larger display region in which Frank's video image is displayed. An example of the identified sub-regions 612 for Frank's eyes is shown in Figure 6.
  • the communication software 206 identifies when a touch contact on the tactile interface 204 is incident at a location of the display part within a sub-region, and in response executes a corresponding set of operations. For example, when the communication software 240 identifies that a touch contact is incident within the sub- region for Frank's ears, the communication software mutes Jane's audio stream to Frank's device, as described above. Further operations may also be executed, for example notifications may be displayed on the display part 206 to indicate that Jane has been muted to Frank, as shown in Figure 6. As shown, notification may comprise images overlayed with the video image such as a "mute" icon 640 and/ or images 660 of fingertips positioned over Frank's ears to indicate that Jane has muted her audio to Frank by way of a touch interaction.
  • the operations performed in response to a touch contact on Jane's device 100 may also include operations performed on Frank's device 110. Such operations may be performed in response to signals sent from Jane's device 100 to Frank's device 110 to indicate that particular operations should be performed on Frank's device. For example, as illustrated in Figure 7, when Jane touches Frank's ears on her device 100, notifications are displayed on the display part 204 of Frank's device 110, including fingers 702 positioned over Frank's ears in the video image 720 and a notification icon 730 to indicate that audio from Jane has been muted.
  • the communication software 260 also allows Jane to interact with the video image 620 of herself by way of tactile input.
  • the communication software 240 may be configured to identify when Jane touches a sub-region displaying her own ears and in response to mute the audio stream from Frank to Jane. It will be appreciated that interaction with the video image 620 is implemented in a similar way to interaction with the video image 610.
  • Frank's device is configured in a similar way to that of Jane's device and as a result Frank can also interact with Jane by way of touch contact with the video images of Jane and himself which are displayed on his device 110. For example, Frank may mute the audio stream from Frank's device 110 to Jane's device 100 by touch contact with a sub-region corresponding to Jane's ears, causing corresponding notifications to be displayed on Frank and Jane's display.
  • a set of operations may be performed in response to touch contact in different sub-regions. For example, as shown in Figures 8 and 9, when Frank touches a sub- region for Jane's eyes, a set of operations are performed comprising 1) reducing the brightness of the video image of Frank on Jane's device 100, 2) displaying an image of fingers covering Jane's eyes over the video of Jane displayed on Franks device 110 and 2) displaying an image of hands covering Jane's eyes over the video of Jane displayed on Jane's device 100.
  • the video data stream to Jane's device 100 may be blocked so that Jane cannot see Frank.
  • a set of operations may be performed when a specific touch gesture is recognised by the touch gesture recognition software 450, the touch gesture comprising one or more touch contacts in a sub-region displaying a facial feature.
  • the audio may mute the corresponding device 100, 110 and display notifications to both devices 100, 110 by displaying an image denoting a pinch on the mouth, along with a mute icon.
  • the devices 100, 110 may also be configured to execute particular operations in response to a combination of touch and one or more hand or other gestures. For example, if the gesture recognition software 430 identifies a raised index finger by processing video data from the camera 106, and after a short moment the touch gesture recognition software identifies a touch contact on the mouth, a "mute" operation may be executed.
  • the sub-regions may be determined to be larger than the region on which the corresponding facial feature is displayed, so that for example a tactile input representing a pinch on the lips may be identified even where the gesture is not precisely incident on the lips.
  • the sub-regions corresponding to specific facial features may not necessarily display those facial features. For example, sub-regions for Frank's lips may be determined to be above or below the region of the display part 204 where the lips are actually displayed, so that an exaggerated pinch gesture on Frank's face may be recognised as a user input for muting Frank's audio stream.
  • different communication options such as muting and adjusting the video brightness may be selected by tactile user input comprising touch contact with sub-regions of the display parts 202 of the devices 100, 110.
  • communication options may be directly adjusted by touch, without the need to navigate through menus and options of the user interface and without it being necessary to exit a full-screen video mode.
  • a video processing operation in which the appearance of a participants face is modified may be performed in response to a corresponding tactile user input.
  • the devices 100, 110 are configured to stretch or enlarge a participants face 610, 620, 710, 720 in response to a corresponding tactile user input, for example in response to a multi- touch dragging input.
  • individual facial features may be enlarged or reduced in size by way of tactile user input comprising one or more touch contacts with sub-regions for the features. In this way, users may distort participants faces for fun and to express themselves.
  • a user may enlarge a particular facial feature to make that feature easier for the other user to interact with.
  • the video processing software 420 is configured to visually enhance the participants faces or bodies, thereby to enable an avatar-like appearance.
  • FIG. 10 is a flow chart illustrating a method according to an exemplary embodiment of the invention.
  • a video call is initiated.
  • graphical information representing a participant in the video call is displayed, as shown in step 1010.
  • sub-regions are dynamically identified by identifying body features of the participant displayed on the display.
  • User input comprising at least one touch contact with a sub-region is received at step 1030 and in response, at least one operation is caused to be performed by the device 100 or the remote device 110, as shown at step 1040.
  • graphical information in the form of video images of the participants may be displayed on the devices 100, 110.
  • users may interact with the still images by tactile input, thereby to cause corresponding operations to be performed.
  • Figure 11 illustrates an exemplary communication software configuration for this type of communication.
  • the communication software 240 of Figure 11 has avatar display software 1100 for displaying graphical information comprising avatars of participants in the
  • the avatars may be two dimensional or three dimensional models or icons representing the participants, and may be generated manually by avatar generation software 1100 running on the devices 100, 110.
  • avatar generation software 1100 for manual avatar generation is well known per se and will not be described in detail here.
  • the avatars may be generated automatically by the devices 100, 110 from video or still images of the participants.
  • Jane's device 100 displays an avatar for Frank and also an avatar for Jane.
  • avatars are displayed for Jane and Frank.
  • Jane's avatar may be generated on Jane's device 100 and transmitted to Frank's device 110 over the wireless link.
  • Franks' avatar may be generated on Frank's device 100 and transmitted to Jane's device 110.
  • the devices 100, 100 are responsive to tactile input comprising touch contact with sub-regions of the display and are configured to respond to these tactile inputs by executing operations on one or both of the devices 100,110.
  • the sub-regions relate to regions of the display in which body or facial features of the avatars are displayed. Accordingly, Frank and Jane may interact with the avatars in a similar manner to the manner described above in relation to the video call images 610, 620, 710, 720 and similar or the same operations may be performed by the devices 100, 110 in response.
  • the devices 100, 110 may be configured so that in response to a multi touch dragging gesture comprising touch contact with a sub-region for Jane's avatar's mouth, the shape of the mouth is changed, for by twisting the shape of the mouth or by changing the shape from a frown to a smile.
  • the devices 100, 110 exchange data so that any such change to the appearance of an avatar is made on both devices. That is, both parties to the call get informed about the interaction in real time. It will be appreciated that many other interactions are possible.
  • the avatar's faces may be enlarged or distorted, or some facial features may be enlarged to make them easier to interact with.
  • a "pinch" on the mouth of one participant may disable the audio stream from that participant.
  • a user can pinch an avatar's mouth to reduce its size and in this way slightly lower the volume, instead of totally muting it.
  • user distortion of an avatar's mouth may cause changes in the audio stream associated with the avatar so that, for entertainment, the avatar adopts a distorted, frequency shifted or modulated voice.
  • the other avatar may fade or temporarily disappear from view.
  • Many other operations may be performed in response to tactile input, for example operations involving audio and/ or vibration on one or both devices.
  • the avatars may be animated or may be still images.
  • the avatars are animated to reflect body movements captured by the video cameras 106 of the devices 100, 110. That is, video captured by the video cameras 106 may be processed to identify gestures or body/ facial movements of the participants and the avatars may be animated accordingly. For example, in response to identifying that Jane has moved her head to the left, Jane's avatar's head also be caused to move to the left.
  • the animation need not necessarily reflect all body movements exactly and for example if a participant's face moves outside of the camera view-finder, the avatar may still be caused to remain fully on-screen for further interaction.

Abstract

A method comprises causing a first device to send information to and receive information from a second device, causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, causing the first device to define at least one sub-region of said region of the display, each ofthe at least one sub-region displaying a part of said graphical information and causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/or the second device.

Description

Sending and Receiving Information Field of the Invention
This invention relates to sending and receiving information.
Background of the Invention
Modern smartphones such as the N97 from Nokia Corporation are configured for sending and receiving video information so as to allow users at different locations to communicate face-to-face in real time.
Summary of the Invention
A first aspect of the invention provides a method comprising causing a first device to send information to and receive information from a second device, causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, causing the first device to define at least one sub-region of said region of the display, each of the at least one sub- region displaying a part of said graphical information and causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
A second aspect of the invention provides an apparatus comprising at least one processor and at least one memory having stored therein computer program code, wherein the computer program code is configured, when executed by the at least one processor using the at least one memory, to cause the apparatus to perform a method comprising causing a first device to send information to and receive information from a second device, causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information, and causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device. A third aspect of the invention provides an apparatus comprising means for causing a first device to send information to and receive information from a second device, means for causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, means for causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information and means for causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
A fourth aspect of the invention provides a non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to cause a first device to send information to and receive information from a second device, cause the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display, cause the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information and cause the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
Brief Description of the Drawings
Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
Figure 1 shows a device according to aspects of the invention;
Figure 2 and Figure 3 illustrate some aspects of the hardware and software
configuration of the device of Figure 1;
Figure 4 is a block diagram illustrating a configuration of communication software of the device of Figure 1.
Figure 5 shows the device of Figure 1 in communication with a remote device. Figure 6 to 9 show display outputs of the device of Figure 1 and the remote device during a video call;
Figure 10 is a flow chart illustrating operation of the Figure 1 device; and.
Figure 11 is a block diagram illustrating an alternative configuration of the device communication software.
Detailed Description
Referring firstly to Figure 1, a device 100 is shown. The device 100 embodies aspects of the invention and is not limiting on the scope of the invention. The device 100 has a touch sensitive display, or touchscreen 102 for displaying content and for receiving tactile user inputs comprising one or more touch inputs. The touchscreen 102 may be a resistive or a capacitive touchscreen, an optical touch touchscreen, an infra-red touchscreen or a surface acoustic wave touchscreen. The device may also comprise one or more physical keys 104 for receiving inputs from a user. The device 100 also has a front-facing video camera 106 configured for electronic motion picture acquisition for use in video calling and other applications. The device 100 may be a mobile computer, mobile phone, PDA, internet tablet, smartphone or other portable communications device. Figure 2 is a block diagram illustrating some aspects of the hardware and software configuration of the device 100. The device 100 includes a controller 200 comprising one or more processors. The controller 200 communicates with other hardware components via a system bus 201. Each other hardware component is connected to the system bus 201 either directly or via an interface. The hardware components include the touchscreen 102, video camera 106 and also a speaker 220 and microphone 230. The device 100 also contains a transceiver 210 connected to the system bus 201 via an interface for communicating over a wireless link, such as a GSM, CDMA, UMTS, LTE, WiMax or Wi-Fi link. The device 100 includes both volatile memory 212 and non-volatile memory 214.
Both types of memory are accessed by controller 200 via the system bus 201 and both types are connected directly to the system bus 201. The volatile memory 212 is RAM, DRAM or any other type of volatile memory. The volatile memory 212 is used by the controller 200 for the temporary storage of data, for instance when controlling the operation of a hardware or software component or moving data between components. The non-volatile memory 214 is ROM, flash memory, a hard disk or any other type of non- volatile memory. The non- volatile memory 214 stores computer readable instructions used in operating the device 100 as well as storing content and personal data such as images, videos, music and contacts.
The controller 200 operates under the control of computer-readable instructions stored in the non-volatile memory 214, which include an operating system 216 and additional software, middleware or firmware modules, and software applications. The operating system 216 is the Maemo operating system, developed by Nokia
Corporation. It may alternatively be the Meego operating system, Symbian, or another operating system. The touch sensitive display 102 comprises a display part 202 and a tactile interface part 204. The tactile interface part 204 may be a multi-touch device configured to simultaneously detect multiple touch contacts, or may alternatively be a single-touch device. The controller 200 receives inputs from the tactile interface 204 and controls the display shown on the display part 202.
Referring to Figure 3, the device 100 is shown as comprising a user interface 300. This incorporates the touchscreen 102 and physical keys 104, but also includes outputs, particularly in the form of text and graphics provided on the display part 202. The user interface 300 incorporates the user interface software 218, which is configured to operate along with user interface hardware, including the touchscreen 102 and physical keys 104. The user interface software 218 may be separate from the operating system 216, in which case it interacts closely with the operating system as well as the applications. Alternatively, the user interface software 218 may be integrated with the operating system.
Referring again to Figure 2, software applications include communication software 240 configured for controlling one or more of the hardware components to effect communication between a user of the device and user(s) of remote device(s). It will be appreciated that the communication software 240 receives user inputs by way of the user interface 300 and also interacts with the user interface in producing device outputs. Figure 4 shows a communication software configuration according to an exemplary embodiment of the invention. As shown, the communication software 240 includes a video telephony software application 410 configured for controlling the display part 202, video camera 106, speaker 220, microphone 230, transceiver 210 and other hardware components to effect video communication with users of remote devices. The communication software 240 also includes video processing software 420 for processing video data generated by the video camera 106 during a video call mediated by the video telephony software 410. As shown, the video processing software 420 includes face recognition software 422 for identifying faces and facial features displayed on the display part 202. The video processing software 420 also includes region identification software 424 for identifying regions of the display part in which faces or facial features such as ears, noses or eyes are displayed. The region
identification process may be said to be dynamic because as the video image on the display changes, for example when a call participant moves their head position or the device 100, the region identification software 424 updates the identified region accordingly to account for the change. Dynamically identifying regions in which a face or facial feature is displayed is known per se to those skilled in the art. Relevant references include, inter alia:
1. M. Kass, A. WItkin, D. Terzopoulos, Snakes: active contour models, Int. J. Comput. Vision 1 (4) (1988) 321-331.
2. A. Yuille, P. Haallinan, D.S. Cohen, Feature extraction from faces using deformable templates, Int. J. Comput. Vision 8 (2) (1992) 99-111.
3. G. E. Hintion, C. K. I. Williams, and M. D. Revow, Adaptive elastic models for handprinted character recognition, in Advances in Neural Information Processing Systems 4, Morgan Kauffmann, San Mateo, CA, 1992
4. T.F. Cootes, C.J. Taylor, D.H. Cooper, J. Graham, Active shape models— their training and application, Comput. Vision Image Understanding 61 (1) (1995) 38-59. 5. T.F. Cootes, G.J. Edwards, C. Taylor, Active appearance models, IEEE Trans. Pattern Anal. Mach. Intell. 23 (6) (2001) 681-685.
6. X.W. Hou, S.Z. Li, H.J. Zhang, Q.S. Cheng, Direct appearance models, Proceedings of CVPR01, vol. 1, 2001, pp. 828-833.
7. Zhong Xuea, Stan Z. Lib and Eam Khwang Teoh, AI-EigenSnake: an affine-invariant deformable contour model for object matching, Image and Vision Computing Volume 20, Issue 2, 1 February 2002, Pages 77-84
8. S. Sclaroff, J. Isidoro, Active blobs: region-based, deformable appearance models, Comput. Vision Image Understanding 89 (2-3) (2003) 197-225.
9. "Computer-Assisted Lip Reading Recognition for Hearing Impaired", Lecture Notes in Computer Science published by Springer Berlin /Heidi eberg, ISSN 0302- 9743.
The video processing software 420 further comprises video modifying software 426 for modifying the video image, for example by cropping the video image or by enlarging or reducing some areas in size, or by modifying the displayed video in other ways.
The video processing software 420 may further comprise gesture recognition software 430 for processing video data to identify human gestures, for example facial or hand gestures. Gesture recognition is known per se and described for example in US5454043 and US2008005703, and will not be described in more detail here.
The communication software 240 also includes tactile input processing software 440 comprising touch gesture recognition software 450 for processing tactile user inputs received by the touchscreen 102 to identify touch gestures. Different touch gestures may be associated with different operations for execution on the device 100. Some gestures may involve movements such as a "pinch" gesture at a specific location, for example.
The gesture recognition software 430 and/ or the touch gesture recognition software 450 may have a behaviour recognition engine which can be "trained" to identify user- defined gestures by an iterative process whereby a user repeats the gesture until it can be unambiguously identified by the device 100.
Operation of the communication software 240 to mediate communication between a user of the device 100 and a user of a remote device 110 will now be described. For ease of understanding, the remote device 110 will be described as identical to the device 100 and like reference numerals will be used to identify like components.
However, it will be appreciated that although the devices may include the same communication software 240, the invention is not limited to communication between identical or even generally similar devices and the remote device 110 may be of a completely different type to the device 100 and may have substantially different features and configuration.
Figures 5 to 9 illustrate operations of the device 100 and the remote device 110 during a communication session between the devices 100, 110. It will be appreciated that in the operations of Figures 5 to 9, the controllers 200 of the devices 100, 110 perform the stated functions and actions, including controlling the display of content on the display parts 202, under control of instructions that form part of the communication software 240 and the user interface software 218 running on the operating systems 216 of the devices 100, 110.
Figure 5 shows the first and second devices 100, 110 connected via the wireless link by way of the transceivers 210. As shown, the first device 100 and the second devices are exchanging information under control of the communication software 240 running on the devices 100, 110, thereby effecting a video call communication between Jane, a user of the first device, and Frank, a user of the second device.
During the video call, the display part 202 of Jane's device 100 displays graphical information comprising video of Frank. The video is derived from video information captured by the video camera 106 of Frank's device 110 and transmitted to Jane's device 100 over the wireless link. The video data received from Frank's device 110 may be processed at Jane's device 100 and/or Frank's device and modified before it is displayed. For example, the region identification software 424 of Jane's device 100 may dynamically identify a region of the video in which Frank's face is displayed and the video modification software 426 may then crop the video so that Frank's face is appropriately and centrally framed by the display part 202. Figure 6 shows the output 600 of the display part 202 of Jane's device 100 at one particular instant during the video call and illustrates a video 610 of Frank being displayed on the display part 202.
The display part 600 also displays graphical information in the form of video 620 of Jane, which is derived from video information captured by the video camera 106 of Jane's device 100. It will be appreciated that this video information may be processed and modified before it is displayed, in a similar way to that described above in relation to the video 610 of Frank. Figure 7 shows the corresponding output on Frank's device 110. As shown, the display of Frank's device 110 has a similar layout to that of Jane's device but in this case the main video image 710 is of Jane and the secondary video image 720 is of Frank. The devices 100, 110 may exchange signals and process the video images so as to ensure that the video image 710 of Jane on Frank's display is the same as the corresponding video image 620 on Jane's display and so that the video image 610 of Frank on Jane's display is the same as the corresponding video image 720 on Frank's display.
The video call allows Jane to interact with Frank by way of audio and visual communication, for example by speaking and by way of gestures. As will be described in more detail below, Jane can also interact with Frank by way of tactile user input comprising touch contact with the region of the display part 206 where Frank's video image 610 is displayed. The additional "touch" option presents an intuitive and fun option for enhancing the communication. Further, where verbal communication is not possible or not wanted, the "touch" interaction allows discrete, fast and silent communication. The communication software 206 is configured to execute different operations depending on the location of the touch interaction. As illustrated in the example of Figure 6, when Jane touches Frank's ears, the communication software 206 disables the audio stream from Jane's device 100 to Frank's device 110 so that Jane's voice is muted to Frank.
The location of Frank's ears and other facial features are determined by the region identification software 424. During the video call, the region identification software 424 dynamically identifies different regions of the display part 206 where Frank's facial features are displayed. For example, the region identification software 424 identifies regions where Frank's ears are displayed and regions were Frank's eyes are displayed. These display regions can be referred to as sub-regions since they are sub-regions of the larger display region in which Frank's video image is displayed. An example of the identified sub-regions 612 for Frank's eyes is shown in Figure 6.
The communication software 206 identifies when a touch contact on the tactile interface 204 is incident at a location of the display part within a sub-region, and in response executes a corresponding set of operations. For example, when the communication software 240 identifies that a touch contact is incident within the sub- region for Frank's ears, the communication software mutes Jane's audio stream to Frank's device, as described above. Further operations may also be executed, for example notifications may be displayed on the display part 206 to indicate that Jane has been muted to Frank, as shown in Figure 6. As shown, notification may comprise images overlayed with the video image such as a "mute" icon 640 and/ or images 660 of fingertips positioned over Frank's ears to indicate that Jane has muted her audio to Frank by way of a touch interaction.
The operations performed in response to a touch contact on Jane's device 100 may also include operations performed on Frank's device 110. Such operations may be performed in response to signals sent from Jane's device 100 to Frank's device 110 to indicate that particular operations should be performed on Frank's device. For example, as illustrated in Figure 7, when Jane touches Frank's ears on her device 100, notifications are displayed on the display part 204 of Frank's device 110, including fingers 702 positioned over Frank's ears in the video image 720 and a notification icon 730 to indicate that audio from Jane has been muted.
The communication software 260 also allows Jane to interact with the video image 620 of herself by way of tactile input. For example, the communication software 240 may be configured to identify when Jane touches a sub-region displaying her own ears and in response to mute the audio stream from Frank to Jane. It will be appreciated that interaction with the video image 620 is implemented in a similar way to interaction with the video image 610.
Frank's device is configured in a similar way to that of Jane's device and as a result Frank can also interact with Jane by way of touch contact with the video images of Jane and himself which are displayed on his device 110. For example, Frank may mute the audio stream from Frank's device 110 to Jane's device 100 by touch contact with a sub-region corresponding to Jane's ears, causing corresponding notifications to be displayed on Frank and Jane's display.
Other sets of operations may be performed in response to touch contact in different sub-regions. For example, as shown in Figures 8 and 9, when Frank touches a sub- region for Jane's eyes, a set of operations are performed comprising 1) reducing the brightness of the video image of Frank on Jane's device 100, 2) displaying an image of fingers covering Jane's eyes over the video of Jane displayed on Franks device 110 and 2) displaying an image of hands covering Jane's eyes over the video of Jane displayed on Jane's device 100. Optionally, rather than reducing the brightness of the video image representing Frank, the video data stream to Jane's device 100 may be blocked so that Jane cannot see Frank.
In other examples, a set of operations may be performed when a specific touch gesture is recognised by the touch gesture recognition software 450, the touch gesture comprising one or more touch contacts in a sub-region displaying a facial feature. For example, in response to a touch gesture comprising pinching the mouth, the audio may mute the corresponding device 100, 110 and display notifications to both devices 100, 110 by displaying an image denoting a pinch on the mouth, along with a mute icon. The devices 100, 110 may also be configured to execute particular operations in response to a combination of touch and one or more hand or other gestures. For example, if the gesture recognition software 430 identifies a raised index finger by processing video data from the camera 106, and after a short moment the touch gesture recognition software identifies a touch contact on the mouth, a "mute" operation may be executed.
It will be appreciated that not all of the touch contacts involved with a particular touch gesture need to be in a specific sub-region in order for the corresponding operation to be executed. Further, the sub-regions may be determined to be larger than the region on which the corresponding facial feature is displayed, so that for example a tactile input representing a pinch on the lips may be identified even where the gesture is not precisely incident on the lips. In other examples, the sub-regions corresponding to specific facial features may not necessarily display those facial features. For example, sub-regions for Frank's lips may be determined to be above or below the region of the display part 204 where the lips are actually displayed, so that an exaggerated pinch gesture on Frank's face may be recognised as a user input for muting Frank's audio stream.
As described above, different communication options such as muting and adjusting the video brightness may be selected by tactile user input comprising touch contact with sub-regions of the display parts 202 of the devices 100, 110. In this way, communication options may be directly adjusted by touch, without the need to navigate through menus and options of the user interface and without it being necessary to exit a full-screen video mode.
Instead of or in addition to changing a communication option, other operations such as a video processing operation may be performed. For example, a video processing operation in which the appearance of a participants face is modified may be performed in response to a corresponding tactile user input. In some examples, the devices 100, 110 are configured to stretch or enlarge a participants face 610, 620, 710, 720 in response to a corresponding tactile user input, for example in response to a multi- touch dragging input. Furthermore, individual facial features may be enlarged or reduced in size by way of tactile user input comprising one or more touch contacts with sub-regions for the features. In this way, users may distort participants faces for fun and to express themselves. In addition, a user may enlarge a particular facial feature to make that feature easier for the other user to interact with.
Many other operations could be performed in response to tactile interaction with the video images 610, 620, 710, 720, for example vibration and/or audio operations on one or both devices. Further, in some examples, the video processing software 420 is configured to visually enhance the participants faces or bodies, thereby to enable an avatar-like appearance.
Figure 10 is a flow chart illustrating a method according to an exemplary embodiment of the invention. At step 1000, a video call is initiated. During the video call graphical information representing a participant in the video call is displayed, as shown in step 1010. At step 1020, sub-regions are dynamically identified by identifying body features of the participant displayed on the display. User input comprising at least one touch contact with a sub-region is received at step 1030 and in response, at least one operation is caused to be performed by the device 100 or the remote device 110, as shown at step 1040.
Although the interaction above has been described with reference to graphical information in the form of video images of the participants, alternatively graphical information in the form of still images of the participants may be displayed on the devices 100, 110. As with video images, users may interact with the still images by tactile input, thereby to cause corresponding operations to be performed.
Further, in some embodiments, instead of interacting with video or still images of the participants, users of the device 100 and the remote device 110 communicate in real time by interacting with avatars of themselves and of each other. Figure 11 illustrates an exemplary communication software configuration for this type of communication. Instead of video telephony software 410 and video processing software 420, the communication software 240 of Figure 11 has avatar display software 1100 for displaying graphical information comprising avatars of participants in the
communication session. The avatars may be two dimensional or three dimensional models or icons representing the participants, and may be generated manually by avatar generation software 1100 running on the devices 100, 110. Avatar generation software for manual avatar generation is well known per se and will not be described in detail here. Alternatively, the avatars may be generated automatically by the devices 100, 110 from video or still images of the participants.
In one example, during an "avatar interaction" communication session between Frank and Jane, instead of displaying a main video call image 610 of Frank and a secondary video call image 620 of Jane, Jane's device 100 displays an avatar for Frank and also an avatar for Jane. On Frank's device 110, instead of displaying video call images 710, 720, avatars are displayed for Jane and Frank. Jane's avatar may be generated on Jane's device 100 and transmitted to Frank's device 110 over the wireless link. Similarly, Franks' avatar may be generated on Frank's device 100 and transmitted to Jane's device 110.
As with a video call interaction, in an "avatar interaction" communication, the devices 100, 100 are responsive to tactile input comprising touch contact with sub-regions of the display and are configured to respond to these tactile inputs by executing operations on one or both of the devices 100,110. The sub-regions relate to regions of the display in which body or facial features of the avatars are displayed. Accordingly, Frank and Jane may interact with the avatars in a similar manner to the manner described above in relation to the video call images 610, 620, 710, 720 and similar or the same operations may be performed by the devices 100, 110 in response.
For example, the devices 100, 110 may be configured so that in response to a multi touch dragging gesture comprising touch contact with a sub-region for Jane's avatar's mouth, the shape of the mouth is changed, for by twisting the shape of the mouth or by changing the shape from a frown to a smile. The devices 100, 110 exchange data so that any such change to the appearance of an avatar is made on both devices. That is, both parties to the call get informed about the interaction in real time. It will be appreciated that many other interactions are possible. For example, in response to tactile input the avatar's faces may be enlarged or distorted, or some facial features may be enlarged to make them easier to interact with.
Further, where audio is enabled, a "pinch" on the mouth of one participant may disable the audio stream from that participant. In some implementations a user can pinch an avatar's mouth to reduce its size and in this way slightly lower the volume, instead of totally muting it. Alternatively, or in addition, user distortion of an avatar's mouth may cause changes in the audio stream associated with the avatar so that, for entertainment, the avatar adopts a distorted, frequency shifted or modulated voice. Further alternatively or in addition, when a participant covers an avatar's eyes, the other avatar may fade or temporarily disappear from view. Many other operations may be performed in response to tactile input, for example operations involving audio and/ or vibration on one or both devices.
The avatars may be animated or may be still images. In some examples, the avatars are animated to reflect body movements captured by the video cameras 106 of the devices 100, 110. That is, video captured by the video cameras 106 may be processed to identify gestures or body/ facial movements of the participants and the avatars may be animated accordingly. For example, in response to identifying that Jane has moved her head to the left, Jane's avatar's head also be caused to move to the left. The animation need not necessarily reflect all body movements exactly and for example if a participant's face moves outside of the camera view-finder, the avatar may still be caused to remain fully on-screen for further interaction.
Although communication between only two participants is described above, in some embodiments more than two participants communicate, for example in a video conference. In such group communications, graphical information relating to all of the participants may be displayed on the device of each participant and any two participants may interact as described above. In a group communication, signals may be exchanged between the devices so that each user is informed of each interaction. It should be realized that the foregoing embodiments should not be construed as limiting. Other variations and modifications will be apparent to persons skilled in the art upon reading the present application. Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or any application derived therefrom, new claims may be formulated to cover any such features and/ or combination of such features.

Claims

Claims
1. A method comprising:
causing a first device to send information to and receive information from a second device;
causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display;
causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information;
causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device. 2. A method according to claim 1, wherein causing said first device to send information to and to receive information from a second device comprises effecting a video call and wherein the graphical information comprises video of the user of the first device or of the second device. 3. A method according to claim 1, wherein the graphical information comprises an avatar of the user of the first device or of the second device.
4. A method according to any preceding claim, further comprising the first device defining said sub-region based on identification of a body feature of a user displayed on the display
5. A method according to claim 4, comprising dynamically defining said sub- region. 6. A method according to any preceding claim, comprising causing the first device to define at least two sub-regions of said region of the display, and causing a different operation to be performed dependent on the sub-region at which the touch user input was received.
7. A method according to claim 6, further comprising identifying each sub- region by identifying a body feature of a user displayed on the display. 8. A method according to claim 6 or claim 7, comprising dynamically identifying each sub-region.
9. A method according to any preceding claim, wherein the first device is responsive to a user input comprising user selection of a sub-region.
10. A method according to any preceding claim, wherein the first device is responsive to a user input comprising a touch gesture, wherein said gesture is one of a plurality of gestures, each gesture corresponding to a different operation. 11. A method according to any preceding claim, wherein said operation comprises a command for changing a communication option.
12. A method according to any preceding claim, wherein said operation comprises changing the graphical information, the method comprising causing display of said changed graphical information, further comprising the first device causing generation of a signal configured to control the second device to display the changed graphical information, and causing said signal to be transmitted to the second device so that said changed graphical information can be displayed by the second device.
13. A method according to claim 12, further comprising identifying a sub-region of said region of the display by identifying a body feature of a user displayed on the display, and causing the first device to be responsive to a user input comprising at least one touch contact with said sub-region, wherein said changed graphical information comprises changed graphical information representing said body feature.
14. A method according to any preceding claim, wherein said at least one operation comprises an operation performed on the first device and a
corresponding operation performed on the second device. 15. A method according to any preceding claim, comprising causing display of graphical information representing all participants in the communication.
16. Computer program code, optionally stored on a computer readable medium, which when executed by a computer apparatus causes the apparatus to perform the method of any of claims 1 to 15.
20. An apparatus comprising
at least one processor; and
at least one memory having stored therein computer program code; wherein the computer program code is configured, when executed by the at least one processor using the at least one memory, to cause the apparatus to perform a method comprising:
causing a first device to send information to and receive information from a second device;
causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display;
causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information;
causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device. 21. An apparatus according to claim 20, wherein causing said first device to send information to and to receive information from a second device comprises effecting a video call and wherein the graphical information comprises video of the user of the first device or of the second device.
22. An apparatus according to claim 20, wherein the graphical information comprises an avatar of the user of the first device or of the second device. 23. An apparatus according to any of claims 20 to 22, wherein the method comprises the first device defining said sub-region based on identification of a body feature of a user displayed on the display
24. An apparatus according to claim 23, comprising dynamically defining said sub-region.
25. An apparatus according to any of claims 20 to 24, wherein the method comprises causing the first device to define at least two sub-regions of said region of the display, and causing a different operation to be performed dependent on the sub-region at which the touch user input was received.
26. An apparatus according to claim 25, wherein the method further comprises identifying each sub-region by identifying a body feature of a user displayed on the display.
27. An apparatus according to claim 25 or claim 26, wherein the method further comprises dynamically identifying each sub-region.
28. An apparatus according to any of claims 20 to 27, wherein the first device is responsive to a user input comprising user selection of a sub-region.
29. An apparatus according to any of claims 20 to 28, wherein the first device is responsive to a user input comprising a touch gesture, wherein said gesture is one of a plurality of gestures, each gesture corresponding to a different operation.
30. An apparatus according to any of claims 20 to 29, wherein said operation comprises a command for changing a communication option.
31. An apparatus according to any of claims 20 to 30, wherein said operation comprises changing the graphical information, the method comprising causing display of said changed graphical information and further comprising the first device causing generation of a signal configured to control the second device to display the changed graphical information, and causing said signal to be transmitted to the second device so that said changed graphical information can be displayed by the second device.
32. An apparatus according to claim 31, wherein the method further comprises identifying a sub-region of said region of the display by identifying a body feature of a user displayed on the display, and causing the first device to be responsive to a user input comprising at least one touch contact with said sub-region, wherein said changed graphical information comprises changed graphical information
representing said body feature.
33. A method according to any of claims 20 to 32, wherein said at least one operation comprises an operation performed on the first device and a
corresponding operation performed on the second device. 34. A method according to any of claims 20 to 33, wherein the method further comprises causing display of graphical information representing all participants in the communication.
35. An apparatus comprising:
means for causing a first device to send information to and receive information from a second device;
means for causing the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display;
means for causing the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information; means for causing the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
36. A non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by computing apparatus, causes the computing apparatus to:
cause a first device to send information to and receive information from a second device;
cause the first device to display graphical information representing a user of the first device or a user of the second device in a region of a display;
cause the first device to define at least one sub-region of said region of the display, each of the at least one sub-region displaying a part of said graphical information;
cause the first device to be responsive to a touch user input in one of said at least one sub-region to cause at least one operation to be performed by the first device and/ or the second device.
PCT/EP2010/060081 2010-07-13 2010-07-13 Sending and receiving information WO2012007034A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/EP2010/060081 WO2012007034A1 (en) 2010-07-13 2010-07-13 Sending and receiving information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2010/060081 WO2012007034A1 (en) 2010-07-13 2010-07-13 Sending and receiving information

Publications (1)

Publication Number Publication Date
WO2012007034A1 true WO2012007034A1 (en) 2012-01-19

Family

ID=43827631

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2010/060081 WO2012007034A1 (en) 2010-07-13 2010-07-13 Sending and receiving information

Country Status (1)

Country Link
WO (1) WO2012007034A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014241093A (en) * 2013-06-12 2014-12-25 株式会社リコー Information processing system, operation apparatus, information display method, and program
EP4220369A4 (en) * 2020-10-27 2024-03-20 Vivo Mobile Communication Co Ltd Interaction method and apparatus for video call

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5454043A (en) 1993-07-30 1995-09-26 Mitsubishi Electric Research Laboratories, Inc. Dynamic and static hand gesture recognition through low-level image analysis
US20080005703A1 (en) 2006-06-28 2008-01-03 Nokia Corporation Apparatus, Methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications
US20080055272A1 (en) * 2006-09-06 2008-03-06 Freddy Allen Anzures Video Manager for Portable Multifunction Device
US20090040289A1 (en) * 2007-08-08 2009-02-12 Qnx Software Systems (Wavemakers), Inc. Video phone system
EP2098947A2 (en) * 2008-03-04 2009-09-09 Apple Inc. Selecting of text using gestures
EP2114073A2 (en) * 2008-04-30 2009-11-04 LG Electronics Inc. Mobile terminal and method for controlling video call thereof

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5454043A (en) 1993-07-30 1995-09-26 Mitsubishi Electric Research Laboratories, Inc. Dynamic and static hand gesture recognition through low-level image analysis
US20080005703A1 (en) 2006-06-28 2008-01-03 Nokia Corporation Apparatus, Methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications
US20080055272A1 (en) * 2006-09-06 2008-03-06 Freddy Allen Anzures Video Manager for Portable Multifunction Device
US20090040289A1 (en) * 2007-08-08 2009-02-12 Qnx Software Systems (Wavemakers), Inc. Video phone system
EP2098947A2 (en) * 2008-03-04 2009-09-09 Apple Inc. Selecting of text using gestures
EP2114073A2 (en) * 2008-04-30 2009-11-04 LG Electronics Inc. Mobile terminal and method for controlling video call thereof

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
A. YUILLE; P. HAALLINAN; D.S. COHEN: "Feature extraction from faces using deformable templates", INT. J. COMPUT. VISION, vol. 8, no. 2, 1992, pages 99 - 111
M. KASS; A. WITKIN; D. TERZOPOULOS: "Snakes: active contour models", INT. J. COMPUT. VISION, vol. 1, no. 4, 1988, pages 321 - 331
S. SCLAROFF; J. ISIDORO: "Active blobs: region-based, deformable appearance models", COMPUT. VISION IMAGE UNDERSTANDING, vol. 89, no. 2-3, 2003, pages 197 - 225
T.F. COOTES; C.J. TAYLOR; D.H. COOPER; J. GRAHAM: "Active shape models-their training and application", COMPUT. VISION IMAGE UNDERSTANDING, vol. 61, no. 1, 1995, pages 38 - 59
T.F. COOTES; G.J. EDWARDS; C. TAYLOR: "Active appearance models", IEEE TRANS. PATTERN ANAL. MACH. INTELL., vol. 23, no. 6, 2001, pages 681 - 685
X.W. HOU; S.Z. LI; H.J. ZHANG; Q.S. CHENG: "Direct appearance models", PROCEEDINGS OF CVPR01, vol. 1, 2001, pages 828 - 833
ZHONG XUEA; STAN Z. LIB; EAM KHWANG TEOH; AI-EIGENSNAKE: "an affine-invariant deformable contour model for object matching", IMAGE AND VISION COMPUTING, vol. 20, no. 2, 1 February 2002 (2002-02-01), pages 77 - 84

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014241093A (en) * 2013-06-12 2014-12-25 株式会社リコー Information processing system, operation apparatus, information display method, and program
EP4220369A4 (en) * 2020-10-27 2024-03-20 Vivo Mobile Communication Co Ltd Interaction method and apparatus for video call

Similar Documents

Publication Publication Date Title
KR102062310B1 (en) Method and apparatus for prividing control service using head tracking in an electronic device
US11513608B2 (en) Apparatus, method and recording medium for controlling user interface using input image
CN110471596B (en) Split screen switching method and device, storage medium and electronic equipment
CN105814522B (en) Device and method for displaying user interface of virtual input device based on motion recognition
WO2015188614A1 (en) Method and device for operating computer and mobile phone in virtual world, and glasses using same
EP4002066A1 (en) Air gesture interaction method and electronic device
US11546457B2 (en) Electronic device and method of operating electronic device in virtual reality
CN108920069B (en) Touch operation method and device, mobile terminal and storage medium
CN111045511B (en) Gesture-based control method and terminal equipment
US11740754B2 (en) Method for interface operation and terminal, storage medium thereof
US20180239526A1 (en) Method and systems for touch input
CN111610912B (en) Application display method, application display device and storage medium
WO2017076017A1 (en) Method and apparatus for operating application
EP3575917A1 (en) Collecting fingerprints
JP2021531589A (en) Motion recognition method, device and electronic device for target
CN113655929A (en) Interface display adaptation processing method and device and electronic equipment
CN111988522B (en) Shooting control method and device, electronic equipment and storage medium
CN109547696B (en) Shooting method and terminal equipment
WO2012007034A1 (en) Sending and receiving information
WO2020034763A1 (en) Gesture recognition method, and gesture processing method and apparatus
KR20230168568A (en) Method and apparatus for inputting information, electronic device and storage medium
CN108877742A (en) Luminance regulating method and device
CN109213349A (en) Exchange method and device, computer readable storage medium based on touch screen
CN113325948A (en) Air-isolated gesture adjusting method and terminal
CN111093035A (en) Image processing method, electronic device, and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10737827

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10737827

Country of ref document: EP

Kind code of ref document: A1