Optimized telepresence using mobile device gestures

• US 9,294,722 B2
• Filed: 10/19/2010
• Issued: 03/22/2016
• Est. Priority Date: 10/19/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Pahud, Michel, Hinckley, Ken, Buxton, William A. S.
• Primary Examiner(s): Laekemariam, Yosef K
• Application Number:
  US12/907,244
• Publication Number:
  US 20120092436A1
• Time in Patent Office: 1,981 Days
• Field of Search: 348/14.01, 348/14.02, 348/14.04, 348/14.05, 348/14.07, 348/14.08, 348/14.09, 348/52, 348/E7.087, 348/14.1
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Telepresence of a mobile user (MU) utilizing a mobile device (MD) and remote users who are participating in a telepresence session is optimized. The MD receives video of a first remote user (FRU). Whenever the MU gestures with the MD using a first motion, video of the FRU is displayed. The MD can also receive video and audio of the FRU and a second remote user (SRU), display a workspace, and reproduce the audio of the FRU and SRU in a default manner. Whenever the MU gestures with the MD using the first motion, video of the FRU is displayed and audio of the FRU and SRU is reproduced in a manner that accentuates the FRU. Whenever the MU gestures with the MD using a second motion, video of the SRU is displayed and audio of the FRU and SRU is reproduced in a manner that accentuates the SRU.

Claims

1. A computer-implemented process for optimizing the telepresence of two or more users participating in a telepresence session, said users comprising a mobile user who is utilizing a mobile device (MD) comprising a display screen, and one or more remote users each of whom are utilizing a computer that is connected to the MD via a network, comprising:

using the MD to perform the following process actions;

receiving live video of a first remote user over the network;

displaying default information on the display screen; and

whenever it is detected by the MD that the mobile user has performed a gesture with the MD using a first motion,

displaying the video of the first remote user on the display screen, instead of the default information, wherein gesture detection comprises detecting physical movement of the MD itself,

receiving gaze information for the first remote user over the network, wherein said information specifies what the first remote user is currently looking at on their display device, and

whenever the gaze information for the first remote user specifies that they are currently looking at either a workspace or a video of another remote user, differentiating the video of the first remote user in a manner that indicates to the mobile user that the first remote user is not looking at the mobile user.

2. The process of claim 1, further comprising the actions of:

receiving live video of a second remote user over the network; and

whenever the default information is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD using a second motion, displaying the video of the second remote user on the display screen, instead of the default information or the video of the first remote user.

3. The process of claim 2, wherein the default information comprises a workspace, further comprising the actions of:

whenever the video of the first remote user is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD using the second motion, displaying the workspace on the display screen, instead of the video of the first remote user or the video of the second remote user; and

whenever the video of the second remote user is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD using the first motion, displaying the workspace on the display screen, instead of the video of the second remote user or the video of the first remote user.

4. The process of claim 3, wherein,

the process action of displaying the video of the first remote user on the display screen and the process action of displaying the video of the second remote user on the display screen both comprise an action of providing haptic feedback to the mobile user, said feedback notifying the mobile user that what is displayed on the display screen has changed, and

the process action of displaying the workspace on the display screen also comprises the action of providing haptic feedback to the mobile user, said feedback notifying the mobile user that what is displayed on the display screen has changed.

5. The process of claim 4, wherein,

the MD further comprises a vibration motor, and

the process action of providing haptic feedback to the mobile user comprises activating the vibration motor for a prescribed period of time.

6. The process of claim 4, wherein the haptic feedback is accompanied by either audio feedback, or video feedback, or both audio and video feedback.

7. The process of claim 2, wherein either,

the first motion comprises a leftward motion and the second motion comprises a rightward motion, or

the first motion comprises an upward motion and the second motion comprises a downward motion.

8. The process of claim 7, wherein either,

the leftward motion comprises the MD being tilted about its left edge and the rightward motion comprises the MD being tilted about its right edge, or

the leftward motion comprises the MD being moved horizontally leftward from its vertical axis and the rightward motion comprises the MD being moved horizontally rightward from its vertical axis.

9. The process of claim 7, wherein either,

the upward motion comprises the MD being tilted about its top edge and the downward motion comprises the MD being tilted about its bottom edge, or

the upward motion comprises the MD being moved vertically upward from its horizontal axis and the downward motion comprises the MD being moved vertically downward from its horizontal axis.

10. The process of claim 2, further comprising an action of transmitting information over the network to the computer of each remote user specifying that the video of the first remote user is displayed upon detection by the MD that the mobile user has performed a gesture with the MD using the first motion, and the video of the second remote user is displayed upon detection by the MD that the mobile user has performed a gesture with the MD using the second motion, wherein,

whenever the computer of a given remote user comprises a single display device said computer utilizes said information to determine where on the display device the video of the first remote user and the video of the second remote user are to be displayed, said determination being made in a manner that maintains an intuitive and consistent spatial interrelationship between the mobile user, first remote user and second remote user, and

whenever the computer of a given remote user comprises a separate display device for each remote user participating in the session said computer utilizes said information to determine on which display device the live video of each remote user is to be displayed, said determination being made in a manner that maintains an intuitive and consistent spatial interrelationship between the mobile user, first remote user and second remote user.

11. The process of claim 2, wherein the MD further comprises a front-facing video capture device that captures live video of the mobile user, further comprising the actions of:

(a) transmitting gaze information over the network to the computer of each remote user, said information specifying what the mobile user is currently looking at on the display screen;

(b) whenever what the mobile user is looking at on the display screen changes, transmitting revised gaze information over the network to the computer of each remote user; and

(c) repeating action (b) until the end of the telepresence session.

12. The process of claim 11, wherein,

whenever the mobile user is currently looking at the video of the first remote user on the display screen, the gaze information comprises the name of the first remote user, and

whenever the mobile user is currently looking at the video of the second remote user on the display screen, the gaze information comprises the name of the second remote user.

13. The process of claim 2, further comprising the actions of:

receiving gaze information for the second remote user over the network, wherein said information specifies what the second remote user is currently looking at on their display device;

whenever the gaze information for the second remote user specifies that they are currently looking at either a workspace or the video of another remote user, differentiating the video of the second remote user in a manner that indicates to the mobile user that the second remote user is not looking at the mobile user.

14. The process of claim 2, wherein,

the first motion comprises a leftward motion and the second motion comprises a rightward motion,

the workspace comprises a personal workspace comprising one or more data objects,

the personal workspace is viewable and manipulatable by just the mobile user, and

the display screen is touch-sensitive, further comprising the actions of:

upon sensing that the mobile user touched a particular data object and dragged it leftward along the display screen, transmitting the particular data object over the network to the computer of the first remote user; and

upon sensing that the mobile user touched a particular data object and dragged it rightward along the display screen, transmitting the particular data object over the network to the computer of the second remote user.

15. The process of claim 1, wherein,

the workspace comprises a shared workspace comprising either one or more data objects, or one or more annotations, or a combination of one or more data objects and one or more annotations, and

the shared workspace is collaboratively viewable, manipulatable and annotatable by the mobile user and remote users.

16. A computer-implemented process for optimizing the telepresence of three or more users participating in a telepresence session, said users comprising a mobile user who is utilizing a mobile device (MD) comprising a display screen and an audio output device, and two or more remote users each of whom are utilizing a computer that is connected to the MD via a network, comprising:

using the MD to perform the following process actions;

receiving live video and live audio of a first remote user over the network;

receiving live video and live audio of a second remote user over the network;

displaying a workspace on the display screen, and reproducing the audio of the first and second remote users via the audio output device in a default manner;

whenever the workspace is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD using a first motion,

displaying the video of the first remote user on the display screen instead of the workspace, and

reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the first remote user; and

whenever the workspace is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD using a second motion,

displaying the video of the second remote user on the display screen instead of the workspace, and

reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the second remote user; and

wherein

gesture detection comprises detecting physical movement of the MD itself.

17. The process of claim 16, wherein,

the audio output device comprises a single audio channel,

the process action of reproducing the audio of the first and second remote users via the audio output device in a default manner comprises the actions of reproducing the audio of the first remote user through said channel at a first volume, and reproducing the audio of the second remote user through said channel at the first volume,

the process action of reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the first remote user comprises the actions of reproducing the audio of the first remote user through said channel at the first volume, and reproducing the audio of the second remote user through said channel at a second volume which is less than the first volume, and

the process action of reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the second remote user comprises the actions of reproducing the audio of the second remote user through said channel at the first volume, and reproducing the audio of the first remote user through said channel at said second volume.

18. The process of claim 16, wherein,

the audio output device comprises a left audio channel and a right audio channel,

the process action of reproducing the audio of the first and second remote users via the audio output device in a default manner comprises the actions of reproducing the audio of the first remote user through just the left audio channel, and reproducing the audio of the second remote user through just the right audio channel,

the process action of reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the first remote user comprises the actions of reproducing the audio of the first remote user through both the left and right audio channels, and reproducing the audio of the second remote user through just the right audio channel, and

the process action of reproducing the audio of the first and second remote users via the audio output device in a manner that accentuates the second remote user comprises the actions of reproducing the audio of the first remote user through just the left audio channel, and reproducing the audio of the second remote user through both the left and right audio channels.

19. The process of claim 18 wherein,

the process action of reproducing the audio of the second remote user through just the right audio channel whenever the first remote user is accentuated comprises an action of reducing the volume of said audio by a prescribed percentage, and

the process action of reproducing the audio of the first remote user through just the left audio channel whenever the second remote user is accentuated comprises an action of reducing the volume of said audio by the prescribed percentage.

20. A computer-implemented process for optimizing the telepresence of three or more users participating in a telepresence session, said users comprising a mobile user who is utilizing a mobile device (MD) comprising a display screen and a front-facing video capture device that captures live video of the mobile user, and two or more remote users each of whom are utilizing a computer that is connected to the MD via a network, comprising:

using the MD to perform the following process actions;

(a) receiving live video of a first remote user over the network;

(b) receiving live video of a second remote user over the network;

(c) displaying a workspace on the display screen;

(d) transmitting gaze information over the network to the computer of each remote user, said information specifying what the mobile user is currently looking at on the display screen;

(e) whenever what the mobile user is looking at on the display screen changes, transmitting revised gaze information over the network to the computer of each remote user;

(f) receiving gaze information for the first remote user over the network, wherein said information specifies what the first remote user is currently looking at on their display device;

(g) receiving gaze information for the second remote user over the network, wherein said information specifies what the second remote user is currently looking at on their display device;

(h) whenever the workspace is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD by tilting the MD about its left edge, displaying the video of the first remote user on the display screen rather than the workspace, wherein whenever the gaze information for the first remote user specifies that they are currently looking at either a workspace or the video of the second remote user, said displayed video is differentiated in a manner that indicates to the mobile user that the first remote user is not looking at the mobile user;

(i) whenever the workspace is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD by tilting the MD about its right edge, displaying the video of the second remote user on the display screen rather than the workspace, wherein whenever the gaze information for the second remote user specifies that they are currently looking at either a workspace or the video of the first remote user, said displayed video is differentiated in a manner that indicates to the mobile user that the second remote user is not looking at the mobile user;

(j) whenever the video of the first remote user is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD by tilting the MD about its right edge, re-displaying the workspace on the display screen, instead of the first remote user or the second remote user;

(k) whenever the video of the second remote user is displayed on the display screen and it is detected by the MD that the mobile user has performed a gesture with the MD by tilting the MD about its left edge, re-displaying the workspace on the display screen, instead of the video of the second remote user or the video of the first remote user; and

(l) repeating actions (e)-(k) until the end of the telepresence session.