Telepresence experience

• US 9,883,138 B2
• Filed: 07/16/2014
• Issued: 01/30/2018
• Est. Priority Date: 02/26/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chen, Yinpeng, Chou, Philip A., Zhang, Zhengyou
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US14/333,437
• Publication Number:
  US 20150244976A1
• Time in Patent Office: 1,294 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-357, 37920201-20701, 709201-207, 709217-248
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 9/00 Details of colour television systems

Abstract

The description relates to remote collaboration via a telepresence experience. One example can include an interactive digital display. The example can also include a virtual user presentation component configured to generate a graphical user interface that includes a virtual representation of a remote user on the interactive digital display. The graphical user interface can be configured to present the remote user in a side by side or mirror image relationship to a local user of the interactive digital display.

Claims

1. A system, comprising:

a touch sensitive display;

a processor; and

storage having computer-readable instructions which, when executed by the processor, cause the processor to;

obtain video of a local user from a camera that is configured to capture the local user in a region in front of the touch sensitive display;

render a tilted view of a local drawing surface on a graphical user interface (GUI) on the touch sensitive display, the tilted view showing the local drawing surface as tilted at an angle away from the local user;

temporarily rectify the local drawing surface to provide a rectified view and receive inputs from the local user while the local drawing surface is shown in the rectified view;

after the inputs are received, show a modified local drawing surface in the tilted view, the modified local drawing surface reflecting the inputs received from the local user;

send display content reflecting the video of the local user and the modified local drawing surface to a remote device;

receive other display content from the remote device reflecting interaction by a remote user with the remote device;

and

using the other display content, update the GUI to include a virtual representation of the remote user of the remote device interacting with the modified local drawing surface.

2. The system of claim 1, wherein the computer-readable instructions, when executed by the processor, cause the processor to:

cause the video of the local user to be captured;

separate a portion of the video containing the local user from background portions of the video; and

include the portion of the video containing the local user in the display content sent to the remote device.

3. The system of claim 1, wherein the system includes the remote device.

4. The system of claim 1, embodied as a single computing device comprising the touch sensitive display, the processor, and the storage.

5. The system of claim 1, further comprising the camera, wherein the camera is a red, green, blue, +depth (RGBD) camera positionable to capture the region in front of the touch sensitive display.

6. The system of claim 5, wherein the RGBD camera is fixedly mounted to the touch sensitive display.

7. The system of claim 1, wherein the GUI comprises a first portion configured to present the virtual representation of the remote user and another portion configured to present the local drawing surface.

8. The system of claim 1, wherein the computer-readable instructions, when executed by the processor, cause the processor to:

detect that the local user has moved a hand close to the touch sensitive display; and

temporarily rectify the local drawing surface responsive to detecting that the local user has moved the hand close to the touch sensitive display.

9. The system of claim 8, wherein the computer-readable instructions, when executed by the processor, cause the processor to:

after the local drawing surface has been rectified, detect that the local user has withdrawn their hand away from the touch sensitive display; and

return the local drawing surface to the tilted view responsive to detecting that the local user has withdrawn their hand away from the touch sensitive display.

10. The system of claim 1, wherein the local drawing surface is a rectangle that is shown as tilted between 10 and 45 degrees away from the local user.

11. A system, comprising:

a processor;

a touch sensitive display; and

storage having computer-executable instructions which, when executed by the processor, cause the processor to;

obtain red, green, blue, plus depth (RGBD) data from an RGBD camera oriented to sense a local user in a region in front of the touch sensitive display;

render a tilted view of a local drawing surface on the touch sensitive display, the local drawing surface being tilted at an angle away from the local user when shown in the tilted view;

detect user input to the touch sensitive display and, in response, rectify the local drawing surface to provide a rectified view that is in line with the touch sensitive display;

update the local drawing surface displayed on the touch sensitive display to reflect changes by the local user via the user input;

return the local drawing surface to the tilted view;

and

transmit the changes made by the local user to a remote system.

12. The system of claim 11, further comprising the RGBD camera.

13. The system of claim 12, wherein the RGBD camera is at an orientation that is neither horizontal nor vertical relative to the touch sensitive display.

14. The system of claim 12, wherein the RGBD camera comprises a first RGBD camera that is oriented generally parallel to a plane defined by the touch sensitive display and a second RGBD camera that is oriented to capture the region in front of the plane and wherein the first and second RGBD cameras and the touch sensitive display are mounted in a housing.

15. The system of claim 14, manifest as a mobile device or a device that is configured to be mounted on a vertical wall.

16. The system of claim 11, wherein the computer-executable instructions, when executed by the processor, cause the processor to:

separate a first portion of the RGBD data relating to the local user in the region from a second portion of background data; and

transmit the first portion without the second portion to the remote system.

17. A method comprising:

obtaining video data from a camera oriented to sense a local user in a region in front of a local touch sensitive display;

presenting a tilted view of a local drawing surface on the local touch sensitive display, the local drawing surface being tilted at an angle away from the local user when shown in the tilted view;

detecting user input by the local user to the local touch sensitive display and, in response, rectifying the local drawing surface to provide a rectified view that shows the local drawing surface in line with the local touch sensitive display;

updating the local drawing surface presented on the local touch sensitive display to reflect changes by the local user on the local touch sensitive display via the user input;

returning the local drawing surface to the tilted view; and

transmitting the changes by the local user to a remote display.

18. The method of claim 17, wherein the local drawing surface is a rectangle that is shown as tilted between 10 and 45 degrees away from the local user when shown in the tilted view.

19. The method of claim 17, wherein the video data comprises depth data.

20. The method of claim 17, performed by a server connected via a network to both the local touch sensitive display and the remote display.