TELEPRESENCE DEVICES OPERATION METHODS

• US 20190320142A1
• Filed: 04/17/2018
• Published: 10/17/2019
• Est. Priority Date: 04/17/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: CUTLER, Ross Garrett
• Granted Patent: US 11,089,265 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A method of operating a telepresence device is disclosed, which includes capturing images of a local subject using a plurality of cameras to generate a plurality of local captured images. Remote subject point of view (POV) data is received from a remote device. A plurality of tiles is selected from the plurality of local captured images based on the remote subject POV data. Local image data is sent to the remote device. The local image data includes the plurality of selected tiles.

Claims

1. A method of operating a telepresence device, comprising:

capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;

receiving remote subject point of view (POV) data from a remote device via a communication network, the remote subject POV data comprising an estimated future POV of a remote subject;

selecting a plurality of tiles from the plurality of local subject captured images based on the received remote subject POV data;

generating local image data comprising the plurality of selected tiles; and

sending the local image data to the remote device via the communication channel.

2. The method of claim 1, wherein capturing the local subject comprises capturing, using the plurality of cameras, the local subject through a display.

3. The method of claim 1, wherein the plurality of cameras comprises a light field camera.

4. The method of claim 1, further comprising estimating a depth between the plurality of cameras and the local subject.

5. The method of claim 4, further comprising adding depth information to the local image data, the depth information comprising the estimated depth.

6. The method of claim 1, wherein the remote subject POV data further comprises a previous or current POV of the remote subject.

7. The method of claim 1, further comprising:

tracking a POV of the local subject to generate local subject POV data; and

sending the local subject POV data to the remote device via the communication network.

8. The method of claim 7, wherein the local subject POV data comprises at least one of previous, current and estimated future POVs of the local subject.

9. A method of operating a telepresence device, comprising:

tracking a point of view (POV) of a local subject;

generating local subject POV data based on the POV of the local subject, the local subject POV data comprising an estimated future POV of the local subject;

receiving remote image data from a remote device via a communication network, the remote image data comprising a plurality of remote tiles;

selecting, based on the local subject POV data, a plurality of tiles from the plurality of remote tiles;

compositing an image of a remote subject based on the plurality of selected tiles to produce a composited image; and

displaying, on a display, the image of the composited image of the remote subject.

10. The method of claim 9, wherein capturing the local subject comprises capturing, using a plurality of cameras, the local subject through the display.

11. The method of claim 9, wherein the plurality of cameras comprises a light field camera.

12. The method of claim 9, wherein the local subject POV data further comprises a previous or current POV of the local subject.

13. The method of claim 9, further comprising sending the local subject POV data to the remote device via the communication network, wherein the remote device selects the plurality of remote tiles the local subject POV data.

14. The method of claim 13, wherein the plurality of remote tiles are selected based on the estimated future POV of the local subject.

15. The method of claim 9, wherein the remote image data comprises depth information of the remote subject, the depth information comprising an estimated depth between the remote subject and the remote device.

16. A method of operating telepresence devices, comprising:

capturing an image of a first subject located at a first location, using a plurality of first cameras of a first telepresence device, to generate a plurality of first captured images of the first subject;

capturing an image of a second subject located at a second location, using a plurality of second cameras of a second telepresence device, to generate a plurality of second captured images of the second subject;

processing, at the first telepresence device, the plurality of first captured images to generate first subject point of view (POV) data, the first POV data comprising an estimated future POV of the first subject;

processing, at the second telepresence device, the plurality of second captured images to generate second subject POV data, the second POV data comprising an estimated future POV of the second subject;

sending the first subject POV data from the first telepresence device to the second telepresence device;

sending the second subject POV data from the second telepresence device to the first telepresence device;

selecting, at the first telepresence device, a plurality of first tiles from the plurality of first captured images based on the second subject POV data;

sending first image data from the first telepresence device to the second telepresence device, the first image data comprising the plurality of first tiles;

sending second image data from the second telepresence device to the first telepresence device, the second image data comprising the plurality of second tiles;

selecting, at the first telepresence device, a plurality of third tiles from the plurality of second tiles based on the first subject POV data; and

displaying, on a first display of the first telepresence device, an image of the second subject composited based on the plurality of third tiles.

17. The method of claim 16, wherein the plurality of first and second cameras comprise a light field camera.

18. The method of claim 16, further comprising:

selecting, at the second telepresence device, a plurality of fourth tiles from the plurality of first tiles based on the first subject POV data; and

displaying, on a second display of the second telepresence device, an image of the first subject composited based on the plurality of fourth tiles.

19. The method of claim 18, wherein the plurality of first cameras and second cameras capture the images of the first and second subjects through the first and second displays, respectively.

20. The method of claim 18, wherein the first subject POV data further comprises a previous or current and estimated future POV of the first subject, and the second subject POV data further comprises a previous or current and estimated future POV of the second subject.

21. The method of claim 16, further comprising estimating a first depth between the first subject and a plurality of first cameras,

wherein the first tiles first image data comprises the estimated first depth.

22. The method of claim 16, further comprising estimating a second depth between the second subject and a plurality of second cameras,

wherein the second image data comprises the estimated second depth.

23. A non-transient computer readable medium comprising instructions for causing a computer to perform functions of:

capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;

receiving remote subject point of view (POV) data from a remote device via a communication network, the remote subject POV data comprising an estimated future POV of a remote subject;

selecting a plurality of first tiles from the plurality of local subject captured images based on the received remote subject POV data; and

sending local image data to the remote device, the local image data comprising the plurality of selected first tiles.

24. The non-transient computer readable medium of claim 23, wherein the plurality of cameras are arranged to capture the local subject through a display.

25. The non-transient computer readable medium of claim 23, wherein the plurality of cameras comprises a light field camera.

26. The non-transient computer readable medium of claim 23, further comprising instructions for causing the computer to perform a function of estimating a depth between the plurality of cameras and the local subject.

27. The non-transient computer readable medium of claim 26, further comprising instructions for causing the computer to perform functions of:

estimating a depth between the local subject and the plurality of cameras; and

adding depth information to the local image data, the depth information comprising the estimated depth.

28. The non-transient computer readable medium of claim 23, wherein the remote subject POV data further comprises a previous or current of the remote subject.

29. The non-transient computer readable medium of claim 23, further comprising instructions for causing the computer to perform functions of:

generating local subject POV data;

sending the local subject POV data to the remote device;

receiving remote image data from the remote device, the remote image data comprising a plurality of second tiles selected at the remote device based on the local subject POV data;

selecting a plurality of third tiles from the plurality of second tiles based on the local subject POV data; and

displaying, on a display, the image of the remote subject composited based on the plurality of third tiles.

30. The non-transient computer readable medium of claim 29, wherein the local subject POV data comprises at least one of previous, current and estimated future POVs of the local subject.

31. A method comprising:

capturing an image of a local subject using a plurality of cameras to generate a plurality of local subject captured images;

tracking, using the plurality of cameras, a point of view (POV) of the local subject;

generating local subject POV data based on the POV of the local subject, the local subject POV data comprising an estimated future POV of the local subject;

selecting a plurality of tiles from the local subject captured image; and

converting the plurality of selected tiles to local subject image data that is usable by a remote device to render and display a 3D visual representation of the local subject,

wherein the local subject image data comprises the local subject POV data that is usable by the remote device to maintain eye gazing between a remote device user and the 3D visual representation of the local subject.

32. The method of claim 31, wherein the 3D visual representation is a 3D avatar of the local subject.