Telepresence device

• US 10,554,928 B2
• Filed: 04/17/2018
• Issued: 02/04/2020
• Est. Priority Date: 04/17/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cutler, Ross Garrett
• Primary Examiner(s): El-Zoobi, Maria
• Application Number:
  US15/955,669
• Publication Number:
  US 20190320135A1
• Time in Patent Office: 658 Days
• Field of Search: 348 1401- 1409
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A telepresence device is disclosed, which includes a display and a camera array including cameras configured to capture images of a local subject through the display and output a plurality of local captured images. The telepresence device also includes a local tile selector configured to select a plurality of tiles from the plurality of local captured images based on remote subject point of view (POV) data received from a remote device. The local tile selector outputs local image data that comprises the plurality of selected tiles. The local image data is sent to the remote device to composite an image of the local subject.

Claims

1. A telepresence device comprising:

a display;

a camera array comprising a plurality of cameras;

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of;

capturing, using the camera array, a local subject to generate a plurality of local subject capture images;

receiving, via a communication network from a remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;

based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data; and

sending, via the communication network to the remote device, local image data comprising the plurality of candidate local tiles.

2. The telepresence device of claim 1, wherein the plurality of cameras comprises a light field camera.

3. The telepresence device of claim 1, wherein the plurality of cameras comprises a depth camera configured to determine a depth estimate between the camera array and the local subject.

4. The telepresence device of claim 1, wherein instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of estimating a depth between the camera array and the local subject.

5. The telepresence device of claim 1, wherein the remote subject POV data further comprises a previous or current POV of the remote user.

6. The telepresence device of claim 1, wherein the camera array comprises a camera module including the camera and an image processor.

7. The telepresence device of claim 1, wherein instructions, when executed by the processor, further cause the processor to control the telepresence device to perform functions of:

tracking a POV of the local subject; and

sending local subject POV data to the remote device via the communication network, the local subject POV data comprising at least one of previous, current and estimated future POVs of the local subject.

26. The telepresence device of claim 1, wherein, for selecting the plurality of candidate local tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject.

8. A telepresence device comprising:

a display;

a camera array comprising a plurality of cameras;

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of;

tracking, using the camera array, a point of view (POV) of a local subject;

generating, based on the POV of the local subject, local subject POV data comprising an estimated future POV of the local subject;

sending, via a communication network to a remote device, the local subject POV data;

receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based the local subject POV data;

based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;

compositing an image of a remote subject based on the plurality of selected remote tiles; and

displaying, on the display, the composited image of the remote subject.

9. The telepresence device of claim 8, wherein the plurality of cameras comprises a light field camera.

10. The telepresence device of claim 8, wherein the camera array comprises a camera module including the camera and an image sensor.

11. The telepresence device of claim 8, wherein the local subject POV data further comprises a previous or current POV of the local subject.

12. The telepresence device of claim 8, wherein the remote image data includes a depth estimate between the remote device and the remote subject.

27. The telepresence device of claim 8, wherein:

the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and

for selecting the plurality of remote tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.

13. A telepresence device comprising:

a display;

a camera array comprising a plurality of cameras;

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to control the telepresence device to perform functions of;

capturing, using the camera array, a local subject to generate a plurality of local subject capture images;

receiving, via a communication network form the remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;

based on the remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data;

sending, via a communication network to the remote device, local image data comprising the plurality of candidate local tiles;

tracking a POV of the local subject and generating local subject POV data;

sending, via the communication network to the remote device, the local POV data comprising an estimate future POV of the local subject;

receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based on the local POV data;

based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;

compositing an image of the remote subject based on the plurality of selected remote tiles; and

displaying the composited image of the remote subject on the display.

14. The telepresence device of claim 13, wherein the plurality of cameras comprises a light field camera.

15. The telepresence device of claim 13, wherein the plurality of cameras includes a depth camera configured to determine a depth estimate between the camera array and the local subject.

16. The telepresence device of claim 13, wherein the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of estimating a depth between the camera array and the local subject, wherein the local image data comprises the estimated depth.

17. The telepresence device of claim 13, wherein the camera array comprises a camera module including the camera and an image processor.

18. The telepresence device of claim 13, wherein:

the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of sending, via the communication network to the remote device, the local POV data, and

the plurality of candidate remote tiles is selected, based on the local POV data, by the remote device based on the local POV data for further selection by the telepresence device.

28. The telepresence device of claim 13, wherein:

for selecting the plurality of candidate local tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject,

the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and

for selecting the plurality of remote tiles, the instructions, when executed by the processor, further cause the processor to control the telepresence device to perform a function of selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.

19. A method of operating a telepresence device comprising a camera array, the method comprising:

capturing, using the camera array, a local subject to generate a plurality of local subject capture images;

receiving, via a communication network from a remote device, remote subject point of view (POV) data comprising an estimated future POV of a remote subject;

based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data; and

sending, via the communication network to the remote device, local image data comprising the plurality of candidate local tiles.

20. The method of claim 19, further comprising estimating a depth between the camera array and the local subject.

21. The method of claim 20, wherein the local image data further comprises the estimated depth between the camera array and the local subject.

22. The method of claim 19, further comprising:

tracking a POV of the local subject; and

sending local subject POV data to the remote device, the local subject POV data comprising an estimated future POV of the local subject.

29. The method of claim 19, wherein selecting the plurality of candidate local tiles comprises selecting, based a first estimate future POV of the remote subject, the plurality of candidate local tiles for further selection by the remote device based on a second estimate future POV of the remote subject.

23. A method of operating a telepresence device comprising a display and camera array, the method comprising:

tracking, using the camera array, a point of view (POV) of a local subject;

generating, based on the POV of the local subject, local subject POV data comprising an estimated future POV of the local subject;

sending, via a communication network to a remote device, the local subject POV data;

receiving, via the communication network from the remote device, remote image data comprising a plurality of candidate remote tiles selected by the remote device based the local subject POV data;

based on the local subject POV data, selecting, from the plurality of candidate remote tiles, a plurality of remote tiles;

compositing an image of a remote subject based on the plurality of remote tiles; and

displaying, on the display, the composited image of the remote subject.

24. The method of claim 23, wherein the remote image data includes a depth estimate between the remote device and the remote subject.

25. The method of claim 23, further comprising:

capturing, using the camera array, the local subject to generate a plurality of local subject capture images;

receiving, via the communication network from the remote device, remote subject POV data comprising an estimated future POV of a remote subject;

based on the received remote subject POV data, selecting, from the plurality of local subject capture images, a plurality of candidate local tiles; and

sending, via the communication network to the remove device, local image data comprising the plurality of candidate local tiles for further selection by the remote device based on the remote subject POV data.

30. The method of claim 23, wherein the received remote image data comprises the plurality of candidate remote tiles selected by the remote device based on a first estimated future POV of the local subject, and

selecting the plurality of remote tiles comprises selecting, from the plurality of candidate remote tiles, the plurality of remote tiled based on a second estimated future POV of the local subject.