Methods, systems, and products for telepresence visualizations

• US 10,447,945 B2
• Filed: 09/10/2018
• Issued: 10/15/2019
• Est. Priority Date: 10/30/2013
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Oetting, John
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US16/126,077
• Publication Number:
  US 20190007628A1
• Time in Patent Office: 400 Days
• Field of Search: 348 1401- 1406, 709204, 715753
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Methods, systems, and products generate telepresence visualizations for a remote participant to a videoconference. A central server superimposes the remote participant onto images or video of the teleconferencing environment. The central server thus generates an illusion that the remote participant is in the same conferencing environment as other conferees.

Claims

1. A method, comprising:

receiving, by a server, a location identifier for a videoconferencing environment;

retrieving, by the server, a cached point cloud map of sensor data of the videoconferencing environment and a cached image of the videoconferencing environment, the cached point cloud map of sensor data captured by one or more room sensors present in the videoconferencing environment, the cached image of the videoconferencing environment captured by one or more image capture devices within the videoconferencing environment;

receiving, by the server, participant video of a remote participant of the video conference, wherein the remote participant is situated in a location that is physically separate from the videoconferencing environment;

removing, by the server, a background portion of the participant video, wherein removing the background portion of the participant video results in a foreground portion of the participant video of the remote participant;

superimposing the foreground portion of the participant video onto the cached image of the videoconferencing environment using the cached point cloud map of sensor data of the video conferencing environment, wherein the superimposing generates composite video of the remote participant; and

transmitting the composite video for display.

2. The method of claim 1, further comprising:

receiving, by the server, a differential update point cloud map of sensor data of the videoconferencing environment captured by one or more room sensors present in the videoconferencing environment;

wherein superimposing the foreground portion of the participant video onto the image of the videoconferencing environment uses the cached point cloud map of sensor data and the differential update point cloud map of sensor data of the videoconferencing environment.

3. The method of claim 1, wherein the cached point cloud map of sensor data of the videoconferencing environment is retrieved at a first time, and wherein the differential update point cloud map of sensor data of the videoconferencing is received at a second time, and wherein the second time is later than the first time.

4. The method of claim 1, further comprising receiving positional information of a display device in the videoconferencing environment, and wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment behind the display device and sensor data of a portion of the videoconferencing environment behind the display device.

5. The method of claim 1, further comprising:

receiving, by the server, a gaze direction of a videoconferencing participant present in the videoconferencing environment based at least in part on one or more sensors of an augmented reality device of the videoconferencing participant; and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment corresponding to the gaze direction of the videoconferencing participant present in the videoconferencing environment.

6. The method of claim 1, further comprising receiving positional information of a conference participant within the videoconferencing environment based at least in part on one or more sensors of a device of the videoconferencing participant, and wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of the portion of the videoconferencing environment viewed by the conference participant, based on the positional information of the conference participant.

7. The method of claim 1, wherein the one or more room sensors present in the videoconferencing environment include one or more of an optical sensor, an acoustic sensor, a proximity sensor, a light detecting and ranging (LIDAR) sensor, and a photogrammetry sensor.

8. A system for conducting a video conference, comprising:

a room sensor within a videoconferencing environment configured to capture a point cloud map of sensor data of the videoconferencing environment;

an image capture device within the videoconferencing environment;

a display device within the videoconferencing environment;

a naturalizing server, coupled to the room sensor, image capture device, and display device, the naturalizing server configured to receive a locational identifier for the videoconferencing environment and retrieve a cached point cloud map of sensor data of the videoconferencing environment captured by the room sensor and a cached image of the videoconferencing environment captured by the image capture device;

the naturalizing server further configured to receive participant video of a remote participant of the video conference, wherein the remote participant is situated in a location that is physically separate from the videoconferencing environment;

the naturalizing server further configured to process the participant video and the videoconferencing environment data, to remove a background portion of the participant video, wherein removing the background portion of the participant video results in a foreground portion of the participant video of the remote participant, and to superimpose the foreground portion of the participant video onto the cached image of the videoconferencing environment using the cached point cloud map of sensor data of the videoconferencing environment, wherein the superimposing generates composite video of the remote participant; and

the naturalizing server supplying the composite video data for display on the display device.

9. The system of claim 8, wherein the naturalizing server is further configured to:

receive a differential update point cloud map of sensor data of the videoconferencing environment captured by one or more room sensors present in the videoconferencing environment;

wherein superimposing the foreground portion of the participant video onto the image of the videoconferencing environment uses the cached point cloud map of sensor data and the differential update point cloud map of sensor data of the videoconferencing environment.

10. The system of claim 8, wherein the cached point cloud map of sensor data of the videoconferencing environment is retrieved at a first time, and wherein the differential update point cloud map of sensor data of the videoconferencing is received at a second time, and wherein the second time is later than the first time.

11. The system of claim 8, wherein the naturalizing server is further coupled to receive positional information of a display device in the videoconferencing environment, and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment behind the display device and sensor data of a portion of the videoconferencing environment behind the display device.

12. The system of claim 8, wherein the naturalizing server is further configured to receive a gaze direction of a videoconferencing participant present in the videoconferencing environment from one or more sensors of an augmented reality device of the videoconferencing participant;

and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment corresponding to the gaze direction of the videoconferencing participant present in the videoconferencing environment.

13. The system of claim 8, wherein the one or more sensors present in the videoconferencing environment include one or more of an optical sensor, an acoustic sensor, a proximity sensor, a light detecting and ranging (LIDAR) sensor, and a photogrammetry sensor.

14. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor to perform operations, comprising:

receiving a location identifier for a videoconferencing environment;

retrieving a cached point cloud map of sensor data of the videoconferencing environment and a cached image of the videoconferencing environment, the cached point cloud map of sensor data captured by one or more room sensors present in the videoconferencing environment, the cached image of the videoconferencing environment captured by one or more image capture devices within the videoconferencing environment;

receiving participant video of a remote participant of the video conference, wherein the remote participant is situated in a location that is physically separate from the videoconferencing environment;

removing a background portion of the participant video, wherein removing the background portion of the participant video results in a foreground portion of the participant video of the remote participant;

superimposing the foreground portion of the participant video onto the cached image of the videoconferencing environment using the cached point cloud map of sensor data of the video conferencing environment, wherein the superimposing generates composite video of the remote participant; and

transmitting the composite video for display.

15. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise:

receiving a differential update point cloud map of sensor data of the videoconferencing environment captured by one or more room sensors present in the videoconferencing environment;

wherein superimposing the foreground portion of the participant video onto the image of the videoconferencing environment uses the cached point cloud map of sensor data and the differential update point cloud map of sensor data of the videoconferencing environment.

16. The non-transitory computer-readable medium of claim 14, wherein the cached point cloud map of sensor data of the videoconferencing environment is retrieved at a first time, and wherein the differential update point cloud map of sensor data of the videoconferencing is received at a second time, and wherein the second time is later than the first time.

17. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise:

receiving a gaze direction of a videoconferencing participant present in the videoconferencing environment based at least in part on one or more sensors of an augmented reality device of the videoconferencing participant; and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment corresponding to the gaze direction of the videoconferencing participant present in the videoconferencing environment.

18. The non-transitory computer-readable medium of claim 14, wherein the one or more room sensors present in the videoconferencing environment include one or more of an optical sensor, an acoustic sensor, a proximity sensor, a light detecting and ranging (LIDAR) sensor, and a photogrammetry sensor.

19. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise:

receiving positional information of a display device in the videoconferencing environment, and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of a portion of the videoconferencing environment behind the display device and sensor data of a portion of the videoconferencing environment behind the display device.

20. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise:

receiving positional information of a conference participant within the videoconferencing environment based at least in part on one or more sensors of a device of the videoconferencing participant, and

wherein the superimposing the foreground portion of the participant video onto the image of the videoconferencing environment utilizes an image of the portion of the videoconferencing environment viewed by the conference participant, based on the positional information of the conference participant.