Depth camera based image stabilization

• US 10,574,892 B2
• Filed: 06/18/2018
• Issued: 02/25/2020
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Burgess, Gregory M., Carpenter, Thor
• Primary Examiner(s): Vu, Ngoc Yen T
• Application Number:
  US16/011,270
• Publication Number:
  US 20180302565A1
• Time in Patent Office: 617 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A processing device collects depth data for frames in a sequence of images of a video stream being provided by a source device to a target device as part of a communication session. The depth data is created by a depth aware camera of the source device. The processing device maps, using the depth data, feature locations of the features of an object in a frame to feature locations of the features of the object in other frames, determines overlapping frame sections between the frames using the mapped feature locations, modifies, in the sequence of images, a set of images corresponding to the frames based on the overlapping frame sections to create a stabilized stream of images for the video stream, and provides the stabilized stream of images in the video stream as part of the communication session.

Claims

1. A method comprising:

collecting, by a processing device, depth data related to a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a first user of the source device and a second user of the target device, wherein the depth data includes a depth measurement of individual pixels in each image of the sequence of images;

creating x, y, and z coordinates for features of an object in the sequence of images of the video stream based on the depth data;

creating a stabilized stream of images for the video stream based on the x, y, and z coordinates for the features of the object in the sequence of images of the video stream by;

building a first model of the object in a first image of the sequence of images based on the depth data;

building a second model of the object in a second image of the sequence of images based on the depth data; and

generating the stabilized stream of images based on the first model of the object and the second model of the object, wherein a first foreground portion of the first image corresponds to a second foreground portion of the second image, and wherein the stabilized stream of images comprises an overlapping frame section that includes the object, the overlapping frame section overlapping between the first foreground portion and the second foreground portion; and

providing, to the target device, the stabilized stream of images in the video stream as part of the communication session.

2. The method of claim 1, wherein the object in the overlapping frame section between the first foreground portion and the second foreground portion includes at least one of the first user or the second user.

3. The method of claim 1, wherein creating the stabilized stream of images includes cropping the overlapping frame section to create corresponding new images.

4. The method of claim 3, wherein creating the x, y, and z coordinates for the features of the object includes building a three-dimensional point cloud based on the depth data.

5. The method of claim 1, wherein creating the stabilized stream of images based on the first model of the object and the second model of the object comprises:

extracting at least one first feature location of a feature of the object based on the first model of the object;

extracting at least one second feature location of the feature of the object based on the second model of the object; and

aligning the first image to the second image based on the first feature location and the second feature location.

6. The method of claim 5, wherein creating the stabilized stream of images based on the first model of the object and the second model of the object further comprises:

identifying, based on the alignment, a first section in the first foreground portion of the first image and a second section in the second foreground portion of the second image; and

identifying the overlapping frame section between the first foreground portion and the second foreground portion based on the first section in the first foreground portion of the first image and the second section in the second foreground portion of the second image.

7. The method of claim 1, further comprising:

creating a copy of the first image and a copy of the second image;

replacing a section of the copy of the first image that contains a portion of a person with the overlapping frame section without modifying a background portion of the copy of the first image; and

replacing a section of the copy of the second image that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the second image.

8. A system comprising:

a memory; and

a processing device, coupled to the memory, the processing device configured to;

collect depth data related to a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a first user of the source device and a second user of the target device, wherein the depth data includes a depth measurement of individual pixels in each image of the sequence of images, the depth measurement being indicative of a distance of an object from a camera that captured the sequence of images of the video;

create x, y, and z coordinates for features of the object in the sequence of images of the video stream based on the depth data;

create a stabilized stream of images for the video stream based on the x, y, and z coordinates for the features of the object in the sequence of images of the video stream by;

building a first model of the object in a first image of the sequence of images based on the depth data;

building a second model of the object in a second image of the sequence of images based on the depth data; and

generating the stabilized stream of images based on the first model of the object and the second model of the object, wherein a first foreground portion of the first image corresponds to a second foreground portion of the second image, and wherein the stabilized stream of images comprises an overlapping frame section that includes the object, the overlapping frame section overlapping between the first foreground portion and the second foreground portion; and

provide, to the target device, the stabilized stream of images in the video stream as part of the communication session.

9. The system of claim 8, wherein the object in the overlapping frame section between the first foreground portion and the second foreground portion includes at least one of the first user or the second user.

10. The system of claim 8, wherein creating the stabilized stream of images includes cropping the overlapping frame section to create corresponding new images.

11. The system of claim 10, wherein creating the x, y, and z coordinates for the features of the object includes building a three-dimensional point cloud based on the depth data.

12. The system of claim 8, wherein, to create the stabilized stream of images based on the first model of the object and the second model of the object, the processing device is further configured to:

extract at least one first feature location of a feature of the object based on the first model of the object;

extract at least one second feature location of the feature of the object based on the second model of the object; and

align the first image to the second image based on the first feature location and the second feature location.

13. The system of claim 12, wherein, to create the stabilized stream of images based on the first model of the object and the second model of the object, the processing device is further configured to:

identify a first section in the first foreground portion of the first image and a second section in the second foreground portion of the second image; and

identify the overlapping frame section between the first foreground portion and the second foreground portion based on the first section in the first foreground portion of the first image and the second section in the second foreground portion of the second image.

14. The system of claim 8, wherein the processing device is further configured to:

create a copy of the first image and a copy of the second image;

replace a section of the copy of the first image that contains a portion of a person with the overlapping frame section without modifying a background portion of the copy of the first image; and

replace a section of the copy of the second image that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the second image.

15. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:

collect depth data related to a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a first user of the source device and a second user of the target device, wherein the depth data includes a depth measurement of individual pixels in each image of the sequence of images;

create x, y, and z coordinates for features of an object in the sequence of images of the video stream based on the depth data;

create a stabilized stream of images for the video stream based on the x, y, and z coordinates for the features of the object in the sequence of images of the video stream by;

building a first model of the object in a first image of the sequence of images based on the depth data;

building a second model of the object in a second image of the sequence of images based on the depth data; and

generating the stabilized stream of images based on the first model of the object and the second model of the object, wherein a first foreground portion of the first image corresponds to a second foreground portion of the second image, and wherein the stabilized stream of images comprises an overlapping frame section that includes the object, the overlapping frame section overlapping between the first foreground portion and the second foreground portion; and

provide, to the target device, the stabilized stream of images in the video stream as part of the communication session.

16. The non-transitory computer readable medium of claim 15, wherein the object in the overlapping frame section between the first foreground portion and the second foreground portion includes at least one of the first user or the second user.

17. The non-transitory computer readable medium of claim 15, wherein creating the stabilized stream of images includes cropping the overlapping frame section to create corresponding new images.

18. The non-transitory computer readable medium of claim 17, wherein creating the x, y, and z coordinates for the features of the object includes building a three-dimensional point cloud based on the depth data.

19. The non-transitory computer readable medium of claim 15, wherein the instructions to create the stabilized stream of images based on the first model of the object and the second model of the object, cause the processing device to:

extract at least one first feature location of a feature of the object based on the first model of the object;

extract at least one second feature location of the feature of the object based on the second model of the object; and

align the first image to the second image based on the first feature location and the second feature location.

20. The non-transitory computer readable medium of claim 19, wherein the instructions to create the stabilized stream of images based on the first model of the object and the second model of the object, cause the processing device to:

identify a first section in the first foreground portion of the first image and a second section in the second foreground portion of the second image; and

identify the overlapping frame section between the first foreground portion and the second foreground portion based on the first section in the first foreground portion of the first image and the second section in the second foreground portion of the second image.