Depth camera based image stabilization

• US 10,033,926 B2
• Filed: 11/06/2015
• Issued: 07/24/2018
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Burgess, Gregory M, Carpenter, Thor
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US14/935,092
• Publication Number:
  US 20170134656A1
• Time in Patent Office: 991 Days
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A processing device collects depth data for frames in a sequence of images of a video stream being provided by a source device to a target device as part of a communication session. The depth data is created by a depth aware camera of the source device. The processing device maps, using the depth data, feature locations of the features of an object in a frame to feature locations of the features of the object in other frames, determines overlapping frame sections between the frames using the mapped feature locations, modifies, in the sequence of images, a set of images corresponding to the frames based on the overlapping frame sections to create a stabilized stream of images for the video stream, and provides the stabilized stream of images in the video stream as part of the communication session.

Claims

1. A method comprising:

collecting, by a processing device, depth measurements for pixels of frames in a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a user of the source device and a user of the target device, the depth measurements being created by a depth aware camera of the source device;

mapping, using the depth measurements for the pixels of the frames, feature locations of one or more features of an object in a frame in the sequence of images to feature locations of the one or more features of the object in at least one other frame in the sequence of images;

determining one or more overlapping frame sections between the frame and the at least one other frame using the mapped feature locations;

modifying, in the sequence of images, a set of images corresponding to the frame and the at least another frame based on the overlapping frame sections to create a stabilized stream of images for the video stream; and

providing the stabilized stream of images in the video stream as part of the communication session,

wherein the overlapping sections comprise at least a portion of a person, and modifying the set of images to create the stabilized stream of images comprises;

creating a copy of the frame and the at least one other frame;

replacing a section of the copy of the frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the frame; and

replacing a section of the copy of the at least one other frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the at least one other frame.

2. The method of claim 1, wherein the object comprises at least a portion of a face or a facial feature.

3. The method of claim 1, wherein modifying the set of images comprises:

identifying the person in the images as a foreground object;

identifying one or more objects in the set of images, other than the person, as background objects; and

removing one or more sections of the frames that correspond to the set of images containing the background objects.

4. The method of claim 1, wherein determining the one or more overlapping frame sections comprises:

aligning the frame and the at least one other frame using the mapped feature locations; and

identifying, as the overlapping frame sections, one or more sections in a foreground portion of the frame and one or more sections in a foreground portion of the at least one other frame comprising at least one of same objects or same portions of objects.

5. The method of claim 1, wherein the communication session is a video chat via a mobile device.

6. A system comprising:

a memory; and

a processing device, coupled to the memory, to;

collect depth measurements for pixels of frames in a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a user of the source device and a user of the target device, the depth measurements being created by a depth aware camera of the source device;

map, using the depth measurements for the pixels of the frames, feature locations of one or more features of an object in a frame in the sequence of images to feature locations of the one or more features of the object in at least one other frame in the sequence of images;

determine one or more overlapping frame sections between the frame and the at least one other frame using the mapped feature locations;

modify, in the sequence of images, a set of images corresponding to the frame and the at least another frame based on the overlapping frame sections to create a stabilized stream of images for the video stream; and

provide the stabilized stream of images in the video stream as part of the communication session,

wherein the overlapping sections comprise at least a portion of a person, and to modify the set of images to create the stabilized stream of images, the processing device is to;

create a copy of the frame and the at least one other frame;

replace a section of the copy of the frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the frame; and

replace a section of the copy of the at least one other frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the at least one other frame.

7. The system of claim 6, wherein the object comprises at least a portion of a face or a facial feature.

8. The system of claim 6, wherein to modify the set of images, the processing device is to:

identify the person in the images as a foreground object;

identify one or more objects in the set of images, other than the person, as background objects; and

remove one or more sections of the frames that correspond to the set of images containing the background objects.

9. The system of claim 6, wherein to determine the one or more overlapping frame sections, the processing device is to:

align the frame and the at least one other frame using the mapped feature locations; and

identify, as the overlapping frame sections, one or more sections in a foreground portion of the frame and one or more sections in a foreground portion of the at least one other frame comprising at least one of same objects or same portions of objects.

10. The system of claim 6, wherein the communication session is a video chat via a mobile device.

11. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:

collecting, by the processing device, depth measurements for pixels of frames in a sequence of images of a video stream being provided by a source device to a target device as part of a communication session between a user of the source device and a user of the target device, the depth measurements being created by a depth aware camera of the source device;

mapping, using the depth measurements for the pixels of the frames, feature locations of one or more features of an object in a frame in the sequence of images to feature locations of the one or more features of the object in at least one other frame in the sequence of images;

determining one or more overlapping frame sections between the frame and the at least one other frame using the mapped feature locations;

modifying, in the sequence of images, a set of images corresponding to the frame and the at least another frame based on the overlapping frame sections to create a stabilized stream of images for the video stream; and

providing the stabilized stream of images in the video stream as part of the communication session,

wherein the overlapping sections comprise at least a portion of a person, and modifying the set of images to create the stabilized stream of images comprises;

creating a copy of the frame and the at least one other frame;

replacing a section of the copy of the frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the frame; and

replacing a section of the copy of the at least one other frame that contains the portion of the person with the overlapping frame section without modifying a background portion of the copy of the at least one other frame.

12. The non-transitory computer readable medium of claim 11, wherein the object comprises at least a portion of a face or a facial feature.

13. The non-transitory computer readable medium of claim 11, wherein modifying the set of images comprises:

identifying the person in the images as a foreground object;

identifying one or more objects in the set of images, other than the person, as background objects; and

removing one or more sections of the frames that correspond to the set of images containing the background objects.

14. The non-transitory computer readable medium of claim 11, wherein determining the one or more overlapping frame sections comprises:

aligning the frame and the at least one other frame using the mapped feature locations; and

identifying, as the overlapping frame sections, one or more sections in a foreground portion of the frame and one or more sections in a foreground portion of the at least one other frame comprising at least one of same objects or same portions of objects.