Dynamically adjusting sampling of a real-time depth map

• US 10,609,355 B2
• Filed: 10/27/2017
• Issued: 03/31/2020
• Est. Priority Date: 10/27/2017
• Status: Active Grant

• Current Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Chen, Yin Hu, Marchevsky, Valeriy, Xu, Susan Yanqing
• Primary Examiner(s): Harvey, David E
• Application Number:
  US15/796,425
• Publication Number:
  US 20190132570A1
• Time in Patent Office: 886 Days
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules

Abstract

A method, a system, and a computer program product for dynamically adjusting sampling of a depth map based on detected motion in a current scene. The method includes capturing real-time scan data at a first resolution by a first camera and a second camera of an image capturing device. The method further includes synchronizing a first plurality of frames of the real-time scan data to create a plurality of synchronized frames at a first frame rate. The method further includes analyzing the synchronized frames to determine whether motion exists. The method further includes, in response to determining motion exists: determining, based on the plurality of synchronized frames, a rate of motion within the current scene; and dynamically calculating a target resolution and a target frame rate for a real-time depth map. The method further includes generating a real-time depth map at the target resolution and target frame rate.

Claims

1. A method comprising:

capturing real-time scan data by a first camera and a second camera of an image capturing device;

synchronizing, by a processor, a first plurality of frames of the real-time scan data captured by the first camera with a second plurality of frames captured by the second camera to create a plurality of synchronized frames at a first frame rate;

analyzing the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene;

determining, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculating a target resolution and a target frame rate based on the rate of motion; and

generating, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to the analysis of the plurality of synchronized frames indicating that the current scene is a static scene, combining, by the processor, the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and a full resolution of at least one of the first camera and the second camera.

2. The method of claim 1, wherein calculating the target resolution and the target frame rate further comprises:

selecting, based on the determined rate of motion, a second resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution; and

selecting, based on the determined rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

3. The method of claim 2, wherein generating the real-time depth map further comprises:

down-sampling the real-time scan data to the second resolution based on the determined rate of motion; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

4. The method of claim 1, further comprising:

in response to determining motion exists within the current scene;

determining whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range;

selecting, from a database, a high-motion resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution;

selecting, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-sampling the real-time scan data to the high-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

5. The method of claim 4, further comprising:

in response to determining motion exists within the current scene;

determining whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range;

selecting, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selecting, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-sampling the real-time scan data to the low-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

6. The method of claim 1, wherein dynamically calculating the target resolution and the target frame rate further comprises:

calculating, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculating a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishing the minimum frame rate as the target frame rate; and

establishing the maximum resolution as the target resolution.

7. An image capturing device comprising:

a first camera that captures a first scan data of a current scene at a full resolution;

a second camera that captures a second scan data of the current scene at the full resolution; and

at least one processor that;

synchronizes the first scan data and the second scan data to create a plurality of synchronized frames at a first frame rate;

analyzes the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene;

determines, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculates a target resolution and a target frame rate based on the rate of motion; and

generates, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to determining that the analysis of the plurality of synchronized frames indicates that the current scene is a static scene, combines the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and the full resolution.

8. The image capturing device of claim 7, wherein in calculating the target resolution and the target frame rate, the at least one processor:

selects, based on the rate of motion, a second resolution that is lower than the full resolution as the target resolution; and

selects, based on the rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

9. The image capturing device of claim 8, wherein in generating the real-time depth map, the at least one processor:

down-samples the real-time scan data to the second resolution based on the determined rate of motion; and

combines the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

10. The image capturing device of claim 7, wherein in response to determining motion exists within the current scene, the at least one processor:

determines whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range;

selects, from a database, a high-motion resolution that is lower than the full resolution as the target resolution;

selects, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-samples the real-time scan data to the high-motion resolution; and

combines the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

11. The image capturing device of claim 10, wherein in response to determining motion exists within the current scene, the at least one processor:

determines whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range;

selects, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selects, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-samples the real-time scan data to the low-motion resolution; and

combines, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

12. The image capturing device of claim 7, wherein in calculating the target resolution and the target frame rate, the processor:

calculates, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculates a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishes the minimum frame rate as the target frame rate; and

establishes the maximum resolution as the target resolution.

13. A computer program product comprising:

a computer readable storage device; and

program code on the computer readable storage device that, when executed by a processor associated with an image capturing device, configures the image capturing device to provide the functionality of;

capturing real-time scan data at a full resolution by a first camera and a second camera of an image capturing device;

synchronizing, by a processor, a first plurality of frames of the real-time scan data captured by the first camera with a second plurality of frames captured by the second camera to create a plurality of synchronized frames at a first frame rate;

analyzing the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene;

determining, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculating a target resolution and a target frame rate based on the rate of motion; and

generating, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to the analysis of the plurality of synchronized frames indicating that the current scene is a static scene, combining the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and the full resolution.

14. The computer program product of claim 13, the program code for calculating the target resolution and the target frame rate further comprising code for:

selecting, based on the determined rate of motion, a second resolution that is lower than the full resolution as the target resolution; and

selecting, based on the determined rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

15. The computer program product of claim 14, the program code for generating the real-time depth map further comprising code for:

down-sampling the real-time scan data to the second resolution based on the determined rate of motion; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

16. The computer program product of claim 13, the program code further comprising code for:

in response to determining motion exists within the current scene;

determining whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range;

selecting, from a database, a high-motion resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution;

selecting, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-sampling the real-time scan data to the high-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

17. The computer program product of claim 16, the program code further comprising code for:

in response to determining motion exists within the current scene;

determining whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range;

selecting, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selecting, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-sampling the real-time scan data to the low-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

18. The computer program product of claim 13, the program code for dynamically calculating the target resolution and the target frame rate further comprising code for:

calculating, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculating a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishing the minimum frame rate as the target frame rate; and

establishing the maximum resolution as the target resolution.