Scene stability detection

• US 9,686,470 B2
• Filed: 09/30/2014
• Issued: 06/20/2017
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Brunner, Ralph, Nariani-Schulze, Anita, Olson, Benjamin M., Lim, Suk Hwan, Doepke, Frank
• Primary Examiner(s): FOSSELMAN, JOEL W
• Application Number:
  US14/502,128
• Publication Number:
  US 20150350547A1
• Time in Patent Office: 994 Days
• Field of Search: 3482086, 382232
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 23/6811 based on the image signal

Abstract

Techniques to detect subject and camera motion in a set of consecutively captured image frames are disclosed. More particularly, techniques disclosed herein temporally track two sets of downscaled images to detect motion. One set may contain higher resolution and the other set lower resolution of the same images. For each set, a coefficient of variation may be computed across the set of images for each sample in the downscaled image to detect motion and generate a change mask. The information in the change mask can be used for various applications, including determining how to capture a next image in the sequence.

Claims

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause the programmable control device to:

obtain a set of two or more image frames in an image sequence;

downscale each of the obtained image frames in the set in its entirety, thereby generating a first downscaled version of each image frame, wherein the first downscaled version of each image frame consists of a first plurality of samples;

calculate a coefficient of variation for each of the first plurality of samples in the first downscaled version of a last received image frame in the set, wherein the coefficient of variation for each sample in the first downscaled version of the last received image frame in the set is calculated based, at least in part, on the corresponding sample in the first downscaled version of each of the other image frames in the set; and

detect motion in the last image frame in the set by comparing each of the calculated coefficients of variation in the first downscaled version of the last received image frame in the set to a threshold value.

2. The non-transitory program storage device of claim 1, wherein the instructions to detect motion in the last image frame comprise instructions to cause the programmable control device to generate a change mask based on determining if each coefficient of variation exceeds the threshold value.

3. The non-transitory program storage device of claim 2, further comprising instructions stored thereon to cause the programmable control device to apply a Gaussian distribution to the change mask.

4. The non-transitory program storage device of claim 3, wherein the Gaussian distribution is applied to one or more areas of interest in the last received image frame.

5. The non-transitory program storage device of claim 1, further comprising instructions stored thereon to cause the programmable control device to cause the programmable control device to exposure normalize each image frame in the set.

6. The non-transitory program storage device of claim 1, further comprising instructions stored thereon to cause the programmable control device to cause the programmable control device to:

perform a second downscaling operation on each of the obtained image frames in its entirety, thereby generating a second downscaled version of each image frame, wherein the second downscaled version of each image frame consists of a second plurality of samples, and wherein the second plurality of samples is a larger number than the first plurality of samples; and

calculate a coefficient of variation for each of the second plurality of samples in the second downscaled version of the last received image frame in the set, wherein the coefficient of variation for each sample in the second downscaled version of the last received image frame in the set is calculated based, at least in part, on the corresponding sample in the second downscaled version of each of the other image frames in the set.

7. The non-transitory program storage device of claim 6, further comprising instructions stored thereon to cause the programmable control device to cause the programmable control device to:

detect motion in the last image frame in the set by comparing each of the calculated coefficients of variation in the second downscaled version of the last received image frame in the set to a threshold value when no motion is detected by comparing the calculated coefficients of variation in the first downscaled version of the last received image frame in the set.

8. The non-transitory program storage device of claim 6, further comprising instructions stored thereon to cause the programmable control device to:

exposure normalize the first downscaled version of each image frame in the set; and

exposure normalize the second downscaled version of each image frame in the set.

9. An electronic device, comprising:

an image capture unit;

a memory operatively coupled to the image capture unit; and

one or more processors operatively coupled to the memory and configured to execute instructions stored in the memory, the instructions configured to cause the one or more processors to;

capture, by the image capture unit, a set of two or more consecutively captured images;

store the set of images in the memory;

downscale each of the captured images in the set in its entirety, thereby generating a first downscaled version of each image, wherein the first downscaled version of each image consists of a first plurality of samples;

calculate a coefficient of variation for each of the first plurality of samples in the first downscaled version of a last received image in the set, the coefficient of variation for each sample in the first downscaled version of the last received image in the set is calculated based, at least in part, on the corresponding sample in the first downscaled version of each of the other images in the set; and

detect motion in the last image in the set by comparing the coefficients of variation in the first downscaled version of the last received image in the set to a threshold value.

10. The electronic device of claim 9, wherein the instructions to detect motion in the last image comprise instructions to cause the one or more processors to:

generate a change mask based on determining if each coefficient of variation exceeds the threshold value.

11. The electronic device of claim 10, further comprising instructions to cause the one or more processors to apply a Gaussian distribution to the change mask.

12. The electronic device of claim 9, further comprising instructions to cause the one or more processors to exposure normalize each image in the set.

13. The electronic device of claim 9, further comprising instructions to cause the one or more processors to:

perform a second downscaling operation on each of the obtained images in its entirety, thereby generating a second downscaled version of each image, wherein the second downscaled version of each image consists of a second plurality of samples, and wherein the second plurality of samples is a larger number than the first plurality of samples; and

calculate a coefficient of variation for each of the second plurality of samples in the second downscaled version of the last received image in the set, wherein the coefficient of variation for each sample in the second downscaled version of the last received image in the set is calculated based, at least in part, on the corresponding sample in the second downscaled version of each of the other images in the set.

14. The electronic device of claim 13, further comprising instructions to cause the one or more processors to:

detect motion in the last image in the set by comparing each of the calculated coefficients of variation in the second downscaled version of the last received image in the set to a threshold value when no motion is detected by comparing the calculated coefficients of variation in the first downscaled version of the last received image in the set.

15. The electronic device of claim 14, further comprising instructions to cause the one or more processors to register the last image in the set of captured images with an image preceding it in the image sequence.

16. The electronic device of claim 14, further comprising instructions to cause the one or more processors to:

exposure normalize the first downscaled version of each image in the set; and

exposure normalize the second downscaled version of each image in the set.

17. A method comprising:

obtaining a set of two or more image frames in an image sequence;

downscaling each of the obtained image frames in the set in its entirety, thereby generating a first downscaled version of each image frame, wherein the first downscaled version of each image frame consists of a first plurality of samples;

calculating a coefficient of variation for each of the first plurality of samples in the first downscaled version of a last received image frame in the set, wherein the coefficient of variation for each sample in the first downscaled version of the last received image frame in the set is calculated based, at least in part, on the corresponding sample in the first downscaled version of each of the other image frame in the set; and

detecting motion in the last image frame in the set by comparing each of the calculated coefficients of variation in the first downscaled version of the last received image frame in the set to a threshold value.

18. The method of claim 17, further comprising:

generating a change mask based on determining if each coefficient of variation exceeds the threshold value.

19. The method of claim 18, further comprising applying a Gaussian distribution to the change mask.

20. The method of claim 17, further comprising fusing two or more of the image frames in the set together based, at least in part, on the detected motion.