Focus specification and focus stabilization

• US 9,549,125 B1
• Filed: 09/01/2015
• Issued: 01/17/2017
• Est. Priority Date: 09/01/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Goyal, Dushyant, Mishra, Pragyana K.
• Primary Examiner(s): Vieaux, Gary C
• Application Number:
  US14/842,754
• Time in Patent Office: 504 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 7/36 using image sharpness techniques , e.g. image processing techniques for generating autofocus signals image data processing per se G06T; in cameras having a solid state image sensor H04N23/67
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions
• CPC Class Codes: H04N 23/676 Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An imaging device may be configured to monitor a field of view for various objects or events occurring therein. The imaging device may capture a plurality of images at various focal lengths, identify a region of interest including one or more semantic objects therein, and determine measures of the levels of blur or sharpness within the regions of interest of the images. Based on their respective focal lengths and measures of their respective levels of blur or sharpness, a focal length for capturing subsequent images with sufficient clarity may be predicted. The imaging device may be adjusted to capture images at the predicted focal length, and such images may be captured. Feedback for further adjustments to the imaging device may be identified by determining measures of the levels of blur or sharpness within the subsequently captured images.

Claims

1. An imaging device comprising:

a lens;

an imaging sensor having a photosensitive surface with a first plurality of pixel sensors;

at least one memory device; and

at least one computer processor,

wherein the at least one computer processor is configured to at least;

capture a first image, wherein the first image is captured with the lens at a first focal length from the imaging sensor, and wherein the first image is captured at a first level of resolution;

recognize at least one object within the first image;

automatically define a region of interest within the first image based at least in part on the at least one object within the first image; and

determine a first blur metric associated with the region of interest within the first image;

adjust the lens to a second focal length from the imaging sensor;

capture at least a second image, wherein the second image is captured with the lens at the second focal length from the imaging sensor, and wherein the second image is captured at a second level of resolution;

recognize the at least one object within the second image;

automatically identify the region of interest within the second image based at least in part on the at least one object within the second image;

determine a second blur metric associated with the region of interest within the second image;

determine a third focal length for capturing a third image based at least in part on at least one of the first focal length, the first blur metric, the second focal length or the second blur metric;

adjust the lens to the third focal length from the imaging sensor; and

capture the third image, wherein the third image is captured with the lens at the third focal length from the imaging sensor, wherein the third image is captured at a third level of resolution, and wherein the third level of resolution exceeds at least one of the first level of resolution or the second level of resolution.

2. The imaging device of claim 1, wherein the at least one computer processor is further configured to at least:

identify the region of interest within the third image;

determine a third blur metric associated with the region of interest within the third image;

determine a fourth focal length for capturing a fourth image based at least in part on at least one of the third focal length or the third blur metric;

adjust the lens to the fourth focal length from the imaging sensor; and

capture the fourth image, wherein the fourth image is captured with the lens at the fourth focal length from the imaging sensor.

3. The imaging device of claim 1, wherein the at least one computer processor is further configured to at least:

cause a display of the first image on a computer display having a touchscreen element;

receive a selection of a portion of the touchscreen element;

identify a portion of the first image associated with the portion of the touchscreen element; and

identify the region of interest within the first image based at least in part on the portion of the first image associated with the portion of the touchscreen element.

4. The imaging device of claim 1, wherein the at least one computer processor is further configured to at least:

provide at least a portion of the first image as an input to a Laplacian operator; and

determine the first blur metric associated with the region of interest within the first image based at least in part on an output of the Laplacian operator.

5. A computer-implemented method comprising:

identifying a first digital image captured using a first imaging device, wherein the first digital image is captured with a lens of the first imaging device at a first focal length with respect to an image sensor of the first imaging device;

identifying a first region of interest in at least a first portion of the first digital image comprising at least one semantic object, wherein the first region of interest is defined based at least in part on the at least one semantic object;

determining a first blur metric for at least the first portion of the first digital image;

predicting a second focal length for capturing a second digital image based at least in part on at least one of the first focal length or the first blur metric, wherein the second focal length is predicted to maintain the at least one semantic object in focus within the second digital image;

adjusting the lens to the second focal length with respect to the image sensor;

capturing the second digital image using the first imaging device; and

storing the second digital image in at least one data store.

6. The computer-implemented method of claim 5, wherein identifying the first digital image captured using the first imaging device further comprises:

capturing a burst of digital images using the first imaging device;

determining blur metrics for at least a first plurality of the digital images of the burst; and

selecting one of the first plurality of the digital images of the burst having the lowest blur metric,

wherein the first digital image is the selected one of the first plurality of the digital images of the burst.

7. The computer-implemented method of claim 6, wherein capturing the burst of digital images using the first imaging device further comprises:

identifying a plurality of focal lengths, wherein the plurality of focal lengths includes the first focal length, and

wherein each of the digital images of the burst is captured at a respective one of the plurality of focal lengths.

8. The computer-implemented method of claim 7, wherein identifying the plurality of focal lengths comprises at least one of:

selecting at least one of the plurality of focal lengths at random;

selecting the at least one of the plurality of focal lengths based at least in part on historical data regarding at least one of the first imaging device or an environment in which the first imaging device is provided;

or

selecting the at least one of the plurality of focal lengths based at least in part on at least one attribute of the at least one semantic object.

9. The computer-implemented method of claim 6, wherein identifying the first region of interest in at least the first portion of the first digital image further comprises:

identifying a plurality of semantically related pixels within the first portion of the first digital image;

associating the plurality of semantically related pixels with the at least one semantic object; and

defining the first region of interest based at least in part on the plurality of semantically related pixels.

10. The computer-implemented method of claim 9, wherein identifying the plurality of semantically related pixels within the first portion of the first digital image further comprises:

causing a display of at least the first portion of the first digital image on a touchscreen display; and

receiving information regarding an interaction with at least a portion of the touchscreen display, wherein the portion of the touchscreen display is associated with at least one of the plurality of semantically related pixels, and

wherein defining the first region of interest based at least in part on the plurality of semantically related pixels further comprises;

defining the first region of interest based at least in part on the interaction with the portion of the touchscreen display.

11. The computer-implemented method of claim 9, wherein identifying the plurality of semantically related pixels within the first portion of the first digital image further comprises:

automatically recognizing a plurality of pixels corresponding to an object within the first portion of the first digital image;

determining that each of the plurality of pixels corresponding to the object is semantically related, wherein the plurality of pixels corresponding to the object is the plurality of semantically related pixels; and

determining that the plurality of semantically related pixels corresponds to one of a plurality of semantic objects, wherein the one of the plurality of semantic objects is the at least one semantic object, and

wherein defining the first region of interest based at least in part on the plurality of semantically related pixels further comprises;

defining the first region of interest based at least in part on the at least one semantic object within the first portion of the digital image.

12. The computer-implemented method of claim 9, wherein identifying the plurality of semantically related pixels within the first portion of the first digital image further comprises:

automatically recognizing a plurality of pixels corresponding to motion of an object within the first portion of the first digital image; and

determining that each of the plurality of pixels corresponding to the motion of the object is semantically related, wherein the plurality of pixels corresponding to the motion is the plurality of semantically related pixels,

wherein defining the first region of interest based at least in part on the plurality of semantically related pixels further comprises;

defining the first region of interest based at least in part on the motion of the object.

13. The computer-implemented method of claim 12, wherein associating the plurality of semantically related pixels with the at least one semantic object further comprises:

determining that the plurality of semantically related pixels is associated with at least one of a predetermined shape, a predetermined size, a predetermined velocity,

a predetermined feature, a predetermined color, a predetermined marking, a predetermined level of reflectance or a predetermined frequency, and

wherein defining the first region of interest based at least in part on the plurality of semantically related pixels further comprises;

defining the first region of interest based at least in part on the predetermined shape, the predetermined size, the predetermined velocity, the predetermined feature, the predetermined color, the predetermined marking, the predetermined level of reflectance or the predetermined frequency.

14. The computer-implemented method of claim 12, wherein automatically recognizing the plurality of pixels corresponding to the motion of the object within the first portion of the first digital image further comprises:

providing at least the first portion of the first digital image as an input to at least one classifier;

receiving an output from the at least one classifier; and

recognizing that the plurality of pixels corresponds to the motion of the object based at least in part on the output from the classifier.

15. The computer-implemented method of claim 5, wherein identifying the first region of interest in at least the first portion of the first digital image further comprises:

identifying a plurality of regions of interest within the first digital image, wherein each of the regions of interest is associated with a respective one of a plurality of semantic objects; and

determining, for each of the plurality of regions of interest, a value of at least one attribute of the respective one of the plurality of semantic objects;

ranking the plurality of regions of interest based at least in part on the value of the at least one attribute of the respective one of the semantic objects,

wherein the first region of interest is a highest ranking of the plurality of regions of interest, and

wherein the at least one attribute is at least one of a shape, a size, a velocity, a feature, a color, a velocity, a marking, a level of reflectance or a frequency associated with the respective one of the semantic objects.

16. The computer-implemented method of claim 5, further comprising:

identifying the first region of interest in at least a second portion of the second digital image;

determining a second blur metric for at least the second portion of the second digital image;

predicting a third focal length for capturing a third digital image based at least in part on at least one of the first focal length, the first blur metric, the second focal length or the second blur metric;

adjusting the lens to the third focal length with respect to the image sensor;

capturing the third digital image using the first imaging device; and

storing the third digital image in the at least one data store.

17. The computer-implemented method of claim 16, wherein predicting the third focal length for capturing the third digital image further comprises:

determining at least one of a minimum level of blur or a maximum level of sharpness of the first region of interest based at least in part on the at least one of the first focal length, the first blur metric, the second focal length or the second blur metric; and

predicting a focal length for capturing a digital image with the at least one of the minimum level of blur or the maximum level of sharpness of the first region of interest using the first imaging device,

wherein the third focal length is the predicted focal length.

18. The computer-implemented method of claim 5, wherein the first digital image is captured at a first level of resolution,

wherein the second digital image is captured at a second level of resolution, and

wherein the second level of resolution is greater than the first level of resolution.

19. A mobile computing device comprising:

an imaging sensor having a photosensitive surface with a plurality of pixel sensors;

a lens;

at least one memory device;

at least one touchscreen display;

at least one computer processor,

wherein the at least one computer processor is configured to at least;

capture a burst of image frames, wherein each of the image frames of the burst is captured with the lens at a different focal length with respect to the imaging sensor;

cause a display of a first one of the image frames of the burst on the touchscreen display;

receive a selection of a portion of the touchscreen display;

identify a portion of the first image frame corresponding to the selected portion of the touchscreen display;

identify a plurality of semantically related pixels within the portion of the first image frame;

define a first region of interest within the first image frame based at least in part on the plurality of semantically related pixels;

determine a first value of a perceptual score for the first region of interest within the portion of the first image frame, wherein the perceptual score is indicative of a level of sharpness of the first region of interest within the first image frame;

identify a first focal length at which the first image frame was captured;

predict a second focal length for capturing an image frame with the first region of interest at a value of the perceptual score greater than the first value, wherein the second focal length is predicted based at least in part on the first focal length and the first value;

position the lens at the second focal length with respect to the imaging sensor; and

capture a second image frame with the lens at the second focal length with respect to the imaging sensor.

20. The mobile computing device of claim 19, wherein the at least one computer processor is further configured to at least:

identify the first region of interest within at least a portion of the second image frame;

determine a second value of the perceptual score for the first region of interest within at least the portion of the second image frame;

predict a third focal length for capturing an image frame with the first region of interest at a value of the perceptual score greater than at least one of the first value or the second value, wherein the third focal length is predicted based at least in part on at least one of the first focal length, the first value, the second focal length or the second value;

position the lens at the third focal length with respect to the imaging sensor; and

capture a third image frame with the lens at the third focal length with respect to the imaging sensor.

21. The mobile computing device of claim 19, wherein the at least one computer processor is further configured to at least:

provide at least the portion of the first image frame as an input to at least one classifier;

identify the plurality of semantically related pixels based at least in part on an output from the at least one classifier;

determine that the plurality of semantically related pixels corresponds to a semantic object; and

define the first region of interest within the first image frame based at least in part on the semantic object.

22. The mobile computing device of claim 19, wherein the at least one computer processor is further configured to at least:

identify a plurality of semantic objects within the first image frame;

cause a display of a plurality of interactive identifiers on the first image frame on the touchscreen display,

wherein each of the interactive identifiers is associated with a respective one of the plurality of semantic objects, and

wherein the selected portion of the first region of interest corresponds to one of the interactive features.

23. The mobile computing device of claim 19, wherein the at least one computer processor is further configured to at least:

define a plurality of regions of interest based at least in part on the plurality of semantically related pixels, wherein each of the regions of interest is associated with a respective object within the first image frame, and wherein the first region of interest is one of the plurality of regions of interest;

determine a value of at least one attribute for each of the plurality of regions of interest;

rank the plurality of regions of interest based at least in part on the values of the at least one attribute,

wherein the first region of interest is a highest ranking of the plurality of regions of interest, and

wherein the at least one attribute is at least one of a shape, a size, a velocity, a feature, a color, a velocity, a marking, a level of reflectance or a frequency of the respective object associated with each of the regions of interest.