Camera setting adjustment based on predicted environmental factors and tracking systems employing the same

• US 10,506,172 B2
• Filed: 08/17/2017
• Issued: 12/10/2019
• Est. Priority Date: 08/17/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Welker, Stefan
• Primary Examiner(s): Haliyur, Padma
• Application Number:
  US15/679,649
• Publication Number:
  US 20180054559A1
• Time in Patent Office: 845 Days
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions

Abstract

Systems, devices, methods, computer-readable storage media, and electronic apparatuses for camera setting adjustment based on predicted environmental factors are provided. An example system includes a camera assembly, at least one processor, and memory storing instructions. When executed by the at least one processor, the instructions may cause the system to determine a direction of movement of the camera assembly. The instructions may also cause the system to predict environmental factors based on the direction of movement. Additionally, the instructions may cause the system to determine camera settings based on the environmental factors. The instructions may also capture an image with the camera assembly using the determined camera settings. In some implementations, the captured image is used to track or one or more entities.

Claims

1. A system comprising:

a camera assembly;

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to;

determine a direction of movement of the camera assembly;

predict a location of the camera assembly based on the direction of movement;

responsive to predicting the location of the camera assembly, predict a light level at the predicted location by identifying a brightness gradient in a previously captured image and using the brightness gradient to predict the light level at the predicted location;

determine an exposure setting for the camera assembly based on the predicted light level;

predictively adjust the camera assembly based on the determined exposure setting; and

capture an image with the camera assembly using the determined exposure setting.

2. The system of claim 1, wherein the instructions that cause the system to determine the direction of movement of the camera assembly, comprise instructions that cause the system to:

extract a first set of features from a first previously captured image;

extract a second set of features from a second previously captured image, the second previously captured image having been captured after the first previously captured image;

identify a plurality of common features that are in both the first set of features and the second set of features; and

predict the direction of movement based on the positions of at least some of the common features in the first previously captured image relative to the second previously captured image.

3. The system of claim 1, wherein the system further comprises an inertial measurement unit (IMU) and the instructions that cause the system to determine the direction of movement of the camera assembly, comprise instructions that cause the system to determine the direction of movement using the IMU.

4. The system of claim 1, wherein the instructions that cause the system to predict the light level at the predicted location, comprise instructions that cause the system to predict the light level based on data about the predicted location that is retrieved from an environmental map.

5. The system of claim 1, wherein the camera assembly includes a first camera and a second camera, and wherein the instructions that cause the system to determine the exposure setting based on the predicted light level, comprise instructions that cause the system to:

determine a main exposure setting based on the predicted light level;

apply a negative offset to the main exposure setting to generate a negative offset exposure setting;

apply a positive offset to the main exposure setting to generate a positive offset exposure setting;

adjust the first camera based on the negative offset exposure setting; and

adjust the second camera based on the positive offset exposure setting.

6. The system of claim 5, wherein the instructions that cause the system to determine the exposure setting based on the predicted light level, further comprise instructions that cause the system to:

determine a number of usable features captured by the first camera and a number of usable features captured by the second camera;

responsive to determining the number of usable features captured by the first camera is greater than the number of usable features captured by the second camera;

generate an adjusted main exposure setting based on the main exposure setting and the negative offset exposure setting;

apply the negative offset to the adjusted main exposure setting to generate an adjusted negative offset exposure setting;

apply the positive offset to the adjusted main exposure setting to generate an adjusted positive offset exposure setting;

adjust the first camera based on the adjusted negative offset exposure setting; and

adjust the second camera based on the adjusted positive offset exposure setting.

7. The system of claim 1, wherein the instructions further comprise instructions that cause the system to track an entity using images captured by the camera assembly.

15. The system of claim 1, wherein the instructions that cause the system to predictively adjust the camera assembly include instructions that cause the system to adjust camera settings of the camera assembly based on a predicted future location of the camera assembly.

8. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to at least:

capture a first image with a camera of a moving camera assembly;

predict a future field of view of the moving camera assembly;

responsive to predicting the future field of view, divide the predicted field of view of the camera into a plurality of regions;

predict light levels within at least some of the plurality of regions;

estimate a number of useable features in at least some of the plurality of regions;

combine the predicted light levels by weighting the regions based on the estimated numbers of usable features;

determine an exposure setting based on the combined predicted light level;

adjust the camera based on the exposure setting determined based on the predicted light level; and

capture a second image with the camera using the determined exposure setting.

9. The non-transitory computer-readable storage medium of claim 8, further comprising instructions that cause the computing system to use a map to estimate the number of useable features in at least some of the plurality of regions.

10. The non-transitory computer-readable storage medium of claim 9, wherein a usable feature is determined based on a quality score assigned to the feature exceeding a threshold value.

14. The non-transitory computer-readable storage medium of claim 8, further comprising instructions that cause the computing system to extract a set of features from the first image and wherein the instructions that cause the computing system to determine the exposure setting based on the predicted light level include instructions that cause the computing system to determine the exposure setting based on the predicted light level to improve capture of at least some of the extracted set of features.

11. A method of tracking an entity:

capturing a first image with a camera of a moving camera assembly;

predicting a location of the moving camera assembly;

predicting a future field of view of the camera based on the predicted location of the moving camera assembly;

responsive to predicting the future field of view of the camera, predicting a light level within the predicted future field of view;

determining an exposure setting based on the predicted light level to improve capture of at least some features of a set of features extracted from a previously captured image, wherein the features are associated with quality values and the exposure setting is determined to improve capture of the at least some of the extracted set of features based on the quality values;

adjusting the camera based on the exposure setting determined based on the predicted light level; and

capturing a second image with the camera using the determined exposure setting.

12. The method of claim 11, wherein the predicted light level is retrieved from an environment map.

13. The method of claim 11, further comprising extracting a set of features from the first image, and wherein the determining the exposure setting based on the predicted light level includes determining the exposure setting based on the predicted light level to improve capture of at least some of the extracted set of features.

16. A system comprising:

a camera assembly that includes a first camera and a second camera;

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to;

determine a direction of movement of the camera assembly;

predict environmental factors based on the direction of movement;

determine main camera settings based on the environmental factors;

apply a negative offset to the main camera settings to generate negative offset camera settings;

apply a positive offset to the main camera settings to generate positive offset camera settings;

adjust the first camera based on the negative offset camera settings;

adjust the second camera based on the positive offset camera settings; and

capture an image with the first camera using the negative offset camera settings and an image with the adjusted second camera using the positive offset camera settings.

17. The system of claim 16, wherein the instructions that cause the system to determine the main camera settings based on the environmental factors, further comprise instructions that cause the system to:

determine a number of usable features captured by the first camera and a number of usable features captured by the second camera;

responsive to determining the number of usable features captured by the first camera is greater than the number of usable features captured by the second camera;

generate adjusted main camera settings based on the main camera settings and the negative offset camera settings;

apply the negative offset to the adjusted main camera settings to generate adjusted negative offset camera settings;

apply the positive offset to the adjusted main camera settings to generate adjusted positive offset camera settings;

adjust the first camera based on the adjusted negative offset camera settings; and

adjust the second camera based on the adjusted positive offset camera settings.

18. A system comprising:

a camera assembly;

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, cause the system to;

determine a direction of movement of the camera assembly;

predict a location of the camera assembly based on the direction of movement;

responsive to predicting the location of the camera assembly, predict a light level at the predicted location based on data about the predicted location that is retrieved from an environmental map;

determine an exposure setting for the camera assembly based on the predicted light level;

predictively adjust the camera assembly based on the determined exposure setting; and

capture an image with the camera assembly using the determined exposure setting.

19. The system of claim 18, wherein the system further comprises an inertial measurement unit (IMU) and the instructions that cause the system to determine the direction of movement of the camera assembly, comprise instructions that cause the system to determine the direction of movement using the IMU.