CAMERA SETTING ADJUSTMENT BASED ON PREDICTED ENVIRONMENTAL FACTORS AND TRACKING SYSTEMS EMPLOYING THE SAME

• US 20180054559A1
• Filed: 08/17/2017
• Published: 02/22/2018
• Est. Priority Date: 08/17/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: WELKER, Stefan
• Granted Patent: US 10,506,172 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions

Abstract

Systems, devices, methods, computer-readable storage media, and electronic apparatuses for camera setting adjustment based on predicted environmental factors are provided. An example system includes a camera assembly, at least one processor, and memory storing instructions. When executed by the at least one processor, the instructions may cause the system to determine a direction of movement of the camera assembly. The instructions may also cause the system to predict environmental factors based on the direction of movement. Additionally, the instructions may cause the system to determine camera settings based on the environmental factors. The instructions may also capture an image with the camera assembly using the determined camera settings. In some implementations, the captured image is used to track or one or more entities.

Claims

1. A system comprising:

a camera assembly;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

determine a direction of movement of the camera assembly;

predict environmental factors based on the direction of movement;

determine camera settings based on the environmental factors; and

capture an image with the camera assembly using the determined camera settings.

2. The system of claim 1, wherein the instructions that cause the system to determine the direction of movement of the camera assembly, comprise instructions that cause the system to:

extract a first set of features from a first previously captured image;

extract a second set of features from a second previously captured image, the second previously captured image having been captured after the first previously captured image;

identify a plurality of common features that are in both the first set of features and the second set of features; and

predict the direction of movement based on the positions of at least some of the common features in the first previously captured image relative to the second previously captured image.

3. The system of claim 1, wherein the system further comprises an inertial measurement unit (IMU) and the instructions that cause the system to determine the direction of movement of the camera assembly, comprise instructions that cause the system to determine the direction of movement using the IMU.

4. The system of claim 1, wherein the environmental factors include a light level.

5. The system of claim 4, wherein the instructions that cause the system to predict environmental factors based on the direction of movement, comprise instructions that cause the system to:

predict a location of the camera assembly based on the direction of movement; and

predict the light level at the predicted location.

6. The system of claim 5, wherein the instructions that cause the system to predict the light level at the predicted location, comprise instructions that cause the system to:

identify a brightness gradient in a previously captured image; and

use the brightness gradient to predict the light level at the predicted location.

7. The system of claim 5, wherein the instructions that cause the system to predict the light level at the predicted location, comprise instructions that cause the system to predict the light level based on data about the predicted location that is retrieved from an environmental map.

8. The system of claim 5, wherein the instructions further comprise instructions that cause the system to:

extract a set of features from a previously captured image; and

determine the camera settings based on the predicted light level to improve capture of at least some of the extracted set of features.

9. The system of claim 8, wherein the features are associated with a quality value and the cameras settings are determined to improve capture of the at least some of the extracted set of features based on the quality level.

10. The system of claim 1, wherein the camera assembly includes a first camera and a second camera, and wherein the instructions that cause the system to determine the camera settings based on the environmental factors, comprise instructions that cause the system to:

determine main camera settings based on the environmental factors;

apply a negative offset to the main camera settings to generate negative offset camera settings;

apply a positive offset to the main camera settings to generate positive offset camera settings;

adjust the first camera based on the negative offset camera settings; and

adjust the second camera based on the positive offset camera settings.

11. The system of claim 10, wherein the instructions that cause the system to determine the camera settings based on the environmental factors, further comprise instructions that cause the system to:

determine a number of usable features captured by the first camera and a number of usable features captured by the second camera;

responsive to determining the number of usable features captured by the first camera is greater than the number of usable features captured by the second camera;

generate adjusted main camera settings based on the main camera settings and the negative offset camera settings;

apply the negative offset to the adjusted main camera settings to generate adjusted negative offset camera settings;

apply the positive offset to the main camera settings to generate adjusted positive offset camera settings;

adjust the first camera based on the adjusted negative offset camera settings; and

adjust the second camera based on the adjusted positive offset camera settings.

12. The system of claim 1, wherein the camera settings include exposure settings.

13. The system of claim 1, wherein the instructions further comprise instructions that cause the system to track an entity using images captured by the camera assembly.

14. A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to at least:

capture a first image with a camera of a moving camera assembly;

predict a future location of the moving camera assembly;

predict a light level at the predicted future location;

determine exposure settings based on the predicted light level; and

capture a second image with the camera using the determined exposure settings.

15. The non-transitory computer-readable storage medium of claim 14, further comprising instructions that cause the computing system to:

divide a predicted field of view of the camera into a plurality of regions;

predict light levels within at least some of the plurality of regions; and

determine the exposure settings based on combining the predicted light levels.

16. The non-transitory computer-readable storage medium of claim 15, further comprising instructions that cause the computing system to:

estimate a number of useable features in at least some of the plurality of regions; and

combine the predicted light levels by weighting the regions based on the estimated numbers of usable features.

17. The non-transitory computer-readable storage medium of claim 16, further comprising instructions that cause the computing system to use a map to estimate the number of useable features in at least some of the plurality of regions.

18. The non-transitory computer-readable storage medium of claim 17, wherein a usable feature is determined based on a quality score assigned to the feature exceeding a threshold value.

19. A method of tracking an entity:

capturing a first image with a camera of a moving camera assembly;

predicting a location of the moving camera assembly;

predicting a future field of view of the camera based on the predicted location of the moving camera assembly;

predicting environmental factors within the predicted future field of view;

determining camera setting based on the predicted environmental factors; and

capturing a second image with the camera using the determined camera settings.

20. The method of claim 19, wherein the environmental factors are retrieved from an environment map and include a light level, and wherein the camera settings include exposure settings.