Using a scene illuminating infrared emitter array in a video monitoring camera to estimate the position of the camera

• US 9,886,620 B2
• Filed: 06/12/2015
• Issued: 02/06/2018
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Dixon, Michael, Shin, Dongeek
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Colon, Maria Vazquez
• Application Number:
  US14/738,811
• Publication Number:
  US 20160364612A1
• Time in Patent Office: 970 Days
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A process estimates height and tilt of a camera having an image sensor array and IR illuminators. The process identifies multiple distinct subsets of the illuminators. For multiple heights and tilts, the process constructs dictionary entries that correspond to the camera having the height and tilt above a floor. Each entry includes light intensity values for pixels in images corresponding to activating each of the subsets of illuminators. For each of the subsets, the process receives an image of a scene captured while the subset is emitting light and the other illuminators are not. The process uses the captured images to identify a floor region corresponding to a floor in the scene. The process forms a vector including pixels from the captured images in the identified floor region and estimates the height and tilt of the camera by comparing the vector to the dictionary entries.

Claims

1. A method of estimating height and tilt angle of a camera system having a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors, comprising:

at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

identifying a plurality of distinct subsets of the IR illuminators;

partitioning the image sensors into a plurality of pixels;

for each of the plurality of distinct subsets of the IR illuminators, receiving a captured IR image of a scene taken by the array of image sensors while the respective subset of the IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light;

using at least one of the captured IR images to identify a floor region corresponding to a floor in the scene;

forming a light intensity vector including pixels from the captured IR images in the identified floor region; and

estimating a camera height and camera tilt angle relative to the floor by comparing the light intensity vector to a dictionary, wherein the dictionary includes entries that correspond to the camera system having a plurality of heights and tilt angles.

2. The method of claim 1, wherein each respective dictionary entry includes a respective height and tilt angle above a floor and includes respective IR light intensity values for pixels in images corresponding to activating individually each of the distinct subsets of the IR illuminators.

3. The method of claim 2, wherein each dictionary entry is constructed based on simulating the camera, the floor, and the images, and computing expected IR light intensity values for pixels in the simulated images.

4. The method of claim 3, wherein each expected IR light intensity value is based on characteristics of the IR illuminators, including one or more characteristics selected from the group consisting of lux, orientation of the IR illuminators relative to the array of image sensors, and location of the IR illuminators relative to the array of image sensors.

5. The method of claim 1, wherein a respective dictionary entry for a respective height and respective tilt angle is based on measuring IR light intensity values of actual images captured by the camera having the respective height and respective tilt angle with respect to an actual floor.

6. The method of claim 1, wherein each respective dictionary entry is normalized.

7. The method of claim 6, wherein normalizing a respective dictionary entry comprises determining a respective total magnitude of the light intensity features in the respective dictionary entry and dividing each component of the respective dictionary entry by the respective total magnitude.

8. The method of claim 1, wherein each pixel comprises a respective single image sensor.

9. The method of claim 1, wherein each pixel comprises a respective plurality of image sensors.

10. The method of claim 1, wherein:

each of the distinct subsets of the IR illuminators comprises two adjacent IR illuminators, and the distinct subsets of the IR illuminators are non-overlapping.

11. The method of claim 1, wherein the computing device is included in the camera system.

12. The method of claim 1, wherein the computing device is a server distinct from the camera system.

13. The method of claim 12, wherein the dictionary entries are downloaded to the camera system from the computing device during an initialization process.

14. The method of claim 1, further comprising:

receiving a baseline IR image of the scene captured by the array of image sensors while none of the IR illuminators are emitting IR light; and

subtracting a light intensity at each pixel of the baseline IR image from the light intensity at the corresponding pixel of each of the other captured IR images.

15. The method of claim 1, wherein estimating the camera height and camera tilt angle relative to the floor comprises:

computing a respective distance between the light intensity vector and respective dictionary entries;

selecting a first dictionary entry whose corresponding computed distance is less than the other computed distances; and

estimating the camera height and tilt angle to be the height and tilt angle associated with the first dictionary entry.

16. The method of claim 15, wherein computing a respective distance between the light intensity vector and respective dictionary entries comprises computing a Euclidean distance that uses only vector components corresponding to pixels in the identified floor region.

17. The method of claim 15, further comprising normalizing the light intensity vector and the dictionary entries prior to computing the distances.

18. The method of claim 1, wherein identifying the floor region comprises:

constructing a depth map of the scene using the captured IR images;

identifying a region bounded by depth discontinuities; and

determining that the region is substantially planar and facing upwards.

19. A computing device, comprising:

one or more processors;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for estimating height and tilt angle of a camera system having a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors, wherein the instructions are configured to;

identify a plurality of distinct subsets of the IR illuminators;

partition the image sensors into a plurality of pixels;

for each of the plurality of distinct subsets of the IR illuminators, receive a captured IR image of a scene taken by the array of image sensors while the respective subset of the IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light;

use at least one of the captured IR images to identify a floor region corresponding to a floor in the scene;

form a light intensity vector including pixels from the captured IR images in the identified floor region; and

estimate a camera height and camera tilt angle relative to the floor by comparing the light intensity vector to a dictionary, wherein the dictionary includes entries that correspond to the camera system having a plurality of heights and tilt angles.

20. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for estimating height and tilt angle of a camera system having a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors, wherein the instructions are configured to:

identify a plurality of distinct subsets of the IR illuminators;

partition the image sensors into a plurality of pixels;

for each of the plurality of distinct subsets of the IR illuminators, receive a captured IR image of a scene taken by the array of image sensors while the respective subset of the IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light;

use at least one of the captured IR images to identify a floor region corresponding to a floor in the scene;

form a light intensity vector including pixels from the captured IR images in the identified floor region; and

estimate a camera height and camera tilt angle relative to the floor by comparing the light intensity vector to a dictionary, wherein the dictionary includes entries that correspond to the camera system having a plurality of heights and tilt angles.