Using depth maps of a scene to identify movement of a video camera

• US 9,489,745 B1
• Filed: 06/12/2015
• Issued: 11/08/2016
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Heitz, George Alban III, Shin, Dongeek
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US14/738,825
• Time in Patent Office: 515 Days
• Field of Search: 348/135, 348/142
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A process recomputes zones for a scene. The process is performed at a computing device having one or more processors and memory. The memory stores one or more programs configured for execution by the one or more processors. The process receives a first image of a scene taken by an array of image sensors of a camera system at a first time and receives designation from a user of a zone within the first image. The process also receives a second image of the scene taken by the array of image sensors at a second time that is after the first time. The process compares the first and second images to identify movement of the camera and notifies the user about a change to the zone when the camera has moved.

Claims

1. A method of recomputing zones for a scene, comprising:

at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

receiving a first plurality of images of a scene captured by an array of image sensors of a camera system at a first time, wherein each of the first plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

receiving designation from a user of a zone within a first image of the first plurality of images;

receiving a second plurality of images of the scene captured by the array of image sensors at a second time that is after the first time, wherein each of the second plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

building a first depth map of the scene using the first plurality of images;

building a second depth map of the scene using the second plurality of images;

comparing points in the first depth map to points in the second depth map to identify movement of the camera; and

notifying the user about a change to the zone when the camera has moved.

2. The method of claim 1, wherein:

when movement is identified and there is substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes identifying an adjusted zone in a second image of the second plurality of images, wherein the adjusted zone corresponds to the zone in the first image, and recommending to the user to replace the zone with the adjusted zone.

3. The method of claim 1, wherein:

when movement is identified and there is not substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes recommending removing the zone.

4. The method of claim 1, wherein the first plurality of images and the second plurality of images are RGB images.

5. The method of claim 1, wherein the first plurality of images and the second plurality of images are IR images.

6. The method of claim 1, wherein building the first depth map of the scene comprises:

partitioning the image sensors into a plurality of pixels; and

for each pixel;

forming a respective vector of the received first plurality of images at the respective pixel; and

estimating a depth in the scene at the respective pixel by looking up the respective vector in a respective lookup table.

7. The method of claim 1, wherein identifying movement of the camera system comprises:

forming a first point cloud using a first plurality of points from the first depth map;

forming a second point cloud using a second plurality of points from the second depth map; and

computing a minimal transformation that aligns the first point cloud with the second point cloud.

8. The method of claim 1, wherein the computing device is included in the camera system.

9. A computing device, comprising:

one or more processors;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for;

receiving a first plurality of images of a scene captured by an array of image sensors of a camera system at a first time, wherein each of the first plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

receiving designation from a user of a zone within a first image of the first plurality of images;

receiving a second plurality of images of the scene captured by the array of image sensors at a second time that is after the first time, wherein each of the second plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

building a first depth map of the scene using the first plurality of images;

building a second depth map of the scene using the second plurality of images;

comparing points in the first depth map to points in the second depth map to identify movement of the camera; and

notifying the user about a change to the zone when the camera has moved.

10. The computing device of claim 9, wherein:

when movement is identified and there is substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes identifying an adjusted zone in a second image of the second plurality of images, wherein the adjusted zone corresponds to the zone in the first image, and recommending to the user to replace the zone with the adjusted zone.

11. The computing device of claim 9, wherein:

when movement is identified and there is not substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes recommending removing the zone.

13. The computing device of claim 9, wherein building the first depth map of the scene comprises:

partitioning the image sensors into a plurality of pixels; and

for each pixel;

forming a respective vector of the received first plurality of images at the respective pixel; and

estimating a depth in the scene at the respective pixel by looking up the respective vector in a respective lookup table.

14. The computing device of claim 9, wherein identifying movement of the camera system comprises:

forming a first point cloud using a first plurality of points from the first depth map;

forming a second point cloud using a second plurality of points from the second depth map; and

computing a minimal transformation that aligns the first point cloud with the second point cloud.

15. The computing device of claim 9, wherein the computing device includes the camera system.

12. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for:

receiving a first plurality of images of a scene captured by an array of image sensors of a camera system at a first time, wherein each of the first plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

receiving designation from a user of a zone within a first image of the first plurality of images;

receiving a second plurality of images of the scene captured by the array of image sensors at a second time that is after the first time, wherein each of the second plurality of images is captured when a different subset of illuminators of the camera system are emitting light;

building a first depth map of the scene using the first plurality of images;

building a second depth map of the scene using the second plurality of images;

comparing points in the first depth map to points in the second depth map to identify movement of the camera; and

notifying the user about a change to the zone when the camera has moved.

16. The computer readable storage medium of claim 12, wherein:

when movement is identified and there is substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes identifying an adjusted zone in a second image of the second plurality of images, wherein the adjusted zone corresponds to the zone in the first image, and recommending to the user to replace the zone with the adjusted zone.

17. The computer readable storage medium of claim 12, wherein:

when movement is identified and there is not substantial overlap between portions of the scene as captured in the first plurality of images and the second plurality of images, notifying the user includes recommending removing the zone.

18. The computer readable storage medium of claim 12, wherein the first plurality of images and the second plurality of images are RGB images.

19. The computer readable storage medium of claim 12, wherein the first plurality of images and the second plurality of images are IR images.

20. The computer readable storage medium of claim 12, wherein building the first depth map of the scene comprises:

partitioning the image sensors into a plurality of pixels; and

for each pixel;

forming a respective vector of the received first plurality of images at the respective pixel; and

estimating a depth in the scene at the respective pixel by looking up the respective vector in a respective lookup table.