Method and Apparatus for Projective Volume Monitoring

• US 20130094705A1
• Filed: 10/12/2012
• Published: 04/18/2013
• Est. Priority Date: 10/14/2011
• Status: Active Grant

• Current Assignee: Omron Corporation
• Original Assignee: Omron Corporation
• Inventors: Tyagi, Ambrish, Kinoshita, Koichi, Drinkard, John, Tani, Yoshiharu
• Granted Patent: US 9,501,692 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: F16P 3/142 using image capturing devices
• CPC Class Codes: F16P 3/144 using light grids
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition

Abstract

According to one aspect of the teachings presented herein, a projective volume monitoring apparatus is configured to detect objects intruding into a monitoring zone. The projective volume monitoring apparatus is configured to detect the intrusion of objects of a minimum object size relative to a protection boundary, based on an advantageous processing technique that represents range pixels obtained from stereo correlation processing in spherical coordinates and maps those range pixels to a two-dimensional histogram that is defined over the projective coordinate space associated with capturing the stereo images used in correlation processing. The histogram quantizes the horizontal and vertical solid angle ranges of the projective coordinate space into a grid of cells. The apparatus flags range pixels that are within the protection boundary and accumulates them into corresponding cells of the histogram, and then performs clustering on the histogram cells to detect object intrusions.

Claims

1. A method of detecting objects intruding into a monitoring zone, said method performed by a projective volume monitoring apparatus and comprising:

capturing a stereo image from a pair of image sensors;

correlating the stereo image to obtain a depth map comprising range pixels represented in three-dimensional Cartesian coordinates;

converting the range pixels into spherical coordinates, so that each range pixel is represented as a radial distance along a respective pixel ray and a corresponding pair of solid angle values within the horizontal and vertical fields of view associated with capturing the stereo image;

obtaining a set of flagged pixels by flagging those range pixels that fall within a protection boundary defined for the monitoring zone;

accumulating the flagged pixels into corresponding cells of a two-dimensional histogram that quantizes the solid angle ranges of the horizontal and vertical fields of view; and

clustering cells in the histogram to detect intrusions of objects within the protection boundary that meet a minimum object size threshold.

2. The method of claim 1, wherein clustering cells in the histogram comprises identifying qualified cells as those cells that accumulated at least a minimum number of the flagged pixels, and identifying qualified clusters as those clusters of qualified cells that meet a minimum cluster size corresponding to the minimum object size threshold.

3. The method of claim 2, wherein clustering cells in the histogram further comprises determining whether any qualified clusters persist for a defined window of time and, if so, determining that an object intrusion has occurred.

4. The method of claim 2, wherein qualified clusters are identified for the stereo image captured in each of a number of successive image frames, and further comprising filtering the qualified clusters over time by determining whether a same qualified cluster persists over a defined number of image frames and, if so, determining that an object intrusion has occurred.

5. The method of claim 4, wherein filtering the qualified clusters over time further includes determining whether equivalent qualified clusters persist over the defined number of image frames and, if so, determining that an object intrusion has occurred, wherein equivalent qualified clusters are those qualified clusters that split or merge within a same region of the histogram over the defined number of image frames.

6. The method of claim 1, further comprising generating one or more sets of mitigation pixels from the stereo image, wherein the mitigation pixels represent pixels in the stereo image that are flagged as being faulty or unreliable, and integrating the one or more sets of mitigation pixels into the set of flagged pixels for accumulation in corresponding cells of the histogram, along with the range pixels flagged for inclusion in the set of flagged pixels, so that any given cluster includes cells that accumulated range pixels, mitigation pixels, or a mix of both.

7. The method of claim 6, wherein generating the one or more sets of mitigation pixels comprises detecting at least one of:

pixels in the stereo image that are outside of a usable dynamic range;

pixels in the stereo image that are stuck;

pixels in the stereo image that have reduced sensitivity;

pixels in the stereo image that are noisy; and

pixels in the stereo image that are identified as corresponding to shadowed regions of the monitoring zone.

8. The method of claim 1, wherein capturing the stereo image comprises capturing corresponding high exposure and low exposure stereo images and forming a high dynamic range stereo image from the corresponding high and low exposure stereo images, and further comprising obtaining a rectified high dynamic range stereo image by rectifying the high dynamic range stereo image, so that correlation processing is performed on the rectified high dynamic range stereo image.

9. The method of claim 8, further comprising flagging pixels in the rectified high dynamic range stereo image as mitigation pixels that are deemed bad or unreliable for range detection purposes, and further comprising adding the mitigation pixels to the set of flagged pixels, such that clustering considers both the range pixels and the mitigation pixels in the set of flagged pixels.

10. The method of claim 8, further comprising capturing the stereo image in each one of a number of image frames, and correspondingly obtaining a rectified high dynamic range stereo image in each such image frame, and controlling exposure timing used for capturing the low and high exposure stereo images in each image frame as a function of intensity statistics determined from the high dynamic range stereo images or from the rectified high dynamic range stereo images.

11. The method of claim 1, wherein capturing the stereo image comprises capturing a first stereo image in one or more image frames using a first pair of image sensors operating with a first baseline, and further comprising capturing a redundant second stereo image in the one or more image frames using a second pair of image sensors operating with a second baseline, and wherein said method steps of correlating, converting, obtaining, accumulating and clustering are performed independently for the first and second stereo images, and wherein the method further includes deciding that an object intrusion has occurred if there is a threshold level of correspondence between object intrusions detected with respect to the first stereo images and object intrusions detected with respect to the second stereo images.

12. The method of claim 1, further comprising suppressing false detection of objection intrusions by re-computing depth information at a higher reliability for the ranged pixels in the depth map that were flagged for inclusion in the set of flagged pixels and evaluating the recomputed depth information to verify that each range pixel in the set of flagged pixels falls within the protection boundary.

13. The method of claim 1, further comprising suppressing epipolar errors in the clusters to obtain error-corrected clusters.

14. The method of claim 1, where the monitoring zone comprises two overlapping protection zones defined by respective protection boundaries, and further comprising monitoring a first one of the overlapping protection zones using detection parameters configured for detecting persons walking or running and monitoring a second one of the overlapping protection zones using detection parameters configured for detecting crawling and prone persons.

15. The method of claim 1, wherein clustering cells in the histogram to detect intrusions of objects within the protection boundary that meet a minimum object size threshold includes temporally filtering detected clusters to distinguish between clusters arising from noise and clusters arising from object intrusions.

16. A projective volume monitoring apparatus configured to detect objects intruding into a monitoring zone, said projective volume monitoring apparatus comprising:

image sensors configured to capture a stereo image; and

image processing circuits operatively associated with the image sensors and configured to;

correlate the stereo image to obtain a depth map comprising pixels represented in three-dimensional Cartesian coordinates;

convert the range pixels into spherical coordinates, so that each range pixel is represented as a radial distance along a respective pixel ray and a corresponding pair of solid angle values within the horizontal and vertical fields of view associated with capturing the stereo image;

obtain a set of flagged pixels by flagging those range pixels that fall within a protection boundary defined for the monitoring zone;

accumulate the flagged pixels into corresponding cells of a two-dimensional histogram that quantizes the solid angle ranges of the horizontal and vertical fields of view; and

cluster cells in the histogram to detect intrusions of objects within the protection boundary that meet a minimum object size threshold.

17. The projective volume monitoring apparatus of claim 16, wherein the image processing circuits are configured to cluster cells in the histogram by identifying qualified cells as those cells that accumulated at least a minimum number of the flagged pixels, and identifying qualified clusters as those clusters of qualified cells that meet a minimum cluster size corresponding to the minimum object size threshold.

18. The projective volume monitoring apparatus of claim 17, wherein the image processing circuits are configured to cluster cells in the histogram further by determining whether any qualified clusters persist for a defined window of time and, if so, determining that an object intrusion has occurred.

19. The projective volume monitoring apparatus of claim 17, wherein the image processing circuits are configured to identify qualified clusters for the stereo image captured in each of a number of successive image frames and to filter qualified clusters over time by determining whether a same qualified cluster persists over a defined number of image frames and, if so, determining that an object intrusion has occurred.

20. The projective volume monitoring apparatus of claim 19, wherein the image processing circuits are configured to filter qualified clusters over time further by determining whether equivalent qualified clusters persist over the defined number of image frames and, if so, determining that an object intrusion has occurred, wherein equivalent qualified clusters are those qualified clusters that split or merge within a same region of the histogram over the defined number of image frames.

21. The projective volume monitoring apparatus of claim 16, wherein the image processing circuits are configured to generate one or more sets of mitigation pixels from the stereo image, wherein the mitigation pixels represent pixels in the stereo image that are flagged as being faulty or unreliable, and integrate the one or more sets of mitigation pixels into the set of flagged pixels for accumulation in corresponding cells of the histogram, along with the range pixels flagged for inclusion in the set of flagged pixels, so that any given cluster includes cells that accumulated range pixels, mitigation pixels, or a mix of both.

22. The projective volume monitoring apparatus of claim 21, wherein the image processing circuits are configured to generate the one or more sets of mitigation pixels by detecting at least one of:

pixels in the stereo image that are outside of a usable dynamic range;

pixels in the stereo image that are stuck;

pixels in the stereo image that have reduced sensitivity;

pixels in the stereo image that are noisy; and

pixels in the stereo image that are identified as corresponding to shadowed regions of the monitoring zone.

23. The projective volume monitoring apparatus of claim 22, wherein the image processing circuits are configured to capture the stereo image by capturing corresponding high exposure and low exposure stereo images and forming a high dynamic range stereo image from the corresponding high and low exposure stereo images, and further are configured to obtain a rectified high dynamic range stereo image by rectifying the high dynamic range stereo image, so that correlation processing is performed on the rectified high dynamic range stereo image.

24. The projective volume monitoring apparatus of claim 23, wherein the image processing circuits are configured to flag pixels in the rectified high dynamic range stereo image as mitigation pixels that are deemed bad or unreliable for range detection purposes, and further configured to add the mitigation pixels to the set of flagged pixels, such that the image processing circuits perform clustering in consideration of both the range pixels and the mitigation pixels in the flagged set of pixels.

25. The projective volume monitoring apparatus of claim 23, wherein the image processing circuits are configured to capture the stereo image in each one of a number of successive image frames and to correspondingly obtain a rectified high dynamic range stereo image in each such image frame, and to control exposure timing used for capturing the low and high exposure stereo images in each image frame as a function of intensity statistics determined from the high dynamic range stereo images or from the rectified high dynamic range stereo images.

26. The projective monitoring apparatus of claim 16, wherein the projective monitoring apparatus includes a first pair of image sensors operating with a first baseline to capture the stereo image as a first stereo image obtained in each of one or more image frames, and further includes a second pair of image sensors operating with a second baseline to capture a redundant, second stereo image in each of the one or more image frames, and wherein the image processing circuits are configured to process the first and second stereo images independently, and to decide that an object intrusion has occurred if there is a threshold level of correspondence between object intrusions detected with respect to the first stereo images and object intrusions detected with respect to the second stereo images.

27. The projective monitoring apparatus of claim 16, wherein the image processing circuits are configured to suppress false detection of objection intrusions by re-computing depth information at a higher reliability for the ranged pixels in the depth map that were flagged for inclusion in the set of flagged pixels and evaluating the recomputed depth information to verify that each range pixel in the set of flagged pixels falls within the protection boundary.

28. The projective monitoring apparatus of claim 16, wherein the image processing circuits are configured to suppress epipolar errors in the clusters to obtain error-corrected clusters.

29. The projective monitoring apparatus of claim 16, where the monitoring zone comprises two overlapping protection zones defined by respective protection boundaries, and wherein the image processing circuits are configured to monitor a first one of the overlapping protection zones using detection parameters configured for detecting persons walking or running and to monitor a second one of the overlapping protection zones using detection parameters configured for detecting crawling and prone persons.

30. The projective monitoring apparatus of claim 16, wherein the image processing circuits are configured to detect object intrusions based on being configured to cluster cells in the histogram, detect any clusters that meet a minimum object size threshold, and to decide that an object intrusion has occurred if any of the detected clusters persist over a defined window of time.

31. The projective volume monitoring apparatus of claim 16, wherein the projective volume monitoring apparatus includes a sensor unit including a pair of image sensors configured to capture the stereo image and further including the image processing circuits, and wherein the projective volume monitoring apparatus further includes a control unit that is configured to communicate with the sensor unit, said control unit including control circuitry that controls one or more output signals responsive to intrusion detection signaling from the sensor unit, and wherein the image processing circuits of the sensor unit are configured to generate the intrusion detection signaling responsive to the detection of intrusions of objects within the protection boundary that meet the minimum object size threshold.

32. A method of protecting against the intrusion of crawling and walking persons into a monitoring zone comprising:

acquiring stereo images of a first monitoring zone that begins at a defined height above a floor and of a second monitoring zone that lies below the first monitoring zone and extends downward to a floor or other surface on which persons may walk or crawl;

processing the stereo images to obtain range pixels;

detecting object intrusions within the first monitoring zone by processing those range pixels corresponding to the first monitoring zone using first object detection parameters that are tuned for the detection of walking or running persons; and

detecting object intrusions within the second monitoring zone by processing those range pixels corresponding to the second monitoring zone using second object detection parameters that are tuned for the detection of crawling or prone persons.

33. The method of claim 32, wherein the first object detection parameters include a first minimum object size threshold that defines a minimum size for detectable objects within the first monitoring zone, and further include a first maximum object speed that defines a maximum speed for detectable objects within the first monitoring zone, and wherein the second object detection parameters include a second minimum object size threshold that is larger than the first minimum object size threshold, and further include a second maximum object speed that is lower than the first maximum object speed.

34. The method of claim 32, wherein the first monitoring zone begins about 420 mm above the floor or other surface, and wherein the second monitoring zone extends from 300 mm above the floor or other surface up to the beginning of the first monitoring zone, or up to the beginning of a defined protection zone.

35. The method of claim 32, wherein the first minimum object size threshold is at or about 200 mm in cross section, second minimum object size threshold is more than 200 mm in cross section, the first maximum object speed is about 1.6 m/s, and the second maximum object speed is about 0.8 m/s.