SMART AREA MONITORING WITH ARTIFICIAL INTELLIGENCE

• US 20190294889A1
• Filed: 03/26/2019
• Published: 09/26/2019
• Est. Priority Date: 03/26/2018
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Sriram, Parthasarathy, Kumar, Ratnesh, Aghdasi, Farzin, Toorians, Arman, Naphade, Milind, Biswas, Sujit, Kolar, Vinay, Pisupati, Bhanu, Bartholomew, Aaron
• Granted Patent: US 11,182,598 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30264 Parking
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/625 : License plates
• CPC Class Codes: G06V 2201/08 : Detecting or categorising vehicles
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The present disclosure provides various approaches for smart area monitoring suitable for parking garages or other areas. These approaches may include ROI-based occupancy detection to determine whether particular parking spots are occupied by leveraging image data from image sensors, such as cameras. These approaches may also include multi-sensor object tracking using multiple sensors that are distributed across an area that leverage both image data and spatial information regarding the area, to provide precise object tracking across the sensors. Further approaches relate to various architectures and configurations for smart area monitoring systems, as well as visualization and processing techniques. For example, as opposed to presenting video of an area captured by cameras, 3D renderings may be generated and played from metadata extracted from sensors around the area.

Claims

1. A method comprising:

learning, by a machine learning model, a region of interest for one or more designated spaces based at least in part on observing object behavior patterns that correspond to at least one of a frequency of occupancy or a duration of occupancy of one or more objects in an environment;

receiving image data representative of a field of view of at least one image sensor, the image data representative of an object and the one or more designated spaces within the field of view;

determining, from the image data, an object region that includes at least a portion of the object;

determining an amount of overlap from a perspective of the at least one image sensor between the object region and the region of interest that corresponds to a designated space of the one or more designated spaces;

determining an occupancy status for the designated space based at least in part on the amount of overlap; and

generating metadata representative of the occupancy status for the designated space.

2. The method of claim 1, wherein the occupancy status comprises a level of confidence in the object at least partially occupying the designated space, the level of confidence being based at least in part on the amount of overlap.

3. The method of claim 1, wherein the amount of overlap is a measurement associated with a proportion of the region of interest that is occluded by the object region in the field of view of the image sensor.

4. The method of claim 1, wherein the object comprises a vehicle and the determining the object region includes detecting the object in the image data using one or more machine learning models (MLMs) trained to identify one or more of a vehicle back or a vehicle front.

5. The method of claim 1, wherein the learning of the region of interest includes:

receiving first image data representative of the field of view at a first time;

determining, from the first image data, a first object bounding box relative to the perspective of the at least one image sensor based at least in part on a first object detection;

receiving second image data representative of the field of view at a second time;

determining, from the second image data, a second object bounding box relative to the perspective of the at least one image sensor based at least in part on a second object detection;

aggregating the first object bounding box with the second object bounding box; and

based at least in part on the aggregating, generating the region of interest for the designated space.

6. The method of claim 1, further comprising:

receiving initial image data representative of the field of view over a period of time;

determining, from the initial image data, locations of object regions that correspond to respective ones of the designated spaces based at least in part on at least one of the object or different objects being detected within the field of view at the locations of the object regions over the period of time; and

determining a horizon line that divides a first area of the field of view including the locations of the object regions from a second area of the field of view not including the locations of the object regions,

wherein the determining of the occupancy status is based at least in part on the object region being at least partially within the first area of the field of view.

7. The method of claim 1, further comprising identifying, from the image data, a first surface within the field of view and associated with the one or more designated spaces, and a second surface within the field of view and associated with an aisle that extends along the one or more designated spaces, wherein the learning of the region of interest includes identifying the object behavior patterns within the first surface based at least in part on detecting the one or more objects within the first surface.

8. The method of claim 1, wherein the field of view comprises a fish-eye view of a first row of the one or more designated spaces, a second row of the one or more designated spaces, and an aisle separating the first row from the second row.

9. A method comprising:

receiving first image data representative of a first field of view of a first image sensor and second image data representative of a second field of view of a second image sensor;

identifying first image coordinates of a first object from the first image data based at least in part on detecting the first object in a first region of interest of the first field of view;

tracking, based at least in part on the first image coordinates, a first trajectory of a first object in the first region of interest;

identifying second image coordinates of a second object from the second image data based at least in part on detecting the second object in a second region of interest of the second field of view;

tracking, based at least in part on the second image coordinates, a second trajectory of the object in the second region of interest; and

generating a combined trajectory from the first trajectory and the second trajectory based at least in part on determining that the first object and the second object are a same object.

10. The method of claim 9, wherein the first region of interest and the second region of interest correspond to an aisle in a monitored area.

11. The method of claim 9, wherein the determining that the first object and the second object are a same object comprises evaluating attributes associated with the first trajectory and the second trajectory, the attributes for each trajectory comprising one or more of an object speed, a vehicle license plate number, a vehicle make, a vehicle model, or person information for a detected face.

12. The method of claim 9, wherein the receiving of the first image data, the identifying of the first image coordinates, and the tracking of the first trajectory are each performed by a first data processing stream and the receiving of the second image data, the identifying of the second image coordinates, and the tracking of the second trajectory are each performed by a second data processing stream that operates in parallel to the first data processing stream.

13. The method of claim 9, wherein the generating of the combined trajectory includes clustering location coordinates from the first trajectory with location coordinates from the second trajectory for the same object.

14. The method of claim 9, wherein the generating of the combined trajectory includes assigning one or more object attributes of the first trajectory or the second trajectory to the combined trajectory.

15. The method of claim 9, wherein the first image coordinates and the second image coordinates each correspond to the same object located in an overlapping portion of the first region of interest and the second region of interest.

16. A method comprising:

receiving first global coordinates of an object in a monitored area that correspond to first image coordinates of the object as depicted in a first field of view of at least a first image sensor;

receiving second global coordinates of the object in the monitored area that correspond to second image coordinates of the object as depicted in a second field of view of at least a second image sensor;

grouping at least the first global coordinates and the second global coordinates into a cluster based at least in part on evaluating attributes associated with the first global coordinates and the second global coordinates; and

generating at least a portion of a trajectory of the object in the monitored area based at least in part on the cluster.

17. The method of claim 16, wherein the evaluating comprises computing a distance between the first global coordinates and the second global coordinates.

18. The method of claim 16, wherein the grouping of the first global coordinates and the second global coordinates into the cluster is based at least in part on locations of the first image sensor and the second image sensor in the monitored area.

19. The method of claim 16, wherein the attributes associated with the first global coordinates and the second global coordinates identify one or more sensors used to generate one or more of the first global coordinates or the second global coordinates.

20. The method of claim 16, wherein the grouping is based at least in part on analyzing overlap data indicating which sensors are capable of detecting a same object.