Operations monitoring in an area

• US 10,202,135 B2
• Filed: 05/16/2014
• Issued: 02/12/2019
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: International Electronic Machines Corp.
• Original Assignee: International Electronic Machines Corp.
• Inventors: Mian, Zahid F., Gamache, Ronald W., Spoor, Ryk E.
• Primary Examiner(s): Werner, David N
• Application Number:
  US14/279,385
• Publication Number:
  US 20140339374A1
• Time in Patent Office: 1,733 Days
• CPC Class Codes: B61L 23/041 Obstacle detection
• CPC Class Codes: B61L 29/30 Supervision, e.g. monitoring arrangements
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/1436 with motion detection
• CPC Class Codes: G08G 1/005 including pedestrian guidance indicator
• CPC Class Codes: G08G 1/017 identifying vehicles G08G1/015, G08G1/054 take precedence
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/04 : using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

An assembly for monitoring an area is provided. The assembly can include two or more cameras sensitive to radiation of distinct wavelength ranges. The fields of view of the cameras can be substantially co-registered at the area to be monitored. The assembly can include a computer system which can process the image data to monitor the area. The computer system can be configured to identify relevant objects present in the area, update tracking information for the relevant objects, and evaluate whether an alert condition is present using the tracking information.

Claims

1. A system comprising:

a monitoring assembly deployed to monitor an area including a restricted path periodically used by authorized vehicles, wherein the monitoring assembly includes;

a plurality of cameras including at least two cameras sensitive to radiation of distinct wavelength ranges, wherein the fields of view of the at least two cameras are substantially co-registered at the area; and

a computer system configured to monitor the area by performing a method including;

operating the plurality of cameras to acquire substantially concurrently image data of the area;

identifying, prior to an authorized vehicle entering the area, a set of relevant objects in the area using the image data acquired by the plurality of cameras, wherein the identifying includes evaluating a size of an object at least partially visible in the image data, a shape of the object, and a consistency of movement of the object, to classify the object as either a relevant object associated with at least one alert condition or an irrelevant object not associated with any alert condition;

updating tracking information for each object in the set of relevant objects; and

evaluating an alert condition using the updated tracking information for each of the set of relevant objects, wherein the evaluating includes evaluating a presence of a relevant object in a restricted zone corresponding to the restricted path with respect to the alert condition using tracking information regarding;

an amount of time the relevant object has been in the restricted zone, a manner in which the relevant object entered the restricted zone, and a manner in which the relevant object has been moving within the area.

2. The system of claim 1, wherein the identifying includes:

independently processing the image data acquired by each of the plurality of cameras to identify objects present in the area;

fusing similar objects identified in the image data acquired by each of the plurality of cameras; and

verifying objects identified in the image data acquired by only a portion of the plurality of cameras.

3. The system of claim 1, wherein the identifying a set of relevant objects further includes classifying an object identified in the area as being relevant or not relevant based on an orientation of the object.

4. The system of claim 1, wherein the identifying a set of relevant objects further includes classifying an object identified in the area as being relevant or not relevant based on a temperature difference between the object and at least one other object in the image data.

5. The system of claim 1, wherein the computer system is further configured to self-calibrate using a set of static features in the area.

6. The system of claim 5, wherein the set of static features includes at least one of:

rails or ties for railroad tracks.

7. The system of claim 1, wherein the evaluating an alert condition includes discriminating between possible objects having similar appearance but different threat levels using at least one of:

tracking data for the object in the monitored area or tracking data for another object associated with the object in the restricted area.

8. The system of claim 1, wherein the evaluating further uses an amount of time to a critical event.

9. The system of claim 8, wherein the restricted area comprises an at-grade railroad crossing and the critical event comprises a train using the at-grade railroad crossing.

10. The system of claim 1, wherein the plurality of distinct wavelength ranges includes:

near infrared and visible radiation and infrared radiation, the monitoring assembly further comprising a near infrared illuminator, wherein the method further includes operating the near infrared illuminator to provide sufficient light for imaging the area with the first camera.

11. A method comprising:

acquiring multispectral image data of an area on a computer system of a monitoring assembly including a plurality of cameras including at least two cameras sensitive to radiation of distinct wavelengths, wherein the area includes a restricted path periodically used by authorized vehicles;

the computer system identifying, prior to an authorized vehicle entering the area, a set of relevant objects in the area using the multispectral image data, wherein the identifying includes evaluating a size of an object at least partially visible in the image data, a shape of the object, and a consistency of movement of the object, to classify the object as either a relevant object associated with at least one alert condition or an irrelevant object not associated with any alert condition;

the computer system updating tracking information for each object in the set of relevant objects;

the computer system evaluating an alert condition using the updated tracking information for each of the set of relevant objects, wherein the evaluating includes evaluating a presence of a relevant object in a restricted zone corresponding to the restricted path with respect to the alert condition using tracking information regarding;

an amount of time the relevant object has been in the restricted zone, a manner in which the relevant object entered the restricted zone, and a manner in which the relevant object has been moving within the area; and

the computer system performing an action response to evaluating a presence of an alert condition.

12. The method of claim 11, further comprising the computer system self-calibrating and registering two-dimensional pixel data of the area into a three-dimensional model of the area using a plurality of static features in the area.

13. The method of claim 12, wherein the plurality of static features include at least one of:

rails or ties for railroad tracks in the area.

14. The method of claim 11, wherein the identifying a set of relevant objects further includes classifying an object identified in the area as being relevant or not relevant based on an orientation of the object.

15. The method of claim 11, wherein the identifying a set of relevant objects further includes classifying an object identified in the area as being relevant or not relevant based on a temperature difference between the object and at least one other object in the image data.

16. The method of claim 11, wherein the evaluating an alert condition includes discriminating between possible objects having similar appearance but different threat levels using at least one of:

tracking data for the object in the monitored area or tracking data for another object associated with the object in the restricted area.

17. An at-grade railroad crossing comprising:

a road intersecting with railroad tracks; and

a monitoring assembly deployed to monitor the railroad crossing, wherein the monitoring assembly includes;

a first camera sensitive to near infrared and visible radiation, wherein each image acquired by the first camera is based on sensed near infrared and visible radiation;

a second camera sensitive to infrared radiation, wherein the fields of view of the first and second cameras are substantially co-registered at the intersection of the road and the railroad tracks; and

a computer system configured to monitor the railroad crossing by performing a method including;

identifying, prior to a rail vehicle using the railroad tracks entering the area, a set of relevant objects in the area using image data substantially concurrently acquired by the first and second cameras, wherein the identifying includes evaluating a size of an object at least partially visible in the image data, a shape of the object, and a consistency of movement of the object, to classify the object as either a relevant object associated with at least one alert condition or an irrelevant object not associated with any alert condition;

updating location and tracking information for each object in the set of relevant objects; and

evaluating an alert condition using the location and tracking information for each of the set of relevant objects and an amount of time until a train next uses the railroad crossing, wherein the evaluating includes evaluating a presence of a relevant object in a restricted zone with respect to the alert condition using tracking information regarding;

an amount of time the relevant object has been in the restricted zone, a manner in which the relevant object entered the restricted zone, and a manner in which the relevant object has been moving within the area.

18. The crossing of claim 17, wherein the updating uses a registration map mapping the image data to a three-dimensional model of the railroad crossing.

19. The crossing of claim 17, wherein the identifying a set of relevant objects further includes:

identifying all objects visible in the image data;

determining a size and orientation of each object visible in the image data; and

classifying an object as not relevant based on at least one of;

the size and orientation of the object or tracking information corresponding to movement of the object.

20. The crossing of claim 17, the monitoring assembly further comprising a near infrared illuminator, wherein the method further includes operating the near infrared illuminator to provide sufficient light for imaging the railroad crossing with the first camera.