Collision avoidance system and method for an underground mine environment

• US 9,747,802 B2
• Filed: 03/10/2014
• Issued: 08/29/2017
• Est. Priority Date: 09/19/2011
• Status: Active Grant

• Current Assignee: Innovative Wireless Technologies Inc.
• Original Assignee: Innovative Wireless Technologies Inc.
• Inventors: Fisher, Matthew A., Carpenter, Paul R., Silverstrim, James E.
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): Fereja, Samuel D
• Application Number:
  US14/202,228
• Publication Number:
  US 20150254985A1
• Time in Patent Office: 1,268 Days
• Field of Search: None
• CPC Class Codes: B60K 28/04 responsive to presence or absence of the driver, e.g. to weight or lack thereof
• CPC Class Codes: B60K 28/10 responsive to conditions relating to the vehicle
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2556/20 Data confidence level
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/085 Taking automatic action to adjust vehicle attitude in preparation for collision, e.g. braking for nose dropping
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 30/0953 the prediction being responsive to vehicle dynamic parameters
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: H04N 23/23 from thermal infrared radiation
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described are methods and systems for collision avoidance in an underground mine environment that use one or more of a computer vision component, an asset tracking system, and a motion detection component for the purpose of determining and responding to potential collision threats. Imaging is captured and processed in real time, so that assets of interest can be identified and used in evaluating potential for collision with other assets. Location data from an asset tracking system is likewise evaluated and used to determine proximity of assets in relation to the host. A final input is provided by the motion detection component that intelligently determines movement patterns and direction of travel. Once these components'"'"' inputs are collectively evaluated, a proximity or a threat value is generated which determine an audible or visual signal or action to prevent collision and increase safety in unfavorable conditions.

Claims

1. A collision avoidance system, the system comprising:

a computer vision component comprising;

an imaging modality comprising one or more thermal infrared or ultraviolet cameras configured to provide an image capture of a region of interest;

a computer processor; and

a memory comprising a set of computer-executable instructions configured for instructing the computer processor to analyze the thermal image capture to identify assets present in the region of interest;

an asset tracking component based on fixed mesh radio nodes and mobile mesh radio nodes, wherein a mobile mesh radio node is placed on a first asset and the asset tracking component is configured to determine the location of the mobile mesh radio node based on a Received Signal Strength Indication (RSSI) between the mobile mesh radio node and surrounding fixed mesh radio nodes;

at least one motion detection component capable of determining a directional velocity component for the asset tracking component and comprising an accelerometer-based motion sensor device placed on the first asset, wherein the directional velocity component comprises a speed and direction of travel; and

a collision avoidance component which is configured to receive inputs from the computer vision component, the asset tracking component, and the motion detection component and combine the inputs into a collision avoidance algorithm programmed in a set of computer-executable instructions which instruct a computer processor to calculate a Threat Rating Value that determines a warning or action for the first asset to avoid collision with a second asset;

wherein the computer-executable instructions are configured to instruct the computer processor to calculate the Threat Rating Value as;

TRV=(K_VH·

A_VH·

V_VH)+(K_VO·

A_VO·

V_VO)+(K_TS·

max[TRV_TS1 . . . TRV_TSn])

wherein;

TRV_TS1=C_TS1·

D_TS1·

V_TS1

TRV_TSn=C_TSn·

D_TSn·

V_TSn

K_VH=Weight constant for a host computer vision component input

A_VH=Amplitude level for the host computer vision component input

V_VH=Value of the host computer vision component input

K_VO=Weight constant for an object computer vision component input

A_VO=Amplitude level for the object computer vision component input

V_VO=Value of the object computer vision component input

K_TS=Weight constant for the asset tracking component input

V_TS1=Value of a first asset tracking component input

TRV_TS1=Threat rating value for an n^th asset tracking component input

C_TSn=Confidence level for the n^th asset tracking component input

D_TSn=Directional velocity component for the n^th asset tracking component input

V_TSn=Value of the n^th asset tracking component input

TRV_TSn=Threat rating value for the n^th asset tracking component input

TRV=Threat rating value for the Collision Avoidance Component.

2. The collision avoidance system of claim 1, wherein the first asset is a vehicle or human and the second asset is a vehicle or human.

3. The collision avoidance system of claim 1, wherein object recognition is achieved by training the computer vision component with positive samples of objects to be detected and with negative samples wherein no objects to be detected exist,

wherein positive sample measurements are manually calibrated using the formula:

F=P*D/S where

F is the focal length of the camera;

P is the number of pixels wide of the subject;

D is the distance from the camera to the subject; and

S is the size of the subject.

4. The collision avoidance system of claim 1, wherein the collision avoidance algorithm is capable of detecting co-location of humans riding in a vehicle and overriding the warning or action on a user interface for the system and the mobile mesh radio nodes where a vehicle must be stopped for on-boarding and off-boarding of human assets.

5. The collision avoidance system of claim 1, wherein:

the motion tracking component comprises an accelerometer placed on each vehicle and each human asset that is configured to provide a speed, direction of travel and unique ID for each human and vehicle;

the imaging component comprises a thermal or ultraviolet imaging component on each vehicle to capture images in dark and dusty environments using one or more passive long wave infrared cameras and/or ultraviolet cameras configured for imaging one or more areas not otherwise capable of being seen by a driver to display real-time live video to the driver and real-time object recognition corresponding to known objects including humans, vehicles, and electrical infrastructure; and

the collision avoidance algorithm is on each vehicle and is configured to use position information, directional velocity information, a unique ID for each mobile object, and object recognition information as input values to calculate proximity between assets, speed and direction of travel between assets, and a threat rating value between the first asset and the second asset.

6. The collision avoidance system of claim 1, wherein the collision avoidance component is configured to receive inputs comprising object recognition information from the computer vision component, position information from the asset tracking component, and directional velocity information from the motion detection component and combine the inputs into the algorithm.

7. The system of claim 1, wherein the asset tracking component comprises a plurality of mobile mesh radio nodes and a plurality of fixed mesh radio nodes wherein:

1. a mobile mesh radio node is placed on each mobile vehicular asset and mobile human asset and the fixed mesh radio nodes are placed on fixed objects in the underground mine; and

2. the mobile mesh radio nodes and the fixed mesh radio nodes together form a Wireless Mesh Network capable of determining position information of each of the mobile vehicular assets and each of the mobile human assets in non-line-of-sight (NLOS) conditions in an underground mine environment based on a tracking algorithm that uses Received Signal Strength Indication (RSSI) calculations from the fixed mesh radio nodes which together comprise multiple surrounding fixed mesh radio nodes with known locations.

8. The system of claim 1, wherein the computer vision component comprises one or more camera and one or more object recognition algorithms.

9. The system of claim 1, wherein the computer vision component comprises one or more long wave infrared cameras or ultraviolet cameras and the imaging is capable of being captured at a rate of 1 to 100 frames per second to form a continual live video feed for analysis to perform real-time image processing and object recognition to determine short range line-of-sight collision threats, distance of threats, and/or speed of threats.

10. The system of claim 8, wherein one or more of the object recognition algorithms comprise appearance-based or feature-based techniques chosen from edges, gradients, Histogram of Oriented Gradients (HOG), Haar wavelets, linear binary patterns, extracted features and boosted learning algorithms, bag-of-words models, gradient-based and derivative-based matching approaches, Viola-Jones algorithm, template matching, image segmentation and blob analysis, local feature detectors, Speeded Up Robust Features (SURF), blob detection methods, or Maximally Stable Extremal Regions (MSER) and provides a realtime computer vision system capable of performing object recognition on thermal image frames received as a video feed for one or more of:

object recognition of humans;

object recognition of human faces;

object recognition of vehicles; and

object recognition of heat emitting infrastructure.

11. The system of claim 1, wherein the asset tracking component is configured to determine direction of travel and speed of a mobile mesh radio node using data from an accelerometer placed on the mobile node to measure directional velocity.

12. The system of claim 1, comprising collision avoidance software which provides user interface instructions for avoiding collisions between objects.

13. The system of claim 1, comprising at least one motion detection component comprising an accelerometer to provide directional velocity information for at least one mobile unit.

14. The system of claim 1, wherein the computer vision component comprises:

a video imaging modality configured to provide an image capture of an asset;

wherein the computer processor is configured to receive asset tracking information and video image frames for analysis to identify assets according to a set of computer-executable instructions stored in the memory.

15. The system of claim 1, wherein the computer-executable instructions are configured to calculate the proximity of a human form by the size and heat intensity of the thermal or ultraviolet image capture whereby a brighter image pixel indicates warmer areas and closer and larger humans span a greater number of pixels than do farther and smaller humans.

16. The system of claim 1, wherein:

in addition to at least one of the motion detection components being placed on the first asset, at least one of the motion detection components is placed on the second asset; and

inputs from the asset tracking component and one or more of the motion detection components are combined into the collision avoidance algorithm to determine an intersection point and time to intersection point of the first asset and the second asset.

17. A method for avoiding asset collisions, the method comprising:

thermal imaging a first asset with a computer vision component and identifying the type of asset with the computer vision component, the computer vision component comprising;

a video imaging component comprising one or more thermal infrared or ultraviolet cameras which provide video image frames comprising a thermal or ultraviolet image capture of an object;

a computer processor; and

a memory comprising a set of computer-executable instructions which instruct the computer processor to analyze video image frames received from the video imaging component to identify assets present in the thermal or ultraviolet image capture, wherein the set of computer-executable instructions employ object classification algorithms to identify the asset;

tracking the location of the first asset with an asset tracking component comprising fixed mesh radio nodes and mobile mesh radio nodes, wherein a mobile mesh radio node is placed on a first vehicle or a human and the asset tracking component determines the location of the mobile mesh radio node based on a Received Signal Strength Indication (RSSI) between the mobile mesh radio node and surrounding fixed mesh radio nodes;

tracking the speed and direction of travel of the first asset with a motion detection component which determines a directional velocity component for the asset tracking component based on an accelerometer-based motion sensor device placed on the first asset, wherein the directional velocity component comprises a speed and direction of travel;

determining a Threat Rating Value through a collision avoidance component which receives inputs from the computer vision component, asset tracking component, and motion detection component and combines the inputs into a collision avoidance algorithm programmed in a set of computer-executable instructions which instruct a processor to calculate the Threat Rating Value; and

issuing a warning or instruction for action for the first asset to avoid collision with a second asset based on the Threat Rating Value;

wherein the Threat Rating Value is calculated as;

TRV=(K_VH·

A_VH·

V_VH)+(K_VO·

A_VO·

V_VO)+(K_TS·

max[TRV_TS1 . . . TRV_TSn])

wherein;

TRV_TS1=C_TS1·

D_TS1·

V_TS1

TRV_TSn=C_TSn·

D_TSn·

V_TSn

K_VH=Weight constant for a host computer vision component input

A_VH=Amplitude level for the host computer vision component input

V_VH=Value of the host computer vision component input

K_VO=Weight constant for an object computer vision component input

A_VO=Amplitude level for the object computer vision component input

V_VO=Value of the object computer vision component input

K_TS=Weight constant for the asset tracking component input

V_TS1=Value of a first asset tracking component input

TRV_TS1=Threat rating value for an n^th asset tracking component input

C_TSn=Confidence level for the n^th asset tracking component input

D_TSn=Directional velocity component for the n^th asset tracking component input

V_TSn=Value of the n^th asset tracking component input

TRV_TSn=Threat rating value for the n^th asset tracking component input

TRV=Threat rating value for the Collision Avoidance Component.

18. The method of claim 17, wherein the first asset is a vehicle or human and the second asset is a vehicle or human.

19. The method of claim 17, wherein the collision avoidance component receives inputs comprising object recognition information from the computer vision component, position information from the asset tracking component, and directional velocity information from the motion detection component and combines the inputs into the collision avoidance algorithm.