Video content analysis system and method for transportation system

• US 10,392,038 B2
• Filed: 05/15/2017
• Issued: 08/27/2019
• Est. Priority Date: 05/16/2016
• Status: Active Grant

• Current Assignee: Wi-Tronix, LLC
• Original Assignee: Wi-Tronix, LLC
• Inventors: Jordan, Lawrence B., Patel, Savankumar V., Mueller, Jeffrey A., Rathinavel, Jagadeeswaran, Martinez, Roger
• Primary Examiner(s): Shin, Soo
• Application Number:
  US15/595,712
• Publication Number:
  US 20170327138A1
• Time in Patent Office: 834 Days
• Field of Search: None
• CPC Class Codes: B61L 15/0027 Radio-based, e.g. using GSM-R
• CPC Class Codes: B61L 15/0072 On-board train data handling
• CPC Class Codes: B61L 15/0081 On-board diagnosis or maintenance
• CPC Class Codes: B61L 15/0094 Recorders on the vehicle
• CPC Class Codes: B61L 23/041 Obstacle detection
• CPC Class Codes: B61L 23/047 Track or rail movements
• CPC Class Codes: B61L 27/40 Handling position reports or trackside vehicle data
• CPC Class Codes: B61L 27/57 for vehicles or trains, e.g. trackside supervision of train conditions
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0841 Registering performance data recording measured values G01D; information storage G11B
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 23/617 Upgrading or updating of programs or applications for camera control
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A video content analysis system for mobile assets that includes track detection and infrastructure monitoring component. The track detection and infrastructure monitoring includes a reinforcement learning component, an object detection and location component, and an obstruction detection component to analyze video data, audio data, vehicle data, weather data, and route/manifest data to determine internal and/or external conditions relating to an the asset. A data acquisition and recording system uploads the data, internal and/or external condition information, object detection information, object location information, and obstruction detection information to a remote memory module and provides streaming video data in real-time to a remotely located user. Remotely located users can view the data in various view modes through a web browser or virtual reality device, which provides for quicker emergency response, validates the effectiveness of repairs and rerouting, and monitors crew performance and safety.

Claims

1. A method for processing data from a mobile asset comprising:

receiving, using a video analytics system onboard the mobile asset, data based on at least one data signal from at least one of;

at least one data source onboard the mobile asset, comprising at least one of a mobile asset data component, at least one microphone, at least one fixed camera, and at least one 360 degree camera; and

at least one data source remote from the mobile asset, comprising at least one of a weather component, a route, a manifest, and geographic information system information received from a geographic information system (GIS), and a remote data repository;

processing, using an artificial intelligence component of the video analytics system, the data into processed data;

sending, using the video analytics system, at least one of the data and the processed data to a data recorder onboard the mobile asset;

encoding, using a data encoder of the data recorder, a first record based on the processed data; and

storing, using an onboard data manager of the data recorder, at least one of the data, the processed data, and the first record at a configurable first predetermined rate in at least one local memory of the data recorder, wherein the first predetermined rate is greater than zero seconds and up to five minutes.

2. The method of claim 1, wherein the data received from the mobile asset data component includes global positioning system data received from a global positioning system (GPS) and inertial sensor data received from an inertial sensor.

3. The method of claim 1, wherein the data comprises at least one of video information received from at least one of the at least one fixed camera and the at least one 360 degree camera and audio information received from the at least one microphone.

4. The method of claim 1, further comprising:

determining, using an object detection and location component of the video analytics system, at least one of first object detection data and first object location data of a first object based on the processed data; and

identifying, using the object detection and location component, an internal condition involving the mobile asset based on at least one of the first object detection data on a first condition that the first object detection data was determined and the first object location data on a second condition that the first object location data was determined.

5. The method of claim 1, further comprising:

receiving, using a digital video recorder onboard the mobile asset, multimedia data based on at least one data signal from at least one of;

the at least one 360 degree camera;

the at least one fixed camera; and

the at least one microphone;

receiving, using the data recorder, the multimedia data;

encoding, using the data encoder of the data recorder, a second record based on the multimedia data; and

storing, using the onboard data manager of the data recorder, at least one of the multimedia data and the second record at the configurable first predetermined rate in the at least one local memory of the data recorder.

6. The method of claim 4, further comprising:

determining, using an obstruction detection component of the video analytics system, obstruction detection data based on at least one of the processed data, the first object detection data on a third condition that the first object detection data was determined, and first the object location data on a fourth condition that the first object location data was determined; and

identifying, using the obstruction detection component, an external condition related to the mobile asset based on the obstruction detection data.

7. The method of claim 4, wherein at least one of the internal condition comprises cab occupancy of the mobile asset and the external condition comprises track detection.

8. The method of claim 4, further comprising:

receiving, using the data recorder, at least one of the internal condition, the first object detection data on a third condition that the first object detection data was determined, and the first object location data on a fourth condition that the first object location data was determined;

encoding, using the data encoder of the data recorder, a second record based on the at least one of the internal condition, the first object detection data on the third condition that the first object detection data was determined, and the first object location data on the fourth condition that the first object location data was determined; and

storing, using the onboard data manager of the data recorder, at least one of the internal condition, the first object detection data on the third condition that the first object detection data was determined, the first object location data on the fourth condition that the first object location data was determined, and the second record at the configurable first predetermined rate in the at least one local memory of the data recorder.

9. The method of claim 6, wherein the obstruction detection data is based on a fifth condition that at least one of the first object location data, on a sixth condition that the first object location data was determined, and the first object detection data, on a seventh condition that the first object detection data was determined, corresponds to a second object location data and a second object detection data of a second object.

10. The method of claim 6, further comprising:

receiving, using the data recorder, at least one of the external condition and the obstruction detection data;

encoding, using the data encoder of the data recorder, a second record based on the at least one of the external condition and the obstruction detection data; and

storing, using the onboard data manager of the data recorder, at least one of the external condition, the obstruction detection data, and the second record at the configurable first predetermined rate in the at least one local memory of the data recorder.

11. A method for processing data from a mobile asset comprising:

receiving, using a video analytics system onboard the mobile asset, data based on at least one data signal from at least one of;

at least one data source onboard the mobile asset, comprising at least one of mobile asset data component, at least one microphone, at least one fixed camera, and at least one 360 degree camera; and

at least one data source remote from the mobile asset, comprising at least one of a weather component, a route, a manifest, and geographic information system information received from a geographic information system (GIS), and a remote data repository;

processing, using an artificial intelligence component of the video analytics system, the data into processed data;

sending, using the video analytics system, at least one of the data and the processed data to a data recorder onboard the mobile asset;

encoding, using a data encoder of the data recorder, a first record based on the processed data;

storing, using an onboard data manager of the data recorder, at least one of the data, the processed data, and the first record at a configurable first predetermined rate in at least one local memory of the data recorder; and

the first predetermined rate is configurable between zero seconds and five minutes, wherein at least one of the data, the processed data, and the first record is stored continuously in the at least one local memory of the data recorder on a first condition that the first predetermined rate is zero seconds and at least one of the data, the processed data, and the first record is stored in specified time intervals based on the first predetermined rate in the at least one local memory of the data recorder on a second condition that the first predetermined rate is greater than zero seconds and up to five minutes.

12. A method for processing data from a mobile asset comprising:

receiving, using a video analytics system onboard the mobile asset, data based on at least one data signal from at least one of;

at least one data source onboard the mobile asset, comprising at least one of mobile asset data component, at least one microphone, at least one fixed camera, and at least one 360 degree camera; and

at least one data source remote from the mobile asset, comprising at least one of a weather component, a route, a manifest, and geographic information system Information received from a geographic information system (GIS), and a remote data repository;

processing, using an artificial intelligence component of the video analytics system, the data Into processed data;

sending, using the video analytics system, at least one of the data and the processed data to a data recorder onboard the mobile asset;

encoding, using a data encoder of the data recorder, a first record based on the processed data;

storing, using an onboard data manager of the data recorder, at least one of the data, the processed data, and the first record at a configurable first predetermined rate in at least one local memory of the data recorder;

sending, using the onboard data manager, the first record to a remote data manager via a wireless data link at a configurable second predetermined rate, the second predetermined rate is configurable between zero seconds and five minutes, wherein the first record is sent continuously to the remote data manager on a first condition that the second predetermined rate is zero seconds and the first record is sent to the remote data manager in specified time intervals based on the second predetermined rate on a second condition that the second predetermined rate is greater than zero seconds and up to five minutes; and

storing, using the remote data manager, the first record in a remote data repository.

13. A system for analyzing video content comprising:

at least one of at least one 360 degree camera, at least one fixed camera, and at least one microphone;

a video analytics system onboard a mobile asset, the video analytics system comprising an artificial intelligence component, an object detection and location component, and an obstruction detection component, the video analytics system adapted to receive data based on at least one data signal from the at least one of the at least one 360 degree camera, the at least one fixed camera, and the at least one microphone, the artificial intelligence component adapted to process the data into processed data, the object detection and location component adapted to determine object detection data and object location data of a first object based on the processed data, and the obstruction detection component adapted to determine obstruction detection information based on at least one of the processed data, the object detection information, and the object location information;

a digital video recorder, onboard the mobile asset, adapted to receive data based on at least one data signal from the at least one of the at least one 360 degree camera, the at least one fixed camera, and the at least one microphone; and

a data recorder, onboard the mobile asset, comprising a data encoder, an onboard data manager, and at least one local memory, the data recorder adapted to store the data at a configurable first predetermined rate in the at least one local memory, wherein the first predetermined rate is greater than zero seconds and up to five minutes.

14. The system of claim 13, further comprising:

a vehicle data component, onboard the mobile asset, comprising a global positioning system (GPS) and an inertial sensor, the vehicle data component adapted to send at least one of GPS data received from the GPS and inertial sensor data received from the inertial sensor to the video analytics system;

a weather component adapted to send at least one of the current weather information and the forecasted weather information to the video analytics system;

geographical information system (GIS) adapted to send at least one of route information, crew information, manifest information, and GIS information to the video analytics system; and

wherein the artificial intelligence component is adapted to use at least one of the GPS data, the inertial sensor data, the current weather information, the forecasted weather information, the route information, the crew information, the manifest information, and the GIS information to process the data into processed data.

15. The system of claim 13, further comprising:

a data recorder onboard the mobile asset comprising at least one local memory, an onboard data manager, and a data encoder, the data recorder adapted to receive at least one of the processed data, the object detection information, the object location information, and the obstruction information from the video analytics system;

the data encoder adapted to encode a record based on at least one of the processed data, the object detection information, the object location information, and the obstruction information; and

the onboard data manager adapted to store at least one of the record, the processed data, the object detection information, the object location information, and the obstruction information at a configurable first predetermined rate in the at least one local memory.

16. The system of claim 15, further comprising:

a remote data manager remote from the mobile asset, the remote data manager adapted to receive the record from the onboard data manager via a wireless data link at a configurable second predetermined rate, wherein the second predetermined rate is configurable between zero seconds and five minutes; and

a remote data repository remote from the mobile asset, the remote data repository adapted to store the record received from the remote data manager.

17. The system of claim 16, further comprising:

a data decoder remote from the mobile asset, the data decoder adapted to receive the record from the remote data repository and decode the record; and

an external monitoring component remote from the mobile asset, the external monitoring component adapted to identify at least one of the object detection information, the object location information, and the obstruction information.

18. The system of claim 16, further comprising:

a web client comprising a display device;

a web server in wireless communication with the web client, the web server adapted to receive a request comprising specified data relating to the mobile asset and a specified view mode;

a localizer in wireless communication with the web server, the localizer adapted to receive the specified data from the data decoder and modify the specified data based on a time setting and unit of measure setting specified by a remote user, the specified data based on at least one of the record, the processed data, the object detection information, the object location information, and the obstruction information;

the web server adapted to receive the specified data; and

the display device adapted to display the specified data in the specified view.