System and method for managing mixed fleet worksites using video and audio analytics

• US 9,685,009 B2
• Filed: 04/01/2015
• Issued: 06/20/2017
• Est. Priority Date: 04/01/2015
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Sprock, Christopher, Baumann, Ryan, Talmaki, Sanat, Halepatali, Praveen
• Primary Examiner(s): Nguyen, Cuong H
• Application Number:
  US14/676,243
• Publication Number:
  US 20160292933A1
• Time in Patent Office: 811 Days
• Field of Search: 701 293, 701 31, 701 319, 701 337, 701 50, 348143, 348 61, 382103, 340438
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 50/08 Construction
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera

Abstract

Systems and methods for managing and optimizing mixed fleet worksite operations based on video and or audio data are disclosed. One method includes receiving one or more models relating to a fleet of machines at the worksite, wherein the fleet of machines comprises an in-network machine and an out-of-network machine, receiving first sensor data associated with the out-of-network machine at the worksite, receiving second sensor data associated with the in-network machine at the worksite, determining a machine state of each of the in-network machine and the out-of-network machine based at least on the first sensor data and the second sensor data, comparing the determined machine states to a modeled machine state represented by the received one or more models to classify site operations and/or detect an irregularity in site operations or an inefficiency in site operations, and generating a response based at least on the detected irregularity or inefficiency.

Claims

1. A method for managing mixed-fleet worksites, comprising:

receiving, by one or more processors of a central station, one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with the central station, and an out-of-network machine that is not configured to communicate with the central station;

receiving, by the one or more processors of the central station, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;

receiving, by the one or more processors of the central station, second sensor data associated with the in-network machine at the worksite;

determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a ground crew location relative to machine;

comparing the determined machine states to a modeled machine state represented by the received one or more models to detect a non-failure irregularity in site operations or an inefficiency in site operations;

generating a response based at least on the detected irregularity or inefficiency, the response including a warning or a remote reconfiguration of operational parameters of the in-network machine; and

transmitting the response to an operator, a display at the worksite, or to the in-network machine,

wherein the one or more models include a machine model, a payload model or a worksite model

wherein the worksite includes a mining site or a construction site,

wherein the out-of-network machine is an out-of-network digging machine, an out-of-network loading machine or an out-of-network hauling machine,

wherein the in-network machine is an in-network digging machine, an in-network loading machine or an in-network hauling machine,

wherein the out-of-network hauling machine and in-network hauling machine are each configured to carry excavated materials between different locations at the worksite.

3. The method of claim 1, wherein the one or more models comprises a worksite model and the method further comprises updating the worksite model based at least on the generated response.

4. The method of claim 1, wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

5. The method of claim 1, wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

6. The method of claim 1, wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

7. The method of claim 1, wherein the determining the machine state comprises generating a current signature representing the machine state.

8. The method of claim 1, wherein the response comprises one or more of an audible indicator and a visual indicator.

2. A computer program product comprising a non-transitory storage medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for managing a mixed-fleet worksite, the method comprising:

receiving, by the one or more processors, one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with a central station and an out-of-network machine that is not configured to communicate with the central station;

receiving, by the one or more processors, first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;

receiving, by the one or more processors, second sensor data associated with the in-network machine at the worksite;

determining, by the one or more processors, a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a ground crew location relative to machine;

comparing the determined machine states to a modeled machine state represented by the received one or more models to detect a non-failure irregularity in site operations or an inefficiency in site operations;

generating a response based at least on the detected irregularity or inefficiency, the response including a warning or a remote reconfiguration of operational parameters of the in-network machine; and

transmitting the response to an operator, a display at the worksite, or to the in-network machine,

wherein the one or more models include a machine model, a payload model or a worksite model

wherein the worksite includes a mining site or a construction site,

wherein the out-of-network machine is an out-of-network digging machine, an out-of-network loading machine or an out-of-network hauling machine,

wherein the in-network machine is an in-network digging machine, an in-network loading machine or an in-network hauling machine,

wherein the out-of-network hauling machine and in-network hauling machine are each configured to carry excavated materials between different locations on the worksite.

15. The computer readable program product of claim 2, wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

16. The computer readable program product of claim 2, wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

17. The computer readable program product of claim 2, wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

9. A system comprising:

a processor configured to

receive one or more models relating to a worksite or a fleet of machines at the worksite, the fleet of machines including an in-network machine configured to communicate with a central station and an out-of-network machine that is not configured to communicate with the central station;

receive first sensor data associated with the out-of-network machine at the worksite, the first sensor data comprising one or more of image data and audio data;

receive second sensor data associated with the in-network machine at the worksite;

determine a machine state of each of the in-network machine and the out-of-network machine, the machine state of the out-of-network machine based at least on a comparison of a feature of an object detected in the image data of the first sensor data to a signature that represents a known or learned machine state, the machine state of the in-network machine based at least on the second sensor data, the machine state one of a full load, an empty load, a payload material type, a payload to air ratio, a payload placement, a payload compaction, a payload water content, a material amount moved, a drop placement, an excavator position, an idle state, a swing state, a dump position, a deformation of machine, an operator characteristic and a ground crew location relative to machine; and

compare the determined machine states to a modeled machine state represented by the received one or more models to classify one or more site operations,

wherein the one or more models include a machine model, a payload model or a worksite model,

wherein the worksite includes a mining site or a construction site,

wherein the out-of-network machine is an out-of-network digging machine, an out-of-network loading machine or an out-of-network hauling machine,

wherein the in-network machine is an in-network digging machine, an in-network loading machine or an in-network hauling machine,

wherein the out-of-network hauling machine and in-network hauling machine are each configured to carry excavated materials between different locations at the worksite.

10. The system of claim 9, wherein the one or more models comprises a worksite model and the processor is further configured to update the worksite model based at least on the classified one or more site operations.

11. The system of claim 9, wherein the first sensor data is received via an onsite sensor and the second sensor data is received via an offsite sensor.

12. The system of claim 9, wherein the first sensor data is received via an offsite sensor and the second sensor data is received via an onsite sensor.

13. The system of claim 9, wherein one or more of the determined machine state and the modeled machine state comprise a characteristic of a material load.

14. The system of claim 9, wherein the determining the machine state comprises generating a current signature representing the machine state.