System and method for determination of machine state based on video and audio analytics
First Claim
1. A method for determining machine state, the method comprising:
- receiving, by one or more processors, one or more predetermined models relating to a worksite or a machine at the worksite;
receiving, by the one or more processors, first sensor data associated with the machine at the worksite, the first sensor data comprising one or more of image data and audio data;
receiving, by the one or more processors, second sensor data associated with an operation of the machine at the worksite, wherein the second sensor data is sourced from a sensor that is different from a sensor sourcing the first sensor data;
determining, by the one or more processors, a machine state based at least on the first sensor data and the second sensor data, wherein the machine state includes at least one of a full load, empty load, payload material type, payload vs. air ratio, payload placement, payload compaction, payload water content, material amount moved, drop placement, excavator position, idle state, swing, dump position, deformation of machine, operator characteristic, and ground crew location relative to machine;
comparing the determined machine state to a modeled machine state represented by the received one or more predetermined models to detect an irregularity in site operations or an inefficiency in site operations; and
generating a response based at least on the detected irregularity or inefficiency and transmitting the response to one or more of an operator and a central station.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods for analyzing and optimizing worksite operations based on video and or audio data are disclosed. One method includes receiving one or more models relating to a worksite, receiving first sensor data associated with the machine at the worksite, receiving second sensor data associated with an operation of the machine at the worksite, wherein the second sensor data is sourced from a sensor that is different from a sensor sourcing the first sensor data, determining, by the one or more processors, a machine state based at least on the first data and the second data, comparing the determined machine state to a modeled machine state represented by the received one or more models to classify site operations and/or detect an irregularity in site operations or an inefficiency in site operations, and generating a response based at least on the detected irregularity or inefficiency.
9 Citations
18 Claims
-
1. A method for determining machine state, the method comprising:
-
receiving, by one or more processors, one or more predetermined models relating to a worksite or a machine at the worksite; receiving, by the one or more processors, first sensor data associated with the machine at the worksite, the first sensor data comprising one or more of image data and audio data; receiving, by the one or more processors, second sensor data associated with an operation of the machine at the worksite, wherein the second sensor data is sourced from a sensor that is different from a sensor sourcing the first sensor data; determining, by the one or more processors, a machine state based at least on the first sensor data and the second sensor data, wherein the machine state includes at least one of a full load, empty load, payload material type, payload vs. air ratio, payload placement, payload compaction, payload water content, material amount moved, drop placement, excavator position, idle state, swing, dump position, deformation of machine, operator characteristic, and ground crew location relative to machine; comparing the determined machine state to a modeled machine state represented by the received one or more predetermined models to detect an irregularity in site operations or an inefficiency in site operations; and generating a response based at least on the detected irregularity or inefficiency and transmitting the response to one or more of an operator and a central station. - View Dependent Claims (2, 3, 4, 5, 6, 17, 18)
-
-
7. A system for determining machine state, the system comprising:
-
a processor; and a memory bearing instructions that, upon execution by the processor, cause the system at least to; receive one or more predetermined models relating to a worksite or a machine at the worksite; receive first sensor data associated with the machine at the worksite, the first sensor data comprising one or more of image data and audio data; receive second sensor data associated with an operation of the machine at the worksite, wherein the second sensor data is sourced from a sensor that is different from a sensor sourcing the first sensor data; determine a machine state based at least on the first sensor data and the second sensor data, wherein the machine state includes at least one of a full load, empty load, payload material type, payload vs. air ratio, payload placement, payload compaction, payload water content, material amount moved, drop placement, excavator position, idle state, swing, dump position, deformation of machine, operator characteristic, and ground crew location relative to machine; and compare the determined machine state to a modeled machine state represented by the received one or more predetermined models to classify one or more site operations. - View Dependent Claims (8, 9, 10, 11)
-
-
12. A non-transitory computer readable storage medium bearing instructions that, upon execution by one or more processors, effectuate operations for determining machine state, the non-transitory computer readable storage medium comprising:
-
receiving, by the one or more processors, first sensor data associated with a machine at a worksite, the first sensor data comprising one or more of image data and audio data; receiving, by the one or more processors, second sensor data associated with an operation of the machine at the worksite, wherein the second sensor data is sourced from a sensor that is different from a sensor sourcing the first sensor data; determining, by the one or more processors, a state signature based at least on the first sensor data and the second sensor data; comparing the state signature to a stored state signature to determine a machine state, wherein the machine state includes at least one of a full load, empty load, payload material type, payload vs. air ratio, payload placement, payload compaction, payload water content, material amount moved, drop placement, excavator position, idle state, swing, dump position, deformation of machine, operator characteristic, and ground crew location relative to machine; detecting an irregularity in site operations or an inefficiency in site operations based on the determined machine state; and generating a response based at least on the detected irregularity or inefficiency. - View Dependent Claims (13, 14, 15, 16)
-
Specification