Vehicle classification and vehicle pose estimation

• US 10,163,033 B2
• Filed: 12/13/2016
• Issued: 12/25/2018
• Est. Priority Date: 12/13/2016
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Mianzo, Lawrence Andrew, Kriel, Bradley Scott
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/376,920
• Publication Number:
  US 20180165502A1
• Time in Patent Office: 742 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A method of classifying a work machine within a worksite is provided. The method may include receiving one or more captured images of the work machine from one or more image capture devices, identifying at least two classifiers within the captured images of the work machine, determining spatial relationships between the classifiers within the captured images, determining a classification and a pose of the work machine based on the spatial relationships, and displaying analytics based on the classification and the pose of the work machine.

Claims

1. A method of classifying a work machine within a worksite, the method comprising:

receiving one or more captured images of the work machine from one or more image capture devices positioned around the worksite and remote from the work machine;

identifying at least two classifiers within the captured images of the work machine, wherein each classifier is indicative of a known component that is common to all work machines of the work machine type of the work machine;

determining spatial relationships between the classifiers within the captured images based on the relative positions of the classifiers as measured within the captured images;

determining a classification and a pose of the work machine based on the spatial relationships;

tracking the determined poses of the work machine over time;

determining analytics related to the work machine and the worksite based on the tracked poses, the analytics including actual data derived from the poses of the work machine and representing a volume of work actually performed by the work machine within a given timeframe, and model data representing a predetermined volume of work performed by the work machine within the given timeframe; and

displaying the analytics based on the classifications and the poses of the work machine.

2. The method of claim 1, wherein the captured images include at least one video and the image capture devices includes at least one video camera installed in relation to one of the work machine and the worksite.

3. The method of claim 1, wherein each of the classifiers, the classification and the pose is visually recognized based on reference data retrieved from a database, the reference data visually associating each of the classifiers to one of a plurality of known machine components and visually associating different spatial relationships to known classifications and poses.

4. The method of claim 3, wherein the known machine components include two or more of wheels, tracks, linkages, machine implements, operator cabs, and work machines.

5. The method of claim 1, wherein the analytics include one or more of work productivity of the work machine, work progress, a state of the work machine, and a state of a terrain within the worksite.

6. A system for classifying a work machine within a worksite, the system comprising:

one or more image capture devices configured to generate one or more captured images of the work machine; and

a visual recognition device in communication with the one or more image capture devices positioned around the worksite and remote from the work machine, the visual recognition device configured to

receive the captured images from the image capture devices,

identify at least two classifiers within the captured images wherein each classifier is indicative of a known component that is common to all work machines of the work machine type of the work machine,

determine spatial relationships between the classifiers within the captured images based on the relative positions of the classifiers as measured within the captured images,

determine a classification and a pose of the work machine based on the spatial relationships,

track the determined poses of the work machine over time,

determine analytics related to the work machine and the worksite based on the tracked poses, the analytics including actual data derived from the poses of the work machine and representing a volume of work actually performed by the work machine within a given timeframe, and model data representing a predetermined volume of work performed by the work machine within the given timeframe, and

cause the display of the analytics based on the classifications and the poses of the work machine.

7. The system of claim 6, wherein the image capture devices include at least one video camera installed in relation to one of the work machine and the worksite, and the captured images include at least one video.

8. The system of claim 6, further comprising one or more databases in communication with the visual recognition device and configured to retrievably store reference data, the reference data visually associating each of the classifiers to one of a plurality of known machine components and visually associating different spatial relationships to known classifications and poses.

9. The system of claim 8, wherein the visual recognition device is configured to visually recognize each of the classifiers based on the reference data.

10. The system of claim 8, wherein the visual recognition device is configured to visually recognize the classification and the pose of the work machine based on the reference data.

11. The system of claim 8, wherein the database retrievably stores reference data visually associating the classifiers to known machine components including two or more of wheels, tracks, linkages, machine implements, operator cabs, and work machines.

12. The system of claim 6, further comprising one or more tracking devices configured to acquire terrain measurements from the worksite, the visual recognition device being configured to track the pose of the work machine and the terrain measurements over time, and determine the analytics related to the work machine and the worksite based on the tracked pose and terrain measurements, the analytics including one or more of work productivity of the work machine, work progress, a state of the work machine, and a state of a terrain within the worksite.

13. A device for classifying a work machine within a worksite, the device comprising:

a communication device in communication with one or more image capture devices positioned around the worksite and remote from the work machine;

an output device;

a memory configured to retrievably store one or more algorithms; and

a controller in communication with each of the communication device, the output device and the memory, the controller being configured to

receive one or more captured images of the work machine from the image capture devices,

identify at least two classifiers within the captured images, wherein each classifier is indicative of a known component that is common to all work machines of the work machine type of the work machine,

determine spatial relationships between the classifiers within the captured images based on the relative positions of the classifiers as measured within the captured images,

determine a classification and a pose of the work machine based on the spatial relationships,

track the determined poses of the work machine over time and store the poses in the memory,

determine analytics related to the work machine and the worksite based on the tracked poses, the analytics including actual data derived from the poses of the work machine and representing a volume of work actually performed by the work machine within a given timeframe, and model data representing a desired volume of work performed by the work machine within the given timeframe, and

cause the output device to display the analytics based on the classifications and the poses of the work machine.

14. The device of claim 13, wherein the communication device is in communication with one or more databases configured to retrievably store reference data, the reference data visually associating each of the classifiers to one of a plurality of known machine components and visually associating different spatial relationships to known classifications and poses.

15. The device of claim 14, wherein the controller is configured to visually recognize each of the classifiers, the classification and the pose of the work machine based on the reference data.

16. The device of claim 14, wherein the database retrievably stores reference data visually associating the classifiers to known machine components including two or more of wheels, tracks, linkages, machine implements, operator cabs, and work machines.

17. The device of claim 13, wherein the communication device is in communication with one or more tracking devices configured to acquire terrain measurements from the worksite, the controller being configured to track the pose of the work machine and the terrain measurements over time, and determine the analytics related to the work machine and the worksite based on the tracked pose and terrain measurements, the analytics including one or more of work productivity of the work machine, work progress, a state of the work machine, and a state of a terrain within the worksite.