System and method for detecting missing tooth in mining shovel

• US 9,886,754 B2
• Filed: 04/05/2016
• Issued: 02/06/2018
• Est. Priority Date: 04/05/2016
• Status: Active Grant

• Current Assignee: GE Global Sourcing LLC (Westinghouse Air Brake Technologies Corporation)
• Original Assignee: General Electric Company
• Inventors: Lim, Ser Nam, Zhou, Ning, Soares, Joao Vitor Baldini
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/090,646
• Publication Number:
  US 20170287124A1
• Time in Patent Office: 672 Days
• CPC Class Codes: E02F 9/267 Diagnosing or detecting failure of vehicles
• CPC Class Codes: E02F 9/2808 Teeth
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

A method for detecting missing tooth in mining shovel, implemented using a processing device, includes receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load. A tooth line region corresponding to the pair of image frames is detected to generate a pair of tooth line regions based on a shovel template set. A difference image is determined based on the pair of image frames and the pair of tooth line regions. Further, a response map representative of possible tooth positions is determined based on the difference image using a tooth template matching technique. A tooth line is selected among a plurality of candidate tooth lines based on the response map. Further, a tooth condition is determined based on the tooth line and the difference image. The tooth condition is notified to an operator of the rope mine shovel.

Claims

1. A method, comprising:

receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load, wherein the pair of image frames comprises a first image frame and a second image frame;

detecting a corresponding tooth line region for the pair of image frames to generate a pair of tooth line regions based on a shovel template set, wherein the shovel template set comprises a plurality of bucket template images, a plurality of tooth line images, a plurality of locations corresponding to the plurality of bucket template images, and a plurality of locations of tooth line images, and wherein the pair of tooth line regions comprises a first tooth line region in the first image and a second tooth line region in the second image;

determining a difference image based on the pair of tooth line regions;

determining a response map representative of possible tooth positions based on the difference image using a tooth template matching technique;

selecting a tooth line among a plurality of candidate tooth lines based on the response map;

determining a tooth condition based on the tooth line and the difference image and

notifying the tooth condition to an operator of the rope mine shovel.

2. The method of claim 1, wherein notifying the tooth condition comprises emitting an alarm.

3. The method of claim 1, wherein detecting the tooth line region comprises:

generating a plurality of image tiles corresponding to an image frame among the pair of image frames;

determining a correlation between a bucket template image among a plurality of bucket template images and an image tile among the plurality of image tiles to generate a plurality of matching scores;

selecting a bucket template image and an associated image tile corresponding to a minimum value among the plurality of matching scores; and

determining a location of the tooth line region based on location of the selected image tile, location of the selected bucket template image, and a location of tooth line region associated with the selected bucket template image.

4. The method of claim 1, wherein determining the difference image comprises:

determining a homography transformation of the first tooth line image with respect to the second tooth line image; and

generating a transformed image by applying the homography transformation to the first image.

5. The method of claim 3, wherein determining the difference image comprises:

determining a first gradient image corresponding to the transformed image and a second gradient image corresponding to the second image frame; and

performing a differencing operation between the first gradient image and the second gradient image.

6. The method of claim 1, wherein the tooth template matching technique comprises:

consecutively positioning a tooth template image at a plurality of locations in the difference image; and

determining a plurality of correlation coefficients corresponding to the plurality of locations in the difference image based on the tooth template image and a plurality of pixels of the difference image.

7. The method of claim 1, wherein determining the tooth line comprises generating a plurality of candidate tooth lines based on a template tooth line associated with the selected bucket template image, wherein each of the plurality of candidate tooth lines are scaled and rotated version of the template tooth line.

8. The method of claim 7, wherein determining the tooth line comprises determining a fit score value as a sum of pixel values of the response map overlapping with the determined tooth line.

9. The method of claim 1, wherein determining the tooth condition comprises:

determining a plurality of pixels in the difference image around a position of a tooth of a tooth line;

determining a missing score value as a sum of the plurality of pixels of the difference image; and

comparing the missing score value with a missing measurement threshold value and setting a missing flag to a value of one if the sum exceeds the missing measurement threshold value.

10. The method of claim 9, wherein determining the tooth condition comprises:

initiating a missing count value to zero;

determining a plurality of missing flags corresponding to a plurality of image frames;

adding the plurality of missing flags to the missing count; and

determining a tooth condition when the missing count is greater than a missing count threshold value.

11. The method of claim 1, wherein receiving the pair of image frames comprises rejecting an image frame based on the difference image.

12. A system, comprising:

an imaging device fitted on a rope mine shovel disposed on a rope mine shovel configured to carry a mining load, configured to acquire a plurality of image frames;

a processor communicatively coupled to the imaging device and configured to;

receive a pair of image frames wherein the pair of image frames comprises a first image frame and a second image frame;

detect a corresponding tooth line region for the pair of image frames to generate a pair of tooth line regions based on a shovel template set, wherein the shovel template set comprises a plurality of bucket template images, a plurality of tooth line images, a plurality of locations corresponding to the plurality of bucket template images, and a plurality of locations of tooth line images, and wherein the pair of tooth line regions comprises a first tooth line region in the first image and a second tooth line region in the second image;

determine a difference image based on the pair of tooth line regions;

determine a response map representative of possible tooth positions based on the difference image using a tooth template matching technique;

select a tooth line among a plurality of candidate tooth lines based on the response map;

determine a tooth condition based on the tooth line and the difference image; and

notify the tooth condition to an operator of the rope mine shovel.

13. The system of claim 12, wherein the processor device is further configured to emit an alarm, or stop the rope mine shovel, or both.

14. The system of claim 12, wherein the processor device is further configured to:

generate a plurality of image tiles corresponding to an image frame among the pair of image frames;

determine a correlation between a bucket template image among a plurality of bucket template images and an image tile among the plurality of image tiles to generate a plurality of matching scores;

select a bucket template image and an associated image tile corresponding to a minimum value among the plurality of matching scores; and

determine a location of the tooth line region based on location of the selected image tile, location of the selected bucket template image, and a location of tooth line region associated with the selected bucket template image.

15. The system of claim 12, wherein the processor device is further configured to:

determine a homography transformation of the first tooth line image with respect to the second tooth line image; and

generate a transformed image by applying the homography transformation to the first image.

16. The system of claim 15, wherein the processor is further configured to:

determine a first gradient image corresponding to the transformed image and a second gradient image corresponding to the second image frame; and

determine a difference between the first gradient image and the second gradient image.

17. The system of claim 12, wherein the processor is further configured to:

consecutively position a tooth template image at a plurality of locations on the difference image; and

determine a plurality of correlation coefficients corresponding to the plurality of locations in the difference image based on the tooth template image and a plurality of pixels of the difference image.

18. The system of claim 12, wherein the processor is further configured to generate a plurality of candidate tooth lines based on a template tooth line associated with the selected bucket template image, wherein each of the plurality of candidate tooth lines are scaled and rotated version of the template tooth line.

19. The system of claim 18, wherein the processor is further configured to determine a fit score value as a sum of pixel values of the response map overlapping with a candidate tooth line among the plurality of tooth lines.

20. The system of claim 12, wherein the processor is further configured to:

determine a plurality of pixels in the difference image around a position of a tooth of a tooth line;

determine a missing score value as a sum of the plurality of pixels of the difference image; and

compare the missing score value with a missing measurement threshold value and setting a missing flag to a value of one if the sum exceeds the missing measurement threshold value.

21. The system of claim 12, wherein the processor is further configured to:

initiate a missing count value to zero;

determine a plurality of missing flags corresponding to a plurality of image frames;

add the plurality of missing flags to the missing count; and

determine a tooth condition when the missing count is greater than a missing count threshold value.

22. The system of claim 12, wherein the processor is configured to reject an image frame based on the difference image.

23. A method, comprising:

receiving a pair of image frames from a camera disposed on a rope mine shovel configured to carry a mining load, wherein the pair of image frames comprises a first image frame and a second image frame;

detecting a corresponding metal part region for the pair of image frames to generate a pair of metal part regions based on a shovel template set, wherein the shovel template set comprises a plurality of shovel template images, a plurality of metal part images, a plurality of locations corresponding to the plurality of shovel template images, and a plurality of locations of metal part images, and wherein the pair of metal part regions comprises a first metal part region in the first image and a second metal part region in the second image;

determining a difference image based on the pair of metal part regions;

determining a response map representative of possible metal part positions based on the difference image using a metal part template matching technique;

determining a metal part condition based on the response map and the difference image and

notifying the metal part condition to an operator of the rope mine shovel.