Command-driven automatic and semi-automatic mobile wear detection

• US 10,429,272 B2
• Filed: 09/27/2016
• Issued: 10/01/2019
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Lu, Stephen P., Pauly, Ted J., Aberle, Conor D., Diekevers, Mark S., Hasselbusch, Michael D., O'Brien, Kevin N.
• Primary Examiner(s): Chad, Aniss
• Assistant Examiner(s): Moll, Nithya J.
• Application Number:
  US15/276,822
• Publication Number:
  US 20170091375A1
• Time in Patent Office: Days
• Field of Search: None
• CPC Class Codes: G01M 13/023 Power-transmitting endless elements, e.g. belts or chains
• CPC Class Codes: G05B 23/0283 Predictive maintenance, e.g. involving the monitoring of a system and, based on the monitoring results, taking decisions on the maintenance schedule of the monitored system; Estimating remaining useful life [RUL] preventive maintenance, i.e. planning maintenance according to the available resources without monitoring the system G06Q10/06
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/001 using an image reference approach

Abstract

Systems and methods for command-driven automatic and semi-automatic mobile wear detection are disclosed. One method may include: transmitting, by a mobile device, a request indicative of a desire to determine a wear metric associated with a part; receiving, by the mobile device, first data associated with the part and indicative of a directive for the mobile device to display an instruction for the mobile device to be situated in a specified position; responsive to a determination, by the mobile device, that the mobile device is situated in the specified position, receiving, by a camera of the mobile device, visual data associated with the part; receiving, by the mobile device, second data indicative of a directive for the mobile device to process the visual data; determining, by the mobile device, the wear metric based, at least, on the processed visual data.

Claims

1. A method comprising:

receiving, by a mobile device, first data associated with a part and indicative of a directive for the mobile device to provide an instruction for the mobile device to be situated in a specified position;

responsive to a determination, by the mobile device, that the mobile device is situated in the specified position, receiving, by a camera of the mobile device, visual data associated with the part;

receiving, by the mobile device, second data indicative of a directive for the mobile device to process the visual data;

determining, by the mobile device, a wear metric associated with the part, the wear metric based, at least, on the processed visual data;

identifying a user type of a user of the mobile device;

displaying the wear metric on the mobile device in a format selected based on the user type;

determining from inertial measurement unit (IMU) data of the mobile device, an amount of movement of the mobile device that the mobile device experienced during receiving the visual data associated with the part via the camera; and

outputting a confidence metric with the wear metric based on the amount of movement, the confidence metric indicative of confidence that the wear metric is accurate based on the amount of movement of the mobile device.

2. The method of claim 1, wherein the instruction for the mobile device to be situated in a specified position comprises a textual instruction to be displayed on a display of the mobile device.

3. The method of claim 1, wherein the instruction for the mobile device to be situated in a specified position comprises a visual indicator displayed on a display of the mobile device and overlaid upon a view of the part on the display.

4. The method of claim 1, wherein the instruction for the mobile device to be situated in a specified position comprises an auditory signal.

5. The method of claim 1, further comprising:

determining that one or more features of the part are recognized in the visual data, wherein the processing of the visual data is responsive to the determining that the one or more features of the part are recognized in the visual data.

6. The method of claim 1, further comprising:

receiving, by the mobile device, an input from a user of the mobile device, the input modifying the visual data; and

processing the modified visual data to determine a second wear metric.

7. A method comprising:

receiving, by a mobile device, first data associated with a part of equipment and indicative of a directive for the mobile device to communicate an instruction for the mobile device to be situated in a specified position;

receiving, by the mobile device, second data comprising one or more dimensions of the part in an unworn state;

responsive to a determination, by the mobile device, that the mobile device is situated in the specified position, receiving, by a camera of the mobile device, visual data associated with the part;

generating, by the mobile device, a digital model of the part based, at least, on the visual data;

determining, by the mobile device, a wear metric based, at least, on the digital model of the part and the second data;

identifying a user type of a user of the mobile device;

displaying the wear metric on the mobile device in a format selected based on the user type;

determining from inertial measurement unit (IMU) data of the mobile device, an amount of movement of the mobile device that the mobile device experienced during receiving the visual data associated with the part via the camera; and

outputting a confidence metric with the wear metric based on the amount of movement, the confidence metric indicative of confidence that the wear metric is accurate based on the amount of movement of the mobile device.

8. The method of claim 7, wherein the digital model comprises a three-dimensional reconstruction of the part.

9. The method of claim 7, further comprising:

receiving, by the mobile device, third data indicative of a directive for the mobile device to process the digital model of the part; and

determining, by the mobile device, the wear metric based, at least, on the digital model of the part, the second data, and the third data.

10. The method of claim 7, further comprising:

receiving, by the mobile device, an input from a user of the mobile device, the input modifying the digital model of the part; and

processing the modified digital model of the part to determine a second wear metric.

11. The method of claim 10, wherein the input from the user of the mobile device comprises a modification of at least one of a line, curve, or point in the digital model of the part.

12. A mobile device comprising:

a processor;

a camera; and

a memory communicatively coupled with the processor, the memory having stored thereon executable instructions that when executed by the processor cause the processor to effectuate operations comprising;

receiving first data associated with a part and indicative of a directive for the mobile device to provide an instruction for the mobile device to be situated in a specified position;

responsive to a determination that the mobile device is situated in the specified position, receiving, by the camera, visual data associated with the part;

receiving second data indicative of a directive for the mobile device to process the visual data;

processing, based at least on the second data, the visual data to determine a wear metric associated with the part;

identify a user type of a user of the mobile device;

display the wear metric on the mobile device in a format selected based on the user type;

determining from inertial measurement unit (IMU) data of the mobile device, an amount of movement of the mobile device that the mobile device experienced during receiving the visual data associated with the part via the camera; and

outputting a confidence metric with the wear metric based on the amount of movement, the confidence metric indicative of confidence that the wear metric is accurate based on the amount of movement of the mobile device.

13. The mobile device of claim 12, wherein the operations further comprise:

determining that one or more features of the part are recognized in the visual data, wherein the processing of the visual data is responsive to the determining that the one or more features of the part are recognized in the visual data.

14. The mobile device of claim 12, wherein the operations further comprise:

receiving an input from a user of the mobile device, the input modifying the visual data; and

processing the modified visual data to determine a second wear metric.

15. The mobile device of claim 12, wherein the operations further comprise:

transmitting the wear metric to a central server.