Excavator bucket positioning via mobile device

• US 10,586,349 B2
• Filed: 08/24/2017
• Issued: 03/10/2020
• Est. Priority Date: 08/24/2017
• Status: Active Grant

• Current Assignee: Trimble Inc.
• Original Assignee: Trimble Inc.
• Inventors: Li, Ding, Crozier, Scott T., Wilson, Eric K.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Mangialaschi, Tracy
• Application Number:
  US15/685,617
• Publication Number:
  US 20190066323A1
• Time in Patent Office: 929 Days
• Field of Search: None
• CPC Class Codes: E02F 9/264 Sensors and their calibration for indicating the position of the work tool
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

Various embodiments provide novel tools and techniques for position and/or motion tracking, including without limitation solutions that can be used for excavation and similar applications. In a particular aspect of some embodiments, a communication device with a camera may be provided to track the position and motion of a stick of an excavator or reference features located on a stick of an excavator. A location of a bucket of an excavator may then be calculated relative to the stick or reference features on the stick of the excavator. Thus, an operator of the excavator can use the communication device to effectively and efficiently track the location of the excavator bucket without the need for a second person to provide instructions about the location of the bucket.

Claims

1. A mobile communication device, comprising:

an image sensor;

an accelerometer;

a display;

one or more processors in communication with the image sensor, the accelerometer, and the display; and

a computer readable medium in communication with the one or more processors, the computer readable medium having encoded thereon a set of instructions executable by the mobile communication device to;

capture, with the image sensor of the mobile communication device, a reference image of a plurality of reference features on a stick of an excavator while the stick of the excavator is at a known orientation;

capture input from the accelerometer of the mobile communication device;

determine an orientation of the mobile communication device, based at least in part on the input from the accelerometer of the mobile communication device;

calibrate the mobile communication device, based on orientations of the plurality of reference features in the reference image and the orientation of the mobile communication device;

capture, with the image sensor of the mobile communication device, a second image of the plurality of reference features on the stick of the excavator;

determine, from orientations of the plurality of reference features in the second image, a second orientation of the stick of the excavator; and

display, with the display of the mobile communication device, an indication of the determined second orientation of the stick of the excavator.

2. The mobile communication device of claim 1, wherein the set of instructions are further executable by the mobile communication device to determine, from the determined second orientation of the stick of the excavator, an orientation of a bucket of the excavator, wherein displaying the indication of the determined second orientation of the stick of the excavator comprises displaying an indication of the determined orientation of the bucket of the excavator.

3. The mobile communication device of claim 1, wherein the plurality of reference features comprise one or more reference features of one or more designated optical targets on the stick of the excavator.

4. The mobile communication device of claim 3, wherein the set of instructions are further executable by the mobile communication device to automatically identify, without human input, depictions of the one or more designated optical targets in the reference image.

5. The mobile communication device of claim 1, wherein the plurality of reference features comprises one or more reference features identified from a structure of the stick of the excavator.

6. The mobile communication device of claim 1, wherein determining the second orientation of the stick of the excavator comprises determining the second orientation relative to the reference orientation of the stick of the excavator.

7. The mobile communication device of claim 6, wherein determining the second orientation of the stick of the excavator comprises determining a reach of the stick of the excavator in the second orientation.

8. The mobile communication device of claim 6, wherein determining the second orientation of the stick of the excavator comprises determining a depth of the stick of the excavator in the second orientation.

9. The mobile communication device of claim 1, wherein determining the second orientation of the stick of the excavator comprises comparing orientations of the one or more reference features in the second image with orientations of the one or more reference features in the reference image.

10. The mobile communication device of claim 9, wherein orientations of the one or more reference features comprise positions of the one or more reference features in the reference image and in the second image.

11. The mobile communication device of claim 9, wherein orientations of the one or more reference features comprise dimensions of the one or more reference features in the reference image and in the second image.

12. The mobile communication device of claim 1, wherein determining the second orientation of the stick of the excavator comprises determining an azimuth of the stick of the excavator in the second orientation.

13. The mobile communication device of claim 12, wherein the set of instructions are further executable by the mobile communication device to capture, with the image sensor of the mobile communication device, a second reference image of one or more external reference features of an environment around the excavator, wherein the second image comprises the one or more external reference features, and wherein determining the azimuth of the stick of the excavator comprises comparing positions of the one or more external reference features in the second image with positions of the one or more external reference features in the second reference image.

14. The mobile communication device of claim 1, wherein determining the second orientation of the stick of the excavator comprises determining the second orientation relative to a local or global coordinate system.

15. The mobile communication device of claim 1, wherein the set of instructions are further executable by the mobile communication device to:

capture, with the image sensor of the mobile communication device, a second reference image of the plurality of reference features on the stick of the excavator while the stick of the excavator is at a known orientation;

calculate, based on orientations of the plurality of reference features in the reference image and the second reference image, a reference feature estimation error associated with each reference feature;

determine whether the reference feature estimation error exceeds a specified threshold;

based on a determination that the reference feature estimation error exceeds the specified threshold, capture, with the image sensor of the mobile communication device, at least one additional reference image of the plurality of reference features;

recalculate the reference feature estimation error associated with each reference feature; and

repeat the operations of capturing additional reference images of the plurality of reference features on the stick of the excavator, calculating the reference feature estimation error, and determining whether the reference feature estimation error exceeds the specified threshold until the reference feature estimation error no longer exceeds the specified threshold;

wherein calibrating the mobile communication device comprises calibrating the mobile communication device based on orientations of the plurality of reference features in at least two of the reference images.

16. The mobile communication device of claim 1, wherein the set of instructions are further executable by the mobile communication device to:

calculate, based on the captured input from the accelerometer, an orientation estimation error associated with the orientation of the mobile communication device;

determine whether the orientation estimation error exceeds a specified threshold;

based on a determination that the orientation estimation error exceeds the specified threshold, capture additional input from the accelerometer of the mobile communication device;

recalculate the orientation estimation error based upon the additional input from the accelerometer; and

repeat the operations of capturing input from the accelerometer, calculating the orientation estimation error, and determining whether the orientation estimation error exceeds the specified threshold until the orientation estimation error no longer exceeds the specified threshold;

wherein calibrating the mobile communication device further comprises calibrating the mobile communication device based on one or more of the captured inputs from the accelerometer.

17. The mobile communication device of claim 1, wherein the mobile communication device is one of a wireless phone or a tablet computer.

18. The mobile communication device of claim 1, wherein the mobile communication device is mounted in a cabin of the excavator.

19. The mobile communication device of claim 1, further comprising a gyroscope in communication with the one or more processors, wherein the instructions to determine the orientation of the mobile communication device comprise instructions to determine the orientation based at least in part on input from the accelerometer and input from the gyroscope.

20. An apparatus, comprising:

a non-transitory computer readable medium having encoded thereon a set of instructions executable by a mobile communication device to;

capture, with an image sensor of the mobile communication device, a reference image of a plurality of reference features on a stick of an excavator while the stick of the excavator is at a known orientation;

capture input from an accelerometer of the mobile communication device;

determine an orientation of the mobile communication device, based at least in part on the input from the accelerometer of the mobile communication device;

calibrate the mobile communication device, based on orientations of the plurality of reference features in the reference image and the orientation of the mobile communication device;

capture, with the image sensor of the mobile communication device, a second image of the plurality of reference features on the stick of the excavator;

determine, from orientations of the plurality of reference features in the second image, a second orientation of the stick of the excavator; and

display, with a display of the mobile communication device, an indication of the determined second orientation of the stick of the excavator.

21. A method, comprising:

capturing, with an image sensor of a mobile communication device, a reference image of a plurality of reference features on a stick of an excavator while the stick of the excavator is at a known orientation;

capturing input from an accelerometer of the mobile communication device;

determining an orientation of the mobile communication device, based at least in part on the input from the accelerometer of the mobile communication device;

calibrating the mobile communication device, based on orientations of the plurality of reference features in the reference image and the orientation of the mobile communication device;

capturing, with the image sensor of the mobile communication device, a second image of the plurality of reference features on the stick of the excavator;

determining, from orientations of the plurality of reference features in the second image, a second orientation of the stick of the excavator; and

displaying, with a display of the mobile communication device, an indication of the determined second orientation of the stick of the excavator.