3D tool tracking and positioning using cameras

• US 10,339,670 B2
• Filed: 08/29/2017
• Issued: 07/02/2019
• Est. Priority Date: 08/29/2017
• Status: Active Grant

• Current Assignee: Trimble Inc.
• Original Assignee: Trimble Inc.
• Inventors: Kahle, Kent, Xiao, Changlin
• Primary Examiner(s): Bhatnagar, Anand P
• Application Number:
  US15/689,514
• Publication Number:
  US 20190066331A1
• Time in Patent Office: 672 Days
• CPC Class Codes: B25F 5/00 Details or components of portable power-driven tools not particularly related to the operations performed and not otherwise provided for
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A tool in a scene is detected using one or more cameras by dividing an image into patches. Multiple different patch sizes are used in conjunction with deep learning to identify the tool in the image. After the tool is identified, a position of the tool in three dimensions is calculated using images from two or more cameras.

Claims

1. A system for detecting a tool in a scene, the system comprising:

a camera unit for acquiring an image of a scene; and

one or more processors configured to;

divide the image into a first number of patches to generate a first set of patches, wherein a patch is an area of the image,

perform object detection on the first set of patches to identify the tool in a patch of the first set of patches,

determine that the tool is not detected in the first set of patches,

divide the image into a second number of patches to generate a second set of patches, after determining that the tool is not detected in the first set of patches,

perform object detection on the second set of patches to identify the tool in a patch of the second set of patches, and

determine that the tool is detected in a patch of the second set of patches.

2. The system as recited in claim 1, wherein the camera unit is a first camera unit, the one or more processors is a first processor unit, the image is a first image, and the system further comprises:

a second camera unit configured to acquire a second image, and

a second processor unit configured to determine a position of the tool in three dimensions based on the first image and the second image.

3. The system as recited in claim 1, further comprising a display, and the one or more processors is further configured to display an indication of the tool overlaying an image on the display.

4. The system as recited in claim 3, wherein the indication of the tool is a heat map.

5. A method for detecting a tool in a scene, the method comprising:

acquiring an image of the scene;

dividing the image into a first number of patches to generate a first set of patches, wherein a patch is an area of the image;

performing object detection on the first set of patches to identify the tool in a patch of the first set of patches;

determining that the tool is not detected in the first set of patches;

dividing the image into a second number of patches to generate a second set of patches, after determining that the tool is not detected in the first set of patches;

performing object detection on the second set of patches to identify the tool in a patch of the second set of patches; and

determining that the tool is detected in a patch of the second set of patches.

6. The method as recited in claim 5, wherein the first number of patches is less than the second number of patches.

7. The method as recited in claim 5, further comprising:

dividing the image into a third number of patches to generate a third set of patches, after determining that the tool is not detected in the first set of patches and before dividing the image into a second number of patches;

performing object detection on the third set of patches to identify the tool in a patch of the third set of patches; and

determining that the tool is not detected in the third set of patches.

8. The method as recited in claim 7, wherein the third number of patches is greater than the first number of patches and less than the second number of patches.

9. The method as recited in claim 5, wherein performing object detection comprises:

dividing a patch into a number of cells; and

calculating a probability of a part of the tool being in a cell.

10. The method as recited in claim 5, wherein the patches of the first set of patches and/or the second set of patches are rectangular.

11. The method as recited in claim 5, wherein the patches of the first set of patches are equal in size.

12. The method as recited in claim 5, wherein a first patch of the first set of patches is not equal in size to a second patch of the first set of patches.

13. The method as recited in claim 5, further comprising calculating a three-dimensional position of the tool based on the image from a first camera unit, wherein the image is a first image, and a second image from a second camera unit.

14. The method as recited in claim 5, wherein the tool is a first tool and the method further comprises detecting a second tool in the first set of patches.

15. The method as recited in claim 5, further comprising training a computer neural network to identify the tool before acquiring the image of the scene.

16. The method as recited in claim 15, wherein the computer neural network is trained at a scale, and the scale is used for:

performing object detection on the first set of patches, and

performing object detection on the second set of patches.

17. The method as recited in claim 5, further comprising determining not to divide the image into a third number of patches based on the determining that the tool is detected in a patch of the second set of patches, wherein the third number of patches is greater than the second number of patches.

18. The method as recited in claim 5, further comprising acquiring the image of the scene using a camera unit having two or more image sensors.

19. A memory device comprising instructions that cause one or more processors to perform the following steps for detecting a tool in a scene:

obtain an image of the scene;

divide the image into a first number of patches to generate a first set of patches, wherein a patch is an area of the image;

perform object detection on the first set of patches to identify the tool in a patch of the first set of patches;

determine that the tool is not detected in the first set of patches;

divide the image into a second number of patches to generate a second set of patches, after determining that the tool is not detected in the first set of patches;

perform object detection on the second set of patches to identify the tool in a patch of the second set of patches; and

determine that the tool is detected in a patch of the second set of patches.

20. The memory device as recited in claim 19, wherein the memory device is part of a camera unit.