APPLYING AN ANNOTATION TO AN IMAGE BASED ON KEYPOINTS

• US 20200137326A1
• Filed: 10/24/2018
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Jahagirdar, Manik Mahesh
• Granted Patent: US 10,778,916 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: H04N 25/445 by skipping some contiguous pixels within the read portion of the array

Abstract

In some examples, a computing device includes a display, processing circuitry configured to present a first image via the display, and an input device configured to receive user inputs. The processing circuitry is further configured to determine an annotation to the first image based on the user inputs and determine an image patch in the first image overlapping with the annotation. The processing circuitry is also configured to determine, in the image patch, a first set of keypoints associated with the annotation and match the first set of keypoints in the first image to a second set of keypoints in the corresponding image patch in the second image. The processing circuitry is configured to determine a position of the corresponding image patch in the second image based on matching the first set of keypoints to the second set of keypoints and apply the annotation to the second image.

Claims

1. A computing device comprising:

a display;

processing circuitry configured to present a first image via the display; and

an input device configured to receive user inputs,

wherein the processing circuitry is further configured to;

determine an annotation to the first image based on the user inputs;

determine an image patch in the first image overlapping with the annotation;

determine, in the image patch in the first image, a first set of keypoints associated with the annotation;

match the first set of keypoints in the image patch in the first image to a second set of keypoints in a corresponding image patch in a second image;

determine a position of the corresponding image patch in the second image based on matching the first set of keypoints to the second set of keypoints; and

apply the annotation to the second image based on the position of the corresponding image patch in the second image.

2. The computing device of claim 1,

wherein the processing circuitry is configured to determine the first set of keypoints in the image patch at least in part by determining pixel values of a first block of pixels in the image patch, and

wherein the processing circuitry is configured to match the first set of keypoints to the second set of keypoints at least in part by determining that a difference between the pixel values of the first block of pixels and pixel values of a second block of pixels in the second image is less than a threshold level.

3. The computing device of claim 1, wherein the processing circuitry is further configured to:

select the second image based on a time of capture of the second image;

determine a set of possible keypoints in the second image using a keypoint detection method, wherein the set of possible keypoints includes the second set of keypoints;

determine the first set of keypoints in the image patch using the keypoint detection method;

determine whether the second image includes the corresponding image patch at least in part by matching the first set of keypoints with the set of possible keypoints in the second image;

determine the second set of keypoints by matching the first set of keypoints with the set of possible keypoints in the second image;

determine the position of the corresponding image patch in the second image based on a centroid of the second set of keypoints in the second image;

determine a scaling factor based on the second set of keypoints in the second image and the first set of keypoints in the image patch;

create a third image based on a resolution of the second image; and

apply the annotation to the third image based on the scaling factor and the position of the image patch in the second image.

4. The computing device of claim 1, wherein the image patch is centered on a centroid of the annotation.

5. The computing device of claim 1,

wherein the corresponding image patch is less than fully visible in the second image,

wherein the processing circuitry is configured to match the first set of keypoints to the second set of keypoints at least in part by matching fewer than all of the first set of keypoints in the image patch in the first image to the second set of keypoints in the corresponding image patch in the second image.

6. The computing device of claim 1,

wherein the processing circuitry is further configured to determine a scaling factor based on positions of the first set of keypoints in the first image and positions of the second set of keypoints in the second image,

wherein the processing circuitry is configured to apply the annotation to the second image at least in part by:

determining a scaled version of the annotation based on the scaling factor; and

applying the scaled version of the annotation to the second image.

7. The computing device of claim 6, wherein the processing circuitry is configured to determine the scaling factor at least in part by:

determining a centroid of the image patch in the first image;

determining a centroid of the corresponding image patch in the second image;

determining an average distance in a horizontal dimension and in a vertical dimension between the first set of keypoints and the centroid of the image patch in the first image;

determining an average distance in the horizontal dimension and in the vertical dimension between the second set of keypoints and the centroid of the image patch in the second image;

determining a horizontal scaling factor based on a ratio of the average distance in the horizontal dimension for the first set of keypoints and the average distance in the horizontal dimension for the second set of keypoints; and

determining a vertical scaling factor based on a ratio of the average distance in the vertical dimension for the first set of keypoints and the average distance in the vertical dimension for the second set of keypoints.

8. The computing device of claim 1, wherein the first image comprises a low-resolution version of an image, and wherein the second image comprises a high-resolution version of the image.

9. The computing device of claim 1, wherein the processing circuitry is configured to determine the first set of keypoints in the image patch at least in part by determining a set of unique image features in the image patch.

10. The computing device of claim 1, wherein the processing circuitry is further configured to:

select the second image from a set of images at least in part by determining whether each image of the set of images includes the image patch; and

apply the annotation to each image of the set of images that includes the image patch.

11. The computing device of claim 1, wherein the processing circuitry is further configured to select the second image from a set of images at least in part by determining that a difference between a time of capture for the first image and a time of capture for the second image is less than a threshold time duration.

12. The computing device of claim 1,

wherein the second image comprises a panoramic image, and

wherein the processing circuitry is further configured to create the second image at least in part by stitching together two or more images.

13. The computing device of claim 1, wherein the processing circuitry is further configured to:

build a three-dimensional model based on the first image and the second image; and

apply the annotation to the three-dimensional model.

14. A method comprising:

receiving, by a computing device, a first image and a second image;

receiving, by an input device of the computing device, user inputs;

determining, by processing circuitry of the computing device, an annotation to the first image based on the user inputs;

determining, by the processing circuitry, an image patch in the first image overlapping with the annotation;

determining, by the processing circuitry, a first set of keypoints in the image patch in the first image, the first set of keypoints being associated with the annotation;

matching, by the processing circuitry, the first set of keypoints on the image patch in the first image to a second set of keypoints in a corresponding image patch in the second image;

determining, by processing circuitry, a position of the corresponding image patch in the second image based on matching the first set of keypoints to the second set of keypoints; and

applying, by the processing circuitry, the annotation to the second image based on the position of the corresponding image patch in the second image.

15. The method of claim 14,

wherein determining the first set of keypoints in the image patch comprises determining pixel values of a first block of pixels in the image patch, and

wherein matching the first set of keypoints to the second set of keypoints comprises determining that a difference between the pixel values of the first block of pixels and pixel values of a second block of pixels in the second image is less than a threshold level.

16. The method of claim 14, further comprising determining a scaling factor based on positions of the first set of keypoints in the first image and positions of the second set of keypoints in the second image,

wherein applying the annotation to the second image comprises:

determining a scaled version of the annotation based on the scaling factor; and

applying the scaled version of the annotation to the second image.

17. A device comprising a computer-readable medium having executable instructions stored thereon, configured to be executable by processing circuitry for causing the processing circuitry to:

receive a first image and a second image;

determine an annotation to the first image based on user inputs;

determine an image patch in the first image overlapping with the annotation;

determine, in the image patch in the first image, a first set of keypoints associated with the annotation;

match the first set of keypoints in the image patch in the first image to a second set of keypoints in a corresponding image patch in the second image;

determine a position of a corresponding image patch in the second image based on matching the first set of keypoints to the second set of keypoints; and

apply the annotation to the second image based on the position of the corresponding image patch in the second image.

18. The device of claim 17,

wherein the instructions to determine the first set of keypoints in the image patch comprise instructions to determine pixel values of a first block of pixels in the image patch, and

wherein the instructions to match the first set of keypoints to the second set of keypoints comprise instructions to determining that a difference between the pixel values of the first block of pixels and pixel values of a second block of pixels in the second image is less than a threshold level.

19. The device of claim 17,

wherein the instructions are configured to be executable by the processing circuitry for causing the processing circuitry to determine a scaling factor based on positions of the first set of keypoints in the first image and positions of the second set of keypoints in the second image,

wherein the instructions to apply the annotation to the second image comprise instructions to:

determine a scaled version of the annotation based on the scaling factor; and

apply the scaled version of the annotation to the second image.

20. The device of claim 17, wherein the instructions are configured to be executable by the processing circuitry for causing the processing circuitry to:

build a three-dimensional model based on the first image and the second image; and

apply the annotation to the three-dimensional model.