Registration of low contrast images

• US 9,530,208 B1
• Filed: 01/14/2016
• Issued: 12/27/2016
• Est. Priority Date: 08/04/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ivanchenko, Volodymyr V.
• Primary Examiner(s): Chu, David H
• Application Number:
  US14/996,125
• Time in Patent Office: 348 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The registration of images captured at multiple locations, for purposes such as location mapping, can be improved by utilizing multiple image capture elements pointing in at least two different directions or having different viewable ranges. At least one primary image is captured at each location. If the primary image is not able to be correlated with at least one other image based on one or more matching features, image information captured by at least one other camera at substantially the same times as those images can be analyzed to attempt to determine a change in position and/or orientation of the device between those times, which can assist in correlating the primary images. In some embodiments, motion or orientation determining elements can assist in the determination of device movement, and in at least some embodiments can reduce the amount of image information to be processed for a match.

Claims

1. A computer-implemented method for registering a set of images, comprising:

capturing, with a primary imaging element of a device, primary image data at a first time and at a first position of the device;

capturing, with the primary imaging element, second primary image data at a second time and at a second position of the device, a first field of view of the primary imaging element at the first time at least partially overlapping a second field of view of the primary imaging element at the second time;

capturing, with the primary imaging element, third primary image data at a third time and at a third position of the device, the first field of view of the primary imaging element at the first time at least partially overlapping a third field of view of the primary imaging element at the third time;

determining a common feature represented by the first primary image data;

selecting as registration image data one of the second primary image data, or the third primary image data based at least in part on a first movement between a first relative position of the common feature in the first primary image data and a second relative position of the common feature in the second primary image data; and

storing registration information enabling the first primary image data and the registration image data to be used to generate a three-dimensional representation of an area around the device.

2. The computer-implemented method of claim 1, further comprising:

capturing, with a secondary imaging element of the device having a different image capture area than the primary imaging element, first secondary image data at the first time and at the first position;

capturing, with the secondary imaging element, second secondary image data at the second time and at the second position; and

capturing, with the secondary imaging element, third secondary image data at the third time and at the third position.

3. The computer-implemented method of claim 2, further comprising:

sending the first primary image data and the first secondary image data to a remote server for analysis.

4. The computer-implemented method of claim 1, further comprising:

determining a relative change in orientation of the device between the first time and the second time using at least one orientation determining element,

wherein selecting as the registration image data one of the second primary image data, or the third primary image data is further based at least in part on the relative change in orientation.

5. The computer-implemented method of claim 1, further comprising determining the registration information based at least in part upon an amount and direction of overlap between the first primary image data and the registration image data.

6. The computer-implemented method of claim 1, further comprising determining with at least a minimum level of confidence that the common feature is represented in at least one of the second primary image data and the third primary image data.

7. The computer-implemented method of claim 1, further comprising:

prompting a user of the device to capture fourth primary image data.

8. The computer-implemented method of claim 1, further comprising:

capturing, at the first position, distance information between the device and at least one object; and

using the distance information with the first primary image data and the registration image data to generate a three-dimensional mapping of an area around the device.

9. A computing device, comprising:

a processor;

at least one primary imaging element oriented in a primary image capture direction; and

memory storing instructions, that when executed by the processor, cause the computing device to;

capture, with the primary imaging element, primary image data at a first time and at a first position of the device;

capture, with the primary imaging element, second primary image data at a second time and at a second position of the device, a first field of view of the primary imaging element at the first time at least partially overlapping a second field of view of the primary imaging element at the second time;

capture, with the primary imaging element, third primary image data at a third time and at a third position of the device, the first field of view of the primary imaging element at the first time at least partially overlapping a third field of view of the primary imaging element at the third time;

determine a common feature represented by the first primary image data;

select as registration image data one of the second primary image data, or the third primary image data based at least in part on a first movement between a first relative position of the common feature in the first primary image data and a second relative position of the common feature in the second primary image data; and

store registration information enabling the first primary image data and the registration image data to be used to generate a three-dimensional representation of an area around the device.

10. The computing device of claim 9, further comprising:

a first secondary imaging element having a different image capture area than the primary imaging element,

the instructions that, when executed by the processor, further cause the computing device to;

capture, with the first secondary imaging element, first secondary image data at the first time and at the first position;

capture, with the first secondary imaging element, second secondary image data at the second time and at the second position; and

capture, with the first secondary imaging element, third secondary image data at the third time and at the third position.

11. The computing device of claim 10, the instructions that, when executed by the processor, further cause the computing device to:

send the first primary image data and the first secondary image data to a remote server for analysis.

12. The computing device of claim 10, further comprising:

a second secondary imaging element on a second edge of the device;

wherein the first secondary imaging element is on a first edge of the device;

the instructions that, when executed by the processor, further cause the computing device to;

capture, with the second secondary imaging element, a fourth secondary image at the first time and at the first position;

capture, with the second secondary imaging element, a fifth secondary image at the second time and at the second position; and

capture, with the second secondary imaging element, a sixth secondary image at the third time and at the third position.

13. The computing device of claim 9, further comprising:

a secondary imaging element,

wherein the primary imaging element has at least one of a greater resolution, a lower field of view, and a substantially different image capture direction than the secondary imaging element.

14. The computing device of claim 9, claim further comprising:

a secondary imaging element,

wherein the secondary imaging element is positioned to provide for stereo imaging.

15. The computing device of claim 9, further comprising:

a secondary imaging element,

wherein the primary imaging element and the secondary imaging element include at least one of a digital still camera, a digital video camera, and an infrared sensor.

16. The computing device of claim 9, further comprising:

an orientation determining element;

the instructions that, when executed by the processor, further cause the computing device to;

determine a relative change in orientation of the device between the first time and the second time using at least the orientation determining element,

wherein selecting as the registration image data one of the second primary image data, or the third primary image data is further based at least in part on the relative change in orientation.

17. The computing device of claim 9, the instructions that, when executed by the processor, further cause the computing device to:

determine the registration information based at least in part upon an amount and direction of overlap between the first primary image data and the registration image data.

18. The computing device of claim 9, the instructions that, when executed by the processor, further cause the computing device to:

determine with at least a minimum level of confidence that the common feature is represented in at least one of the second primary image data and the third primary image data.

19. The computing device of claim 9, the instructions that, when executed by the processor, further cause the computing device to:

prompt a user of the device to capture fourth primary image data.

20. The computing device of claim 9, the instructions that, when executed by the processor, further cause the computing device to:

capture, at the first position, distance information between the device and at least one object; and

use the distance information with the first primary image data and the registration image data to generate a three-dimensional mapping of an area around the device.