Collaborative sighting

• US 10,499,045 B2
• Filed: 02/28/2017
• Issued: 12/03/2019
• Est. Priority Date: 04/30/2012
• Status: Active Grant

• Current Assignee: 2D3 Limited
• Original Assignee: The Boeing Co.
• Inventors: Sommerlade, Eric, Spier, Emmet, Morris, Julian
• Primary Examiner(s): Teitelbaum, Michael E
• Application Number:
  US15/444,554
• Publication Number:
  US 20170171542A1
• Time in Patent Office: 1,008 Days
• CPC Class Codes: F41G 1/345 for illuminating the sights
• CPC Class Codes: F41G 1/38 Telescopic sights specially adapted for smallarms or ordnance telescopic sights in general G02B; Supports or mountings therefor
• CPC Class Codes: F41G 3/02 using an independent line of sight
• CPC Class Codes: G01C 11/12 the pictures being supported in the same relative position as when they were taken
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

A method includes generating calibration data by geometrically calibrating first image data from a first camera unit relative to second image data from a second camera unit based on first descriptor data and second descriptor data. The first descriptor data is based on the first image data. The second descriptor data is based on the second image data. The method includes identifying, based on the calibration data, a target location relative to the first image data. The method further includes generating an output image that includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data.

Claims

1. A method comprising:

receiving first position data, wherein the first position data identifies a first position of a first camera unit while first image data was captured;

receiving second position data, wherein the second position data identifies a second position of a second camera unit while second image data was captured;

generating, at a processor, calibration data by geometrically calibrating the first image data from the first camera unit relative to the second image data from the second camera unit based on first descriptor data and second descriptor data, wherein the first descriptor data is based on the first image data, wherein the second descriptor data is based on the second image data, and wherein the calibration data is generated based on the first position data and the second position data;

identifying, based on the calibration data, a target location relative to the first image data;

generating an output image, wherein the output image includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data; and

determining that the first camera unit is focused on a second target location based on user input designating the second target location, a plurality of image data captured by the first camera unit, or a combination thereof.

2. The method of claim 1, wherein the calibration data is generated in real time.

3. The method of claim 1, further comprising receiving the second image data from the second camera unit via a communication network.

4. The method of claim 1, wherein the target location corresponds to a center of a second scene depicted in the second image data.

5. The method of claim 1, further comprising generating a second output image, wherein the second output image includes third image data and a second indication of where the second target location is relative to a second scene depicted in the third image data.

6. The method of claim 1, further comprising:

detecting a plurality of features within the first image data;

generating, by the processor, descriptors for each feature of the plurality of features, wherein the first descriptor data includes the descriptors; and

receiving the second descriptor data, wherein the second descriptor data is generated by a second processor coupled to the second camera unit, and wherein the second descriptor data is received from the second processor via a communications network.

7. The method of claim 6, wherein the descriptors include histograms of image gradients in different directions across respective positions of the plurality of features.

8. The method of claim 1, wherein an indicator is overlaid on a picture of the target location in response to the target location being represented in the first image data, and wherein the indicator points in a direction of the target location in response to the target location being located outside of the scene depicted in the first image data.

9. The method of claim 1, wherein the first camera unit is determined to be focused on a second target location in response to receiving the user input designating the second target location, and further comprising, in response to determining that the first camera unit focused on the second target location, transmitting target data indicating the second target location to the second camera unit.

10. The method of claim 1, wherein the first camera unit is determined to be focused on a second target area in response to the plurality of image data indicating that the first camera unit focusing on the second target area for a particular period of time.

11. A apparatus comprising:

a first camera unit configured to generate first image data;

a processor coupled to the first camera unit;

a memory storing instruction that when executed by the processor cause the processor to perform operations including;

generating calibration data by geometrically calibrating first image data from the first camera unit relative to second image data from a second camera unit based on first descriptor data and second descriptor data, wherein the first descriptor data is based on the first image data, and wherein the second descriptor data is based on the second image data;

identifying, based on the calibration data, a target location relative to the first image data;

generating an output image, wherein the output image includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data;

determining that the first camera unit is focused on a second target area; and

in response to determining that the first camera unit is focused on the second target area;

generating, at the processor, third descriptor data based on third image data from the first camera unit; and

transmitting the third descriptor data to a second processor coupled to the second camera unit.

12. The apparatus of claim 11, wherein the first camera unit is coupled to a vehicle.

13. The apparatus of claim 11, further comprising a display unit, wherein the display unit is configured to display the output image.

14. The apparatus of claim 13, further comprising a housing, wherein the first camera unit is coupled to a first side of the housing, and wherein the display unit is coupled to a second side of the housing opposite of the first side.

15. The apparatus of claim 14, wherein the housing is coupled to a weapon, and wherein the first camera unit and display unit are configured to act as a sight for the weapon.

16. The apparatus of claim 11, further comprising an Attitude and Reference System sensor, a Global Positioning System sensor, or a combination thereof.

17. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving first position data, wherein the first position data identifies a first position of a first camera unit while first image data was captured;

receiving second position data, wherein the second position data identifies a second position of a second camera unit while second image data was captured;

generating calibration data by geometrically calibrating the first image data from the first camera unit relative to the second image data from the second camera unit based on first descriptor data and second descriptor data, wherein the first descriptor data is based on the first image data, wherein the second descriptor data is based on the second image data, and wherein the calibration data is generated based on the first position data and the second position data;

identifying, based on the calibration data, a target location relative to the first image data based on the calibration data;

generating an output image, wherein the output image includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data; and

determining that the first camera unit is focused on a second target location based on user input designating the second target location, a plurality of image data captured by the first camera unit, or a combination thereof.

18. The non-transitory computer-readable medium of claim 17, wherein the operations further comprise receiving, from a second processor coupled to the second camera unit, target data indicating the target location.

19. A method comprising:

generating calibration data by geometrically calibrating first image data from a first camera unit relative to second image data from a second camera unit based on first descriptor data and second descriptor data, wherein the first descriptor data is based on first image data, and wherein the second descriptor data is based on second image data;

identifying, based on the calibration data, a target location relative to the first image data;

generating an output image, wherein the output image includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data; and

determining that the first camera unit is focused on a second target area; and

in response to determining that the first camera unit is focused on the second target area;

generating third descriptor data based on third image data from the first camera unit; and

transmitting the third descriptor data to a second processor coupled to the second camera unit.

20. An apparatus comprising:

a processor coupled to a first camera unit;

a memory storing instruction that when executed by the processor cause the processor to perform operations including;

receiving first position data, wherein the first position data identifies a first position of a first camera unit while first image data was captured;

receiving second position data, wherein the second position data identifies a second position of a second camera unit while second image data was captured;

generating, at the processor, calibration data by geometrically calibrating the first image data from the first camera unit relative to the second image data from the second camera unit based on first descriptor data and second descriptor data, wherein the first descriptor data is based on the first image data, wherein the second descriptor data is based on the second image data, and wherein the calibration data is generated based on the first position data and the second position data;

identifying, based on the calibration data, a target location relative to the first image data;

generating an output image, wherein the output image includes the first image data and an indication of where the target location is relative to a scene depicted in the first image data; and

determining that the first camera unit is focused on a second target location based on user input designating the second target location, a plurality of image data captured by the first camera unit, or a combination thereof.