Collaborative sighting

• US 9,509,987 B2
• Filed: 04/30/2013
• Issued: 11/29/2016
• Est. Priority Date: 04/30/2012
• Status: Active Grant

• Current Assignee: 2D3 Limited
• Original Assignee: The Boeing Co.
• Inventors: Sommerlade, Eric, Spier, Emmet, Morris, Julian
• Primary Examiner(s): TEITELBAUM, MICHAEL E
• Application Number:
  US14/397,576
• Publication Number:
  US 20150146015A1
• Time in Patent Office: 1,309 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: F41G 1/345 for illuminating the sights
• CPC Class Codes: F41G 1/38 Telescopic sights specially adapted for smallarms or ordnance telescopic sights in general G02B; Supports or mountings therefor
• CPC Class Codes: F41G 3/02 using an independent line of sight
• CPC Class Codes: G01C 11/12 the pictures being supported in the same relative position as when they were taken
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

Collaborative sighting is performed using plural optical sighting apparatuses that each comprise: a camera unit and a display device. Geometric calibration of the camera units with respect to the imaged scene is performed by detecting features within captured images, generating descriptors from respective patches of the image at the features; detecting corresponding descriptors from different images, and deriving the geometric calibration from the positions in the respective images of the corresponding descriptors. A target location in the image captured by a first optical sighting apparatus is designated and a corresponding location relative to the image captured by the second one optical sighting apparatus is identified from the geometric calibration. The display device of the second optical sighting apparatus indicates where the corresponding location lies relative to the displayed image.

Claims

1. A method of collaborative sighting using plural optical sighting apparatuses that each comprise:

a camera unit arranged to capture an image of a portion of a scene within a field of view of the camera unit; and

a display device arranged to display images captured by the camera unit, the method comprising;

detecting features within images captured by the camera units of the optical sighting apparatuses;

generating descriptors in respect of each image from respective patches of the image at a position of each detected feature by the optical sighting apparatuses whose camera units capture the images;

detecting corresponding descriptors generated from different images;

deriving geometric calibration of the camera units with respect to the scene from positions in respective images of the features corresponding to the detected corresponding descriptors, wherein the geometric calibration is derived in real time as the corresponding descriptors are detected;

in respect of at least one target location in the image captured by the camera unit of a first optical sighting apparatus, identifying, from the derived geometric calibration of the camera units, a corresponding location relative to the image captured by the camera unit of a second optical sighting apparatus that corresponds to a target location in the scene that itself corresponds to the target location in the image captured by the camera unit of the first optical sighting apparatus; and

indicating on the display device of the second optical sighting apparatus where the corresponding location lies relative to the displayed image, wherein

detecting the features and generating the descriptors are performed in respect of each image by the optical sighting apparatus whose camera unit captures that image.

2. A method according to claim 1, wherein in a case that the target location in the scene that corresponds to the target location in the image captured by the camera unit of the first optical sighting apparatus is outside the field of view of the camera unit of the second optical sighting apparatus, the step of indicating on the display device of the second optical sighting apparatus where the corresponding location lies relative to the displayed image comprises displaying a pointer towards the corresponding location.

3. A method according to claim 1, wherein the at least one target location in the image captured by the camera unit of the first optical sighting apparatus is at a predetermined position within that image.

4. A method according to claim 1, wherein the optical sighting apparatuses and a central processing apparatus are arranged to communicate over a communications network.

5. A method according to claim 4, wherein one of the optical sighting apparatuses or the central processing apparatus acts as an analysis apparatus that performs said steps of detecting corresponding descriptors and deriving the geometric calibration of the camera units with respect to the scene.

6. A method according to claim 5, wherein the analysis apparatus is the second optical sighting apparatus.

7. A method according to claim 5, wherein

the optical sighting apparatuses that are not the analysis apparatus communicate the descriptors and the positions in the respective images of the features corresponding to the descriptors to the analysis apparatus over the communication network.

8. A method according to claim 1, wherein the step of detecting features comprises detecting features with scale-invariance and rotation-invariance.

9. A method according to claim 1, wherein the step of generating descriptors comprises generating descriptors with scale-invariance and rotation-invariance.

10. A method according to claim 1, wherein the step of detecting features comprises detecting maxima and/or minima in differences between different scale spaces of the image.

11. A method according to claim 10, wherein the step of generating descriptors comprises generating descriptors from respective patches of the image in the scale spaces of the image in which the maxima and/or the minima in different samples is detected.

12. A method according to claim 1, wherein the step of generating descriptors comprises generating descriptors from respective patches of the image around each detected feature that are each oriented to a dominant gradient in the respective patches.

13. A method according to claim 1, wherein the descriptors comprise histograms of image gradients in different directions across the respective positions of the detected features.

14. A method according to claim 1, wherein the step of detecting corresponding descriptors generated from different images comprises detecting descriptors generated from different images that match each other and that conform to a common mapping between the positions in the respective images of the features corresponding to those descriptors.

15. A method according to claim 1, wherein deriving the geometric calibration of the camera units with respect to the scene uses non-linear optimisation.

16. A method according to claim 1, wherein the optical sighting apparatus is a weapon sight.

17. A system for performing collaborative sighting, the system comprising plural optical sighting apparatuses that each comprise:

a camera unit arranged to capture an image of a portion of a scene within a field of view of the camera unit; and

a display device arranged to display the image captured by the camera unit, and a central processing apparatus, the plural optical sighting apparatuses and the central processing apparatus, being arranged to communicate over a communications network, wherein the system is arranged to perform a method according to claim 1.

18. A system comprising a plurality of optical sighting apparatuses for performing collaborative sighting, each optical sighting apparatus comprising a camera unit arranged to capture an image of a portion of a scene within a field of view of the camera unit;

a display device arranged to display the image captured by the camera unit, and a processing unit, wherein the processing unit is arranged to;

detect features within an image captured by a first camera unit;

generate descriptors from respective patches of the image at a position of each detected feature;

detect corresponding descriptors generated from the image captured by the camera unit of the said optical sighting apparatus and the image captured by a camera unit of at least one other optical sighting apparatus;

derive geometric calibration of the camera unit of the said optical sighting apparatus and the camera unit of the at least one other optical sighting apparatus with respect to the scene from positions in respective images of the features corresponding to the detected corresponding descriptors, wherein the geometric calibration is derived in real time as the corresponding descriptors are detected;

in respect of at least one target location in the image captured by the camera unit of another optical sighting apparatus, identifying, from the derived geometric calibration of the camera units, a corresponding location relative to the image captured by the camera unit of the said optical sighting apparatus that corresponds to a target location in the scene that itself corresponds to the target location in the image captured by the camera unit of the at least one other optical sighting apparatus; and

control the display device of the said optical sighting apparatus to indicate where the corresponding location lies relative to the displayed image.

19. A system according to claim 18, wherein the processing unit is further arranged to receive, from the other optical sighting apparatus, descriptors and the positions, in an image captured by a camera unit of the at least one other optical sighting apparatus, of features corresponding to the descriptors.

20. A system according to claim 19, wherein the at least one optical sighting apparatus is a weapon sight.