Method for recognizing objects in a set of images recorded by one or more cameras

• US 9,117,269 B2
• Filed: 12/01/2009
• Issued: 08/25/2015
• Est. Priority Date: 12/01/2008
• Status: Active Grant

• Current Assignee: Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno
• Original Assignee: Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno
• Inventors: van der Mark, Wannes, Tettelaar, Peter, den Hollander, Richard Jacobus Maria
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US13/132,007
• Publication Number:
  US 20110293142A1
• Time in Patent Office: 2,093 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

Method for improving the visibly of objects and recognizing objects in a set of images recorded by one or more cameras, the images of said set of images being made from mutual different geometric positions, the method comprising the steps or recording a set or subset of images by means of one camera which is moved rather freely and which makes said images during its movement, thus providing an array of subsequent images, estimating the camera movement between subsequent image recordings, also called ego-motion hereinafter, based on features of those recorded images, registering the camera images using a synthetic aperture method, recognizing said objects.

Claims

1. A method for recognizing objects in recorded images, the images being obtained from mutually different geometric positions, the method comprising the following steps:

recording a set of images by means of a camera which is freely moved and which optically makes said images during its movement, thus providing an array of successive images;

estimating the camera movement between successive image recordings based on positions of features detected in those recorded images;

using a synthetic aperture method based on the camera movements to compute synthetic aperture images for focal surfaces at different depths, the synthetic aperture images being computed from the array of successive images, wherein said computing comprises;

computing respective image transformations from each image of the array of successive images to each of a plurality of respective 3D surfaces, each of the 3D surfaces corresponding to a respective focal surface at a respective depth;

warping each of the images to each of the 3D surfaces according to the respective image transformations; and

computing each of the synthetic aperture images by fusing the images warped to a respective one of the 3D surfaces.

2. A method according to claim any one of claims 1, recognizing an object from a result of the synthetic aperture method.

3. A method according to claim 1, comprising the following steps:

recording the images at mutually different 3D positions;

detecting image features in the images;

matching detected image features from mutually different images;

estimating camera motion from the matched features;

selecting a 3D surface;

computing image transformations from the images to the selected 3D surface;

warping the images to the 3D surface according to the image transformations;

fusing the warped images to the 3D surface and either displaying or further processing or evaluating the results and optionally add or remove additional images.

4. A method according to claim 1, wherein the one camera is a stereo camera, the set of images being a stereo image sequence, the estimation of camera movement comprising detecting relationships between left and right stereo images from a common stereo camera position and tracking features over time through the stereo image sequence.

5. A method according to claim 1, wherein said estimating comprises estimating camera positions relative to the features of those recorded images that are visible in the recorded images, said computing the plurality of synthetic aperture images comprising computing the synthetic aperture image for the respective depths based on the camera positions.

6. A device for recognizing objects in a set of images, the device comprising

a camera for recording a set of images,

a controller configured to actuate said camera to record the images of said set of images obtained optically from mutual different geometric positions during movements of the device, thus providing an array of successive images;

a motion estimator or a connection to a motion estimator arranged to estimate the camera movements between the recorded successive images based on positions of features detected in those recorded successive images;

a synthetic aperture processor arranged to process the camera images by means of a synthetic aperture method to compute synthetic aperture images at different depths, wherein the synthetic aperture images are computed from the array of successive images, wherein said synthetic aperture processor is configured to;

compute respective image transformations from each image of the array of successive images to each of a plurality of respective 3D surfaces, each of the 3D surfaces corresponding to a respective focal surface at a respective depth;

warp each of the images to each of the 3D surfaces according to the respective image transformations; and

compute each of the synthetic aperture images by fusing the images warped to a respective one of the 3D surfaces.

7. A device according to claim 6, wherein the motion estimator is configured to detect image features, match said features and estimate camera motion;

the synthetic aperture processor being configured to select a 3-D surface, compute image transformations to the selected 3-D surface, warp images to the 3-D surface, fuse images to the 3-D surface.

8. Device according to claim 6, the device being, comprising or being comprised in a handheld device selected from the group consisting of a set of electronic binoculars and a mobile phone with a camera.

9. Device according to claim 6, the device being, comprising or being comprised in a head mounted device selected from the group consisting of night-vision, goggles and a head mounted display.

10. Device according to claim 6, the device being, comprising or being comprised in an optical system, that is intended for targeting or aiming, selected from the group consisting of a gun scope, bombsight and a target designator.

11. Device according to claim 6, the device being, comprising or being comprised in an observation, reconnaissance or surveillance system that is mounted on a flying platform selected from the group consisting of an unmanned airborne vehicle, airplane and a helicopter.

12. Device according to claim 6, the device being, comprising or being comprised in an observation, reconnaissance or surveillance system that is mounted on a mobile platform selected from the group consisting of a ground vehicle, motorcycle, ship and submarine.

13. Device according to claim 6, the device being, comprising or being comprised in a camera system mounted on a space born system selected from the group consisting of a space ship, satellite and a planetary rover.

14. Device according to claim 6, the device being, comprising or being comprised in a remote viewing system selected from the group consisting of a periscope and a telescope.

15. Device according to claim 6, the device being, comprising or being comprised in a camera system that is moved through the 3-D space with a robot manipulator.

16. Device according to claim 6, the device being, comprising or being comprised in an imaging system that are sensitive to the visual, infra-red, ultraviolet or other parts of the electromagnetic spectrum.

17. Device according to claim 6, the device being, comprising or being comprised in a biometric identification system that is used for surveillance, security or access control.

18. A device according to claim 6, wherein said motion estimator is configured to estimate camera positions relative to the features of those recorded images that are visible in the recorded images, the synthetic aperture processor being configured to compute the synthetic aperture image for the respective focal surface at the respective depth based on the camera positions.

19. A computer, comprising a program of instructions for a programmable processor system that, when executed by the programmable processor system, will cause the programmable processor system to

record a set or subset of images from one camera which is freely moved and which makes said images during its movement, thus providing an array of subsequent images;

estimate the camera movement between subsequent image recordings based on positions of features detected in those recorded images;

use a synthetic aperture method based on the camera movements to compute synthetic aperture images for focal surfaces at different depths, the synthetic aperture images being computed from the array of successive images, wherein the synthetic aperture image computation includes computing respective image transformations from each image of the array of successive images to each of a plurality of respective 3D surfaces, each of the 3D surfaces corresponding to a respective focal surface at a respective depth;

warping each of the images to each of the 3D surfaces according to the respective image transformations; and

computing each of the synthetic aperture images by fusing the images warped to a respective one of the 3D surfaces.