Method and system for determining a relation between a first scene and a second scene

• US 9,792,701 B2
• Filed: 11/08/2011
• Issued: 10/17/2017
• Est. Priority Date: 11/08/2011
• Status: Active Grant

• Current Assignee: Saab AB
• Original Assignee: Saab AB
• Inventors: Olofsson, Michael, Isaksson, Folke, Brege, Magnus, Andersson, Ingmar, Carlbom, Per, Borg, Johan, Haglund, Leif, Rosenqvist, Fredrik
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US14/355,144
• Publication Number:
  US 20150078652A1
• Time in Patent Office: 2,170 Days
• Field of Search: None
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

The present invention relates to a system (200) and method for determining a relation between a first scene and a second scene. The method comprises the steps of generating at least one sensor image of a first scene with at least one sensor; accessing information related to at least one second scene, said second scene encompassing said first scene, and matching the sensor image with the second scene to map the sensor image onto the second scene. The step of accessing information related to the at least one second scene comprises accessing a 3D map comprising geocoded 3D coordinate data. The mapping involves associating geocoding information to a plurality of positions in the sensor image based on the coordinate data of the second scene.

Claims

1. Method for determining a relation between a first scene and a second scene, said method comprising the steps of

generating at least one sensor image of a first scene with at least one sensor;

accessing information related to at least one second scene by a processing and control module, said second scene encompassing said first scene; and

matching, by the processing and control module, the sensor image with the second scene to map the sensor image onto the second scene,

wherein;

the step of accessing information related to the at least one second scene comprises;

accessing a textured 3D map comprising geocoded 3D coordinate data by the processing and control module, wherein the textured 3D map comprises a 3D model having texture information associated to one or more surfaces of the 3D model; and

extracting, from the textured 3D map and by the processing and control module, the texture information comprising the corresponding geocoded 3D coordinate data related to the at least one second scene, said extracting generating 2D extracted texture information; and

the step of matching the sensor image with the second scene to map the sensor image onto the second scene comprises;

matching, by the processing and control module, the 2D extracted texture information corresponding to at least one of the one or more surfaces of the 3D model and comprising the corresponding geocoded 3D coordinate data with 2D image texture information of the sensor image, the 2D image texture information of the sensor image corresponding to one or more surfaces within the sensor image, and

associating, by the processing and control module, the extracted corresponding geocoded 3D coordinate data to a plurality of positions in the sensor image based on the matching of the 2D extracted texture information with the 2D image texture information of the sensor image.

2. Method according to claim 1, wherein the 3D map comprises a 3D model of the environment comprising 3D coordinate data given in a geo-referenced coordinate system.

3. Method according to claim 1, further comprising:

displaying the sensor image on a display;

receiving a selection of a point on the displayed image of the sensor image;

extracting coordinate information related to the selected point in the sensor image.

4. Method according to claim 1, further comprising a step of forming a geocoded reference image from the second scene, wherein a geographical extension of the geocoded reference image corresponds to a geographical extension of the sensor image.

5. Method according to claim 4, further comprising a step of identifying geocoded additional information from the geocoded sensor image in relation to the corresponding geocoded reference image.

6. Method according to claim 5, wherein the geocoded additional information comprises information related to a difference in at least one of shape or size of at least one object in the sensor image in relation to the corresponding geocoded reference image.

7. Method according to claim 5, wherein the additional information comprises information relating to at least one texture deviation in the sensor image in relation to the corresponding geocoded reference image.

8. Method according to claim 5, wherein the additional information comprises additional geocoded coordinate data providing enhanced resolution in relation to the reference image.

9. Method according to claim 8, further comprising a step of updating the geocoded reference image based on the additional information.

10. Method according to claim 5, wherein the at least one sensor comprises an IR sensor, and wherein the geocoded additional information comprises IR data.

11. Method according to claim 5, further comprising a step of presenting the geocoded additional information.

27. Method according to claim 1, wherein the matching of the extracted texture information comprises directly matching the extracted texture information with the sensor image.

28. Method according to claim 1, wherein the sensor image generated is unaltered during the matching process.

29. Method according to claim 1, wherein the method further comprises the steps of:

identifying geocoded additional information from the sensor image in relation to the corresponding reference image, the geocoded additional information comprising changes in a portion of the 3D map; and

updating the 3D map based on the identified geocoded additional information.

30. Method according to claim 1, wherein the texture information includes height data.

32. Method according to claim 1, wherein the first scene is captured from a first point of reference and the second scene is captured from a second point of reference different from the first point of reference.

12. System for determining a relation between a first scene and a second scene, said system comprising:

at least one sensor configured to capture at least one image of the first scene;

a map comprising geographic information; and

a processing and control module comprising a processor and coupled to a storage medium, wherein the processing and control module accesses from the map information related to at least one second scene, said second scene encompassing said sensor image, and matches the sensor image with the second scene to map the sensor image onto the second scene,

wherein;

the map is a textured 3D map comprising geocoded 3D coordinate data, wherein the textured 3D map comprises a 3D model having texture information associated to one or more surfaces of the 3D model;

the accessing of the map by the processing and control module comprises;

accessing the textured 3D map, and

extracting, from the textured 3D map, the texture information comprising the corresponding geocoded 3D coordinate data related to the at least one second scene, said extracting generating 2D extracted texture information; and

the processing and control module matches the 2D extracted texture information corresponding to at least one of the one or more surfaces of the 3D model and comprising the corresponding geocoded 3D coordinate data with 2D image texture information of the sensor image, the 2D image texture information of the sensor image corresponding to one or more surfaces within the sensor image, and associates the extracted corresponding geocoded 3D coordinate data to a plurality of positions in the sensor image based on the coordinate data of the second scene based on the matching of the 2D extracted texture information with the 2D image texture information of the sensor image.

13. System according to claim 12, wherein the 3D map comprises a 3D model of the environment comprising 3D coordinate data given in a geo-referenced coordinate system.

14. System according to claim 12 further comprising means for extracting coordinate information related to a selected point in the sensor image.

15. System according to claim 14, further comprising a display configured to display the sensor image and selection means for selecting the point from the displayed sensor image.

16. System according to claim 12, wherein the processing and control module is configured to form a geocoded reference image from the second scene, wherein a geographical extension of the geocoded reference image corresponds to the geographical extension of a sensor image.

17. System according to claim 16, wherein the processing and control module further is configured to identify geocoded additional information from the geocoded sensor image in relation to the corresponding geocoded reference image.

18. System according to claim 17, wherein the geocoded additional information comprises information relating to at least one of:

at least one of addition or removal of at least one object in the sensor image in relation to the corresponding geocoded reference image;

at least one of a difference in at least one of shape or size of at least one object in the sensor image in relation to the corresponding geocoded reference image;

or

at least one of texture deviation in the sensor image in relation to the corresponding geocoded reference image.

19. System according to claim 17 wherein the additional information comprises additional geocoded coordinate data providing enhanced accuracy in relation to the reference image.

20. System according to claim 19, wherein the 3D map is updated with the additional information based on the geocoded reference image based on the additional information.

21. System according to claim 17, wherein the at least one sensor comprises an IR sensor and in that the processing and control module is configured to providing IR data as geocoded additional information.

22. System according to claim 12, further comprising a presentation unit configured to present information related to the sensor image associated with geocoding information.

23. System according to claim 22, wherein the presentation unit further is configured to present information related to the geocoded reference image.

24. System according to claim 22, wherein the presentation unit further is configured to present information related to the geocoded additional information.

25. System according to claim 22, further comprising:

a first platform comprising at least the at least one sensor and a first communication unit for transmittal of at least first data related to the captured at least one image of the first scene; and

a second platform comprising at least the presentation unit and a second communication unit for reception of at least second data related to the sensor image associated with geocoding information.

31. System according to claim 17, wherein the processing and control module is further configured to:

identify geocoded additional information from the sensor image in relation to the corresponding reference image, the geocoded additional information comprising changes in a portion of the 3D map; and

update the 3D map based on the identified geocoded additional information.

33. System according to claim 12, wherein the first scene is captured from a first point of reference and the second scene is captured from a second point of reference different from the first point of reference.

26. A computer program product for determining a relation between a first scene and a second scene, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

an executable portion configured for receiving first data related to at least one sensor image of a first scene from at least one sensor;

an executable portion configured for receiving information related to at least one second scene, said second scene encompassing said first scene; and

an executable portion configured for matching the sensor image with the second scene to map the sensor image onto the second scene,

wherein;

the receiving of information related to the at least one second scene comprises;

accessing a textured 3D map comprising geocoded 3D coordinate data, wherein the textured 3D map comprises a 3D model having texture information associated to one or more surfaces of the 3D model; and

extracting, from the textured 3D map, the texture information comprising the corresponding geocoded 3D coordinate data related to the at least one second scene, said extracting generating 2D extracted texture information; and

the step of matching the sensor image with the second scene to map the sensor image onto the second scene comprises;

matching the 2D extracted texture information corresponding to at least one of the one or more surfaces of the 3D model and comprising the corresponding geocoded 3D coordinate data with 2D image texture information of the sensor image, the 2D image texture information of the sensor image corresponding to one or more surfaces within the sensor image, and

associating the extracted corresponding geocoded 3D coordinate data to a plurality of positions in the sensor image based on the matching of the 2D extracted texture information with the 2D image texture information of the sensor image.