System and method for mixing a scene with a virtual scenario

• US 10,187,589 B2
• Filed: 12/19/2008
• Issued: 01/22/2019
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Saab AB
• Original Assignee: Saab AB
• Inventors: Hagstrom, Mans, Erlandsson, Ulf, Borg, Johan, Isaksson, Folke, Andersson, Ingmar, Tengblad, Adam
• Primary Examiner(s): Harper, Tramar
• Assistant Examiner(s): Wong, Jeffrey
• Application Number:
  US13/140,973
• Publication Number:
  US 20120115598A1
• Time in Patent Office: 3,686 Days
• Field of Search: 463 31- 34, 463 37, 348E15001, 348E1301-E1305, 348578, 348580
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/003 for military purposes and tactics
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A system and method for mixing a scene with a virtual scenario. An image capturing unit is arranged to capture at least one image so as to cover the scene from a first viewpoint. An image representation generation unit is arranged to generate at least one image representation based on the captured image. A game engine unit is arranged to generate a virtual scenario. An image processing unit is arranged to adapt the at least one image representation based on the generated virtual scenario so as to provide a virtual video sequence.

Claims

1. A system for providing a virtual video sequence, comprising:

an image capturing unit configured to capture images so as to cover a scene viewed from a first viewpoint in a global geographical coordinate system, wherein the first viewpoint is a viewpoint of the scene from a first position in the global geographical coordinate system;

an image representation generation unit configured to generate at least one image representation based on said captured images, said image representation comprising an image depth map comprising information about a distance to objects in said captured images provided in the global geographical coordinate system and texture data;

a game engine unit configured to generate a virtual scenario comprising virtual objects in a coordinate system aligned with the global geographical coordinate system, wherein generating the virtual scenario comprises forming depth map data and texture data for at least one generated virtual object;

a warping unit configured to warp the image depth map to a warped depth map related to a second viewpoint so as to form a warped representation of the scene viewed from the second viewpoint, the warped depth map comprises texture data, wherein the second viewpoint is obtained from a position of the coordinate system aligned with the global geographical coordinate system that is different than the first position of the first viewpoint;

an image processing unit configured to mix the scene with the virtual scenario based on the warped representation of the scene viewed from the second viewpoint and on the generated scenario;

wherein the second viewpoint is a virtual viewpoint that is a view at a distance from the image capturing unit and is arbitrarily chosen within an area around the first viewpoint,

wherein the image processing unit is configured to adapt the warped depth map comprising texture data based on the formed depth map data for said generated virtual scenario so as to provide the virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario, and

a position/posture estimation unit configured to estimate position and/or posture information at said second viewpoint in relation to said first viewpoint,

wherein the position/posture estimation unit is configured to determine a field of view from the second viewpoint,

wherein the position/posture estimation unit is configured to elect one or a plurality of images with associated depth maps corresponding to the determined field of view,

wherein the warping unit is configured to process the depth maps of the images elected by the position/posture estimation unit, and

wherein the warping unit is configured to adapt the depth map for the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.

2. The system according to claim 1, wherein the game engine unit is configured to generate a sequence of virtual objects in accordance with a tactical scenario.

3. The system according to claim 1, further comprising:

a display unit configured to display the virtual video sequence.

4. The system according to claim 3, wherein the display unit is configured to act as a sight.

5. The system according to claim 4, wherein the display unit is configured to act as a sight of a weapon.

6. The system according to claim 4, wherein the display unit is formed at a sight.

7. The system according to claim 3, wherein the display unit is a computer screen.

8. The system according to claim 1, wherein the warping unit is configured to update the depth map in real time to follow movements of the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.

9. The system according to claim 1, wherein the virtual video sequence is formed based on the texture data.

10. A method for generating a virtual scenario, the method comprising:

providing a representation of a scene related to a first viewpoint, said representation comprising an image depth map data and texture data, the image depth map comprising information about a distance to objects in images provided in a global geographical coordinate system and texture data, wherein the first viewpoint is a viewpoint of the scene from a first position in the global geographical coordinate system;

providing a representation of the scene viewed from a second viewpoint that is a virtual viewpoint that is different from the first viewpoint, wherein providing a warped image representation of the scene viewed from the second viewpoint comprises warping at least the depth map data of the representation of the scene so as to provide a warped depth map corresponding to the second viewpoint and comprising texture data;

generating the virtual scenario utilizing a game engine unit, wherein the virtual scenario comprising virtual objects is generated in a coordinate system aligned with a coordinate system of the image depth map, wherein generating the virtual scenario comprises forming depth map data and texture data for at least one generated virtual object;

mixing the scene with the virtual scenario based on the representation of the scene viewed from the second viewpoint and on the generated scenario;

choosing the second viewpoint arbitrarily within an area around the first viewpoint, wherein the second viewpoint is from a position that is different than the first position of the first viewpoint,

wherein mixing the scene with the virtual scenario comprises adapting the warped image representation based on the formed depth map data for said generated virtual scenario so as to provide a virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario,

estimating position and posture information at said second viewpoint in relation to said first viewpoint;

determining a field of view from the second viewpoint;

electing at least one image with associated depth maps corresponding to the determined field of view; and

processing the depth maps of the elected images so as to provide the representation of the scene viewed from the second viewpoint.

11. The method according to claim 10, wherein the generation of a virtual scenario comprises forming at least one of depth map data or texture data for at least one virtual object.

12. A virtual scenario generating device, comprising:

a memory unit configured to store images covering a scene from a first viewpoint, wherein the images each are associated to position information and posture information, wherein the position information comprises a coordinate in a global coordinate system and the posture information comprises a compass bearing, wherein the first viewpoint is a viewpoint of the scene from a first position in a global geographical coordinate system;

an image representation generation unit configured to generate at least one image representation in the global coordinate system based on the images stored in the memory unit, said image representation comprising an image depth map comprising information about a distance to objects in the images provided in the global geographical coordinate system and texture data;

a game engine unit configured to generate a virtual scenario comprising virtual objects in a coordinate system aligned with the global coordinate system, wherein generating the virtual scenario comprises forming depth map data and texture data for at least one generated virtual object;

a warping unit configured to warp the image depth map to a warped depth map related to a second viewpoint so as to form a representation of the scene viewed from the second viewpoint, wherein the second viewpoint is from a position of the coordinate system aligned with the global geographical coordinate system that is different than the first position of the first viewpoint, and the warped depth map comprising texture data;

an image processing unit configured to mix the scene with the virtual scenario based on the representation of the scene viewed from the second viewpoint and on the virtual scenario;

wherein the second viewpoint is a virtual viewpoint that is arbitrarily chosen within an area around the first viewpoint,

wherein the image processing unit is configured to adapt the warped depth map based on the formed depth map data for the generated virtual scenario so as to provide a virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario, and

a position/posture estimation unit configured to estimate position and/or posture information at said second viewpoint in relation to said first viewpoint,

wherein the position/posture estimation unit is configured to determine a field of view from the second viewpoint,

wherein the position/posture estimation unit is configured to elect one or a plurality of images with associated depth maps corresponding to the determined field of view and wherein the warping unit is configured to process the depth maps of the images elected by the position/posture estimation unit and wherein the warping unit is configured to adapt the depth map for the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.

13. The virtual scenario generating device of claim 12, wherein the warping unit is configured to update the depth map in real time to follow movements of the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.

14. A training system, comprising:

a memory unit configured to store images covering a scene from a first viewpoint, wherein the images are each associated with position information and posture information, wherein the position information comprises a coordinate in a global coordinate system and the posture information comprises a compass bearing, wherein the first viewpoint is a viewpoint of the scene from a first coordinate in a global geographical coordinate system;

an image representation generation unit configured to generate at least one image representation in the global coordinate system based on the images stored in the memory unit, said image representation comprising an image depth map comprising texture data and comprising information about a distance to objects in the images provided in the global geographical coordinate system;

a game engine unit configured to generate a virtual scenario comprising virtual objects in a coordinate system aligned with the global coordinate system, wherein generating the virtual scenario comprises forming depth map data and texture data for at least one generated virtual object;

a warping unit configured to warp the image depth map to a warped depth map related to a second viewpoint so as to form a warped representation of the scene viewed from the second viewpoint, wherein the second viewpoint is from a position of the coordinate system aligned with the global geographical coordinate system that is different than the first coordinate of the first viewpoint, the warped depth map comprising texture data;

an image processing unit configured to mix the scene with the virtual scenario based on the warped representation of the scene viewed from the second viewpoint and on the virtual scenario,

wherein the second viewpoint is a virtual viewpoint that is arbitrarily chosen within an area around the first viewpoint,

wherein the image processing unit is configured to adapt the warped depth map based on the formed depth map data for the generated virtual scenario so as to provide a virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario,

a weapon having a sight configured to display the virtual video sequence and a position/posture estimation unit for estimating position and/or posture information at said second viewpoint in relation to said first viewpoint,

wherein the position/posture estimation unit is configured to determine a field of view from the second viewpoint,

wherein the position/posture estimation unit is configured to elect one or a plurality of images with associated depth maps corresponding to the determined field of view and wherein the warping unit is configured to process the depth maps of the images elected by the position/posture estimation unit and wherein the warping unit is configured to adapt the depth map for the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.

15. The training system according to claim 14, wherein the warping unit is configured to update the depth map in real time to follow movements of the second viewpoint as it is moving based on updated images with the associated depth maps elected by the position/posture estimation unit.