SYSTEM AND METHOD FOR MIXING A SCENE WITH A VIRTUAL SCENARIO

• US 20120115598A1
• Filed: 12/19/2008
• Published: 05/10/2012
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Saab AB
• Original Assignee: Saab AB
• Inventors: Borg, Johan, Isaksson, Folke, Andersson, Ingmar, Tengblad, Adam, Hagström, Måns, Erlandsson, Ulf
• Granted Patent: US 10,187,589 B2
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/003 for military purposes and tactics
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A system and method for mixing a scene with a virtual scenario. An image capturing unit is arranged to capture at least one image so as to cover the scene from a first viewpoint. An image representation generation unit is arranged to generate at least one image representation based on the captured image. A game engine unit is arranged to generate a virtual scenario. An image processing unit is arranged to adapt the at least one image representation based on the generated virtual scenario so as to provide a virtual video sequence.

Claims

1. A system for providing a virtual video sequence, comprising:

an image capturing unit arranged to capture images so as to cover a scene from a first viewpoint,

an image representation generation unit arranged to generate at least one image representation based on said captured images,

a game engine unit arranged to generate a virtual scenario,

a warping unit for warping each image representation generated by said image representation generation unit to a representation related to a second viewpoint so as to form a representation of the scene viewed from the second viewpoint, and

an image processing unit arranged to mix the scene with the virtual scenario based on the representation of the scene viewed from the second viewpoint and on the generated scenario,

wherein the second viewpoint is arbitrary chosen within an area around the first viewpoint and

wherein the image processing unit is arranged to adapt the warped image representation based on said generated virtual scenario so as to provide the virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario.

2. The system according to claim 1, wherein the game engine unit is arranged to generate the virtual scenario in a coordinate system aligned with the coordinate system of the image representations.

3. The system according to claim 1, wherein the game engine unit is arranged to generate at least one virtual object.

4. The system according to claim 1, wherein the game engine unit is arranged to generate a sequence of virtual objects in accordance with a tactical scenario.

5. The system according to claim 1, further comprising:

a position/posture estimation unit for estimating position and/or posture information at said second viewpoint in relation to said first viewpoint.

6. The system according to claim 5, wherein the position/posture estimation unit is arranged to determine the field of view from the second viewpoint.

7. The system according to claim 6, wherein the position/posture estimation unit is arranged to elect one or a plurality of images with associated depth maps corresponding to the determined field of view.

8. The system according to claim 7, wherein the warping unit is arranged to process the depth maps of the images elected by the position/posture estimation unit.

9. The system according to claim 8, wherein the warping unit is arranged to adapt the depth map for the second viewpoint as it is moving based on updated elections by the position/posture estimation unit.

10. The system according to claim 1, further comprising:

a display unit arranged to display the virtual video sequence.

11. The system according to claim 10, wherein the display unit is arranged to act as a sight.

12. The system according to claim 11, wherein the display unit is arranged to act as a sight of a weapon.

13. The system according to claim 11, wherein the display unit is formed at a sight.

14. The system according to claim 10, wherein the display unit is a computer screen.

15. A method for generating a virtual scenario, the method comprising:

providing a representation of a scene related to a first viewpoint, said representation comprising depth map data and texture data,

providing a representation of the scene viewed from a second viewpoint, wherein providing a representation of the scene viewed from a second viewpoint comprises warping at least the depth map data of the representation of the scene so as to correspond to the second viewpoint,

generating a virtual scenario utilizing a game engine unit, and

mixing the scene with the virtual scenario based on the representation of the scene viewed from the second viewpoint and on the generated scenario, and

choosing the second viewpoint arbitrary within an area around the first viewpoint, and

wherein mixing the scene with the virtual scenario comprises adapting the warped image representation based on said generated virtual scenario so as to provide a virtual video sequence comprising the warped representation of the scene viewed from the second viewpoint mixed with the virtual scenario.

16. The method according to claim 15, wherein the generation of a virtual scenario comprises forming depth map data and/or texture data for at least one virtual object.