Driving simulation apparatus, wide-angle camera video simulation apparatus, and image deforming/compositing apparatus

• US 8,907,950 B2
• Filed: 09/29/2009
• Issued: 12/09/2014
• Est. Priority Date: 12/01/2008
• Status: Expired due to Fees

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Miyata, Ryosuke, Fujita, Takushi
• Primary Examiner(s): Ma, Tize
• Application Number:
  US12/568,841
• Publication Number:
  US 20100136507A1
• Time in Patent Office: 1,897 Days
• Field of Search: None
• US Class Current: 345/427
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles
• CPC Class Codes: G09B 9/05 the view from a vehicle being simulated G09B9/052, G09B9/058 take precedence

Abstract

A driving simulation apparatus includes an input unit that receives an input of an operation instruction, a vehicle simulation unit that calculates a position and an orientation of a vehicle in a virtual environment in real-time, a camera state calculation unit that calculates a viewpoint position and a viewing direction of a on-vehicle wide-angle camera based on the position and the orientation of the vehicle, and a wide-angle camera simulation unit. The wide-angle camera simulation unit includes an intermediate image rendering unit that generates multiple intermediate images by, for multiple intermediate projection planes that cover a given field of view from the on-vehicle wide-angle camera, performing perspective projection on the virtual environment from the viewpoint position and the viewing direction of the on-vehicle wide-angle camera, and a video rendering unit that generates a on-vehicle wide-angle camera video by deforming and compositing the intermediate images based on a parameter generated based on a lens characteristic of the on-vehicle wide-angle camera that has been recorded in advance.

Claims

1. A driving simulation apparatus comprising:

a processor; and

a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to execute;

an input instruction that receives an input of an operation instruction with respect to a vehicle in a virtual environment;

a vehicle simulation instruction that calculates a position and an orientation of the vehicle in the virtual environment in real-time in accordance with the input of the operation instruction;

a camera state calculation instruction that obtains a viewpoint position and a viewing direction of a on-vehicle wide-angle camera based on the position and the orientation of the vehicle; and

a wide-angle camera simulation instruction including;

an intermediate image rendering instruction that generates a plurality of intermediate images by, for a plurality of intermediate projection planes that cover a given field of view from the on-vehicle wide-angle camera, performing perspective projection on the virtual environment from both the viewpoint position and the viewing direction of the on-vehicle wide-angle camera that are calculated by the camera state calculation unit; and

a video rendering instruction that generates a on-vehicle wide-angle camera video by deforming and compositing the plurality of intermediate images,

wherein the intermediate image rendering instruction generates the plurality of intermediate images by performing perspective projection on the virtual environment from the viewpoint position and the viewing direction of the on-vehicle wide-angle camera, with use of, as the plurality of intermediate projection planes, at least two planes among six planes of a cube in which the viewpoint of the on-vehicle camera is included,

the driving simulation apparatus further comprises a direction vector attachment instruction that, for each node of a two-dimensional mesh arranged on a screen of the on-vehicle wide-angle camera video to be generated, calculates a direction vector that associates the node and a position on an intermediate projection plane from among the plurality of intermediate projection planes, based on the parameter obtained based on a relationship between a direction of an incident light that incidents the lens of the on-vehicle wide-angle camera and a two-dimensional position of an image that is formed by the incident light which transmits through the lens on a capturing plane of the on-vehicle wide-angle camera,

wherein the video rendering instruction generates the on-vehicle wide-angle camera video by mapping textures of the plurality of intermediate images to the two-dimensional mesh with use of the direction vectors.

2. The driving simulation apparatus according to claim 1,

wherein a cube mapping function is used as processing in which the textures of the plurality of intermediate images are mapped to the two-dimensional mesh with use of the direction vectors.

3. The driving simulation apparatus according to claim 1,

wherein partition lines of the two-dimensional mesh arranged on the screen of the on-vehicle wide-angle camera video to be generated have been set so as to overlap with, among boundary lines of the plurality of intermediate projection planes, boundary lines that are to be straight lines on the screen of the on-vehicle wide-angle camera video to be generated.

4. A driving simulation apparatus comprising:

a processor; and

a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to execute;

an input instruction that receives an input of an operation instruction with respect to a vehicle in a virtual environment;

a vehicle simulation instruction that calculates a position and an orientation of the vehicle in the virtual environment in real-time in accordance with the input of the operation instruction;

a camera state calculation instruction that obtains a viewpoint position and a viewing direction of a on-vehicle wide-angle camera based on the position and the orientation of the vehicle; and

a wide-angle camera simulation instruction including;

an intermediate image rendering instruction that generates a plurality of intermediate images by, for a plurality of intermediate projection planes that cover a given field of view from the on-vehicle wide-angle camera, performing perspective projection on the virtual environment from both the viewpoint position and the viewing direction of the on-vehicle wide-angle camera that are calculated by the camera state calculation unit; and

a video rendering instruction that generates a on-vehicle wide-angle camera video by deforming and compositing the plurality of intermediate images, wherein the intermediate image rendering unit generates the plurality of intermediate images by performing perspective projection on the virtual environment from the viewpoint position and the viewing direction of the on-vehicle wide-angle camera, with use of, as the plurality of intermediate projection planes, at least two planes among six planes of a cuboid in which the viewpoint of the on-vehicle camera is included,

the driving simulation apparatus further comprises

a direction vector attachment instruction that, for each node of a two-dimensional mesh arranged on a screen of the on-vehicle wide-angle camera video, calculates a direction vector that associates the node and a position on an intermediate projection plane from among the plurality of intermediate projection planes, based on the parameter obtained based on a relationship between a direction of an incident light that incidents the lens of the on-vehicle wide-angle camera and a two-dimensional position of an image that is formed by the incident light which transmits through the lens on a capturing plane of the on-vehicle wide-angle camera, and

a vector conversion instruction that, based on a positional relationship between the cuboid and the viewpoint of the on-vehicle wide-angle camera, converts the direction vectors calculated by the direction vector attachment unit into vectors that, for each node of the two-dimensional mesh, associate the node and a position on a plane of a post-deformation cube obtained when the cuboid is deformed into a cube whose center is the camera viewpoint,

wherein the video rendering instruction generates the on-vehicle wide-angle camera video by mapping textures of the plurality of intermediate images to the two-dimensional mesh with use of the direction vectors.

5. The driving simulation apparatus according to claim 4,

wherein a cube mapping function is used as processing in which the textures of the plurality of intermediate images are mapped to the two-dimensional mesh with use of the direction vectors.

6. The driving simulation apparatus according to claim 4,

wherein partition lines of the two-dimensional mesh arranged on the screen of the on-vehicle wide-angle camera video to be generated have been set so as to overlap with, among boundary lines of the plurality of intermediate projection planes, boundary lines that are to be straight lines on the screen of the on-vehicle wide-angle camera video to be generated.

7. A driving simulation method, comprising:

a method performed by a processor and a memory which stores a plurality of instructions, which when performed by the processor, cause the processor to perform;

an input instruction that receives an input of an operation instruction with respect to a vehicle in a virtual environment;

a vehicle simulation instruction that calculates a position and an orientation of the vehicle in the virtual environment in real-time in accordance with the input of the operation instruction;

a camera state calculation instruction that obtains a viewpoint position and a viewing direction of a on-vehicle wide-angle camera based on the position and the orientation of the vehicle; and

a wide-angle camera simulation instruction including;

an intermediate image rendering instruction that generates a plurality of intermediate images by, for a plurality of intermediate projection planes that cover a given field of view from the on-vehicle wide-angle camera, performing perspective projection on the virtual environment from both the viewpoint position and the viewing direction of the on-vehicle wide-angle camera that are calculated by the camera state calculation unit; and

a video rendering instruction that generates a on-vehicle wide-angle camera video by deforming and compositing the plurality of intermediate images,

wherein the intermediate image rendering instruction generates the plurality of intermediate images by performing perspective projection on the virtual environment from the viewpoint position and the viewing direction of the on-vehicle wide-angle camera, with use of, as the plurality of intermediate projection planes, at least two planes among six planes of a cube in which the viewpoint of the on-vehicle camera is included,

the driving simulation apparatus further comprises a direction vector attachment instruction that, for each node of a two-dimensional mesh arranged on a screen of the on-vehicle wide-angle camera video to be generated, calculates a direction vector that associates the node and a position on an intermediate projection plane from among the plurality of intermediate projection planes, based on the parameter obtained based on a relationship between a direction of an incident light that incidents the lens of the on-vehicle wide-angle camera and a two-dimensional position of an image that is formed by the incident light which transmits through the lens on a capturing plane of the on-vehicle wide-angle camera,

wherein the video rendering instruction generates the on-vehicle wide-angle camera video by mapping textures of the plurality of intermediate images to the two-dimensional mesh with use of the direction vectors.

8. A non-transitory computer readable storage storing a driving simulation method, the a method performed by a processor and a memory which stores a plurality of instructions, which when performed by the processor, cause the processor to perform;

an input instruction that receives an input of an operation instruction with respect to a vehicle in a virtual environment;

a vehicle simulation instruction that calculates a position and an orientation of the vehicle in the virtual environment in real-time in accordance with the input of the operation instruction;

a camera state calculation instruction that obtains a viewpoint position and a viewing direction of a on-vehicle wide-angle camera based on the position and the orientation of the vehicle; and

a wide-angle camera simulation instruction including;

an intermediate image rendering instruction that generates a plurality of intermediate images by, for a plurality of intermediate projection planes that cover a given field of view from the on-vehicle wide-angle camera, performing perspective projection on the virtual environment from both the viewpoint position and the viewing direction of the on-vehicle wide-angle camera that are calculated by the camera state calculation unit; and

a video rendering instruction that generates a on-vehicle wide-angle camera video by deforming and compositing the plurality of intermediate images,

wherein the intermediate image rendering instruction generates the plurality of intermediate images by performing perspective projection on the virtual environment from the viewpoint position and the viewing direction of the on-vehicle wide-angle camera, with use of, as the plurality of intermediate projection planes, at least two planes among six planes of a cube in which the viewpoint of the on-vehicle camera is included,

the driving simulation apparatus further comprises a direction vector attachment instruction that, for each node of a two-dimensional mesh arranged on a screen of the on-vehicle wide-angle camera video to be generated, calculates a direction vector that associates the node and a position on an intermediate projection plane from among the plurality of intermediate projection planes, based on the parameter obtained based on a relationship between a direction of an incident light that incidents the lens of the on-vehicle wide-angle camera and a two-dimensional position of an image that is formed by the incident light which transmits through the lens on a capturing plane of the on-vehicle wide-angle camera,

wherein the video rendering instruction generates the on-vehicle wide-angle camera video by mapping textures of the plurality of intermediate images to the two-dimensional mesh with use of the direction vectors.