IMAGE RENDERING DEVICE, IMAGE RENDERING METHOD, AND IMAGE RENDERING PROGRAM FOR RENDERING STEREOSCOPIC IMAGES

• US 20130100132A1
• Filed: 03/05/2012
• Published: 04/25/2013
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Katayama, Tomoko, Tanaka, Keiichi, Oto, Hidetaka, Yamaji, Osamu
• Granted Patent: US 9,055,277 B2
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

An image rendering device realizes stereoscopic viewing of composite images generated by compositing background three-dimensional models and foreground three-dimensional models each defined in three-dimensional modeling space. A texture mapping unit converts background image data into two or more viewpoint textures and maps each viewpoint texture to a background three-dimensional model in the three-dimensional modeling space. A viewport conversion unit extracts, for each of the two or more viewpoint textures, a viewport image from the background three-dimensional model mapped with the viewpoint texture and from the foreground three-dimensional model. By comparing a parallax of the foreground three-dimensional model occurring in stereoscopic playback with a parallax of the intra-texture object occurring in stereoscopic playback, the device determines depth relationship between the foreground three-dimensional model and the intra-texture object and adjusts positional relationship between the foreground three-dimensional model and the intra-texture object based on a result of the determination.

Claims

1. An image rendering device for realizing stereoscopic viewing of composite images generated by compositing background three-dimensional models and foreground three-dimensional models each defined in three-dimensional modeling space, the image rendering device comprising:

a coordinate converter that performs a coordinate conversion on modeling data defining a geometry of a background three-dimensional model and a geometry of a foreground three-dimensional model, so that coordinates describing vertices of the modeling data are converted into three-dimensional coordinates describing corresponding points in the three-dimensional modeling space;

a texture mapper that converts background image data into two or more viewpoint textures and maps each viewpoint texture to a background three-dimensional model in the three-dimensional modeling space;

a viewport converter that extracts, for each of the two or more viewpoint textures, a viewport image from the background three-dimensional model mapped with the viewpoint texture and from the foreground three-dimensional model; and

a manager that manages the foreground three-dimensional model and the background three-dimensional model, wherein

the two or more viewpoint textures include an intra-texture object, and

the manager determines depth relationship between the foreground three-dimensional model and the intra-texture object, by comparing a parallax of the foreground three-dimensional model occurring in stereoscopic playback with a parallax of the intra-texture object occurring in stereoscopic playback and adjusts positional relationship between the foreground three-dimensional model and the intra-texture object based on a result of the determination.

2. The image rendering device according to claim 1, wherein

the adjustment of positional relationship is carried out by causing the coordinate converter to change a position of the foreground three-dimensional model in the three-dimensional modeling space to a position further away from the background three-dimensional model.

3. The image rendering device according to claim 1, wherein

the conversion of the background image data into the two or more viewpoint textures is carried out by acquiring a depth image associated with the background image data and performing depth image based rendering based on the acquired depth image, and

the adjustment of positional relationship is carried out by correcting the depth image data and performing depth image based rendering based on the corrected depth image to newly generate two or more textures.

4. The image rendering device according to claim 1, wherein

the adjustment of positional relationship is carried out by masking, among pixels of the foreground three-dimensional model projected in each viewport image, every pixel having a depth greater than a depth of the intra-texture object and rendering unmasked pixels of the foreground three-dimensional model.

5. The image rendering device according to claim 1, wherein

the two or more viewpoint textures are generated by acquiring background depth image data and performing depth image rendering based on the acquired background depth image data, and

the adjustment of positional relationship is carried out by:

converting the intra-texture object into modeling data;

storing, into a depth buffer, three-dimensional coordinates describing a geometry of modeling data converted from the intra-texture object, along with three-dimensional coordinates describing the foreground three-dimensional model; and

enabling a depth testing function to calculate lighting values based on the sets of three-dimensional coordinates stored in the depth buffer.

6. The image rendering device according to claim 5, wherein of the three-dimensional coordinates describing the modeling data converted from the intra-texture object, a Z coordinate is calculated based on a parallax between a pixel in one of the two or more viewpoint textures and a corresponding pixel in another one of the two or more viewpoint textures.

7. The image rendering device according to claim 1, wherein

when a line-of-sight direction is changed, the viewport converter:

newly extracts a viewport image corresponding to the new line-of-sight direction from the background three-dimensional model mapped with one of the two or more viewpoint textures and from the foreground three-dimensional model; and

newly acquires two or more viewport images by performing depth image based rendering of the newly extracted viewport image based on background depth image data.

8. The image rendering device according to claim 1, wherein

the adjustment of positional relationship refers to a geometry transformation applied to the foreground three-dimensional model by manipulating the coordinates describing vertices defining the geometry of the foreground three-dimensional model.

9. The image rendering device according to claim 1, wherein

the extraction of viewport images is carried out according to a viewpoint position in the three-dimensional modeling space, and

the adjustment of positional relationship is carried out by changing the viewpoint position.

10. The image rendering device according to claim 1, wherein

the parallax of the foreground three-dimensional model refers to a parallax PS between points SL and SR on a precondition that a point S in the foreground three-dimensional model corresponds to a set of coordinates SL on a left-view viewport and to a set of coordinates SR on a right-view viewport, and is expressed in Equation 1 below,

PS=2p*(Sz−

D)/Sz,  

 

Equation 1

where

Sz denotes a coordinate in depth direction of the foreground image in a view coordinate system,

D denotes a distance between a camera position and a display, and

2p denotes a distance between a left-view camera and a right-view camera.

11. The image rendering device according to claim 1, wherein the parallax of the intra-texture object refers to a parallax PB between points BL and BR on a precondition that a point B in the intra-texture object applied to an inner surface of the background three-dimensional model corresponds to a set of coordinates (BLx, BLy, BLz) on a left-view viewport and to a set of coordinates (BRx, BRy, BRz) on a right-view viewport, and is expressed in Equation 2 below,

PB={(D*BRx+p*(BRz−

D))/BRz}−

{(D*BLx−

p*(BLz−

D))/BLz},  

 

Equation 2

where

D denotes a distance between a camera position to a display, and

p denotes half a distance between a left-view camera and a right-view camera.

12. The image rendering device according to claim 1, further comprising:

a position and direction determiner that determines, in accordance with a user operation, a current viewpoint location on a map and a current line-of-sight direction; and

a downloader that generates an image acquisition request using geographic information corresponding to the current viewpoint location on the map and transmits the image acquisition request to an image collection server, thereby to download a street view file, wherein

the street view file transmitted from the image collection server has a location attribute matching the geographic information included in the image acquisition request, and

the background image data is contained in the downloaded street view file.

13. The image rendering device according to claim 12, wherein

the background image data is a panoramic viewpoint image representing a view of surroundings as seen from a geographic location corresponding to the geographic information included in the image acquisition information.

14. An image rendering method for realizing stereoscopic viewing of composite images generated by compositing background three-dimensional models and foreground three-dimensional models each defined in three-dimensional modeling space, the image rendering method comprising:

a coordinate conversion step of performing a coordinate conversion on modeling data defining a geometry of a background three-dimensional model and a geometry of a foreground three-dimensional model, so that coordinates describing vertices of the modeling data are converted into three-dimensional coordinates describing corresponding points in the three-dimensional modeling space;

a texture mapping step of converting background image data into two or more viewpoint textures and mapping each viewpoint texture to a background three-dimensional model in the three-dimensional modeling space;

a viewport converting step of extracting, for each of the two or more viewpoint textures, a viewport image from the background three-dimensional model mapped with the viewpoint texture and from the foreground three-dimensional model; and

a managing step of managing the foreground three-dimensional model and the background three-dimensional model, wherein

the two or more viewpoint textures include an intra-texture object, and

in the managing step, depth relationship between the foreground three-dimensional model and the intra-texture object is determined, by comparing a parallax of the foreground three-dimensional model occurring in stereoscopic playback with a parallax of the intra-texture object occurring in stereoscopic playback and positional relationship between the foreground three-dimensional model, and the intra-texture object is adjusted based on a result of the determination.

15. An image rendering program for causing a computer to execute processing for realizing stereoscopic viewing of composite images generated by compositing background three-dimensional models and foreground three-dimensional models each defined in three dimensional modeling space, the image rendering program comprising code operable to cause the computer to perform:

a coordinate conversion step of performing a coordinate conversion on modeling data defining a geometry of a background three-dimensional model and a geometry of a foreground three-dimensional model, so that coordinates describing vertices of the modeling data are converted into three-dimensional coordinates describing corresponding points in the three-dimensional modeling space;

a texture mapping step of converting background image data into two or more viewpoint textures and mapping each viewpoint texture to a background three-dimensional model in the three-dimensional modeling space;

a viewport converting step of extracting, for each of the two or more viewpoint textures, a viewport image from the background three-dimensional model mapped with the viewpoint texture and from the foreground three-dimensional model; and

a managing step of managing the foreground three-dimensional model and the background three-dimensional model, wherein

the two or more viewpoint textures include an intra-texture object, and

in the managing step, depth relationship between the foreground three-dimensional model and the intra-texture object is determined, by comparing a parallax of the foreground three-dimensional model occurring in stereoscopic playback with a parallax of the intra-texture object occurring in stereoscopic playback and positional relationship between the foreground three-dimensional model, and the intra-texture object is adjusted based on a result of the determination.