Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images

• US 8,907,968 B2
• Filed: 03/02/2012
• Issued: 12/09/2014
• Est. Priority Date: 03/31/2011
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Tanaka, Keiichi, Katayama, Tomoko, Yamaji, Osamu
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US13/635,820
• Publication Number:
  US 20130044108A1
• Time in Patent Office: 1,012 Days
• Field of Search: 345/419, 345/420, 345/422, 345/582, 345/585
• US Class Current: 345/585
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation

Abstract

An image rendering device realizes stereoscopic viewing of a joint background image. A DIBR unit 17 applies pixel shifting to each background image constructing the joint background image and also conducts boundary processing relative to the pixel shifting. The joint background image is composed of one or more background images stitched together along their edges on a three-dimensional model. The pixel shifting is to shift pixels of each background image in the row direction. The amount of shift applied to each pixel is determined based on the depth value of a corresponding pixel of a depth image. The boundary processing is to extract pixels shifted out of the display region of a given background image as a result of pixel shifting and add the extracted pixels to an edge of another background image adjacent to the given background image in the row direction.

Claims

1. An image rendering device for realizing stereoscopic viewing of joint image data,

the joint image data being constructed of a plurality of pieces of background image data that are stitched together in a row direction,

the image rendering device comprising:

a pixel shifter that applies pixel shifting to each piece of background image data and conducts boundary processing relative to the pixel shifting, thereby to acquire two or more viewpoint textures for the joint image data;

a mapper that places the two or more viewpoint textures onto inner surfaces of three-dimensional models in a three-dimensional modeling space;

a viewport converter that extracts, for each of two or more viewpoints, a viewport image from the textured three-dimensional models, wherein

the pixel shifting is to shift coordinates of respective pixels in each piece of background image data in the row direction, and the amount of shift applied to each pixel is determined based on a depth of a corresponding pixel in depth image data associated with the joint image data, and

the boundary processing is to extract out-of-region pixels which are pixels to be located outside a display region of a given one of the pieces of background image data as a result of the pixel shifting, and write the extracted out-of-region pixels at an edge of another piece of background image data that is adjacent to the given piece of background image data in the row direction.

2. The image rendering device according to claim 1, further comprising:

a display region buffer;

an out-of-region buffer; and

a texture memory that stores a piece of background image data already processed by the pixel shifting, wherein

of the plurality of pieces of background image data constructing the joint image data, a piece of background image data currently subjected to the pixel shifting is designated as a current piece of background image data,

the pixel shifting is carried out by;

storing the current piece of background image data in the display region buffer;

shifting, on the display region buffer, coordinates of respective pixels in the current piece of background image data in the row direction; and

extracting out-of-region pixels shifted to outside a display region of the current piece of background image data and storing the extracted out-of-region pixels in the out-of-region buffer, and

the boundary processing is carried out by;

reading the out-of-region pixels from the out-of-region buffer; and

writing the read out-of-region pixels at the edge of the piece of already processed background image data stored in the texture memory.

3. The image rendering device according to claim 1, wherein

the joint image data is joint Mercator image data obtained by stitching together one edge and another edge of a single piece of Mercator image data, and

the boundary processing is carried out by:

extracting out-of-region pixels which are pixels to be located outside a display region of the Mercator image data as a result of the pixel shifting applied to respective pixel in the joint Mercator image data based on the depth image data; and

writing the extracted out-of-region pixels into the joint Mercator image at the opposite edge from the edge where the out-of-region pixels are extracted.

4. The image rendering device according to claim 1, further comprising:

a position and direction determiner that determines, in accordance with a user operation, a current viewpoint location on a map and a current line-of-sight direction; and

an acquisitor that generates an image acquisition request using geographic information corresponding to the current viewpoint location on the map and transmits the image acquisition request to an image collection server, thereby to download a street view file, wherein

the street view file downloaded from the image collection server has a location attribute matching the geographic information included in the image acquisition request, and

the joint image data is contained in the downloaded street view file.

5. The image rendering device according to claim 1, wherein

the joint image data is a panoramic image and constructed of the plurality of pieces of background image data stitched together in the row direction to represent a view of surroundings as seen from a geographic location corresponding to the geographic information included in the image acquisition information.

6. An image rendering method for realizing stereoscopic viewing of joint image data,

the joint image data being constructed of a plurality of pieces of background image data that are stitched together in a row direction,

the image rendering method comprising:

applying pixel shifting to each piece of background image data and conducting boundary processing relative to the pixel shifting, thereby to acquire two or more viewpoint textures for the joint image data;

placing the two or more viewpoint textures onto inner surfaces of three-dimensional models in a three-dimensional modeling space;

extracting, for each of two or more viewpoints, a viewport image from the textured three-dimensional models, wherein

the pixel shifting is to shift coordinates of respective pixels in each piece of background image data in the row direction, and the amount of shift applied to each pixel is determined based on a depth of a corresponding pixel in depth image data associated with the joint image data, and

the boundary processing is to extract out-of-region pixels which are pixels to be located outside a display region of a given one of the pieces of background image data as a result of the pixel shifting, and write the extracted out-of-region pixels at an edge of another piece of background image data that is adjacent to the given piece of background image data in the row direction.

7. A non-transitory recording medium recording an image rendering program for causing a computer to realize stereoscopic viewing of joint image data,

the joint image data being constructed of a plurality of pieces of background image data that are stitched together in a row direction,

the image rendering program comprising code that causes the computer to perform:

applying pixel shifting to each piece of background image data and conducting boundary processing relative to the pixel shifting, thereby to acquire two or more viewpoint textures for the joint image data;

placing the two or more viewpoint textures onto inner surfaces of three-dimensional models in a three-dimensional modeling space;

extracting, for each of two or more viewpoints, a viewport image from the textured three-dimensional models, wherein

the pixel shifting is to shift coordinates of respective pixels in each piece of background image data in the row direction, and the amount of shift applied to each pixel is determined based on a depth of a corresponding pixel in depth image data associated with the joint image data, and

the boundary processing is to extract out-of-region pixels which are pixels to be located outside a display region of a given one of the pieces of background image data as a result of the pixel shifting, and write the extracted out-of-region pixels at an edge of another piece of background image data that is adjacent to the given piece of background image data in the row direction.