Image providing device, image providing method, and image providing program for providing past-experience images

• US 9,020,241 B2
• Filed: 02/29/2012
• Issued: 04/28/2015
• Est. Priority Date: 03/03/2011
• Status: Expired due to Fees

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Leichsenring, Germano, Katayama, Tomoko, Oto, Hidetaka, Tominaga, Nobuki
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Kholdebarin, Iman K
• Application Number:
  US13/580,027
• Publication Number:
  US 20130071012A1
• Time in Patent Office: 1,154 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information

Abstract

A image providing device provides a user with realistic and natural past-experience simulation through stereoscopic photographs. Specifically, feature-point extractors extract feature points from a foreground image and a background image, respectively. A stereoscopic matching module searches for pairs of feature points matching between the foreground image and the background image and obtains using the feature point pairs a transformation matrix for projecting the foreground image onto the background image. The transformation by the transformation matrix obtained by the matching unit is applied to foreground depth data, which is depth data of the foreground image. Lastly, depth based rendering is performed based on the transformed foreground depth data to obtain two or more viewpoint images corresponding to the foreground image.

Claims

1. An image providing device for compositing foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing device comprising:

a reader operable to access a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

a transmitter operable to generate, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and to transmit the image acquisition request to an image collection server;

a receiver operable to receive a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

an extractor operable to extract feature points from foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

a matcher operable to search for a pair of feature points matching between the foreground image data and the background image data and obtain a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

a transformer operable to determine the one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background data with an image capturing direction of each photographic image, and to apply, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained by the matcher;

a depth-based renderer operable to perform depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

a compositor operable to composite the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.

2. The image providing device according to claim 1, wherein

a transformation matrix H is given by Formula 1 below,

3. The image providing device according to claim 1, further comprising:

an acquirer operable to acquire photographic image data captured by a stereoscopic camera, the captured photographic image data including the right-view photograph data and the left-view photograph data.

4. The image providing device according to claim 3, further comprising:

a depth image generator operable to generate a depth image of the photographic image data from the right-view photograph data and the left-view photograph data, wherein

the foreground depth data subjected to the transformation by the transformer is the depth image generated by the depth image generator.

5. The image providing device according to claim 3, wherein

one of the right-view photograph data and the left-view photograph data not determined as the base image data is designated as non-base image data,

the depth-based renderer is operable to re-generate image data by applying horizontal shift to pixels of the non-base image data according to depths represented by the foreground depth data, and

the two or more pieces of viewpoint image data are composed of the non-base image data and the re-generated image data.

6. The image providing device according to claim 1, wherein

a permissible level of a depth of feature points to be searched for is set in advance in the image providing device, and

when searching the foreground image data for feature points for matching with a street view viewpoint, the matcher is operable to exclude any feature point having a median of depth distribution falling outside a permissible range, and perform matching between the background image data and the foreground image data using feature points each having a median of depth distribution falling within the permissible range.

7. The image providing device according to claim 1, wherein

the background image is a panoramic image as seen from a geographic location corresponding to the geographic information included in the image acquisition request.

8. An image providing method for compositing foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing method comprising:

accessing a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

generating, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and transmitting the image acquisition request to an image collection server;

receiving a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

extracting feature points from the foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

searching for a pair of feature points matching between the foreground image data and the background image data and obtaining a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

determining one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background image data with an image capturing direction of each photographic image, and applying, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained in the searching;

performing depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

compositing the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.

9. A non-transitory computer-readable medium including an image providing program for causing a computer to execute processing to composite foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing program causing the computer to perform:

accessing a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

generating, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and transmitting the image acquisition request to an image collection server;

receiving a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

extracting feature points from the foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

searching for a pair of feature points matching between the foreground image data and the background image data and obtaining a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

determining one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background image data with an image capturing direction of each photographic image, and applying, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained in the searching;

performing depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

compositing the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.