Display apparatus, image processing apparatus and image processing method, imaging apparatus, and program

• US 8,102,395 B2
• Filed: 04/19/2007
• Issued: 01/24/2012
• Est. Priority Date: 10/04/2002
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Okumura, Yuji, Haga, Tsugihiko, Murakami, Tetsuya, Tanaka, Kenji
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): Prendergast, Roberta D
• Application Number:
  US11/737,512
• Publication Number:
  US 20070188493A1
• Time in Patent Office: 1,741 Days
• Field of Search: 345/427, 345/473, 353/94, 434/40, 434/44
• US Class Current: 345/427
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6669 using a plurality of virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character change rooms
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8023 the game being played by multiple players at a common site, e.g. in an arena, theatre, shopping mall using a large public display
• CPC Class Codes: G03B 21/56 Projection screens
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G03B 37/06 involving anamorphosis G03B37/02, G03B37/04 take precedence
• CPC Class Codes: G06F 1/1601 Constructional details related to the housing of computer displays, e.g. of CRT monitors, of flat displays constructional details related to flat displays integrated in a portable computer, e.g. laptop, handheld computer G06F1/1637; constructional details related to television receivers H04N5/64
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G09G 2340/0492 Change of orientation of the displayed image, e.g. upside-down, mirrored
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80

Abstract

A system and method displays a realistic image that allows a user to readily grasp his/her own positional relationship and also to experience a sense of presence. An outer dome screen is disposed so as to surround a user, and an immersion image such as the scenery of the surroundings of an object is displayed thereon. An inner dome screen is disposed inside the outer dome screen, and it displays a bird'"'"'s-eye image of the object as perceived by the vision of the user when the object is viewed from a viewpoint of the user. In this case, the user is allowed to readily grasp his/her own positional relationship by the bird'"'"'s-eye image and is allowed to experience a sense of presence by the immersion image.

Claims

1. An image processing method for processing images captured from a plurality of positions, the image processing method comprising:

detecting, by a viewpoint detecting unit, a plurality of viewpoints of a plurality of users;

determining, by a viewpoint processing unit, whether the plurality of viewpoints are within a predetermined distance of each other;

selecting, by an image selecting mechanism, viewpoint images of a predetermined object as viewed from the plurality of viewpoints of the plurality of users, based on the plurality of detected viewpoints of the plurality of users, from the images captured respectively from the plurality of positions;

integrating, by an integrating mechanism, the plurality of detected viewpoints of the plurality of users into a position of a barycenter of the plurality of detected viewpoints of the plurality of users, the integrating step further integrating the plurality of detected viewpoints of the plurality of users into a single viewpoint, based on the determination in the determining step; and

generating, by a viewpoint-image converting mechanism, a projected image based on the selected viewpoint images, the plurality of detected viewpoints of the plurality of users, the determination in the determining step, the integration in the integrating step, and a predetermined three dimensional shape of an interior display surface, the projected image being projected onto the interior display surface by a projector and viewed on an outer surface of the interior display surface.

2. The image processing method according to claim 1, wherein:

the selecting step comprises selecting, by the image selecting mechanism, the viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users;

the generating step comprises converting, by the viewpoint-image converting mechanism, the selected viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users to generate the projected image; and

the image processing method further comprises displaying, by the projector, the projected image on the interior display surface so as to display images of the predetermined object as viewed respectively from the plurality of viewpoints of the plurality of users.

3. The image processing method according to claim 1, wherein

the selecting step comprises selecting, by the image selecting mechanism, the viewpoint images respectively associated with the plurality of users'"'"' detected viewpoints from the images respectively captured by a plurality of imaging devices, and generating a viewpoint image for the single viewpoint from the selected viewpoint images, and

the generating the projected image step comprises converting, by the viewpoint-image converting mechanism, the generated viewpoint image for the single viewpoint into the projected image.

4. An image processing apparatus for processing images captured from a plurality of positions , the image processing apparatus comprising:

a viewpoint detecting unit configured to detect a plurality of viewpoints of a plurality of users;

a viewpoint processing unit configured to determine whether the plurality of viewpoints are within a predetermined distance of each other;

an image selecting mechanism configured to select a viewpoint image of a predetermined object, as viewed from the viewpoints of the plurality of users, based on the plurality of detected viewpoints of the plurality of users, from the images captured respectively from the plurality of positions;

an integrating mechanism configured to integrate the plurality of detected viewpoints of the plurality of users into a position of a barycenter of the plurality of detected viewpoints of the plurality of users, and configured to integrate the plurality of detected viewpoints of the plurality of users into a single viewpoint, based on the determination by the viewpoint processing unit; and

a viewpoint-image converting mechanism configured to Generate a projected image based on the selected viewpoint images, the plurality of detected viewpoints of the plurality Of users, the determination by the viewpoint processing unit, the integration by the integrating mechanism, and a predetermined three dimensional shape of an interior display surface, the projected image being projected onto the interior display surface by a projector and viewed on an outer surface of the interior display surface.

5. The image processing apparatus according to claim 4, further comprising a plurality of imaging devices.

6. The image processing apparatus according to claim 4, wherein:

the image selecting mechanism is configured to select the viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users;

the viewpoint-image converting mechanism is configured to convert the selected viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users to generate the projected image; and

the image processing apparatus further comprises the projector configured to display the projected image on the interior display surface so as to display images of the predetermined object as viewed respectively from the plurality of viewpoints of the plurality of users.

7. The image processing apparatus according to claim 4, wherein

the image selecting mechanism is configured to select viewpoint images respectively associated with the plurality of users'"'"' detected viewpoints from the images respectively captured by the plurality of imaging devices, and to generate a viewpoint image for the single viewpoint from the selected viewpoint images, and

the viewpoint-image converting mechanism is configured to convert the generated viewpoint image for the single viewpoint into the projected image.

8. The image processing apparatus according to claim 7,

wherein the image selecting mechanism is configured to calculate a weighted sum of the viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users in accordance with distances of the respective detected viewpoints of the plurality of users and the single viewpoint so as to generate the viewpoint image for the single viewpoint.

9. The image processing apparatus according to claim 7, further comprising:

a viewpoint-distance calculator configured to calculate distances among the plurality of detected viewpoints of plurality of users,

wherein the integrating mechanism is configured to integrate a plurality of detected viewpoints of the plurality of users having viewpoint distances not greater than the predetermined distance into the single viewpoint.

10. A non-transitory computer-readable storage medium having embedded therein instructions, which when executed by a computer, cause the computer to perform a method of image processing for processing images captured from a plurality of positions, the method comprising:

detecting a plurality of viewpoints of a plurality of users;

determining whether the plurality of viewpoints are within a predetermined distance of each other;

selecting viewpoint images of a predetermined object as viewed from the plurality of viewpoints of the plurality of users, based on the plurality of detected viewpoints of the plurality of users, from the images captured respectively from the plurality of positions;

integrating the plurality of detected viewpoints of the plurality of users into a position of a barycenter of the plurality of detected viewpoints of the plurality of users, the integrating step further integrating the plurality of detected viewpoints of the plurality of users into a single viewpoint, based on the determination in the determining step; and

generating a projected image based on the selected viewpoint images, the plurality of detected viewpoints of the plurality of users, the determination in the determining step, the integration in the integrating step, and a predetermined three dimensional shape of an interior display surface, the projected image being projected onto the interior display surface by a projector and viewed on an outer surface of the interior display surface.

11. The non-transitory computer-readable storage medium according to claim 10, wherein:

the selecting step comprises selecting the viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users;

the generating step comprises converting the selected viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users to generate the projected image; and

the image processing method further comprises displaying, by the projector, the projected image on the interior display surface so as to display images of the predetermined object as viewed respectively from the plurality of viewpoints of the plurality of users.

12. The non-transitory computer-readable storage medium according to claim 10, wherein

the selecting step comprises selecting the viewpoint images respectively associated with the plurality of users'"'"' detected viewpoints from the images respectively captured by a plurality of imaging devices, and generating a viewpoint image for the single viewpoint from the selected viewpoint images, and

the generating the projected image step comprises converting, by the viewpoint-image converting mechanism, the generated viewpoint image for the single viewpoint into the projected image.

13. An imaging apparatus for capturing an image, comprising:

an omnidirectional camera configured to obtain omnidirectional-image data including an image of surroundings of an object in all directions;

an object-imaging unit, including a plurality of cameras surrounding the object in horizontal and vertical directions, configured to obtain surrounding-image data including an image of the object from the plurality of positions corresponding to respective different polar and azimuth angles of a spherical surface in a spherical coordinate system; and

a multiplexer configured to receive the omnidirectional-image data and the surrounding-image data, and to multiplex the surrounding-image data and the omnidirectional-image data in association with each other, wherein

the image of the surrounding obtained by the omnidirectional imaging unit is displayed as an immersion image for presenting a plurality of users with a sense of immersion, and

the image of the object obtained by the object-imaging unit is displayed on an interior display surface as at least one bird'"'"'s-eye image of the object as viewed from a plurality of detected viewpoints of a plurality of users based on a determination of whether the plurality of viewpoints are within a predetermined distance of each other, an integration of the plurality of detected viewpoints of the plurality of users into a position of a barycenter of the plurality of detected viewpoints of the plurality of users, and an integration of the plurality of detected viewpoints of the plurality of users into a single viewpoint, based on the determination of whether the plurality of viewpoints are within the predetermined distance of each other.

14. The imaging apparatus according to claim 13, wherein

the plurality of cameras included in the object-imaging unit are disposed along a hemisphere of a spherical plane surrounding the object.

15. An image processing apparatus for processing images captured from a plurality of positions, the image processing apparatus comprising:

a viewpoint detecting unit configured to detect a plurality of viewpoints of a plurality of users;

a viewpoint processing unit configured to determine whether the plurality of viewpoints are within a predetermined distance of each other;

an integrating mechanism configured to integrate the plurality of detected viewpoints of the plurality of users into a single viewpoint, based on the determination by the viewpoint processing unit;

an image selecting mechanism configured to select viewpoint images of a predetermined object, as viewed from the plurality of viewpoints of the plurality of users, based on the plurality of detected viewpoints of the plurality of users, from the images captured respectively from the plurality of positions, and to calculate a weighted sum of the viewpoint images respectively associated with the plurality of detected viewpoints of the plurality of users in accordance with distances of the respective detected viewpoints of the plurality of users and the single viewpoints to generate a viewpoint image for the single viewpoint; and

a viewpoint-image converting mechanism configured to generate a projected image based on the selected viewpoint images, the plurality of detected viewpoints of the plurality of users, the determination by the viewpoint processing unit, the viewpoint image for the single viewpoint, and a predetermined three dimensional shape of an interior display surface, the projected image being projected onto the interior display surface by a projector and viewed on an outer surface of the interior display surface.