Information processing apparatus, information processing method, and data structure of content files

• US 9,047,680 B2
• Filed: 06/08/2010
• Issued: 06/02/2015
• Est. Priority Date: 09/11/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.), Sony Corporation (Sony Group Corp.)
• Inventors: Kawai, Aritoki, Nozaki, Takashi
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US13/391,160
• Publication Number:
  US 20120206491A1
• Time in Patent Office: 1,820 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting

Abstract

As a user inputs instruction to start up content, an initial image is displayed. A viewpoint shift request is made, viewpoint coordinates are calculated according to the viewpoint shift request. Then verified is whether a hierarchical level has been changed in hierarchical image data due to a shift of the viewpoint coordinates or not. If the hierarchical level has been changed, the node of a hierarchical image is extracted from image correspondence data and then a coordinate transform matrix associated with the node is modified. A new elemental image is decoded, each image is subjected to the coordinate transform based on the image correspondence data so as to evaluate the coordinates on display coordinates. Each elemental image is placed on the evaluated coordinates, which in turn integrates the elemental images and updates a displayed imaged.

Claims

1. An information processing apparatus for processing content and displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from the single elemental image are hierarchized according to resolution, the information processing apparatus comprising:

a viewpoint coordinate acquiring unit configured to receive a request for a shift of a viewpoint for the displayed image and configured to acquire viewpoint coordinates of an image to be displayed next based at least in part on a hierarchical coordinate system comprising levels of hierarchical image data representing different resolutions, wherein image data at a particular hierarchical level is decoded according to the position of the viewpoint;

an elemental image rendering unit configured to render the elemental image; and

a coordinate conversion unit configured to convert a hierarchical coordinate system unique to each elemental image into a display coordinate system, based on image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image, and configured to calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next; and

an elemental image integrating unit configured to generate a displayed image where the elemental image rendered by the elemental image rendering unit is arranged in the respective positional coordinates.

2. An information processing apparatus according to claim 1, wherein the relative position of the image correspondence data is expressed by a coordinate transform matrix used to convert the coordinate system of the elemental image to the coordinate system of the another elemental image, and

wherein when a hierarchical level of image data used to display the hierarchical image is switched by a shift of the viewpoint, the coordinate conversion unit modifies the coordinate transform matrix, used to convert the coordinate system of the hierarchical image to the coordinate system of the another elemental image in the image correspondence data and then calculates the respective positional coordinates in the display coordinate system of each elemental image.

3. An information processing apparatus according to claim 1, wherein

for the image correspondence image, the layout of the elemental image linked to the another elemental image as a result of the shift of the viewpoint is expressed by the relative position in relation to the another elemental image, and

the relative position in relation to a display is set to an elemental image not linked to the another elemental image.

4. An information processing apparatus according to claim 1, wherein

the elemental image contains a conditional elemental image of which relative position is defined, in the image correspondence data, in relation to an image in a specified hierarchical level which constitutes a part of the hierarchical image data, and

wherein the elemental image integrating unit places the conditional elemental image within the displayed image, only when the hierarchical image is to be displayed using data on the hierarchical level specified for the conditional elemental image.

5. An information processing apparatus according to claim 1, wherein

the image correspondence data is such that a display is set as a root node and each elemental image is set to a node other than the root node, and

the image correspondence data has a tree structure where elemental images expressing relative positions are associated with each other.

6. An information processing apparatus for supporting the preparation of content displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image are hierarchized in the order of resolution, the information processing apparatus comprising:

an input information acquiring unit configured to receive, from a user, elemental image data on the elemental images, a layout of each elemental image, and a designation of an elemental image that moves in conjunction with another elemental image as a result of a shift of a viewpoint when displayed;

a coordinate conversion unit configured to convert a hierarchical coordinate system unique to each elemental image into a display coordinate system, based on image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image, and calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next;

an image correspondence data generator configured to generate image correspondence data where the elemental image data is associated with relative positional information expressing the layout of each elemental image through a relative position of another elemental image that is linked to each elemental image; and

a content file generator configured to generate a content file data where the elemental image data and the image correspondence data are associated with each other.

7. An information processing apparatus according to claim 6, wherein,

in the image correspondence data, the image correspondence data generator associates the relative positional information with relation to a display with an elemental image not specified to move in conjunction with the another elemental image.

8. An information processing apparatus according to claim 6, wherein

the input information acquiring unit further receives from the user a designation of a temporal change in the display region and in the enlargement factor of an image to be realized at the time the content is processed,

the information processing apparatus further comprising a scenario data generator that is configured to calculate positional coordinates in a display coordinate system of an elemental image to be displayed, based on the designation of the temporal change and the relative positional information in the image correspondence data, and that is configured to generate scenario data including the display order and the positional coordinates of the elemental image.

9. An information processing apparatus according to claim 6, wherein

the input information acquiring unit further receives from the user a designation of a device that is to process the content,

the information processing apparatus further comprising a device-dependent data generator configured to generate device-dependent data indicating an optimum value of an internal parameter concerning an image display process, which is used when the content is processed by a device specified by the user, by referencing a pre-prepared table that associates identification information on a device that processes the content with the optimum value of the internal parameter.

10. An information processing apparatus according to claim 6, wherein

the input information acquiring unit further receives from the user a designation to replacing the elemental image with another elemental image depending on a device that processes the content, and

wherein the image correspondence generator generates image correspondence data for each device that processes the content.

11. An information processing method for processing a content displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image are hierarchized in the order of resolution, the method comprising:

receiving a request for a shift of a viewpoint for the displayed image and acquiring viewpoint coordinates of an image to be displayed next based at least in part on a hierarchical coordinate system comprising levels of hierarchical image data representing different resolutions, wherein image data at a particular hierarchical level is decoded according to the position of the viewpoint;

reading, from a memory, data of the elemental images and rendering the elemental image; and

reading, from the memory, image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image and converting a coordinate system unique to each elemental image into a display coordinate system, based on the image correspondence data, and calculating positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next; and

generating a displayed image where the elemental image rendered is arranged in the positional coordinates, and displaying the displayed image on a display apparatus.

12. An information processing method for supporting the preparation of content displaying an image comprised of elemental images of the displayed image, the elemental images including a hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image is are hierarchized in the order of resolution, the method comprising:

receiving, from a user, elemental image data on the elemental images, a layout of each elemental image, and a designation of an elemental image that moves in conjunction with another elemental image as a result of a shift of a viewpoint when displayed;

generating image correspondence data where the elemental image data is associated with relative positional information expressing the layout of each elemental image through a relative position of another elemental image linked to each elemental image, and storing the image correspondence data in a memory;

converting a hierarchical coordinate system unique to each elemental image into a display coordinate system, based on image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image;

calculating positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next; and

generating content file data where the elemental image data and the image correspondence data are associated with each other, and storing the content file in the memory.

13. A program, embedded in a non-transitory computer-readable medium, for processing content displaying an image comprised of elemental images, the elemental images including a hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image is hierarchized in the order of resolution, the program comprising:

a viewpoint coordinate acquiring module operative to receive a request for a shift of a viewpoint for the displayed image and operative to acquire viewpoint coordinates of an image to be displayed next;

an elemental image rendering module operative to read, from a memory, data of the elemental images and operative to render the elemental image; and

a coordinate conversion module operative to read, from the memory, image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image and converting a coordinate system unique to each elemental image into a display coordinate system, based on the image correspondence data, and operative to calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next based at least in part on a hierarchical coordinate system comprising levels of hierarchical image data representing different resolutions, wherein image data at a particular hierarchical level is decoded according to the position of the viewpoint; and

an elemental image integrating module operative to generate a displayed image where the elemental image rendered is arranged in the respective positional coordinates, and operative to display the displayed image on a display apparatus.

14. A program, embedded in a non-transitory computer-readable medium, for supporting the preparation of content displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image are hierarchized in the order of resolution, the program comprising:

an input information acquiring module operative to receive, from a user, elemental image data on the elemental images, a layout of each elemental image, and a designation of an elemental image that moves in conjunction with another elemental image as a result of a shift of a viewpoint when displayed;

an image correspondence data generating module operative to generate image correspondence data where the elemental image data is associated with relative positional information expressing the layout of each elemental image by a relative position of another elemental image linked to each elemental image, and operative to store the image correspondence data in a memory;

a coordinate conversion module configured to convert a hierarchical coordinate system unique to each elemental image into a display coordinate system, based on image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image, and calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next; and

a content file generating module operative to generate content file data where the elemental image data and the image correspondence data are associated with each other, and operative to store the content file in the memory.

15. A non-transitory computer-readable medium encoded with a program, executable by a computer, for processing content displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image are hierarchized in the order of resolution, the medium comprising:

a viewpoint coordinate acquiring module operative to receive a request for a shift of a viewpoint for the displayed image and operative to acquire viewpoint coordinates of an image to be displayed next;

an elemental image rendering module operative to read, from a memory, data of the elemental images and operative to render the elemental image; and

a coordinate conversion module operative to read, from the memory, image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image and operative to convert a coordinate system unique to each elemental image into a display coordinate system, based on the image correspondence data, and operative to calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next based at least in part on a hierarchical coordinate system comprising levels of hierarchical image data representing different resolutions, wherein image data at a particular hierarchical level is decoded according to the position of the viewpoint; and

an elemental image integrating module operative to generate a displayed image where the elemental image rendered is arranged in the positional coordinates, and operative to display the displayed image on a display apparatus.

16. A non-transitory computer-readable medium encoded with a program, executable by a computer, for supporting the preparation of content displaying an image comprised of elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image is are hierarchized in the order of resolution, the medium comprising:

an input information acquiring module operative to receive, from a user, elemental image data on the elemental images, a layout of each elemental image, and a designation of an elemental image that moves in conjunction with another elemental image as a result of a shift of a viewpoint when displayed;

an image correspondence data generating module operative to generate image correspondence data where the elemental image data is associated with relative positional information expressing the layout of each elemental image by a relative position of another elemental image linked to each elemental image, and operative to store the image correspondence data in a memory;

a coordinate conversion module configured to convert a hierarchical coordinate system unique to each elemental image into a display coordinate system, based on image correspondence data where a layout of the elemental image is expressed by a relative position in relation to another elemental image, and calculate positional coordinates in the display coordinate system of each elemental image corresponding to the viewpoint coordinates of the image to be displayed next; and

a content file generating module operative to generate content file data where the elemental image data and the image correspondence data are associated with each other, and operative to store the content file in the memory.

17. A non-transitory computer readable medium having stored thereon a data structure of a content for displaying an image comprising:

elemental images, the elemental images including hierarchical images having a data structure where pieces of image data with different resolutions generated from a single image are hierarchized in the order of resolution,

wherein the structure of the content file associates data on the elemental images with image correspondence data where a layout of each elemental image at the time of displaying is expressed by a coordinate transform matrix by which a coordinate system unique to the elemental image is converted into that which is unique to another elemental image and positional coordinates in the coordinate system of each elemental image are calculated which correspond to the viewpoint coordinates of the image to be displayed next.