Moving image data generation device, moving image display device, moving image data generation method, moving image displaying method, and data structure of moving image file

• US 9,516,310 B2
• Filed: 07/02/2012
• Issued: 12/06/2016
• Est. Priority Date: 08/01/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Inada, Tetsugo, Ohba, Akio, Segawa, Hiroyuki
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): Slater, Alison
• Application Number:
  US14/234,492
• Publication Number:
  US 20140153634A1
• Time in Patent Office: 1,618 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

Frames of a moving image are configured as a hierarchical structure where each frame is represented with a plurality of resolutions. Some layers are set as original image layers, and the other layers are set as difference image layers in hierarchical data representing a frame at each time step. In the case that an area is to be displayed in the resolution of the difference image layer, to respective pixel values of a difference image of the area, respective pixel values of an image of a corresponding area retained by the original image layer of lower resolution, the image enlarged to the resolution of the difference image layer, are added. A layer to be set as a difference image layer is switched to another layer as time passes.

Claims

1. A moving image data generation device comprising a microprocessor operating under the control of a computer program to generate hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image, the moving image data generation device comprising:

a hierarchical data generating unit configured to generate the hierarchical moving image data by;

(i) generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data, and (ii) generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

a compressed data generating unit configured to compress and encode the hierarchical moving image data generated by the hierarchical data generating unit, and to store the compressed and encoded hierarchical moving image data in a storage device; and

a scheduling unit configured to determine a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

2. The moving image data generation device according to claim 1, wherein the switching schedule for allocating the at least one original image layer and the at least two difference image layers includes changing where, within the hierarchy, at least one of the at least two difference image layers is located as a function of the successive time intervals, wherein the hierarchical data generating unit specifies which of the layers within the hierarchy are set as difference image layers for each image frame in accordance with the switching schedule determined by the scheduling unit, and generates the hierarchical moving image data.

3. The moving image data generation device according to claim 2, wherein the scheduling unit determines whether to retain the difference image data for the at least two difference image layers at each successive time interval, and switches the allocation except a layer with a lowest resolution among the layers of the hierarchy constituting the hierarchical moving image data.

4. The moving image data generation device according to claim 2, wherein the scheduling unit determines whether to retain the difference image data for the at least two difference image layers at each successive time interval, and switches the allocation except a layer with a lowest resolution.

5. The moving image data generation device according to claim 2, wherein the scheduling unit determines whether to retain the difference image data for at least an area in at least one of the layers.

6. The moving image data generation device according to claim 2, wherein the scheduling unit switches the allocation for each of a predetermined number of image frames.

7. The moving image data generation device according to claim 2, wherein the scheduling unit acquires information regarding a timing of a scene change in the moving image, and switches the allocation at the timing of the scene change.

8. The moving image data generation device according to claim 2, wherein the scheduling unit switches the allocation when an accumulated amount of data size of a switching unit per a unit of area reaches a predetermined threshold value.

9. The moving image data generation device according to claim 1, wherein the compressed data generating unit:

groups chronological data on tile images generated by dividing an image on each layer in the hierarchical moving image data generated by the hierarchical data generating unit by a predetermined size, on the basis of whether the data is on a-one of the at least two difference image layers;

generates new chronological data by combining the grouped chronological data for each group of the grouping in accordance with a predetermined rule; and

compresses and encodes the new chronological data in order of the data.

10. The moving image data generation device according to claim 1, wherein the compressed data generating unit compresses and encodes the hierarchical moving image data, which is generated by the hierarchical data generating unit, for respective chronological data on tile images, which are generated by dividing an image on each layer by a predetermined size, regardless of whether the data is on one of the at least two difference image layers.

11. The moving image data generation device according to claim 9, wherein the compressed data generating unit sets at least data on a first tile image after switching to/from a difference image, the data on a first tile image included in chronological data to be compressed, as an intra frame that can be decoded independently.

12. The moving image data generation device according to claim 1,

wherein the compressed data generating unit combines and compresses chronological data on tile images, which are generated by dividing by a predetermined size an image on each layer in the hierarchical moving image data generated by the hierarchical data generating unit, for each same area on images on plurality of layers so as to form a unit of compressed data, and

wherein one unit of compressed data is configured so that the top layer among a plurality of layers included in the unit retains data on an original image required in order to restore a difference image retained by another layer.

13. A moving image display device comprising a microprocessor operating under the control of a computer program, the moving image display device comprising:

a moving image data storage unit configured to store hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image;

an input information acquisition unit configured to acquire a signal requesting to move a display area in the moving image being displayed;

a display image processing unit configured to generate the display image according to the signal requesting to move for each image frame, and in accordance with a required resolution determined by the signal requesting to move acquired by the input information acquisition unit, wherein the hierarchical moving data is produced by;

a hierarchical data generating unit configured to generate the hierarchical moving image data by;

(i) generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data, and (ii) generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

a compressed data generating unit configured to compress and encode the hierarchical moving image data generated by the hierarchical data generating unit, and to store the compressed and encoded hierarchical moving image data in a storage device; and

a scheduling unit configured to determine a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

14. The moving image display device according to claim 13, wherein the switching schedule for allocating the at least one original image layer and the at least two difference image layers includes changing where, within the hierarchy, at least one of the at least two difference image layers is located as a function of the successive time intervals, wherein the hierarchical data generating unit specifies which of the layers within the hierarchy are set as difference image layers for each image frame in accordance with the switching schedule determined by the scheduling unit, and generates the hierarchical moving image data.

15. The moving image display device according to claim 14, wherein the allocation is changed for each of a predetermined number of frames, and

the display image processing unit derives whether data used for displaying is difference image data by specifying a layer having the difference image data for each image frame on the basis of the predetermined number of image frames.

16. The moving image display device according to claim 14,

wherein the moving image data storage unit further stores information regarding the switching schedule while associating the information with the hierarchical moving image data, and

wherein the display image processing unit specifies whether data used for displaying is difference image data by referring to the switching schedule.

17. The moving image display device according to claim 13,

wherein the hierarchical moving image data includes a plurality of units of compressed data, which are compressed and encoded chronological data that is different from chronological order of image frames in a moving image and that is grouped depending on whether the data is data on a difference image,

wherein the moving image data storage unit further stores correspondence information that associates order of image frames in the moving image and order of data in the unit of compressed data with each other while associating the correspondence information with the hierarchical moving image data, and

wherein the display image processing unit rearranges the decoded data in the order of original image frames based on the correspondence information.

18. An apparatus comprising a microprocessor operating under the control of a computer program to implement a moving image data generation method comprising:

generating hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image by;

reading moving image data comprising a series of image frames depicted with one resolution from a storage device;

generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data;

generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

compressing and encoding the hierarchical moving image data and storing the compressed encoded hierarchical moving image data in the storage device; and

determining a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

19. A moving image displaying method, comprising:

reading from a storage device at least a portion of hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image;

displaying the moving image on a display device by using the at least a portion of hierarchical moving data;

acquiring a signal requesting to move a display area in the moving image being displayed;

generating the display image according to the signal requesting to move for each image frame, and in accordance with a required resolution determined by the signal requesting to move; and

displaying the generated display image on the display device, wherein the hierarchical moving data is produced by;

generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data;

generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

compressing and encoding the hierarchical moving image data and storing the compressed encoded hierarchical moving image data in the storage device; and

determining a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

20. A non-transitory, computer-readable recording medium embodying a computer program, which when executed by a computer, causes the computer to carry out actions, comprising:

generating hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image by;

reading moving image data comprising a series of image frames depicted with one resolution from a storage device;

generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data;

generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

compressing and encoding the hierarchical moving image data and storing the compressed encoded hierarchical moving image data in the storage device; and

determining a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

21. A non-transitory, computer-readable recording medium embodying a computer program, which when executed by a computer, causes the computer to carry out actions, comprising:

reading from a storage device at least a portion of hierarchical moving data, including a plurality of series of images, each series of images being a representation of image frames constituting one moving image at a respective differing resolution and at a respective layer within a hierarchy, the hierarchical moving data serving as a basis for generating a display image;

displaying the moving image on a display device by using the at least a portion of hierarchical moving data;

acquiring a signal requesting to move a display area in the moving image being displayed;

generating the display image according to the signal requesting to move for each image frame, and in accordance with a required resolution determined by the signal requesting to move; and

displaying the generated display image on the display device, wherein the hierarchical moving data is produced by;

generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data;

generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

compressing and encoding the hierarchical moving image data and storing the compressed encoded hierarchical moving image data in the storage device; and

determining a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

22. A non-transitory computer-readable recording medium embodying a data structure of a moving image file generated from hierarchical moving data, the hierarchical moving data is produced by:

generating a respective plurality of the image frames at each layer within the hierarchy at a respective one of the resolutions, wherein at least one of the layers is an original image layer in which the plurality of image frames therein represent original image data, and at least two of the layers are difference image layers in which the respective plurality of image frames of each of the difference image layers represent respective difference image data;

generating each image frame of the plurality of image frames of each of the difference image layers by subtracting respective data of two corresponding image frames at two different layers within the hierarchy, such that the display image is produced by enlarging a plurality of image frames containing original image data to a desired size and adding a plurality of the image frames containing difference image data;

compressing and encoding the hierarchical moving image data and storing the compressed encoded hierarchical moving image data in the storage device; and

determining a switching schedule for allocating the at least one original image layer and the at least two difference image layers within the hierarchy at successive time intervals,

wherein at least one of the time intervals includes the at least two difference image layers being allocated to vertically-consecutive layers such that a first of the at least two difference image layers is directly above a second of the at least two difference image layers, and wherein the difference image data of the second of the at least two difference image layers is generated by a subtraction from the difference image data of the first of the at least two difference image layers.

23. The medium according to claim 22,

wherein a layer including the difference image data is switched to another layer in the hierarchical data with passage of time of the moving image, and

wherein the switching schedule switches a layer having the difference image data to another layer, and information used for specifying whether data used for displaying is referred to when displaying data on a difference image.