THREE-DIMENSIONAL MOTION MAPPING FOR CLOUD GAMING

• US 20130072299A1
• Filed: 09/16/2011
• Published: 03/21/2013
• Est. Priority Date: 09/16/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Lee, Hung-Ju
• Granted Patent: US 8,913,664 B2
• Time in Patent Office: Days
• US Class Current: 463/32
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Three-dimensional motion mapping is disclosed. A reverse three-dimensional transform for a set of anchor pixels in the current frame is performed using a model-view matrix, a projection matrix, and view parameters associated with the computer game. Reverse transformed anchor pixels are indicated as occluded if the estimated depth for the pixel in the previous frame is greater than a real depth for the pixel in the previous frame or otherwise indicated as visible. A motion vector map for a subset of the reverse transformed pixels is constructed. A best target motion vector for a first set of sections in the current frame is derived or motion estimation is performed for sections in a second set that contain occluded pixels. Target motion vectors for groups of two or more sections of the current frame comprising two or more adjoining sections each with best target motion vectors are derived.

Claims

1. A method for three-dimensional motion mapping in video encoding, comprising:

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of transformed anchor pixels;

b) for each reverse transformed pixel, indicating that the reverse transformed pixel is occluded if the estimated depth for a corresponding pixel in the previous frame is greater than a real depth for the corresponding pixel in the previous frame, or indicating that the corresponding pixel in the current frame is visible, or indicating that the reverse transformed pixel is not occluded if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame.

c) constructing a motion vector map for at least a subset of the set of reverse transformed pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.

2. The method of claim 1, wherein performing the three-dimensional transform includes:

i) mapping two-dimensional window coordinates associated with the pixel in the current frame to three-dimensional object coordinates associated with the pixel in the current frame;

ii) determining a real depth for the pixel in the current frame;

iii) transforming the three-dimensional object coordinates associated with the pixel in the current frame into three-dimensional object coordinates associated with a corresponding pixel in a previous frame;

iv) deriving an estimated depth for the pixel in the previous frame; and

v) mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional window coordinates associated with the corresponding pixel in the previous frame.

3. The method of claim 2, wherein mapping the two-dimensional window coordinates associated with the pixel in the current frame to three-dimensional object coordinates associated with the pixel in the current frame involves using a model-view matrix, a projection matrix, and view parameters associated with the computer game.

4. The method of claim 2, wherein transforming the three-dimensional object coordinates associated with the pixel in the current frame into three-dimensional object coordinates associated with the pixel in the previous frame involves using a model-view matrix, a projection matrix, and view parameters associated with the computer game.

5. The method of claim 2, wherein mapping the three-dimensional object coordinates associated with the pixel in the previous frame to two-dimensional window coordinates associated with the pixel in the previous frame involves using a model-view matrix, a projection matrix, and view parameters associated with the video game.

6. The method of claim 2, wherein performing the three-dimensional transform further comprises updating a model-view matrix, a projection matrix, and view parameters.

7. The method of claim 1, wherein constructing the complete per pixel motion vector map involves determining a motion vector for each pixel in the current frame by calculating the difference between the 2-D window coordinates of the pixel in the current frame and the 2-D window coordinates of the pixel in the previous frame for a visible pixel or performing regular motion estimation for an occluded pixel.

8. The method of claim 1, wherein each section is a 4×

4 array of pixels.

9. The method of claim 1, wherein each group of two or more sections is an 8×

8 array of pixels.

10. The method of claim 1, wherein each group of two or more sections is a 16×

16 array of pixels.

11. The method of claim 1, wherein deriving target motion vectors in e) involves using a pyramid motion vector formation.

12. The method of claim 1, wherein deriving the best target motion vector includes minimizing a cost for encoding a motion vector and a distortion given a target motion vector.

13. The method of claim 1, wherein d) includes deriving the best target motion vector for each section in the current frame with a majority of visible pixels or performing motion estimation for each section of pixels in the current frame with a majority of occluded pixels;

15. The system of claim 13, wherein performing the three-dimensional transform includes:

i) mapping two-dimensional window coordinates associated with the pixel in the current frame to three-dimensional object coordinates associated with the pixel in the current frame;

ii) determining a real depth for the pixel in the current frame;

iii) transforming the three-dimensional object coordinates associated with the pixel in the current frame into three-dimensional object coordinates associated with a corresponding pixel in a previous frame;

iv) deriving an estimated depth for the pixel in the previous frame; and

v) mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional window coordinates associated with the corresponding pixel in the previous frame.

17. The computer program product of claim 15, wherein performing the three-dimensional transform includes:

i) mapping two-dimensional window coordinates associated with the pixel in the current frame to three-dimensional object coordinates associated with the pixel in the current frame;

ii) determining a real depth for the pixel in the current frame;

iii) transforming the three-dimensional object coordinates associated with the pixel in the current frame into three-dimensional object coordinates associated with a corresponding pixel in a previous frame;

iv) deriving an estimated depth for the pixel in the previous frame; and

v) mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional window coordinates associated with the corresponding pixel in the previous frame.

14. A system for implementing three-dimensional motion mapping, comprising:

a processor;

a memory; and

computer-coded instructions embodied in the memory and executable by the processor, wherein the computer coded instructions are configured to implement a method for three-dimensional motion mapping of a current frame of a video game, the method comprising;

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of transformed anchor pixels;

b) for each reverse transformed pixel, indicating that the reverse transformed pixel is occluded if the estimated depth for a corresponding pixel in the previous frame is greater than a real depth for the corresponding pixel in the previous frame, or indicating that the corresponding pixel in the current frame is visible, or indicating that the reverse transformed pixel is not occluded if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame.

c) constructing a motion vector map for at least a subset of the set of reverse transformed pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.

16. A computer program product comprising:

a non-transitory, computer-readable storage medium having computer readable program code embodied in said medium for implementing a method for three-dimensional motion mapping of a current video frame in a computer game, the method comprising;

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of transformed anchor pixels;

b) for each reverse transformed pixel, indicating that the reverse transformed pixel is occluded if the estimated depth for a corresponding pixel in the previous frame is greater than a real depth for the corresponding pixel in the previous frame, or indicating that the corresponding pixel in the current frame is visible, or indicating that the reverse transformed pixel is not occluded if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame.

c) constructing a motion vector map for at least a subset of the set of reverse transformed pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.