Three-dimensional motion mapping for cloud gaming

• US 8,913,664 B2
• Filed: 09/16/2011
• Issued: 12/16/2014
• Est. Priority Date: 09/16/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Lee, Hung-Ju
• Primary Examiner(s): Torrente, Richard
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US13/235,042
• Publication Number:
  US 20130072299A1
• Time in Patent Office: 1,187 Days
• Field of Search: 375/240, 375/240.16
• US Class Current: 375/240.16
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Three-dimensional motion mapping is disclosed. A reverse three-dimensional transform for a set of anchor pixels in the current frame is performed using a model-view matrix, a projection matrix, and view parameters associated with the computer game. Reverse transformed anchor pixels are indicated as occluded if the estimated depth for the pixel in the previous frame is greater than a real depth for the pixel in the previous frame or otherwise indicated as visible. A motion vector map for a subset of the reverse transformed pixels is constructed. A best target motion vector for a first set of sections in the current frame is derived or motion estimation is performed for sections in a second set that contain occluded pixels. Target motion vectors for groups of two or more sections of the current frame comprising two or more adjoining sections each with best target motion vectors are derived.

Claims

1. A method for three-dimensional motion mapping in video encoding, comprising:

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of reverse transformed anchor pixels by mapping two-dimensional coordinates associated with a pixel in the current frame to three-dimensional object coordinates, transforming the three-dimensional object coordinates into three-dimensional object coordinates associated with a corresponding pixel in a previous frame, and mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional coordinates associated with the corresponding pixel in a previous frame;

b) for each reverse transformed anchor pixel, indicating whether the reverse transformed anchor pixel is occluded or visible, whereby the reverse transformed anchor pixel is indicated as occluded if an estimated depth for a corresponding pixel in a previous frame is greater than a real depth for the corresponding pixel in the previous frame, and whereby the reverse transformed anchor pixel is indicated as not occluded and therefore visible if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame;

c) constructing a motion vector map for at least a subset of the set of reverse transformed anchor pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.

2. The method of claim 1, wherein the performing the reverse three-dimensional transform includes:

deriving the estimated depth for the pixel in the previous frame.

3. The method of claim 1, wherein mapping the two-dimensional window coordinates associated with the pixel in the current frame to three-dimensional object coordinates associated with the pixel in the current frame involves using a model-view matrix, a projection matrix, and view parameters associated with the computer game.

4. The method of claim 1, wherein transforming the three-dimensional object coordinates associated with the pixel in the current frame into three-dimensional object coordinates associated with the pixel in the previous frame involves using a model-view matrix, a projection matrix, and view parameters associated with the computer game.

5. The method of claim 1, wherein mapping the three-dimensional object coordinates associated with the pixel in the previous frame to two-dimensional window coordinates associated with the pixel in the previous frame involves using a model-view matrix, a projection matrix, and view parameters associated with the video game.

6. The method of claim 1, wherein performing the three-dimensional transform further comprises updating a model-view matrix, a projection matrix, and view parameters.

7. The method of claim 1, wherein constructing the complete per pixel motion vector map involves determining a motion vector for each pixel in the current frame by calculating the difference between the 2-D window coordinates of the pixel in the current frame and the 2-D window coordinates of the pixel in the previous frame for a visible pixel or performing regular motion estimation for an occluded pixel.

8. The method of claim 1, wherein each section is a 4×

4 array of pixels.

9. The method of claim 1, wherein each group of two or more sections is an 8×

8 array of pixels.

10. The method of claim 1, wherein each group of two or more sections is a 16×

16 array of pixels.

11. The method of claim 1, wherein deriving target motion vectors in e) involves using a pyramid motion vector formation.

12. The method of claim 1, wherein deriving the best target motion vector includes minimizing a cost for encoding a motion vector and a distortion given a target motion vector.

13. The method of claim 1, wherein d) includes deriving the best target motion vector for each section in the current frame with a majority of visible pixels or performing motion estimation for each section of pixels in the current frame with a majority of occluded pixels.

14. A system for implementing three-dimensional motion mapping, comprising:

a processor;

a memory; and

computer-coded instructions embodied in the memory and executable by the processor, wherein the computer coded instructions are configured to implement a method for three-dimensional motion mapping of a current frame of a video game, the method comprising;

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of transformed anchor pixels by mapping two-dimensional coordinates associated with a pixel in the current frame to three-dimensional object coordinates, transforming the three-dimensional object coordinates into three-dimensional object coordinates associated with a corresponding pixel in a previous frame, and mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional coordinates associated with the corresponding pixel in a previous frame;

b) for each reverse transformed anchor pixel, indicating whether the reverse transformed anchor pixel is occluded or visible, whereby the reverse transformed anchor pixel is indicated as occluded if an estimated depth for a corresponding pixel in a previous frame is greater than a real depth for the corresponding pixel in the previous frame, and whereby the reverse transformed anchor pixel is indicated as not occluded and therefore visible if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame;

c) constructing a motion vector map for at least a subset of the set of reverse transformed anchor pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.

15. The method of claim 14, wherein the performing the reverse three-dimensional transform includes:

deriving the estimated depth for the pixel in the previous frame.

16. A computer program product comprising:

a non-transitory, computer-readable storage medium having computer readable program code embodied in said medium for implementing a method for three-dimensional motion mapping of a current video frame in a computer game, the method comprising;

a) performing a reverse three-dimensional transform for a set of anchor pixels in a current frame of video, thereby producing a set of transformed anchor pixels by mapping two-dimensional coordinates associated with a pixel in the current frame to three-dimensional object coordinates, transforming the three-dimensional object coordinates into three-dimensional object coordinates associated with a corresponding pixel in a previous frame, and mapping the three-dimensional object coordinates associated with the corresponding pixel in the previous frame to two-dimensional coordinates associated with the corresponding pixel in a previous frame;

b) for each reverse transformed anchor pixel, indicating whether the reverse transformed anchor pixel is occluded or visible, whereby the reverse transformed anchor pixel is indicated as occluded if an estimated depth for a corresponding pixel in a previous frame is greater than a real depth for the corresponding pixel in the previous frame, and whereby the reverse transformed anchor pixel is indicated as not occluded and therefore visible if the estimated depth for the corresponding pixel in the previous frame is not greater than a real depth for the corresponding pixel in the previous frame;

c) constructing a motion vector map for at least a subset of the set of reverse transformed anchor pixels;

d) deriving a best target motion vector for each section in a first set of sections in the current frame using the motion vector map or performing motion estimation for each section in a second set of sections in the current frame using the motion vector map, wherein the sections in the second set include occluded pixels;

e) deriving target motion vectors for one or more groups of two or more sections of the current frame, each group comprising two or more adjoining sections each with best target motion vectors; and

f) encoding the current frame using the target motion vectors, the best target motion vector for each section, or motion estimation results.

17. The computer program product of claim 16, wherein the performing the reverse three-dimensional transform includes:

deriving the estimated depth for the pixel in the previous frame.