Reprojecting holographic video to enhance streaming bandwidth/quality
First Claim
Patent Images
1. A computer system for compressing video data, comprising:
- one or more processors; and
one or more computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to perform at least the following;
capture successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information;
construct a first 3D representation of a shape of a user'"'"'s environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image;
project the previous frame image onto the first 3D representation;
detect a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image;
construct a second 3D representation of the shape of the user'"'"'s environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user'"'"'s environment as viewed from the second location and orientation (pose) of the camera, and render the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera;
pass the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and
generate compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.
1 Assignment
0 Petitions
Accused Products
Abstract
Improved video compression and video streaming systems and methods are disclosed for environments where camera motion is common, such as cameras incorporated into head-mounted displays. This is accomplished by combining a 3D representation of the shape of the user'"'"'s environment (walls, floor, ceiling, furniture, etc.), image data, and data representative of changes in the location and orientation (pose) of the camera between successive image frames, thereby reducing data bandwidth needed to send streaming video in the presence of camera motion.
38 Citations
20 Claims
-
1. A computer system for compressing video data, comprising:
-
one or more processors; and one or more computer-readable media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to perform at least the following; capture successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information; construct a first 3D representation of a shape of a user'"'"'s environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image; project the previous frame image onto the first 3D representation; detect a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image; construct a second 3D representation of the shape of the user'"'"'s environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user'"'"'s environment as viewed from the second location and orientation (pose) of the camera, and render the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera; pass the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and generate compressed video data comprising only the differences between the re-projected previous frame image and the current frame image. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A video compression method comprising:
-
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information; constructing a first 3D representation of a shape of a user'"'"'s environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image; projecting the previous frame image onto the first 3D representation; detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image; constructing a second 3D representation of the shape of the user'"'"'s environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user'"'"'s environment as viewed from the second location and orientation (pose) of the camera, and rendering the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera; passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image. - View Dependent Claims (11, 12, 13, 14, 15, 16, 17)
-
-
18. A video compression method comprising:
-
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information; rendering the depth of pixels in the previous frame image for a first location into a z-buffer; detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of a second location and orientation (pose) of the camera for the current frame image; performing a screenspace transformation operation, wherein each pixel of the image has color (from the camera) and depth (from the z-buffer), to re-project the previous frame image as viewed from the second location and orientation (pose) of the camera based on the 3D position of each pixel; passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image. - View Dependent Claims (19)
-
-
20. One or more hardware storage devices comprising stored computer-executable instructions that are executable by one or more processors for implementing a method for streaming video from a forward-facing depth camera mounted to the front of a head mounted augmented reality display, wherein the method includes:
-
capturing successive frames of video from a camera incorporated into a first device, the successive frames of video comprising a previous frame image and a current frame image and including surface (depth) information; constructing a first 3D representation of a shape of a user'"'"'s environment based on the surface (depth) information of the previous frame image and a first camera location and orientation (pose) of the camera corresponding to the previous frame image; projecting the previous frame image onto the first 3D representation; detecting a change in the location and orientation (pose) of the camera between the previous frame image and the current frame image and generating current frame camera pose data representative of the location and orientation (pose) of the camera for the current frame image; constructing a second 3D representation of the shape of the user'"'"'s environment based on a second camera location and orientation (pose) corresponding to the current frame image, the second 3D representation corresponding to the shape of the user'"'"'s environment as viewed from the second location and orientation (pose) of the camera, and rending the second 3D representation to generate a re-projected previous frame image as viewed from the second location and orientation (pose) of the camera; passing the re-projected previous frame image to a video compressor for computing differences between the re-projected previous frame image and the current frame image; and generating compressed video data comprising only the differences between the re-projected previous frame image and the current frame image.
-
Specification