Model-based view extrapolation for interactive virtual reality systems

• US 6,330,281 B1
• Filed: 08/01/2000
• Issued: 12/11/2001
• Est. Priority Date: 08/06/1999
• Status: Expired due to Term

• Current Assignee: Mind Fusion, LLC (Ascend Innovation Management, LLC)
• Original Assignee: Richfx Incorporated
• Inventors: Shemer, Yehuda, Arad, Uri, Mann, Yair, Cohen-or, Daniel, Monsa, Assaf
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): VO, TUNG T
• Application Number:
  US09/629,514
• Time in Patent Office: 497 Days
• Field of Search: 375/240, 375/240.01, 375/240.05, 375/240.09, 375/240.1, 375/240.11, 375/240.12, 375/240.13, 375/240.14, 375/240.16, 375/240.24 , 375/240.25, 348/415.1, 348/416.1, 348/417.1, 348/419.1, 348/420.1, 348/402.1, 345/419, 345/473, 382/238
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A video compression system is provided. An encoder may encode video that is decoded by a decoder. The video may be made up of video frames. The decoder may generate approximated display frames for viewing on a display device based on corrected reference frames. The corrected reference frames at the decoder may be created based on approximated reference frames generated based on model state information and based on difference frames provided by the encoder. The encoder may create the difference frames by subtracting approximated reference frames from exact reference frames. The exact reference frames may be created based on a model of a three-dimensional virtual reality environment. The encoder may create the approximated reference frames based on corrected reference frames that are generated based on the difference reference frames.

Claims

1. A non-client-server system for compressing and decompressing video using a sequence of reference frames, comprising:

an encoder; and

a decoder, wherein;

the encoder and decoder are each configured to generate approximated reference frames from previous corrected reference frames in the sequence using three-dimensional mapping;

the encoder is configured to generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames;

the encoder is configured to provide the difference reference frames to the decoder and is configured to use the difference reference frames in creating the corrected reference frames at the encoder; and

the decoder is configured to receive the difference reference frames from the encoder and is configured to create the corrected reference frames at the decoder based on the difference reference frames and the approximated reference frames.

2. The non-client-server system defined in claim 1 wherein the decoder is further configured to generate approximated display reference frames based on the corrected reference frames.

3. The non-client-server system defined in claim 1 wherein the decoder is further configured to generate approximated display frames based on the corrected reference frames, wherein more approximated display frames are generated than approximated reference frames.

4. The non-client-server system defined in claim 1 wherein the decoder is further configured to generate approximated display frames based on the corrected reference frames, wherein more than ten approximated display frames are generated for each approximated reference frame.

5. The non-client-server system defined in claim 1 wherein the encoder is further configured to compress the difference reference frames before providing the difference reference frames to the decoder.

6. The non-client-server system defined in claim 1 wherein the encoder is further configured to compress the difference reference frames using JPEG compression before providing the difference reference frames to the decoder.

7. The non-client-server system defined in claim 1 wherein the encoder is further configured to compress the difference reference frames using lossy compression before providing the difference reference frames to the decoder.

8. The non-client-server system defined in claim 1, wherein the encoder and decoder are further configured to perform the three-dimensional mapping based on three-dimensional model state information and wherein the model state information involves a camera point-of-view that is used to create range data.

9. The non-client-server system defined in claim 1, wherein the encoder and decoder are further configured to perform the three-dimensional mapping based on three-dimensional model state information, wherein the model state information involves a camera point-of-view that is used to create range data, and wherein the encoder is further configured to stream the difference reference frames to the decoder over a communications network.

10. The non-client-server system defined in claim 1, wherein the encoder is further configured to stream the difference reference frames to the decoder over a communications network.

11. The non-client-server system defined in claim 1, wherein the encoder is further configured to stream the difference reference frames to the decoder over a communications network and wherein the encoder is further configured to encode information based on user inputs.

12. The non-client-server system defined in claim 1, wherein the encoder is further configured to prepare the difference reference frames offline for subsequent use by the decoder.

13. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein the generation of the approximated reference frames at the encoder involves calculating a quality factor for pixels in the approximated reference frames that are being mapped between the corrected reference frames and the approximated reference frames by the three-dimensional mapping.

14. The system defined in claim 13 wherein the quality factor is based on a combination of pixel visibility and local scaling factor.

15. The system defined in claim 13 wherein the difference reference frames include only pixels that have been determined to be of low quality.

16. The system defined in claim 13 wherein the difference reference frames include only pixels that have been determined to be of low quality and wherein the difference reference frames are packed before being provided to the decoder.

17. The system defined in claim 13 wherein pixels in the approximated reference frames have similar depth values and a high quality.

18. The system defined in claim 13 wherein pixels in the approximated reference frames have similar depth values and a high quality, wherein given pixels in the approximated reference frames are generated from pixels in the corrected reference frames by calculating a weighted average RGB value for a group of pixels neighboring each given pixel, and wherein the weighted average uses weights that are based on the inverse of the distance between the neighboring pixels and each given pixel.

19. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein a pool of the corrected reference frames is selected by the encoder and a quality factor is computed for the three-dimensional mapping of pixels in the corrected reference frames to pixels in the approximated reference frame, and wherein pixels are mapped to the approximated reference frame from the corrected reference frames associated with the highest quality pixels.

20. The system defined in claim 19, wherein pixels are mapped to each approximated reference frame from multiple corrected reference frames.

21. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein the three-dimensional mapping is performed using a Z-buffer.

22. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein the encoder is further configured to perform the three-dimensional mapping based on three-dimensional model state information, wherein the model state information involves a camera point-of-view that is used to create range data, and wherein the three-dimensional model state information is simplified before being provided to the decoder.

23. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames;

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames;

stream the difference reference frames to the decoder over a communications network; and

encode information based on user inputs, wherein the decoder uses an extrapolator to extrapolate future user inputs and transmit them to the encoder.

24. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein a real-world camera is used to create the exact reference frames.

25. The system defined in claim 24 wherein the real-world camera includes a range scanner that generates depth information for the exact reference frames.

26. A system for compressing video using a sequence of reference frames, wherein the compressed video is decompressed by a decoder, the system comprising an encoder configured to:

generate approximated reference frames from previous corrected reference frames in the sequence using a three-dimensional mapping;

generate a plurality of difference reference frames by subtracting the approximated reference frames from corresponding exact reference frames; and

provide the difference reference frames to the decoder for use in creating corrected reference frames at the decoder based on the difference reference frames and approximated reference frames, wherein at least some of the exact reference frames are associated with vertices in a directional graph.

27. The system defined in claim 26 wherein the directional graph includes edges and wherein a sequence of difference reference frames is associated with each edge.

28. The system defined in claim 26 wherein the directional graph includes edges and source and target vertices and wherein a sequence of difference reference frames including a first difference reference frame and a last difference reference frame is associated with a given one of the edges so that the source and target vertices are respectively associated with the first and last difference reference frames.

29. The system defined in claim 26 wherein the directional graph includes edges and source and target vertices, wherein a sequence of difference reference frames including a first difference reference frame and a last difference reference frame is associated with a given one of the edges so that the source and target vertices are respectively associated with the first and last difference reference frames, and wherein the encoder prepares the difference reference frames offline.

30. The system defined in claim 26 wherein the directional graph includes edges and source and target vertices, wherein a sequence of difference reference frames including a first difference reference frame and a last difference reference frame is associated with a given one of the edges so that the source and target vertices are respectively associated with the first and last difference reference frames, wherein the encoder prepares the difference reference frames offline, and wherein the decoder is used to select a path through the graph.

31. The system defined in claim 26 wherein the directional graph includes edges and source and target vertices, wherein a sequence of reference frames is associated with each edge and a first one of these reference frames is associated with the source vertex and a last one of these reference frames is associated with the target vertex, wherein the first reference frame is a last corrected reference frame from a sequence of corrected reference frames that share the source vertex with the given edge.

32. The system defined in claim 26 wherein the directional graph includes edges and source and target vertices, wherein a sequence of reference frames is associated with each edge and a first one of these reference frames is associated with the source vertex and a last one of these reference frames is associated with the target vertex, wherein the encoder and decoder perform the three-dimensional mapping based on three-dimensional model state information, wherein the encoder is configured to provide no three-dimensional model state information to the decoder for the source vertex of a given edge, and wherein the decoder uses instead model state information associated with a final reference frame of the sequence of reference frames used for another edge.