Mixed reality coding with overlays

• US 10,424,082 B2
• Filed: 04/24/2017
• Issued: 09/24/2019
• Est. Priority Date: 04/24/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Boyce, Jill M., Rowe, Keith W., Holland, James M., Fu, Fangwen, Yedidi, Satya N., Mohan, Sumit
• Primary Examiner(s): Tseng, Charles
• Application Number:
  US15/495,238
• Publication Number:
  US 20180308257A1
• Time in Patent Office: 883 Days
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence

Abstract

A system includes a camera to capture real world content and a semiconductor package apparatus. The semiconductor package apparatus includes a substrate and logic. The logic includes a graphics pipeline to generate rendered content, a base layer encoder to encode real world content into a base layer and a first layer encoder to encode rendered content into a first non-base layer, a multiplexer to interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content, and a transmitter to transmit the single output signal. The system further includes a second layer encoder to encode map data into a second non-base layer. The multiplexer to interleave the second non-base layer with the first non-base layer and the base layer. The first and second layer encoders encode the rendered content and the map data into overlay auxiliary pictures.

Claims

1. A system comprising:

a camera to capture real world content;

a semiconductor package apparatus coupled to the camera, wherein the semiconductor package apparatus includes;

a substrate and logic coupled to the substrate, wherein the logic includes;

a graphics pipeline to generate rendered content,

a base layer encoder to encode the real world content into a base layer and a first layer encoder to encode the rendered content into a first non-base layer, and

a multiplexer to interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content; and

a transmitter to transmit the single output signal,

wherein a second layer encoder is to encode map data into a second non-base layer, and the multiplexer is to interleave the second non-base layer with the first non-base layer and the base layer; and

wherein the first and second layer encoders encode the rendered content and the map data into overlay auxiliary pictures and the map data distinguishes between placement of objects within another overlay auxiliary picture on another layer.

2. The system of claim 1 wherein the real world content includes captured video.

3. The system of claim 1, wherein the base layer and the first non-base layer bypass composition.

4. A system comprising:

a camera to capture real world content;

a semiconductor package apparatus coupled to the camera, wherein the semiconductor package apparatus includes;

a substrate and logic coupled to the substrate, wherein the logic includes;

a graphics pipeline to generate rendered content,

a base layer encoder to encode the real world content into a base layer and a first layer encoder to encode the rendered content into a first non-base layer, and

a multiplexer to interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content; and

a transmitter to transmit the single output signal,

wherein a second layer encoder is to encode map data into a second non-base layer, and the multiplexer is to interleave the second non-base layer with the first non-base layer and the base layer; and

wherein a third layer encoder is to encode blending data into a third non-base layer, and the multiplexer is to interleave the third non-base layer with the second non-base layer, the first non-base layer and the base layer.

5. The system of claim 4, wherein the third layer encoder encodes the blending data into an overlay auxiliary picture and the blending data represents transparency of an overlay object from one or more overlay auxiliary pictures.

6. An apparatus comprising:

a substrate; and

logic coupled to the substrate, wherein the logic is implemented in one or more of configurable logic or fixed-functionality hardware logic, the logic including;

a demultiplexer to receive a single complex signal having multiple interleaved signals and to separate a base layer signal, a first non-base layer signal, a second non-base layer signal, and a third non-base layer signal from the single complex signal;

a base layer decoder to decode the base layer signal into real world content and a first layer decoder to decode the first non-base layer into a mixed reality content overlay;

a second layer decoder to decode the second non-base layer into a map overlay and a third layer decoder to decode the third non-base layer into a blend overlay; and

a compositor to allow a user to select or deselect one or more overlays and to composite the selected overlays with the decoded base layer for display.

7. The apparatus of claim 6, wherein the real world content includes captured video.

8. The apparatus of claim 6, wherein the map overlay distinguishes between placement of objects within an overlay auxiliary picture on another layer.

9. The apparatus of claim 6, wherein the blend overlay represents transparency of an overlay object from one or more overlay auxiliary pictures.

10. A method comprising:

encoding real world content into a base layer;

encoding rendered content into a first non-base layer;

interleaving the base layer with the first non-base layer to obtain a single output signal having mixed reality content; and

encoding map data into a second non-base layer,

wherein the rendered content is encoded into an overlay auxiliary picture and the map data distinguishes between placement of objects within one or more overlay auxiliary pictures on another layer.

11. The method of claim 10, wherein the real world content includes captured video.

12. The method of claim 10, wherein the base layer and the first non-base layer bypass composition.

13. A method comprising:

encoding real world content into a base layer;

encoding rendered content into a first non-base layer;

interleaving the base layer with the first non-base layer to obtain a single output signal having mixed reality content;

encoding map data into a second non-base layer;

interleaving the second non-base layer with the first non-base layer and the base layer;

encoding blending data into a third non-base layer; and

interleaving the third non-base layer with the second non-base layer, the first non-base layer and the base layer.

14. At least one non-transitory computer readable medium comprising a set of instructions, which when executed by a computing system, cause the computing system to:

encode real world content into a base layer;

encode rendered content into a first non-base layer;

interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content; and

encode map data into a second non-base layer,

wherein the rendered content is encoded into an overlay auxiliary picture and the map data distinguishes between placement of objects within one or more overlay auxiliary pictures on another layer.

15. The at least one non-transitory computer readable medium of claim 14, wherein the real world content includes captured video.

16. The at least one non-transitory computer readable medium of claim 14, wherein the base layer and the first non-base layer bypass composition.

17. At least one non-transitory computer readable medium comprising a set of instructions, which when executed by a computing system, cause the computing system to:

encode real world content into a base layer;

encode rendered content into a first non-base layer;

interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content;

encode map data into a second non-base layer;

interleave the second non-base layer with the first non-base layer and the base layer;

encode blending data into a third non-base layer; and

interleave the third non-base layer with the second non-base layer, the first non-base layer and the base layer.