EXPERIENCE OR "SENTIO" CODECS, AND METHODS AND SYSTEMS FOR IMPROVING QoE AND ENCODING BASED ON QoE EXPERIENCES

• US 20160219279A1
• Filed: 03/31/2016
• Published: 07/28/2016
• Est. Priority Date: 08/12/2010
• Status: Abandoned Application

• Current Assignee: Wickr Inc. (Amazon.com, Inc.)
• Original Assignee: Net Power & Light, Inc (Amazon.com, Inc.)
• Inventors: Vonog, Stanislav, Surin, Nikolay, Lemmey, Tara
• Application Number:
  US15/087,657
• Publication Number:
  US 20160219279A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics

Abstract

Certain embodiments teach a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoS) engine which allows the sentio codec to select a suitable encoding engine or device. The sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features. As will be appreciated, the following description provides one paradigm for understanding the multi-dimensional experience available to the participants, and as implemented utilizing a sentio codec. There are many suitable ways of describing, characterizing and implementing the sentio codec and experience platform contemplated herein.

Claims

1. A hybrid codec for encoding and decoding a plurality of multi-dimensional data streams for a multi-dimensional experience, the hybrid codec comprising:

a plurality of codecs suitable for encoding and decoding multi-dimensional experience data streams related to a multi-dimensional experience shared over a network between one or more transmitting devices and a receiving device;

a Quality of Experience (QoE) decision engine configured to;

receive an output associated with the multi-dimensional experience, the encoded output including the plurality of multi-dimensional data streams;

wherein the output is divided into a plurality of regions;

analyze the output in each of the plurality of regions; and

for each of the plurality of regions, select one codec from the plurality of codecs to decode the encoded output in that region;

wherein, the selection of the codec is made to improve a human perception of the multi-dimensional experience and is based on data associated with the capabilities of a transmitting device, the capabilities of the receiving device, and the characteristics of the multi-dimensional experience; and

a network engine configured to implement a low-latency transfer protocol for transmitting and receiving of encoded multi-dimensional data streams;

wherein the low latency transfer protocol takes into account the current conditions of the network, the capabilities of the one or more transmitting devices, and the capabilities of the receiving device.

2. The hybrid codec of claim 1, wherein the plurality of codecs includes an audio codec and a video codec.

3. The hybrid codec of claim 2, wherein the plurality of codecs further includes a gesture command codec.

4. The hybrid codec of claim 2, wherein the plurality of codecs further includes a sensor data codec.

5. The hybrid codec of claim 2, wherein the plurality of codecs further includes an emotion data codec.

6. The hybrid codec of claim 1, wherein improving human perception of the multi-dimensional experience includes prioritizing the encoding and decoding of audio streams over the encoding and decoding of video streams.

7. The hybrid codec of claim 1, wherein the capabilities of the transmitting and receiving device include the availability of hardware codecs.

8. The hybrid codec of claim 1, wherein the capabilities of the receiving device include graphical processing capabilities.

9. The hybrid codec of claim 1, wherein the received encoded output associated with the multi-dimensional experience includes a plurality of layers, each of the plurality of layers associated with a subset of the plurality of multi-dimensional data streams.

10. A computer implemented method for providing an experience using a hybrid codec for encoding, decoding, and transmitting experiences, the hybrid codec including a quality of experience (QoE) decision engine, a network engine, and plurality of codecs suitable for encoding and decoding multi-dimensional data streams associated with the experience, the computer implemented method comprising:

receiving, by the QoE decision engine, an output including a plurality of data streams, the plurality of data streams including video, audio, graphics, text, gestures and at least one emotion;

wherein the output is divided into a plurality of regions;

analyzing, by the QoE decision engine, the output in each of the plurality of regions;

for each of the plurality of regions, selecting, by the QoE decision engine, one codec from the plurality of codecs to encode the output within that region;

wherein, the selection of the codec is made to improve a human perception of the multi-dimensional experience and is based on data associated with the capabilities of a transmitting device, the capabilities of the receiving device, and the characteristics of the multi-dimensional experience;

wherein the selection is informed by the network engine, the network engine including a hybrid network stack with network intelligence configured to implement a low-latency transfer protocol; and

for each of the plurality of regions, encoding, by the selected codec, the plurality of data streams associated with the output in that region.

11. The computer implemented method of claim 10, wherein the experience includes a plurality of layers, and the encoding generates the plurality of layers.

12. The computer implemented method of claim 10, wherein the quality of experience engine affects the encoding by taking into consideration the nature and type of devices involved in providing the experience.

13. The computer implemented method of claim 10 further comprising encoding and transmitting virtual goods as part of the experiences.

14. The computer implemented method of claim 10, wherein the received output includes a plurality of layers, each of the plurality of layers associated with a subset of the plurality of multi-dimensional data streams.

15. The computer implemented method of claim 10, wherein improving human perception of the multi-dimensional experience includes prioritizing the encoding and decoding of audio streams over the encoding and decoding of video streams.

16. The computer implemented method of claim 10, further characterized in that a network engine provides instructions on how to encode, the network engine utilizing network information including bandwidth, latency, and jitter.

17. A system comprising:

a plurality of codecs suitable for encoding and decoding multi-dimensional data streams

one or more processors; and

a memory unity having instructions stored thereon, which when executed by the one or more processors, cause the system to;

receiving an output including a plurality of data streams, the plurality of data streams including video, audio, graphics, text, gestures and at least one emotion, wherein the output is divided into a plurality of regions;

analyze the output in each of the plurality of regions;

for each of the plurality of regions, select one codec from the plurality of codecs to encode the output within that region;

wherein, the selection of the codec is made to improve a human perception of the multi-dimensional experience and is based on data associated with the capabilities of a transmitting device, the capabilities of the receiving device, and the characteristics of the multi-dimensional experience;

wherein the selection is informed by a network engine, the network engine including a hybrid network stack with network intelligence configured to implement a low-latency transfer protocol; and

for each of the plurality of regions, cause the selected codec to encode the plurality of data streams associated with the output in that region.