Experience or “sentio” codecs, and methods and systems for improving QoE and encoding based on QoE experiences

• US 9,172,979 B2
• Filed: 08/12/2011
• Issued: 10/27/2015
• Est. Priority Date: 08/12/2010
• Status: Expired due to Fees

• Current Assignee: Wickr Inc. (Amazon.com, Inc.)
• Original Assignee: Net Power & Light, Inc (Amazon.com, Inc.)
• Inventors: Vonog, Stanislav, Surin, Nikolay, Lemmey, Tara
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): BRUMFIELD, SHANIKA M
• Application Number:
  US13/136,870
• Publication Number:
  US 20120039382A1
• Time in Patent Office: 1,537 Days
• Field of Search: 348/474, 348/578, 375240-24029, 345/757, 345/173, 345/158, 463/42, 455/566
• US Class Current: 1/1
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics

Abstract

Certain embodiments teach a variety of experience or “sentio” codecs, and methods and systems for enabling an experience platform, as well as a Quality of Experience (QoS) engine which allows the sentio codec to select a suitable encoding engine or device. The sentio codec is capable of encoding and transmitting data streams that correspond to participant experiences with a variety of different dimensions and features. As will be appreciated, the following description provides one paradigm for understanding the multi-dimensional experience available to the participants, and as implemented utilizing a sentio codec. There are many suitable ways of describing, characterizing and implementing the sentio codec and experience platform contemplated herein.

Claims

1. A sentio codec for encoding and decoding a plurality of multi-dimensional data streams for a multi-dimensional experience, the sentio codec comprising:

a plurality of codecs suitable for encoding and decoding multi-dimensional experience data streams related to a multi-dimensional experience;

a quality of experience (“

QoE”

) engine;

wherein the QoE engine is configured to;

receive an output associated with the multi-dimensional experience, wherein the output is divided into a plurality of regions;

receive data associated with the capabilities of a transmitting device and a receiving device;

wherein, the transmitting device and the receiving device are connected via a network;

receive data associated with the current conditions of the network;

cause one or more of the plurality of codecs to encode each of the regions of the output into one or more of the plurality of multi-dimensional data streams;

wherein, the encoding of each region is based on the data associated with the capabilities of the transmitting device, the data associated with the capabilities of the receiving device, and the data associated with the current conditions of the network; and

a network stack;

wherein, the network stack implements a low-latency transfer protocol suitable for enabling a multi-dimensional experience;

wherein the network stack is configured to cause the transmitting device to transmit the encoded plurality of multi-dimensional data streams according to the low-latency transfer protocol;

wherein, the transmitting includes;

parsing the encoded plurality of data streams into a reliable queue, an unreliable queue, and an unreliable with forward error correction queue, wherein each queue has a plurality of channels, each channel associated with a priority;

transmitting the reliable queue, wherein each packet in the reliable queue is acknowledged by the receiving device and includes a reliable packet sequence number;

transmitting the unreliable queue, wherein each packet in the unreliable queue includes an unreliable packet sequence number; and

transmitting the unreliable with forward error correction queue, wherein each packet in the unreliable with forward error correction queue includes a transform code containing redundant information to recover lost packets.

2. A sentio codec as recited in claim 1, wherein the plurality of codecs includes an audio codec and a video codec.

3. A sentio codec as recited in claim 2, wherein the plurality of codecs further includes a gesture command codec.

4. A sentio codec as recited in claim 2, wherein the plurality of codecs further includes a sensor data codec.

5. A sentio codec as recited in claim 2, wherein the plurality of codecs further includes an emotion data codec.

6. A sentio codec as recited in claim 1, wherein the encoding of each region is further based on a characteristics of the output in each region.

7. A method for encoding and transmitting a plurality of multi-dimensional data streams for a multi-dimensional experience, by a sentio codec, the method comprising:

receiving, by a quality of experience (“

QoE”

) engine, an output associated with the multi-dimensional experience, wherein the output is divided into a plurality of regions;

receiving, by the QoE engine, data associated with the capabilities of a transmitting device and a receiving device, wherein, the transmitting device and the receiving device are connected via a network;

receiving, by the QoE engine, data associated with the current conditions of the network;

selecting, by the QoE engine, one or more codecs out of a plurality of codecs with which to encode each region of the output based on the data associated with the capabilities of the transmitting device, the data associated with the capabilities of the receiving device, and the data associated with the current conditions of the network;

encoding, by one or more of the plurality codecs, based on the selection by the QoE engine, each of the regions of the output into one or more of the plurality of multi-dimensional data streams; and

transmitting, by the transmitting device, using the network stack, the encoded plurality of multi-dimensional data streams, according to a low-latency transfer protocol;

wherein, the transmitting includes;

parsing the encoded plurality of data streams into a reliable queue, an unreliable queue, and an unreliable with forward error correction queue, wherein each queue has a plurality of channels, each channel associated with a priority;

transmitting the reliable queue, wherein each packet in the reliable queue is acknowledged by the receiving device and includes a reliable packet sequence number;

transmitting the unreliable queue, wherein each packet in the unreliable queue includes an unreliable packet sequence number; and

transmitting the unreliable with forward error correction queue, wherein each packet in the unreliable with forward error correction queue includes a transform code containing redundant information to recover lost packets.

8. A method as recited in claim 7, wherein the plurality of codecs includes an audio codec and a video codec.

9. A method as recited in claim 8, wherein the plurality of codecs further includes a gesture command codec.

10. A method as recited in claim 8, wherein the plurality of codecs further includes a sensor data codec.

11. A method as recited in claim 8, wherein the plurality of codecs further includes an emotion data codec.

12. A method as recited in claim 7, wherein the encoding of each region is further based on a characteristics of the output in each region.