Altering streaming video encoding based on user attention

• US 9,942,556 B2
• Filed: 01/06/2016
• Issued: 04/10/2018
• Est. Priority Date: 12/01/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McCuller, Patrick G.
• Primary Examiner(s): LUO, KATE H
• Application Number:
  US14/989,076
• Publication Number:
  US 20160119627A1
• Time in Patent Office: 825 Days
• Field of Search: None
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate

Abstract

Disclosed are various embodiments for adjusting the encoding of a video signal into a video stream based on user attention. A temporary lapse of attention by a user is predicted. The encoding of the video signal into the video stream is adjusted from an initial state to a conservation state in response to predicting the temporary lapse of attention by the user. The conservation state is configured to conserve one or more resources used for the video stream relative to the initial state.

Claims

1. A system, comprising:

at least one computing device; and

at least one application executable in the at least one computing device, wherein when executed the at least one application causes the at least one computing device to at least;

predict a temporary lapse of attention by a user to a video stream;

adjust an encoding of the video stream from an initial state to a conservation state in response to predicting the temporary lapse of attention, wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state;

predict that the temporary lapse of attention has ended; and

readjust the encoding of the video stream from the conservation state to the initial state in response to predicting that the temporary lapse of attention has ended.

2. The system of claim 1, wherein the video stream includes a video signal generated by an interactive application.

3. The system of claim 2, wherein the user interacts with the interactive application by sending at least one input to the interactive application over a data communications network.

4. The system of claim 1, wherein the temporary lapse of attention is predicted in response to a decrease in an input by the user.

5. The system of claim 1, wherein the temporary lapse of attention is predicted in response to receiving an indication that the user is idle from a client computing device configured to render the video stream.

6. The system of claim 1, wherein the temporary lapse of attention is predicted in response to determining that an attention of the user is likely to be drawn to a saccadic event.

7. The system of claim 1, wherein the temporary lapse of attention is predicted in response to determining that an input has not been received from the user for at least a predetermined time period.

8. The system of claim 1, wherein adjusting the encoding further comprises stopping the encoding of a video signal into the video stream.

9. A method, comprising:

sending, by at least one computing device, a video stream to a client computing device;

determining, by the at least one computing device, that an input from the client computing device has decreased;

adjusting, by the at least one computing device, an encoding of the video stream from an initial state to a conservation state based at least in part on determining that the input from the client computing device has decreased, wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state;

determining, by the at least one computing device, that the input from the client computing device has increased after having decreased; and

readjusting, by the at least one computing device, the encoding of the video stream from the conservation state to the initial state in response to determining that the input from the client computing device has increased after having decreased.

10. The method of claim 9, wherein, as compared to the initial state, the conservation state is associated with at least one of:

a lower resolution, a lower frame rate, a lower bitrate, or a lower color depth.

11. The method of claim 9, further comprising encoding, by the at least one computing device, a video signal generated by an interactive application into the video stream.

12. The method of claim 11, further comprising providing, by the at least one computing device, the input from the client computing device to the interactive application.

13. The method of claim 9, wherein adjusting the encoding further comprises stopping, by the at least one computing device, the encoding of a video signal into the video stream.

14. The method of claim 9, wherein the input from the client computing device comprises data encoding at least one input command from a user of the client computing device.

15. The method of claim 9, wherein the at least one input command comprises an input command generated by a mouse.

16. A non-transitory computer-readable medium embodying a program executable in at least one computing device, wherein when executed the program causes the at least one computing device to at least:

send a video stream to a client computing device associated with a user;

receive that an indication from the client computing device that the user is idle;

adjust an encoding of the video stream from an initial state to a conservation state based at least in part on the indication from the client computing device that the user is idle, wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state;

receive that a subsequent indication from the client computing device that the user is no longer idle; and

readjust the encoding of the video stream from the conservation state to the initial state in response to the subsequent indication from the client computing device that the user is no longer idle.

17. The non-transitory computer-readable medium of claim 16, wherein when executed the program further causes the at least one computing device to at least encode a video signal generated by an interactive application into the video stream.

18. The non-transitory computer-readable medium of claim 17, wherein the user interacts with the interactive application by sending at least one input to the interactive application over a data communications network.

19. The non-transitory computer-readable medium of claim 16, wherein the indication that the user is idle is sent by the client computing device in response to a client application being minimized.

20. The non-transitory computer-readable medium of claim 16, wherein adjusting the encoding further comprises stopping the encoding of a video signal into the video stream.