Altering streaming video encoding based on user attention

• US 9,253,494 B2
• Filed: 11/26/2014
• Issued: 02/02/2016
• Est. Priority Date: 12/01/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McCuller, Patrick G.
• Primary Examiner(s): LUO, KATE H
• Application Number:
  US14/554,322
• Publication Number:
  US 20150078459A1
• Time in Patent Office: 433 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate

Abstract

Disclosed are various embodiments for adjusting the encoding of a video signal into a video stream based on user attention. A video signal is encoded into a video stream. A temporary lapse of attention by a user of the interactive application is predicted. The encoding of the video signal into the video stream is adjusted from an initial state to a conservation state in response to predicting the temporary lapse of attention by the user. The conservation state is configured to conserve one or more resources used for the video stream relative to the initial state.

Claims

1. A method, comprising:

encoding, by at least one computing device, a video signal generated by an interactive application into a video stream, wherein the interactive application is executed by the at least one computing device, and a user interacts with the interactive application by sending at least one input to the interactive application over a data communications network;

predicting, by the at least one computing device, a temporary lapse of attention by the user to the video stream, wherein the temporary lapse of attention is predicted in response to a decrease in input by the user to the interactive application;

adjusting, by the at least one computing device, an encoding of the video stream from an initial state to a conservation state in response to predicting the temporary lapse of attention;

predicting, by the at least one computing device, that the temporary lapse of attention has ended;

readjusting, by the at least one computing device, the encoding of the video stream from the conservation state to the initial state in response to predicting that the temporary lapse of attention has ended; and

wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state.

2. The method of claim 1, wherein predicting the temporary lapse of attention further comprises detecting, by the at least one computing device, that the user is not paying attention to the video stream.

3. The method of claim 1, wherein the interactive application comprises a game application that is executed by the at least one computing device, and the user interacts with the game application by sending at least one input to the game application over a data communications network.

4. The method of claim 1, wherein predicting the temporary lapse of attention further comprises determining, by the at least one computing device, that the interactive application has not obtained an input from the user in at least a predetermined time period.

5. The method of claim 1, wherein adjusting the encoding further comprises reconfiguring, by the at least one computing device, the interactive application to conserve at least one resource used in generating the video signal.

6. The method of claim 1, wherein adjusting the encoding further comprises stopping, by the at least one computing device, the encoding of a video signal into the video stream.

7. The method of claim 1, wherein adjusting the encoding further comprises reconfiguring, by the at least one computing device, a video encoder that performs the encoding to use a first encoding configuration that is associated with a lower quality video stream than a second encoding configuration that is used in the initial state.

8. The method of claim 1, wherein predicting that the temporary lapse of attention has ended further comprises determining, by the at least one computing device, that an interactive application has obtained at least one input from the user.

9. The method of claim 1, wherein readjusting the encoding further comprises restarting, by the at least one computing device, a stopped encoding of the video stream.

10. The method of claim 1, wherein readjusting the encoding further comprises reconfiguring, by the at least one computing device, a video encoder that performs the encoding to use an encoding configuration associated with the initial state.

11. A system, comprising:

at least one computing device; and

at least one application executed in the at least one computing device, wherein when executed the at least one application causes the at least one computing device to at least;

predict a temporary lapse of attention by a user to a video stream, the video stream including a video signal generated by an interactive application, wherein the user interacts with the interactive application by sending at least one input to the interactive application over a data communications network, and the temporary lapse of attention is predicted in response to a decrease in input by the user to the interactive application;

adjust an encoding of the video stream from an initial state to a conservation state in response to predicting the temporary lapse of attention;

predict that the temporary lapse of attention has ended;

readjust the encoding of the video stream from the conservation state to the initial state in response to predicting that the temporary lapse of attention has ended; and

wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state.

12. The system of claim 11, wherein when executed the at least one application further causes the at least one computing device to receive an indication that the user is idle from a client device configured to render the video stream.

13. The system of claim 11, wherein when executed the at least one application further causes the at least one computing device to predict that an attention of the user is drawn to a saccadic event.

14. The system of claim 11, wherein when executed the at least one application further causes the at least one computing device to determine that input from the user is not received for at least a predetermined time period.

15. A non-transitory computer-readable medium embodying a program executable by at least one computing device, wherein when executed the program causes the at least one computing device to at least:

encode a video signal generated by an interactive application into a video stream, wherein the interactive application is executed by the at least one computing device, and a user interacts with the interactive application by sending at least one input to the interactive application over a data communications network;

send the video stream to a plurality of client devices corresponding to a plurality of users;

predict a temporary lapse of attention by all of the plurality of users to the video stream, wherein the temporary lapse of attention is predicted in response to a decrease in input by the plurality of users to the interactive application;

adjust an encoding of the video stream from an initial state to a conservation state in response to predicting the temporary lapse of attention;

predict that the temporary lapse of attention has ended;

readjust the encoding of the video stream from the conservation state to the initial state in response to predicting that the temporary lapse of attention has ended; and

wherein the conservation state is configured to conserve at least one resource used for the encoding of the video stream relative to the initial state.

16. The non-transitory computer-readable medium of claim 15, wherein the plurality of users are participants in a session of the interactive application.

17. The non-transitory computer-readable medium of claim 15, wherein, as compared to the initial state, the conservation state is associated with at least one of:

a lower resolution, a lower frame rate, a lower bitrate, or a lower color depth.

18. The non-transitory computer-readable medium of claim 15, wherein when executed the program further causes the at least one computing device to determine that the interactive application has not obtained an input from any of the plurality of users in at least a predetermined time period.

19. The non-transitory computer-readable medium of claim 15, wherein predicting the temporary lapse of attention further comprises detecting that the plurality of users are not paying attention to the video stream.

20. The non-transitory computer-readable medium of claim 15, wherein the interactive application comprises a game application.