Controlling display content based on a line of sight of a user

• US 10,120,441 B2
• Filed: 06/30/2014
• Issued: 11/06/2018
• Est. Priority Date: 09/13/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Imoto, Maki, Noda, Takuro, Yasuda, Ryouhei
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Lemieux, Ian
• Application Number:
  US14/917,244
• Publication Number:
  US 20160195926A1
• Time in Patent Office: 1,590 Days
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

There is provided an information processing apparatus including an image acquisition unit configured to acquire a captured image of users, a determination unit configured to determine an operator from among the users included in the acquired captured image, and a processing unit configured to conduct a process based on information about user line of sight corresponding to the determined operator.

Claims

1. An information processing apparatus, comprising:

a display screen; and

one or more processors configured to;

acquire a captured image that includes a plurality of users;

detect a plurality of face regions that correspond to the plurality of users in the captured image;

determine an operator from the plurality of users included in the captured image, based on a position of a line of sight of a first user of the plurality of users in a specific region on the display screen and based on a size of each of the plurality of face regions; and

control a first process based on first information associated with the line of sight corresponding to the determined operator.

2. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to determine the first user of the plurality of users as the operator based on a first face region of the plurality of face regions, wherein the first face region corresponds to the first user that has a largest face region size among the plurality of face regions.

3. The information processing apparatus according to claim 2, wherein based on each of a set of face regions of the plurality of face regions, that have a face region size that is largest among the plurality of face regions, the one or more processors are further configured to determine the first face region of the set of face regions as the operator, wherein the first face region is detected prior to a second face region of the set of face regions.

4. The information processing apparatus according to claim 3, wherein the one or more processors are further configured to determine a configured number of users of the plurality of users as the operator, based on the set of face regions that have the face region size that is largest among the plurality of face regions.

5. The information processing apparatus according to claim 2, wherein

the one or more processors are further configured to determine the first user is a user other than the operator based on a first difference value that is equal to or greater than a first threshold value,

the first difference value indicates a difference of a first face region size corresponding to the first user and a second face region size corresponding to a second user other than the first user.

6. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to determine the operator based on a corresponding distance of each face region of the plurality of face regions from the display screen.

7. The information processing apparatus according to claim 6, wherein the one or more processors are further configured to determine the first user of the plurality of users as the operator based on a first face region of the first user that has a shortest distance among the corresponding distance of the each face region of the plurality of face regions from the display screen.

8. The information processing apparatus according to claim 7, wherein based on each of a set of distances having the shortest distance among the corresponding distance of the each face region of the plurality of face regions, the one or more processors are further configured to determine the first user as the operator, wherein the first user that corresponds to the set of distances is detected prior to a second user that corresponds to the set of distances.

9. The information processing apparatus according to claim 8, wherein the one or more processors are further configured to determine a configured number of users of the plurality of users as the operator, based on the set of distances that have the shortest distance.

10. The information processing apparatus according to claim 7, wherein

the one or more processors are further configured to determine the first user is a user other than the operator based on a second difference value that is equal to or greater than a second threshold value,

the second difference value indicates a difference between a first distance from the display screen and a second distance from the display screen, wherein the first distance corresponds to the first face region of the first user and the second distance corresponds to a second face region of a second user of the plurality of users.

11. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to determine the operator based on a specific gesture detected from the captured image.

12. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to determine the operator based on a rank associated with each of the plurality of users.

13. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to:

determine an operation level for the first user of the plurality of users, wherein the first user is determined as the operator; and

dynamically change available processes based on the determined operation level.

14. The information processing apparatus according to claim 1, wherein based on a determination that a known speech is spoken by a second user of the plurality of users, the one or more processors are further configured to control a second process based on second information about a line of sight corresponding to the second user.

15. The information processing apparatus according to claim 1, wherein based on an unavailability of the first information, the one or more processors are further configured to determine a new operator from among the plurality of users.

16. The information processing apparatus according to claim 1, wherein based on an unavailability of the first information, the one or more processors are further configured to determine the operator based on a detection of a head among the plurality of users.

17. The information processing apparatus according to claim 1, wherein based on a detection of a known combination of gestures by the first user determined as the operator and a second user other than the first user, the one or more processors are further configured to change the operator from the first user to the second user.

18. The information processing apparatus according to claim 1, wherein the one or more processors are further configured to determine the first user of the plurality of users as the operator based on the position of the line of sight of the first user that is included in the specific region on the display screen, and wherein the first information associated with the line of sight of the first user indicates that the position of the line of sight of the first user is included in the specific region on the display screen.

19. An information processing method, comprising:

in an information processing apparatus;

acquiring a captured image that includes a plurality of users;

detecting a plurality of face regions that correspond to the plurality of users in the captured image;

determining an operator from the plurality of users included in the captured image, based on a position of a line of sight of a user of the plurality of users in a specific region on a display screen and based on a size of the plurality of face regions of the plurality of users; and

controlling a process based on information associated with the line of sight corresponding to the determined operator.