Detecting natural user-input engagement

• US 9,785,228 B2
• Filed: 02/11/2013
• Issued: 10/10/2017
• Est. Priority Date: 02/11/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schwesinger, Mark, Escardo Raffo, Eduardo, Murillo, Oscar, Bastien, David, Ahn, Matthew H., Giusti, Mauro, Endres, Kevin, Klein, Christian, Schwarz, Julia, Marais, Charles Claudius
• Primary Examiner(s): Spar, Ilana
• Assistant Examiner(s): Castiaux, Brent D
• Application Number:
  US13/764,056
• Publication Number:
  US 20140225820A1
• Time in Patent Office: 1,702 Days
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An NUI system to provide user input to a computer system. The NUI system includes a logic machine and an instruction-storage machine. The instruction-storage machine holds instructions that, when executed by the logic machine, cause the logic machine to detect an engagement gesture from a human subject or to compute an engagement metric reflecting the degree of the subject'"'"'s engagement. The instructions also cause the logic machine to direct gesture-based user input from the subject to the computer system as soon as the engagement gesture is detected or the engagement metric exceeds a threshold.

Claims

1. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions executable by the logic machine, the instructions comprising:

instructions to receive posture information derived from depth video of a person, the depth video acquired with a depth camera;

instructions to analyze the received posture information to compute an engagement metric for the person, the engagement metric increasing with greater indication that the person wishes to engage the computer system as a user, and decreasing with lesser indication that the person wishes to engage the computer system;

instructions to assess user input being provided to the computer system;

instructions to permit facile initial engagement of the person as the user and prevent loss of user engagement to another person in the depth video by maintaining a variable threshold at a first level responsive to posture information from a currently engaged user being processed as user input to the computer system, and at a second level, lower than the first level, responsive to there being no currently engaged user;

instructions to process the received posture information as user input to the computer system only after the engagement metric exceeds the variable threshold;

instructions to forego processing the received posture information as input of an engaged user of the computer system and to save the received posture information for subsequent processing until the engagement metric exceeds the variable threshold; and

instructions to retroactively process the saved posture information as input of an engaged user of the computer system after the engagement metric exceeds the variable threshold.

2. The system of claim 1 wherein the person is one of a plurality of persons for whom the logic machine receives and analyzes posture information, wherein the engagement metric is computed for each of the persons, and wherein the posture information for the person having the highest engagement metric is processed as user input to the computer system.

3. The system of claim 1 wherein the variable threshold decreases with increasing vehemence of an engagement gesture of the person.

4. The system of claim 1 wherein the engagement metric increases with increasing vehemence of an engagement gesture of the person.

5. The system of claim 1 wherein the engagement metric is further based on microphonic data from the person.

6. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions executable by the logic machine, the instructions comprising:

instructions to receive, for the human subject, posture information derived from depth video of that subject, the depth video acquired with a depth camera;

instructions to analyze the received posture information to detect an engagement gesture from the human subject, the engagement gesture including a raising of a hand of the subject to a zone above the subject'"'"'s waist and between the subject'"'"'s torso and a display plane of the computer system, the raising followed by a pause during which the hand is stationary;

instructions to process the received posture information as input of an engaged user of the computer system after detection of the engagement gesture;

instructions to forego processing the received posture information as input of an engaged user of the computer system and to save the received posture information for subsequent processing until the engagement gesture is detected; and

instructions to retroactively process the saved posture information as input of an engaged user of the computer system upon detection of the engagement gesture.

7. The system of claim 6 wherein the raising is followed immediately by the pause, and wherein the posture information is processed as the user input immediately after the pause.

8. The system of claim 6 wherein the engagement gesture is detected absent side-to-side motion of the hand.

9. The system of claim 6 wherein the engagement gesture is an open-palm gesture.

10. The system of claim 6 wherein the engagement gesture is one in which a palm of the hand faces the display plane.

11. The system of claim 6 wherein the engagement gesture is one in which a finger of the hand is pointed toward the display plane.

12. The system of claim 6 wherein the engagement gesture is one in which a torso of the human subject faces the display plane.

13. The system of claim 6 wherein the engagement gesture is one in which a centroid of the human subject is substantially stationary.

14. The system of claim 6 wherein the engagement gesture is one in which a face or gaze of the human subject is directed toward the display plane.

15. The system of claim 6 wherein the zone extends above a head of the human subject and one arm'"'"'s length on both sides of the human subject.

16. The system of claim 6 wherein the pause is one-quarter second to one second in duration, and wherein the posture information is processed as the user input immediately after the pause.

17. A natural user-input (NUI) system to provide user input to a computer system, the NUI system including a logic machine and an instruction-storage machine, the instruction-storage machine holding instructions executable by the logic machine, the instructions comprising:

instructions to select a human subject to provide user input to the computer system;

instructions to receive, for the human subject, posture information derived from depth video of that subject, the depth video acquired with a depth camera;

instructions to analyze the received posture information to compute an engagement metric for the human subject, the engagement metric increasing with greater indication that the user wishes to engage the computer system and decreasing with lesser indication that the user wishes to engage the computer system;

instructions to detect an engagement gesture of the human subject when the engagement metric exceeds a threshold;

instructions to process the received posture information as input of an engaged user of the computer system after detection of the engagement gesture;

instructions to forego processing the received posture information as input of an engaged user of the computer system and to save the received posture information for subsequent processing until the engagement gesture is detected; and

instructions to retroactively process the saved posture information as input of an engaged user of the computer system upon detection of the engagement gesture.

18. The system of claim 17 wherein the human subject is selected based on proximity to the depth camera and/or position in a field of view of the depth camera.