User movement tracking feedback to improve tracking

• US 9,898,675 B2
• Filed: 06/22/2010
• Issued: 02/20/2018
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yee, Dawson, Perez, Kathryn Stone
• Primary Examiner(s): Shih, Haoshian
• Application Number:
  US12/820,954
• Publication Number:
  US 20100277411A1
• Time in Patent Office: 2,800 Days
• Field of Search: 715863, 345156, 382107, 382154, 463 36- 39
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/67 by interactive indications to the user

Abstract

Technology is presented for providing feedback to a user on an ability of an executing application to track user action for control of the executing application on a computer system. A capture system detects a user in a capture area. Factors in the capture area and the user'"'"'s actions can adversely affect the ability of the application to determine if a user movement is a gesture which is a control or instruction to the application. One example of such factors is a user being out of the field of view of the capture system. Some other factor examples include lighting conditions and obstructions in the capture area. Responsive to a user tracking criteria not being satisfied, feedback is output to the user. In some embodiments, the feedback is provided within the context of an executing application.

Claims

1. In a computer-implemented system operable to track user action for control of a displayed object, a method for providing feedback to a user on an ability of the system to track user action comprising:

detecting the user in a capture area with a capture system, the user interacting with an application on the system having interactive display objects, the application configured to display an application interface using interactive display objects during execution of the application, the user interacting with an interactive display object of the interactive display objects for a purpose of furthering an objective of the application, the interactive display object moving independently of the user, the user moving left, right, up or down in order to be in a position to interact with the interactive display object;

tracking at least one body feature of the user based on data generated by the capture system;

while executing the application and responsive to user tracking criteria not being satisfied during execution of the application,

rendering the interactive display object during execution of the application in a new position on the display, moved toward a center of the display, such that a user moves toward a center of the capture area in order to interact with the interactive display object.

2. The method of claim 1, wherein the application comprises a first game application, and the interactive display object comprising a first interactive display object, the method further comprising:

receiving user interaction with a second, different game application on the system and repeating the tracking for the second application;

while executing the second game application and, responsive to user tracking criteria not being satisfied during game play in the second game application, determining a rendering of a second interactive display object of the second game application, the second interactive display object being different than the first interactive display object,

rendering the second interactive display object of the second game application during game play such that interaction by the user with the second display object of the second application repositions the at least one body part of the user.

3. The method of claim 2 further comprising:

wherein the capture system is an image capture system;

wherein the user tracking criteria not being satisfied is that a body part of the user which can control the displayed object is at least partially out of a field of view of the image capture system; and

wherein the at least one action of the object on the display induces user movement of the body part towards a center of the field of view.

4. The method of claim 1 wherein the interactive display object is determined based on a profile of the user including an age of the user such that the interactive display object is an age appropriate response.

5. The method of claim 1 wherein the interactive display object includes no representation of any body part of the user.

6. The method of claim 1 wherein outputting a feedback response to the user comprises outputting an audio indicator to the user indicating a user tracking criteria is not being satisfied.

7. The method of claim 1 wherein outputting a feedback response to the user further comprises changing a visual characteristic of a displayed view of the user.

8. The method of claim 1 wherein the interactive display object is an element of a game.

9. The method of claim 1 further comprising wherein the user tracking criteria is based on a visibility factor of the group consisting of the following:

a location within a capture system'"'"'s field of view of a body part of the user which can control the displayed object;

lighting;

contrast; and

an obstruction of the body part of the user.

10. The method of claim 1 further comprising wherein a user tracking criteria is based on a distinguishability factor which is an audio factor consisting of one of the following:

volume; and

distinguishability of speech.

11. A system for providing feedback to a user on an ability of the system to track user action comprising:

an image capture system for capturing image data of a user within a capture area;

one or more processors communicatively coupled to the image capture system and accessible to a memory storing the image data and instructions;

a display communicatively coupled to the one or more processors for displaying feedback to the user;

the one or more processors executing instructions comprising;

executing a first game application of a plurality of applications, the first game application configured to display an application interface using interactive display objects, an interactive display object of the interactive display objects comprising at least one of a virtual ball, a virtual target, a virtual weapon, a virtual opponent and a virtual musical instrument allowing a user to participate in game play,

detecting at least one body part of the user in the image data,

tracking movement of the at least one body part, the user interacting with the first game application on the system via the capture area, the application responsive to user movement of the at least one body part;

while executing the first game application and responsive to user tracking criteria of at least one body part not being satisfied, determining a rendering of the interactive display object in an application context in a new position moved toward a center of the display such that movement by the user to interact with the interactive display object would reposition the at least one body part in a direction towards a center of the capture area while advancing game play, the interactive display object moving independently of the user; and

displaying the interactive display object on the display during game play.

12. The system of claim 11 wherein the one or more processors executing instructions further comprises executing instructions for assisting the user in selecting a capture area by:

displaying instructions to make at least one movement of the at least one body part;

tracking movement of the at least one body part;

responsive to user tracking criteria based on one or more visibility factors being satisfied, determining a tracking quality score based on visibility factors for a test capture area;

responsive to user tracking criteria based on one or more visibility factors not being satisfied;

determining how one or more visibility factors for the image data can be increased to satisfy the user tracking criteria; and

displaying feedback to the user identifying at least one change in the capture area to increase a visibility factor.

13. The system of claim 12 wherein the one or more processors executing instructions further comprises executing instructions comprising:

determining a tracking quality score for each of a plurality of test capture areas based on tracking criteria of visibility factors;

identifying the test capture area with a highest score; and

displaying a recommendation of the test capture area with the highest score.

14. The system of claim 12 wherein the one or more visibility factors includes one or more of the following:

a location within a capture system'"'"'s field of view of at least one body part of the user;

lighting;

contrast; and

an obstruction of at least one body part of the user.

15. The system of claim 11 wherein the display is an off-screen display device.

16. The system of claim 11 wherein the display displays the feedback on a screen of the display.