System and method for guided continuous body tracking for complex interaction

• US 9,727,778 B2
• Filed: 05/12/2014
• Issued: 08/08/2017
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Wipro Limited
• Original Assignee: Wipro Limited
• Inventors: Kumar, Amit, Raj, Sheeba, Madhusudhanan, Manoj
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/275,492
• Publication Number:
  US 20150278586A1
• Time in Patent Office: 1,184 Days
• Field of Search: None
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method, non-transitory computer readable medium and apparatus that provides feedback to a user for interacting continuously without any disconnect. It gives real time visual clues and gestures based on application need. This solves the problem of a disconnected user found in current system and applications as user will be continuously aware whether and how much he is being tracked by the application that will allow him to continuously interact with the application. The requirements from various applications like FBT (Full body Tracking), HBT (Half Body Tracking), LTO (Leg Tracking Only), HTO (Hand Tracking Only), STO (Specific Tracking only) is sent to the CFBTE (Continuous Full Body Tracking Engine). The CFBTE process the requirements and generates the visual clues and gesture events. In visual clues the tracking level/intensity is continuously displayed, which will guide the user on system readiness and effectiveness for interaction.

Claims

1. A method for continuous tracking of body of at least one user, the method comprising:

capturing, by a tracking management computing apparatus, tracking points on body of the at least one user with a capture system;

listening, by the tracking management computing apparatus, continuously to at least one application to receive data related to one or more gestures required by the at least one application and body tracking points associated with the one or more gestures;

filtering, by the tracking management computing apparatus, the tracking points on the body based on the body tracking points associated with the one or more gestures;

monitoring, by the tracking management computing apparatus, continuously a variation of intensity of the filtered one or more tracking points, the variation of the intensity being associated with tracking of each of the filtered one or more tracking points;

generating, by the tracking management computing apparatus, an audiovisual feedback in real time for the at least one user when the variation of the intensity fails to meet a predetermined body tracking criterion, the predetermined body tracking criterion based on tracking of the filtered one or more tracking points, wherein at least one of the capturing, the listening, the filtering, the monitoring, or the generating is performed by one or more hardware processors; and

confirming, by the tracking management computing apparatus, the variation of the intensity associated with tracking of each of the filtered one or more tracking points are above a threshold before any tracking of the one or more gestures required by the at least one application.

2. The method of claim 1, wherein the generating an audiovisual feedback comprises generating, by the tracking management computing apparatus, a human silhouette of the at least one user.

3. The method of claim 2, further comprising adjusting position by the at least one user to meet the predetermined tracking criterion in response to the audiovisual feedback to the at least one user.

4. The method of claim 1, wherein the capture system comprises an infrared (IR) device.

5. The method of claim 4, wherein the IR device captures a color map, depth map, and a skeleton map.

6. The method of claim 1, wherein the method further comprises sending, by the tracking management computing apparatus, the filtered tracking points associated with the one or more gestures to a gesture generator component.

7. The method of claim 6, wherein the method further comprises generating, by the gesture generator component, the one or more gestures based on the filtered tracking points.

8. The method of claim 7, wherein the method further comprises storing the generated one or more gestures in a sequence in a gesture queue, the sequence based on the requirement of the at least one application.

9. The method of claim 1, further comprising identifying, by the tracking management computing apparatus, action of the at least one user by the gesture event listener.

10. The method of claim 1, further comprising matching, by the tracking management computing apparatus, action of the at least one user with the stored one or more gestures to trigger at least one event specific to matched gesture.

11. A tracking management computing apparatus comprising:

a capture system to capture tracking points on body of the at least one user;

one or more hardware processors; and

a memory storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising;

listening continuously to at least one application to receive data related to one or more gestures required by the at least one application and body tracking points associated with the one or more gestures;

filtering the tracking points on the body based on the body tracking points associated with the one or more gestures;

monitoring continuously a variation of intensity of the filtered one or more tracking points, the variation of the intensity being associated with tracking of each of the filtered one or more tracking points;

generating an audiovisual feedback in real time for the at least one user when the variation of the intensity fails to meet a predetermined body tracking criterion, the predetermined body tracking criterion based on tracking of the filtered one or more tracking points; and

confirming the variation of the intensity associated with tracking of each of the filtered one or more tracking points are above a threshold before any tracking of the one or more gestures required by the at least one application.

12. The apparatus of claim 11, wherein generating the audiovisual feedback comprises generating a human silhouette of the at least one user.

13. The apparatus of claim 12, wherein the at least one user adjusts position to meet the predetermined tracking criterion in response to the audiovisual feedback to the at least one user.

14. The apparatus of claim 11, wherein the capture system comprises an infrared (IR) device.

15. The apparatus of claim 11, wherein the one or more hardware processors receive the filtered tracking points associated with the one or more gestures to generate the one or more gestures.

16. The apparatus of claim 15, wherein the memory stores the generated one or more gestures in a sequence, the sequence based on the requirement of the at least one application.

17. The apparatus of claim 11, wherein the memory stores further instructions that when executed by the one or more hardware processors cause the one or more hardware processors to perform operations comprising:

identifying action of the at least one user by gesture event listener.

18. The apparatus of claim 17, wherein the memory stores further instructions that, when executed by the one or more hardware processors cause the one or more hardware processors to perform operations comprising:

matching action of the at least one user with the stored one or more gestures to trigger at least one event specific to matched gesture.

19. A non-transitory computer-readable medium that stores programmed instructions for continuous tracking of body of at least one user that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising:

capturing tracking points on body of the at least one user with a capture system;

listening continuously to at least one application to receive data related to one or more gestures required by the at least one application and body tracking points associated with the one or more gestures;

filtering the tracking points on the body based on the body tracking points associated with the one or more gestures;

monitoring continuously a variation of intensity of the filtered one or more tracking points, the variation of the intensity being associated with tracking of each of the filtered one or more tracking points;

generating an audiovisual feedback in real time for the at least one user when variation of intensity fails to meet a predetermined body tracking criterion, the predetermined body tracking criterion based on tracking of the filtered one or more tracking points; and

confirming the variation of the intensity associated with tracking of each of the filtered one or more tracking points are above a threshold before any tracking of the one or more gestures required by the at least one application.

20. The non-transitory computer-readable medium of claim 19, wherein generating the audiovisual feedback comprises generating a human silhouette of the at least one user.

21. The non-transitory computer-readable medium of claim 20, wherein the at least one user adjusts position to meet the predetermined tracking criterion in response to the audiovisual feedback to the user.

22. The non-transitory computer-readable medium of claim 19, wherein the non-transitory computer-readable medium stores further instructions that, when executed by the one or more hardware processors causes the one or more hardware processors to perform operations comprising:

receiving the filtered tracking points associated with the one or more gestures to generate the one or more gestures.

23. The non-transitory computer-readable medium of claim 22, wherein the non-transitory computer-readable medium stores further instructions that, when executed by the one or more hardware processors causes the one or more hardware processors to perform operations comprising:

storing the generated one or more gestures in a sequence, the sequence based on the requirement of the at least one application.