Systems and methods of detecting body movements using globally generated multi-dimensional gesture data

• US 9,639,746 B2
• Filed: 04/18/2012
• Issued: 05/02/2017
• Est. Priority Date: 07/28/2011
• Status: Active Grant

• Current Assignee: ARB Labs Inc.
• Original Assignee: ARB Labs Inc.
• Inventors: Bulzacki, Adrian
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US14/235,594
• Publication Number:
  US 20140198954A1
• Time in Patent Office: 1,840 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: Y02A 90/10 Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

The disclosure describes systems and methods of detecting body movements using gesture data. The gesture data may be self-referenced and may be comprised by frames which may identify locations or positions of body parts of a subject with respect to a particular reference point within the frame. A classifier may process frames to learn body movements and store the frames of gesture data in a database. Data comprising frames of self-referenced gesture data may be received by a recognizer which recognizes movements of the subject identified by the frames by matching gesture data of the incoming frames to the classified self-referenced gesture data stored in the database.

Claims

1. A method of identifying a movement of a subject based on data, the method comprising:

receiving at a network interface, a stream of frames comprising gesture data, each frame of the stream of frames identifying positions of one or more body parts of a subject with respect to the waist of the subject'"'"'s body, the stream of frames being stored in storage memory;

extrapolating, by a processor, from the stream of frames comprising gesture data, one or more frames corresponding to a first movement;

assigning, by a classifier, the one or more frames to the first movement, the classifier applying a scale invariant feature transform is used to determine a descriptor of the first movement;

dividing, by the processor using a self-organizing map, the stream of frames into separate phases;

identifying, by the processor using a scalar vector machine, transition conditions within a feature space between phases;

receiving, by the network interface, a new gesture data identifying positions of one or more body parts of a new subject with respect to the waist of the new subject'"'"'s body;

extrapolating, by a processor, from the new gesture data one or more frames comprising gesture data identifying one or more features of the body movement of the new subject;

determining, by the processor, that movement of the new subject corresponds to the first movement responsive to comparing at least a portion of the new gesture data to at least a portion of the gesture data of the one or more frames corresponding to a first movement by applying at least both the descriptor provided by the scale invariant feature transform and the transition conditions identified by the scalar vector machine.

2. The method of claim 1, further comprising identifying, by the processor, within a first threshold of accuracy that the at least the portion of the new gesture data matches the at least the portion of the gesture data, and

further determining that the movement of the new subject is the first movement based on the identification within the first threshold.

3. The method of claim 2, further comprising:

additionally identifying, by the processor, within a second threshold of greater certainty than the first threshold of certainty that at least a second portion of the new gesture data matches at least a second portion of the gesture data, and

further determines with a greater certainty that the movement of the new subject corresponds to the first movement based on the identification within the second threshold;

wherein the first threshold is used in determining a fixed minimum threshold by averaging the gesture across the stream of frames, and wherein the second threshold is used to provide a separate minimum threshold that is used to provide a recognition value.

4. The method of claim 1, further comprising determining that the movement of the new subject corresponds to the first movement responsive to comparing one or more features of the gesture data of the frame to the one or more features of the new gesture data.

5. The method of claim 1, further comprising:

receiving a plurality of frames comprising gesture data via the network from a plurality of users at different geographical locations,

receiving the frame via the network from a user of the plurality of users,

storing the gesture data comprising the frame in a database, and

retrieving the frame from the database upon detecting that gesture data in the frame substantially matches the new gesture data.

6. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a shoulder of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a shoulder of the subject with respect to the subject'"'"'s waist.

7. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a hip of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a hip of the subject with respect to the subject'"'"'s waist.

8. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of an elbow of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of an elbow of the subject with respect to the subject'"'"'s waist.

9. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a palm of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a palm of the subject with respect to the subject'"'"'s waist.

10. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of one or more fingers of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of one or more fingers of the subject with respect to the subject'"'"'s waist.

11. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a knee of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a knee of the subject with respect to the subject'"'"'s waist.

12. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a heel of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data identifying the position of a heel of the subject with respect to the subject'"'"'s waist.

13. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of toes of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data identifying the position of toes of the subject with respect to the subject'"'"'s waist.

14. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position a portion of the head of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a portion of the head of the subject with respect to the subject'"'"'s waist.

15. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of the neck of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying a position of the neck of the subject with respect to the subject'"'"'s waist.

16. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a pelvis of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a pelvis of the subject with respect to the subject'"'"'s waist.

17. The method of claim 1, further comprising comparing, by the processor, a feature of the new gesture data identifying a position of a belly of the new subject with respect to the new subject'"'"'s waist to a feature of the gesture data in the frame identifying the position of a belly of the subject with respect to the subject'"'"'s waist.

18. The method of claim 1, further comprising receiving, by the network interface from a detector, the frame comprising gesture data, the detector comprising a camera comprising a functionality to extrapolate self-referential gesture data, and

receiving, by the processor, from a different detector the new gesture data, the different detector comprising a different camera that comprises the functionality to extrapolate self-referential gesture data.

26. The method of claim 1, wherein the classifier engine utilizes a radial basis function scalar vector machine.

27. The method of claim 26, wherein the feature space of the radial basis function scalar vector machine has a Hilbert space of infinite dimensions defined as:

k(x_i,x_j)=exp(−

γ

∥

x_i−

x_j∥

²)

for γ

>

0.

28. The method of claim 26, wherein the scalar vector machine is non-linear and utilizes a hyperplane that is defined as:

k(x_i,x_j)=(x_i·

x_j)^d or k(x_i,x_j)=(x_i·

x_j+1)^d.

19. A system for identifying a movement of a subject based on data, the system comprising:

a database storing a stream of frames received at a network interface, each frame of the stream of frames comprising gesture data identifying positions of one or more body parts of a subject with respect to the waist of the subject'"'"'s body;

a processor configured to extrapolate, from the the stream of frames comprising gesture data, one or more frames corresponding to a first movement, wherein the processor is configured to divide the stream of frames into separate phases using a self-organizing map, and wherein the processor is configured to identify, using a scalar vector machine, transition conditions within a feature space between phases;

a classifier configured to assign the one or more frames to a first movement, the classifier configured to apply a scale invariant feature transform to determine a descriptor of the first movement;

the network interface being configured to receive a new gesture data identifying positions of one or more body parts of a new subject with respect to the waist of the new subjects body;

the processor configured to extrapolate from the new gesture data one or more frames comprising gesture data identifying one or more features of the body movement of the new subject;

the processor further configured to determine that a movement of the new subject corresponds to the first movement responsive to comparing at least a portion of the new gesture data to the at least a portion of the new gesture data in the one or more frames of the stream of frames stored in the database.

20. The system of claim 19, wherein the processor is configured to determine within a first threshold of certainty that the movement of the new subject corresponds to the first movement.

21. The system of claim 20, further comprising:

the processor additionally configured to determine within a second threshold of greater certainty than the first threshold, that the movement of the new subject corresponds to the first movement responsive to a comparison of the new gesture data to a second frame assigned to the first movement;

wherein the first threshold is used in determining a fixed minimum threshold by averaging the gesture across the stream of frames, and wherein the second threshold is used to provide a separate minimum threshold that is used to provide a recognition value.

22. The system of claim 19, wherein processor is configured to determine that the movement of the new subject corresponds to the first movement responsive to a comparison of the one or more positions of the frame to the one or more positions of the new gesture data.

23. The system of claim 19, wherein the processor is configured to determine that the movement of the new subject corresponds to the first movement responsive to a comparison of the one or more positions identified by the new gesture data to the one or more positions identified by the gesture data in the frame.

24. The system of claim 19, wherein a plurality of frames comprising the gesture data are received via the network from a plurality of users at different geographical locations, and wherein the frame is received via the network from a user of the plurality of users.

25. The system of claim 19, wherein the network interface is configured to receive from a detector the frame comprising gesture data, the detector comprising a camera comprising a functionality to extrapolate self-referential gesture data, and wherein the processor receives from a different detector the new gesture data, the different detector comprising a different camera that comprises the functionality to extrapolate self-referential gesture data.