Multiple-detection gesture recognition

• US 10,496,879 B2
• Filed: 08/25/2017
• Issued: 12/03/2019
• Est. Priority Date: 08/25/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Park, Edwin Chongwoo
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/687,432
• Publication Number:
  US 20190065836A1
• Time in Patent Office: 830 Days
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204

Abstract

Systems and methods are disclosed for recognizing a multi-component gesture within a sequence of image frames. A method comprises selecting a first object model as the selected object model, wherein the first object model corresponds to a first component of the multi-component gesture, receiving a frame in the sequence of frames and analyzing the frame to determine whether the frame includes a first object that matches the selected first object model, selecting a second object model as the selected object model, wherein the second object model corresponds to a second component of the multi-component gesture, receiving at least one subsequent frame and analyzing the at least one subsequent frame to determine whether the at least one subsequent frame includes a second object that matches the selected second object model, and recognizing that the multi-component gesture is present in the sequence of image frames.

Claims

1. A method for recognizing a multi-component gesture within a sequence of image frames, the method comprising:

selecting a first object model as a selected object model, wherein the first object model corresponds to a first component of the multi-component gesture;

in response to selecting the first object model as the selected object model, receiving a frame in the sequence of image frames and analyzing the frame to determine whether the frame includes a first object that matches the selected first object model;

in response to a determination that the frame includes the first object that matches the selected first object model, selecting a second object model as the selected object model, wherein the second object model corresponds to a second component of the multi-component gesture;

in response to selecting the second object model as the selected object model, receiving at least one subsequent frame in the sequence of image frames and analyzing the at least one subsequent frame to determine whether the at least one subsequent frame includes a second object that matches the selected second object model; and

in response to a determination that the at least one subsequent frame includes the second object that matches the selected second object model, recognizing that the multi-component gesture is present in the sequence of image frames.

2. The method of claim 1, further comprising:

in response to the determination that the frame includes the first object that matches the selected first object model, estimating one or more characteristics of the first object, wherein the one or more characteristics of the first object include a first object size and/or a first object position.

3. The method of claim 2, further comprising:

identifying, based on the one or more characteristics of the first object, a portion of the at least one subsequent frame to be analyzed, wherein the identified portion is smaller than the at least one subsequent frame and corresponds to the first object size and/or the first object position;

wherein the analyzing of the at least one subsequent frame comprises analyzing the identified portion of the at least one subsequent frame to determine whether the identified portion of the at least one subsequent frame includes the second object that matches the selected second object model.

4. The method of claim 2, further comprising:

in response to the determination that the at least one subsequent frame includes the second object that matches the selected second object model, estimating one or more characteristics of the second object; and

comparing the one or more characteristics of the second object to the one or more characteristics of the first object to determine whether a difference between the one or more characteristics of the first object and the one or more characteristics of the second object exceeds a difference threshold.

5. The method of claim 4, wherein the recognizing that the multi-component gesture is present in the sequence of image frames is performed further in response to a determination that a difference between the one or more characteristics of the first object and the one or more characteristics of the second object does not exceed the difference threshold.

6. The method of claim 5, wherein:

the one or more characteristics of the second object include a second object size and/or a second object position; and

the determining that the difference does not exceed the difference threshold comprises;

determining whether a difference between the first object size and the second object size does not exceed the difference threshold; and

/or

determining whether the difference between the first object position and the second object position does not exceed the difference threshold.

7. The method of claim 1, wherein the analyzing of the at least one subsequent frame comprises:

analyzing a predetermined number of frames in the sequence of image frames that are immediately subsequent to the frame; and

/or

analyzing all frames in the sequence of image frames that are captured within a predetermined amount of time immediately subsequent to the frame.

8. The method of claim 7, further comprising:

determining that none of the analyzed frames includes a second object that matches the selected second object model; and

in response to the determination that none of the analyzed frames includes a second object that matches the selected second object model, reselecting the first object model as the selected object model.

9. The method of claim 1, further comprising:

in response to a determination that the frame does not include the first object that matches the selected first object model, selecting a next frame in the sequence of image frames as the frame and repeating the analyzing of the frame to determine whether the frame includes a first object that matches the selected first object model.

10. The method of claim 1, wherein the analyzing of the frame and the analyzing of the at least one subsequent frame is performed using hardware acceleration.

11. An apparatus configured to recognize a multi-component gesture within a sequence of image frames, the apparatus comprising:

a memory system, to store related data and instructions; and

a processing system coupled to the memory system, the processing system being configured to;

select a first object model as a selected object model, wherein the first object model corresponds to a first component of the multi-component gesture;

in response to the selection of the first object model as the selected object model, receive a frame in the sequence of image frames and analyze the frame to determine whether the frame includes a first object that matches the selected first object model;

in response to a determination that the frame includes the first object that matches the selected first object model, select a second object model as the selected object model, wherein the second object model corresponds to a second component of the multi-component gesture;

in response to the selection of the second object model as the selected object model, receive at least one subsequent frame in the sequence of image frames and analyze the at least one subsequent frame to determine whether the at least one subsequent frame includes a second object that matches the selected second object model; and

in response to a determination that the at least one subsequent frame includes the second object that matches the selected second object model, recognize that the multi-component gesture is present in the sequence of image frames.

12. The apparatus of claim 11, wherein the processing system is further configured to estimate one or more characteristics of the first object in response to the determination that the frame includes the first object that matches the selected first object model, wherein the one or more characteristics of the first object include a first object size and/or a first object position.

13. The apparatus of claim 12, wherein the processing system is further configured to:

identify, based on the one or more characteristics of the first object, a portion of the at least one subsequent frame to be analyzed, wherein the identified portion is smaller than the at least one subsequent frame and corresponds to the first object size and/or the first object position;

wherein to analyze the at least one subsequent frame, the processing system is further configured to analyze the identified portion of the at least one subsequent frame to determine whether the identified portion of the at least one subsequent frame includes the second object that matches the selected second object model.

14. The apparatus of claim 12, wherein the processing system is further configured to:

estimate one or more characteristics of the second object in response to the determination that the at least one subsequent frame includes the second object that matches the selected second object model; and

compare the one or more characteristics of the second object to the one or more characteristics of the first object to determine whether a difference between the one or more characteristics of the first object and the one or more characteristics of the second object exceeds a difference threshold.

15. The apparatus of claim 14, wherein the processing system is further configured to:

recognize that the multi-component gesture is present in the sequence of image frames further in response to a determination that a difference between the one or more characteristics of the first object and the one or more characteristics of the second object does not exceed the difference threshold.

16. The apparatus of claim 15, wherein:

the one or more characteristics of the second object include a second object size and/or a second object position; and

to determine that the difference does not exceed the difference threshold, the processing system is further configured to;

determine whether a difference between the first object size and the second object size does not exceed the difference threshold; and

/or

determine whether a difference between the first object position and the second object position does not exceed the difference threshold.

17. The apparatus of claim 11, wherein to analyze the at least one subsequent frame, the processing system is further configured to:

analyze a predetermined number of frames in the sequence of image frames that are immediately subsequent to the frame; and

/or

analyze all frames in the sequence of image frames that are captured within a predetermined amount of time immediately subsequent to the frame.

18. The apparatus of claim 17, wherein the processing system is further configured to:

determine that none of the analyzed frames includes a second object that matches the selected second object model; and

in response to the determination that none of the analyzed frames includes a second object that matches the selected second object model, reselect the first object model as the selected object model.

19. The apparatus of claim 11, wherein the processing system is further configured to:

in response to a determination that the frame does not include the first object that matches the selected first object model, select a next frame in the sequence of image frames as the frame and repeat the analyzing of the frame to determine whether the frame includes the first object that matches the selected first object model.

20. The apparatus of claim 11, wherein to analyze the at least one subsequent frame, the processing system is further configured to:

analyze the at least one subsequent frame using hardware acceleration.

21. An apparatus configured to recognize a multi-component gesture within a sequence of image frames, the apparatus comprising:

means for selecting a first object model as a selected object model, wherein the first object model corresponds to a first component of the multi-component gesture;

means for receiving a frame in the sequence of image frames in response to the selection of the first object model as the select object model and analyzing the frame to determine whether the frame includes a first object that matches the selected first object model;

means for selecting a second object model as the selected object model, wherein the second object model corresponds to a second component of the multi-component gesture, in response to a determination that the frame includes the first object that matches the selected first object model;

means for receiving at least one subsequent frame in the sequence of image frames in response to the selection of the second object model as the selected object model and analyzing the at least one subsequent frame to determine whether the at least one subsequent frame includes a second object that matches the selected second object model; and

means for recognizing that the multi-component gesture is present in the sequence of image frames in response to a determination that the at least one subsequent frame includes the second object that matches the selected second object model.

22. The apparatus of claim 21, further comprising:

means for estimating one or more characteristics of the first object in response to the determination that the frame includes the first object that matches the selected first object model, wherein the one or more characteristics of the first object include a first object size and/or a first object position.

23. The apparatus of claim 22, further comprising:

means for identifying, based on the one or more characteristics of the first object, a portion of the at least one subsequent frame to be analyzed, wherein the identified portion is smaller than the at least one subsequent frame and corresponds to the first object size and/or the first object position;

wherein the means for receiving and analyzing the at least one subsequent frame comprises means for analyzing the identified portion of the at least one subsequent frame to determine whether the identified portion of the at least one subsequent frame includes the second object that matches the selected second object model.

24. The apparatus of claim 22, further comprising:

means for estimating one or more characteristics of the second object in response to the determination that the at least one subsequent frame includes the second object that matches the selected second object model; and

means for comparing the one or more characteristics of the second object to the one or more characteristics of the first object to determine whether a difference between the one or more characteristics of the first object and the one or more characteristics of the second object exceeds a difference threshold.

25. The apparatus of claim 21, wherein the means for analyzing the at least one subsequent frame comprises:

means for analyzing a predetermined number of frames in the sequence of image frames that are immediately subsequent to the frame; and

/or

means for analyzing all frames in the sequence of image frames that are captured within a predetermined amount of time immediately subsequent to the frame.

26. A non-transitory computer-readable medium comprising code, which, when executed by a processor, causes the processor to perform operations for recognizing a multi-component gesture within a sequence of image frames, the non-transitory computer-readable medium comprising:

code for selecting a first object model as a selected object model, wherein the first object model corresponds to a first component of the multi-component gesture;

code for receiving a frame in the sequence of image frames in response to the selection of the first object model as the selected object model and analyzing the frame to determine whether the frame includes a first object that matches the selected first object model;

code for selecting a second object model as the selected object model, wherein the second object model corresponds to a second component of the multi-component gesture, in response to a determination that the frame includes the first object that matches the selected first object model;

code for receiving at least one subsequent frame in the sequence of image frames in response to the selection of the second object model as the selected object model and analyzing the at least one subsequent frame to determine whether the at least one subsequent frame includes a second object that matches the selected second object model; and

code for recognizing that the multi-component gesture is present in the sequence of image frames in response to a determination that the at least one subsequent frame includes the second object that matches the selected second object model.

27. The non-transitory computer-readable medium of claim 26, further comprising:

code for estimating one or more characteristics of the first object in response to the determination that the frame includes the first object that matches the selected first object model, wherein the one or more characteristics of the first object include a first object size and/or a first object position.

28. The non-transitory computer-readable medium of claim 26, further comprising:

code for identifying, based on the one or more characteristics of the first object, a portion of the at least one subsequent frame to be analyzed, wherein the identified portion is smaller than the at least one subsequent frame and corresponds to the first object size and/or the first object position;

wherein the code for receiving and analyzing the at least one subsequent frame comprises code for analyzing the identified portion of the at least one subsequent frame to determine whether the identified portion of the at least one subsequent frame includes the second object that matches the selected second object model.

29. The non-transitory computer-readable medium of claim 27, further comprising:

code for estimating one or more characteristics of the second object in response to the determination that the at least one subsequent frame includes the second object that matches the selected second object model; and

code for comparing the one or more characteristics of the second object to the one or more characteristics of the first object to determine whether a difference between the one or more characteristics of the first object and the one or more characteristics of the second object exceeds a difference threshold.

30. The non-transitory computer-readable medium of claim 26, wherein the code for analyzing the at least one subsequent frame comprises:

code for analyzing a predetermined number of frames in the sequence of image frames that are immediately subsequent to the frame; and

/or

code for analyzing all frames in the sequence of image frames that are captured within a predetermined amount of time immediately subsequent to the frame.