Method and apparatus for identifying input features for later recognition

• US 9,747,306 B2
• Filed: 05/23/2013
• Issued: 08/29/2017
• Est. Priority Date: 05/25/2012
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Itani, Sleiman, Yang, Allen Yang
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/901,402
• Publication Number:
  US 20130336528A1
• Time in Patent Office: 1,559 Days
• Field of Search: None
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Disclosed are methods and apparatuses to recognize actors during normal system operation. The method includes defining actor input such as hand gestures, executing and detecting input, and identifying salient features of the actor therein. A model is defined from salient features, and a data set of salient features and/or model are retained, and may be used to identify actors for other inputs. A command such as “unlock” may be executed in response to actor input. Parameters may be applied to further define where, when, how, etc. actor input is executed, such as defining a region for a gesture. The apparatus includes a processor and sensor, the processor defining actor input, identifying salient features, defining a model therefrom, and retaining a data set. A display may also be used to show actor input, a defined region, relevant information, and/or an environment. A stylus or other non-human actor may be used.

Claims

1. A method, comprising:

defining an actor input for an actor;

defining a command associated with said actor input

defining a region for said actor input;

detecting said actor input in said region;

executing said command in response to said actor input;

identifying at least one structural geometric salient feature of said actor from said actor input associated with said executed command;

defining a structural geometric actor model of said actor from said at least one structural geometric salient feature;

retaining a data set comprising at least one of said at least one structural geometric salient feature or said structural geometric actor model; and

using said data set to identify subsequent actor inputs.

2. The method of claim 1 further comprising:

defining limiting parameters for executing said actor input;

detecting said actor input within said parameters.

3. The method of claim 1, wherein:

said actor comprises a human vocal system; and

said actor input comprises human speech.

4. The method of claim 1, wherein:

said actor comprises a human hand; and

said actor input comprises at least one of a hand gesture or a hand posture.

5. The method of claim 1, further comprising:

establishing an imaging field of view; and

defining a said region in said field of view;

wherein;

detecting said actor input comprises receiving at least one image of said field of view, with said actor input therein and in said region.

6. The method of claim 5, wherein:

said region is smaller than said field of view, and is substantially fully within said field of view.

7. The method of claim 6, further comprising:

establishing first and second imaging fields of view;

defining said region in said first imaging field of view;

defining said region in said second imaging field of view;

executing said actor input in said region;

receiving at least one first image of said first imaging field of view, said first image comprising at least a portion of said actor input therein;

receiving at least one second image of said second imaging field of view, said second image comprising at least a portion of said actor input therein; and

identifying at least one structural geometric salient feature of said actor from said actor input in said region in said first and second images.

8. The method of claim 5, wherein:

said actor comprises an end-effector.

9. The method of claim 8, wherein:

said end-effector comprises at least one of a hand, a finger, or a stylus.

10. The method of claim 5, further comprising:

identifying a plurality of structural geometric salient features of said actor from said region in said image.

11. The method of claim 5, further comprising:

establishing first and second imaging fields of view;

defining a first region in said first imaging field of view;

defining a second region in said second imaging field of view corresponding to said first region in said first imaging field of view;

detecting said actor input in said first and second regions;

receiving at least one first image of said first field of view, said first image comprising at least a portion of said actor input therein;

receiving at least one second image of said second imaging field of view, said second image comprising at least a portion of said actor input therein;

identifying at least one first structural geometric salient feature of said actor from said first region in said first image;

identifying at least one second structural geometric salient feature of said actor from said second region in said second image;

defining said structural geometric actor model of said actor from said at least one first structural geometric salient feature and said at least one structural geometric second salient feature; and

retaining said data set comprising at least one of said at least one first salient feature, said at least one second salient feature, or said actor model.

12. The method of claim 11, wherein:

said first and second imaging fields of view form a stereo pair.

13. The method of claim 5, wherein:

said actor input comprises a substantially static posture of an end-effector.

14. The method of claim 5, wherein:

said actor input comprises a dynamic gesture of an end-effector.

15. The method of claim 5, further comprising:

outputting at least a portion of said field of view to a display.

16. The method of claim 15, further comprising:

outputting to said display a graphic occupying at least a portion of said region.

17. The method of claim 16, wherein:

said graphic comprises a virtual object.

18. The method of claim 17, wherein:

said image comprises an unlock target.

19. The method of claim 18, wherein said region is substantially shaped as a silhouette of said actor executing said actor input.

20. The method of claim 19, wherein said region is substantially shaped as an outline of a hand executing said input.

21. The method of claim 18, wherein:

said image comprises unlock instructions.

22. The method of claim 5 further comprising:

defining said region as substantially two dimensional.

23. The method of claim 5 further comprising:

defining said region as three dimensional.

24. The method of claim 5 further comprising:

defining a minimum rate of apparent motion;

identifying said at least one structural geometric salient feature in said region, ignoring portions of said region exhibiting less than said minimum rate of apparent motion.

25. The method of claim 5, further comprising:

identifying said at least one structural geometric salient feature in said region, beginning at a bottom of said region.

26. The method of claim 5, further comprising:

identifying said at least one structural geometric salient feature in said region, beginning in a horizontal stripe.

27. The method of claim 5, further comprising:

determining a position of said at least one structural geometric salient feature in said at least one image;

obtaining a second image of said field of view; and

searching for said at least one structural geometric salient feature in a second position in said second image, said second position corresponding to said position of said at least one structural geometric salient feature in said at least one image.

28. The method of claim 1, wherein:

said command comprises an unlock command.

29. The method of claim 1, comprising:

executing said command only if said data set substantially matches a reference data set.

30. An apparatus, comprising:

at least one sensor; and

a processor in communication with said sensor;

wherein;

said processor is adapted to define an actor input and a command associate with said actor input;

said sensor is adapted to detect said actor input;

said processor is adapted to execute said command in response to said input;

said processor is adapted to identify at least one structural geometric salient feature of an actor from said actor input associated with said executed command;

said processor is adapted to define a structural geometric actor model of said actor from said at least one structural geometric salient feature;

said processor is adapted to retain a data set comprising at least one of said at least one structural geometric salient feature or said structural geometric actor model; and

said processor is adapted to use said data set to identify subsequent actor inputs.

31. The apparatus of claim 30, wherein:

said sensor comprises an image sensor.

32. The apparatus of claim 30, wherein:

said sensor comprises an audio sensor.

33. The apparatus of claim 30, further comprising:

a display in communication with said processor;

wherein;

said sensor comprises an image sensor, said image sensor having a field of view;

said processor is adapted to define a region in said field of view, said region being smaller than said field of view;

said image sensor is adapted to obtain at least one image of said field of view with said actor input therein and said actor input in said region;

said display is adapted to output at least a portion of said field of view;

said processor is adapted to identify at least one structural geometric salient feature of said actor from said actor input in said region in said at least one image.

34. The apparatus of claim 33 further comprising:

first and second image sensors.

35. The apparatus of claim 34, wherein:

said first and second image sensors are disposed in a stereo configuration.

36. The apparatus of claim 33, wherein:

said sensor, said processor, and said display are disposed on a head mounted display.

37. The apparatus of claim 30, wherein:

said actor comprises a stylus.

38. The apparatus of claim 37, wherein:

said stylus comprises at least one indicator;

said sensor is adapted to sense said at least one indicator;

said processor is adapted to identify said at least one indicator as said at least one structural geometric salient feature.

39. A method, comprising:

establishing an imaging field of view;

defining a region in said imaging field of view;

defining an input comprising at least one of a group consisting of a hand posture and a hand gesture;

defining an unlock command;

outputting said imaging field of view to a display;

outputting to said display a virtual object occupying at least a portion of said region;

obtaining at least one image of at least a portion of said imaging field of view with said input therein and said input in said region;

identifying a plurality of structural geometric salient features of said hand from said input associated with said executed command in said region in said image;

executing said unlock command in response to said input;

defining a structural geometric hand model of said hand from said structural geometric salient features;

retaining a data set comprising at least one of said structural geometric salient features or said structural geometric hand model; and

identifying said hand in subsequent images using said data set.

40. An apparatus, comprising:

a body adapted to be worn on a head of a user;

at least one image sensor disposed on said body, said at least one image sensor having a field of view;

a processor in communication with said at least one image sensor and disposed on said body; and

a display in communication with said processor and disposed on said body such that said display is disposed in front of and proximate to at least one of said eyes of the user when said body is worn on said head of the user;

wherein;

said processor is adapted to define a region in said field of view;

said processor is adapted to define a visible input with a hand, said visible input comprising at least one of a hand gesture or a hand posture;

said processor is adapted to define a command associated with said visible input;

said image sensor is adapted to obtain at least one image of said field of view with said input therein and said visible input in said region;

said processor is adapted to execute said command in response to said input;

said display is adapted to output at least a portion of said field of view;

said display is adapted to output a virtual object occupying at least a portion of said region;

said processor is adapted to identify a plurality of structural geometric salient features of said hand from said visible input associated with said executed command in said region in said image; and

said processor is adapted to define a structural geometric hand model of said hand from said structural geometric salient features;

said processor is adapted to retain a data set comprising at least one of said structural geometric salient features or said structural geometric and model; and

said processor is adapted to identify said hand in subsequent images using said data set.

41. An apparatus, comprising:

means for establishing an imaging field of view;

means for defining a region in said imaging field of view for an end-effector input;

means for defining said end-effector input;

means for defining a command associated with said end-effector input;

means for obtaining at least one image of said imaging field of view with said end-effector input therein and said end-effector input in said region;

means for executing said command in response to said end-effector input;

means for identifying at least one structural geometric salient feature of said end-effector from said input associated with said executed command in said region in said image;

means for defining a structural geometric end-effector model of said end-effector with said at least one structural geometric salient feature;

means for retaining a data set comprising at least one of said at least one structural geometric salient feature or said structural geometric end-effector model.