METHOD AND APPARATUS FOR IDENTIFYING INPUT FEATURES FOR LATER RECOGNITION

• US 20180165304A1
• Filed: 06/01/2017
• Published: 06/14/2018
• Est. Priority Date: 05/25/2012
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Itani, Sleiman, Yang, Allen Yang
• Granted Patent: US 10,331,731 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Disclosed are methods and apparatuses to recognize actors during normal system operation. The method includes defining actor input such as hand gestures, executing and detecting input, and identifying salient features of the actor therein. A model is defined from salient features, and a data set of salient features and/or model are retained, and may be used to identify actors for other inputs. A command such as “unlock” may be executed in response to actor input. Parameters may be applied to further define where, when, how, etc. actor input is executed, such as defining a region for a gesture. The apparatus includes a processor and sensor, the processor defining actor input, identifying salient features, defining a model therefrom, and retaining a data set. A display may also be used to show actor input, a defined region, relevant information, and/or an environment. A stylus or other non-human actor may be used.

Claims

1. A method, comprising:

defining an actor input;

executing said actor input;

detecting said actor input;

identifying at least one salient feature of an actor from said actor input;

defining an actor model using said at least one salient feature; and

retaining a data set comprising at least one of said at least one salient feature or said actor model.

2. The method of claim 1 further comprising:

defining a command to a computer system associated with said actor input; and

executing said command in response to said actor input.

3. The method of claim 1 further comprising:

defining limiting parameters for executing said actor input;

executing said actor input within said parameters; and

detecting said actor input within said parameters.

4. The method of claim 1, wherein:

said actor comprises a human vocal system; and

said actor input comprises human speech.

5. The method of claim 1, wherein:

said actor comprises a human hand; and

said actor input comprises at least one of a hand gesture or a hand posture.

6. The method of claim 1, further comprising:

using said data set to identify other actor inputs.

7. The method of claim 1, further comprising:

establishing an imaging field of view; and

defining a region in said field of view;

wherein;

detecting said actor input comprises receiving at least one image of said field of view, with said actor input therein.

8. The method of claim 7, wherein:

said region is smaller than said field of view, and is substantially fully within said field of view.

9. The method of claim 7, wherein:

said actor comprises an end-effector.

10. The method of claim 7, further comprising:

identifying a plurality of salient features of said actor from said region in said image.

11. The method of claim 7, further comprising:

using said data set to identify other actor inputs.

12. The method of claim 7, further comprising:

establishing first and second imaging fields of view;

defining a first region in said first imaging field of view;

defining a second region in said second imaging field of view corresponding to said first region in said first imaging field of view;

executing said actor input in said first and second regions;

receiving at least one first image of said first field of view, said first image comprising at least a portion of said actor input therein;

receiving at least one second image of said second imaging field of view, said second image comprising at least a portion of said actor input therein;

identifying at least one first salient feature of said actor from said first region in said first image;

identifying at least one second salient feature of said actor from said second region in said second image;

defining the actor model using said at least one first salient feature and said at least one second salient feature; and

retaining the data set comprising at least one of a group consisting of said at least one first salient feature, said at least one second salient feature, and said actor model.

13. The method of claim 8, further comprising:

establishing first and second imaging fields of view;

defining the region in said first imaging field of view;

defining said region in said second imaging field of view;

executing said actor input in said region;

receiving at least one first image of said first imaging field of view, said first image comprising at least a portion of said actor input therein;

receiving at least one second image of said second imaging field of view, said second image comprising at least a portion of said actor input therein; and

identifying at least one salient feature of said actor from said region in said first and second images.

14. The method of claim 12, wherein:

said first and second imaging fields of view form a stereo pair.

15. The method of claim 7, wherein:

said actor input comprises a substantially static posture of an end-effector.

16. The method of claim 7, wherein:

said actor input comprises a dynamic gesture of an end-effector.

17. The method of claim 9, wherein:

said end-effector comprises at least one of a hand, a finger, or a stylus.

18. The method of claim 7, further comprising:

outputting at least a portion of said field of view to a display.

19. The method of claim 18, further comprising:

outputting to said display a graphic occupying at least a portion of said region.

20. The method of claim 19, wherein:

said graphic comprises a virtual object.

21. The method of claim 20, wherein:

said image comprises an unlock target.

22. The method of claim 21, wherein said region is substantially shaped as a silhouette of said actor executing said actor input.

23. The method of claim 22, wherein said region is substantially shaped as an outline of a hand executing said input.

24. The method of claim 21, wherein:

said image comprises unlock instructions.

25. The method of claim 2, wherein:

said command comprises an unlock command.

26. The method of claim 2, comprising:

executing said command only said data set substantially matches a reference data set.

27. The method of claim 7, wherein:

said at least one salient feature comprises at least one of an edge configuration, a shape, a color, a motion, or a convex hull model.

28. The method of claim 7 further comprising:

defining said region as substantially two dimensional.

29. The method of claim 7 further comprising:

defining said region as three dimensional.

30. The method of claim 7 further comprising:

defining a minimum rate of apparent motion;

in identifying said at least one salient feature in said region, ignoring portions of said region exhibiting less than said minimum rate of apparent motion.

31. The method of claim 7, further comprising:

in identifying said at least one salient feature in said region, beginning at a bottom of said region.

32. The method of claim 7, further comprising:

in identifying said at least one salient feature in said region, beginning in a horizontal stripe.

33. The method of claim 7, further comprising:

determining a position of said at least one salient feature in said at least one image;

obtaining a second image of said field of view; and

searching for said at least one salient feature in a second position in said second image, said second position corresponding to said position of said at least one salient feature in said at least one image.

34. The method of claim 1, further comprising:

defining a preliminary data set for said actor, said preliminary data set comprising at least one of an at least one preliminary salient feature of said actor or a preliminary actor model;

defining said actor input such that in executing said actor input, said actor exhibits said at least one preliminary salient feature in said preliminary data set;

wherein;

detecting said actor input comprises identifying said actor using said preliminary data set.

35. The method of claim 34, wherein:

said at least one preliminary salient feature comprises an outline feature of said actor, and said preliminary actor model comprises a shape of said actor executing said actor input.

36. The method of claim 34, wherein:

said at least one salient feature comprises a color feature of said actor, and said actor model comprises a color characterization of said actor.

44. The apparatus of claim 31, wherein:

said sensor, said processor, and said display are disposed on a head mounted display.

37. An apparatus, comprising:

at least one sensor; and

a processor in communication with said sensor;

wherein;

said processor is adapted to define an actor input;

said sensor is adapted to detect said actor input;

said processor is adapted to identify at least one salient feature of an actor from said actor input;

said processor is adapted to define an actor model using said at least one salient feature; and

said processor is adapted to retain a data set comprising at least one of said at least one salient feature or said actor model.

38. The apparatus of claim 37, wherein:

said processor is adapted to use said data set to identify other actor inputs.

39. The apparatus of claim 37, wherein:

said sensor comprises an image sensor.

40. The apparatus of claim 37, wherein:

said sensor comprises an audio sensor.

41. The apparatus of claim 37, further comprising:

a display in communication with said processor;

wherein;

said sensor comprises an image sensor, said image sensor having a field of view;

said processor is adapted to define a region in said field of view, said region being smaller than said field of view;

said image sensor is adapted to obtain at least one image of said field of view with said actor input therein;

said display is adapted to output at least a portion of said field of view;

said processor is adapted to identify at least one salient feature of said actor from said actor input in said region in said at least one image.

42. The apparatus of claim 41 further comprising:

first and second image sensors.

43. The apparatus of claim 42, wherein:

said first and second image sensors are disposed in a stereo configuration.

45. The apparatus of claim 37, wherein:

said actor comprises a stylus.

46. The apparatus of claim 45, wherein:

said stylus comprises at least one indicator;

said sensor is adapted to sense said at least one indicator;

said processor is adapted to identify said at least one indicator as said at least one salient feature.

47. A method, comprising:

establishing an imaging field of view;

defining a region in said imaging field of view;

defining an input comprising at least one of a group consisting of a hand posture and a hand gesture;

outputting said imaging field of view to a display;

outputting to said display a virtual object occupying at least a portion of said region;

executing said input in said region;

obtaining at least one image of at least a portion of said imaging field of view with said input therein;

identifying a plurality of salient features of said hand from said input in said region in said image;

executing an unlock command in response to said input;

defining an actor model using said salient features;

retaining a data set comprising at least one of said salient features or said actor model; and

identifying said hand in other images using said data set.

48. A method, comprising:

establishing an imaging field of view;

defining a region in said imaging field of view, said region having a shape substantially corresponding to a shape of a human hand;

defining a preliminary data set for said hand, said preliminary data set comprising at least one of a plurality of preliminary salient features of said hand or a preliminary model of said hand, said preliminary salient features comprising outline features of said hand, and said preliminary model comprising a shape of said hand executing an input;

defining said input to comprise at least one of a hand posture or a hand gesture, and such that in executing said input said hand exhibits at least one of said preliminary salient features;

outputting said imaging field of view to a display;

outputting to said display a virtual object occupying at least a portion of said region, said virtual object having a shape substantially corresponding to said shape of said human hand;

executing said input in said region;

executing an unlock command in response to said input;

obtaining at least one image of said imaging field of view;

detecting said input, the detecting said input comprising identifying said actor using said preliminary data set;

identifying a plurality of salient features of said hand from said input in said region in said image, said salient features comprising color features of said hand;

defining a model of said hand using said salient features, said model comprising a color characterization of said hand;

retaining a data set comprising at least one of said salient features or said actor model; and

identifying said hand in other images using said data set.

49. An apparatus, comprising:

a body adapted to be worn on a head of a user;

at least one image sensor disposed on said body, said at least one image sensor having a field of view;

a processor in communication with said at least one image sensor and disposed on said body; and

a display in communication with said processor and disposed on said body such that said display is disposed in front of and proximate to at least one of said eyes of the user when said body is worn on said head of the user;

wherein;

said processor is adapted to define a region in said field of view;

said processor is adapted to define a visible input with a hand, said visible input comprising at least one of a hand gesture or a hand posture;

said image sensor is adapted to obtain at least one image of said field of view with said input therein;

said display is adapted to output at least a portion of said field of view;

said display is adapted to output a virtual object occupying at least a portion of said region;

said processor is adapted to identify a plurality of salient features of said hand from said region in said image; and

said processor is adapted to define an actor model using said salient features;

said processor is adapted to retain a data set comprising at least one of said salient features or said actor model; and

said processor is adapted to identify said hand in other images using said data set.

50. An apparatus, comprising:

means for establishing an imaging field of view;

means for defining a region in said imaging field of view;

means for defining an end-effector input;

means for obtaining at least one image of said imaging field of view with said end-effector input therein;

means for identifying at least one salient feature of said end-effector from said region in said image;

means for defining an end-effector model using said at least one salient feature;

means for retaining a data set comprising at least one of said at least one salient feature or said actor model.