Acquisition of a user expression and an environment of the expression

• US 8,787,706 B2
• Filed: 03/31/2005
• Issued: 07/22/2014
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: The Invention Science Fund I, L.L.C. (Intellectual Ventures LLC)
• Original Assignee: The Invention Science Fund I, L.L.C. (Intellectual Ventures LLC)
• Inventors: Cohen, Alexander J., Jung, Edward K. Y., Levien, Royce A., Lord, Robert W., Malamud, Mark A., Rinaldo, John D. Jr.
• Primary Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US11/097,980
• Publication Number:
  US 20060209017A1
• Time in Patent Office: 3,400 Days
• Field of Search: 235/472.01, 235/462.01, 235/462.07, 235/375, 713/186
• US Class Current: 382/321
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability

Abstract

Embodiments include an apparatus, device, system, and method. In an embodiment, a human wearable apparatus is provided. The apparatus includes an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item and an annotating device operable to generate information indicative of a user expression associated with the recognizable aspect of the item.

Claims

1. A human wearable apparatus, the apparatus comprising:

a) an annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item while at least one user'"'"'s eye gazes at the item, the annotation environment capture device being further operable to generate information indicative of content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and

b) an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item.

2. The apparatus of claim 1, wherein the annotation environment capture device includes at least a head mountable annotation environment device.

3. The apparatus of claim 1, wherein the annotating device includes at least a head mountable annotating device.

4. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes;

an annotation environment capture device operable to capture at least one parameter of an environment that includes at least the machine-recognizable aspect of an item.

5. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine recognizable aspect of an item includes:

an annotation environment capture device operable to capture an optical representation of an environment that includes at least the machine-recognizable aspect of an item.

6. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to capture an audio representation of an environment that includes at least the machine-recognizable aspect of an item.

7. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a machine-recognizable radio frequency identification signal corresponding to an aspect of an item.

8. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of an optically-recognizable aspect of an item.

9. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a human recognizable aspect of an item.

10. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable native text of an item.

11. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable visual aspect of an item.

12. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable image aspect of an item.

13. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable shape aspect of an item.

14. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable dimensional aspect of an item.

15. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable human readable content.

16. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item in response to an input received from the user.

17. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

an annotation environment capture device operable to generate information indicative of a recognizable aspect of an item in response to an input responsive to a gesture by the user.

18. The apparatus of claim 1, wherein the annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item includes:

a module operable to recognize data indicative of the recognizable aspect of an item.

19. The apparatus of claim 18, wherein the module operable to recognize data indicative of the machine-recognizable aspect of an item includes:

an artificial intelligence module operable to recognize data indicative of the recognizable aspect of an item.

20. The apparatus of claim 18, wherein the module operable to recognize data indicative of the machine-recognizable aspect of an item includes:

a pattern recognition module operable to recognize data indicative of the recognizable aspect of an item.

21. The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item, wherein the user expression includes at least one of a hand formed user expression, a verbal user expression, a preformed user expression, or a gestural user expression.

22. The apparatus of claim 1, further comprising:

c) a communications device.

23. The apparatus of claim 1, further comprising:

c) a storage device.

49. The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises:

an annotating device operable to generate information indicative of a visual handwritten expression associated with the machine-recognizable aspect of the item.

50. The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises:

an annotating device operable to generate information indicative of a spatial handwritten expression associated with the machine-recognizable aspect of the item.

51. The apparatus of claim 1, wherein the annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item comprises:

an annotating device operable to generate information indicative of an audible user expression associated with the machine-recognizable aspect of the item.

52. The apparatus of claim 1, wherein the content portion includes an incidental element, wherein the incidental element includes at least an indication of a torn page.

53. The apparatus of claim 1, wherein the content portion includes an incidental element, the incidental element includes at least an indication of a stain correlated with the content portion.

54. The apparatus of claim 1, wherein the information indicative of the content portion includes at least contextual information, wherein the contextual information is acquired from another source.

55. The apparatus of claim 1, wherein at least a portion of the contextual information comprises:

contextual information acquired by ultrasound.

24. A head mountable apparatus, the apparatus comprising:

a) an annotation environment capture device operable to generate information indicative of a machine-recognizable aspect of an item while at least one user'"'"'s eye gazes at the item, the annotation environment capture device being further operable to generate information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in each of the at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and

b) an annotating device operable to generate information indicative of a user expression associated with the machine-recognizable aspect of the item.

25. The apparatus of claim 24, further comprising:

c) a communications device.

26. The apparatus of claim 24, further comprising:

c) a storage device.

27. A non-transitory computer-readable medium having a set of instructions for a computer-implemented method, the set of instructions comprising:

a) one or more instructions for generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item while at least one user'"'"'s eye gazes at the item, wherein the generating the first information as responsive to a user direction to capture the machine-recognizable aspect of the item;

b) one or more instructions for generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item; and

c) one or more instructions for generating in the portable apparatus information indicative of a content portion of an item, the generation of the information indicative of content existing in at least two planes and including at least detection of detection of at least a portion of content in at least two planes substantially simultaneously the information indicative of the content portion including at least contextual information, wherein at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion.

28. The computer-readable medium of claim 27, wherein the one or more instructions for generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item includes:

one or more instructions for generating in a portable head mountable apparatus a first information indicative of the machine-recognizable aspect of an item.

29. The computer-readable medium of claim 28, wherein the one or more instructions for generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

one or more instructions for generating in the portable head mountable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item.

30. The computer-readable medium of claim 27, further comprising:

c) one or more instructions for saving at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

31. The computer-readable medium of claim 27, further comprising:

c) one or more instructions for transmitting at least one of the first information indicative of a machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

32. A wearable apparatus, the comprising:

a) a context module operable to generate a first information indicative of a machine-recognizable aspect of an item while at least one user'"'"'s eye gazes at the item, the context module being further operable to generate information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion includes at least contextual information, at least a portion of the contextual information being acquired by ultrasound and being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion; and

b) an annotation module operable to generate a second information indicative of a user expression associated with the machine-recognizable aspect of the item.

33. The apparatus of claim 32, further comprising:

c) a communications module operable to transmit at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

34. The apparatus of claim 32, further comprising:

c) a storage module operable to save at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

35. A method comprising:

a) generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item while at least one user'"'"'s eye gazes at the item;

b) generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item, wherein the user expression is recognized in response to the portable apparatus recognizing the machine-recognizable aspect of the item; and

c) generating in the portable apparatus information indicative of a content portion of an item, the generating of the information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in each of the at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, wherein at least a portion of the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion.

36. The method of claim 35, wherein generating in a portable apparatus a first information indicative of a machine-recognizable aspect of an item includes:

generating in a portable head mountable apparatus a first information indicative of the machine-recognizable aspect of an item.

37. The method of claim 36, wherein generating in the portable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

generating in the portable head mountable apparatus a second information indicative of a user expression associated with the machine-recognizable aspect of the item.

38. The method of claim 35, further comprising:

c) saving at least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

39. The method of claim 35, further comprising:

c) transmitting least one of the first information indicative of the machine-recognizable aspect of an item and the second information indicative of a user expression associated with the machine-recognizable aspect of the item.

40. A computer-implemented method comprising:

a) tracking a gaze of at least one user'"'"'s eye;

b) acquiring an image of a machine-recognizable aspect of an item corresponding to the gaze of the at least one user'"'"'s eye;

c) generating a first information indicative of the machine-recognizable aspect of an item corresponding to the gaze of the at least one user'"'"'s eye while a user views the item;

d) generating a second information indicative of a user expression associated with the machine-recognizable aspect of an item corresponding to the gaze of the at least one user'"'"'s eye; and

e) generating information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion.

41. The method of claim 40, further comprising:

e) receiving a signal indicative of the user expression associated with the machine-recognizable aspect of the item.

42. The method of claim 40, wherein acquiring an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user'"'"'s eye includes:

acquiring an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user'"'"'s eye in response to a received user input.

43. The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

generating a second information indicative of a user verbal expression associated with the machine-recognizable aspect of the item.

44. The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

generating a second information indicative of a user hand formed expression associated with the machine-recognizable aspect of the item.

45. The method of claim 40, wherein generating a second information indicative of a user expression associated with the machine-recognizable aspect of the item includes:

generating a second information indicative of a user gesture associated with the machine-recognizable aspect of the item.

46. The method of claim 40, wherein generating information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion includes:

generating, in a head mountable apparatus, information indicative of a content portion of an item, the generation of information being indicative of content existing in at least two planes and including at least detection of at least a portion of content in at least two planes substantially simultaneously, the information indicative of the content portion including at least contextual information, the contextual information being associated with a handwriting movement on at least one of the at least two planes proximate to the content portion.

47. A head mountable apparatus, the apparatus comprising:

a) a tracking module operable to track a gaze of at least one user'"'"'s eye;

b) a sensor module operable to acquire an image of a machine-recognizable aspect of the item corresponding to the gaze of the at least one user'"'"'s eye;

c) a context module operable to generate a first information indicative of the machine-recognizable aspect of an item corresponding to the gaze of the at least one user'"'"'s eye while a user views the item;

d) an annotation module operable to generate a second information indicative of a user expression associated with the machine-recognizable aspect of the item; and

e) wherein the context module is further operable to generate information indicative of a content portion of an item, at least a portion of the information being indicative of content existing in at least two planes and including at least detection of at least a portion of the content in at least two planes substantially simultaneously, and wherein the content portion includes at least an incidental element configured to aid in distinguishing the item, wherein the incidental element is not readable by a human, the information indicative of the content portion includes at least contextual information associated with a handwriting movement on at least one of the at least two planes proximate to the content portion.

48. The apparatus of claim 47, further comprising:

a receiver module operable to receive a signal indicative of the user expression associated with the machine-recognizable aspect of the item.