NOVEL MAN MACHINE INTERFACES AND APPLICATIONS

• US 20020036617A1
• Filed: 08/21/1998
• Published: 03/28/2002
• Est. Priority Date: 08/21/1998
• Status: Abandoned Application

• Current Assignee: Timothy R. Pryor
• Original Assignee: Timothy R. Pryor
• Inventors: PRYOR, TIMOTHY R.
• Application Number:
  US09/138,339
• Publication Number:
  US 20020036617A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The invention is aimed at providing affordable methods and apparatus for inputting position, attitude(orientation) or other object characteristic data to computers for the purpose of Computer Aided learning, Teaching, Gaming, Toys, Simulations, Aids to the disabled, Word Processing and other applications.

Preferred embodiments of the invention utilize electro-optical sensors, and particularly TV Cameras, providing optically inputted data from specialized datum'"'"'s on objects and/or natural features of objects. Objects can be both static and in motion, from which individual datum positions and movements can be derived, also with respect to other objects both fixed and moving. Real-time photogrammetry is preferably used to determine relation ships of portions of one or more datums with respect to a plurality of cameras or a single camera processed by a conventional PC.

Claims

1. Apparatus for input by a person of data to a computer having a display comprising

One or more Datum means provided on said person, said datum means distinguishable in reflected light

At least one TV Camera having an output

Means for determining from said TV camera output, the position of said datums and/or the orientation of a portion of said person

Means for creating on said display, a representation of at least one object, and;

Means for modifying, manipulating, or positioning said at least one object representation on said screen as a function of the position or orientation of datums or person

2. Apparatus according to claim 1 further including light source means for directing light at said member

3. Apparatus according to claim 1 wherein at least one of said datums is retroreflective

4. Apparatus according to claim 1 wherein at least one of said datums is a natural feature of said member

5. Apparatus according to claim 2 wherein said light source is an LED light source

6. Apparatus according to claim 2 wherein light from said light source is substantially invisible

7. Apparatus according to claim 1 wherein at least one of said datums is distinctive in color

8. Apparatus according to claim 1 wherein at least one of said datums is a distinctive shape

9. Apparatus according to claim 1 wherein at least two cameras are used

10. Apparatus according to claim 9 wherein said cameras provide stereo pair of images of said object

11. Apparatus according to claim 9 wherein said cameras look at different sides of said person

12. Apparatus according to claim 9 wherein said cameras look at different times at said person

13. Apparatus according to claim 1 wherein said cameras are provided with the display

14. Apparatus according to claim 1 including further means of affixing a datum

15. Apparatus according to claim 1 including further voice input means to said computer

16. Apparatus according to claim 1 including further means to allow said camera to see objects associated with said person

17. Apparatus according to claim 1 including bandpass filter means associated with at least one of said cameras

18. A method by which a person may input data to a computer, the method comprising:

providing a target on said person

providing a source of light to create an illumination field;

providing at least one TV camera proximate said light source such that the camera can detect reflection of light from said object in said illumination field

detecting radiation reflected from said within the illumination field to create at least one tv image containing an image of said person

determining from said tv image information concerning the position and/or orientation of said target, and

providing a desired input to said computer using said determined information

22. A Method for input of information by a person to a computer having a display representing at least one object comprising the steps of

Providing a datum associated with said person

Electro-optically determining, the position of at least one datum on said person in 3 dimensions

Providing a representation of at least one computer generated virtual object on said display, and

Using said determined position or orientation data, manipulating said object displayed by said computer to provide a desired visual display or audio response

19. A method according to claim 20 wherein said member contains at least one retroreflective datum

20. A method according to claim 20 wherein said light source is an LED light source

21. A method according to claim 20 wherein said Light source is substantially invisible

23. A method according to claim 24 wherein at least one of said datums is retroreflective

24. A method according to claim 24 wherein said datum is distinctive in color

25. A method according to claim 24 wherein said datum is a distinctive shape

26. A method according to claim 24 wherein at least two cameras are used

27. A method according to claim 24 wherein said cameras provide stereo pair of images of said datum

28. A method according to claim 24 wherein said cameras look at different sides of said datum

29. A method according to claim 24 wherein said cameras are provided with said display

30. A method according to claim 24 including further step of affixing a datum

31. A method according to claim 24 wherein at least one of said datums is a natural object feature

32. A method according to claim 24 including the further step of recognizing voice input

33. A method according to claim 24 including temporary filter means for at least one lens of said cameras

34. A method according to claim 24 including the further step of sensing the gray level image of a portion of said user.

35. A method according to claim 24 including the further step of changing Sound output as a function of said data

36. A method according to claim 24 including the further step of using said display or audio for learning

37. A method according to claim 24 including the further step of analyzing movement of said datum

38. A method according to claim 24 including the further step of determining the position or orientation of a member

40. Apparatus according to claim 39, wherein said decoration means is retroreflective

41. Apparatus according to claim 39, wherein said decoration is selected from a group comprising rings, bracelets, watches, lipstick, nail polish,

42. Apparatus according to claim 39, wherein said decoration is part of clothing

39. Means for aiding the determination of locations of points on a human, comprising

means providing decoration for said human, said means easily visible by a TV camera or other electro-optical device, and

Means for temporarily providing said decoration means on said human

43. A Method for producing a display based experience for a user comprising the steps of;

Providing a computer

Providing a large screen TV display of size greater than 42 inches diagonal, the display being controlled by said computer

Providing at least one electro-optical sensor having an output

Processing in said computer said sensor output

From said processing, determining the position or orientation of a portion of a person and/or object camera, and using said computer,

Modifying said display to create a response to an action of said person.

44. A method according to claim 44 wherein said display is approximately lifesize.

45. A method according to claim 44 wherein said user touches or points at virtual objects depicted on said display

46. A method according to claim 44 wherein said user pinches, or grips virtual objects depicted on said display

47. A method according to claim 44 wherein said display varies as the users view changes

53. A method according to claim 48 wherein said special datum is easily affixed by hand

54. A method according to claim 48 wherein said special datum is retroreflective

55. A method according to claim 48 wherein said special datum is linear

56. A method according to claim 48 wherein said special datum is curvilinear

58. A method according to claim 57 wherein said member is movable by said human

59. A method according to claim 57 wherein said member moves as a result of the action of a physical variable

60. A method according to claim 57 including the additional step of determining the position or orientation of a portion of said human

61. A method according to claim 57 Wherein said sensor is comprised of at least one TV camera

62. A method according to claim 57, wherein said position or motion is determined relative to another member or said object

48. Method for activity involving an object, comprising the steps of

49. Providing an object

50. Determining if features can be sensed by a tv camera

51. Affixing special datums to said object where features are required for best sensing results,

52. Recording the locations of features and special datums into a data base.

57. A method of providing a game or other human activity comprising

Providing an object

Providing a member attached to said object and movable with respect thereto

Determining the position or orientation, or change therein, of said member with an electro-optical sensing system

From said determined position or orientation, or change therein, determining an input parameter to a computer program, and

Using said program, provide said game or other activity