Fingertip visual haptic sensor controller

• US 9,024,874 B2
• Filed: 03/07/2008
• Issued: 05/05/2015
• Est. Priority Date: 03/12/2007
• Status: Active Grant

• Current Assignee: University of Pittsburgh of The Commonwealth System of Higher Education
• Original Assignee: University of Pittsburgh of The Commonwealth System of Higher Education
• Inventors: Stetten, George DeWitt, Klatzky, Roberta
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US12/044,138
• Publication Number:
  US 20080226134A1
• Time in Patent Office: 2,615 Days
• Field of Search: 340/4.12, 3404071-4072, 345156-158, 345183-182
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037

Abstract

Systems, methods and devices for capturing environmental image data and providing tactile feedback about the visual environment to a user. One or more image sensing devices are mounted on various appendages of a user and are used to sense the environment. The image data is sent to a computer vision system where it is processed in order to identify specific objects within the environment. When the computer vision system makes such an identification it may send a signal to a haptic stimulator mounted on a user'"'"'s appendage or may be used to control an object in the environment.

Claims

1. A visual sensing and control system, comprising:

a first image sensing device, adapted to conduct measurements of an environment surrounding a user and generate image data based on said measurements;

a first controller associated with said first image sensing device;

a computer vision system in communication with said first image sensing device and first controller, wherein said computer vision system receives image data from said first image sensing device, further wherein said computer vision system is adapted to process said image data to recognize an object based on its texture and three-dimensional shape located at a location in said environment and sends a first signal through said controller to operate the recognized object, the operation occurring at the location of the recognized object; and

a haptic stimulator in communication with computer vision system, wherein said haptic stimulator is adapted to generate a haptic stimulus upon receipt of a second signal from said computer vision system, wherein said computer vision system is adapted to generate said second signal upon recognition of said object.

2. The system according to claim 1, wherein the first image sensing device is a video camera.

35. The system of claim 1, wherein the controller is configured to control movement of the recognized object in three-dimensions.

36. The system of claim 1, wherein the recognized object comprises a mechanical or electrical device, and the controller operates the mechanical or electrical device.

3. A visual sensing and tactile stimulation system, comprising:

a first image sensing device which transmits image data corresponding to three-dimensional objects in an environment surrounding a user, the first image sensing device configured to collect and transmit image data for multiple images of three-dimensional objects corresponding to different locations of the image-sensing device as the image-sensing device is moved in three-dimensional space;

a first haptic stimulator associated with said first image sensing device; and

a computer vision system in communication with said first image sensing device and first haptic stimulator, wherein said computer vision system receives the image data from said first image sensing device, compares the image data to stored data of a predetermined three-dimensional object, and sends a first signal to said first haptic stimulator when said received image data corresponds to the predetermined three-dimensional object,

wherein said haptic stimulator is adapted to provide a haptic stimulus after receipt of said first signal from said computer vision system.

4. The system according to claim 3, further comprising:

a second image sensing device; and

a second haptic stimulator associated with said second image sensing device, wherein said second image sensing device and second haptic stimulator are in communication with said computer vision system.

5. The system according to claim 3, wherein the first image sensing device is a video camera.

6. The system according to claim 3, wherein the first image sensing device is a laser range finder.

7. The system according to claim 3, wherein the first haptic stimulator is a vibratory transducer.

8. The system according to claim 3, wherein the first haptic stimulator is a pneumatic device.

9. The system according to claim 3, further comprising a plurality of navigational beacons.

39. The system of claim 3, wherein the predetermined three-dimensional object is recognized based on its three-dimensional shape.

40. The system of claim 3, wherein the image sensing device comprises a laser configured to illuminate the three-dimensional objects and a phototransister configured to receive reflected light from the three-dimensional objects, wherein the reflected light is produced in response to the illumination.

41. The system of claim 40, wherein the haptic stimulator is attached to the laser and comprises a coil and a plurality of magnets.

10. A method for providing information about the physical structure of a three-dimensional environment of a user, comprising the steps of:

gathering image data about the three-dimensional environment surrounding said user using an image sensing device mounted on an appendage of said user;

communicating the image data to a computer vision system;

processing the image data by the computer vision system to identify features of a non-textual three-dimensional object in the three-dimensional environment;

creating a signal based on said processing step; and

transmitting a signal to a haptic stimulator mounted on said user to provide a haptic representation of the three-dimensional environment, wherein said haptic representation of the three-dimensional environment includes information about said features found in the environment;

wherein the act of gathering image data comprises moving the image-sensing device in three-dimensions, collecting data for multiple images corresponding to different locations of the image-sensing device as it is moved in three-dimensional space;

wherein the act of processing the image data comprises processing the image data of the multiple images, comparing the image data of the multiple images to predefined features of the three-dimensional object in a database to identify the features of the three-dimensional object.

11. A method according to claim 10, wherein the image sensing device and haptic stimulator are mounted on the dorsal aspect of a fingertip.

12. A method according to claim 10, wherein said processing step identifies edges or obstructions of the three-dimensional object in the environment, and further wherein said transmitting step includes transmitting signals to the haptic stimulator when edges or obstructions in the environment are identified.

13. A method according to claim 10, wherein the features in said processing step are selected from said database via voice command.

14. A method according to claim 10, further comprising transmitting a command signal from said computer vision system to a controller associated with the object in the environment.

15. A method according to claim 10, wherein said appendage is a finger and the image sensing device is mounted on the dorsal aspect of said finger.

16. A method according to claim 10, wherein said step of gathering image data includes gathering image data using a plurality of image sensing devices mounted on said user.

17. A method according to claim 16, wherein said step of processing the image data includes the computer vision system identifying a different feature of a different three-dimensional object in the environment for each of said plurality of image sensing devices.

18. The method of claim 17, wherein the different features of the three-dimensional objects identified by the plurality of image sensing devices are located at different physical locations.

19. The method of claim 10, further comprising:

transmitting instructions from the user to the computer vision system to search the environment for a specific three-dimensional object.

20. The method of claim 10, wherein the act of processing the image data comprises recognizing a three-dimensional object based on its shape.

21. The method of claim 10, wherein the act of processing the image data comprises recognizing a piece of furniture or a three-dimensional portion thereof.

22. The method of claim 10, wherein the act of processing the image data comprises recognizing an appliance or a three-dimensional portion thereof.

23. The method of claim 10, wherein the act of processing the image data comprises recognizing a light switch.

24. The method of claim 10, wherein the three-dimensional environment comprises an open, outdoor environment.

25. The method of claim 24, wherein the outdoor environment comprises a surface of a body of water, and wherein the act of processing the image data comprises recognizing a nautical navigational beacon.

26. The method of claim 10, wherein the act of processing the image data comprises analyzing the environment to detect at least one predefined feature which is selected from the database.

27. The method of claim 10, wherein the database houses a plurality of predefined objects associated with the predefined features.

28. The method of claim 15, wherein the act of gathering image data comprises waving the finger in three-dimensions within the environment to recognize the three-dimensional object.

31. The method of claim 15, further comprising moving the finger to operate the three-dimensional object.

32. The method of claim 31, wherein the movement of the finger comprises a downward movement, an upward movement, or a rotational movement.

33. The method of claim 32, wherein the downward or upward movement of the finger comprises bending or pivoting of the finger down or up.

34. The method of claim 32, wherein the rotational movement comprises turning of a hand comprising the finger.

29. A method for providing information about the physical structure of a three-dimensional environment of a user, comprising the steps of:

gathering image data about the three-dimensional environment surrounding said user using an image sensing device mounted on an appendage of said user;

communicating the image data to a computer vision system;

processing the image data by the computer vision system to identify features of a three-dimensional object in the three-dimensional environment;

creating a signal based on said processing step; and

transmitting a signal to a haptic stimulator mounted on said user to provide a haptic representation of the three-dimensional environment, wherein said haptic representation of the three-dimensional environment includes information about said features found in the environment;

wherein said appendage is a finger and the image sensing device is mounted on the dorsal aspect of said finger;

wherein the act of gathering image data comprises waving the finger in three-dimensions within the environment to recognize the three-dimensional object;

wherein the act of gathering image data further comprises waving a second image sensing device mounted on a dorsal aspect of a second finger in three-dimensions within the environment to recognize a second three-dimensional object, wherein the first and second image sensing devices are waved simultaneously.

30. The method of claim 29, wherein the first and second image sensing devices are mounted on different fingers of the same hand, and are waved simultaneously by waving the hand in three-dimensions.

37. A visual sensing and tactile stimulation system, comprising:

a first image sensing device which transmits image data corresponding to a three-dimensional object in an environment surrounding a user;

a first haptic stimulator associated with said first image sensing device; and

a computer vision system in communication with said first image sensing device and first haptic stimulator, wherein said computer vision system receives the image data from said first image sensing device and sends a first signal to said first haptic stimulator when said received image data corresponds to a first recognized three-dimensional object, wherein said haptic stimulator is adapted to provide a haptic stimulus after receipt of said first signal from said computer vision system; and

a second image sensing device which transmits image data corresponding to a second three-dimensional object in the environment surrounding the user, wherein the computer vision system is in communication with the second image sensing device and a second haptic stimulator,

wherein said computer vision system receives image data from said second image sensing device and said first image sensing device simultaneously, and sends a signal to said second haptic stimulator when said received image data corresponds to a second recognized three-dimensional object, and

wherein the second haptic stimulator is adapted to provide a haptic stimulus after receipt of said signal from said computer vision system.

38. The system of claim 37, wherein the first three-dimensional object and the second three-dimensional object are at different locations within the environment.