System and method for storing and playback of information for blind users

• US 9,972,216 B2
• Filed: 03/20/2015
• Issued: 05/15/2018
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Chen, Tiffany L.
• Primary Examiner(s): Jiang, Yong Hang
• Application Number:
  US14/664,234
• Publication Number:
  US 20160275817A1
• Time in Patent Office: 1,152 Days
• Field of Search: 340 412
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: G09B 21/007 using both tactile and audible presentation of the information

Abstract

A system for providing information to a user includes a haptic display including a plurality of sensors positioned thereon, a map positioned on the haptic display and including points of interest where each point of interest is positioned adjacent to or directly on at least one of the plurality of sensors, and a memory for storing a plurality of breadcrumb files where each of the plurality of breadcrumb files corresponds to at least one of the points of interest and contains information related to at least one of an environment, an object, an obstacle, a path or a location. The system includes a processor for determining which point of interest has been selected based on contact with at least one of the points of interest and an output device configured to provide the information contained in the breadcrumb file corresponding to the selected point of interest to the user.

Claims

1. A system for providing information to a blind user, comprising:

a haptic display including a plurality of display sensors positioned thereon;

a map positioned on or superimposed onto the haptic display and including a plurality of points of interest, each point of interest of the plurality of points of interest being positioned adjacent to or directly on at least one of the plurality of display sensors;

a memory configured to store a plurality of media files, each media file of the plurality of media files corresponding to at least one of the plurality of points of interest and including information related to at least one of an environment, an object, an obstacle, a path or a location of the at least one of the plurality of points of interest;

a media sensor configured to detect new media that is at least one of operator-generated media or media captured from the environment and that corresponds to one of the plurality of points of interest or a new point of interest;

an input device configured to receive a media capture request corresponding to a request for the new media to be captured or stored;

a processor coupled to the haptic display, the memory, the media sensor, and the input device and configured to;

update the plurality of media files to include a new media file that includes the new media when the input device receives the media capture request;

control the haptic display to output data indicating which one or more sensor of the plurality of display sensors is associated with at least one stored media file of the plurality of media files, each of the one or more sensor corresponding to a respective point of interest of the plurality of points of interest, and

determine that the respective point of interest has been selected based on contact of the one or more sensor; and

an output device configured to be held or worn by the blind user and configured to output the information included in the stored media file associated with the one or more sensor that corresponds to the respective point of interest.

2. The system of claim 1 wherein the output device is configured to output the information included in the stored media file to the blind user via a combined audio and tactile immersive experience by activating a vibration unit within the output device to generate vibrations while simultaneously reproducing audio from the stored media file.

3. The system of claim 1 wherein each sensor of the plurality of display sensors has an active state and a deactive state and is activated when there is a corresponding media file associated with a particular point of interest and deactivated when there is no corresponding media file associated with the particular point of interest.

4. The system of claim 1 wherein each sensor of the plurality of display sensors is tactile only when there is a corresponding media file associated with a particular point of interest that the user is touching.

5. The system of claim 1 wherein the media sensor is further configured to create or obtain the information related to the at least one of the environment, the object, the obstacle or the location and wherein the information corresponding to at least some of the plurality of media files is provided by the media sensor or by a second media sensor of another user device.

6. The system of claim 5 wherein the media sensor includes at least one of a camera, a computer, a necklace device, a smart phone, a video recorder or a voice recorder capable of creating or obtaining the information.

7. The system of claim 1 further comprising a receiver, coupled to the memory, and configured to receive at least some of the plurality of media files from a cloud storage device.

8. The system of claim 1 wherein the haptic display is a haptic pin display or a haptic vibrating display.

9. The system of claim 1 wherein the haptic display is a haptic pin display including a surface and a plurality of pins each corresponding to one of the plurality of display sensors such that the processor is configured to control the haptic display to output data indicating which one or more sensor is associated with the at least one stored media file by controlling a corresponding pin of each of the one or more sensor that is associated with the at least one stored media file to raise above the surface of the haptic pin display.

10. The system of claim 1 wherein the plurality of display sensors are force sensors that are activated by physical contact by the blind user.

11. A system for providing information to a blind user, comprising:

a haptic device including a plurality of display sensors positioned thereon, a surface, and a plurality of pins each corresponding to one of the plurality of display sensors and capable of being raised above the surface;

a map positioned on or superimposed onto the haptic device and including a plurality of points of interest, each point of interest of the plurality of points of interest corresponding to at least one of the plurality of display sensors;

a memory configured to store a plurality of media files, each media file of the plurality of media files corresponding to at least one of the plurality of points of interest and including information related to at least one of an environment, an object, an obstacle, a path or a location of the at least one of the plurality of points of interest;

a media sensor configured to detect new media that is at least one of operator-generated media or media captured from the environment and that corresponds to one of the plurality of points of interest or a new point of interest;

an input device configured to receive a media capture request corresponding to a request for the new media to be captured or stored;

a processor coupled to the haptic device, the memory, the media sensor, and the input device and configured to;

update the plurality of media files to include a new media file that includes the new media when the input device receives the media capture request,

control the haptic display to raise a pin corresponding to each of the plurality of display sensors that is associated with a stored media file of the plurality of media files, each of the plurality of display sensors that is associated with a stored media file corresponding to a respective point of interest of the plurality of points of interest, and

determine that the respective point of interest has been selected based on contact of a respective sensor; and

a necklace having a speaker and a vibration unit, configured to be worn by the blind user and configured to provide the information included in the stored media file associated with the respective sensor that corresponds to the respective point of interest to the blind user via a combined audio and tactile immersive experience by activating the vibration unit to generate vibrations while simultaneously reproducing audio from the stored media file using the speaker.

12. The system of claim 11 wherein each sensor of the plurality of display sensors has an active state and a deactive state and is activated when there is a corresponding media file associated with a particular point of interest and deactivated when there is no corresponding media file associated with the particular point of interest.

13. The system of claim 11 wherein each sensor of the plurality of display sensors is tactile only when there is a corresponding media file associated with a particular point of interest that the blind user is touching.

14. The system of claim 11 wherein the input device is further configured to create or obtain the information related to the at least one of the environment, the object, the obstacle or the location, and wherein the information corresponding to at least some of the plurality of media files is provided by the media sensor or by a second media sensor of another user device.

15. The system of claim 14 wherein the media sensor includes at least one of a camera, a computer, a necklace device, a smart phone, a video recorder or a voice recorder capable of creating or obtaining the information.

16. The system of claim 11 further comprising a receiver, coupled to the memory, and configured to receive at least some of the plurality of media files from a cloud storage device.

17. The system of claim 11 wherein the haptic device further includes a haptic vibrating feature.

18. The system of claim 11 wherein the plurality of display sensors are force sensors that are activated by physical contact by the blind user.

19. The system of claim 11 wherein each media file of the plurality of media files includes a title to allow the blind user the ability to listen to the title without having to play the entire media file.

20. The system of claim 11 wherein the memory is further configured to store a table with data indicating the relationship between each point of interest of the plurality of points of interest on the map and at least one of the plurality of display sensors on the haptic device.