Vision-assist devices and methods of detecting a classification of an object

• US 10,037,712 B2
• Filed: 01/30/2015
• Issued: 07/31/2018
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Dayal, Rajiv
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Dang, Philip
• Application Number:
  US14/609,874
• Publication Number:
  US 20160225286A1
• Time in Patent Office: 1,278 Days
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 21/006 using audible presentation of the information
• CPC Class Codes: H04N 23/51 Housings

Abstract

Vision-assist devices and methods are disclosed. In one embodiment, a vision-assist device includes an image sensor for generating image data corresponding to a scene, a processor, and an auditory device. The processor is programmed to receive the image data from the image sensor, perform object recognition on the image data to determine a classification of a detected object that is present within the scene, and determine a confidence value with respect to the classification of the detected object. The confidence value is based on a confidence that the classification of the detected object matches an actual classification of the detected object. The processor further generates an auditory signal based on the confidence value. The audio device receives the auditory signal from the processor and produces an auditory message from the auditory signal. The auditory message is indicative of the classification of the detected object and the confidence value.

Claims

1. A vision-assist device configured to be worn by a user, comprising:

at least one image sensor for generating image data corresponding to a scene;

a processor, wherein the processor is programmed to;

receive the image data from the at least one image sensor;

perform object recognition on the image data to determine a classification of a detected object that is present within the scene;

determine a confidence value with respect to the classification of the detected object, wherein the confidence value is based on a confidence that the classification of the detected object matches an actual classification of the detected object; and

generate an auditory signal based on the confidence value; and

an audio device for receiving the auditory signal from the processor and for producing an auditory speech message configured to be heard by the user from the auditory signal, wherein the auditory speech message is indicative of the classification of the detected object and a qualifying statement associated with a sub-increment comprising the confidence value.

2. The vision-assist device of claim 1, wherein:

the confidence value is a percentage based on the confidence that the classification of the detected object matches the actual classification of the detected object; and

the auditory speech message includes the percentage of the confidence value.

3. The vision-assist device of claim 1, wherein:

the confidence value is selected from a confidence value range that ranges from a minimum confidence value to a maximum confidence value on a confidence value scale;

the confidence value range includes a plurality of sub-increments from the minimum confidence value to the maximum confidence value, each sub-increment of the plurality of sub-increments spanning a plurality of confidence values on the confidence value scale;

each sub-increment of the plurality of sub-increments is associated with an auditory speech description representing a degree of confidence; and

the auditory signal is generated by selecting a selected sub-increment of the plurality of sub-increments that includes the selected confidence value, and incorporating the auditory speech description associated with the selected sub-increment into the auditory signal such that the auditory speech message comprises an audible speech message to the user of the auditory description associated with the selected sub-increment.

4. The vision-assist device of claim 3, wherein the processor is further programmed to:

receive a user feedback signal in response to the auditory speech message; and

adjust the plurality of sub-increments of the confidence value range based on the user feedback signal.

5. The vision-assist device of claim 1, wherein the at least one image sensor comprises a first image sensor and a second image sensor.

6. The vision-assist device of claim 1, further comprising an eyeglass frame, wherein the at least one image sensor is coupled to the eyeglass frame.

7. The vision-assist device of claim 6, further comprising an earpiece configured to be inserted in an ear of the user, wherein the audio device is disposed within the earpiece.

8. The vision-assist device of claim 6, wherein the processor is disposed within the eyeglass frame.

9. The vision-assist device of claim 1, further comprising a housing, wherein the housing comprises a neck portion, a first chest portion, and a second chest portion.

10. The vision-assist device of claim 9, wherein the at least one image sensor is disposed within the first chest portion and/or the second chest portion.

11. The vision-assist device of claim 1, wherein the processor is further programmed to:

receive a user feedback signal in response to the auditory speech message, the user feedback signal corresponding to the user disregarding the auditory speech message associated with the detected object; and

for each future detected object having a same classification as the classification of the detected object associated with the user feedback signal disregarding the auditory speech message, generate an auditory signal only when a confidence value is greater than the confidence value associated with the auditory speech message that was disregarded.

12. The vision-assist device of claim 1, wherein the processor is further programmed to:

receive a user feedback signal in response to the auditory speech message, the user feedback signal corresponding to the user confirming that the classification of the detected object associated with the auditory speech message is the actual classification of the detected object; and

in response to receiving the user feedback signal, store, in a non-transitory computer-readable medium, the image data representing the detected object and the classification.

20. The vision-assist device of claim 1, wherein the scene is a user-navigable environment surrounding the user, the detected object comprises one of a person and a feature present within the user-navigable environment, and the auditory speech message further comprises distance information indicative of a distance between the user and the detected object.

21. The vision-assist device of claim 20, wherein the feature present within the scene comprises at least one a trashcan, a sign indicative of a location, an item of furniture, and a staircase.

13. A method of detecting a classification of an object comprising:

receiving image data of a scene from at least one image sensor configured to be worn by a user;

determining from the image data, by a processor, a classification of a detected object that is present within the scene;

determining a confidence value with respect to the classification of the detected object, wherein the confidence value is based on a confidence that the classification of the detected object matches an actual classification of the detected object; and

producing an auditory speech message configured to be heard by the user that is indicative of the classification of the detected object and a qualifying statement associated with a sub-increment comprising the confidence value.

14. The method of claim 13, wherein:

the confidence value is a percentage based on the confidence that the classification of the detected object matches the actual classification of the detected object; and

the auditory speech message includes the percentage of the confidence value.

15. The method of claim 13, wherein:

the confidence value is selected from a confidence value range that ranges from a minimum confidence value to a maximum confidence value on a confidence value scale;

the confidence value range includes a plurality of sub-increments from the minimum confidence value to the maximum confidence value, each sub-increment of the plurality of sub-increments spanning a plurality of confidence values on the confidence value scale;

each sub-increment of the plurality of sub-increments is associated with an auditory description; and

the method further comprises selecting a selected sub-increment of the plurality of sub-increments that includes the selected confidence value; and

the auditory speech message includes the auditory description associated with the selected sub-increment.

16. The method of claim 15, further comprising:

receiving a user feedback signal in response to the auditory speech message; and

adjusting the plurality of sub-increments of the confidence value range based on the user feedback signal.

17. The method of claim 13, further comprising:

receiving a user feedback signal in response to the auditory speech message, the user feedback signal corresponding to the user disregarding the auditory speech message associated with the detected object; and

for each future detected object having a same classification as the classification of the detected object associated with the user feedback signal disregarding the auditory speech message, generating an auditory signal only when a confidence value is greater than the confidence value associated with the auditory speech message that was disregarded.

18. The method of claim 13, further comprising:

receiving a user feedback signal in response to the auditory speech message, the user feedback signal corresponding to the user confirming that the classification of the detected object associated with the auditory speech message is the actual classification of the detected object; and

in response to receiving the user feedback signal, storing, in a non-transitory computer-readable medium, the image data representing the detected object and the classification.

19. A vision-assist device configured to be worn by a user, the vision-assist device comprising:

at least one image sensor for generating image data corresponding to a scene;

a processor, wherein the processor is programmed to;

receive the image data from the at least one image sensor;

perform object recognition on the image data to determine a classification of a detected object that is present within the scene;

determine a confidence value with respect to the classification of the detected object, wherein the confidence value is based on a confidence that the classification of the detected object matches an actual classification of the detected object and the confidence value is selected from a confidence value range on a confidence value scale including a plurality of sub-increments respectively spanning a plurality of confidence values on the confidence value scale and respectively associated with an audio speech description comprising a qualifying statement; and

generate an auditory signal based on the confidence value and a sub-increment comprising the confidence value; and

an audio device for receiving the auditory signal from the processor and for producing an auditory speech message configured to be heard by the user from the auditory signal, wherein the auditory speech message is an audible voice message configured to be heard by the user and is indicative of the classification of the detected object, the confidence value, and the qualifying statement associated with the sub-increment comprising the confidence value;

wherein the processor of the vision-assist device is programmed to store in a memory communicatively coupled to the processor at least one of the classification of the detected object, the confidence value of the detected object, and image data of the detected object based on positive user feedback to the auditory speech message.