Information processing device, information processing method, and program

• US 10,712,998 B2
• Filed: 06/30/2016
• Issued: 07/14/2020
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Yasuda, Ryouhei
• Primary Examiner(s): Monikang, George C
• Application Number:
  US15/759,628
• Publication Number:
  US 20180260187A1
• Time in Patent Office: 1,475 Days
• Field of Search: 381 56- 57, 381312, 381 74, 381104, 381110
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04M 11/06 Simultaneous speech and data transmission, e.g. telegraphic transmission over the same conductors
• CPC Class Codes: H04R 1/1083 Reduction of ambient noise active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/304 For headphones

Abstract

There is provided an information processing device to improve communication between a user and a person speaking to the user by specifying speaking motion information indicating a motion of a surrounding person speaking to the user for whom information from the surroundings is auditorily or visually restricted, the information processing device including: a detecting unit configured to detect a speaking motion of a surrounding person speaking to a user using a device that auditorily or visually restricts information from surroundings; and a specifying unit configured to specify speaking motion information indicating the speaking motion on a basis of monitored surrounding information in a case in which the speaking motion is detected.

Claims

1. An information processing device, comprising:

a detecting unit configured to detect each speaking motion of a surrounding person speaking near a user using a wearable display device that auditorily and visually restricts information from surroundings;

a specifying unit configured to specify sound information of the surrounding person corresponding to each speaking motion based on monitored surrounding information in a case in which the speaking motion is detected;

a history generating unit configured to generate speaking history information based on the monitored surrounding information over time; and

an internal notification control unit configured to control the wearable display device to

display a single text message corresponding to a continuously received piece of the specified sound information according to a first operation, and

display a list of the speaking history information including a plurality of text messages corresponding to a plurality of pieces of the specified sound information according to a second operation,

wherein the detecting unit, the specifying unit, the history generating unit, and the internal notification control unit are each implemented via at least one processor.

2. The information processing device according to claim 1,

wherein the specifying unit is further configured to specify image information based on the monitored surrounding information in the case in which the speaking motion is detected.

3. The information processing device according to claim 1,

wherein the internal notification control unit is further configured to control a notification to the user based on the monitored surrounding information.

4. The information processing device according to claim 3,

wherein the internal notification control unit is further configured to control the notification to the user by performing time shift reproduction on the sound information.

5. The information processing device according to claim 4,

wherein the internal notification control unit controls the notification to the user by gradually reducing a delay time caused by the time shift reproduction and performing switching to a real time sound output.

6. The information processing device according to claim 3,

wherein the internal notification control unit is further configured to acquire position information of the surrounding person corresponding to the speaking motion based on the monitored surrounding information, and

wherein the internal notification control unit controls the notification to the user based on the acquired position information together with the speaking motion information.

7. The information processing device according to claim 6,

wherein the internal notification control unit controls the notification to the user by causing the sound information to be output based on the acquired position information of the surrounding person.

8. The information processing device according to claim 6,

wherein the internal notification control unit controls the notification to the user by causing the wearable display device to display a map based on the acquired position information of the surrounding person.

9. The information processing device according to claim 3,

wherein the internal notification control unit controls the notification to the user by decreasing a volume of content sound being used by the user and then causing the sound information to be output.

10. The information processing device according to claim 3,

wherein the wearable display device includes a headphone device or an earphone device having a noise cancellation function, and

wherein the internal notification control unit controls the notification to the user by suppressing or stopping the noise cancellation function and then causing the sound information to be output.

11. The information processing device according to claim 3,

wherein the internal notification control unit controls the notification to the user by stopping content being used by the user and then causing the notification to be given to the user.

12. The information processing device according to claim 11,

wherein the internal notification control unit controls the notification to the user by causing a notice operation to be performed before stopping the content.

13. The information processing device according to claim 3,

wherein the internal notification control unit controls the notification to the user by causing image information including an image of the surrounding person to be superimposed and displayed on a content image being used by the user.

14. The information processing device according to claim 3, further comprising:

an external notification control unit configured to cause an external notification to be given to the surrounding person indicating a permission state of the notification control to the user,

wherein the external notification control unit is implemented via at least one processor.

15. The information processing device according to claim 14,

wherein the external notification control unit causes the external notification indicating the permission state of the notification control to the user to be given to the surrounding person only when it is detected that the surrounding person is facing in a direction of the user.

16. The information processing device according to claim 14,

wherein the external notification control unit causes the external notification to indicate at least one of a recording state of the sound information or a recording state of image information associated with the speaking motion to be given to the surrounding person.

17. The information processing device according to claim 1,

wherein the speaking history information includes information of at least one of a speaking time of each surrounding person, content used by the user at a time of speaking, an identification of each surrounding person who makes the speaking motion, utterance content of each surrounding person at the time of speaking, or a prohibition state of the notification control to the user.

18. The information processing device according to claim 1,

wherein the specifying unit is further configured to specify position information of the surrounding person corresponding to the speaking motion based on the monitored surrounding information in the case in which the speaking motion is detected, and

wherein the specified position information of the surrounding person comprises a distance between the surrounding person and the user.

19. The information processing device according to claim 1, further comprising:

a storage unit configured to store preset message data,

wherein the specifying unit is further configured to specify position information of the surrounding person corresponding to the speaking motion based on the monitored surrounding information in the case in which the speaking motion is detected, and

wherein the internal notification control unit controls the wearable display device to display the text message according to the specified position information and the stored preset message data.

20. The information processing device according to claim 1,

wherein the specifying unit is further configured to convert a voice recognition result of an utterance included in the sound information into text, and

wherein the internal notification control unit controls the wearable display device to display the text message by indicating the text corresponding to the utterance.

21. The information processing device according to claim 1,

wherein the speaking history information further includes an identification of the surrounding person corresponding to each piece of the plurality of pieces of the specified sound information.

22. The information processing device according to claim 21,

wherein the speaking history information further includes at least one of a speaking time of each surrounding person or content used by the user at a time of speaking.

23. An information processing method, executed via at least one processor, the method comprising:

detecting each speaking motion of a surrounding person speaking near a user using a wearable display device that auditorily and visually restricts information from surroundings;

specifying sound information of the surrounding person corresponding to each speaking motion based on monitored surrounding information in a case in which the speaking motion is detected;

generating speaking history information based on the monitored surrounding information over time; and

controlling the wearable display device to

display a single text message corresponding to a continuously received piece of the specified sound information according to a first operation, and

display a list of the speaking history information including a plurality of text messages corresponding to a plurality of pieces of the specified sound information according to a second operation.

24. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising:

detecting a speaking motion of a surrounding person speaking near a user using a wearable display device that auditorily and visually restricts information from surroundings;

specifying sound information of the surrounding person corresponding to the speaking motion based on monitored surrounding information in a case in which the speaking motion is detected;

generating speaking history information based on the monitored surrounding information over time; and

controlling the wearable display device to

display a single text message corresponding to a continuously received piece of the specified sound information according to a first operation, and

display a list of the speaking history information including a plurality of text messages corresponding to a plurality of pieces of the specified sound information according to a second operation.