Conference support device, conference support method, and computer-readable medium storing conference support program

• US 8,560,315 B2
• Filed: 03/12/2010
• Issued: 10/15/2013
• Est. Priority Date: 03/27/2009
• Status: Active Grant

• Current Assignee: Brother Kogyo Kabushiki Kaisha
• Original Assignee: Brother Kogyo Kabushiki Kaisha
• Inventors: Yasoshima, Mizuho
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US12/659,570
• Publication Number:
  US 20100250252A1
• Time in Patent Office: 1,313 Days
• Field of Search: 704/231, 704/235, 704/243, 704/246, 704/258, 704/260, 704/261, 704/270, 704/270.1, 704/275
• US Class Current: 704/246
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/234336 by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A conference support device includes an image receiving portion that receives captured images from conference terminals, a voice receiving portion that receives, from one of the conference terminals, a voice that is generated by a first participant, a first storage portion that stores the captured images and the voice, a voice recognition portion that recognizes the voice, a text data creation portion that creates text data that express the words that are included in the voice, an addressee specification portion that specifies a second participant, whom the voice is addressing, an image creation portion that creates a display image that is configured from the captured images and in which the text data are associated with the first participant and a specified image is associated with at least one of the first participant and the second participant, and a transmission portion that transmits the display image to the conference terminals.

Claims

1. A conference support device that, by controlling communication among a plurality of conference terminals, is configured to support a video conference that is conducted among conference participants who are using the conference terminals, the conference support device comprising:

a memory configured to store computer-readable instructions; and

a processor that is configured to execute the computer-readable instructions to;

receive, from the plurality of the conference terminals, captured images that are captured by image capture devices of the conference terminals and in each of which at least one of the conference participants is visible;

receive, from a first conference terminal that is one of the plurality of the conference terminals, a voice that is generated by a first participant, the first participant being one of the conference participants and being using the first conference terminal, the voice being input from a voice input device of the first conference terminal;

identify words that are included in the received voice by voice recognition processing;

create text data that express the identified words;

specify a second participant who is a different conference participant from the first participant based on a result of the voice recognition processing, the second participant corresponding to the identified words and being at least one of the conference participants;

create a display image that is to be displayed on display devices of the plurality of the conference terminals from the captured images, and in which the text data are associated with a first portion of the captured image that corresponds to the first participant and a specified image is associated with a second portion of the captured image that corresponds to the second participant, the specified image being an image that indicate that the second participant is addressed by the first participant; and

transmit the created display image to the plurality of the conference terminals, in order for the display image to be displayed on the display devices of the plurality of the conference terminals.

2. The conference support device according to claim 1, wherein

the specified image is a blank display frame in which text will be displayed, and

the creating the display image includes creating the display image by associating the blank display frame with the second portion of the captured image that corresponds to the second participant.

3. The conference support device according to claim 1, wherein

the specifying the second participant includes, in a case where the identified words by the voice recognition processing include a first specified word, specifying all of the conference participants except the first participant as the second participant, and

the creating the display image includes, in a case where all of the conference participants except the first participant are specified as the second participant, creating the display image by associating the specified image with the second portion of the captured image that correspond to the second participant.

4. The conference support device according to claim 1, wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant;

and

the creating the display image includes specifying the first portion of the captured image that corresponds to the first participant, based on a participant image that is associated with the information that specifies the first participant, among participant images that are stored in a storage portion, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively, and associating the text data with the specified first portion of the captured image.

5. The conference support device according to claim 1, wherein

the processor is further configured to execute the computer-readable instructions to:

select, as a second participant image, from among participant images that are stored in a storage portion, one of the participant images that is associated with information that specifies the second participant, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively,

wherein

the creating the display image includes;

associating the text data with the first portion of the captured image that corresponds to the first participant, and

associating the selected second participant image, as the specified image, with the first portion of the captured image that corresponds to the first participant.

6. The conference support device according to claim 5,

wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant, and

the creating the display image includes:

specifying the first portion of the captured image that corresponds to the first participant, based on one of the participant images that is associated with the information that specifies the first participant, among the participant images that are stored in the storage portion, and

associating the text data and the selected second participant image with the specified first portion of the captured image.

7. The conference support device according to claim 1, wherein

the specifying the second participant includes:

determining whether a silent state has been continued for a specified time immediately after a second specified word was spoken, and

specifying, in a case where the silent state has been continued for the specified time, the second participant based on the second specified word.

8. A conference support method that, by controlling communication among a plurality of conference terminals, is configured to support a video conference that is conducted among conference participants who are using the conference terminals, the conference support method comprising:

receiving, from the plurality of the conference terminals, captured images that are captured by image capture devices of the conference terminals and in each of which at least one of the conference participants is visible;

receiving, from a first conference terminal that is one of the plurality of the conference terminals, a voice that is generated by a first participant, the first participant being one of the conference participants and being using the first conference terminal, the voice being input from a voice input device of the first conference terminal;

identifying words that are included in the received voice by voice recognition processing;

creating text data that express the identified words;

specifying a second participant who is a different conference participant from the first participant based on a result of voice recognition processing, the second participant corresponding to the identified words and being at least one of the conference participants;

creating a display image that is to be displayed on display devices of the plurality of the conference terminals from the captured images that have been received, and in which the text data that have been created are associated with a first portion of the captured image that corresponds to the first participant and a specified image is associated with a second portion of the captured image that corresponds to the second participant, the specified image being an image that indicates that the second participant is addressed by the first participant; and

transmitting the created display image to the plurality of the conference terminals, in order for the display image to be displayed on the display devices of the plurality of the conference terminals.

9. The conference support method according to claim 8, wherein

the specified image is a blank display frame in which text will be displayed, and

the creating the display image includes creating the display image by associating the blank display frame with the second portion of the captured image that corresponds to the second participant.

10. The conference support method according to claim 8, wherein

the specifying the second participant includes, in a case where the identified words by the voice recognition processing include a first specified word, specifying all of the conference participants except the first participant as the second participant, and

the creating the display image includes, in a case where all of the conference participants except the first participant are specified as the second participant, creating the display image by associating the specified image with the second portion of the captured image that correspond to the second participant.

11. The conference support method according to claim 8, wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant;

and

the creating the display image includes;

specifying the first portion of the captured image that corresponds to the first participant, based on a participant image that is associated with the information that specifies the first participant, among participant images that are stored in a storage portion, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively, and

associating the text data with the specified first portion of the captured image.

12. The conference support method according to claim 8, further comprising:

selecting, as a second participant image, from among participant images that are stored in a storage portion, one of the participant images that is associated with information that specifies the second participant, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively,

wherein the creating the display image includes;

associating the text data with the first portion of the captured image that corresponds to the first participant, and

associating the selected second participant image, as the specified image, with the first portion of the captured image that corresponds to the first participant.

13. The conference support method according to claim 12, wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant, and

the creating the display image includes:

specifying the first portion of the captured image that corresponds to the first participant, based on one of the participant images that is associated with the information that specifies the first participant, among the participant images that are stored in the storage portion, and

associating the text data and the selected second participant image with the specified first portion of the captured image.

14. A non-transitory computer-readable medium that stores a conference support program for a conference support device that, by controlling communication among a plurality of conference terminals, is configured to support a video conference that is conducted among conference participants who are using the conference terminals, the program comprising instructions that cause a computer of the conference support device to perform:

receiving, from the plurality of the conference terminals, captured images that are captured by image capture devices of the conference terminals and in each of which at least one of the conference participants is visible;

receiving, from a first conference terminal that is one of the plurality of the conference terminals, a voice that is generated by a first participant, the first participant being one of the conference participants and being using the first conference terminal, the voice being input from a voice input device of the first conference terminal;

identifying words that are included in the received voice by voice recognition processing;

creating text data that express the identified words;

specifying a second participant who is a different conference participant from the first participant based on a result of voice recognition processing, the second participant corresponding to the identified words and being at least one of the conference participants;

creating a display image that is to be displayed on display devices of the plurality of the conference terminals from the captured images that have been received, and in which the text data that have been created are associated with a first portion of the captured image that corresponds to the first participant and a specified image is associated with a second portion of the captured image that corresponds to the second participant, the specified image being an image that indicates that the second participant is addressed by the first participant; and

transmitting the created display image to the plurality of the conference terminals, in order for the display image to be displayed on the display devices of the plurality of the conference terminals.

15. The non-transitory computer-readable medium according to claim 14, wherein

the specified image is a blank display frame in which text will be displayed, and

the creating the display image includes creating the display image by associating the blank display frame with the second portion of the captured image that corresponds to the second participant.

16. The non-transitory computer-readable medium according to claim 14, wherein

the specifying the second participant includes, in a case where the identified words by the voice recognition processing include a first specified word, specifying all of the conference participants except the first participant as the second participant, and

the creating the display image includes, in a case where all of the conference participants except the first participant are specified as the second participant, creating the display image by associating the specified image with the second portion of the captured image that correspond to the second participant.

17. The non-transitory computer-readable medium according to claim 14, wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant;

and

the creating the display image includes;

specifying the first portion of the captured image that corresponds to the first participant, based on a participant image that is associated with the information that specifies the first participant, among participant images that are stored in a storage portion, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively, and

associating the text data with the specified first portion of the captured image.

18. The non-transitory computer-readable medium according to claim 14, wherein

the instructions further cause the computer of the conference support device to perform:

selecting, as a second participant image, from among participant images that are stored in a storage portion, one of the participant images that is associated with information that specifies the second participant, the participant images being images of the conference participants and being associated with information that specifies the conference participants, respectively,

wherein the creating the display image includes;

associating the text data with the first portion of the captured image that corresponds to the first participant, and

associating the selected second participant image, as the specified image, with the first portion of the captured image that corresponds to the first participant.

19. The non-transitory computer-readable medium according to claim 18, wherein

the receiving the voice includes receiving, along with the voice that is generated by the first participant, information that specifies the first participant, and

the creating the display image includes:

specifying the first portion of the captured image that corresponds to the first participant, based on one of the participant images that is associated with the information that specifies the first participant, among the participant images that are stored in the storage portion, and

associating the text data and the selected second participant image with the specified first portion of the captured image.

20. The non-transitory computer-readable medium according to claim 14, wherein

the specifying the second participant includes:

determining whether a silent state has been continued for a specified time immediately after a second specified word was spoken, and

specifying, in a case where the silent state has been continued for the specified time, the second participant based on the second specified word.