Voice tracking camera with speaker identification

• US 9,723,260 B2
• Filed: 05/18/2010
• Issued: 08/01/2017
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (Plantronics, Inc.)
• Inventors: Feng, Jinwei
• Primary Examiner(s): Tsang, Fan
• Assistant Examiner(s): LYTLE, JEFFREY P
• Application Number:
  US12/782,173
• Publication Number:
  US 20110285807A1
• Time in Patent Office: 2,632 Days
• Field of Search: 348 141- 1416
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A videoconferencing apparatus automatically tracks speakers in a room and dynamically switches between a controlled, people-view camera and a fixed, room-view camera. When no one is speaking, the apparatus shows the room view to the far-end. When there is a dominant speaker in the room, the apparatus directs the people-view camera at the dominant speaker and switches from the room-view camera to the people-view camera. When there is a new speaker in the room, the apparatus switches to the room-view camera first, directs the people-view camera at the new speaker, and then switches to the people-view camera directed at the new speaker. When there are two near-end speakers engaged in a conversation, the apparatus tracks and zooms-in the people-view camera so that both speakers are in view.

Claims

1. An automated videoconferencing method for a near-end endpoint located in a near-end environment at a near-end site to conduct a videoconference with one or more far-end endpoints located in one or more far-end environments at one or more far-end sites, the method comprising:

detecting one or more first audio indicative of speech during the videoconference with the near-end endpoint in the near-end environment at the near-end site;

determining one or more first speech frequency characteristics of the one or more first audio, the one or more first speech frequency characteristics characterizing frequency information indicative of the speech detected in the one or more first audio;

determining one or more first source locations of the one or more first audio with the near-end endpoint in the near-end environment at the near-end site;

directing at least one camera of the near-end endpoint at at least one of the one or more first source locations in the near-end environment;

storing the one or more first speech frequency characteristics with information of at least one camera directed at the at least one of the one or more first source locations;

detecting second audio indicative of speech with the near-end endpoint in the near-end environment at the near-end site;

determining a second speech frequency characteristic of the second audio, the second speech frequency characteristic characterizing frequency information indicative of the speech detected in the second audio; and

determining that the second speech frequency characteristic does not match the stored one or more first speech frequency characteristics;

determining a second source location of the second audio with the near-end endpoint in the near-end environment at the near-end site;

directing the at least one camera of the near-end endpoint at the second source location in the near-end environment; and

storing the second speech frequency characteristic with information of the at least one camera directed at the second source location.

2. The method of claim 1, wherein the speech frequency characteristics comprise mel-frequency cepstrum coefficients.

3. The method of claim 1, further comprising obtaining the one or more first speech frequency characteristics for storage by detecting the first audio and determining the one or more first speech frequency characteristics when initiating the videoconference.

4. The method of claim 3, further comprising determining the one or more first source locations for storage with the one or more first speech frequency characteristics obtained when initiating the videoconference.

5. The method of claim 1, further comprising obtaining the one or more first speech frequency characteristics for storage by detecting the first audio and determining the one or more first speech frequency characteristics during the videoconference.

6. The method of claim 5, further comprising determining the one or more first source locations for storage with the one or more first speech frequency characteristics obtained during the videoconference.

7. The method of claim 1, wherein the method comprises associating personal information identifying the one or more near-end participants with the one or more first stored speech frequency characteristics.

8. The method of claim 1, wherein the at least one camera comprises a steerable camera of the near-end endpoint, and wherein directing the at least one camera of the near-end endpoint comprises adjusting one or more of a pan, a tilt, and a zoom of the steerable camera.

9. The method of claim 1, further comprising:

determining a current speech characteristic of current audio indicative of speech detected with the near-end endpoint in the near-end environment;

matching the current speech characteristics to one of the stored first speech characteristics;

obtaining the one of the stored first source locations having the stored first speech characteristic that matches the current speech characteristic; and

directing the at least one camera of the near-end endpoint at the matching stored first source location.

10. The method of claim 1, wherein the at least one camera comprises first and second cameras;

and wherein before directing the at least one camera, the method comprises switching output from first video captured with the first camera to second video captured with the second camera of the near-end endpoint.

11. The method of claim 10, wherein the second video is captured by the second camera in a wide view of the near-end environment.

12. The method of claim 10, wherein after directing the first camera, the method comprises switching output for the videoconference from the second video to third video captured with the first camera.

13. The method of claim 12, wherein the third video is captured by the first camera in a tight view of the first stored source location.

14. The method of claim 1, wherein directing the at least one camera comprises directing the at least one camera at both of the first and second stored source locations for the first and second audio detected.

15. The method of claim 1, wherein directing the at least one camera of the near-end endpoint at the at least one first source location in the near-end environment comprises:

detecting current audio indicative of speech with the near-end endpoint in the near-end environment;

determining a current speech frequency characteristic of the detected audio;

matching the current speech frequency characteristic to at least one of the one or more first stored speech frequency characteristics; and

directing the at least one camera at the at least one first source location associated with the match.

16. The method of claim 1, wherein determining that the second speech frequency characteristic does not match the stored first speech frequency characteristic comprises comparing the second speech frequency characteristic to the stored first speech frequency characteristic.

17. The method of claim 1, wherein determining the one or more first speech frequency characteristics comprises matching one of the one or more first speech frequency characteristics to one of the one or more first stored speech frequency characteristics for a first of the near-end participants.

18. The method of claim 17, wherein directing the at least one camera of the near-end endpoint at the at least one first source location in the near-end environment for the first participant having the matching one of the first stored speech frequency characteristics comprises

obtaining personal information associated with the first participant;

associating the obtained personal information with first video captured with the at least one camera; and

sending the first video with the associated personal information to the one or more far-end endpoints.

19. The method of claim 18, further comprising:

logging a speech statistic during the videoconference for the near-end participant associated with the matching one of the stored speech frequency characteristics during the videoconference; and

making the logged speech statistic available.

20. The method of claim 19, wherein the speech statistic includes one or more of:

a duration that the near-end participant spoke during the videoconference, a number of times that the near-end participant spoke during the videoconference, a time in the videoconference when the near-end participant spoke; and

a section in a recording of the videoconference in which the near-end participant spoke.

21. A non-transitory program storage device having program instructions stored thereon for causing a programmable control device to perform an automated videoconferencing method for a near-end endpoint located in a near-end environment at a near-end site conducting a videoconference with one or more far-end endpoints located in one or more far-end environments at one or more far-end sites, the method comprising:

detecting one or more first audio indicative of speech during the videoconference with the near-end endpoint in the near-end environment at the near-end site;

determining one or more first speech frequency characteristics of the one or more first audio, the one or more first speech frequency characteristics characterizing frequency information indicative of the speech detected in the one or more first audio;

determining one or more first source locations of the one or more first audio with the near-end endpoint in the near-end environment at the near-end site;

directing at least one camera of the near-end endpoint at at least one of the one or more first source locations in the near-end environment;

storing the one or more first speech frequency characteristics with information of the at least one camera directed at the at least one of the one or more first source locations;

detecting second audio indicative of speech with the near-end endpoint in the near-end environment at the near-end site;

determining a second speech frequency characteristic of the second audio, the second speech frequency characteristic characterizing frequency information indicative of the speech detected in the second audio; and

determining that the second speech frequency characteristic does not match the stored one or more first speech frequency characteristics;

determining a second source location of the second audio with the near-end endpoint in the near-end environment at the near-end site;

directing the at least one camera of the near-end endpoint at the second source location in the near-end environment; and

storing the second speech frequency characteristic with information of the at least one camera directed at the second source location.

22. A videoconferencing apparatus located in a near-end environment at a near-end site and operatively coupling in a videoconference with one or more far-end endpoints located in one or more far-end environments at one or more far-end sites, the apparatus comprising:

at least one camera being controllable to pan, tilt, and zoom in the near-end environment at the near-end site;

one or more microphones for capturing near-end audio in the near-end environment at the near-end site;

memory storing speech frequency characteristics for near-end participants located in the near-end environment at the near-end site and storing source locations in the near-end environment of the near participants associated with the stored speech frequency characteristics; and

a processing unit operatively coupled to the at least one camera, the one or more microphones, and the memory, the processing unit being operable to;

detect one or more first audio indicative of speech during the videoconference with the near-end endpoint in the near-end environment at the near-end site;

determine one or more first speech frequency characteristics of the one or more first audio, the one or more first speech frequency characteristics characterizing frequency information indicative of the speech detected in the one or more first audio;

determine one or more first source locations of the one or more first audio with the near-end endpoint in the near-end environment at the near-end site;

direct at least one camera of the near-end endpoint at at least one of the one or more first source locations in the near-end environment;

store the one or more first speech frequency characteristics with information of the at least one camera directed at the at least one of the one or more first source locations;

detect second audio indicative of speech with the near-end endpoint in the near-end environment at the near-end site;

determine a second speech frequency characteristic of the second audio, the second speech frequency characteristic characterizing frequency information indicative of the speech detected in the second audio; and

determine that the second speech frequency characteristic does not match the stored one or more first speech frequency characteristics;

determine a second source location of the second audio with the near-end endpoint in the near-end environment at the near-end site;

direct the at least one camera of the near-end endpoint at the second source location in the near-end environment; and

store the second speech frequency characteristic with information of the at least one camera directed at the second source location.