Visual speech detection using facial landmarks

• US 9,190,061 B1
• Filed: 03/15/2013
• Issued: 11/17/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Shemer, Mikhal
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US13/839,655
• Time in Patent Office: 977 Days
• Field of Search: 704/215, 704/233, 704/236, 382/118, 382/170
• US Class Current: 1/1
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

A data processing apparatus for detecting a probability of speech based on video data is disclosed. The data processing apparatus may include at least one processor, and a non-transitory computer-readable storage medium including instructions executable by the at least one processor, where execution of the instructions by the at least one processor causes the data processing apparatus to execute a visual speech detector. The visual speech detector may be configured to receive a coordinate-based signal. The coordinate-based signal may represent movement or lack of movement of at least one facial landmark of a person in a video signal. The visual speech detector may be configured to compute a probability of speech of the person based on the coordinate-based signal.

Claims

1. A data processing apparatus for detecting a probability of speech based on video data, the data processing apparatus comprising:

at least one processor;

a non-transitory computer-readable storage medium including instructions executable by the at least one processor, wherein execution of the instructions by the at least one processor causes the data processing apparatus to execute;

a visual speech detector configured to receive a coordinate-based signal, the coordinate-based signal representing movement or lack of movement of at least one facial landmark of a person in a video signal;

the visual speech detector configured to calculate a short-term value representing short-term characteristics of the coordinated-based signal and a long-term value representing long-term characteristics of the coordinate-based signal,

the visual speech detector configured to compute a probability of speech of the person based on a comparison of the short-term value and the long-term value, wherein, when the short-term value is greater than the long-term value, the visual speech detector computes the probability of speech as a value indicating that speech as occurred.

2. The data processing apparatus of claim 1, wherein execution of the instructions by the at least one processor further causes the data processing apparatus to execute:

a facial landmark detector configured to receive the video signal including video data captured from an image capturing device, and determine coordinates of the at least one facial landmark based on the video data; and

a signal generator configured to generate the coordinate-based signal based on the coordinates of the at least one facial landmark.

3. The data processing apparatus of claim 1, wherein the at least one facial landmark includes a mouth region of the person.

4. The data processing apparatus of claim 3, wherein the coordinate-based signal includes values indicating a ratio of height and width of the mouth region for a plurality of video frames.

5. The data processing apparatus of claim 1, wherein the visual speech detector includes:

a quantizer configured to partition the coordinate-based signal into a plurality of bins;

a filtered value calculator configured to calculate the short-term value and the long-term value;

a long-term bin selector configured to select a long-term bin from the plurality of bins that corresponds to the long-term value;

a short-term bin selector configured to select a short-term bin from the plurality of bins that corresponds to the short-term value;

a bin comparator configured to compare the long-term bin with the short-term bin; and

a probability calculator configured to calculate the probability of speech based on the comparison.

6. The data processing apparatus of claim 5, wherein the bin comparator configured to compare the long-term bin with the short-term bin includes determining that the short-term bin is greater than the long-term bin, wherein the probability calculator calculates the probability of speech as a value indicating that speech has occurred.

7. The data processing apparatus of claim 5, wherein the bin comparator configured to compare the long-term bin with the short-term bin includes determining that the short-term bin is less than the long-term bin, wherein the probability calculator calculates the probability of speech as a value indicating that speech as not occurred.

8. The data processing apparatus of claim 5, wherein the bin comparator configured to compare the long-term bin with the short-term bin includes determining that the short-term bin is equal to the long-term bin, wherein the probability calculator calculates the probability of speech as an intermediate value between a first value indicating that speech has occurred and a second value indicating that speech has not occurred.

9. The data processing apparatus of claim 8, wherein the probability calculator is configured to adjust the intermediate value based on at least one signal characteristic of the coordinate-based signal.

10. The data processing apparatus of claim 5 wherein execution of the instructions by the at least one processor further causes the data processing apparatus to execute:

a histogram builder configured to build a histogram of the coordinate-based signal;

a data validator configured to determine an active range of the person based on the histogram, and determine a validity of the coordinate-based signal based on the active range.

11. A method for detecting a probability of speech based on video data,

the method comprising:

receiving, by at least one processor, a coordinate-based signal, the coordinate-based signal representing movement or lack of movement of at least one facial landmark of a person in a video signal;

building, by the at least one processor, a histogram of the coordinate-based signal;

determining, by the at least one processor, an active range of the person based on the histogram;

validating, by the at least one processor, the coordinate-based signal based on the active range;

calculating, by the at least one processor, a short-term value representing short-term characteristics of the validated coordinated-based signal and a long-term value representing long-term characteristics of the validated coordinate-based signal; and

computing, by the at least one processor, a probability of speech of the person based on a comparison of the short-term value and the long-term value of the validated coordinate-based signal.

12. The method of claim 11, wherein the at least one facial landmark includes a mouth region.

13. The method of claim 12, wherein the coordinate-based signal includes values indicating a ratio of height and width of the mouth region for a plurality of video frames.

14. The method of claim 11, further comprising:

partitioning, by the at least one processor, the validated coordinate-based signal into a plurality of bins;

selecting, by the at least one processor, a long-term bin from the plurality of bins that corresponds to the long-term value and a short-term bin from the plurality of bins that corresponds to the short-term value;

comparing, by the at least one processor, the long-term bin with the short-term bin; and

calculating, by the at least one processor, the probability of speech based on the comparison.

15. A non-transitory computer-readable medium storing instructions that when executed cause at least one processor to detect a probability of speech based on video data, the instructions comprising instructions to:

receive a coordinate-based signal, the coordinate-based signal representing movement or lack of movement of at least one facial landmark of a person in a video signal;

partition the coordinate-based signal into a plurality of bins;

calculate a short-term value representing short-term characteristics of the partitioned coordinated-based signal and a long-term value representing long-term characteristics of the partitioned coordinate-based signal;

select a long-term bin from the plurality of bins that corresponds to the long-term value and a short-term bin from the plurality of bins that corresponds to the short-term value;

compare the long-term bin with the short-term bin; and

calculate a probability of speech of the person based on the comparison.

16. The non-transitory computer-readable medium of claim 15, wherein the at least one facial landmark includes a mouth region, and the coordinate-based signal includes values indicating a ratio of height and width of the mouth region for a plurality of video frames.

17. The non-transitory computer-readable medium of claim 15, wherein the instructions to compare the long-term bin with the short-term bin includes determining that the short-term bin is greater than the long-term bin, and the instructions to calculate a probability of speech includes calculating the probability of speech as a value indicating that speech has occurred.

18. The non-transitory computer-readable medium of claim 15, wherein the instructions to compare the long-term bin with the short-term bin includes determining that the short-term bin is less than the long-term bin, and the instructions to calculate a probability of speech includes calculating the probability of speech as a value indicating that speech has not occurred.

19. The non-transitory computer-readable medium of claim 15, wherein the instructions to compare the long-term bin with the short-term bin includes determining that the short-term bin is equal to the long-term bin, and the instructions to calculate a probability of speech includes calculating the probability of speech as an intermediate value between a first value indicating that speech has occurred and a second value indicating that speech has not occurred.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions to calculate a probability of speech includes adjusting the intermediate value based on a simultaneous analysis of multiple signal characteristics of the coordinate-based signal.