System and method for communication

• US 9,465,981 B2
• Filed: 05/08/2015
• Issued: 10/11/2016
• Est. Priority Date: 05/09/2014
• Status: Active Grant

• Current Assignee: Barron Associates Inc
• Original Assignee: Barron Associates Inc
• Inventors: Bateman, Alec, DeVore, Michael
• Primary Examiner(s): OLSON, JASON C
• Application Number:
  US14/707,755
• Publication Number:
  US 20150323989A1
• Time in Patent Office: 522 Days
• Field of Search: 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers

Abstract

Computer based systems, methods, and computer readable media for sensing and decoding a sequence of facial expression exhibited by an individual sender, in order to determine a message encoded by that sequence of the individual. Digital image sensor devices may capture facial images of a sending individual. An enrollment processes is used by which an enrolled message is associated with a sequence of facial expressions. A background model is used with relevant expression models to locate a region of interest of the individual'"'"'s face, which models accommodate the stand-off placement of the image sensors relative to the individual. An expression identifier is used with the expression model to classify captured images as including an enrolled facial expression. The classified images may then be decoded to identify the message sent.

Claims

1. A system that enables an individual sender to communicate messages to a receiver by exhibiting a sequence of at least a first and second facial expression, the system comprising:

at least one stand-off digital image sensing device configured to capture digital images of facial expressions of the sender;

a digital control unit having at least one computer processor, memory, non-volatile data storage, and communication interface, such that the at least one computer processor is in operable communication with the memory, data storage, and image sensing device;

a receiver interface in communication with the digital control unit via the communication interface;

the data storage comprising a background model, a first expression model, and a second expression model, the background model adapted to characterize a plurality of pixels from at least a portion of an image outside the region of interest of the sender'"'"'s face, the first expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled first facial expression, the second expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled second facial expression, wherein the first and second expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

the data storage further comprising at least one enrolled message associated with a sequence of a first and second enrolled expressions of the sender;

the digital control unit comprising computer programming code having an expression identifier and a sequence decoder, the at least one computer processor configured by the computer programming code, the computer programming code adapted to inter-relate with the digital imaging device, the background model, and the first and second expression models to locate the facial region of interest of the sender and to capture first and second images of the region of interest of the sender, the computer programming code further adapted to perform the steps of;

(a) applying the expression identifier to the first and second captured images with the first and second expression models to classify each of the first and second captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, or (iii) no enrolled facial expression;

(b) decoding the classified first and second captured images with the sequence decoder based on a matching score to determine if the classified first and second images satisfy a predetermined likelihood of communicating the at least one enrolled message; and

(c) communicating the at least one enrolled message to a receiver via the receiver interface.

2. The system of claim 1, wherein the first and second facial expressions comprise closed eyelids and open eyelids of the sender.

3. The system of claim 2, wherein the at least one message is a virtual button press, and the receiver is a person.

4. The system of claim 1, wherein the at least one digital image sensing device comprises an infrared sensor.

5. The system of claim 1, wherein the system further comprises a facial feature detector that locates the sender'"'"'s face within an image, allowing the system to detect changes of facial expression from a stand-off location without requiring the sender to wear a facial detection device.

6. The system of claim 5, wherein the digital image sensing device includes the facial feature detector, and the facial feature detector includes a tracker that determines movement of the sender'"'"'s face from sequences of images using the background model and the first and second expression models.

7. The system of claim 1 wherein the system enables an individual sender to communicate messages to a receiver by exhibiting a sequence of first, second, and third facial expressions, wherein:

the data storage further comprises a third expression model, the third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled third facial expression;

wherein the third expression model characterizes the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

wherein the digital control unit further comprises computer programming code adapted to inter-relate with the digital imaging device, the background model, and the third expression model to locate the facial region of interest of the sender and to capture a third image of the region of interest of the sender, with the computer programming code further adapted to perform the steps of;

(a) applying the expression identifier to the third captured image with the first, second, and third expression models to classify each of the first, second, and third captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the third enrolled facial expression, or (iv) no enrolled facial expression; and

(b) decoding the classified first, second, and third captured images with the sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, and third images satisfy a predetermined likelihood of communicating the at least one enrolled message.

8. The system of claim 1, wherein the expression identifier classifies each of the first and second captured images using a maximum likelihood probability calculation.

9. The system of claim 1 wherein the system enables an individual sender to communicate messages to a receiver by exhibiting a sequence of a first and second facial expression or a third and fourth facial expression, wherein:

the at least one enrolled message comprises a first and a second enrolled message, with the first enrolled message associated with a sequence of a first and second enrolled expression of the sender and the second enrolled message associated with a sequence of the third and fourth enrolled expressions of the sender;

the data storage further comprises a third expression model, the third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled third facial expression;

the data storage further comprises a fourth expression model, the fourth expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled fourth facial expression;

wherein the third and fourth expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

wherein the digital control unit further comprises computer programming code adapted to inter-relate with the digital imaging device, the background model, and the third and fourth expression models to locate the facial region of interest of the sender and to capture a third and fourth images of the region of interest of the sender, with the computer programming code further adapted to perform the steps of;

(a) applying the expression identifier to the third and fourth captured image with the first, second, third, and fourth expression models to classify each of the first, second, third, and fourth captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the third enrolled facial expression, (iv) the enrolled fourth facial expression, or (v) no enrolled facial expression, wherein the expression identifier classifies each of the captured images using a maximum likelihood probability calculation;

(b) decoding the classified first, second, third, and fourth captured images with the sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, third, and fourth images satisfy a predetermined likelihood of communicating the first or second enrolled message.

10. A method in a computer system for decoding a sequence of at least a first and a second facial expression exhibited by an individual sender to determine a message encoded by the individual sender, the method comprising:

receiving with a digital image sensing device first and second enrollment digital images of first and second facial expressions of a sender in sequence, including a region of interest of the sender, to form enrolled first and second facial expressions with at least one enrolled message associated with the sequence of the enrolled first and second facial expressions;

providing a background model, a first expression model, and a second expression model, the background model adapted to characterizing a plurality of pixels from at least a portion of an image outside a region of interest of the sender'"'"'s face, the first expression model adapted to characterizing a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled first facial expression, the second expression model adapted to characterizing a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled second facial expression,

wherein the first and second expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender;

inter-relating at least one digital image sensing device with the background model and the first and second expression models to locate the facial region of interest of the sender and capturing first and second images of the region of interest of the sender;

applying an expression identifier to the first and second captured images with the first and second expression models to classify each of the first and second captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, or (iii) no enrolled facial expression;

decoding the classified first and second captured images with a sequence decoder based on a matching score to determine if the classified first and second images satisfy a predetermined likelihood of communicating the at least one enrolled message; and

wherein the imaging, storing, providing, inter-relating, applying, and decoding are performed on at least one computer processor.

11. The method of claim 10, wherein the at least first and second facial expressions comprise closed eyelids and open eyelids of the sender.

12. The method of claim 11, wherein the at least one message is a virtual button press.

13. The method of claim 10, wherein the at least one digital image sensing device comprises an infrared sensor.

14. The method of claim 10 wherein the computer system decodes a sequence of a first, second, and third facial expression to determine a message encoded by the individual sender, the method further comprising:

receiving with the at least one digital image sensing device a third enrollment digital image of a third facial expression in sequence with the first and second facial expressions, and including the region of interest of the sender, to form an enrolled third facial expression;

providing a third expression model, the third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled third facial expression;

wherein the third expression model characterizes the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

inter-relating a digital imaging sensing device with the background model and the third expression model to locate the facial region of interest of the sender and to capture a third image of the region of interest of the sender;

applying the expression identifier to the third captured image with the first, second, and third expression models to classify each of the first, second, and third captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the third enrolled facial expression, or (iv) no enrolled facial expression; and

decoding the classified first, second, and third captured images with the sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, and third images satisfy a predetermined likelihood of communicating the at least one enrolled message.

15. The method of claim 14, wherein the expression identifier classifies each of the first, second, and third captured images using a maximum likelihood probability calculation.

16. The method of claim 10 wherein the computer system decodes a sequence of a first, second facial expressions or a third and fourth facial expression to determine a first or second enrolled message encoded by the individual sender, the method further comprising:

receiving with the at least one digital image sensing device a third and fourth enrollment digital images of a third facial and fourth facial expressions in sequence, including the region of interest of the sender, with the third and fourth facial expressions to form enrolled third and fourth facial expressions with the second enrolled message associated with the sequence of the enrolled third and fourth facial expressions

providing a third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled third facial expression;

providing a fourth expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled fourth facial expression;

wherein the third and fourth expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

inter-relating at least one digital imaging sensing device with the background model and the third and fourth expression models to locate the facial region of interest of the sender and to capture a third and fourth images of the region of interest of the sender;

applying the expression identifier to the third and fourth captured image with the first, second, third, and fourth expression models to classify each of the first, second, third, and fourth captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the enrolled third facial expression, (iv) the enrolled fourth facial expression, or (v) no enrolled facial expression, wherein the expression identifier classifies each of the captured images using a maximum likelihood probability calculation; and

decoding the classified first, second, third, and fourth captured images with the sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, third, and fourth images satisfy a predetermined likelihood of communicating the first or second enrolled message.

17. A non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, perform steps for decoding a sequence of at least a first and a second facial expression exhibited by an individual sender to determine a message encoded by the individual sender, the steps comprising:

receiving with a digital image sensing device first and second enrollment digital images of first and second facial expressions of a sender in sequence, including a region of interest of the sender, to form enrolled first and second facial expressions with at least one enrolled message associated with the sequence of the enrolled first and second facial expressions;

providing a background model, a first expression model, and a second expression model, the background model adapted to characterizing a plurality of pixels from at least a portion of an image outside the region of interest of the sender'"'"'s face, the first expression model adapted to characterizing a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled first facial expression, the second expression model adapted to characterizing a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled second facial expression, wherein the first and second expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender, and at least one enrolled message associated with a sequence of a first and second enrolled expressions of the sender;

inter-relating at least one digital image sensing device with the background model and the first and second expression models to locate the facial region of interest of the sender and capturing first and second images of the region of interest of the sender;

applying an expression identifier to the first and second captured images with the first and second expression models to classify each of the first and second captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, or (iii) no enrolled facial expression;

decoding the classified first and second captured images with a sequence decoder based on a matching score to determine if the classified first and second images satisfy a predetermined likelihood of communicating the at least one enrolled message.

18. The non-transitory computer-readable medium method of claim 17 wherein the instructions stored thereon, when executed by a processor perform steps for decoding a sequence of at least a first, second, and third facial expressions to determine a message encoded by the individual sender, the steps further comprising:

receiving with the at least one digital image sensing device a third enrollment digital image of a third facial expression in sequence with the first and second facial expressions, and including the region of interest of the sender, to form an enrolled third facial expression;

providing a third expression model, the third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting the enrolled third facial expression;

wherein the third expression model characterizes the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

inter-relating at least one digital imaging sensing device with the background model and the third expression model to locate the facial region of interest of the sender and to capture a third image of the region of interest of the sender;

applying the expression identifier to the third captured image with the first, second, and third expression models to classify each of the first, second, and third captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the third enrolled facial expression, or (iv) no enrolled facial expression; and

decoding the classified first, second, and third captured images with the Viterbi sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, and third images satisfy a predetermined likelihood of communicating the at least one enrolled message.

19. The non-transitory computer-readable medium method of claim 18, wherein the expression identifier classifies each of the first, second, and third captured images using a maximum likelihood probability calculation.

20. The non-transitory computer-readable medium method of claim 17 wherein the instructions stored thereon, when executed by a processor, perform steps for decoding a sequence of a first, second facial expressions or a third and fourth facial expression to determine a first or second enrolled message encoded by the individual sender, the method further comprising:

receiving with the at least one digital image sensing device a third and fourth enrollment digital images of a third facial and fourth facial expressions in sequence, including the region of interest of the sender, with the third and fourth facial expressions to form enrolled third and fourth facial expressions with the second enrolled message associated with the sequence of the enrolled third and fourth facial expressions

providing a third expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled third facial expression;

providing a fourth expression model adapted to characterize a plurality of pixels within an image of the region of interest of the sender'"'"'s face when the sender is exhibiting an enrolled fourth facial expression;

wherein the third and fourth expression models characterize the plurality of pixels as a function of location (m, n), orientation (φ

, θ

, ψ

), and size(s) of the region of interest within a captured image to accommodate variation in relative stand-off distance and orientation of the at least one digital imaging device to the sender,

inter-relating at least one digital imaging sensing device with the background model and the third and fourth expression models to locate the facial region of interest of the sender and to capture a third and fourth images of the region of interest of the sender;

applying the expression identifier to the third and fourth captured image with the first, second, third, and fourth expression models to classify each of the first, second, third, and fourth captured images as exhibiting (i) the enrolled first facial expression, (ii) the enrolled second facial expression, (iii) the enrolled third facial expression, (iv) the enrolled fourth facial expression, or (v) no enrolled facial expression, wherein the expression identifier classifies each of the captured images using a maximum likelihood probability calculation; and

decoding the classified first, second, third, and fourth captured images with the sequence decoder based on a Hidden Markov Model score to determine if the classified first, second, third, and fourth images satisfy a predetermined likelihood of communicating the first or second enrolled message.