Eyelid detection apparatus, eyelid detection method and program therefor

• US 7,957,566 B2
• Filed: 03/13/2008
• Issued: 06/07/2011
• Est. Priority Date: 03/15/2007
• Status: Active Grant

• Current Assignee: Aisin Seiki Co., Ltd. (Aisin Corp.), Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Aisin Seiki Co., Ltd. (Aisin Corp.), Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Nakanishi, Satoru, Hiramaki, Takashi, Suzuki, Tomoharu, Ninagawa, Yuji, Ohue, Kenichi, Adachi, Jun, Kojima, Shinichi, Uozumi, Shigeyasu, Takahashi, Kentaro
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Rosario; Dennis
• Application Number:
  US12/047,612
• Publication Number:
  US 20080226139A1
• Time in Patent Office: 1,181 Days
• Field of Search: 382/116, 382/117
• US Class Current: 382/117
• CPC Class Codes: A61B 3/10 Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: G06V 10/36 Applying a local operator, i.e. means to operate on image points situated in the vicinity of a given point; Non-linear local filtering operations, e.g. median filtering
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

An eyelid detection apparatus includes a face-image storing apparatus for storing an image of a face of a subject, an eyelid-candidate extracting apparatus for processing the image stored in the face-image storing apparatus to extract a pair of lines, which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid, a parameter calculating apparatus for calculating a parameter, which indicates a possibility that a pan of the image stored in the face-image storing apparatus, the part corresponding to the pair of lines extracted by the eyelid-candidate extracting apparatus, includes at least one of outer and inner corners of an eye, and an eyelid detecting apparatus for detecting a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating apparatus.

Claims

1. An eyelid detection apparatus, comprising:

a face-image storing means for storing an image of a face of a subject;

an eyelid-candidate extracting means for processing the image stored in the face-image storing means and extracting a pair of lines, which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating means for calculating an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculating a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels;

an eyelid detecting means for detecting a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means; and

a pattern storing means for storing a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is equal to that of the predetermined part including the both ends of the line, wherein

the parameter calculating means calculates a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face- image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means, and wherein

the eyelid detecting means detects right and left eyelids on the basis of the difference.

2. An eyelid detection apparatus, comprising:

a face-image storing means for storing an image of a face of a subject;

an eyelid-candidate extracting means for processing the image stored in the face-image storing means and extracting a pair of lines, which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating means for calculating an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculating a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels;

an eyelid detecting means for detecting a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means; and

a pattern storing means for storing a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is narrower than that of the predetermined part including the both ends of the line, wherein

the parameter calculating means includes a difference calculating means for relating the pattern stored by the pattern storing means to a portion of the predetermined part including the both ends of the line and calculating a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means,

the difference calculating means scans the predetermined part including the both ends of the line and sets a greatest difference among the calculated plural differences as a maximum difference value, and

the eyelid detecting means detects right and left eyelids on the basis of the maximum difference value.

3. An eyelid detection method, which is executed by an eyelid detection apparatus comprising a face-image storing means for storing an image of a face of a subject, an eyelid-candidate extracting means, a parameter calculating means, an eyelid detecting means and a pattern storing means, the eyelid detection method comprising processes of:

an eyelid-candidate extracting process where the eyelid-candidate extracting means processes the image stored in the face-image storing means and extracts a pair of lines, which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating process where the parameter calculating means calculates an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculates a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels; and

an eyelid detecting process where the eyelid detecting means detects a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means;

wherein

the pattern storing means stores a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is equal to that of the predetermined part including the both ends of the line,

the parameter calculating means calculates a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means, in the parameter calculating process, and

the eyelid detecting means detects right and left eyelids on the basis of the difference in the eyelid detecting process.

4. An eyelid detection method, which is executed by an eyelid detection apparatus comprising a face-image storing means for storing an image of a face of a subject, an eyelid-candidate extracting means, a parameter calculating means, an eyelid detecting means and a pattern storing means, the eyelid detection method comprising processes of:

an eyelid-candidate extracting process where the eyelid-candidate extracting means processes the image stored in the face-image storing means and extracts a pair of lines, which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating process where the parameter calculating means calculates an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculates a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels; and

an eyelid detecting process where the eyelid detecting means detects a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means;

wherein

the pattern storing means stores a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is narrower than that of the predetermined part including the both ends of the line,

the parameter calculating means relates the pattern stored in the pattern storing means to a portion of the predetermined part including the both ends of the line, and calculates a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means, in the parameter calculating process,

the difference calculating means scans the predetermined part including the both ends of the line and sets a greatest difference among the calculated plural differences as a maximum difference value, and

the eyelid detecting means detects right and left eyelids on the basis of the maximum difference value in the eyelid detecting process.

5. A non-transitory computer-readable medium including a program for instructing a computer to function as:

a face-image storing means for storing an image of a face of a subject;

an eyelid-candidate extracting means for processing the image stored in the face-image storing means and extracting a pair of lines which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating means for calculating an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculates a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels;

an eyelid detecting means for detecting a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means; and

a pattern storing means for storing a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is equal to that of the predetermined part including the both ends of the line, wherein

the parameter calculating means calculates a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face- image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means, in the parameter calculating process, and

the eyelid detecting means detects right and left eyelids on the basis of the difference.

6. A non-transitory computer-readable medium including a program for instructing a computer to function as:

a face-image storing means for storing an image of a face of a subject;

an eyelid-candidate extracting means for processing the image stored in the face-image storing means and extracting a pair of lines which becomes a candidate of a combination of a line corresponding to an upper eyelid and a line corresponding to a lower eyelid;

a parameter calculating means for calculating an average value of intensity of plural pixels in a predetermined part of the image, which is stored within the face-image storing means, including both ends of the line, which is one of the pair of the candidate lines extracted by the eyelid-candidate extracting means and corresponding to the lower eyelid, and calculates a parameter, which indicates a possibility that the predetermined part of the image including the both ends of the line includes at least one of an outer corner and an inner corner of an eye on the basis of the calculated average value of the intensity of the plural pixels;

an eyelid detecting means for detecting a position of an eyelid of the subject on the basis of the parameter calculated by the parameter calculating means; and

a pattern storing means for storing a pattern which represents a skin region corresponding to a skin part and an eye region corresponding to an eye part, within an area, which is narrower than that of the predetermined part including the both ends of the line, wherein

the parameter calculating means includes a difference calculating means for relating the pattern stored in the pattern storing means to a portion of the predetermined part including the both ends of the line, and calculating a difference between an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the skin region, which is represented by the pattern stored in the pattern storing means, and an average value of intensity of plural pixels of a part of the image stored in the face-image storing means corresponding to the eye region, which is represented by the pattern stored in the pattern storing means, in the parameter calculating process, and

the difference calculating means scans the predetermined part including the both ends of the line and sets a greatest difference among the calculated plural differences as a maximum difference value, and

the eyelid detecting means detects right and left eyelids on the basis of the maximum difference value.