FACIAL TRACKING METHOD

• US 20140086450A1
• Filed: 03/14/2013
• Published: 03/27/2014
• Est. Priority Date: 09/24/2012
• Status: Active Grant

• Current Assignee: Primax Electronics Limited
• Original Assignee: Primax Electronics Limited
• Inventors: Huang, Ying-Jieh, Liu, Xu-Hua, Jia, Qian-Wen
• Granted Patent: US 9,171,197 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

A facial tracking method for detecting and tracking at least one face image in a region during a time period. The facial tracking method includes a step of performing an image acquiring operation, a step of performing a facial detecting operation to detect whether there is any face image in the entire of a current photo image, and at least one step of performing a facial tracking operation. For performing the facial tracking operation, plural tracking frames are located around a face image of the current photo image, and a similarity between the face image of the current photo image and the image included in each tracking frame in order to judge whether the face image exists in the next photo image. By the facial tracking method of the present invention, the time period of tracking face images is largely reduced.

Claims

1. A facial tracking method for detecting and tracking plural face images within a specified region in a set time period, said facial tracking method comprising steps of:

performing an image acquiring operation for continuously shooting said specified region at the same viewing angle, thereby acquiring plural photo images;

performing a first facial detecting operation for extracting a current photo image from said plural photo images, and detecting whether there is any face image in the entire of said current photo image, wherein if a first face image is included in said current photo image, a first facial data corresponding to said first face image is recorded into a database; and

performing a first facial tracking operation for determining whether said first face image exists in a next photo image, which is posterior to said current photo image, wherein said step of performing said first facial tracking operation comprises sub-steps of locating plural first tracking frames in said current photo image by taking a first face frame including said first face image as a center, extracting said next photo image, calculating plural similarities between images included in said next photo image corresponding to said plural first tracking frames and said image included in said first face frame, and detecting whether there is any face image in said next photo image corresponding to said first tracking frame with the highest similarity relative to said image included in said first face frame, wherein if a face image is included in said next photo image corresponding to said first tracking frame with the highest similarity, said face image is recognized as said first face image, and said first facial data is updated according to information corresponding to said first tracking frame with the highest similarity, wherein if no face image is included in said next photo image corresponding to said first tracking frame with the highest similarity, said first face image is considered to be excluded from said next photo image, and said first facial data is updated.

2. The facial tracking method according to claim 1, wherein said first facial data comprises a position of said first face image in each photo image, a tracking state of said first face image in each photo image, the number of said photo images said first face image exists, a facial texture of said first face image, and a color feature of a specified part under said first face image, wherein said tracking state includes a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said first face image exists in a specified photo image, said first face image is in said continuous tracking state, wherein if said first face image does not exist in said specified photo image, said first face image is in said lost tracking state, wherein if said first face image does not exist in several consecutive photo images, said first face image is in said stop tracking state.

3. The facial tracking method according to claim 2, wherein after said first facial tracking operation is performed, said facial tracking method further comprises a step of performing a second facial tracking operation for determining whether said first face image exists in a subsequent photo image, which is posterior to said next photo image, wherein if a face image is included in said next photo image corresponding to said first tracking frame with the highest similarity, said step of performing said second facial tracking operation comprises sub-steps of:

locating plural second tracking frames in said next photo image by taking each of said plural first tracking frames as a center, wherein the number of said plural second tracking frames relative to each of said plural first tracking frames is determined according to a similarity between said image included in each of said plural first tracking frame and said first face image;

extracting a subsequent photo image, which is posterior to said next photo image;

calculating plural similarities between images included in said subsequent photo image corresponding to said plural second tracking frames and said image included in said next photo image corresponding to said first tracking frame with the highest similarity; and

detecting whether there is any face image in said subsequent photo image corresponding to said second tracking frame with the highest similarity relative to said first tracking frame with the highest similarity, wherein if a face image is included in said subsequent photo image corresponding to said second tracking frame with the highest similarity, said face image is recognized as said first face image, and said first facial data is updated according to information corresponding to said second tracking frame with the highest similarity, wherein if no face image is included in said subsequent photo image corresponding to said second tracking frame with the highest similarity, said first face image is considered to be excluded from said subsequent photo image, and said first facial data is updated.

4. The facial tracking method according to claim 3, wherein after said second facial tracking operation is performed, said facial tracking method further comprises a step of performing a second facial detecting operation for detecting whether there is any additional face image in the entire of said subsequent photo image.

5. The facial tracking method according to claim 4, wherein during said second facial detecting operation is performed, if a second face image is included in said subsequent photo image and said face image is included in said subsequent photo image corresponding to said second tracking frame with the highest similarity, a comparison process is performed to judge whether said second face image and said first face image belong to the same person, wherein said comparison process comprises steps of:

comparing a position of a second face frame including said second face image with a position of said second tracking frame with the highest similarity;

if said position of said second face frame and said position of said second tracking frame with the highest similarity are highly overlapped, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to a second facial data of said second face image;

if said position of said second face frame and said position of said second tracking frame with the highest similarity are not highly overlapped, calculating a facial texture similarity between said second face image and said first face image and a color feature similarity between a specified part under said second face image and a specified part under said first face image;

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both smaller than a first threshold value, confirming that said second face image and said first face image do not belong to the same person, and recording said second facial data of said second face image into said database; and

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both higher than said first threshold value, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to said second facial data of said second face image.

6. The facial tracking method according to claim 5, wherein said second facial data comprises a position of said second face image in each photo image, a tracking state of said second face image in each photo image, the number of said photo images said second face image exists, a facial texture of said second face image, and a color feature of a specified part under said second face image, wherein said tracking state includes a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said second face image exists in a specified photo image, said second face image is in said continuous tracking state, wherein if said second face image does not exist in said specified photo image, said second face image is in said lost tracking state, wherein if said second face image does not exist in several consecutive photo images, said second face image is in said stop tracking state.

7. The facial tracking method according to claim 4, wherein during said second facial detecting operation is performed, if a second face image is included in said subsequent photo image and no face image is included in said subsequent photo image corresponding to said second tracking frame with the highest similarity, a comparison process is performed to judge whether said second face image and said first face image belong to the same person, wherein said comparison process comprises steps of:

calculating a facial texture similarity between said second face image and said first face image and a color feature similarity between a specified part under said second face image and a specified part under said first face image;

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both smaller than a first threshold value, confirming that said second face image and said first face image do not belong to the same person, and recording a second facial data of said second face image into said database; and

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both higher than said first threshold value, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to said second facial data of said second face image.

8. The facial tracking method according to claim 7, wherein said second facial data comprises a position of said second face image in each photo image, a tracking state of said second face image in each photo image, the number of said photo images said second face image exists, a facial texture of said second face image, and a color feature of a specified part under said second face image, wherein said tracking state includes a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said second face image exists in a specified photo image, said second face image is in said continuous tracking state, wherein if said second face image does not exist in said specified photo image, said second face image is in said lost tracking state, wherein if said second face image does not exist in several consecutive photo images, said second face image is in said stop tracking state.

9. The facial tracking method according to claim 2, wherein after said first facial tracking operation is performed, said facial tracking method further comprises a step of performing a second facial tracking operation for determining whether said first face image exists in a subsequent photo image, which is posterior to said next photo image, wherein if no face image is included in said next photo image corresponding to said first tracking frame with the highest similarity, said step of performing said second facial tracking operation comprises sub-steps of:

extracting said subsequent photo image;

locating a second tracking frame with a local area in said subsequent photo image, wherein a center of said local area is corresponding to said first face frame of said current photo image;

detecting whether there is any face image in said second tracking frame of said subsequent photo image;

if no face image is included in said second tracking frame, confirming that said first face image is not included in said subsequent photo image, and further judging whether the number of times said first face image is in said lost tracking state is larger than a second threshold value, wherein if the number of times said first face image is in said lost tracking state is larger than a second threshold value, said first face image is no longer searched from remaining photo images;

if a face image is included in said second tracking frame, calculating a facial texture similarity between said first face image and said face image included in said second tracking frame;

if said facial texture similarity between said first face image and said face image included in said second tracking frame is larger than a third threshold value, confirming that said first face image and said face image included in said second tracking frame belong to the same person, and updating said first facial data according to information corresponding to said second tracking frame;

if said facial texture similarity between said first face image and said face image included in said second tracking frame is smaller than said third threshold value, confirming that said first face image is not included in said subsequent photo image, and judging whether the number of times said first face image is in said lost tracking state is larger than said second threshold value, wherein if the number of times said first face image is in said lost tracking state is larger than said second threshold value, said first face image is no longer searched from said remaining photo images; and

if said facial texture similarity between said first face image and said face image included in said second tracking frame is smaller than said third threshold value, confirming that said face image included in said second tracking frame is a third face image different from said first face image, and recording a third facial data of said third face image into said database.

10. The facial tracking method according to claim 9, wherein said third facial data comprises a position of said third face image in each photo image, a tracking state of said third face image in each photo image, the number of photo images said second face image exists, a facial texture of said third face image, and a color feature of a specified part under said third face image, wherein said tracking state include a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said third face image exists in a specified photo image, said third face image is in said continuous tracking state, wherein if said third face image does not exist in said specified photo image, said third face image is in said lost tracking state, wherein if said third face image does not exist in several consecutive photo images, said third face image is in said stop tracking state.

11. The facial tracking method according to claim 10, wherein after said second facial tracking operation is performed, said facial tracking method further comprises a step of performing a second facial detecting operation for detecting whether there is any additional face image in the entire of said subsequent photo image.

12. The facial tracking method according to claim 11, wherein during said second facial detecting operation is performed, if a second face image in said subsequent photo image is detected and said face image included in said second tracking frame and said first face image belong to the same person, a comparison process is performed to judge whether said second face image and said first face image belong to the same person, wherein said comparison process comprises steps of:

comparing a position of a second face frame including said second face image with a position of said second tracking frame;

if said position of said second face frame and said position of said second tracking frame are highly overlapped, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to a second facial data of said second face image;

if said position of said second face frame and said position of said second tracking frame are not highly overlapped, calculating a facial texture similarity between said second face image and said face image included in said second tracking frame and a color feature similarity between a specified part under said second face image and a specified part under said face image included in said second tracking frame;

if said facial texture similarity and said color feature similarity between said second face image and said face image included in said second tracking frame are both smaller than a first threshold value, confirming that said second face image and said first face image do not belong to the same person, and recording said second facial data of said second face image into said database; and

if said facial texture similarity and said color feature similarity between said second face image and said face image included in said second tracking frame are both higher than said first threshold value, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to said second facial data of said second face image.

13. The facial tracking method according to claim 12, wherein said second facial data comprises a position of said second face image in each photo image, a tracking state of said second face image in each photo image, the number of said photo images said second face image exists, a facial texture of said second face image, and a color feature of a specified part under said second face image, wherein said tracking state includes a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said second face image exists in a specified photo image, said second face image is in said continuous tracking state, wherein if said second face image does not exist in said specified photo image, said second face image is in said lost tracking state, wherein if said second face image does not exist in several consecutive photo images, said second face image is in said stop tracking state.

14. The facial tracking method according to claim 11, wherein during said second facial detecting operation is performed, if a second face image in said subsequent photo image is detected and said face image included in said second tracking frame is said third face image, a comparison process is performed to judge whether said second face image and said third face image or said first face image belong to the same person, wherein said comparison process comprises steps of:

comparing a position of a second face frame including said second face image with a position of said second tracking frame;

if said position of said second face frame and said position of said second tracking frame are highly overlapped, confirming that said second face image and said third face image belong to the same person, and updating said third facial data according to a second facial data of said second face image;

if said position of said second face frame and said position of said second tracking frame are not highly overlapped, calculating a facial texture similarity between said second face image and said third face image and a color feature similarity between a specified part under said second face image and a specified part under said third face image, and calculating a facial texture similarity between said second face image and said first face image and a feature similarity between said specified part under said second face image and a specified part under said first face image;

if said facial texture similarity and said color feature similarity between said second face image and said third face image and said facial texture similarity and said color feature similarity between said second face image and said first face image are all smaller than a first threshold value, confirming that said second face image and said third face image and said first face image do not belong to the same person, and recording said second facial data of said second face image into said database; and

if said facial texture similarity and said color feature similarity between said second face image and said third face image or said facial texture similarity and said color feature similarity between said second face image and said first face image are larger than said first threshold value, confirming that said second face image and said third face image or said first face image belong to the same person, and updating said first facial data or said third facial data according to said second facial data of said second face image.

15. The facial tracking method according to claim 14, wherein said second facial data comprises a position of said second face image in each photo image, a tracking state of said second face image in each photo image, the number of photo images said second face image exists, a facial texture of said second face image, and a color feature of a specified part under said second face image, wherein said tracking state include a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said second face image exists in a specified photo image, said second face image is in said continuous tracking state, wherein if said second face image does not exist in said specified photo image, said second face image is in said lost tracking state, wherein if said second face image does not exist in several consecutive photo images, said second face image is in said stop tracking state.

16. The facial tracking method according to claim 11, wherein during said second facial detecting operation is performed, if a second face image in said subsequent photo image is detected and no face image is included in said second tracking frame, a comparison process is performed to judge whether said second face image and said first face image belong to the same person, wherein said comparison process comprises steps of:

calculating a facial texture similarity between said second face image and said first face image and a color feature similarity between a specified part under said second face image and a specified part under said first face image;

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both smaller than a first threshold value, confirming that said second face image and said first face image do not belong to the same person, and recording a second facial data of said second face image into said database; and

if said facial texture similarity and said color feature similarity between said second face image and said first face image are both higher than said first threshold value, confirming that said second face image and said first face image belong to the same person, and updating said first facial data according to said second facial data of said second face image.

17. The facial tracking method according to claim 16, wherein said second facial data comprises a position of said second face image in each photo image, a tracking state of said second face image in each photo image, the number of said photo images said second face image exists, a facial texture of said second face image, and a color feature of a specified part under said second face image, wherein said tracking state includes a continuous tracking state, a lost tracking state or a stop tracking state, wherein if said second face image exists in a specified photo image, said second face image is in said continuous tracking state, wherein if said second face image does not exist in said specified photo image, said second face image is in said lost tracking state, wherein if said second face image does not exist in several consecutive photo images, said second face image is in said stop tracking state.

18. The facial tracking method according to claim 16, further comprising a step of performing a facial verification process, wherein said facial verification process comprises steps of:

if a first condition is satisfied, moving all facial data of said database to a temporary storing zone;

if a second condition is satisfied, moving a specified facial data of said temporary storing zone to a combination result storing zone;

calculating a similarity between an additional facial data of said temporary storing zone and said specified facial data, wherein if said similarity is larger than a fourth threshold value, said additional facial data is merged into said specified facial data, wherein if said similarity is smaller than said fourth threshold value, said additional facial data is added to said combination result storing zone; and

repeatedly acquiring any facial data from the temporary storing zone and comparing said acquired facial data with all facial data of said combination result storing zone until all facial data of said temporary storing zone are merged into or added to said combination result storing zone.

19. The facial tracking method according to claim 18, wherein said first condition indicates that the number of all facial data in said database reaches a fifth threshold value.

20. The facial tracking method according to claim 19, wherein said first condition indicates that a flowchart of detecting and tracking said plural face images is ended and the number of facial data in the database is not zero.

21. The facial tracking method according to claim 18, wherein said second condition indicates that the number of all facial data in said temporary storing zone reaches a fifth threshold value.

22. The facial tracking method according to claim 21, wherein said second condition indicates that a flowchart of detecting and tracking said plural face images is ended and all facial data in said database are moved to said temporary storing zone.