Method for recognizing content in an image sequence

• US 8,077,930 B2
• Filed: 04/13/2007
• Issued: 12/13/2011
• Est. Priority Date: 04/13/2007
• Status: Active Grant

• Current Assignee: ATG Advanced Swiss Technology Group AG
• Original Assignee: ATG Advanced Swiss Technology Group AG
• Inventors: Hauke, Rudolf
• Primary Examiner(s): Werner, Brian
• Assistant Examiner(s): SHAH, UTPAL D
• Application Number:
  US11/785,027
• Publication Number:
  US 20080253623A1
• Time in Patent Office: 1,705 Days
• Field of Search: 382/115, 382/118, 382/190, 382/305
• US Class Current: 382/115
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method for recognizing content in an image sequence is provided, the method includes the steps of: detecting at least one face appearing in at least one frame of an image sequence under test; recognizing characteristic features of the at least one face; comparing the characteristic features to known features of characters stored in a database, thereby deciding whether the face represents a known character; detecting and recognizing at least one additional feature in at least one frame of the image sequence under test and at least one relation between the appearance of the known character and the at least one additional feature; and comparing the at least one relation to metadata comprising known relations stored in the database each one assigned to a particular known image sequence, thereby recognizing if the image sequence under test at least partially equals one of the known image sequences.

Claims

1. A method for recognizing content in image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the image frames of an image sequence under test;

detecting and recognizing two or more characteristic features of the at least one face;

comparing the characteristic features to known features of characters stored in a database for determining whether the face represents a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test, wherein the at least one additional feature comprises a feature movable independently of the at least one face;

detecting and recognizing at least one spatial relation between the determined known character and the at least one additional feature;

comparing the at least one spatial relation to metadata comprising known spatial relations stored in the database each one assigned to a particular known image sequence for recognizing if the image sequence under test at least partially equals one of the known image sequences;

detecting and recognizing two or more characters in different frames;

detecting an appearance of the two or more characters; and

detecting and recognizing at least one spatiotemporal relation between the appearance of the two or more characters.

2. The method according to claim 1, wherein the relation is spatiotemporal.

3. The method according to claim 1, wherein the at least one character is a real person.

4. The method according to claim 1, wherein the at least one character is an animated character.

5. The method according to claim 1, wherein the at least one additional feature is another face.

6. The method according to claim 1, wherein the at least one additional feature is an object.

7. The method according to claim 6, wherein the object is a car, weapon, building, text, logo, and/or trademark.

8. The method according to claim 7, wherein the text object is identified by pattern matching.

9. The method according to claim 1, wherein the at least one additional feature includes a color of an object.

10. The method according to claim 1, wherein the at least one additional feature is an object touched by the known character.

11. The method according to claim 1, wherein the at least one additional feature is a costume worn by the known character.

12. The method according to claim 1, wherein the at least one additional feature is background scenery.

13. The method according to claim 1, wherein the at least one additional feature is sound.

14. The method according to claim 13, wherein the sound is verbal.

15. The method according to claim 13, wherein the sound is nonverbal.

16. The method according to claim 1, wherein the at least one additional feature is at least detected in the at least one frame in which the at least one face was detected.

17. The method according to claim 1, wherein the at least one additional feature is a spatiotemporal profile of the known character acquired by tracking the known character in the course of the image sequence under test.

18. The method according to claim 1, wherein co-appearance of at least two characters in the at least one frame is detected and compared to the metadata.

19. The method according to claim 1, wherein successive appearance of at least two characters in the image sequence under test along with time intervals between the appearances is detected and compared to the metadata.

20. The method according to claim 1, wherein the image sequence under test is subsampled, thereby reducing the number of frames to be tested.

21. The method according to claim 1, wherein a cast list of the image sequence under test is generated by recognizing characters.

22. The method according to claim 3, wherein the at least one additional feature is an object and wherein at least one of the additional features is a spatiotemporal profile of the known character acquired by tracking the known character in the course of the image sequence under test.

23. The method according to claim 1, further comprising detecting whether the image sequence under test is copyrighted by comparing it to metadata of an image sequence known to be copyrighted.

24. The method according to claim 23, wherein the detection is carried out on a server following an upload of the image sequence under test from a client computer wherein the image sequence under test is incorporated in a video database only if the image sequence under test is recognized as noncopyrighted.

25. The method according to claim 1, wherein, for detecting whether the image sequence, under test is part of a video output of a computer game, the image sequence is compared to metadata of the computer game.

26. An apparatus configured to perform the method according to claim 1, wherein the apparatus comprises at least one of a computer, a portable device, a video game console, a handheld devices or a cellular phone.

27. A method for recognizing content in image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the image frames of an image sequence under test;

detecting and recognizing two or more characteristic features of the at least one face;

comparing the characteristic features to known features of characters stored in a database for determining whether the face represents a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test;

detecting and recognizing at least one relation between the determined known character and the at least one additional feature; and

comparing the at least one relation to metadata comprising known relations stored in the database each one assigned to a particular known image sequence for recognizing if the image sequence under test at least partially equals one of the known image sequences,

wherein the detecting at least one face is carried out on a client computer following an attempt to upload the image sequence under test to a server and wherein the upload is denied if the image sequence under test is recognized as copyrighted,

detecting and recognizing two or more characters in different frames;

detecting an appearance of the two or more characters; and

detecting and recognizing at least one spatiotemporal relation between the appearance of the two or more characters.

28. A method for recognizing content in an image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the frames of an image sequence under test;

detecting and recognizing two or more characteristic features of the at least one face;

storing the characteristic features in a database and assigning them to a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test, wherein the at least one additional feature comprises a feature movable independently of the at least one face;

detecting and recognizing at least one spatial relation between the determined known character and the at least one additional feature;

storing the at least one spatial relation to metadata in the database; and

assigning the at least one spatial relation to the image sequence under test in the database;

detecting and recognizing two or more characters in different frames;

detecting an appearance of the two or more characters; and

detecting and recognizing at least one spatiotemporal relation between the appearance of the two or more characters.

29. A method for recognizing content in image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the image frames of an image sequence under test;

detecting and recognizing two or more characteristic features of the at least one face;

comparing the characteristic features to known features of characters stored in a database for determining whether the face represents a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test, wherein the at least one additional feature comprises a body configuration;

detecting and recognizing and at least one spatial relation between the determined known character and the at least one additional feature;

comparing the at least one spatial relation to metadata comprising known spatial relations stored in the database each one assigned to a particular known image sequence for recognizing if the image sequence under test at least partially equals one of the known image sequences;

detecting and recognizing two or more characters in different frames;

detecting an appearance of the two or more characters; and

detecting and recognizing at least one spatiotemporal relation between the appearance of the two or more characters.

30. The method according to claim 29, wherein the at least one additional feature is a facial expression of the known character.

31. The method according to claim 29, wherein the at least one additional feature is a hand gesture of the known character.

32. The method according to claim 29, wherein the at least one additional feature is a body position of the known character.

33. The method according to claim 29, wherein the at least one additional feature is a position of lips of the known character.