Method for recognizing content in an image sequence

• US 20080253623A1
• Filed: 04/13/2007
• Published: 10/16/2008
• Est. Priority Date: 04/13/2007
• Status: Active Grant

• Current Assignee: ATG Advanced Swiss Technology Group AG
• Original Assignee: Advanced US Technology Group, Inc.
• Inventors: Hauke, Rudolf
• Granted Patent: US 8,077,930 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method for recognizing content in an image sequence is provided, the method includes the steps of: detecting at least one face appearing in at least one frame of an image sequence under test; recognizing characteristic features of the at least one face; comparing the characteristic features to known features of characters stored in a database, thereby deciding whether the face represents a known character; detecting and recognizing at least one additional feature in at least one frame of the image sequence under test and at least one relation between the appearance of the known character and the at least one additional feature; and comparing the at least one relation to metadata comprising known relations stored in the database each one assigned to a particular known image sequence, thereby recognizing if the image sequence under test at least partially equals one of the known image sequences.

Claims

1. A method for recognizing content in image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the image frames of an image sequence under test;

recognizing characteristic features of the at least one face;

comparing the characteristic features to known features of characters stored in a database for determining whether the face represents a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test and at least one relation between the appearance of the known character and the at least one additional feature; and

comparing the at least one relation to metadata comprising known relations stored in the database each one assigned to a particular known image sequence for recognizing if the image sequence under test at least partially equals one of the known image sequences.

2. The method according to claim 1, wherein the relation is spatiotemporal.

3. The method according to claim 1, wherein the at least one character is a real person.

4. The method according to claim 1, wherein the at least one character is an animated character.

5. The method according to claim 1, wherein the at least one additional feature is another face.

6. The method according to claim 1, wherein the at least one additional feature is an object.

7. The method according to claim 6, wherein the object is a car, weapon, building, text, logo, and/or trademark.

8. The method according to claim 7, wherein the text object is identified by pattern matching.

9. The method according to claim 1, wherein the at least one additional feature is a color of an object.

10. The method according to claim 1, wherein the at least one additional feature is an object touched by the known character.

11. The method according to claim 1, wherein the at least one additional feature is a costume worn by the known character.

12. The method according to claim 1, wherein the at least one additional feature is background scenery.

13. The method according to claim 1, wherein the at least one additional feature is sound.

14. The method according to claim 13, wherein the sound is verbal.

15. The method according to claim 13, wherein the sound is nonverbal.

16. The method according to claim 1, wherein the at least one additional feature is a facial expression of the known character.

17. The method according to claim 1, wherein the at least one additional feature is a hand gesture of the known character.

18. The method according to claim 1, wherein the at least one additional feature is a body movement of the known character.

19. The method according to claim 1, wherein the at least one additional feature is a movement of the lips of the known character.

20. The method according to claim 1, wherein the at least one additional feature is at least detected in the at least one frame in which the at least one face was detected.

21. The method according to claim 1, wherein the at least one additional feature is detected in at least one second frame distinct from the at least one frame in which the at least one face was detected.

22. The method according to claim 1, wherein the at least one additional feature is a spatiotemporal profile of the known character acquired by tracking the known character in the course of the image sequence under test.

23. The method according to claim 1, wherein co-appearance of at least two of the characters in the at least one frame is detected and compared to the metadata.

24. The method according to claim 1, wherein successive appearance of at least two of the characters in the image sequence under test along with time intervals between the appearances is detected and compared to the metadata.

25. The method according to claim 1, wherein the image sequence under test is subsampled, thereby reducing the number of frames to be tested.

26. The method according to claim 1, wherein a cast list of the image sequence under test is generated by recognizing characters.

27. The method according to claim 3, wherein the at least one additional feature is an object and wherein at least one of the additional features is a spatiotemporal profile of the known character acquired by tracking the known character in the course of the image sequence under test.

28. Application of the method according to claim 1 for detecting whether the image sequence under test is copyrighted by comparing it to metadata of an image sequence known to be copyrighted.

29. Application according to claim 28, wherein the detection is carried out on a client computer following an attempt to upload the image sequence under test to a server and wherein the upload is denied if the image sequence under test is recognized as copyrighted.

30. Application according to claim 28, wherein the detection is carried out on a server following an upload of the image sequence under test from a client computer wherein the image sequence under test is incorporated in a video database only if the image sequence under test is recognized as noncopyrighted.

31. Application of the method according to claim 1, wherein, for detecting whether the image sequence under test is part of a video output of a computer game, the image sequence is compared to metadata of the computer game.

32. Implementation of the method according to claim 1 in at least one of computer, a portable device, a video game console, a handheld devices and a cellular phone.

33. A method for recognizing content in an image sequence having at least one image frame, the method comprising the steps of:

detecting at least one face appearing in at least one of the frames of an image sequence under test;

recognizing characteristic features of the at least one face;

storing the characteristic features in a database and assigning them to a known character;

detecting and recognizing at least one additional feature in at least one frame of the image sequence under test and at least one relation between the appearance of the known character and the at least one additional feature;

storing the at least one relation to metadata in the database; and

assigning the at least one relation to the image sequence under test in the database.