METHOD AND APPARATUS FOR IDENTIFYING SPEAKERS AND EMPHASIZING SELECTED OBJECTS IN PICTURE AND VIDEO MESSAGES

• US 20110141106A1
• Filed: 12/15/2010
• Published: 06/16/2011
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Deutsche Telekom AG
• Original Assignee: Deutsche Telekom AG
• Inventors: Scheerbarth, Thomas, Runge, Fred
• Granted Patent: US 8,884,982 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 17/10 Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems

Abstract

A method for emphasizing selected objects in digital data of at least one of pictures and video that is stored in digital messages, the messages including sender addresses and being stored in a memory system of a digital messaging system for a plurality of users, includes the step of identifying picture regions including at least one of faces and persons in the digital data of the messages on the digital messaging system having the same sender address. Sender-relevant picture regions in the identified picture regions that represent a sender of the message are determined based on at least one of: a) a comparison with a reference picture of the sender stored on the memory system; b) a comparison of speech data in the digital data with reference speech data using at least one of speaker recognition methods, speaker verification methods and speaker identification methods, taking into account picture data and speech data in the message; and c) a frequency of occurrence of the identified picture regions in the messages that have the same sender address. The digital data of the messages is then modified so as to emphasize the sender-relevant picture region.

Claims

1. A method for emphasizing selected objects in digital data of at least one of pictures and video that is stored in digital messages, the messages including sender addresses and being stored in a memory system of a digital messaging system for a plurality of users, the method comprising:

identifying picture regions including at least one of faces and persons in the digital data of the messages on the digital messaging system having the same sender address;

determining sender-relevant picture regions in the identified picture regions that represent a sender of the message based on at least one of;

a) a comparison with a reference picture of the sender stored on the memory system;

b) a comparison of speech data in the digital data with reference speech data using at least one of speaker recognition methods, speaker verification methods and speaker identification methods, taking into account picture data and speech data in the message; and

c) a frequency of occurrence of the identified picture regions in the messages that have the same sender address; and

modifying the digital data of the messages so as to emphasize the sender-relevant picture region.

2. The method according to claim 1, wherein the modification of the digital data is performed such that a recipient of the message can undo the modification via a menu dialog

3. The method according to claim 1, wherein the identified picture regions are used to generate a 3D model.

4. The method according to claim 1, further comprising:

storing a sender-relevant 2D picture region of the identified picture regions along with the sender address and parameters that characterize a first perspective of the sender image;

transferring the sender-relevant 2D picture region and further sender-relevant picture regions from past messages from the memory system to a module for generating 3D picture data;

determining the first perspective of the sender-relevant 2D picture region based on imaged body proportions;

searching for a second 2D image having parameters that characterize a second perspective in the same horizontal plane so as to provide a 3D impression by evaluating ratios of body proportions;

adapting a size and color of the second 2D image; and

generating at least one 3D image of the sender by supplementing the sender-relevant 2D picture region with a second picture region including a sender image that is copied from the second 2D image, cut out and replaced by a picture from a different perspective.

5. The method according to claim 4, wherein searching for the second 2D image is first performed in a current message for the picture from the different perspective using face recognition methods in the case of a plurality of persons being present and, if a usable one of the picture from the different perspective is not found in the current message, then searching is performed on the memory system, in which case, proportions of features of the sender image are checked in additionally found ones of the picture from the different perspective to determine compliance with predefined limit values that describe a horizontal recording plane and a vertical recording plane.

6. The method according to claim 1, wherein at least one of comparing the speech data in the message with the speech data of a video message and comparing the speech data of a plurality of video messages with one another, is performed so as to find the sender-relevant picture regions of the video messages when there is an overlap of voice messages.

7. The method according to claim 1, wherein at least one of a terminal device and a server accessible via a network is provided for a 3D generation, the speaker recognition methods and a face recognition, wherein parameters that characterize different perspectives of images of the sender in the messages are stored in at least one of a memory system of the server and a local memory of a mobile terminal device.

8. The method according to claim 3, wherein the 3D model includes a plurality of 2D images, which are stored along with parameters that describe a perspective of one of the 2D images.

9. The method according to claim 1, wherein a background is selected and inserted based on an ascertained location of the sender of the message.

10. A digital messaging system having a plurality of person-related messages stored on a memory system, the messages being provided with sender addresses and at least a portion of the messages including digital data of at least one of pictures and video, the digital messaging system comprising:

an identifying unit configured to identify picture regions having at least one of faces and persons in the digital data of messages that have the same sender address;

at least one calculation unit configured to determine, from the identified picture regions, sender-relevant picture regions that represent a sender of the message based on at least one of;

a) reference pictures that are stored on the memory system,

b) a comparison of speech data in the digital data with reference speech data using speaker recognition methods and taking into account speech data in the message, and

c) a frequency of occurrence of sender-relevant picture regions in the messages that have the same sender address; and

a modification unit configured to modify the digital data of the messages that are received by emphasizing the sender-relevant picture regions.

11. The digital messaging system according to claim 10, further comprising an undoing unit configured to undo the modification by the recipient using a menu dialog.

12. The digital messaging system according to claim 10, further comprising:

a first digital interface configured to store a sender-relevant 2D picture region in conjunction with the sender address and parameters that characterize a perspective of a sender image;

a second digital interface configured to transfer the sender-relevant 2D picture region and further sender-relevant picture regions from past messages from the memory system to a module for generating 3D picture data, the calculation unit being configured to determine a perspective of the sender-relevant 2D picture region based on imaged body proportions; and

a search unit configured to search for a second 2D image having parameters that describe a different perspective in the same horizontal plane so as to provide a 3D impression through evaluation of ratios of body proportions;

wherein the calculation unit is configured to generate at least one 3D image of the sender of the message by supplementing the sender-relevant 2D picture region with a second picture region including a sender image that is copied from the second 2D image, cut out and replaced by a picture from the different perspective.

13. The digital messaging system according to claim 12, wherein the search unit is configured to search first for the second 2D image in a current message using face recognition methods, when a plurality of persons are present, and, if no usable one of the picture from the different perspective are present in the current message, then the search unit searches an archive memory of the digital messaging system, in which case, proportions of features of the sender image of additionally found pictures from a different perspective are determined for compliance with predefined limit values that describe a horizontal recording plane and a vertical recording plane.

14. The digital messaging system according to claim 10, wherein the calculation unit is configured to compare at least one of the speech data of the message with the speech data of the video message and the speech data of a plurality of the video messages with one another, so as to find the sender-relevant picture region in the video message, if there is an overlap of voice messages.

15. The digital messaging system according to claim 10, wherein the calculation unit is disposed at least partially on at least one of a terminal device and a server accessible via a network, wherein parameters that characterize different perspectives of sender-relevant picture regions of the messages are stored in at least one of a memory system of the server and a memory of a mobile terminal device.

16. The digital messaging system according to claim 12, wherein the 3D image includes a plurality of 2D images, which are stored on the memory system in conjunction with data that describes the perspective of one of the 2D images.

17. The digital messaging system according to claim 10, further comprising an insertion system configured to insert a background based on an ascertained whereabouts of the sender of the message.

18. A method for controlling a digital messaging system in which a plurality of person-related messages are stored on a memory system, the messages being provided with sender addresses and at least a portion of the messages further comprising digital data of at least one of pictures and video, the method comprising:

identifying picture regions including at least one of faces and persons in the digital data of the messages on the digital messaging system having the same sender address;

determining sender-relevant picture regions in the identified picture regions that represent a sender of the message based on at least one of;

a) a comparison with a reference picture of the sender stored on the memory system,

b) a comparison of speech data in the digital data with reference speech data using at least one of speaker recognition methods, speaker verification methods and speaker identification methods, taking into account picture data and speech data in the message and

c) a frequency of occurrence of the identified picture regions in the messages that have the same sender address; and

generating a digital, graphic model of a speaker using the sender-relevant picture region so as to reproduce content of the message through animation of mouth regions in the reproduction of the message.

19. The method according to claim 18, wherein the identified picture regions are used to generate a 3D model.

20. The method according to claim 19, further comprising:

storing a sender-relevant 2D picture region of the identified picture regions along with the sender address and parameters that characterize a first perspective of the sender image;

transferring the sender-relevant 2D picture region and further sender-relevant picture regions from past messages from the memory system to a module for generating 3D picture data;

determining the first perspective of the sender-relevant 2D picture region based on imaged body proportions;

searching for a second 2D image having parameters that characterize a second perspective in the same horizontal plane so as to provide a 3D impression by evaluating ratios of body proportions;

adapting a size and color of the second 2D image; and

generating at least one 3D image of the sender by supplementing the sender-relevant 2D picture region with a second picture region including a sender image that is copied from the second 2D image, cut out and replaced by a picture from a different perspective.

21. The method according to claim 20, searching for the second 2D image is first performed in a current message for the picture from the different perspective using face recognition methods in the case of a plurality of persons being present and, if a usable one of the picture from the different perspective is not found in the current message, then searching is performed on the memory system, in which case, proportions of features of the sender image are checked in additionally found ones of the picture from the different perspective to determine compliance with predefined limit values that describe a horizontal recording plane and a vertical recording plane.

22. The method according to claim 18, wherein at least one of comparing the speech data in the message with the speech data of a video message and comparing the speech data of a plurality of video messages with one another, is performed so as to find the sender-relevant picture regions of the video messages when there is an overlap of voice messages

23. The method according to claim 18, wherein at least one of a terminal device and a server accessible via a network is provided for a 3D generation, the speaker recognition methods and a face recognition, wherein parameters that characterize different perspectives of images of the sender in the messages are stored in at least one of a memory system of the server and a local memory of a mobile terminal device.

24. The method according to claim 18, wherein the 3D model includes a plurality of 2D images, which are stored along with parameters that describe a perspective of one of the 2D images.

25. The method according to claim 18, wherein a background is selected and inserted based on an ascertained whereabouts of the sender of the message.

26. A digital messaging system having a plurality of person-related messages are stored on a memory system, the messages being provided with sender addresses and at least a portion of the messages including digital data of at least one of pictures and video, the digital messaging system comprising:

an identifying unit configured to identify picture regions having at least one of faces and persons in the digital data of messages that have the same sender address;

at least one calculation unit configured to determine, from the identified picture regions, sender-relevant picture regions that represent a sender of the message based on at least one of;

a) reference pictures that are stored on the memory system,

b) a comparison of speech data in the digital data with reference speech data using speaker recognition methods and taking into account speech data in the message, and

c) a frequency of occurrence of sender-relevant picture regions in the messages that have the same sender address; and

a generation unit configured to generate a digital, graphic model of a speaker using the sender-relevant picture region so as to reproduce content of the message through animation of mouth regions.

27. The digital messaging system according to claim 26, wherein the identified picture regions are used to generate a 3D model.

28. The digital messaging system according to claim 26, further comprising:

a first interface configured to store a sender-relevant 2D picture region in conjunction with the sender address and parameters that characterize a perspective of a sender image;

a second interface configured to transfer the sender-relevant 2D picture region and further sender-relevant picture regions from past messages from the memory system to a module for generating 3D picture data, the calculation unit being configured to determine a perspective of the sender-relevant 2D picture region based on imaged body proportions; and

a search unit configured to search for a second 2D image having parameters that describe a different perspective in the same horizontal plane so as to provide a 3D impression through evaluation of ratios of body proportions, and

wherein the calculation unit is configured to generate at least one 3D image of the sender of the message by supplementing the sender-relevant 2D picture region with a second picture region including a sender image that is copied from the second 2D image, cut out and replaced by a picture from the different perspective.

29. The digital messaging system according to claim 28, wherein the search unit is configured to search first for the second 2D image in a current message using face recognition methods, when a plurality of persons are present, and, if no usable one of the picture from the different perspective are present in the current message, then the search unit searches an archive memory of the digital messaging system, in which case, proportions of features of the sender image of additionally found pictures from a different perspective are determined for compliance with predefined limit values that describe a horizontal recording plane and a vertical recording plane.

30. The digital messaging system according to claim 26, wherein the calculation unit is configured to compare at least one of the speech data of the message with the speech data of the video message and the speech data of a plurality of the video messages with one another, so as to find the sender-relevant picture region in the video message, if there is an overlap of voice messages.

31. The digital messaging system according to claim 26, wherein the calculation unit is disposed at least partially on at least one of a terminal device and a server accessible via a network, wherein parameters that characterize different perspectives of sender-relevant picture regions of the messages are stored in at least one of a memory system of the server and a memory of a mobile terminal device.

32. The digital messaging system according to claim 26, wherein the 3D image includes a plurality of 2D images, which are stored on the memory system in conjunction with data that describes the perspective of one of the 2D images.

33. The digital messaging system according to claim 26, further comprising an insertion system configured to insert a background based on an ascertained whereabouts of the sender of the message.