METHOD OF PROVIDING VISUAL SOUND IMAGE AND ELECTRONIC DEVICE IMPLEMENTING THE SAME

• US 20160054895A1
• Filed: 08/17/2015
• Published: 02/25/2016
• Est. Priority Date: 08/21/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: LEE, Younggyun, PARK, Dooyong, SEO, Eunjung, LEE, Jaehak
• Granted Patent: US 10,684,754 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

A method of providing a visual sound image, which may generate, edit, and play back a visual sound image in which sound data is linked to an image, and an electronic device implementing the same are provided. The method includes an electronic device including a display, an image including at least one object on the display, receiving, by the electronic device, a selection of at least a certain area of the object in the image displayed on the display or a selection of a certain area of the image, and linking, by the electronic device, sound data to the at least the certain area of the object or the certain area of the image. In addition, various embodiments are possible.

Claims

1. A method of operating an electronic device, the method comprising:

displaying, by the electronic device, an image including at least one object on a display;

receiving, by the electronic device, a selection of at least a certain area of the object in the image displayed on the display or a selection of a certain area of the image; and

linking, by the electronic device, sound data to the at least the certain area of the object or the certain area of the image.

2. The method of claim 1, further comprising:

displaying, by the electronic device, at least one of a symbol, an icon, text, a photo, and a picture, which indicates linkage of the sound data, in the image on the display.

3. The method of claim 1, wherein the selecting of the at least one certain area of the object or the certain area of the image comprises at least one of:

selecting at least a certain area of the object or a certain area of the image, based on a user gesture;

detecting a subject from the image, and selecting a certain area of the image to enable the detected subject to be included;

tracking a line of sight of a photographer, and selecting a certain area of the image to enable a subject located in the line of sight of the photographer to be included;

tracking a lip movement of a subject, and selecting a certain area of the image to enable a subject of which a lip moves to be included based on a result of tracking;

recognizing a direction where a sound occurs when occurrence of a sound is detected, and selecting a certain area of the image to enable a subject that is located in the direction where the recognized sound occurs to be included;

recognizing a speaker associated with the sound data and recognizing a subject included in the image, and selecting a certain area of the image to enable a subject corresponding to the recognized speaker to be included, based on a result of the speaker recognition and a result of the subject recognition;

or

when a touch on a photographing menu is detected in a preview screen, displaying at least one icon corresponding to at least one subject included in the preview screen, around the photographing menu, and when the photographing menu is dragged to a location where a predetermined icon is displayed, selecting a certain area of a subject corresponding to the predetermined icon.

4. The method of claim 3, further comprises:

when the subject located in the direction where the recognized sound occurs includes a plurality of persons, grouping the plurality of recognized persons into a group and linking sound data to the group.

5. The method of claim 1, further comprising:

when the selected certain area includes a person, determining information associated with the person through facial recognition and displaying the information.

6. The method of claim 1, further comprising:

when sound data is linked to a person, designating the sound data linked to the person as a notification sound that indicates reception of information associated with the person.

7. The method of claim 1, wherein the linking of the sound data comprises one of:

selecting one of sound data stored in advance, and executing linkage;

generating new sound data using a recording function, and linking the generated sound data to a corresponding area; and

linking address information of a webpage where sound data exists.

8. The method of claim 1, wherein the linking of the sound data further comprises:

displaying the selected area to be visually distinguished from the remaining area of the image.

9. The method of claim 1, wherein the linking of the sound data comprises:

visualizing and displaying the sound data;

selecting a certain section of the visualized sound data; and

linking sound data corresponding to the selected certain section to at least a certain area of the object and a certain area of the image.

10. The method of claim 9, further comprising:

recognizing a speaker for each section of the visualized sound data;

recognizing a face of a person included in the image; and

comparing a result of the speaker recognition and a result of the facial recognition, and displaying a facial image of a matched speaker for each section of the visualized sound data.

11. The method of claim 2, further comprising at least one of:

playing back sound data linked to a corresponding area when an area to which the sound data is linked, or the symbol, the icon, the text, the photo, or the picture is selected; and

when playback of the entire sound data linked to the image is requested, sequentially playing back at least one piece of sound data that is linked to the image, in a sequence designated by a user, in a sequence of linkage of sound data, or in random sequence.

12. The method of claim 11, wherein the sequential playing back comprises:

changing a playback sequence of sound data based on information associated with a receiver that receives an image to which the sound data is linked, and executing playback based on a changed playback sequence.

13. The method of claim 11, wherein the sequential playing back further comprises:

highlighting and displaying an area to which currently played back sound data is linked.

14. The method of claim 1, further comprising at least one of:

when a plurality of pieces of sound data having different qualities are linked to a single area, selecting and playing back one of the plurality of pieces of sound data based on a capability of an audio processing unit;

checking a communication speed when image data to which sound data is linked is requested to be transmitted to another electronic device, and selecting and transmitting one of the plurality of pieces of sound data based on the communication speed; and

when a slideshow associated with a plurality of images to which sound data is linked is requested, changing a slide interval based on a playback time of sound data linked to each image.

15. The method of claim 1, further comprising at least one of:

editing linked sound data in response to editing of an image to which the sound data is linked;

or

when a plurality of images to which sound data is linked is composed, composing the sound data that is linked to the plurality of images.

16. An electronic device comprising:

a display unit configured to display an image including at least one object;

a controller configured to;

receive a selection of at least a certain area of the object in the displayed image or a certain area of the image, and

generate a visual sound image by linking the selected certain area of the object or the certain area of the image to sound data; and

a storage unit configured to store the visual sound image.

17. The electronic device of claim 16, wherein the controller is further configured to display, on the image, at least one of a symbol, an icon, text, a photo, and a picture indicating a linkage of the sound data.

18. The electronic device of claim 16, where the controller includes an area selecting unit configured to at least one of:

select at least a certain area of the object or a certain area of the image, based on a gesture of a user;

detect a subject from the image, and select a certain area of the image to enable the detected subject to be included;

track a line of sight of a photographer, and select a certain area of the image to enable a subject located in the line of sight of the photographer to be included;

track a lip movement of a subject, and select a certain area of the image to enable a subject of which a lip moves to be included;

recognize a direction where a sound occurs when occurrence of a sound is detected, and select a certain area of the image to enable a subject located in the recognized direction where the sound occurs to be included;

recognize a speaker associated with sound data and recognize a subject included in the image, and select a certain area of the image to enable a subject corresponding to the recognized speaker to be included, based on a result of the speaker recognition and a result of the subject recognition;

or

when a touch on a photographing menu is detected in a preview screen, display, around the photographing menu, at least one icon corresponding to at least one subject included in the preview screen, and when the photographing menu is dragged to a predetermined icon, select a certain area of a subject corresponding to the predetermined icon.

19. The electronic device of claim 18, wherein, when a subject located in the direction where the recognized sound occurs includes a plurality of persons, the controller is further configured to group the plurality of recognized persons into a group and link the sound data to the group.

20. The electronic device of claim 16, wherein the controller is further configured to execute a control to recognize information associated with a subject included in the certain area, and to display information associated with the recognized subject around the subject.

21. The electronic device of claim 16, wherein the controller is further configured to designate sound data linked to a person included in the visual sound image, as a notification sound indicating reception of information associated with the person.

22. The electronic device of claim 16, wherein the controller includes a linking unit configured to link one of sound data stored in advance, sound data newly generated using a recording function, and address information of a webpage where sound data exists, to at least certain area of the object or a certain area of the image.

23. The electronic device of claim 16, wherein the controller is further configured to control the display unit to display a selected area of the image to be visually distinguished from the remaining area of the image.

24. The electronic device of claim 16, wherein the controller is further configured to visualize and display sound data when the sound data is linked, receive a selection of a certain section of the visualized sound data, and link sound data corresponding to the certain section to at least a certain area of the object or a certain area of the image, in response to user'"'"'s request.

25. The electronic device of claim 24, wherein the controller is further configured to:

recognize a speaker for each section of the visualized sound data;

recognize a face with respect to a person included in the image; and

compare a result of the speaker recognition and a result of the facial recognition, and display a facial image of a matched speaker for each section of the visualized sound data.

26. The electronic device of claim 17, wherein the controller is further configured to play back sound data linked to a corresponding area when an area to which the sound data is linked, or the symbol, the icon, the text, the photo, or the picture is selected, and sequentially play back at least one piece of sound data linked to the image in a sequence designated by a user, in a sequence of linkage of sound data, or in random sequence, when playback of the entire sound data linked to the image is requested.

27. The electronic device of claim 26, wherein the controller is further configured to determine information associated with a receiver that receives an image to which the sound data is linked when the playback of the entire sound data is requested, change a sequence of playback of the entire sound data based on the determined receiver information, and play back the same.

28. The electronic device of claim 26, wherein the controller is further configured to highlight and display an area to which currently played back sound data is linked when the entire sound data is sequentially played back.

29. The electronic device of claim 16, wherein, when a plurality of pieces of sound data having different qualities is linked to a single area, the controller is further configured to:

select and play back one of the plurality of pieces of sound data, based on a capability of an audio processing unit;

check a communication speed when image data to which sound data is linked is requested to be transmitted to another electronic device, and select and transmit one of the plurality of pieces of sound data based on the communication speed; and

change a slide interval based on a playback time of sound data that is linked to each image when a slideshow of a plurality of images to which sound data is linked is requested.

30. The electronic device of claim 16, wherein the controller is further configured to:

edit sound data linked to the image in response to editing an image to which the sound data is linked;

or

compose, when a plurality of images to which the sound data is linked is composed, sound data linked to the plurality of images.