AUTOCAPTIONING OF IMAGES

• US 20160189414A1
• Filed: 03/07/2016
• Published: 06/30/2016
• Est. Priority Date: 10/18/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: BAKER, Simon, RAMNATH, Krishnan
• Application Number:
  US15/063,323
• Publication Number:
  US 20160189414A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

The description relates to sentence autocaptioning of images. One example can include a set of information modules and a set of sentence generation modules. The set of information modules can include individual information modules configured to operate on an image or metadata associated with the image to produce image information. The set of sentence generation modules can include individual sentence generation modules configured to operate on the image information to produce a sentence caption for the image.

Claims

1-20. -20. (canceled)

21. A device, comprising:

a display;

a processor; and

,

storage storing computer-readable instructions which, when executed by the processor, cause the processor to;

obtain an image comprising image data and associated metadata,

receive a label from a user, the label corresponding to an individual non-human element that is visible in the image,

access previous sentence captions associated with the user,

automatically generate sentence captions for the image based at least in part on the image data, the associated metadata, the label, and the previous sentence captions, and,

present the sentence captions on the display.

22. The device of claim 21, wherein the sentence captions comprise full sentence captions or sentence fragment captions.

23. The device of claim 21, wherein the computer-readable instructions further cause the processor to develop a sentence template style based at least in part on previous sentence captions associated with the user.

24. The device of claim 23, wherein the sentence template style further comprises preferences of the user.

25. The device of claim 21, wherein the computer-readable instructions further cause the processor to access a sentence template style of the previous sentence captions.

26. The device of claim 25, wherein the computer-readable instructions further cause the processor to automatically generate the sentence captions for the image using the sentence template style as a positive parameter.

27. The device of claim 25, wherein the computer-readable instructions further cause the processor to automatically generate the sentence captions for the image using the sentence template style as a negative parameter.

28. The device of claim 25, wherein the sentence template style is selected by the user.

29. The device of claim 21, wherein the computer-readable instructions further cause the processor to automatically generate sentence captions for the image based at least in part on the image being included in a set of images.

30. The device of claim 29, wherein the previous sentence captions are associated with other images in the set and the sentence captions for the image are contextually related to the previous sentence captions.

31. A method, comprising:

capturing an image with an image sensor, the image comprising image data;

associating metadata with the image;

based at least in part on the image data and the metadata, identifying a scenario involving a human and an object of the image;

based at least in part on the scenario, identifying a relative relationship between the human and the object in the image;

producing multiple sentence captions for the image that reflect the relative relationship between the human and the object; and

presenting a display of the multiple sentence captions.

32. The method of claim 31, wherein the method is performed by a device that includes the image sensor and presents the display to a user.

33. The method of claim 31, wherein identifying the scenario further comprises identifying a scene that includes the human and the object, and wherein identifying the relative relationship further comprises identifying an orientation of the human relative to the object.

34. A device, comprising:

an image sensor configured to capture images comprising pixel data;

a processor; and

,

storage storing computer-readable instructions which, when executed by the processor, cause the processor to;

generate image information from the pixel data of an individual image captured by the image sensor and metadata associated with the image, the image containing a human and a non-human element,

obtain a relative relationship between the human and the non-human element in the image,

access a template style of a user, and

automatically generate sentence captions for the image based at least in part on the pixel data, the metadata, the relative relationship, and the template style of the user.

35. The device of claim 34, wherein the computer-readable instructions further cause the processor to determine the relative relationship between the human and the non-human element in the image.

36. The device of claim 34, wherein the computer-readable instructions further cause the processor to obtain the relative relationship from another device.

37. The device of claim 34, wherein the computer-readable instructions further cause the processor to send the pixel data and the metadata to another device for processing to determine the relative relationship.

38. The device of claim 37, wherein the another device comprises cloud-based computing resources.

39. The device of claim 34, wherein the image is associated with a set of related images, and wherein the computer-readable instructions further cause the processor to automatically generate the sentence captions by considering other sentence captions for other images in the set of related images.

40. The device of claim 39, wherein the set of related images comprises an album and the sentence captions and the other sentence captions comprise a contextually related narrative of the album.