Voice-Based Image Tagging and Searching

• US 20130346068A1
• Filed: 03/13/2013
• Published: 12/26/2013
• Est. Priority Date: 06/25/2012
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Solem, Jan Erik, Stalenhoef, Thijs Willem
• Application Number:
  US13/801,534
• Publication Number:
  US 20130346068A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

The electronic device with one or more processors and memory provides a digital photograph of a real-world scene. The electronic device provides a natural language text string corresponding to a speech input associated with the digital photograph. The electronic device performs natural language processing on the text string to identify one or more terms associated with an entity, an activity, or a location. The electronic device tags the digital photograph with the one or more terms and their associated entity, activity, or location.

Claims

1. A method for tagging or searching images using a voice-based digital assistant, comprising:

at an electronic device with a processor and memory storing instructions for execution by the processor;

providing a digital photograph of a real-world scene;

providing a natural language text string corresponding to a speech input associated with the digital photograph;

performing natural language processing on the text string to identify one or more terms associated with an entity, an activity, or a location; and

tagging the digital photograph with the one or more terms and their associated entity, activity, or location.

2. The method of claim 1, further comprising:

receiving the speech input; and

converting the speech input into the text string.

3. The method of claim 1, wherein the entity is selected from the group consisting of:

an object and a person.

4. The method of claim 1, wherein the natural language processing comprises:

determining whether each of the one or more terms in the text string is one of an entity, an activity, and a location.

5. The method of claim 1, wherein natural language processing comprises disambiguating ambiguous terms.

6. The method of claim 5, wherein disambiguating comprises:

identifying that a first term of the one or more terms has multiple candidate meanings;

prompting a user for additional information about the first term;

receiving the additional information from the user in response to the prompt; and

identifying the entity, activity, or location associated with the first term in accordance with the additional information

7. The method of claim 6, wherein prompting the user for additional information comprises providing a voice prompt to the user.

8. The method of claim 1, further comprising displaying, at a client device, the one or more terms on or near the digital photograph.

9. The method of claim 8, wherein the one or more terms are displayed on the digital photograph in spatial proximity to their corresponding entity, activity, or location.

10. The method of claim 1, further comprising storing the one or more terms and their associated entity, activity, or location in association with at least one of the digital photograph or a representation of the digital photograph.

11. The method of claim 1, wherein:

the electronic device is a handheld electronic device; and

providing the digital photograph comprises retrieving the digital photograph from a plurality of digital photographs stored on the handheld electronic device.

12. The method of claim 1, wherein:

the electronic device is a handheld electronic device; and

providing the digital photograph comprises capturing the digital photograph at the handheld electronic device using a camera.

13. The method of claim 1, wherein:

the electronic device is a handheld electronic device; and

the speech input is acquired at the handheld electronic device using one or more microphones.

14. The method of claim 1, the natural language processing comprising:

identifying one of the one or more terms as a pronoun; and

determining a noun to which the pronoun refers.

15. The method of claim 14, wherein the noun is a name of an entity, an activity, or a location identified in a previous speech input associated with a previously tagged digital photograph.

16. The method of claim 14, wherein the noun is a name of a person identified using a contact list associated with a user of the electronic device.

17. The method of claim 14, wherein the noun is a name of a person identified based on a previous speech input associated with a previously tagged digital photograph.

18. The method of claim 1,

wherein the electronic device is a handheld electronic device;

and

wherein performing the natural language processing on the text string further comprises accessing information obtained from one or more sensors of the handheld electronic device for determining a meaning of one or more of the terms, wherein the one or more sensors are selected from the group consisting of;

a proximity sensor, a light sensor, a GPS receiver, a temperature sensor, and an accelerometer.

19. The method of claim 1, further comprising:

providing an additional digital photograph;

determining that the additional digital photograph is graphically similar to the digital photograph in one or more respects; and

suggesting to a user that the additional digital photograph be tagged with the one or more terms and their associated entity, activity, or location identified with respect to the digital photograph.

20. The method of claim 19, further comprising receiving an input from the user indicating that the additional digital photograph should be tagged in accordance with the suggestion.

21. The method of claim 20, wherein determining that the additional digital photograph is graphically similar to the digital photograph in one or more respects comprises:

generating a first fingerprint of the digital photograph;

generating a second fingerprint of the additional digital photograph; and

determining that the first fingerprint and the second fingerprint match to within a predetermined threshold.

22. The method of claim 21, wherein the first fingerprint is a fingerprint of a graphical feature within the digital photograph, and wherein the second fingerprint is a fingerprint of a graphical feature within the additional digital photograph.

23. The method of claim 1, wherein the natural language processing identifies two terms each associated with one of an entity, an activity, or a location, and the digital photograph is tagged with the two terms and their respective associated entity, activity, or location.

24. The method of claim 23, wherein a first of the two terms refers to a person, and a second of the two terms refers to a location.

25. The method of claim 1, wherein the natural language processing identifies three terms each associated with one of an entity, an activity, or a location, and the digital photograph is tagged with the three terms and their respective associated entity, activity, or location.

26. A computer system, comprising:

one or more processors; and

memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for;

providing a digital photograph of a real-world scene;

providing a natural language text string corresponding to a speech input associated with the digital photograph;

performing natural language processing on the text string to identify one or more terms associated with an entity, an activity, or a location; and

tagging the digital photograph with the one or more terms and their associated entity, activity, or location.

27. A non-transitory computer readable storage medium storing one or more programs configured for execution by an electronic device, the one or more programs comprising instructions for:

providing a digital photograph of a real-world scene;

providing a natural language text string corresponding to a speech input associated with the digital photograph;

performing natural language processing on the text string to identify one or more terms associated with an entity, an activity, or a location; and

tagging the digital photograph with the one or more terms and their associated entity, activity, or location.