Annotating images with instructions

• US 8,301,996 B2
• Filed: 03/19/2009
• Issued: 10/30/2012
• Est. Priority Date: 03/19/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Chen, Billy
• Primary Examiner(s): Hong, Stephen
• Assistant Examiner(s): Kells, Asher
• Application Number:
  US12/406,979
• Publication Number:
  US 20100241946A1
• Time in Patent Office: 1,321 Days
• Field of Search: None
• US Class Current: 715/231
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance

Abstract

A method described herein includes the acts of receiving an image captured by a mobile computing device and automatically annotating the image to create an annotated image, wherein annotations on the annotated image provide instructions to a user of the mobile computing device. The method further includes transmitting the annotated image to the mobile computing device.

Claims

1. A method executed by a computer processor, the method comprising:

receiving a first image of a first object from a computing device of a first user, the first image having first annotations thereon relative to a first location of the first object, the first annotations generated by the first user, the first annotations being instructions for performing a first step in a task;

receiving a second image of the first object from the computing device of the first user, the second image having second annotations thereon relative to a second location of the first object, the second annotations generated by the first user, the second annotations being instructions for performing a second step in the task;

receiving an indication from a second user that the second user desires to complete the task;

subsequent to receiving the first image, the second image, and the indication, receiving a first image of a second object captured by a mobile computing device of the second user by way of a network, the second object corresponding to but being different than the first object in the first image;

responsive to receipt of the first image of the second object, accessing the first image of the first object;

transferring the first annotations from the first image of the first object to the first image of the second object to create a first annotated image of the second object, the first annotations positioned relative to the second object in the first image of the second object at a third location that corresponds to the first location relative to the first object in the first image of the first object;

transmitting the first annotated image of the second object to the mobile computing device of the second user by way of the network;

receiving an indication from the second user that the first step in the task has been completed but the task is incomplete;

receiving a second image of the second object from the mobile computing device by way of the network subsequent to receiving the indication from the user that the first step in the task has been completed but the task is incomplete;

transferring the second annotations from the second image of the first object to the second image of the second object to create a second annotated image of the second object, the second annotations positioned relative to the second object in the second image of the second object at a fourth location that corresponds to the second location relative to the first object in the first image; and

transmitting the second annotated image of the second object to the mobile computing device by way of the network.

2. The method of claim 1, wherein at least one of the first annotations or the second annotations comprise at least one of animation overlaid on the first image of the second object or the second image of the second object received from the mobile computing device or images overlaid on the first image of the second object or the second image of the second object from the mobile computing device.

3. The method of claim 1, further comprising transmitting at least one advertisement to the mobile computing device that pertains to at least one of the first annotations or the second annotations.

4. The method of claim 3, wherein the at least one advertisement is in the form of a selectable hyperlink.

5. The method of claim 1, wherein the mobile computing device is a portable telephone that comprises a digital camera.

6. The method of claim 1, further comprising:

retaining the first annotated image of the second object; and

using the first annotated image of the second object in connection with annotating a subsequently received image from a second mobile computing device.

7. The method of claim 1, further comprising transmitting audio instructions to the mobile computing device that correspond to at least one of the first annotations or the second annotations.

8. The method of claim 1, wherein the second image of the second object is a portion of a video stream received from the mobile computing device.

9. The method of claim 1, further comprising transmitting comments of other users corresponding to at least one of the first annotations or the second annotations to the mobile computing device by way of the network.

10. A system comprising:

a processor; and

a memory that comprises a plurality of components that are executable by the processor, the components comprising;

an annotator component that receives a first image of a first object from a mobile computing device of a first user by way of a network, wherein the first image of the first object corresponds to a multi-step task desirably undertaken with respect to the first object by the first user, wherein responsive to receiving the first image, the annotator component;

accesses a first image of a second object from a data repository, the first image of the second object previously received from a computing device of a second user, the first image of the second object comprising first annotations generated by the second user and positioned in the first image of the second object at a first location relative to the second object, the second object corresponding to but different than the first object, the first annotations corresponding to a first step of the multi-step task; and

automatically visually annotates the first image of the first object with the first annotations to generate a first annotated image of the first object, the annotator component transferring the first annotations from the first image of the second object to the first image of the first object to visually annotate the first image of the first object, the annotator component positioning the first annotations at a second location in the first image of the first object relative to the first object that corresponds to the first location in the first image of the second object; and

a transmitter component that transmits the first annotated image to the mobile computing device of the first user by way of the network, wherein subsequent to the transmitter component transmitting the first annotated image to the mobile computing device of the user, the annotator component receives an indication from the first user that the first step is complete but the multi-step task is incomplete, and wherein the annotator component receives a second image of the first object from the first user and transfers second annotations from a second image of the second object to the second image of the first object to generate a second annotated image, the second annotations corresponding to a second step of the multi-step task, and wherein subsequent to the annotator component generating the second annotated image, the transmitter component transmits the second annotated image to the mobile computing device of the first user by way of the network.

11. The system of claim 10, wherein at least one of the first annotations or the second annotations comprise image data.

12. The system of claim 10, wherein the plurality of components further comprises an advertisement component that provides an advertisement to the mobile computing device that pertains to the first object.

13. The system of claim 10, wherein the plurality of components further comprises:

a comparator component that compares the first image of the first object with a plurality of other retained images and determines an orientation of the first object in the first image of the first object with respect to one or more corresponding but different objects in the plurality of other retained images; and

a determiner component that determines the second location of the first object based at least in part upon the orientation of the first object in the first image of the first object relative to orientations of the corresponding but non-identical objects in the plurality of other retained images.

14. The system of claim 10, wherein the first annotations and the second annotations pertain to repairing the first object in the first image.

15. The system of claim 10 comprised by a server that is accessible to the mobile computing device by way of a browser.

16. The system of claim 10, wherein the plurality of components further comprises:

an extractor component that extracts a feature from the first image of the first object responsive to receiving the first image of the first object; and

a comparator component that compares the feature extracted from the first image of the first object with at least one feature in the first image of the second object to determine an orientation of the first object in the first image of the first object relative to the second object in the first image of the second object, wherein the annotator component positions the first annotations in the first image of the first object at the second location based upon the orientation of the first object in the first image of the first object relative to the second object in the first image of the second object.

17. A computer-readable storage device comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving, by way of a first network connection, a first image of a first object from a computing device of a first user, the first image of the first object comprising first annotations at a first location relative to the first object in the first image of the first object, the first annotations applied to the first image of the first object at the first location by the first user prior to receiving the first image of the first object, the first annotations being first instructions for completing a first step of a multi-step task;

receiving, by way of the first network connection, a second image of the first object from the computing device of the first user, the second image of the first object comprising second annotations at a second location relative to the first object in the second image of the first object, the second annotations applied to the second image of the first object at the second location prior to receiving the second image, the second annotations being second instructions for completing a second step of the multi-step task;

receiving a first image of a second object from a mobile computing device of a second user by way of a second network connection, the second object corresponding to but different from the first object;

receiving input from the mobile computing device that the second object in the first image of the second object corresponds to the first object;

responsive to receiving the input from the mobile computing device, extracting at least one feature from the first image of the second object;

comparing the at least one feature extracted from the first image of the second object with a corresponding feature in the first image of the first object;

determining orientation of the second object in the first image of the second object relative to the first object in the first image of the first object based upon the comparing of the at least one feature extracted from the first image of the second object with the corresponding feature in the first image of the first object;

determining a second location in the first image of the second object that corresponds to the first location in the first image of the first object based at least in part upon the determining of the orientation of the second object in the first image of the second object relative to the first object in the first image of the first object;

automatically transferring the first annotations from the first location in the first image of the first object to the second location in the first image of the second object responsive to determining the second location in the first image of the second object to create a first annotated image;

transmitting the first annotated image to the mobile computing device by way of the second network connection;

receiving a second image of the second object from the mobile computing device of the second user, wherein receiving the second image of the second object indicates that the first step in the multi-step task is complete but the task is incomplete;

transferring the second annotations from the second image of the first object to the second image of the second object subsequent to receiving the second image of the second object to generate a second annotated image; and

transmitting the second annotated image to the mobile computing device.

18. The computer-readable storage device of claim 17, wherein the first annotations and the second annotations pertain to repairing the first object and the second object.

19. The computer-readable data storage device of claim 17 comprised by a web server.

20. The computer-readable data storage device of claim 17, wherein the mobile computing device is a mobile telephone that comprises a digital camera.