MOBILE AND SERVER-SIDE COMPUTATIONAL PHOTOGRAPHY

• US 20110312374A1
• Filed: 06/18/2010
• Published: 12/22/2011
• Est. Priority Date: 06/18/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Billy, Ofek, Eyal
• Granted Patent: US 8,488,040 B2
• Time in Patent Office: Days
• US Class Current: 455/556.1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/23213 with fixed number of clusters, e.g. K-means clustering
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: H04N 1/00183 Photography assistance, e.g. displaying suggestions to the user
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00336 with an apparatus performing pattern recognition, e.g. of a face or a geographic feature image or video recognition or understanding of scenes G06V20/00
• CPC Class Codes: H04N 1/00442 Simultaneous viewing of a plurality of images, e.g. using a mosaic display arrangement of thumbnails
• CPC Class Codes: H04N 1/00477 Indicating status, e.g. of a job for control or supervision between transmitter and receiver or between image input and image output device H04N2201/3219
• CPC Class Codes: H04N 1/00488 providing an audible output to the user
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/0039 Connection via a network
• CPC Class Codes: H04N 2201/0055 By radio
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0098 : User intervention not otherwise provided for, e.g. placing documents, responding to an alarm
• CPC Class Codes: H04N 2201/3247 : Data linking a set of images to one another, e.g. sequence, burst or continuous capture mode
• CPC Class Codes: H04N 2201/3252 : Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3253 : Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 23/64 : Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

Automated photographic capture assistance and analysis is effectuated to assist users in capturing sufficient and optimal images of a desired image scene for use in a photographic end product. Photographic capture assistance is implemented on the device that includes a user'"'"'s camera. Photographic capture assistance can include audio and/or graphic information generated in real time locally on the device that includes the user'"'"'s camera and informs the user where additional images of the image scene ought to be captured and/or whether or not sufficient captured images currently exist for the image scene.

Claims

1. A method for assisting a user in capturing images of an image scene, the method comprising:

identifying at least two captured images of the image scene;

mining meta data from at least two captured images of the image scene;

executing a capture algorithm with at least two captured images, wherein the execution of the capture algorithm comprises using meta data mined from at least two captured images of the image scene to generate information identifying a region of the image scene that at least one image is recommended to be captured for; and

informing a user of the location of the region of the image scene for at least one image that is recommended to be captured.

2. The method for assisting a user in capturing images of an image scene of claim 1, wherein the user is using a camera to capture images of the image scene and wherein the camera comprises a camera lens and at least one camera sensor, the method further comprising:

obtaining information generated from at least one camera sensor;

using the information generated from at least one camera sensor to determine where the camera lens is directed; and

using the information generated from at least one camera sensor and information generated from the execution of the capture algorithm to provide instruction to a user wherein the instruction provided to the user comprises information as to a region of the image scene where the user is recommended to direct the camera lens of the camera.

3. The method for assisting a user in capturing images of an image scene of claim 2, wherein the instruction provided to the user comprises audio directions the user can follow to locate the region of the image scene where the user is recommended to direct the camera lens of the camera.

4. The method for assisting a user in capturing images of an image scene of claim 2, wherein the instruction provided to the user comprises an output display comprising a list of one or more directions the user can follow to locate the region of the image scene where the user is recommended to direct the camera lens of the camera.

5. The method for assisting a user in capturing images of an image scene of claim 2, wherein the camera lens sees a current image view comprising the region of the image scene the camera lens is presently directed at, and wherein the instruction provided to the user comprises an output display comprising the current image view seen by the camera lens with the regions of the current image view that there are existing captured images for identified in the output display.

6. The method for assisting a user in capturing images of an image scene of claim 5, wherein the regions of the current image view identified in the output display comprise those regions of the image scene that there are sufficient existing captured images for and wherein sufficient existing captured images comprises a set of at least one captured image that comprises adequate detail of a region of the image scene for use in generating a photographic end product.

7. The method for assisting a user in capturing images of an image scene of claim 6, wherein the photographic end product comprises a photosynth.

8. The method for assisting a user in capturing images of an image scene of claim 5, wherein the identification of a region of the current image view that there are existing captured images for comprises a graphic symbol imposed on top of the region of the current image view in the output display.

9. The method for assisting a user in capturing images of an image scene of claim 2, wherein the user is utilizing a camera of a mobile device to capture images of the image scene.

10. The method for assisting a user in capturing images of an image scene of claim 1, wherein the user is utilizing a camera of a mobile device to capture images of the image scene, the method further comprising:

identifying the images stored on the mobile device that comprise images of at least one region of the image scene;

mining meta data from at least two captured images of the image scene that are stored on the mobile device;

communicating with a computing device to request a search for at least one captured image of the image scene that is not currently stored on the mobile device;

receiving a captured image of the image scene that is not currently stored on the mobile device;

storing the received captured image of the image scene on the mobile device; and

executing a capture algorithm with at least two captured images wherein a captured image of the at least two captured images is the received captured image of the image scene, and wherein the execution of the capture algorithm comprises the generation of information on a region of the image scene that at least one image is recommended to be captured for.

11. The method for assisting a user in capturing images of an image scene of claim 1, wherein the user is utilizing a camera of a mobile device to capture images of the image scene, and wherein the mobile device comprises an output display comprising the capability to display photographic end products, the method further comprising:

communicating with a computing device to provide the computing device a captured image of the image scene;

receiving from the computing device a representation of a photographic end product comprising the captured image communicated to the computing device; and

outputting the representation of the photographic end product to the user via the output display.

12. The method for assisting a user in capturing images of an image scene of claim 1, wherein the user is utilizing a camera to capture images of the image scene and wherein the camera comprises a camera lens and at least one camera sensor, the method further comprising:

obtaining information generated from at least one camera sensor;

using information generated from at least one camera sensor and information generated from the execution of the capture algorithm to determine when the camera lens is directed to the location of a region of the image scene for at least one image that is recommended to be captured comprising the current image view; and

automatically capturing an image of the current image view.

13. The method for assisting a user in capturing images of an image scene of claim 1, wherein the user is utilizing a camera to capture images of the image scene and wherein the camera is a component of a mobile device that comprises a cell phone, and wherein captured images of the image scene are stored on the cell phone, the method further comprising:

executing at least one capture algorithm with captured images of the image scene to generate information on the sufficiency of the captured images for generating a photographic end product; and

informing the user that sufficient captured images of the image scene exist for generating the photographic end product wherein no additional images of the image scene are required to be captured to generate the photographic end product.

14. A method for supporting the capture of images of an image scene to be used to create a photographic end product, the method comprising:

identifying a set of at least two captured images of the image scene that are stored wherein the identified set of at least two captured images are accessible to a computing device, and wherein the set of at least two captured images that are stored comprises a set of images;

mining meta data from at least two images of the set of images;

executing at least one capture algorithm with at least two images of the set of images, wherein the execution of at least one capture algorithm comprises using mined meta data to generate information comprising the sufficiency of the set of images for creating the photographic end product; and

outputting information comprising the sufficiency of the set of images for creating the photographic end product.

15. The method for supporting the capture of images of an image scene to be used to create a photographic end product of claim 14, further comprising:

outputting at least one image of the set of images;

searching the internet for at least one image of the image scene;

retrieving at least one image of the image scene that is discovered from a search of the internet; and

outputting an image retrieved from the internet.

16. The method for supporting the capture of images of an image scene to be used to create a photographic end product of claim 14, wherein the photographic end product comprises a photosynth.

17. The method for supporting the capture of images of an image scene to be used to create a photographic end product of claim 14, further comprising:

receiving a captured image from a device that was utilized to capture the image wherein the received captured image depicts at least a segment of the image scene;

mining meta data from the received captured image, wherein the mined meta data comprises information of the image scene depicted in the received captured image;

organizing the storage of the received captured image using meta data comprising information of the image scene depicted in the received captured image;

executing a capture algorithm with at least two captured images wherein one of the at least two captured images is the received captured image, and wherein the execution of the capture algorithm comprises the generation of information identifying a region of the image scene that an image is recommended to be captured for; and

outputting information identifying a region of the image scene that an image is recommended to be captured for.

18. The method for supporting the capture of images of an image scene to be used to create a photographic end product of claim 17, wherein the information identifying a region of the image scene that an image is recommended to be captured for comprises directions for a user to utilize to locate the region of the image scene that an image is recommended to be captured for.

19. A mobile device photo capture application for assisting a user in capturing images of an image scene, the mobile device photo capture application comprising:

a component comprising the capability to direct the capture and processing of images of the image scene;

a component comprising stored captured images of the image scene;

a component comprising the capability to mine meta data from stored captured images of the image scene;

a component comprising the capability to control at least one sensor of a camera;

a component comprising the capability to analyze stored captured images of the image scene;

a component comprising the capability to generate an analysis that comprises information that identifies when there are sufficient stored captured images of the image scene to generate a photographic end product without the need to capture any additional images of the image scene;

a component comprising the capability to execute a comparison algorithm using at least two captured images of the image scene, wherein the execution of the comparison algorithm comprises the generation of information comprising an identification of a region of the image scene that at least one image is recommended to be captured for; and

a component comprising the capability to output a display comprising information that can be referred to to locate a region of the image scene that at least one image is recommended to be captured for.

20. The mobile device photo capture application for assisting a user in capturing images of an image scene of claim 19, further comprising:

a component comprising the capability to determine when the camera lens of a camera is directed to a region of the image scene that at least one image is recommended to be captured for; and

a component comprising the capability to automatically capture an image when it is determined that the camera lens of the camera is directed to a region of the image scene that at least one image is recommended to be captured for.