Camera-based scanning

• US 8,345,106 B2
• Filed: 10/13/2009
• Issued: 01/01/2013
• Est. Priority Date: 09/23/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Nijemcevic, Djordje, Vukosavljevic, Magdalena, Vugdelija, Milan, Mitrovic, Ana, Petschnigg, Georg F., Dresevic, Bodin
• Primary Examiner(s): Misleh, Justin P
• Application Number:
  US12/578,445
• Publication Number:
  US 20110069180A1
• Time in Patent Office: 1,176 Days
• Field of Search: None
• US Class Current: 348/207.1
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: H04N 1/00251 with an apparatus for taking photographic images, e.g. a camera
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Embodiments of camera-based scanning are described. In various embodiments, scanned documents can be created using images captured by a camera associated with a device. An image captured by the camera is processed to identify portions within the image that correspond to rectangular objects such as paper, business cards, whiteboards, screens, and so forth. One or more of these portions can be selected for scanning automatically based on a scoring scheme and/or semi-automatically with the aid of input from a user. One or more scanned documents are created from the selected portions by un-warping the selected portions to remove effects of perspective (e.g., rectify the portions to rectangles) and applying various image enhancements to improve appearance.

Claims

1. A method implemented by a computing device, the method comprising:

capturing an image responsive to initiation of camera-based scanning of the image;

automatically selecting one or more quadrangular objects in the captured image for scanning;

determining whether the automatic selection of the one or more quadrangular objects is successful based on the one or more quadrangular objects having an associated score that is compared to a predefined threshold;

in an event that the automatic selection is unsuccessful based on the one or more quadrangular objects having an associated score that fails to exceed the predefined threshold, employing semi-automatic correction based upon user input to make a user modification of the automatic selection; and

creating one or more scanned documents from portions of the image corresponding to the selected one or more quadrangular objects, including correcting the portions for perspective distortion of the one or more quadrangular objects in the captured image.

2. A method as recited in claim 1, wherein automatically selecting the one or more quadrangular objects further comprises:

detecting edges in the captured imaged based upon visual differences; and

determining the one or more quadrangular objects as combinations of the detected edges.

3. A method as recited in claim 2, further comprising:

employing Canny'"'"'s algorithm to detect the edges and a Hough transform to form potential quadrangles from the identified edges; and

applying a scoring scheme to the potential quadrangles to determine the one or more quadrangular objects and identify the associated score of the one or more quadrangular objects.

4. A method as recited in claim 1, further comprising exposing a user interface having a portion to display the captured image and another portion to concurrently display at least one scanned document created from the captured image.

5. A method as recited in claim 4, wherein the user interface is configured to:

present indicators within the captured image to identify the selected one or more quadrangular objects;

enable user interaction with the indicators to make a user modification of the automatic selection of the one or more quadrangular objects; and

responsive to user interaction with the indicators, update and display the at least one scanned document created in accordance with the user modification.

6. A method as recited in claim 1, further comprising:

if the automatic selection is successful, performing said creating the one or more scanned documents using the automatic selection; and

if the automatic selection is unsuccessful, performing said creating the one or more scanned documents using quadrangular objects in the captured image selected through the semi-automatic correction.

7. A method as recited in claim 6, wherein employing the semi-automatic correction comprises:

receiving user input to select a location in the captured image; and

automatically deriving a corresponding quadrangle based upon the user input, the selected location being one corner of the corresponding quadrangle.

8. A method as recited in claim 1, further comprising applying one or more visual enhancements to improve appearance of the one or more scanned documents.

9. A method as recited in claim 1, wherein correcting the portions for perspective distortions comprises rectifying each of the portions of the captured image to correspond to a rectangle.

10. A portable computing device, comprising:

a camera;

one or more processors coupled to memory; and

a capture application stored in the memory and executable via the one or more processors to cause the portable computing device to perform camera-based scanning of an image captured via the camera by at least;

selecting at least one quadrangle for scanning from the captured image, the at least one quadrangle corresponding to a rectangular object in the captured image, the selecting including;

conducting an automatic selection to identify the at least one quadrangle based upon a detection of one or more potential quadrangles in the captured image;

determining whether the automatic selection is successful by at least determining whether the at least one quadrangle has an associated score that exceeds a predetermined threshold;

if the automatic selection is successful based on the associated score exceeding the predetermined threshold, selecting the at least one quadrangle identified through the automatic selection;

if the automatic selection is unsuccessful based on the associated score failing to exceed the predetermined threshold, initiating a semi-automatic correction to obtain a user selection to identify the at least one quadrangle and selecting the at least one quadrangle identified through the semi-automatic selection; and

processing the captured image to produce a scanned document corresponding to the selected at least one quadrangle.

11. A portable computing device as recited in claim 10, wherein the capture application is further configured to cause the portable computing device to perform the detection of the one or more potential quadrangles by at least:

detecting edges in the captured image;

determining lines corresponding to the detected edges; and

deriving potential quadrangles by combining the detected edges and lines.

12. A portable computing device as recited in claim 11, wherein detecting the edges comprises applying an algorithm to find sharp changes of brightness corresponding to the edges.

13. A portable computing device as recited in claim 11, wherein determining lines corresponding to the detected edges comprises applying a linear transform to correct for imperfections in the detected edges.

14. A portable computing device as recited in claim 10, wherein conducting the automatic selection includes applying a scoring scheme to the one or more potential quadrangles detected in the captured image that accounts for at least a location of the quadrangles in the image and relative size of the quadrangles in the captured image.

15. A portable computing device as recited in claim 10, wherein processing the captured image to produce the scanned document comprises:

cropping the captured image to correspond to the at least one quadrangle;

un-warping the cropped image to correct for perspective distortions; and

applying one or more visual enhancements to the cropped image.

16. A portable computing device as recited in claim 10, wherein the capture application is further configured to cause the portable computing device to output a user interface having a portion to display the captured image and another portion to concurrently display the scanned document produced by the processing of the captured image.

17. Computer-readable media having stored thereon computer-executable instructions that, if executed by a computing device, initiate the computing device to:

process a captured image to create a scanned document from a selected portion of the captured image that corresponds to a rectangular object in the captured image, the selected portion being automatically selected by the computing device based on the selected portion having an associated score that exceeds a predefined scoring threshold;

initiate a display of a user interface configured to present a split screen having the captured image on one side of the split screen and the scanned document created from the captured image on an other side of the split screen;

enable user interaction with the captured image via the one side of the split screen to modify the selected portion of the captured image; and

update the scanned document displayed in the other side of the split screen to reflect a modification made through the user interaction.

18. Computer-readable media as recited in claim 17, wherein the image is captured via a camera of the computing device.

19. Computer-readable media as recited in claim 18, wherein the computing device is a portable computing device.

20. Computer-readable media as recited in claim 17, wherein the image is captured by a camera separate from the computing device and transferred to the computing device configured to process the captured image to create the scanned document.