Flattening and rectifying a curved image

• US 10,275,858 B2
• Filed: 01/24/2017
• Issued: 04/30/2019
• Est. Priority Date: 01/24/2017
• Status: Active Grant

• Current Assignee: Microsoft Licensing Corporation (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Turner, Jerome Joseph, Yu, Christopher Chinwen
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US15/414,230
• Publication Number:
  US 20180211366A1
• Time in Patent Office: 826 Days
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/13 Edge detection

Abstract

Representative embodiments disclose mechanisms to flatten an image comprising at least one curved or bent edge. An image is received from a capture device and a UI is presented that allows a user to define a general vicinity of an item in the image to be flattened. The system analyzes pixels in the general vicinity to identify the highest probability of a corner in each general vicinity. The gradient of pixels between the corners is calculated and pixels scores are calculated based on the gradient. The pixels with the lowest scores between each corner are used as an edge. Smoothing is applied to each edge. Dimensions of the flat representation are calculated and the system maps pixels bounded by the edges and corners to pixels in the flat representation and copies the mapped pixels to the flat representation.

Claims

1. A method performed by a processor for flattening an image comprising:

receiving, at the processor, an image comprising an object having a non-linear side;

identifying a plurality of corners of the object, the plurality of corners defining a region of the image to be flattened;

defining a gradient between corresponding pairs of each of the plurality of corners;

define a score for pixels between based on the gradient;

using the scores to identify an edge between each of the plurality of corners, one edge approximating the non-linear side;

responsive to identifying the edge between each of the plurality of corners, establishing dimensions of a flat representation;

mapping each pixel in the region of the image to a pixel in the flat representation; and

filling each pixel in the flat representation with the color of the corresponding pixel in the region of the image; and

outputting the image to a display.

2. The method of claim 1 further comprising smoothing each edge.

3. The method of claim 1 wherein identifying the corners of the object comprises:

receiving an indication from a user of a general vicinity of each corner;

analyzing pixels in the general vicinity to identify each corner.

4. The method of claim 3 wherein the indication comprises a widget and wherein the method further comprises:

receiving a location of the widget, the widget defining the general vicinity;

giving a higher weight to pixels at a center of the widget when analyzing pixels to identify a corner.

5. The method of claim 4 wherein the widget is placed by a user.

6. The method of claim 1 wherein the score for a pixel is calculated based on the gradient of the pixel and the gradient of a surrounding pixel.

7. The method of claim 6 wherein identifying the edge comprises:

identifying a current pixel representing a corner;

checking pixels surrounding the current pixel and picking as a next pixel in the edge a pixel with the lowest score;

setting the current pixel as the next pixel;

repeating the checking and setting operations until a second corner pixel is reached.

8. The method of claim 1 wherein mapping and filling comprise:

establishing a percentage in an X and Y dimension in the region of the image; and

copying the color of the pixel in the region of the image at the percentage in the X and Y dimension to a pixel in the flat representation in at the percentage in the X and Y dimension.

9. A system for flattening an image comprising:

a processor and executable instructions accessible on a computer-storage medium that, when executed, cause the processor to perform operations comprising;

receive an image comprising an object having a non-linear side;

identify a plurality of corners of the object, the plurality of corners defining a region of the image to be flattened;

define a gradient between corresponding pairs of each of the plurality of corners;

define a score for pixels based on the gradient;

use the scores to identify an edge between corresponding pairs of each of the plurality of corners, one edge approximating the non-linear side;

establish the dimensions of a flat representation;

map each pixel in the region of the image to a pixel in the flat representation; and

fill each pixel in the flat representation with the color of the corresponding pixel in the region of the image.

10. The system of claim 9 further comprising smooth each edge.

11. The system of claim 9 further comprising:

receive a location of a widget, the widget defining a general vicinity of a corner;

analyze pixels in the general vicinity and give a higher weight to pixels at a center of the widget to identify the corner.

12. The system of claim 11 wherein the widget is placed by a user.

13. The system of claim 9 wherein the score for a pixel is calculated based on the gradient of the pixel and the gradient of a surrounding pixel.

14. The system of claim 13 wherein identifying the edge comprises:

identify a current pixel representing a corner;

check pixels surrounding the current pixel and picking as a next pixel in the edge a pixel with the lowest score;

set the current pixel as the next pixel;

repeat the checking and setting operations until a second corner pixel is reached.

15. The system of claim 9 wherein mapping each pixel in the region of the image to the pixel in the flat representation comprises:

establish a percentage in an X and Y dimension in the region of the image; and

copy the color of the pixel in the region of the image at the percentage in the X and Y dimension to a pixel in the flat representation in at the percentage in the X and Y dimension.

16. A computer storage medium comprising executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receive an image captured by a capture device, the image comprising an object having a non-linear side;

identify a plurality of corners of the object, the plurality of corners defining a region of the image to be flattened;

calculate a gradient value for pixels between each of the plurality of corners;

assign a score to each of the pixels based on the gradient value;

use the score to identify an edge between each of the plurality of corners, one edge approximating the non-linear side;

establish dimensions of a flat representation;

map each pixel in the region of the image to a pixel in the flat representation; and

fill each pixel in the flat representation with the color of the corresponding pixel in the region of the image.

17. The medium of claim 16 further comprising:

receive a location of a widget placed by a user, the widget defining a general vicinity of a corner;

analyze pixels in the general vicinity and give a higher weight to pixels at a center of the widget to identify the corner.

18. The medium of claim 17 further comprising:

identify a current pixel representing a corner;

check pixels surrounding the current pixel and pick as a next pixel in the edge a pixel with the lowest score;

set the current pixel as the next pixel;

repeat the check and set operations until a second corner pixel is reached.