System and method for detecting objects in an image

• US 9,754,163 B2
• Filed: 11/07/2016
• Issued: 09/05/2017
• Est. Priority Date: 06/22/2015
• Status: Active Grant

• Current Assignee: Photomyne Ltd.
• Original Assignee: Photomyne Ltd.
• Inventors: Segalovitz, Yair, Shoor, Omer, Lipman, Yaron, Tzemah, Nir, Verter, Natalie
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US15/344,605
• Publication Number:
  US 20170083762A1
• Time in Patent Office: 302 Days
• Field of Search: 382156-159
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A method for cropping photos images captured by a user from an image of a page of a photo album is described. Corners in the page image are detected using corner detection algorithm or by detecting intersections of line-segments (and their extensions) in the image using edge, corner, or line detection techniques. Pairs of the detected corners are used to define all potential quads, which are then are qualified according to various criteria. A correlation matrix is generated for each potential pair of the qualified quads, and candidate quads are selected based on the Eigenvector of the correlation matrix. The content of the selected quads is checked using a salience map that may be based on a trained neuron network, and the resulting photos images are extracted as individual files for further handling or manipulation by the user.

Claims

1. A method for detecting one or more rectangular-shaped object regions from a background in a captured image, the method comprising:

obtaining the captured image by a digital camera;

analyzing the captured image using a deep convolutional neural network for detecting the object regions;

enhancing the image in each of the detected regions; and

cropping or extracting from the captured image each of the enhanced detected regions into a respective file,

wherein the objects are rectangular-based photographs, receipts, business cards, sticky notes, printed newspapers, or stamps, and

wherein the neural network is further trained to recognize or classify the rectangular-based object regions in the captured image.

2. The method according to claim 1, wherein further comprising training the neural network to detect the rectangular-based object regions in the captured image.

3. The method according to claim 1, wherein the neural network is further trained to recognize or classify the objects in the captured image and having multiple stages or layers, and wherein the analyzing of the content of each of the regions uses an output of an intermediate stage or layer in the neural network.

4. The method according to claim 3, wherein the neural network is ImageNet having 26 stages or layers, the intermediate stage or layer is an eighth stage or layer, the output includes 256 saliency maps, and wherein the analyzing of the content comprises generating an output map calculated by a weighted average of the 256 saliency maps.

5. A non-transitory tangible computer readable storage media comprising code to perform the steps of the method of claim 1.

6. A device housed in a single enclosure and comprising in the single enclosure the digital camera, a memory for storing computer executable instructions, and a processor for executing the instructions, the processor configured by the memory to perform acts comprising the method of claim 1.

7. The device according to claim 6, wherein the single enclosure is a portable or a hand-held enclosure and the device is battery-operated and consists of, comprises, or is part of, a notebook, a laptop computer, a media player, a cellular phone, a tablet, a Personal Digital Assistant (PDA), or an image-processing device.

8. The method according to claim 1, wherein the background comprises a pattern and the captured image comprises an entire or part of a page of a photo album.

9. The method according to claim 1, wherein the obtaining of the captured image comprises capturing the captured image by the digital camera, and wherein the digital camera is part of, or comprises, a single enclosure that is a portable or a hand-held enclosure that includes a battery for powering the digital camera, and the single enclosure further comprises a notebook, a laptop computer, a media player, a cellular phone, a tablet, a Personal Digital Assistant (PDA), or an image-processing device.

10. The method according to claim 1, wherein the obtaining of the captured image comprises fetching the captured image from a volatile memory or a non-volatile memory that consists of, or comprises, a Hard Disk Drive (HDD), a Solid State Drive (SSD), RAM, SRAM, DRAM, TTRAM, Z-RAM, ROM, PROM, EPROM, EEROM, Flash-based memory, CD-RW, DVD-RW, DVD+RW, DVD-RAM BD-RE, CD-ROM, BD-ROM, and DVD-ROM.

11. The method according to claim 1, wherein the captured image is a single image file that is in a compressed format that is according to, based on, or consists of, Portable Network Graphics (PNG), Graphics Interchange Format (GIF), Joint Photographic Experts Group (JPEG), Exchangeable image file format (Exif), Tagged Image File Format (TIFF).

12. The method according to claim 1, wherein the captured image is using a color image that is using a color model that is according to, or based on, the RGB color space.

13. The method according to claim 1, further comprising capturing a first image by the digital camera, and forming the captured image by converting the first image from the color model to a grayscale format, wherein the converting of the captured image to a grayscale format is according to, based on, or using, linearly encoding grayscale intensity from linear RGB.

14. The method according to claim 1, further comprising capturing a first image by the digital camera, and downscaling the first image to form the captured image by downscaling the first image using a downscaling algorithm, wherein the captured image is having less than 10%, 7%, 5%, 3%, or 1% pixels of the first image or wherein the captured image is having less than 10,000, 5,000, 2,000, or 1,000 pixels.

15. The method according to claim 14, wherein the downscaling algorithm is according to, is based on, or uses, an adaptive or non-adaptive image interpolation algorithm.

16. The method according to claim 15, wherein the non-adaptive image interpolation algorithm consists of, comprises, or is part of, nearest-neighbor replacement, bilinear interpolation, bi-cubic interpolation, Spline interpolation, Lanczos interpolation, or a digital filtering technique.

17. The method according to claim 1, wherein the enhancing the region image comprises generating a color balanced image by correcting the color balance of the image.

18. The method according to claim 17, wherein the region image color space is using RGB whereby each pixel is defined by (r, g, b), and the correcting of the color balance comprises:

obtaining a gray reference pixel values (rref, gref, bref);

calculating average pixel values of the region image (ravg, gavg, bavg);

calculating a color shift (rsft, gsft, bsft) of the region image according to, or based on, (rsft, gsft, bsft)=(ravg, gavg, bavg)−

(rref, gref, bref); and

calculating the color balanced image having pixels values (rc, gc, bc), where each pixel value is calculated (rc, gc, bc)=(r, g, b)−

(rsft, gsft, bsft).

19. The method according to claim 18, wherein the obtaining of a gray reference pixel values is based on, or equal to, an average of pixel values in multiple images.

20. The method according to claim 19, wherein the obtaining of the gray reference pixel values is based on, or equal to, an average of pixel values in all of the extracted or cropped regions.

21. The method according to claim 17, wherein the enhancing of the image comprises generating a color-balanced image by correcting the color balance of the region image followed by enhancing the contrast in the color-balanced image.

22. The method according to claim 21, wherein the enhancing of the contrast, comprises, uses, or is based on, a linear contrast enhancement that is Min-Max Linear Contrast Stretch, Percentage Linear Contrast Stretch, or Piecewise Linear Contrast Stretch.

23. The method according to claim 21, wherein the enhancing of the contrast, comprises, uses, or is based on, a non-linear contrast enhancement that is Histogram Equalizations, Adaptive Histogram Equalization, Unsharp Mask, or Homomorphic Filter.

24. The method according to claim 1, wherein the analyzing of the captured image further comprises producing a list of quads in the captured image.

25. The method according to claim 24, for use with plurality of detected corners in the captured image, wherein each of the quads is a quadrilateral having two or more vertices that are selected from the detected corners.

26. The method according to claim 25, further comprising detecting the corners in the captured image, wherein each corner is defined by a point location and two directions from the point.

27. The method according to claim 26, wherein the detecting of the corners is according to, is based on, or consists of, a corner detection algorithm.

28. The method according to claim 27, wherein the detecting of the corners comprises detecting straight-line segments in the captured image according to, or based on, a pattern recognition algorithm.

29. The method according to claim 24, further comprising:

estimating for each pair of quads selected from the list a correlation value corresponding to underlying similarities of the quads pair;

selecting quads from the list based on the estimated correlation values;

analyzing content of each of the regions defined by the selected quads; and

determining that the regions defined by the selected quads correspond to the objects regions.

30. The method according to claim 1, wherein the analyzing of the captured image comprises generating a saliency map identifying salient pixels or saliency region in the captured image.