Visual search methods and systems

• US 10,303,988 B1
• Filed: 08/15/2016
• Issued: 05/28/2019
• Est. Priority Date: 08/14/2015
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Long, Scott M., Rhoads, Geoffrey B.
• Primary Examiner(s): Werner, David N
• Application Number:
  US15/237,110
• Time in Patent Office: 1,016 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 1/005 Robust watermarking, e.g. average attack or collusion attack resistant
• CPC Class Codes: G06T 1/0092 Payload characteristic determination in a watermarking scheme, e.g. number of bits to be embedded
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2201/005 Image watermarking
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 20/95 Pattern authentication; Markers therefor; Forgery detection
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: H04N 1/00846 based on detection of a dedicated indication, e.g. marks or the like
• CPC Class Codes: H04N 1/00848 by detecting a particular original
• CPC Class Codes: H04N 1/00867 with additional data, e.g. by adding a warning message details of embedding additional information in an image H04N1/32144
• CPC Class Codes: H04N 1/0087 with hidden additional data, e.g. data invisible to the human eye
• CPC Class Codes: H04N 1/0088 Detecting or preventing tampering attacks on the reproduction system
• CPC Class Codes: H04N 1/3871 : the composed originals being of different kinds, e.g. low- and high-resolution originals
• CPC Class Codes: H04N 1/409 : Edge or detail enhancement; Noise or error suppression
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/467 : characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/676 : Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/741 : by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 : Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Certain aspects of the present technology involve automated capture of several image frames (e.g., simultaneously in a single exposure, or in a burst of exposures), and application of a data-extraction process (e.g., watermark decoding) to each such image. Other aspects of the technology involve capturing a single scene at two different resolutions, and submitting imagery at both resolutions for watermark decoding. Still other aspects of the technology involve increasing the signal-to-noise ratio of a watermark signal by subtracting one image from another. Yet other aspects of the technology involve receiving focus distance data from a camera, and employing such data in extracting information from camera imagery. Smartphone camera APIs can be employed to simplify implementation of such methods. A great number of features and arrangements are also detailed. Embodiments of such technology can simplify image processing required for data extraction, with attendant reductions in required program memory and battery power consumption. Moreover, they can enlarge a system'"'"'s operational envelope—enabling information to be extracted from imagery under conditions that lead to failure in prior art arrangements.

Claims

1. A watermark decoding method comprising the acts:

obtaining first and second images of a scene captured by a camera, the scene depicting an object having a digital watermark pattern printed thereon, the first image having a first resolution, and the second image having a second resolution less than the first resolution;

interpolating the second image to produce a third image having the first resolution;

subtracting the third image from the first image to yield a fourth, difference image; and

applying a watermark decoding process to data consisting of said fourth image, to yield a plural-bit watermark payload earlier encoded in said object;

wherein said first and second images are gathered in a single exposure, during which light is directed onto a single imaging sensor via a single optic path.

2. An image processing method comprising the acts:

through an API of a device operating system, instructing a device camera to capture a first single exposure of a scene, the scene including an object, the object having artwork printed thereon, the artwork having been encoded to steganographically convey a digital watermark payload, said single exposure being characterized by collection of light directed onto a single image sensor during a single light collection interval;

through said API, also directing first and second different versions of said single exposure to be stored in first and second memories, respectively; and

processing data stored in both the first and second memories in extracting said steganographic digital watermark payload from the scene.

3. The method of claim 2 that includes:

determining a difference image frame based on said first and second different versions of the single exposure; and

applying a watermark decoding process to said difference image frame to extract said payload.

4. The method of claim 2 that includes, through said API,

directing that the first version of said single exposure, stored in the first memory, be of smaller pixel dimensions than the second version, stored in the second memory.

5. The method of claim 4 that includes:

processing the first version of said exposure to produce a third version, the third version of said exposure having pixel dimensions matching the second version of said exposure;

subtracting one of said second or third versions from the other, to determine a difference frame; and

applying a watermark decoding process to an image based on said difference frame to extract said payload.

6. The method of claim 4 that includes applying a watermark decoding process to the first version of the exposure, and applying a watermark decoding process to the second version of the exposure, to determine if a watermark is captured in one of said versions at a resolution that can be decoded.

7. The method of claim 2 that includes:

through said API, directing that the first version of said single exposure, but not the second version, be de-noised prior to storage; and

subtracting one of said versions from the other, to determine a difference frame; and

applying a watermark decoding process to an image based on said difference frame to extract said payload.

8. The method of claim 2 that further includes:

through said API;

(a) instructing the device camera to capture a second exposure of said scene, and (b) directing first and second different versions of said second exposure to be stored in third and fourth memories, respectively, said first and second exposures of the scene being captured within 0.5 seconds of each other;

determining a first difference frame based on the versions of the first exposure stored in the first and second memories;

determining a second difference frame based on the versions of the second exposure stored in the third and fourth memories;

applying a watermark decoding process to an image based on the first difference frame; and

applying a watermark decoding process to an image based on the second difference frame.

9. The method of claim 8 that includes, through said API, directing that the first version of said first exposure, stored in the first memory, be of smaller pixel dimensions than the second version of said first exposure, stored in the second memory.

10. An imaging system comprising:

a camera having an image sensor;

a processor; and

a memory;

wherein the memory contains software that causes the system to perform acts including;

instructing the camera to capture a first single exposure of a scene, said single exposure being characterized by collection of light directed onto said image sensor during a single light collection interval;

directing first and second different versions of said single exposure to be stored separately in said memory; and

processing the first and second different versions of said single exposure, stored separately in said memory, in extracting a steganographic digital watermark payload from a depiction of an object printed with a digital watermark, included in the camera-captured scene.

11. The imaging system of claim 10 in which said software causes the system to perform acts including:

determining a difference image frame based on said first and second different versions of the single exposure; and

applying a watermark decoding process to said difference image frame to extract said payload.

12. The imaging system of claim 10 in which said software causes the system to direct that the first version of said single exposure, stored in the first memory, be of smaller pixel dimensions than the second version, stored in the second memory.

13. The imaging system of claim 12 in which said software causes the system to perform acts including:

processing the first version of said exposure to produce a third version, the third version of said exposure having pixel dimensions matching the second version of said exposure;

subtracting one of said second or third versions from the other, to determine a difference frame; and

applying a watermark decoding process to an image based on said difference frame to extract said payload.

14. The imaging system of claim 12 in which said software causes the system to perform acts including:

applying a watermark decoding process to the first version of the exposure, and applying a watermark decoding process to the second version of the exposure, to determine if a watermark is captured in one of said versions at a resolution that can be decoded.

15. The imaging system of claim 10 in which said software causes the system to perform acts including:

directing that the first version of said single exposure, but not the second version, be de-noised prior to storage;

subtracting one of said versions from the other, to determine a difference frame;

and applying a watermark decoding process to an image based on said difference frame, to extract said payload.

16. The imaging system of claim 10 in which said software causes the system to perform acts including:

(a) instructing the camera to capture a second exposure of said scene, and (b) directing first and second different versions of said second exposure to be stored in third and fourth memories, respectively, said first and second exposures of the scene being captured within 0.5 seconds of each other;

determining a first difference frame based on the versions of the first exposure stored in the first and second memories;

determining a second difference frame based on the versions of the second exposure stored in the third and fourth memories;

applying a watermark decoding process to an image based on the first difference frame; and

applying a watermark decoding process to an image based on the second difference frame.

17. The imaging system of claim 16 in which the first version of said first exposure, stored in the first memory, is of smaller pixel dimensions than the second version of said first exposure, stored in the second memory.

18. The imaging system of claim 16 in which said software causes the system to perform de-noising of the first version of said first exposure, but not the second version of the first exposure, prior to storage.

19. The imaging system of claim 10 in which said software causes the system to perform acts including:

receiving distance information, indicating a distance between the image sensor and the scene; and

scaling the captured exposure by a correction factor corresponding to said distance information.

20. An imaging system comprising:

a memory;

a camera, including an image sensor, to capture a first single exposure of a scene, said single exposure being characterized by collection of light directed onto said image sensor during a single light collection interval;

first means for directing first and second different versions of said single exposure to be stored separately in said memory; and

second means for processing the first and second different versions of said single exposure, stored separately in said memory, in extracting a steganographic digital watermark payload from a depiction of an object printed with a digital watermark, included in the camera-captured scene.