Determining pose for use with digital watermarking, fingerprinting and augmented reality

• US 9,684,941 B2
• Filed: 03/07/2013
• Issued: 06/20/2017
• Est. Priority Date: 10/29/2012
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Filler, Tomas
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US13/789,126
• Publication Number:
  US 20140119593A1
• Time in Patent Office: 1,566 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Image recognition and augmented reality experiences utilize auxiliary data extracted from an image or video, or image fingerprints, or a combination of both. One claim recites a method comprising: receiving a plurality of imagery frames captured by a device sensor; identifying a plurality of key points in each of the plurality of imagery frames; resolving image pose by utilizing relative relationships of the key points between different frames of the plurality of frames; modifying imagery of at least one of the plurality of frames based on resolved image pose; and deriving a fingerprint from modified imagery. Of course other claims and combinations are provided as well.

Claims

1. A method comprising:

receiving imagery captured by a device sensor, in which the device sensor comprises an image or video camera;

analyzing the imagery to decode digital watermarking therefrom, in which the digital watermarking comprises orientation attributes;

determining orientation information from the orientation attributes, the orientation information being associated with a capture position of the device sensor relative to an imaged subject;

providing the orientation information for use by an augmented reality (AR) system; and

using one or more configured processors, refining the orientation information determined from the orientation attributes with image homography, in which the orientation information is utilized as a starting pose for the image homography.

2. The method of claim 1 further comprising:

identifying a plurality of key points in the imagery, in which the key points represent robust image characteristics that are independent of the digital watermarking, in which said refining utilizes pose homographies associated with the key points.

3. The method of claim 2 in which said refining further minimizes a cost function associated with the pose homographies.

4. The method of claim 1 in which the AR system overlays graphics or video on a display screen in an area corresponding to a spatial area in which the digital watermarking was detected from the imagery.

5. The method of claim 1 in which the AR system utilizes the orientation information to transform the imagery for display on a display screen.

6. The method of claim 1 in which the digital watermarking comprises a payload, in which said method further comprises initiating an announcement associated with the AR system utilizing at least a portion of the payload.

7. The method of claim 1 further comprising:

modifying the imagery according to the orientation information to yield modified imagery; and

deriving a fingerprint from the modified imagery, the fingerprint comprising a hash of the modified imagery.

8. The method of claim 7 further comprising providing the fingerprint to the AR system to help facilitate display of graphics or video on a display screen corresponding to a spatial area of modified imagery from which the fingerprint was derived from.

9. A mobile device comprising:

a device sensor comprising an image or video camera;

memory storing instructions;

one or more processors configured by the instructions for;

analyzing imagery captured by said device sensor to extract an encoded signal therefrom, in which the encoded signal comprises orientation attributes;

determining orientation information from the orientation attributes, the orientation information being associated with a capture position of said device sensor relative to an imaged subject;

providing the orientation information for use by an augmented reality (AR) system; and

adapting the orientation information determined from the orientation attributes with image homography, in which the orientation information is utilized as a starting pose for the image homography.

10. The mobile device of claim 9 further comprising the AR system.

11. The mobile device of claim 10 in which said AR system is configured for overlaying graphics or video on a display screen in an area corresponding to a spatial area in which the encoded signal was detected from the imagery.

12. The mobile device of claim 10 in which said mobile device comprises a display screen, and in which said AR system is configured for utilizing the orientation information to transform the imagery for display on said display screen.

13. The mobile device of claim 10 in which the encoded signal comprises a payload, in which at least a portion of the payload is used to initiate an announcement associated with said AR system.

14. The mobile device of claim 10 in which said instructions comprise instructions to configure said one or more processors for:

transforming the imagery according to the orientation information to yield modified imagery; and

extracting a fingerprint from the modified imagery, the fingerprint comprising a hash of the modified imagery.

15. The mobile device of claim 14 further comprising a display screen, and instructions to configure said one or more processors for providing the fingerprint to said AR system to help facilitate display of graphics or video on said display screen corresponding to a spatial area of modified imagery from which the fingerprint was derived from.

16. The mobile device of claim 9 in which said instructions comprise instructions to configure said one or more processors for:

identifying a plurality of key points in the imagery, in which the key points represent robust image characteristics that are independent of the encoded signal, in which said adapting utilizes pose homographies associated with the key points.

17. The mobile device of claim 16 in which said instructions comprise instructions to configure said one or more processors for minimizing a cost function associated with the pose homographies.