Hybrid feature point/watermark-based augmented reality

• US 10,740,613 B1
• Filed: 04/17/2018
• Issued: 08/11/2020
• Est. Priority Date: 04/20/2017
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Sinclair, Eoin C.
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US15/955,481
• Time in Patent Office: 847 Days
• Field of Search: None
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2201/0051 Embedding of the watermark in the spatial domain
• CPC Class Codes: G06T 2201/0061 Embedding of the watermark in each block of the image, e.g. segmented watermarking
• CPC Class Codes: G06T 2201/0601 whereby calibration information is embedded in the watermark, e.g. a grid, a scale, a list of transformations
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations

Abstract

A camera captures video imagery depicting a digitally-watermarked object. A reference signal in the watermark is used to discern the pose of the object relative to the camera, and this pose is used in affine-transforming and positioning a graphic on the imagery as an augmented reality overlay. Feature points are also discerned from the captured imagery, or recalled from a database indexed by the watermark. As the camera moves relative to the object, the augmented reality overlay tracks the changing object depiction, using these feature points. When feature point-based tracking fails, the watermark is again processed to determine pose, and the overlay presentation is updated accordingly. In another arrangement, feature points are extracted from images of supermarket objects captured by multiple users, and are compiled in a database in association with watermark data identifying the objects—serving as a crowd-sourced repository of feature point data. A great number of other features and arrangements are also detailed.

Claims

1. A method employing a subject marked with plural blocks of watermark information in a tiled array, the method comprising the acts:

(a) receiving first image information, based on first imagery depicting said subject captured by a first camera of a first device;

(b) determining a pose of the depicted subject, relative to the first camera, by watermark information in the first image information;

(c) extracting feature points from the first image information;

(d) storing information in a data structure for each of plural of the feature points, the stored information associating the feature point with a corresponding location in a spatial frame of reference defined by blocks of the watermark information; and

(e) at one time after act (d), presenting, on a display device, captured imagery depicting the subject, together with an augmentation, where a displayed position of the augmentation relative to the subject depends on watermark information discerned from captured imagery depicting the subject; and

(f) at another time after act (d), presenting, on said display device, a depiction of the subject, together with an augmentation, where a displayed position of the augmentation relative to the subject depends on said stored information.

2. The method of claim 1 in which acts (b)-(d) are performed by said first device.

3. The method of claim 1 that further includes storing the information in a data structure remote from the first device, for later delivery to a second device different than the first, so that the second device can use said information in overlaying an augmentation graphic on imagery depicting said subject.

4. The method of claim 1 in which, in act (e), the displayed position of the augmentation relative to the subject does not depend on said stored information.

5. The method of claim 1 in which, in act (f), the displayed position of the augmentation relative to the subject does not depend on watermark information discerned from the captured imagery presented on the display device.

6. The method of claim 1 that further includes occasionally performing the acts:

processing imagery of the subject, captured by a camera, to discern first pose information characterizing a pose of the subject relative to the camera, using feature point information;

processing said imagery to discern second pose information characterizing said pose of the subject relative to the camera, using watermark information; and

assessing how closely the first pose information corresponds to the second pose information.

7. The method of claim 3 that further includes occasionally performing the acts:

processing imagery of the subject, captured by a camera, to discern first pose information characterizing a pose of the subject relative to the camera, using feature point information;

processing said imagery to discern second pose information characterizing said pose of the subject relative to the camera, using watermark information; and

assessing how closely the first pose information corresponds to the second pose information.

8. A method employing a subject marked with plural blocks of watermark information in a tiled array, the method comprising the acts:

(a) receiving first image information, based on first imagery depicting said subject captured by a first camera of a first device;

(b) determining a pose of the depicted subject, relative to the first camera, by watermark information in the first image information;

(c) extracting feature points from the first image information;

(d) storing information in a data structure for each of plural of the feature points, the stored information associating the feature point with a corresponding location in a spatial frame of reference defined by blocks of the watermark information; and

(e) presenting video image information captured by the first device on a screen thereof, and overlaying a graphical augmentation on said presentation, wherein the position, scale or rotation of said overlaid augmentation is determined, at one time, by watermark information, and at another time, by extracted feature points.

9. A method employing a subject marked with plural blocks of watermark information in a tiled array, the method comprising the acts:

receiving first image information, based on a first set of imagery depicting said subject captured by a camera-equipped first device;

determining a pose of the depicted subject, relative to the first device camera, by depiction of the watermark information in the first image information;

identifying first feature points depicted by the first image information;

storing, in a data structure, first point information for a plurality of the first feature points, the stored first point information associating said plurality of the first feature points with corresponding locations in a spatial frame of reference defined by the watermark information;

receiving second image information, based on a second set of imagery depicting said subject captured by a second camera-equipped device;

determining a pose of the depicted subject, relative to the second device camera, by depiction of the watermark information in the second image information;

identifying second feature points depicted by the second image information;

storing, in said data structure, second point information for a plurality of the second feature points, the stored second point information associating said plurality of the second feature points with corresponding locations in a spatial frame of reference defined by the watermark information;

wherein said data structure is located remote from both the first and second devices, and stores said first and second point information, for points depicted in plural sets of imagery, captured by plural camera-equipped devices, for delivery to a third camera-equipped device different than the first and second devices, enabling the third device to use said stored point information in overlaying an augmentation graphic on third imagery depicting said subject captured by said third device, by reference to a spatial frame of reference defined by the watermark information depicted in said third imagery.

10. The method of claim 9 in which the first and second imagery were captured under different lighting conditions.

11. The method of claim 9 in which the first and second imagery depict the subject from different viewpoints.

12. The method of claim 9 that further includes determining that one of said feature points is more robust than another of said feature points because said one feature point was identified from both the first and second imagery, whereas said another feature point was identified from just one of the first and second imagery.

13. The method of claim 9 in which the second imagery depicts a portion of the subject not depicted in the first imagery, and the first imagery depicts a portion of the subject not depicted in the second imagery.

14. The method of claim 9 in which said subject is steganographically marked with plural blocks of watermark information in a tiled array.

15. A method employing a portable user device that includes a camera system and a display, the method comprising the acts:

(a) capturing first imagery with the camera system, the first imagery depicting a first portion of a first object, but not depicting a second portion of the first object because the second portion is positioned relative to the camera system so that it falls outside of a capture frame of the camera system;

(b) extracting a first watermark payload from the first imagery, the first payload including a first identifier corresponding to said first object;

(c) including the first identifier in a first query that seeks previously-determined feature point data corresponding to the first object;

(d) receiving, in response to said first query, previously-determined feature point data corresponding to the first object, the previously-determined feature point data including out-of-bounds feature points associated with the second portion of the first object;

(e) using the previously-determined feature point data in determining a pose of the portable user device relative to the first object, and rendering a first augmentation on the display of the portable user device, based on said determined pose;

(f) capturing second imagery with the camera system, the second imagery depicting a first portion of a second object;

(g) extracting a second watermark payload from the second imagery, the second payload including a second identifier corresponding to said second object;

(h) including the second identifier in a second query that seeks previously-determined feature point data corresponding to the second object;

(i) receiving, in response to said second query, a response indicating that a queried source has no previously-determined feature point data corresponding to the second object;

(j) determining feature point data for the second object from the captured second imagery using a processor of said portable device; and

(k) using the device-determined feature point data for the second object, and not using previously-determined feature point data, in determining a pose of the portable user device relative to the second object.

16. The method of claim 15 that further includes rendering a second augmentation on the display of the portable user device, based on the determined pose of the portable user device relative to the second object.

17. The method of claim 16 in which the second augmentation is determined on-the-fly, rather than being pre-scripted.

18. The method of claim 16 in which the second augmentation is not established by a proprietor of the second object.

19. The method of claim 15 in which acts (f)-(k) are performed before acts (a)-(e).

20. The method of claim 15 that further includes rendering a second augmentation on the display of the portable user device, using said device-determined feature point data.

21. The method of claim 15 in which said first and second queries are to a remote data structure.

22. A method comprising the acts:

(a) determining a first pose of an object relative to a camera from a first excerpt of imagery from a video capture, so that an augmented reality graphic can be affine-transformed based on the first pose, and overlaid on the first excerpt;

(b) after act (a), determining a second pose of the object relative to the camera from a second excerpt of imagery from the video capture, so that the augmented reality graphic can be affine-transformed based on the second pose, and overlaid on the second excerpt; and

(c) after act (b), determining a third pose of the object relative to the camera from a third excerpt of imagery from the video capture, so that the augmented reality graphic can be affine-transformed based on the third pose, and overlaid on the third excerpt;

wherein determining the first pose and determining the third pose comprises determining pose using reference signal components in a digital watermark depicted in the first and third imagery excerpts; and

wherein determining the second pose comprises pose using feature points depicted in the second imagery excerpt.

23. The method of claim 22 that further includes:

affine-transforming the augmented reality graphic based on the first pose to yield a first transformed graphic, and overlaying said first transformed graphic on the first excerpt;

affine-transforming the augmented reality graphic based on the second pose to yield a second transformed graphic, and overlaying said second transformed graphic on the second excerpt; and

affine-transforming the augmented reality graphic based on the third pose to yield a third transformed graphic, and overlaying said third transformed graphic on the third excerpt.