Salient point-based arrangements

• US 10,521,873 B2
• Filed: 05/08/2017
• Issued: 12/31/2019
• Est. Priority Date: 04/26/2011
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Aller, Joshua V.
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/589,530
• Publication Number:
  US 20170243318A1
• Time in Patent Office: 967 Days
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06T 1/005 Robust watermarking, e.g. average attack or collusion attack resistant
• CPC Class Codes: G06T 1/0092 Payload characteristic determination in a watermarking scheme, e.g. number of bits to be embedded
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/04 including near field communication means, e.g. RFID
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 1/32144 embedded in the image data, i.e. enclosed or integrated in the image, e.g. watermark, super-imposed logo or stamp
• CPC Class Codes: H04N 1/32203 Spatial or amplitude domain methods H04N1/32309 takes precedence
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3269 of machine readable codes or marks, e.g. bar codes or glyphs

Abstract

A variety of methods and systems involving sensor-equipped portable devices, such as smartphones and tablet computers, are described. One particular embodiment decodes a digital watermark from imagery captured by the device and, by reference to watermark payload data, obtains salient point data corresponding to an object depicted in the imagery. Other embodiments obtain salient point data for an object through use of other technologies (e.g., NFC chips). The salient point data enables the device to interact with the object in a spatially-dependent manner. Many other features and arrangements are also detailed.

Claims

1. A method that processes imagery captured by a first mobile device, the method comprising the acts:

capturing first imagery of a scene with a camera of the first mobile device, the first imagery including depiction of a first part of an object;

processing the first imagery by applying a recognition operation thereto, the recognition operation yielding identification data corresponding to the object;

through use of the identification data, accessing from a database first information associated with the object, the first information indicating a position of the object relative to a reference framework including latitude and longitude;

determining a first device-object pose, of the device relative to the object, using second information derived from the captured first imagery;

by reference to the object position-indicating information from the database, and the determined first pose of the device relative to the object, determining a first device-framework pose, of the device within said reference framework;

obtaining a first graphic, corresponding to the depicted first part of the object, perspectively-warping the first graphic in accordance with the first device-object pose, and overlaying the warped first graphic on the first imagery on a display of the device;

capturing second imagery of the scene with said camera, the second imagery including depiction of a second part of the object, different than the first part;

determining a second device-object pose, of the device relative to the object, using second information derived from the captured second imagery;

by reference to the determined second pose of the device relative to the object, and the first information indicating the position of the object relative to the reference framework, determining a second device-framework pose, of the device within said reference framework; and

obtaining a second graphic, different than the first graphic, the second graphic corresponding to the depicted second part of the object, perspectively-warping the second graphic in accordance with the second device-object pose, and overlaying the warped second graphic on the second imagery on the display of the device.

2. The method of claim 1 in which the object comprises a planar surface, and the object position-indicating first information accessed from the database indicates a compass direction the planar surface is facing, or a vertical tilt of the surface.

3. The method of claim 1 in which:

the second information comprises descriptor information for salient points derived from the captured first imagery; and

the act of determining the first device-object pose uses salient point information for said object provided from a remote computer, in addition to said second information derived from the captured imagery.

4. The method of claim 1 that further includes deriving said second information from the captured imagery, said deriving including identifying salient points in the captured imagery, and generating descriptors for plural of said points, said descriptors each including a feature vector.

5. The method of claim 4 that further includes sending data indicating the first device-object pose to a remote processor, and also sending said second information to a remote processor, the remote processor later delivering salient point information based on said second information to a second mobile device, for use by the second mobile device in determining its pose.

6. The method of claim 1 in which the reference framework comprises latitude, longitude, and elevation.

7. The method of claim 1 in which the recognition operation is performed by one or more processors in the first mobile device.

8. The method of claim 1 that includes determining a 6D pose of the mobile device in said reference framework.

9. The method of claim 1 in which a location of the device in said reference framework is determined with finer resolution than would be possible with GPS.

10. The method of claim 1 in which the first information accessed from the database includes location coordinates for the object in the reference framework.

11. The method of claim 1 in which:

the second information derived from the captured imagery comprises salient point information;

the first information accessed from the database includes salient point information for the object; and

determining the first device-object pose employs both the second information derived from the captured imagery, and the salient point information for the object accessed from the database.

12. The method of claim 1 that further includes sending information relating to the first device-object pose from the first mobile device to a different device.

13. A non-transitory computer-readable medium containing software instructions operable to configure a camera- and display-equipped first mobile device to perform acts including:

capturing first imagery of a scene with the camera;

processing the first imagery by applying a recognition operation thereto, the recognition operation yielding identification data corresponding to an object that is depicted in the first imagery;

through use of the identification data, accessing—

from a database—

first information associated with the object, the first information indicating a position of the object relative to a reference framework including latitude and longitude;

determining a first device-object pose, of the device relative to the object, using second information derived from the captured first imagery;

by reference to the object position-indicating information from the database, and the determined first pose of the device relative to the object, determining a first device-framework pose, of the device within said reference framework;

obtaining a first graphic, corresponding to a depicted first part of the object, perspectively-warping the first graphic in accordance with the first device-object pose, and overlaying the warped first graphic on the first imagery on the display;

capturing second imagery of the scene with said camera;

determining a second device-object pose, of the device relative to the object, using second information derived from the captured second imagery;

by reference to the determined second pose of the device relative to the object, and the first information indicating the position of the object relative to the reference framework, determining a second device-framework pose, of the device within said reference framework; and

obtaining a second graphic, different than the first graphic, the second graphic corresponding to a second part of the object depicted in the second imagery, perspectively-warping the second graphic in accordance with the second device-object pose, and overlaying the warped second graphic on the second imagery on the display.

14. A mobile device comprising:

a camera;

one or more processors;

a memory;

object recognition software stored in the memory;

software instructions stored in the memory that configure the one or more processors to transmit data output by the object recognition software to said remote device, and to receive data locating the object in a reference framework including latitude and longitude, together with reference descriptors for object salient points, in return; and

means for determining a device-object pose, using the salient point descriptors received from the remote device, wherein the determined device-object pose, together with the object location data received from the remote device, enable the mobile device to determine a position of the device within the reference framework;

wherein said means comprises instructions stored in the memory that configure the one or more processors to perform an algorithm including the acts;

identifying salient points in the captured imagery, and generating descriptors for plural of said identified salient points, wherein there are two sets of salient point descriptors—

a reference set received from the remote device, and a set generated by the mobile device from the captured imagery;

matching a first salient point descriptor in a first of said two sets, to a corresponding salient point descriptor in a second of said two sets, by analyzing salient descriptors in the second set to identify a descriptor having a minimum sum-of-squared difference with the first salient point descriptor;

repeating said matching act with plural other salient point descriptors in said first set, to identify a corresponding salient point descriptor in the second set for each of said plural other salient point descriptors in the first set, thereby yielding a plurality of paired salient points; and

determining coordinates of a matrix that relate the salient points in the first set, to their paired salient points in the second set;

wherein said matrix coordinates define said device-object pose.