Salient point-based arrangements
First Claim
Patent Images
1. A method that processes imagery captured by a first mobile device, the method comprising the acts:
- capturing first imagery of a scene with a camera of the first mobile device, the first imagery including depiction of a first part of an object;
processing the first imagery by applying a recognition operation thereto, the recognition operation yielding identification data corresponding to the object;
through use of the identification data, accessing from a database first information associated with the object, the first information indicating a position of the object relative to a reference framework including latitude and longitude;
determining a first device-object pose, of the device relative to the object, using second information derived from the captured first imagery;
by reference to the object position-indicating information from the database, and the determined first pose of the device relative to the object, determining a first device-framework pose, of the device within said reference framework;
obtaining a first graphic, corresponding to the depicted first part of the object, perspectively-warping the first graphic in accordance with the first device-object pose, and overlaying the warped first graphic on the first imagery on a display of the device;
capturing second imagery of the scene with said camera, the second imagery including depiction of a second part of the object, different than the first part;
determining a second device-object pose, of the device relative to the object, using second information derived from the captured second imagery;
by reference to the determined second pose of the device relative to the object, and the first information indicating the position of the object relative to the reference framework, determining a second device-framework pose, of the device within said reference framework; and
obtaining a second graphic, different than the first graphic, the second graphic corresponding to the depicted second part of the object, perspectively-warping the second graphic in accordance with the second device-object pose, and overlaying the warped second graphic on the second imagery on the display of the device.
1 Assignment
0 Petitions
Accused Products
Abstract
A variety of methods and systems involving sensor-equipped portable devices, such as smartphones and tablet computers, are described. One particular embodiment decodes a digital watermark from imagery captured by the device and, by reference to watermark payload data, obtains salient point data corresponding to an object depicted in the imagery. Other embodiments obtain salient point data for an object through use of other technologies (e.g., NFC chips). The salient point data enables the device to interact with the object in a spatially-dependent manner. Many other features and arrangements are also detailed.
-
Citations
14 Claims
-
1. A method that processes imagery captured by a first mobile device, the method comprising the acts:
-
capturing first imagery of a scene with a camera of the first mobile device, the first imagery including depiction of a first part of an object; processing the first imagery by applying a recognition operation thereto, the recognition operation yielding identification data corresponding to the object; through use of the identification data, accessing from a database first information associated with the object, the first information indicating a position of the object relative to a reference framework including latitude and longitude; determining a first device-object pose, of the device relative to the object, using second information derived from the captured first imagery; by reference to the object position-indicating information from the database, and the determined first pose of the device relative to the object, determining a first device-framework pose, of the device within said reference framework; obtaining a first graphic, corresponding to the depicted first part of the object, perspectively-warping the first graphic in accordance with the first device-object pose, and overlaying the warped first graphic on the first imagery on a display of the device; capturing second imagery of the scene with said camera, the second imagery including depiction of a second part of the object, different than the first part; determining a second device-object pose, of the device relative to the object, using second information derived from the captured second imagery; by reference to the determined second pose of the device relative to the object, and the first information indicating the position of the object relative to the reference framework, determining a second device-framework pose, of the device within said reference framework; and obtaining a second graphic, different than the first graphic, the second graphic corresponding to the depicted second part of the object, perspectively-warping the second graphic in accordance with the second device-object pose, and overlaying the warped second graphic on the second imagery on the display of the device. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A non-transitory computer-readable medium containing software instructions operable to configure a camera- and display-equipped first mobile device to perform acts including:
-
capturing first imagery of a scene with the camera; processing the first imagery by applying a recognition operation thereto, the recognition operation yielding identification data corresponding to an object that is depicted in the first imagery; through use of the identification data, accessing—
from a database—
first information associated with the object, the first information indicating a position of the object relative to a reference framework including latitude and longitude;determining a first device-object pose, of the device relative to the object, using second information derived from the captured first imagery; by reference to the object position-indicating information from the database, and the determined first pose of the device relative to the object, determining a first device-framework pose, of the device within said reference framework; obtaining a first graphic, corresponding to a depicted first part of the object, perspectively-warping the first graphic in accordance with the first device-object pose, and overlaying the warped first graphic on the first imagery on the display; capturing second imagery of the scene with said camera; determining a second device-object pose, of the device relative to the object, using second information derived from the captured second imagery; by reference to the determined second pose of the device relative to the object, and the first information indicating the position of the object relative to the reference framework, determining a second device-framework pose, of the device within said reference framework; and obtaining a second graphic, different than the first graphic, the second graphic corresponding to a second part of the object depicted in the second imagery, perspectively-warping the second graphic in accordance with the second device-object pose, and overlaying the warped second graphic on the second imagery on the display.
-
-
14. A mobile device comprising:
-
a camera; one or more processors; a memory; object recognition software stored in the memory; software instructions stored in the memory that configure the one or more processors to transmit data output by the object recognition software to said remote device, and to receive data locating the object in a reference framework including latitude and longitude, together with reference descriptors for object salient points, in return; and means for determining a device-object pose, using the salient point descriptors received from the remote device, wherein the determined device-object pose, together with the object location data received from the remote device, enable the mobile device to determine a position of the device within the reference framework; wherein said means comprises instructions stored in the memory that configure the one or more processors to perform an algorithm including the acts; identifying salient points in the captured imagery, and generating descriptors for plural of said identified salient points, wherein there are two sets of salient point descriptors—
a reference set received from the remote device, and a set generated by the mobile device from the captured imagery;matching a first salient point descriptor in a first of said two sets, to a corresponding salient point descriptor in a second of said two sets, by analyzing salient descriptors in the second set to identify a descriptor having a minimum sum-of-squared difference with the first salient point descriptor; repeating said matching act with plural other salient point descriptors in said first set, to identify a corresponding salient point descriptor in the second set for each of said plural other salient point descriptors in the first set, thereby yielding a plurality of paired salient points; and determining coordinates of a matrix that relate the salient points in the first set, to their paired salient points in the second set; wherein said matrix coordinates define said device-object pose.
-
Specification