Silhouette-based object and texture alignment, systems and methods

• US 9,728,012 B2
• Filed: 10/04/2016
• Issued: 08/08/2017
• Est. Priority Date: 11/18/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Siddiqui, Matheen
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Thomas, Mia M
• Application Number:
  US15/285,389
• Publication Number:
  US 20170024901A1
• Time in Patent Office: 308 Days
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An object-image alignment data generating method for use in an object recognition system is presented. The method obtains a 3D model and a set of 2D images of the object. Each 2D image from the set is captured based on a particular camera point of view. The method then uses the 3D model of the object to generate multiple silhouettes of the object according to different camera point of views. Each silhouette is then matched and aligned with a 2D image based on the corresponding camera point of view. The method also derives at least one descriptor from the 2D images and compiles feature points that correspond to the descriptors. Each feature point includes a 2D location and a 3D location. The method then generates an object-image alignment packet by packaging the 2D images, the descriptors, and the feature points.

Claims

1. A method of presenting augmented reality media on a computing device, the method comprising:

obtaining, by the computing device, at least one object-image alignment packet associated with at least one modeled object, wherein the at least one object-image alignment packet comprises alignment points that include recognition descriptors, feature points, and silhouettes, and wherein the recognition descriptors correspond to feature points located on silhouettes and on corresponding images of the at least one modeled object;

recognizing, at least one portion of a real-world object in a digital representation as the at least one modeled object based at least in part on the recognition descriptors;

tracking, by the computing device, the at least one portion of the real-world object in the digital representation based on the recognition descriptors;

aligning, by the computing device and based on the feature points, at least one of the silhouettes corresponding to the at least one modeled object with the at least one portion of the real-world object in the digital representation; and

displaying, by the computing device, digital media as an overlay on the digital representation according to the at least one of the silhouettes.

2. The method of claim 1, wherein the at least one modeled object comprises a 3D object model.

3. The method of claim 1, wherein the at least one modeled object comprises at least one of a CAD modeled object, a laser-scanned modeled object, and a game engine modeled object.

4. The method of claim 1, wherein the digital representation comprises at least one of an image, a video, and a video frame.

5. The method of claim 1, further comprising:

capturing the digital representation of the portion of the real-world object.

6. The method of claim 1, wherein the at least one portion of the real-world object comprises a view of the real-world object.

7. The method of claim 1, wherein the recognition descriptors include at least one of a SIFT descriptor, a DAISY descriptor, a FREAK descriptor, a histogram of gradients descriptor, and a FAST descriptor.

8. The method of claim 1, wherein the computing device comprises a mobile device.

9. The method of claim 8, wherein the mobile device comprises a cell phone.

10. The method of claim 1, wherein obtaining the at least one object-image alignment packet includes receiving the at least one object-image alignment packet from an object modeling manager.

11. The method of claim 1, wherein the digital media comprises green screen media.

12. The method of claim 1, wherein displaying the digital medial includes overlaying the digital media on a portion of the digital representation that remains unmasked by the at least one of the silhouettes.

13. The method of claim 1, wherein each feature point includes a 2D location within the corresponding images of the at least one modeled object and 3D location on the at least one modeled object.

14. The method of claim 1, wherein the silhouettes represent different points of view of the at least one modeled object.

15. The method of claim 14, wherein the silhouettes represent at least one of a top view of the at least one modeled object, a side view of the at least one modeled object, views of the at least one modeled object from different angles, and a bottom view of the at least one modeled object.

16. The method of claim 1, wherein the at least one object-image alignment packet includes at least one of the following:

a normal vector, an orientation, and sensor metadata.

17. The method of claim 1, wherein the feature points correspond to at least one featureless aspect of the at least one modeled object.

18. The method of claim 17, wherein the at least one featureless aspect comprises a logo on the at least one modeled object.

19. The method of claim 1, further comprising:

determining movement of the real-world object in the digital representation based on the at least one of the silhouettes.

20. The method of claim 19, wherein determining movement of the real-world object includes determining movement frame-by-frame of the digital representation.