Silhouette-based object and texture alignment, systems and methods

• US 9,940,756 B2
• Filed: 07/05/2017
• Issued: 04/10/2018
• Est. Priority Date: 11/18/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Siddiqui, Matheen
• Primary Examiner(s): Niu, Feng
• Assistant Examiner(s): Thomas, Mia M
• Application Number:
  US15/642,276
• Publication Number:
  US 20170301144A1
• Time in Patent Office: 279 Days
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An object-image alignment data generating method for use in an object recognition system is presented. The method obtains a 3D model and a set of 2D images of the object. Each 2D image from the set is captured based on a particular camera point of view. The method then uses the 3D model of the object to generate multiple silhouettes of the object according to different camera point of views. Each silhouette is then matched and aligned with a 2D image based on the corresponding camera point of view. The method also derives at least one descriptor from the 2D images and compiles feature points that correspond to the descriptors. Each feature point includes a 2D location and a 3D location. The method then generates an object-image alignment packet by packaging the 2D images, the descriptors, and the feature points.

Claims

1. An electronic device for presenting augmented reality media, the device comprising:

a tangible, non-transitory, computer-readable memory configured to store at least one object-image alignment packet associated with at least one scanned body of an object, wherein the at least one object-image alignment packet comprises alignment points that include recognition descriptors, feature points, and silhouettes, and wherein the recognition descriptors correspond to feature points located on the silhouettes and on one or more corresponding images of the at least one scanned body;

at least one processor coupled with the tangible, non-transitory computer-readable memory and, upon execution of object recognition and tracking software instructions, is configured to operate to;

recognize at least one portion of a body in a digital representation as the at least one scanned body based at least in part on the recognition descriptors;

track the at least one portion of the body in the digital representation based on the recognition descriptors;

align, based on the feature points, at least one of the silhouettes corresponding to the at least one scanned body with the at least one portion of the body in the digital representation; and

display digital media in the digital representation according to the at least one of the silhouettes.

2. The device of claim 1, wherein the at least one scanned body comprises a 3D object model.

3. The device of claim 1, wherein the digital representation comprises at least one of an image, a video, and a video frame.

4. The device of claim 1, wherein the at least one processor is further configured to facilitate capturing the digital representation.

5. The device of claim 1, wherein the at least one processor is further configured to:

overlay the at least one portion of the body in the digital representation with the at least one of the silhouettes;

determine a movement of the at least one portion of the body in the digital representation according to the at least one of the silhouettes; and

display the digital media based at least in part on the movement.

6. The device of claim 5, wherein the movement of the at least one portion of the body is determined based on a single image or video frame of the digital representation.

7. The device of claim 5, wherein the movement of the at least one portion of the body is determined based on the digital representation from frame to frame.

8. The device of claim 5, wherein the movement of the at least one portion of the body is determined based on a detected micron motion.

9. The device of claim 1, wherein the digital media is displayed as one of an overlay or superposition.

10. The device of claim 1, wherein the at least one portion of the body comprises a view of the body.

11. The device of claim 1, wherein the recognition descriptors include at least one of a SIFT descriptor, a DAISY descriptor, a FREAK descriptor, a histogram of gradients descriptor, and a FAST descriptor.

12. The device of claim 1, wherein the electronic device comprises a mobile device.

13. The device of claim 1, wherein the digital media comprises green screen media.

14. The device of claim 1, wherein displaying the digital media includes overlaying the digital media on a portion of the digital representation that remains unmasked by the at least one of the silhouettes.

15. The device of claim 1, wherein each feature point includes a 2D location within the one or more corresponding images of the at least one scanned body and a 3D location on the at least one scanned body.

16. The device of claim 1, wherein the silhouettes represent different points of view of the at least one scanned body.

17. The device of claim 16, wherein the silhouettes represent views of the at least one scanned body from different angles.

18. The device of claim 1, wherein the at least one object-image alignment packet includes at least one of a normal vector, an orientation, and sensor metadata.

19. The device of claim 1, wherein the aligning includes at least one of adjusting a size of the at least one of the silhouettes and distorting the at least one of the silhouettes.

20. The device of claim 1, wherein the recognition descriptors comprise a facial feature or pattern.

21. The device of claim 1, wherein the at least one portion of body comprises a face.

22. The device of claim 1, wherein the at least one of the silhouettes corresponding to the at least one scanned body are used as a basis for a recognition and tracking strategy.