SILHOUETTE-BASED OBJECT AND TEXTURE ALIGNMENT, SYSTEMS AND METHODS

• US 20150139535A1
• Filed: 11/18/2014
• Published: 05/21/2015
• Est. Priority Date: 11/18/2013
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Siddiqui, Matheen
• Granted Patent: US 9,489,765 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

An object-image alignment data generating method for use in an object recognition system is presented. The method obtains a 3D model and a set of 2D images of the object. Each 2D image from the set is captured based on a particular camera point of view. The method then uses the 3D model of the object to generate multiple silhouettes of the object according to different camera point of views. Each silhouette is then matched and aligned with a 2D image based on the corresponding camera point of view. The method also derives at least one descriptor from the 2D images and compiles feature points that correspond to the descriptors. Each feature point includes a 2D location and a 3D location. The method then generates an object-image alignment packet by packaging the 2D images, the descriptors, and the feature points.

Claims

1. A computer-based method of generating object-image alignment data, the method comprising:

obtaining, via an image processing engine, a digital three-dimensional (3D) model of an object;

obtaining, via the image processing engine, a set of two-dimensional (2D) images of the object from a plurality of image view points;

generating, via the image processing engine, a silhouette of the object from the digital 3D model where the silhouette comprises a collection of edge points associated with edges of the digital 3D model from a silhouette point of view;

registering, via the image processing engine, at least one image from the set of 2D images with the silhouette based an image view point associated with the at least one image and the silhouette point of view;

deriving, via the image processing engine, at least one descriptor from the at least one image;

compiling, via the image processing engine, alignment points based on the at least one descriptor and the collection of edge points, the alignment points including at least some of the edge points; and

generating, via the image processing engine, an object-image alignment packet by packaging the at least one image with alignment points.

2. The computer-based method of claim 1, wherein the object-image alignment packet comprises a key frame packet.

3. The computer-based method of claim 1, wherein the collection of points comprises 3D points.

4. The computer-based method of claim 3, wherein the collection of points comprises relative 3D coordinates to a camera point of view.

5. The computer-based method of claim 1, wherein the alignment points comprises 3D coordinates to a camera point of view.

6. The computer-based method of claim 1, wherein the step of obtaining the digital 3D model of the object includes laser scanning the object.

7. The computer-based method of claim 1, wherein the step of obtaining the digital 3D model of the object includes obtaining CAD data representing the object.

8. The computer-based method of claim 1, wherein the step of obtaining the digital 3D model of the object includes obtaining a game engine asset representing the object.

9. The computer-based method of claim 1, wherein the step of obtaining the set of 2D images of the object includes a capturing image data of the object via an optical sensor.

10. The computer-based method of claim 9, wherein image data comprises still image data.

11. The computer-based method of claim 10, wherein the still image data comprises a frame from a video stream.

12. The computer-based method of claim 9, wherein the image data comprises video data.

13. The computer-based method of claim 1, wherein the step of obtaining the set of 2D images of the object includes a capturing image data of the object via an optical sensor.

14. The computer-based method of claim 1, wherein the step of registering at least one image from the set of 2D images with the silhouette including aligning image data of the object within the at least image to at least some of the edge points.

15. The computer-based method of claim 14, wherein the step of aligning the image data includes aligning edge pixels within the image data of the object to at least some of the edge points.

16. The computer-based method of claim 1, wherein the at least one descriptor comprises an image descriptor.

17. The computer-based method of claim 16, wherein the image descriptor comprises at least one of the following types of descriptors:

a SIFT descriptor, a DAISY descriptor, a FREAK descriptor, and a FAST descriptor.

18. The computer-based method of claim 1, further comprising sending the object-image alignment packet to an electronic device.

19. The computer-based method of claim 18, further comprising sending the object-image alignment packet over a network.

20. The computer-based method of claim 1, wherein the object-image alignment packet includes at least one of the following:

a normal vector, an orientation, and sensor metadata.