Augmented reality content rendering via albedo models, systems and methods

• US 9,805,510 B2
• Filed: 05/13/2015
• Issued: 10/31/2017
• Est. Priority Date: 05/13/2014
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Siddiqui, Matheen, Wnuk, Kamil
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US14/711,763
• Publication Number:
  US 20150332512A1
• Time in Patent Office: 902 Days
• Field of Search: None
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/80 Shading
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Methods for rendering augmented reality (AR) content are presented. An a priori defined 3D albedo model of an object is leveraged to adjust AR content so that is appears as a natural part of a scene. Disclosed devices recognize a known object having a corresponding albedo model. The devices compare the observed object to the known albedo model to determine a content transformation referred to as an estimated shading (environmental shading) model. The transformation is then applied to the AR content to generate adjusted content, which is then rendered and presented for consumption by a user.

Claims

1. A method of rendering augmented reality content, comprising:

obtaining, by a rendering device, a predefined 3D albedo model of an object observed in a physical environment, the predefined 3D albedo model comprising portions corresponding to portions of the object, wherein each portion of the predefined 3D albedo model includes lighting rules selected based on a reflective nature of a corresponding portion of the object;

deriving, by the rendering device, features from a digital representation of the object;

obtaining, by the rendering device, augmented reality (AR) content related to the object based on the features;

deriving, by the rendering device, a pose of the object from the digital representation;

aligning, by the rendering device, the 3D albedo model with the pose;

deriving, by the rendering device, observed shading data from the digital representation and the 3D albedo model;

deriving an estimated object shading model using the 3D albedo model and the observed shading data;

generating, by the rendering device, environmentally adjusted AR content by applying the estimated object shading model to the AR content; and

rendering, by the rendering device, the environmentally adjusted AR content.

2. The method of claim 1, wherein the predefined 3D albedo model comprises a computer-generated object model.

3. The method of claim 2, wherein the computer-generated object model comprises a mesh.

4. The method of claim 2, wherein the computer-generated object model comprises a model of a portion of the object.

5. The method of claim 1, wherein the step of deriving features from the digital representation includes applying at least one feature detection algorithm to the digital representation.

6. The method of claim 5, wherein the feature detection algorithm includes at least one of the following algorithms:

SIFT, BRISK, SURF, FAST, BRIEF, Harris Corners, Edges, DAISY, GLOH, HOG, EOG, and TILT.

7. The method of claim 1, wherein the step of obtaining the AR content includes looking up the AR content based on descriptors associated with the features.

8. The method of claim 1, further comprising the rendering device capturing the digital representation.

9. The method of claim 1, further comprising generating the environmentally adjusted AR content by applying a sensor environment map comprising an environment lighting map.

10. The method of claim 1, further comprising generating the environmentally adjusted AR content by applying a sensor environment map comprising a noise map.

11. The method of claim 1, further comprising generating the environmentally adjusted AR content by applying a sensor environment map comprising a sensor error map.

12. The method of claim 1, wherein the predefined 3D albedo model represents known albedo information of at least one of the following:

a toy, a vehicle, a face, a purchasable product, a printed media, a vending machine, an appliance, a plant, a sign, a tissue, a patient, and a game component.

13. The method of claim 1, wherein the AR content comprises at least one of the following:

a game, an application, a video, an image, an animation, a 3D rendered object, an object mesh, an animated mesh, an age projected animation, a medical image, a clothing, makeup, glasses, furniture, a wearable accessory, a person, an avatar, a pet, a coupon, a store shelf, signage, anatomy, and a sonogram.

14. The method of claim 1, further comprising tracking at least some of the features in real time.

15. The method of claim 14, wherein the step of tracking at least some of the features includes tracking features within frames of a video sequence.

16. The method of claim 14, further comprising re-rendering the environmentally adjusted AR content in response to movement of features.

17. The method of claim 16, further comprising re-rendering the environmentally adjusted AR content at close to the frame rate of the video sequence.

18. The method of claim 17, wherein the frame rate is at least 30 frames per second.

19. The method of claim 1, wherein the environmentally adjusted AR content comprises an animation.

20. The method of claim 1, wherein the step of rendering the environmentally adjusted AR content includes overlaying the environmentally adjusted AR content over an image of at least a portion of the object.

21. The method of claim 1, wherein the step of rendering the environmentally adjusted AR content includes presenting the environmentally adjusted AR content on a display relative to at least a portion of the object.

22. The method of claim 1, wherein the predefined 3D albedo model comprises known features having locations within the model.

23. The method of claim 22, wherein the step of aligning the predefined 3D albedo model with the pose includes aligning the predefined 3D albedo model with the pose based on the known features with corresponding features derived from the digital representation.

24. The method of claim 1, wherein the predefined 3D albedo model comprises a lighting policy, the lighting policy comprises lighting rules sets associated with corresponding portions of the predefined 3D albedo model.

25. The method of claim 24, wherein the step of generating the environmentally adjusted AR content includes applying at least one of the lighting rules sets to a portion of the AR content that corresponds to a portion of the predefined 3D albedo model.

26. The method of claim 24, wherein the lighting rules sets include lighting rules for at least one of the following portions of the predefined 3D albedo model:

a facial feature, a weapon, a panel, an article of clothing, a vehicular feature, a tissue type, a substrate, a game feature, and a material type.

27. The method of claim 1, wherein the predefined 3D albedo model is generated from a plurality of training images captured by the rendering device of the object in a variety of lighting conditions and/or from a variety of viewpoints.

28. The method of claim 27, wherein at least one of the plurality of training images correspond to the digital representation of the object from which features are derived.

29. The method of claim 27, wherein the training images are captured by the rendering device in parallel to deriving the pose of the object from the digital representation, and the predefined 3D albedo model is updated at runtime as the pose alignment of each new training images is complete.

30. The method of claim 29, wherein at least one of online averaging and a Bayesian filter is used to update the predefined 3D albedo model at runtime.

31. The method of claim 1, wherein the step of rendering the environmentally adjusted AR content includes presenting the environmentally adjusted AR content at a spatial location relative to the object that is derived from an environmental feature implied by the estimated shading model.

32. The method of claim 31, wherein the AR content is at least one of a cloud positioned radially from a darkest point in the estimated shading model and a light source positioned radially from a brightest point in the estimated shading model.

33. The method of claim 1, further comprising:

using the estimated object shading model and the predefined 3D albedo model to generate an expected rendering of the object;

using the digital representation of the object and the observed shading data to identify one or more environmental artifacts in the digital representation; and

rendering, by the rendering device, at least some of the one or more environmental artifacts with the environmentally adjusted AR content.

34. The method of claim 1, further comprising:

deriving, in addition to the estimated shading model, one or more other environmental maps usable to modify AR content so that the AR content is rendered more consistently with an environment of the rendering device.

35. A computer program product embedded in a non-transitory computer readable medium comprising instructions executable by one or more computer processors of one or more computers to at least:

derive, from a digital representation of an object observed in a physical environment captured at the rendering device, image features;

align an albedo model of the object with the digital representation of the object using the image features, the albedo model comprising portions corresponding to portions of the object, wherein each portion of the albedo model includes lighting rules selected based on a reflective nature of a corresponding portion of the object;

derive observed shading using the digital representation of the object captured at the rendering device and the albedo model of the object;

derive, from the observed shading, an estimated object shading model; and

generate, at the rendering device, environmentally adjusted augmented reality (AR) content by applying the estimated object shading model to AR content related to the object.