Systems and methods for computer assisted operation

• US 10,282,914 B1
• Filed: 10/26/2017
• Issued: 05/07/2019
• Est. Priority Date: 07/17/2015
• Status: Active Grant

• Current Assignee: Bao Tran, Ha Tran
• Original Assignee: Bao Tran, Ha Tran
• Inventors: Tran, Bao, Tran, Ha
• Primary Examiner(s): Beasley, Deirdre L
• Application Number:
  US15/795,252
• Time in Patent Office: 558 Days
• Field of Search: 348 53
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/1036 Measuring load distribution, e.g. podologic studies
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: A61B 5/6829 Foot or ankle
• CPC Class Codes: A61B 5/6898 Portable consumer electronic devices, e.g. music players, telephones, tablet computers
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A63B 2220/00 Measuring of physical parameters relating to sporting activity
• CPC Class Codes: G01G 19/44 for weighing persons
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 3/18 : Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Systems and methods are disclosed method for rendering reality for an object by performing motion tracking and area learning of an environment; selecting a pattern or color from a plurality of product variations; blending the pattern or color of the object and the environment; and displaying the color on the object as an augmented reality view.

Claims

1. A method for rendering reality, comprising:

performing tracking and area learning of an environment by motion-tracking visual features of the environment using concurrent odometry and mapping (COM) including feature points and planes to estimate a pose of the camera relative to the environment and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment;

capturing images from a plurality of angles of the environment;

acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;

selecting a pattern or color from a plurality of product variations;

blending the pattern or color of the object and the environment;

displaying the color on the object as an augmented reality view;

scaling the 3D model of the product based on dimensions of the environment and the product;

projecting the product in the environment;

selecting a best fit from jewelry variations, shoe variations, clothing variations, apparel variations or footwear variations; and

determining content to be presented based on previous activities or selections while generating an augmented or virtual reality display of the new product in the environment.

2. The method of claim 1, comprising:

performing motion tracking and understanding an environment by detecting points or planes from images and estimating light or color in the environment using the video camera and an accelerometer;

determining a target volume or space;

projecting the product in the environment; and

iteratively changing product variations until satisfied.

3. The method of claim 1, comprising applying a 3D deformable template to the object and warping the template to one or more predetermined points on the model.

4. The method of claim 1, wherein the object comprises hair, foot, hand, finger, toe, wrist, neck, or ear, wherein the object comprises jewelry.

5. The method of claim 1, comprising providing a 3D printer for printing a custom object, or providing a bed of sand having a liquid flowable state and a solid state with an impression for a reformably molded object.

6. The method of claim 1, wherein the object comprises a body portion, a head, a chest region, a leg, and the object comprises body protection gear.

7. The method of claim 1, wherein the object comprises a face and the product variation comprises facial makeup products.

8. The method of claim 1, wherein the object comprises a head and the product variation comprises hairstyles.

9. The method of claim 1, wherein the anatomical portion is a face, comprising:

selecting a makeup pattern or color from a plurality of makeup product variations;

blending the makeup pattern or color onto the model; and

displaying the makeup color on the face.

10. The method of claim 1, comprising modeling one or more applications of light foundation creme, eye disguise, highlighter, concealer or blush and recommending a makeup to the subject.

11. The method of claim 4, wherein the jewelry comprises cosmetic jewelry, a facial jewelry, hand jewelry, foot jewelry, ear jewelry, dental jewelry, navel jewelry or a hair jewelry.

12. The method of claim 1, comprising motion tracking, area learning and depth sensing a body portion.

13. The method of claim 1, comprising matching used jewelry or consigned jewelry to the subject.

14. The method of claim 1, comprising rendering the product variations on an online portal.

15. The method of claim 1, comprising providing augmented reality view or virtual reality view of the one or more products with a user custom view.

16. A method for selecting a product, comprising:

selecting one or more 3D models from a product database;

motion-tracking visual features of the environment using concurrent odometry and mapping (COM) including feature points and planes to estimate a pose of the camera relative to the environment and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment;

capturing images from a plurality of angles of the environment;

acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;

selecting a pattern or color from a plurality of product variations;

blending the pattern or color of the product and the environment; and

displaying the color on the product as an augmented reality view;

scaling the 3D model of the product based on dimensions of the environment and the product;

determining content to be presented based on previous activities or selections while projecting the product in the environment;

selecting a best fit from jewelry variation, shoe variations, clothing variations, apparel variations or footwear variations; and

generating an augmented or virtual reality display of the new product in the environment.

17. A method for fitting, comprising:

selecting one or more 3D models from a product database;

capturing images of a user view relative to a reference frame using a mobile camera, including;

motion-tracking visual features of the environment using concurrent odometry and mapping (COM) including feature points and planes to estimate a pose of the camera relative to the environment and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment;

capturing images from a plurality of angles of the environment;

acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;

selecting a pattern or color from a plurality of product variations;

blending the pattern or color of the object and the environment; and

displaying the color on the object as an augmented reality view;

generating an updated model of the user view by selecting a closest 3D model to the user view and matching points on the closest 3D product model to one or more predetermined points on the user view;

scaling the 3D model of the product based on dimensions of the environment and the product;

determining content to be presented based on previous activities or selections while projecting the product in the environment;

selecting a best fit from jewelry variations, shoe variations, clothing variations, apparel variations or footwear variations.

18. The method of claim 17, comprising providing augmented reality view or virtual reality view of the one or more products with a user custom view.