Augmented reality system

• US 10,157,503 B2
• Filed: 05/05/2018
• Issued: 12/18/2018
• Est. Priority Date: 03/07/2016
• Status: Active Grant

• Current Assignee: Bao Tran
• Original Assignee: Bao Tran
• Inventors: Tran, Bao, Tran, Ha
• Primary Examiner(s): Wu, Sing-Wai
• Application Number:
  US15/972,149
• Publication Number:
  US 20180253906A1
• Time in Patent Office: 227 Days
• CPC Class Codes: A43B 17/00 Insoles for insertion, e.g. footbeds or inlays, for attachment to the shoe after the upper has been joined
• CPC Class Codes: A43B 3/34 with electrical or electronic arrangements
• CPC Class Codes: A43B 3/35 with electric heating arrangements
• CPC Class Codes: A43B 3/38 with power sources
• CPC Class Codes: A43B 7/24 Insertions or other supports preventing the foot canting to one side , preventing supination or pronation
• CPC Class Codes: A43D 1/025 comprising optical means, e.g. mirrors, photo-electric cells, for measuring or inspecting feet
• CPC Class Codes: A43D 2200/60 Computer aided manufacture of footwear, e.g. CAD or CAM
• CPC Class Codes: A45D 2044/007 Devices for determining the condition of hair or skin or for selecting the appropriate cosmetic or hair treatment
• CPC Class Codes: A45D 44/005 for selecting or displaying personal cosmetic colours or hairstyle
• CPC Class Codes: A61F 2/0059 Cosmetic or alloplastic implants A61F2/10 - A61F2/16, A61F2/28 take precedence
• CPC Class Codes: A61F 2/12 Mammary prostheses and implants
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 10/143 : Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 10/462 : Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/754 : involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 2005/2726 : for simulating a person's appearance, e.g. hair style, glasses, clothes
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Systems and methods are disclosed for recommending products or services by receiving a three-dimensional (3D) model of one or more products; performing motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using one video camera without a depth sensor in a mobile phone; acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location; and projecting the product in the environment.

Claims

1. A method for recommending products, comprising:

receiving a three-dimensional (3D) model of one or more products;

performing motion tracking and understanding an environment with points or planes and estimating light or color in the environment;

acquiring sensor data from sensors and optimizing features extracted from each image and sensor data; and

iteratively changing product position until satisfied to identify a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products; and

recommending a new product by looking up the correspondences among different manufacturer products; and

projecting the product in the environment.

2. The method of claim 1, comprising receiving images from an infrared camera.

3. The method of claim 1, wherein the product comprises an appliance or furniture.

4. The method of claim 1, wherein the product comprises a wearable item, a jean, or a shirt.

5. The method of claim 4, comprising rendering an image of the product on a mannequin.

6. The method of claim 4, comprising monitoring user health by analyzing changes in the 3D model over time.

7. The method of claim 4, comprising analyzing a user anatomical portion and selecting a best fit from apparel variations.

8. The method of claim 1, wherein the product comprises cosmetic product, a facial makeup product, a hair product.

9. The method of claim 1, comprising motion tracking, area learning and depth sensing the product.

10. The method of claim 1, comprising creating a 3D model using infrared images.

11. The method of claim 1, comprising identifying one or more best fitting products to the environment and displaying recommendations with one or more best fitting products in the environment.

12. The method of claim 11, wherein the best fitting products comprise clothing, shoes, cosmetics, appliances or furniture.

13. The method of claim 1, comprising:

capturing 3D model of user'"'"'s feet and identifying the subject'"'"'s current best fitting shoe products;

setting each best fitting shoe product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap and correlating different manufacturer'"'"'s shoe sizes and creating correspondences among different manufacturer shoe products.

14. A method for best fitting product variations to an environment, comprising:

receiving a three-dimensional (3D) model of one or more products;

performing motion tracking and understanding an and estimating light or color in the environment;

iteratively changing product position until satisfied to identify a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products;

recommending a new product by looking up the correspondences among different manufacturer products; and

generating an augmented or virtual reality display of the product in the environment.

15. A method for recommending a service, comprising:

receiving a three-dimensional (3D) model of a service to be applied to a target object;

performing motion tracking and understanding an and estimating light or color in the environment;

determining one or more dimensions of an environment of the target object;

generating the 3D model of the service as applied to the target object;

iteratively changing product position until satisfied to identify a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products; and

recommending a new product by looking up the correspondences among different manufacturer products;

creating correspondences among alternative services to apply the target object and scaling the 3D model of the generated 3D model based on dimensions of the environment and the service; and

generating an augmented or virtual reality display of the service as applied to the target object in the environment.

16. The method of claim 15, comprising applying the service to a product from one of:

a cosmetic product, a plastic surgery medical device, a facial makeup product, a hair product.

17. The method of claim 15, comprising:

capturing images of a face and a reference object from a plurality of angles using a mobile camera;

creating a 3D model of the face from the images with dimensions based on dimensions of the reference object;

selecting a makeup pattern or color from a plurality of makeup product variations; and

blending the makeup pattern or color onto the 3D model; and

displaying the makeup color on the face.

18. The method of claim 15, wherein the target object comprises a breast implant, comprising recommending a breast augmentation sizing to a patient.

19. The method of claim 15, wherein the products are facial make-ups, comprising:

capturing images of a face from a plurality of angles using a mobile camera;

detecting a skin tone from the face images;

selecting a makeup pattern or color from a plurality of makeup product variations; and

modeling the makeup pattern or color interaction with the skin tone; and

displaying a makeup on the face.