Augmented reality system

• US 9,996,981 B1
• Filed: 02/27/2017
• Issued: 06/12/2018
• Est. Priority Date: 03/07/2016
• Status: Expired due to Fees

• Current Assignee: Bao Tran
• Original Assignee: Bao Tran
• Inventors: Tran, Bao, Tran, Ha
• Primary Examiner(s): Wu, Sing-Wai
• Application Number:
  US15/444,184
• Time in Patent Office: 470 Days
• CPC Class Codes: A43B 17/00 Insoles for insertion, e.g. footbeds or inlays, for attachment to the shoe after the upper has been joined
• CPC Class Codes: A43B 3/34 with electrical or electronic arrangements
• CPC Class Codes: A43B 3/35 with electric heating arrangements
• CPC Class Codes: A43B 3/38 with power sources
• CPC Class Codes: A43B 7/24 Insertions or other supports preventing the foot canting to one side , preventing supination or pronation
• CPC Class Codes: A43D 1/025 comprising optical means, e.g. mirrors, photo-electric cells, for measuring or inspecting feet
• CPC Class Codes: A43D 2200/60 Computer aided manufacture of footwear, e.g. CAD or CAM
• CPC Class Codes: A45D 2044/007 Devices for determining the condition of hair or skin or for selecting the appropriate cosmetic or hair treatment
• CPC Class Codes: A45D 44/005 for selecting or displaying personal cosmetic colours or hairstyle
• CPC Class Codes: A61F 2/0059 Cosmetic or alloplastic implants A61F2/10 - A61F2/16, A61F2/28 take precedence
• CPC Class Codes: A61F 2/12 Mammary prostheses and implants
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 10/143 : Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/17 : using hand-held instruments
• CPC Class Codes: G06V 10/462 : Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/754 : involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 2005/2726 : for simulating a person's appearance, e.g. hair style, glasses, clothes
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Systems and methods are disclosed for recommending products or services by receiving a 3D model of a product; capturing a reference object with a predetermined dimension in an environment where the product is to be placed using a mobile camera; determining one more dimensions of the environment relative to the predetermined dimension of the reference object; scaling the 3D model of the product based on dimensions of the environment and the product; and generating an augmented or virtual reality display of the product in the environment.

Claims

1. A method for recommending products, comprising:

receiving a three-dimensional (3D) model of one or more products;

performing motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using one video camera without a depth sensor in a mobile phone;

determining a target volume or space;

capturing images from a plurality of angles of the environment;

acquiring sensor data from sensors and optimizing features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;

determining candidate product(s) that fit the dimension(s);

scaling the 3D model of the product based on dimensions of the environment and the product;

projecting the product in the environment;

iteratively changing product position until satisfied;

identifying a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products; and

recommending a new product by looking up the correspondences among different manufacturer products and generating an augmented or virtual reality display of the new product in the environment.

2. The method of claim 1, wherein the mobile camera comprises a smart phone or a portable camera.

3. The method of claim 1, comprising depth-sensing the environment, motion-tracking and capturing predetermined dimension(s) in the environment where the product is to be placed based on camera images.

4. The method of claim 1, wherein the product comprises an appliance or furniture.

5. The method of claim 1, wherein the product comprises a wearable item, a jean, or a shirt.

6. The method of claim 5, comprising rendering an image of the product on a mannequin.

7. The method of claim 5, comprising monitoring user health by analyzing changes in the 3D model of an item worn by a user over time.

8. The method of claim 5, comprising analyzing a user anatomical portion and selecting a best fit from apparel variations.

9. The method of claim 1, wherein the product comprises cosmetic product, a facial makeup product, or a hair product.

10. The method of claim 1, comprising selecting a pattern or color from the product and blending the pattern or color of the product and the environment.

11. The method of claim 1, comprising creating a 3D model using infrared images.

12. The method of claim 1, comprising identifying one or more best fitting products to the environment and displaying recommendations with one or more best fitting products in the environment.

13. The method of claim 12, wherein the best fitting products comprise clothing, shoes, cosmetics, appliances or furniture.

14. The method of claim 1, comprising:

capturing 3D model of user'"'"'s feet and identifying the subject'"'"'s current best fitting shoe products;

setting each best fitting shoe product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gaps; and

correlating different manufacturer'"'"'s shoe sizes and creating correspondences among different manufacturer shoe products.

15. A method for best fitting product variations to an environment, comprising:

receiving a three-dimensional (3D) model of a product with one or more product variations;

performing depth-sensing using an infrared camera and motion-tracking camera as a single electronic module for a mobile phone device in addition to front and rear cameras in the mobile phone device and capturing a predetermined dimension in an environment where the product is to be placed using the electronic module;

scaling the 3D model of each product variation based on dimensions of the environment and characteristics of the product variation;

identifying a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products; and

recommending a new product by looking up the correspondences among different manufacturer products and generating an augmented or virtual reality display of the new product in the environment.

16. A method for recommending a service, comprising:

receiving a three-dimensional (3D) model of a service to be applied to a target object;

performing motion tracking, understanding an environment with points or planes and estimating light or color in the environment as detected by one video camera without a separate depth sensor;

determining dimensions in the environment based on the points or planes captured by the one video camera;

scaling the 3D model of the generated 3D model based on dimensions of the environment and the product;

identifying a best fitting product;

setting each best fitting product'"'"'s inside dimension with dimensions from the 3D model plus a predetermined gap;

correlating different manufacturer'"'"'s product sizes and creating correspondences among different manufacturer products; and

recommending a new product by looking up the correspondences among different manufacturer products and generating an augmented or virtual reality display of the new product in the environment.

17. The method of claim 16, comprising applying the service to a product from one of:

a cosmetic product, a plastic surgery medical device, a facial makeup product, and a hair product.

18. The method of claim 16, comprising:

capturing images of a face and a reference object from a plurality of angles using a mobile camera;

creating a 3D model of the face from the images with dimensions based on dimensions of the reference object;

selecting a makeup pattern or color from a plurality of makeup product variations;

blending the makeup pattern or color onto the 3D model; and

displaying the makeup color on the face.

19. The method of claim 16, wherein the target object comprises a breast implant, comprising recommending a breast augmentation sizing to a patient.

20. The method of claim 16, wherein the products are facial make-ups, comprising:

capturing images of a face from a plurality of angles using a mobile camera;

detecting a skin tone from the face images;

selecting a makeup pattern or color from a plurality of makeup product variations;

modeling the makeup pattern or color interaction with the skin tone; and

displaying a makeup on the face.