Providing a digital model of a corresponding product in a camera feed

• US 10,712,811 B2
• Filed: 12/12/2017
• Issued: 07/14/2020
• Est. Priority Date: 12/12/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Sun, Yi, Zhu, Jiajie
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US15/839,339
• Publication Number:
  US 20190179405A1
• Time in Patent Office: 945 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/16 Cloth

Abstract

One or more embodiments of the present disclosure provide an augmented reality presentation including a digital model of a product in connection with a camera stream. For example, systems and methods described herein provide a call-to-action associated with a digital model of a corresponding product. In response to detecting a selection of the call-to-action, systems and methods described herein provide the digital model. The system and methods described herein also activate a camera interface of a networking system application and render the digital model of the product within the camera interface. In this way, the systems and methods provide a portable user experience of trying on or otherwise sampling a product via an augmented reality presentation on a client device.

Claims

1. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:

provide, for display by a networking system application on the computing device, a digital media feed comprising one or more posts shared by users of a social networking system associated with the networking system application;

provide, within the digital media feed displayed by the networking system application, digital media portraying a product, wherein the digital media comprises a first call-to-action associated with the product;

detect, via the networking system application, a selection of the first call-to-action associated with the product portrayed in the digital media; and

in response to detecting the selection of the first call-to-action;

retrieve a digital model of the product portrayed in the digital media;

activate a camera interface of the networking system application on the computing device, wherein activating the camera interface comprises capturing a camera stream and providing the camera stream within the camera interface on the computing device;

generate an augmented reality experience by rendering, within the camera interface of the networking system application on the computing device, the digital model of the product portrayed in the digital media in connection with the camera stream; and

provide, within the camera interface, a second call-to-action configured to cause the computing device to locally store the digital model of the product portrayed in the digital media on the computing device for access to the augmented reality experience while offline.

2. The non-transitory computer-readable medium of claim 1, wherein the instructions, when executed by the at least one processor, cause the computing device to generate the augmented reality experience by:

detecting a face of a person within the camera stream;

identifying one or more features of the face within the camera stream; and

rendering the digital model of the product portrayed in the digital media in connection with the camera stream by placing the digital model of the product over a portion of the camera stream based on a location of the identified one or more features of the face within the camera stream.

3. The non-transitory computer-readable medium of claim 1, wherein the instructions, when executed by the at least one processor, cause the computing device to retrieve the digital model of the product portrayed in the digital media by:

providing, to a networking system, a request for the digital model of the product portrayed in the digital media; and

receiving, at the computing device, the digital model of the product portrayed in the digital media.

4. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:

provide, within the camera interface of the networking system application, an effects menu comprising a selectable icon corresponding to an additional digital model of an additional product;

detect, via the networking system application, a selection of the selectable icon within the effects menu; and

in response to detecting the selection of the selectable icon within the effects menu, render, within the camera interface of the networking system application on the computing device, the additional digital model of the additional product in connection with the camera stream.

5. The non-transitory computer-readable medium of claim 1, wherein the instructions, when executed by the at least one processor, cause the computing device to provide the digital media comprising the first call-to-action via a live-video feed of a co-user of the social networking system associated with the networking system application and in connection with the digital model of the product.

6. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:

identify a background effect associated with the digital model of the product; and

provide, within the camera interface of the networking system application on the computing device and in response to identifying the background effect associated with the digital model of the product, the background effect in connection with rendering the digital model of the product within the camera interface of the networking system application.

7. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:

in response to detecting the selection of the first call-to-action, receive an additional digital model corresponding to a related product to the product; and

provide, within the camera interface of the networking system application, a selectable icon associated with viewing the additional digital model corresponding to the related product.

8. The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to:

receive a selection of the second call-to-action; and

store, locally on the computing device, the digital model of the product portrayed in the digital media in response to receiving the selection of the second call-to-action.

9. The non-transitory computer-readable medium of claim 8, further comprising instructions that, when executed by the at least one processor, cause the computing device to:

provide, via a graphical user interface displayed on the computing device while the computing device is offline, a selectable icon corresponding to the digital model of the product portrayed in the digital media;

detect a selection of the selectable icon corresponding to the digital model; and

in response to detecting the selection of the selectable icon, provide an offline augmented reality experience for display on the computing device.

10. The non-transitory computer-readable medium of claim 1, wherein the one or more posts shared by the users of the social networking system associated with the networking system application comprises at least one of a video, a message, or an advertisement.

11. A method comprising:

generating, by at least one processor, a digital model of a product;

providing, for display by a networking system application on a client device, a digital media feed comprising one or more posts shared by users of a social networking system associated with the networking system application;

providing, within the digital media feed displayed by the networking system application, digital media portraying the product, wherein the digital media comprises a first call-to-action associated with the product;

detecting, via the networking system application, a selection of the first call-to-action associated with the product portrayed in the digital media; and

in response to detecting the selection of the first call-to-action;

retrieving the digital model of the product portrayed in the digital media;

providing the digital model of the product to the client device such that the client device provides an augmented reality experience by rendering the digital model of the product within a camera interface of the networking system application activated on the client device such that the digital model of the product appears over a camera stream captured by the client device; and

providing, within the camera interface, a second call-to-action configured to cause the client device to locally store the digital model of the product portrayed in the digital media on the client device for access to the augmented reality experience while offline.

12. The method of claim 11, wherein generating the digital model of the product comprises:

generating a mesh model for the product comprising structural and depth information associated with the product;

generating a UV map comprising surface appearance information associated with the product; and

combining the UV map and the mesh model to create the digital model of the product.

13. The method of claim 12, wherein generating the digital model of the product further comprises mapping points of the mesh model to corresponding reference features of a detected face within the camera stream captured by the client device.

14. The method of claim 13, wherein providing the augmented reality experience comprises rendering the digital model of the product at a location within the camera stream based on identified locations of the corresponding reference features of the detected face and corresponding mapping points of the mesh model for the product.

15. The method of claim 11, further comprising:

receiving, at a server device, the camera stream from the networking system application on the client device;

generating, by the at least one processor, the augmented reality experience by rendering the digital model of the product within the camera stream; and

wherein providing the digital model to the client device comprises providing the augmented reality experience to the networking system application on the client device.

16. The method of claim 15, further comprising:

broadcasting the augmented reality experience to one or more client devices of co-users of the social networking system associated with a user of the client device.

17. The method of claim 11, further comprising:

receiving an enhanced video from the client device comprising the camera stream and the digital model of the product appearing over the camera stream; and

broadcasting the enhanced video to one or more client devices of one or more co-users of the social networking system associated with a user of the client device.

18. A system comprising:

at least one processor;

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to;

provide, for display by a networking system application on a computing device, a digital media feed comprising one or more posts shared by users of a social networking system associated with the networking system application;

provide, within the digital media feed displayed by the networking system application, digital media portraying a product, wherein the digital media comprises a first call-to-action associated with the product;

detect, via the networking system application, a selection of the first call-to-action associated with the product portrayed in the digital media; and

in response to detecting the selection of the first call-to-action;

retrieve a digital model of the product portrayed in the digital media;

activate a camera interface of the networking system application on the computing device, wherein activating the camera interface comprises capturing a camera stream and providing the camera stream within the camera interface on the computing device;

generate an augmented reality experience by rendering, within the camera interface of the networking system application on the computing device, the digital model of the product portrayed in the digital media in connection with the camera stream; and

provide, within the camera interface, a second call-to-action configured to cause the computing device to locally store the digital model of the product portrayed in the digital media on the computing device for access to the augmented reality experience while offline.

19. The system of claim 18, wherein the instructions, when executed by the at least one processor, cause the system to generate the augmented reality experience by:

detecting a face of a person within the camera stream;

identifying one or more features of the face within the camera stream; and

rendering the digital model of the product portrayed in the digital media in connection with the camera stream by placing the digital model of the product over a portion of the camera stream based on a location of the identified one or more features of the face within the camera stream.

20. The system of claim 18, wherein:

the digital model comprises a mesh model comprising vertices associated with corresponding reference features of an object, and

the instructions, when executed by the at least one processor, cause the system to generate the augmented reality experience by;

detecting the object within the camera stream;

determining locations of reference features of the detected object within the camera stream; and

rendering the digital model of the product at a location within the camera stream based on the determined locations of reference features of the detected object within the camera stream.