Generating a virtual wall in an augmented reality environment to simulate art displays

• US 10,621,786 B2
• Filed: 01/16/2018
• Issued: 04/14/2020
• Est. Priority Date: 01/16/2018
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Inventors: Shantharam, Nitin, Chi, Christin, Kelleher, Patrick John, Karnatakam, Ravi Theja, Fernandez, Christine Torrefranca
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US15/872,905
• Publication Number:
  US 20190221040A1
• Time in Patent Office: 819 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

The disclosed teachings disclose a method, apparatus, and system for generating a virtual wall in an augmented reality (AR) environment to simulate art displays. In particular, an AR environment may be generated based upon data representing the physical environment. Additionally, virtual artwork may be positioned on a virtual wall corresponding to a physical wall in the physical environment. The AR environment may further facilitate the selection, modification, and purchase of the simulated artwork.

Claims

1. A method for generating a virtual wall in an augmented reality (AR) environment to simulate art displays, the method comprising:

launching an application on a mobile device that includes a camera;

establishing, by the application, a network connection between the mobile device and a network-connected server system responsible for supporting the application;

obtaining, by the application, selection data indicative of a user selected artwork;

obtaining, by the application, sensor data including a stream of visual data of the physical environment from the camera;

generating, by the application, the AR environment updated in real-time based at least in part on the stream of visual data;

obtaining, by the application, user input data including a position of a single horizontal marker positioned by the user in the AR environment indicating the location where a representation of a physical wall meets a representation of a physical floor in the AR environment, wherein the user input data includes touch screen input indicating the position where a user positioned the single horizontal marker using a drag-and-drop gesture;

calculating distance data to the single horizontal marker relative to the mobile device based on the user input data;

determining, by the application, the position of the physical wall based on the calculated distance data;

constructing, by the application, a virtual wall based on the position of the physical wall;

placing, by the application, a virtual artwork representative of the user selected artwork on the virtual wall;

adjusting, by the application, characteristics of the virtual artwork, the adjusting including changing a perspective of the virtual artwork in real time based on the stream of visual data for the location of the user within the physical environment;

purchasing, by the application, a physical copy of the virtual artwork; and

transmitting, by the application, purchase data indicative of the purchase.

2. The method of claim 1, wherein the position of the physical wall is determined using functionality provided by an AR application programming interface.

3. The method of claim 1, wherein the user selected artwork is selected from among artwork data provided by the network-connected server system.

4. A method for generating a virtual wall in an augmented reality (AR) environment, the method comprising:

obtaining, by a processor, sensor data indicative of a physical environment;

generating, by the processor, the AR environment in real-time based on the sensor data;

obtaining, by the processor, user input data including a position of a single horizontal marker indicative of where a physical wall meets a physical floor wherein the user input data includes touch screen input indicating the position where a user positioned the single horizontal marker using a drag-and-drop gesture;

identifying, by the processor, location points along the single horizontal marker;

calculating, by the processor, distance data associated with the location points along the single horizontal marker relative to a user device that facilitates the touch screen input;

determining, by the processor, physical wall data indicating the distance, angle, and/or position of the physical wall based on the calculated distance data;

generating, by the processor, a virtual wall based upon the physical wall data; and

changing a perspective of the virtual wall in real time based on the sensor data for the location of the user within the physical environment.

5. The method of claim 4, wherein the single horizontal marker is positioned by the user in the AR environment to indicate the location where a representation of the physical wall meets a representation of the physical floor in the AR environment.

6. The method of claim 4, wherein the processor is located on a mobile device that includes a camera for capturing the sensor data.

7. The method of claim 6, wherein the distance data indicates the distance between the location points and the mobile device.

8. The method of claim 7, wherein when the physical wall data indicates angle of the physical wall, the angle of the physical wall is determined by comparing the distance data of the location points.

9. The method of claim 7, wherein the position of the physical wall is determined by using the distance data to determine the position of the location points relative to the location of the mobile device.

10. The method of claim 7, wherein the distance data of the location points is obtained using functionality provided by an AR application programming interface.

11. A computer system comprising:

a processor operable to execute instructions stored in a memory; and

instructions stored in the memory for generating a virtual wall in an augmented reality (AR) environment, wherein execution of the instructions causes the processor to;

obtain sensor data indicative of a physical environment;

generate the AR environment in real-time based on the sensor data;

obtain user input data including a position of a single horizontal marker indicative of where a physical wall meets a physical floor, wherein the user input data includes touch screen input indicating the position where a user positioned the single horizontal marker using a drag-and-drop gesture;

identify location points along the single horizontal marker;

calculate distance data associated with the location points along the single horizontal marker relative to a user device that facilitates the touch screen input;

determine physical wall data indicating the distance, angle, and/or position of the physical wall based on the distance data;

generate a virtual wall based upon the physical wall data; and

change a perspective of the virtual wall in real time based on the sensor data for the location of the user within the physical environment.

12. A method for facilitating the purchase of artwork simulated in an augmented reality (AR) environment, the method comprising:

displaying, on a mobile device, an artwork selection mode for obtaining an artwork selection;

displaying, on the mobile device, an instruction mode with instructions to configure the AR environment on the mobile device display;

displaying, on the mobile device, a configuration mode for displaying the AR environment in real-time and obtaining user input data indicative of a position of a single horizontal marker representing where an AR representation of a physical wall meets an AR representation of a physical floor in the AR environment, wherein the user input data includes touch screen input indicating the position where a user positioned the single horizontal marker using a drag-and-drop gesture;

calculating distance data to the single horizontal marker relative to the mobile device based on the user input data; and

displaying, on the mobile device, a purchase mode for viewing the artwork simulated in the AR environment on a virtual wall determined by the calculated distance data and obtaining purchase data indicative of a purchase of the artwork, the purchase mode configured to display a changed perspective of the virtual artwork in real time based on the user input data for the location of the user.

13. The method of claim 12, wherein the artwork selection indicates one or more artworks selected by the user of the mobile device.

14. The method of claim 12, wherein the artwork selection is selected from among artwork data provided by a server connected to the mobile device.

15. The method of claim 12, wherein the purchase data is transmitted to a server connected to the mobile device for processing and order fulfillment.