Motion-based interactive shopping environment

• US 9,098,873 B2
• Filed: 04/01/2010
• Issued: 08/04/2015
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Tsunoda, Kudo, Bennett, Darren, Murphy, Brian S., Latta, Stephen G., Markovic, Relja, Kipman, Alex
• Primary Examiner(s): GEORGALAS, ANNE MARIE
• Application Number:
  US12/752,917
• Publication Number:
  US 20110246329A1
• Time in Patent Office: 1,951 Days
• Field of Search: 705/26.1, 705/27.1
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers

Abstract

An on-screen shopping application which reacts to a human target user'"'"'s motions to provide a shopping experience to the user is provided. A tracking system captures user motions and executes a shopping application allowing a user to manipulate an on-screen representation the user. The on-screen representation has a likeness of the user or another individual and movements of the user in the on-screen interface allows the user to interact with virtual articles that represent real-world articles. User movements which are recognized as article manipulation or transaction control gestures are translated into commands for the shopping application.

Claims

1. A computer implemented method of presenting an interface on a display device, the method implemented by a computer coupled to a capture device, comprising:

capturing a scene using the capture device by creating depth images of the scene and RGB data, the scene including a human target;

rendering by the computer of a representation of the scene and a visual representation of the human target based on a skeletal model based on the depth images and the RGB data;

presenting the interface on the display device allowing the human target to select and position one or more of a plurality of virtual articles representing real articles available for purchase, in the representation on the display device, each of the one or more of the plurality of virtual articles defined by a three dimensional model;

tracking, by the computer, of movements of the human target in the scene using depth information from the depth images, the depth information provided by the capture device to the computer; and

responsive to said tracking of movements, the computer altering the rendering of the representation of the scene and the visual representation on the display device by mimicking movement of the human target in the visual representation on the display, including;

applying the movements of the human target causing the visual representation to engage the one or more of the plurality of virtual articles; and

applying the movements of the human target engaging the one or more of the plurality of virtual articles to place and move the one or more of the plurality of virtual articles when the movements of the human target in the scene indicate an article manipulation gesture; and

applying the movements of the human target to a purchasing interface to purchase a real article corresponding to the one or more of the plurality of virtual articles when the movements of the human target in the scene indicate an transaction manipulation gesture.

2. The computer implemented method of claim 1 wherein the step of applying movements of the human target includes positioning the one or more of the plurality of virtual articles on the visual representation of the human target responsive to the article manipulation gesture.

3. The computer implemented method of claim 2 wherein positioning the one or more of the plurality of virtual articles on the visual representation of the human target includes applying the movements of the human target to the visual representation of the human target.

4. The computer implemented method of claim 1 wherein the step of applying movements of the human target includes selecting the one or more of the plurality of virtual articles, positioning the one or more of the plurality of virtual articles at a location within the scene and releasing the one or more of the plurality of virtual articles within the scene.

5. The computer implemented method of claim 4 wherein positioning comprises rendering the one or more of the plurality of virtual articles in a representation maintaining proportions of the one or more of the plurality of virtual articles to the visual representation of the human target.

6. The computer implemented method of claim 1 further including the step of receiving three dimensional models of the one or more of the plurality of virtual articles for inclusion in the representation of the scene from third party vendors.

7. The computer implemented method of claim 1 further including generating three dimensional models of the one or more of the plurality of virtual articles from captured information in the scene.

8. The computer implemented method of claim 1 further including generating a representation of the human target by gathering image information and the depth information, and building a mesh model of a body of the human target.

9. The computer implemented method of claim 8 wherein the visual representation of the human target is of a different individual than the human target.

10. A method implemented by a first processing device coupled to a first capture device and a first display device, comprising:

scanning a first scene using the first capture device to capture information comprising depth images and RGB data for objects in the first scene;

parsing the information from the first capture device by the first processing device to determine a first human target user in the first scene;

rendering a virtual representation of the first human target user in the first scene on the first display device, the virtual representation having a likeness of the first human target user and based on a skeletal model derived from the depth images and RGB data;

presenting one or more virtual articles in the scene on the display device, the one or more virtual articles corresponding to one or more real world articles;

tracking, by the first processing device, of movements of the first human target user in the first scene responsive to the first capture device and applying said movements to the virtual representation of the first human target user and said one or more virtual articles on the first display device;

the first processing device recognizing at least one article manipulation gesture of the first human target user responsive to the capture device, the article manipulation gesture indicating an instruction to manipulate at least one of the one or more virtual articles relative to the virtual representation of the first human target user;

altering the virtual representation on the display device by applying the article manipulation gesture to change the virtual representation;

recognizing at least one transaction manipulation gesture, the transaction manipulation gesture comprising an instruction to perform a transaction relative to an article corresponding to the one or more virtual articles selected by the first human target user; and

applying the at least one transaction manipulation gesture to a purchase transaction.

11. The method of claim 10 further including capturing movements of the first human target user using the first capture device and the first processing device in a first physical location and transmitting said information to a second processing device at a second physical location, the second processing device including a second capture device and a second display device and detecting a second human target at the second physical location, further including rendering a second virtual representation of the first human target user in the first scene on the second display device, the second virtual representation having a likeness of the first human target user;

presenting the one or more virtual articles in the first scene on the second display device; and

tracking by the second processing device of movements of the second human target in the second physical location responsive to the second capture device and applying said movements to the second virtual representation of the second human target and the one or more virtual articles in the virtual representation on the first display device.

12. The method of claim 11 wherein the first processing device and the second processing device are coupled by a network.

13. The method of claim 12 wherein the first and second processing devices are coupled to a vendor via the network, and the method further includes the step of:

providing a purchasing interface, the purchasing interface responsive to a transaction gesture, the transaction gesture initiating a purchase of a real world article corresponding to the one or more of the virtual articles with the vendor.

14. The method of claim 13 further including the step of receiving three dimensional models of virtual articles for inclusion in the virtual representation of the first scene from third party vendors.

15. The method of claim 14 further including generating the visual representation of the first human target user using depth information and building a mesh model of a body of the first human target user.

16. An apparatus, comprising

a capture device including a first memory;

code stored in the first memory configured to instruct the capture device to continuously capture information comprising depth images and RGB data for a scene, the scene including a human target, the depth images including information on the human target;

a processor;

a second memory associated with the processor; and

code stored in the second memory configured to instruct the processor to;

render a virtual representation of the human target on a display device, including tracking the human target based on a skeletal model derived from the depth images and RGB data, the virtual representation mimicking movement of the human target in the virtual representation on the display;

present an interface allowing the human target to manipulate virtual articles representing real articles available for purchase relative to the virtual representation on the display;

alter the virtual representation to apply at least one of the virtual articles to the virtual representation of the human target based on an article model;

recognize at least one transaction manipulation gesture performed by the human target relative to the at least one of the virtual articles; and

complete a purchase transaction by applying the at least one transaction manipulation gesture to purchase a real world article corresponding to the at least one of the virtual articles.

17. The apparatus of claim 16, the code stored in the second memory further configured to instruct the processor to render a purchasing interface on the display, the purchasing interface responsive to a transaction gesture which initiates a purchase of a real world article corresponding to the at least one of the virtual articles with a vendor.

18. The apparatus of claim 16, the code stored in the second memory further configured to apply movements of the human target to apply the at least one of the virtual articles on the virtual representation of the human target responsive to an article manipulation gesture.