Service Provision Using Personal Audio/Visual System

• US 20130083063A1
• Filed: 03/30/2012
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, McCulloch, Daniel, McIntyre, Ian D., Cole, Jeffrey B.
• Granted Patent: US 9,128,520 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2340/145 related to small screens
• CPC Class Codes: G09G 2360/02 Graphics controller able to handle multiple formats, e.g. input or output formats
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/363 Graphics controllers

Abstract

A collaborative on-demand system allows a user of a head-mounted display device (HMDD) to obtain assistance with an activity from a qualified service provider. In a session, the user and service provider exchange camera-captured images and augmented reality images. A gaze-detection capability of the HMDD allows the user to mark areas of interest in a scene. The service provider can similarly mark areas of the scene, as well as provide camera-captured images of the service provider'"'"'s hand or arm pointing to or touching an object of the scene. The service provider can also select an animation or text to be displayed on the HMDD. A server can match user requests with qualified service providers which meet parameters regarding fee, location, rating and other preferences. Or, service providers can review open requests and self-select appropriate requests, initiating contact with a user.

Claims

1. A method for use of head-mounted display device worn by a service consumer, the method comprising:

receiving image data of a scene from at least one forward-facing camera;

communicating the image data of the scene to a computing device of a service provider, the service provider generating data based on the image data of the scene, to assist the service consumer in performing an activity in the scene;

receiving the data generated by the service provider; and

controlling an augmented reality projection system based on the data generated by the service provider to project at least one augmented reality image to the service consumer, to assist the service consumer in performing the activity.

2. The method of claim 1, further comprising:

obtaining gaze direction data, the gaze detection data indicating an area of the scene at which the service consumer gazes; and

communicating the gaze direction data to the computing device of the service provider, to identify, at the computing device of the service provider, the area of the scene at which the service consumer gazes.

3. The method of claim 1, further comprising:

obtaining gaze direction data, the gaze detection data indicating gaze directions of the service consumer;

receiving a command from the service consumer to mark one of the gaze directions as a marked gaze direction; and

communicating data to the computing device of the service provider informing the service provider of the marked gaze direction.

4. The method of claim 1, further comprising:

communicating data depicting at least one other augmented reality image which is projected by the augmented reality projection system, to the computing device of the service provider for display at the computing device of the service provider.

5. The method of claim 1, wherein:

the data generated by the service provider includes gaze direction data of the service provider; and

the at least one augmented reality image highlights an area in the scene corresponding to the gaze direction data of the service provider.

6. The method of claim 1, wherein:

the at least one augmented reality image highlights an area of the scene which the service provider calls to the attention of the service consumer.

7. The method of claim 1, wherein:

the at least one augmented reality image comprises an animation which is superimposed over the scene.

8. The method of claim 1, wherein:

the at least one augmented reality image comprises a sequence of images which identify a sequence of hand movements of the service provider which is superimposed over the scene.

9. The method of claim 1, further comprising:

in response to a command by the service consumer, communicating a request for assistance in performing the activity to a service server, the request identifies the activity.

10. The method of claim 1, further comprising:

in response to a command by the service consumer, communicating a request for assistance in performing the activity to a service server, the request identifies the activity;

pursuant to the request, receiving data from the service server identifying a plurality of service providers;

controlling the augmented reality projection system to project at least one augmented reality image to the service consumer which identifies the plurality of service providers; and

in response to a command by the service consumer selecting one of the service providers, establishing a connection with the one of the service providers.

11. A computing device of a service provider, the computing device comprising:

a display device;

a communication interface; and

at least one control circuit in communication with the display device and the communication interface, the at least one control circuit;

receives image data of a scene from a head-mounted display device worn by the service consumer, via the communication interface, the image data is obtained from at least one forward-facing camera of the head-mounted display device worn by the service consumer;

controls the display device to display at least one image to the service provider, the at least one image depicts the scene;

receives a command from the service provider;

in response to the command, generates data to assist the service consumer in performing an activity; and

communicates the data to assist the service consumer in performing the activity to the head-mounted display device worn by the service consumer, via the communication interface, the data to assist the service consumer in performing the activity is adapted to be used to project at least one augmented reality image to the service consumer via the head-mounted display device worn by the service consumer.

12. The computing device according to claim 11, further comprising:

an eye tracking camera, the at least one control circuit obtains gaze direction data of the service provider from the eye tracking camera, and communicates the gaze direction data to the head-mounted display device worn by the service consumer, the gaze direction data indicates an area of the scene which the service provider calls to the attention of the service consumer.

13. The computing device according to claim 11, further comprising:

a forward-facing camera which captures a movement of a hand of the service provider, the data to assist the service consumer includes images which depict the movement of the hand.

14. The computing device according to claim 11, wherein:

the at least one control circuit receives gaze direction data from the head-mounted display device worn by the service consumer, the gaze direction data indicates an area in the scene corresponding to the gaze direction data.

15. A method for assisting a service consumer wearing a head-mounted display device, the method comprising:

communicating a request for assistance in performing an activity to a service server, the request includes an identification of the activity;

in response to the at least one service provider accepting the request, communicating with the at least one service provider via the head-mounted display device, the communicating comprises communicating image data from a forward-facing camera of the head-mounted display to a computing device of the at least one service provider, and receiving data generated by the at least one service provider to assist the service consumer in performing the activity; and

controlling the augmented reality projection system based on the data generated by the at least one service provider to project at least one augmented reality image to the service consumer, to assist the service consumer in performing the activity.

16. The method of claim 15, wherein:

the communicating with the at least one service provider comprises communicating gaze direction data from the head-mounted display device to a computing device of the at least one service provider, the gaze direction data indicates an area of a scene at which the service consumer gazes.

17. The method of claim 15, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a fee parameter indicating a maximum desired fee that the service consumer desires to pay for the assistance.

18. The method of claim 15, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a location parameter indicating a desired location of a service provider.

19. The method of claim 15, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a rating parameter indicating a desired minimum rating of a service provider.

20. The method of claim 15, further comprising:

pursuant to the request, receiving data from the service server identifying the at least one service provider;

based on the data, controlling the augmented reality projection system of the head-mounted display device to project at least one augmented reality image to the service consumer which identifies the at least one service provider.