Service provision using personal audio/visual system

• US 9,128,520 B2
• Filed: 03/30/2012
• Issued: 09/08/2015
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A, Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J, Vaught, Benjamin I, Kipman, Alex Aben-Athar, McCulloch, Daniel, Cole, Jeffrey B, McIntyre, Ian D
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Gray, Ryan M
• Application Number:
  US13/436,511
• Publication Number:
  US 20130083063A1
• Time in Patent Office: 1,257 Days
• Field of Search: 345/419
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2340/145 related to small screens
• CPC Class Codes: G09G 2360/02 Graphics controller able to handle multiple formats, e.g. input or output formats
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/363 Graphics controllers

Abstract

A collaborative on-demand system allows a user of a head-mounted display device (HMDD) to obtain assistance with an activity from a qualified service provider. In a session, the user and service provider exchange camera-captured images and augmented reality images. A gaze-detection capability of the HMDD allows the user to mark areas of interest in a scene. The service provider can similarly mark areas of the scene, as well as provide camera-captured images of the service provider'"'"'s hand or arm pointing to or touching an object of the scene. The service provider can also select an animation or text to be displayed on the HMDD. A server can match user requests with qualified service providers which meet parameters regarding fee, location, rating and other preferences. Or, service providers can review open requests and self-select appropriate requests, initiating contact with a user.

Claims

1. A method for use of head-mounted display device worn by a service consumer, the method comprising:

receiving image data of a scene from at least one forward-facing camera;

obtaining gaze direction data, the gaze direction data indicating a gaze direction of the service consumer;

controlling an augmented reality projection system to provide an augmented reality image which highlights one area in the scene corresponding to the gaze direction of the service consumer;

receiving a command from the service consumer to mark the gaze direction as a marked gaze direction; and

in response to the command, communicating the image data of the scene with the augmented reality image which highlights the one area in the scene to a computing device of a service provider, the service provider generating data to assist the service consumer in performing an activity in the scene;

receiving the data generated by the service provider; and

controlling the augmented reality projection system based on the data generated by the service provider to project an additional augmented reality image to the service consumer which highlights another area in the scene, to assist the service consumer in performing the activity.

2. The method of claim 1, wherein:

the gaze direction data indicates gaze directions of the service consumer; and

the augmented reality projection system provides augmented reality images which highlight areas in the scene corresponding to the gaze directions of the service consumer; and

the command from the service consumer marks one of the gaze directions as the marked gaze direction.

3. The method of claim 1, wherein:

the data generated by the service provider includes gaze direction data of the service provider; and

the augmented reality projection system is controlled to provide at least one augmented reality image which highlights an area in the scene corresponding to the gaze direction data of the service provider.

4. The method of claim 1, wherein:

the data generated by the service provider comprises a sequence of images of hand movements of the service provider; and

the augmented reality projection system is controlled to provide the sequence of images of hand movements of the service provider as a sequence of augmented reality images superimposed over the scene.

5. The method of claim 4, wherein:

the sequence of augmented reality images superimposed over the scene comprises the additional augmented reality image which highlights another area in the scene.

6. The method of claim 1, further comprising:

in response to a command by the service consumer, communicating a request for assistance in performing the activity to a service server, the request identifies the activity and comprises an image which describes the activity.

7. The method of claim 1, further comprising:

in response to a command by the service consumer, communicating a request for assistance in performing the activity to a service server, the request identifies the activity and comprises an image which describes the activity, and the request is made available to a plurality of service providers;

pursuant to the request, receiving data from the service server identifying the plurality of service providers;

controlling the augmented reality projection system to project at least one augmented reality image to the service consumer which identifies the plurality of service providers; and

in response to a command by the service consumer selecting one of the service providers, establishing a connection with the one of the service providers.

8. The method of claim 1, wherein:

setting a brightness of the augmented reality image which highlights the one area in the scene to be higher than a brightness of the scene.

9. The method of claim 1, wherein:

setting a color of the augmented reality image which highlights the one area in the scene to contrast with a color of the scene.

10. The method of claim 1, further comprising:

setting a degree of transparency of the augmented reality image which highlights the one area in the scene to allows the service consumer to see the one area behind the augmented reality image which highlights the one area.

11. A computing device of a service provider, the computing device comprising:

a display device;

a communication interface;

an eye tracking camera; and

at least one control circuit in communication with the display device, the communication interface and the eye tracking camera, the at least one control circuit is configured to;

receive image data of a scene from a head-mounted display device worn by a service consumer, via the communication interface, the image data is obtained from at least one forward-facing camera of the head-mounted display device worn by the service consumer;

control the display device to display at least one image to the service provider, the at least one image depicts the scene;

receive a command from the service provider which identifies an area in the scene which the service provider calls to an attention of the service consumer;

in response to the command, generate data to assist the service consumer in performing an activity;

communicate the data to assist the service consumer in performing the activity to the head-mounted display device worn by the service consumer, via the communication interface, the data to assist the service consumer in performing the activity is configured to be used to project to the service consumer via the head-mounted display device worn by the service consumer, at least one augmented reality image which highlights the area in the scene;

obtain gaze direction data of the service provider from the eye tracking camera, the gaze direction data identifies the area in the scene; and

communicate the gaze direction data to the head-mounted display device worn by the service consumer.

12. The computing device according to claim 11, further comprising:

a forward-facing camera which captures a sequence of images of movement of a hand of the service provider, the at least one control circuit is configured to communicate the sequence of images of the movement of the hand to the head-mounted display device worn by the service consumer.

13. The computing device according to claim 11, wherein:

the at least one control circuit is configured to receive gaze direction data from the head-mounted display device worn by the service consumer, the gaze direction data from the head-mounted display device worn by the service consumer indicates another area in the scene, and to superimpose a highlight of the another area in the scene on the at least one image which depicts the scene.

14. A method for assisting a service consumer wearing a head-mounted display device, the method comprising:

communicating a request for assistance in performing an activity to a service server, the request includes an identification of the activity and an image which describes the activity, and the request is made available to a pool of qualified service providers;

in response to at least one service provider of the pool of qualified service providers accepting the request, communicating with the at least one service provider via the head-mounted display device, the communicating comprises communicating image data from a forward-facing camera of the head-mounted display device to a computing device of the at least one service provider, and receiving data generated by the at least one service provider to assist the service consumer in performing the activity; and

controlling an augmented reality projection system based on the data generated by the at least one service provider to project at least one augmented reality image to the service consumer, to assist the service consumer in performing the activity.

15. The method of claim 14, wherein:

the communicating with the at least one service provider comprises communicating gaze direction data from the head-mounted display device to a computing device of the at least one service provider, the gaze direction data indicates an area in a scene at which the service consumer gazes.

16. The method of claim 14, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a fee parameter indicating a maximum desired fee that the service consumer desires to pay for the assistance.

17. The method of claim 14, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a location parameter indicating a desired location of a service provider.

18. The method of claim 14, wherein:

the request includes an identification of the service consumer; and

the identification of the service consumer is associated with a rating parameter indicating a desired minimum rating of a service provider.

19. The method of claim 14, further comprising:

pursuant to the request, receiving data from the service server identifying the at least one service provider; and

based on the data, controlling the augmented reality projection system of the head-mounted display device to project at least one augmented reality image to the service consumer which identifies the at least one service provider.