Identification of objects in a scene using gaze tracking techniques

• US 10,068,134 B2
• Filed: 05/03/2016
• Issued: 09/04/2018
• Est. Priority Date: 05/03/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Gordon, John C.
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/145,568
• Publication Number:
  US 20170323158A1
• Time in Patent Office: 854 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06Q 50/20 Education
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Techniques and systems for identifying objects using gaze tracking techniques are described. A computing system may determine or infer that an individual is requesting to identify an object that is unknown to the individual based at least partly on images of the individual, images of a scene including the object, or both. In some cases, images of the individual may be used to determine a gaze path of the individual and the unknown object may be within the gaze path of the individual. Additionally, a computing system may send a request to identify the object to at least one individual. One or more of the responses received from the at least one individual may be provided in order to identify the object.

Claims

1. A computing device comprising:

one or more processors; and

one or more computer-readable storage media storing instructions that are executable by the one or more processors to perform operations comprising;

obtaining a first image of an eye of an individual;

determining a gaze path of the individual based at least partly on the first image;

obtaining a second image of at least a portion of a scene, wherein the scene includes one or more objects;

determining a position of an object included in the scene based at least partly on the second image;

determining that the object is within the gaze path of the individual based at least partly on the position of the object;

receiving input to identify the object;

determining a plurality of characteristics of the object;

selecting a group of individuals from a plurality of groups of individuals, wherein the group of individuals is known to be associated with a set of object characteristics that correspond, at least in part, to the plurality of characteristics of the object;

sending communications to computing devices of the group of individuals, the communications including a request to identify the object;

receiving a response from a computing device of a member of the group, the response including an identifier of the object; and

providing output including the identifier of the object.

2. The computing device of claim 1, wherein the computing device is in communication with a data store storing a database of object information, the database of object information including identifiers, categories, and characteristics related to a plurality of objects.

3. The computing device of claim 2, wherein:

the operations further comprise;

performing a comparison of the plurality of characteristics of the object with characteristics of at least a portion of the plurality of objects included in the database of object information; and

determining, based at least partly on the comparison, that the object is absent from the database of object information, wherein the communications are sent to the computing devices of the group of individuals based at least partly on determining that the object is absent from the database of object information.

4. The computing device of claim 1, wherein the group of individuals is known to be associated with a category of objects.

5. The computing device of claim 4, wherein the operations further comprise:

performing a comparison of the plurality of characteristics with the set of object characteristics associated with the group;

determining, based on the comparison, a similarity score for the object and the group;

determining that the similarity score is above a threshold score; and

wherein selecting the group from the plurality of groups is based at least partly on the similarity score being above the threshold score.

6. The computing device of claim 1, wherein:

the response is one of a plurality of responses received from the computing devices of the group;

the operations further comprise;

determining a plurality of identifiers of the object included in the plurality of responses;

determining a number of responses associated with each identifier of the plurality of identifiers; and

determining a ranking of the plurality of responses based at least partly on the number of responses for each identifier; and

providing the output including the identifier of the object is based at least partly on the ranking.

7. The computing device of claim 1, wherein the operations further comprise:

determining a category of the object based at least partly on the plurality of characteristics of the object; and

adding the object to a database of object information by storing the plurality of characteristics of the object in the database of object information, and wherein the object is associated with the category in the database of object information.

8. A computing system comprising:

one or more processors;

one or more computer-readable storage media storing instructions that are executable by the one or more processors to perform operations comprising;

determining a gaze path of an individual;

determining a position of an object included in a scene;

determining that the object is within the gaze path of the individual based at least partly on the position of the object;

receiving input data including at least one of audible input, visual input, or sensor input;

determining that the input data corresponds to a request to identify the object;

determining an identifier of the object;

generating output data including the identifier of the object;

determining that additional input data received indicates that the input data corresponded to identifying a different object in the scene;

projecting a visual indicator onto an additional object in the scene;

determining that further input data corresponds to an additional request to identify the additional object;

determining an additional identifier of the additional object; and

generating additional output data including the additional identifier of the additional object.

9. The computing system of claim 8, further comprising:

a first camera to capture first images of an individual; and

a second camera to capture second images of the scene; and

wherein;

the gaze path of the individual is determined based at least partly on at least a portion of the first images; and

the position of the object included in the scene is determined based at least partly on at least a portion of the second images.

10. The computing system of claim 9, wherein:

the first images and the second images are captured by an electronic device;

the operations further comprise;

sending the first images to a server computing device that is located remotely from the electronic device; and

sending the second images to the server computing device; and

determining the gaze path of the individual is based at least partly on first information received from the server computing device indicating the gaze path of the individual; and

determining the position of the object included in the scene is based at least partly on second information received from the server computing device indicating the position of the object.

11. The computing system of claim 8, wherein the operations further comprise:

determining characteristics of the additional object based at least partly on images of the scene, the characteristics of the additional object including at least one of a shape of the additional object, contours of the additional object, at least one color of the additional object, at least one dimension of the additional object, a use of the additional object, or components of the additional object;

determining characteristics of the scene based at least partly on the images of the scene; and

determining a context of the additional object based at least partly on the characteristics of the scene, the context including a location of the additional object.

12. The computing system of claim 11, further comprising:

a data store storing a database of object information, the database of object information including contexts for objects and characteristics of the objects;

wherein the operations further comprise;

performing a first comparison between the characteristics of the additional object with the characteristics of the objects stored in the database of object information;

performing a second comparison between the context of the additional object and the contexts for the objects stored in the database of object information; and

determining that the additional object is absent from the database of object information based at least partly on the first comparison and the second comparison.

13. The computing system of claim 11, wherein the operations further comprise:

selecting a group of individuals from among a plurality of groups of individuals based at least partly on the characteristics of the additional object and the context of the additional object; and

sending communications to computing devices of the group, the communications including at least one of an image of the additional object, the characteristics of the additional object, or the context of the additional object.

14. The computing system of claim 8, wherein determining that the input data corresponds to the request to identify the object includes at least one of:

analyzing the audible input to identify sounds, words, or both that correspond with predetermined sounds, predetermined words, or both that are associated with requests to identify objects;

analyzing the visual input to identify a gesture that corresponds with predetermined gestures that are associated with requests to identify objects;

or

analyzing the sensor input to identify an electroencephalogram (EEG) pattern that corresponds with predetermined EEG patterns that are associated with requests to identify objects.

15. A method comprising:

obtaining a first image of an eye of an individual;

determining, by one or more processors, a gaze path of the individual based at least partly on the first image;

obtaining a second image of at least a portion of a scene, wherein the scene includes one or more objects;

determining a position of an object included in the scene based at least partly on the second image;

determining that the object is within the gaze path of the individual based at least partly on the position of the object;

receiving input to identify the object;

determining a plurality of characteristics of the object;

selecting a group of individuals from a plurality of groups of individuals, wherein the group of individuals is known to be associated with a set of object characteristics that correspond, at least in part, to the plurality of characteristics of the object;

sending communications to computing devices of the group of individuals, the communications including a request to identify the object;

receiving a response from a computing device of a member of the group, the response including an identifier of the object; and

providing output including the identifier of the object.

16. The method of claim 15, further comprising communicating with a data store storing a database of object information, the database of object information including identifiers, categories, and characteristics related to a plurality of objects.

17. The method of claim 16, further comprising:

performing a comparison of the plurality of characteristics of the object with characteristics of at least a portion of the plurality of objects included in the database of object information; and

determining, based at least partly on the comparison, that the object is absent from the database of object information, wherein the communications are sent to the computing devices of the group of individuals based at least partly on determining that the object is absent from the database of object information.

18. The method of claim 15, wherein the group is known to be associated with a category of objects.

19. The method of claim 18, further comprising:

performing a comparison of the plurality of characteristics with the set of object characteristics associated with the group;

determining, based on the comparison, a similarity score for the object and the group;

determining that the similarity score is above a threshold score; and

wherein selecting the group from the plurality of groups is based at least partly on the similarity score being above the threshold score.