Method and system for user initiated query searches based on gaze data

• US 10,055,495 B2
• Filed: 10/29/2012
• Issued: 08/21/2018
• Est. Priority Date: 10/28/2011
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Olsson, Anders, Skogo, Marten
• Primary Examiner(s): Hasan, Syed H
• Application Number:
  US14/354,874
• Publication Number:
  US 20140310256A1
• Time in Patent Office: 2,122 Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range

Abstract

The invention generally relates to computer implemented systems and methods for utilizing detection of eye movements in connection with interactive graphical user interfaces and, in particular, for utilizing eye tracking data to facilitate and improve information search and presentation in connection with interactive graphical user interfaces. A gaze search index is determined based on information that has been presented on an information presentation area and gaze data signals. The gaze search index comprises links between gaze search parameters and presented information that satisfies gaze filter criteria for respective gaze search parameter. A user can initiate query searches for information on the computer device or on information hosts connectable to the computer device via networks by using combinations of gaze search parameters of the gaze search index and non gaze search parameters of a non gaze search index.

Claims

1. A method for searching information using a computer device associated with an information presentation area, the method comprising:

displaying, on the information presentation area, a search window comprising at least one field for presenting search parameters including gaze search parameters and non gaze search parameters for a user, the gaze search parameters;

representing a predefined set of specific mental conditions experienced by the user when the user viewed presented information during a certain period of time prior to displaying the search window; and

being based on at least one of a duration of blinks and a pupil size of the user measured during the certain period of time when the user viewed the presented information;

displaying the gaze search parameters as selectable posts in the search window, the set of selectable posts comprising a set of labels corresponding to the predefined set of specific mental conditions; and

executing a query search on the computer device or on information hosts connectable to the computer device via networks, by using a selected combination of at least one of the gaze search parameters and at least one of the non gaze search parameters.

2. The method for searching information using a computer device according to claim 1, further comprising:

creating the gaze search parameters based on information that has been presented on the information presentation area for the user and gaze data signals connected with the presented information provided from a gaze tracking module associated with the computer device.

3. The method for searching information using a computer device according to claim 1, further comprising:

creating the gaze search parameters based on information that has been presented for the user on information presentation areas, each being associated with a computer device, and gaze data signals connected with the presented information provided from gaze tracking modules associated with the computer devices.

4. The method for searching information using a computer device according to claim 1, further comprising:

creating the gaze search parameters based on information that has been presented on multiple information presentation areas, each being associated with a computer device, for multiple users and gaze data signals connected with the presented information provided from gaze tracking modules associated with the computer devices.

5. The method for searching information using a computer device according to claim 1, further comprising:

displaying a search result presentation window on the information presentation area presenting a search result from the query search.

6. The method for searching information using a computer device according to claim 1, further comprising:

extracting the gaze search parameters based on gaze tracking data and information of data that has been presented on the information presentation area using one or more gaze search parameter filters.

7. The method for searching information using a computer device according to claim 1, wherein the gaze search parameters comprise at least one of the following:

presented information that the user has looked at for at least a certain period of time; and

/or

presented information that the user has looked at during a period of time while being distracted; and

/or

presented information that has caused strong emotional reaction of the user; and

/or

presented information that the user has focused on; and

/or

articles that the user has read at least a predetermined amount of the included text; and

/or

articles or texts that have been thoroughly read by the user; and

/or

articles or texts that have been skimmed by the user; and

/or

articles or texts the user read while being tired.

8. The method for searching information using a computer device according to claim 1, further comprising analyzing at least one set of previous gaze search parameters to determine a behavioral pattern, wherein the behavioral pattern can be used to organize and/or prioritize search results.

9. The method for searching information using a computer device according to claim 1, further comprising presenting a graphical representation on said information presentation area, wherein said graphical representation enables user input for customizing the query search in accordance with user preferences including defining a time criteria as a gaze search parameter.

10. The method for searching information using a computer device according to claim 1, wherein the computer device is one of a plurality of computer devices each of which being associated with a gaze tracking module configured to provide gaze data signals and an information presentation area for presenting data, wherein said method further comprises collecting raw or pre-processed gaze data signals and information on data shown on the information presentation areas from the computer devices and providing a set of aggregated gaze search parameters based on the gaze data signals and the data presented on the information presentation areas.

11. The method for searching information using a computer device according to claim 1, further comprising displaying a search window on the presentation area comprising at least one field for presenting the gaze search parameters reflecting a reading pattern of the user when reading presented information, the gaze search parameters allowing the user to define a specific query search by selecting whether to include or exclude a specific gaze search parameter in the specific query search.

12. The method for searching information using a computer device according to claim 1, further comprising displaying a search window on the presentation area comprising at least one field for presenting the gaze search parameters reflecting a period of time when the user viewed data, the gaze search parameters allowing the user to define a specific query search by selecting whether to include or exclude a specific gaze search parameter in the specific query search.

13. The method for searching information using a computer device according to claim 1, wherein displaying the search window on the information presentation area comprises displaying the search window on the information presentation area of a head-mounted display worn by the user.

14. The method for searching information using a computer device according to claim 1, wherein displaying the search parameters as selectable posts in the search window further comprises:

displaying the gaze search parameters as first set of selectable posts in the search window; and

displaying the non gaze search parameters as a second set of selectable posts in the search window.

15. The method for searching information using a computer device according to claim 14, wherein the selected combination is selected by:

receiving a first user selection of a first label of the set of labels, the first label being of a first selectable post of the set of selectable posts, and the first selectable post corresponding to the at least one gaze search parameter; and

receiving a second user selection of a second selectable post from the second set of selectable posts, the second selectable post corresponding to the at least one non gaze search parameter.

16. A search module for information searches using a computer device associated with an information presentation area, wherein said search module comprises:

a search module GUI client configured to;

display a search window on the presentation area, the search window comprising at least one field for;

presenting search parameters including gaze search parameters and non gaze search parameters for a user, the gaze search parameters;

representing a predefined set of specific mental conditions experienced by the user when the user viewed presented information during a certain period of time prior to displaying the search window; and

being based on at least one of a duration of blinks and a pupil size of the user measured during the certain period of time when the user viewed the presented information; and

displaying the gaze search parameters as selectable posts in the search window, the selectable posts comprising labels corresponding to the predefined set of specific mental conditions; and

provide signals corresponding to specific query searches to a search engine for execution of the specific query searches on the computer device and/or on information hosts connectable to the computer device via networks, by using a selected combination of at least one of the gaze search parameters and at least one of the non gaze search parameters.

17. The search module according to claim 16, wherein the gaze search parameters are created based on information that has been presented on the information presentation area for the user and gaze data signals connected with the presented information provided from a gaze tracking module associated with the computer device.

18. The search module according to claim 16, wherein the gaze search parameters are created based on information that has been presented for the user on information presentation areas, each being associated with a different computer device, and gaze data signals connected with the presented information provided from gaze tracking modules associated with the different computer devices.

19. The search module according to claim 16, wherein the gaze search parameters are created based on information that has been presented on multiple information presentation areas, each being associated with a different computer device, for multiple users and gaze data signals connected with the presented information provided from multiple gaze tracking modules associated with the different computer devices.

20. The search module according to claim 16, wherein the search module GUI client is further configured to display a search result presentation window on the information presentation area presenting a search result from the query search.

21. The search module according to claim 16, wherein the gaze search parameters reflect a reading pattern of the user when reading the presented information, the gaze search parameters allowing the user to define a specific query search by selecting whether to include or exclude a specific gaze search parameter in the specific query search.

22. The search module according to claim 16, wherein the gaze search parameters reflect a period of time when the user viewed an image, the gaze search parameters allowing the user to define a specific query search by selecting whether to include or exclude a specific gaze search parameter in the specific query search.

23. A system for searching information using a computer device associated with an information presentation area and a gaze tracking module, the system comprising:

a search module GUI client configured to;

display a search window on the information presentation area, the search window comprising at least one area for;

presenting search parameters including gaze search parameters and non gaze search parameters for a user, the gaze search parameters;

representing a predefined set of specific mental conditions experienced by the user when the user viewed presented information during a certain period of time prior to displaying the search window; and

being based on at least one of a duration of blinks and a pupil size of the user measured during the certain period of time when the user viewed the presented information; and

displaying the search gaze parameters as selectable posts in the search window, the selectable posts comprising labels corresponding to the predefined set of specific mental conditions; and

provide signals corresponding to a specific query search to a search engine for execution of the specific query search on the computer device and/or on information hosts connectable to the computer device via networks, by using a selected combination of at least one of the gaze search parameters and at least one of the non gaze search parameters.

24. The system according to claim 23, further comprising a gaze filter module configured to determine the gaze search parameters based on information that has been presented and gaze data signals, the gaze search parameters comprising links between the gaze search parameters and presented information that satisfies gaze filter criteria for the respective gaze search parameter.

25. The system according to claim 24 wherein the gaze filter module is configured to create the gaze search parameters based on information that has been presented on the information presentation area for the user and gaze data signals connected with the presented information provided from the gaze tracking module associated with the computer device.

26. The system for searching information using a computer device according to claim 24, wherein the gaze filter module is configured to create, maintain and update a gaze search index based on gaze data signals and information of data that has been presented on the information presentation area using the gaze filter criteria.

27. The system according to claim 24, wherein the gaze filter module is further configured to analyze at least one gaze search index to determine a behavioral pattern, the behavioral pattern usable to organize and/or prioritize search results.

28. The system according to claim 23, wherein the search module GUI client is further configured to display a search result presentation window on the information presentation area presenting a search result from the specific query search.

29. The system for searching information using a computer device according to claim 23, wherein the gaze search parameters comprise:

presented information that the user has looked at for at least a certain period of time; and

/or

presented information that the user has looked at during a period of time while being distracted; and

/or

presented information that has caused strong emotional reaction of the user; and

/or

presented information that the user has focused on; and

/or

articles that the user has read at least a predetermined amount of the included text; and

/or

articles or texts that have been thoroughly read by the user; and

/or

articles or texts that have been skimmed by the user; and

/or

articles or texts the user read while being tired.

30. The system according to claim 23, wherein the search window on the information presentation area comprises at least one field for presenting specific gaze search parameters reflecting a reading pattern of the user when reading the presented information, the specific search parameters allowing the user to define a different specific query search by selecting whether to include or exclude the specific gaze search parameters in the different specific query search.

31. The system according to claim 23, wherein the search window on the information presentation area comprises at least one field for presenting specific gaze search parameters reflecting a period of time when the user viewed an image, the specific gaze search parameters allowing the user to define a different specific query search by selecting whether to include or exclude the specific gaze search parameters in the different specific query search.