Physical object addition and removal based on affordance and view

• US 10,429,926 B2
• Filed: 03/15/2017
• Issued: 10/01/2019
• Est. Priority Date: 03/15/2017
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Bastian, David C., Baughman, Aaron K., McCrory, Nicholas A., Whitman, Todd R.
• Primary Examiner(s): Johnson, Motilewa Good
• Application Number:
  US15/459,431
• Publication Number:
  US 20180267603A1
• Time in Patent Office: 930 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/177 Augmented reality
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/23 Head-up displays [HUD] optical aspects of head-up displays G02B27/01
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06F 30/13 Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

Physical object addition and removal based on affordance and view. A method tracks eye gaze of a group of users who view an environment, the group of users each having a respective field of view to the environment. The method assesses, based on the tracking, whether an object present in the respective fields of view of the group of users is clutter. The method also initiates a physical change to the environment based on the assessing. In some examples, assessing whether the object is clutter includes determining an extent to which the group of users avoid the object by looking around or away from the object, measuring visual dwell time on and around the object by the group of users, and/or identifying affordances of the object based on input from the group of users and/or on correlating the object to another object having known affordances associated therewith.

Claims

1. A computer-implemented method comprising:

tracking, using a first one or more eye gaze tracking devices, eye gaze of each user of a first group of users who view an environment through respective transparent displays, the first group of users each having a respective field of view to the environment;

assessing, based on the tracking, whether an object present in the respective fields of view of the first group of users is clutter, the object being a physical object, in the environment, within the respective fields of view of the first group of users, or the object being an augmented reality model interposed in the respective fields of view of the first group of users on the respective transparent displays; and

initiating a physical change to the environment based on the assessing, the physical change being one selected from the group consisting of;

(i) based on the object being the physical object, deconstruction of that physical object in the environment, and (ii) based on the object being the augmented reality model, construction of a physical object based that augmented reality model.

2. The method of claim 1, wherein the assessing comprises determining, based on the tracking, an extent to which the first group of users avoid the object by looking around or away from the object.

3. The method of claim 1, wherein the assessing comprises measuring, based on the tracking, visual dwell time on and around the object by the first group of users.

4. The method of claim 1, wherein the assessing comprises identifying affordances of the object based on at least one selected from the group consisting of:

(i) input from the first group of users, and (ii) correlating the object to another object having known affordances associated therewith.

5. The method of claim 1, wherein the object present in the respective fields of view of the first group of users is the physical object for which initiation of deconstruction of that physical object is performed, and wherein the method further comprises:

placing an augmented reality model of the physical object in fields of view of a second group of users who view the environment, the second group of users being a same group or different group from the first group of users;

scheduling removal of the physical object from the environment; and

repeating the tracking and the assessing for the augmented reality model of the physical object, to assess based on interaction of the second group of users with the augmented reality model of the physical object whether the physical object is properly removed from the environment.

6. The method of claim 5, wherein the placing the augmented reality model of the physical object in the fields of view of the second group of users comprises interposing the augmented reality model of the physical object on transparent displays in lines of sight of the second group of users.

7. The method of claim 6, wherein the placing the augmented reality model of the physical object in the fields of view of the second group of users further comprises varying transparency of the augmented reality model of the physical object on the transparent displays over time at increasing transparency levels, wherein repeating the assessing comprises assessing interaction of the second group of users with the augmented reality model of the physical object at the increasing transparency levels to determine whether the physical object is properly removed from the environment.

8. The method of claim 5, wherein the initiating the physical change to the environment comprises confirming that the physical object is to be deconstructed.

9. The method of claim 1, wherein the object present in the respective fields of view of the first group of users comprises the augmented reality model, the augmented reality model being an augmented reality model of the physical object, the physical object being proposed to be constructed in the environment, and wherein the method further comprises:

determining that the physical object has been proposed for construction in the environment; and

interposing the augmented reality model of the physical object on the respective transparent displays in lines of sight of the first group of users to present for the first group of users an augmented reality version of the physical object positioned at a location, in the environment, at which the physical object is proposed to be constructed, wherein the assessing assesses whether the augmented reality model of the physical object is clutter as an indicator of whether the physical object, if actually constructed as proposed at the location in the environment, will be clutter in the environment.

10. The method of claim 9, wherein the initiating the physical change to the environment is based on the assessing indicating that the physical object will not be clutter in the environment and comprises approving a proposal to construct the physical object in the environment.

11. The method of claim 9, wherein the physical object comprises an alteration to an existing physical object in the environment.

12. The method of claim 9, further comprising, based on construction of the physical object in the environment, repeating the tracking eye gaze for a second group of users who view the environment including the constructed physical object and the assessing, for the constructed physical object, to assess based on interaction of the second group of users with the constructed physical object whether the constructed physical object is clutter.

13. The method of claim 1, further comprising obtaining, from each user of at least one user of the first group of users, a respective utterance about the environment within the field of view of that user as the tracking tracks the eye gaze of that user, wherein the assessing is further based on the obtaining.

14. A computer system comprising:

a memory; and

a processor in communication with the memory, wherein the computer system is configured to perform a method comprising;

tracking, using a first one or more eye gaze tracking devices, eye gaze of each user of a first group of users who view an environment through respective transparent displays, the first group of users each having a respective field of view to the environment;

assessing, based on the tracking, whether an object present in the respective fields of view of the first group of users is clutter, the object being a physical object, in the environment, within the respective fields of view of the first group of users, or the object being an augmented reality model interposed in the respective fields of view of the first group of users on the respective transparent displays; and

initiating a physical change to the environment based on the assessing, the physical change being one selected from the group consisting of;

(i) based on the object being the physical object, deconstruction of that physical object in the environment, and (ii) based on the object being the augmented reality model, construction of a physical object based on that augmented reality model.

15. The computer system of claim 14, wherein the assessing comprises:

determining, based on the tracking, an extent to which the first group of users avoid the object by looking around or away from the object;

measuring, based on the tracking, visual dwell time on and around the object by the first group of users; and

identifying affordances of the object based on at least one selected from the group consisting of;

(i) input from the first group of users, and (ii) correlating the object to another object having known affordances associated therewith.

16. The computer system of claim 14, wherein the object present in the respective fields of view of the first group of users is the physical object for which initiation of deconstruction of that physical object is performed, and wherein the method further comprises:

placing an augmented reality model of the physical object in fields of view of a second group of users who view the environment, the second group of users being a same group or different group from the first group of users, wherein the placing comprises;

interposing the augmented reality model of the physical object on transparent displays in lines of sight of the second group of users; and

varying transparency of the augmented reality model of the physical object on the transparent displays over time at increasing transparency levels;

scheduling removal of the physical object from the environment; and

repeating the tracking and the assessing for the augmented reality model of the physical object, to assess based on interaction of the second group of users with the augmented reality model of the physical object whether the physical object is properly removed from the environment, wherein repeating the assessing comprises assessing interaction of the second group of users with the augmented reality model of the physical object at the increasing transparency levels to determine whether the physical object is properly removed from the environment.

17. The computer system of claim 14, wherein the object present in the respective fields of view of the first group of users comprises the augmented reality model, the augmented reality model being an augmented reality model of the physical object, the physical object being proposed to be constructed in the environment, and wherein the method further comprises:

determining that the physical object has been proposed for construction in the environment; and

interposing the augmented reality model of the physical object on the respective transparent displays in lines of sight of the first group of users to present for the first group of users an augmented reality version of the physical object positioned at a location, in the environment, at which the physical object is proposed to be constructed, wherein the assessing assesses whether the augmented reality model of the physical object is clutter as an indicator of whether the physical object, if actually constructed as proposed at the location in the environment, will be clutter in the environment.

18. A computer program product comprising:

a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising;

tracking, using a first one or more eye gaze tracking devices, eye gaze of each user of a first group of users who view an environment through respective transparent displays, the first group of users each having a respective field of view to the environment;

assessing, based on the tracking, whether an object present in the respective fields of view of the first group of users is clutter, the object being a physical object, in the environment, within the respective fields of view of the first group of users, or the object being an augmented reality model interposed in the respective fields of view of the first group of users on the respective transparent displays; and

initiating a physical change to the environment based on the assessing, the physical change being one selected from the group consisting of;

(i) based on the object being the physical object, deconstruction of that physical object in the environment, and (ii) based on the object being the augmented reality model, construction of a physical object based on that augmented reality model.

19. The computer program product of claim 18, wherein the object present in the respective fields of view of the first group of users is the physical object for which initiation of deconstruction of that physical object is performed, and wherein the method further comprises:

placing an augmented reality model of the physical object in fields of view of a second group of users who view the environment, the second group of users being a same group or different group from the first group of users, wherein the placing comprises;

interposing the augmented reality model of the physical object on transparent displays in lines of sight of the second group of users; and

varying transparency of the augmented reality model of the physical object on the transparent displays over time at increasing transparency levels;

scheduling removal of the physical object from the environment; and

repeating the tracking and the assessing for the augmented reality model of the physical object, to assess based on interaction of the second group of users with the augmented reality model of the physical object whether the physical object is properly removed from the environment, wherein repeating the assessing comprises assessing interaction of the second group of users with the augmented reality model of the physical object at the increasing transparency levels to determine whether the physical object is properly removed from the environment.

20. The computer program product of claim 18, wherein the object present in the respective fields of view of the first group of users comprises the augmented reality model, the augmented reality model being an augmented reality model of the physical object, the physical object being proposed to be constructed in the environment, and wherein the method further comprises:

determining that the physical object has been proposed for construction in the environment; and

interposing the augmented reality model of the physical object on the respective transparent displays in lines of sight of the first group of users to present for the first group of users an augmented reality version of the physical object positioned at a location, in the environment, at which the physical object is proposed to be constructed, wherein the assessing assesses whether the augmented reality model of the physical object is clutter as an indicator of whether the physical object, if actually constructed as proposed at the location in the environment, will be clutter in the environment.