Modifying device content to facilitate user interaction

• US 10,254,940 B2
• Filed: 04/19/2017
• Issued: 04/09/2019
• Est. Priority Date: 04/19/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Carey, James E., Chen, Jim C., Santosuosso, John M.
• Primary Examiner(s): Duong, Hien L
• Application Number:
  US15/491,550
• Publication Number:
  US 20180307401A1
• Time in Patent Office: 720 Days
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Systems, methods, and computer program products to perform an operation comprising dynamically selecting a first user hand position profile, where in the first user hand position profile includes a first range of reach of a digit of a first hand of the user while the user is holding the mobile device with the first hand in a first position, performing a first predefined operation to modify a first object, wherein modifying the first object causes the first object to be outputted at a location on a touchscreen display within the range of reach of the digit of the first hand, and outputting the modified first object on the touchscreen display.

Claims

1. A method, comprising:

generating a first user hand position profile for a user by;

outputting a plurality of objects on a touchscreen display, wherein each of the plurality of objects are outputted at a respective location on the touchscreen display;

determining that the user can select a first object of the plurality of objects with a digit of a first hand while holding a mobile device with the first hand;

determining that the user cannot select a second object of the plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

determining whether the user can select a remaining plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

storing, in the first user hand position profile for the user, an indication that a range of reach of the digit includes a location of the first object of the plurality of objects and a respective location of each of the remaining plurality of objects the user is determined to be able to select;

storing, in the first user hand position profile for the user, an indication that the range of reach of the digit does not include a location of the second object of the plurality of objects and the respective locations of each of the remaining plurality of objects the user is determined to not be able to select; and

generating a reach line for the user based on the locations of the plurality of objects, wherein the reach line reflects the range of reach of the digit;

dynamically selecting the first user hand position profile, where in the first user hand position profile includes the range of reach of the digit of the first hand of the user while the user is holding the mobile device with the first hand in a first position;

performing a first predefined operation to modify the second object, wherein modifying the second object causes the second object to be outputted at a location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object on the touchscreen display.

2. The method of claim 1, further comprising:

detecting a first user action;

determining that the first user action causes the user to hold the mobile device with the first hand in a second position;

determining, based on the first user hand position profile, that the location of the modified second object is beyond the range of reach of the digit of the first hand of the user when the mobile device is held with the first hand in the second position;

performing a second predefined operation to modify the second object to cause the second object to be outputted at a new location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object at the new location on the touchscreen display.

3. The method of claim 2, wherein second predefined operation is further performed based on one or more of:

(i) detecting a user-defined spoken command, (ii) detecting, based on image data provided by a camera, a user-defined hand gesture, (iii) identifying, based on the image data provided by the camera, a second hand of the user attempting to interact with the mobile device, (iv) determining, based on movement data provided by an accelerometer, that the user is moving the mobile device in the first hand, and (v) determining, based on the movement data provided by the accelerometer, that the user is moving the mobile device from the first hand to the second hand.

4. The method of claim 1, wherein the first predefined operation comprises one or more of:

(i) enlarging the second object, (ii) enlarging each object outputted on the touchscreen display, (iii) moving the second object from a different location to the location, and (iv) moving each object outputted on the touchscreen display.

5. The method of claim 1, further comprising:

determining a first layout of a first plurality of objects including the unmodified first second object outputted on the touchscreen display;

determining that the first layout matches a second layout corresponding to a second plurality of objects including a third object;

determining, based on the first user hand position profile of the user, that a location of the third object of the second plurality of objects is beyond the range of reach of the digit; and

determining that the location of the third object corresponds to the location on the touchscreen display.

6. The method of claim 5, wherein the first predefined operation is further performed upon determining the first layout matches the second layout and determining that the location of the third object corresponds to the location on the touchscreen display.

7. A computer program product, comprising:

a computer-readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor to perform an operation comprising;

generating a first user hand position profile for a user by;

outputting a plurality of objects on a touchscreen display, wherein each of the plurality of objects are outputted at a respective location on the touchscreen display;

determining that the user can select a first object of the plurality of objects with a digit of a first hand while holding a mobile device with the first hand;

determining that the user cannot select a second object of the plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

determining whether the user can select a remaining plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

storing, in the first user hand position profile for the user, an indication that a range of reach of the digit includes a location of the first object of the plurality of objects and a respective location of each of the remaining plurality of objects the user is determined to be able to select;

storing, in the first user hand position profile for the user, an indication that the range of reach of the digit does not include a location of the second object of the plurality of objects and the respective locations of each of the remaining plurality of objects the user is determined to not be able to select; and

generating a reach line for the user based on the locations of the plurality of objects, wherein the reach line reflects the range of reach of the digit;

dynamically selecting the first user hand position profile, where in the first user hand position profile includes the range of reach of the digit of the first hand of the user while the user is holding the mobile device with the first hand in a first position;

performing a first predefined operation to modify the second object, wherein modifying the second object causes the second object to be outputted at a location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object on the touchscreen display.

8. The computer program product of claim 7, the operation further comprising:

detecting a first user action;

determining that the first user action causes the user to hold the mobile device with the first hand in a second position;

determining, based on the first user hand position profile, that the location of the modified second object is beyond the range of reach of the digit of the first hand of the user when the mobile device is held with the first hand in the second position;

performing a second predefined operation to modify the second object to cause the second object to be outputted at a new location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object at the new location on the touchscreen display.

9. The computer program product of claim 8, wherein second predefined operation is further performed based on one or more of:

(i) detecting a user-defined spoken command, (ii) detecting, based on image data provided by a camera, a user-defined hand gesture, (iii) identifying, based on the image data provided by the camera, a second hand of the user attempting to interact with the mobile device, (iv) determining, based on movement data provided by an accelerometer, that the user is moving the mobile device in the first hand, and (v) determining, based on the movement data provided by the accelerometer, that the user is moving the mobile device from the first hand to the second hand.

10. The computer program product of claim 7, wherein the first predefined operation comprises one or more of:

(i) enlarging the second object, (ii) enlarging each object outputted on the touchscreen display, (iii) moving the second object from a different location to the location, and (iv) moving each object outputted on the touchscreen display.

11. The computer program product of claim 7, the operation further comprising:

determining a first layout of a first plurality of objects including the unmodified first second object outputted on the touchscreen display;

determining that the first layout matches a second layout corresponding to a second plurality of objects including a third object;

determining, based on the first user hand position profile of the user, that a location of the third object of the second plurality of objects is beyond the range of reach of the digit; and

determining that the location of the third object corresponds to the location on the touchscreen display.

12. The computer program product of claim 11, wherein the first predefined operation is further performed upon determining the first layout matches the second layout and determining that the location of the third object corresponds to the location on the touchscreen display.

13. A mobile device, comprising:

a processor;

a touchscreen display, and

a memory storing one or more instructions which, when executed by the processor, performs an operation comprising;

generating a first user hand position profile for a user by;

outputting a plurality of objects on a touchscreen display, wherein each of the plurality of objects are outputted at a respective location on the touchscreen display;

determining that the user can select a first object of the plurality of objects with a digit of a first hand while holding a mobile device with the first hand;

determining that the user cannot select a second object of the plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

determining whether the user can select a remaining plurality of objects with the digit of the first hand while holding the mobile device with the first hand;

storing, in the first user hand position profile for the user, an indication that a range of reach of the digit includes a location of the first object of the plurality of objects and a respective location of each of the remaining plurality of objects the user is determined to be able to select;

storing, in the first user hand position profile for the user, an indication that the range of reach of the digit does not include a location of the second object of the plurality of objects and the respective locations of each of the remaining plurality of objects the user is determined to not be able to select; and

generating a reach line for the user based on the locations of the plurality of objects, wherein the reach line reflects the range of reach of the digit;

dynamically selecting the first user hand position profile, where in the first user hand position profile includes the range of reach of the digit of the first hand of the user while the user is holding the mobile device with the first hand in a first position;

performing a first predefined operation to modify the second object, wherein modifying the second object causes the second object to be outputted at a location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object on the touchscreen display.

14. The mobile device of claim 13, the operation further comprising:

detecting a first user action;

determining that the first user action causes the user to hold the mobile device with the first hand in a second position;

determining, based on the first user hand position profile, that the location of the modified second object is beyond the range of reach of the digit of the first hand of the user when the mobile device is held with the first hand in the second position;

performing a second predefined operation to modify the second object to cause the second object to be outputted at a new location on the touchscreen display within the range of reach of the digit of the first hand; and

outputting the modified second object at the new location on the touchscreen display.

15. The mobile device of claim 14, wherein second predefined operation is further performed based on one or more of:

(i) detecting a user-defined spoken command, (ii) detecting, based on image data provided by a camera, a user-defined hand gesture, (iii) identifying, based on the image data provided by the camera, a second hand of the user attempting to interact with the mobile device, (iv) determining, based on movement data provided by an accelerometer, that the user is moving the mobile device in the first hand, and (v) determining, based on the movement data provided by the accelerometer, that the user is moving the mobile device from the first hand to the second hand.

16. The mobile device of claim 13, wherein the first predefined operation comprises one or more of:

(i) enlarging the second object, (ii) enlarging each object outputted on the touchscreen display, (iii) moving the second object from a different location to the location, and (iv) moving each object outputted on the touchscreen display.

17. The mobile device of claim 13, the operation further comprising:

determining a first layout of a first plurality of objects including the unmodified second object outputted on the touchscreen display;

determining that the first layout matches a second layout corresponding to a second plurality of objects including a third object;

determining, based on the first user hand position profile of the user, that a location of the third object of the second plurality of objects is beyond the range of reach of the digit; and

determining that the location of the third object corresponds to the location on the touchscreen display,

wherein the first predefined operation is further performed upon determining the first layout matches the second layout and determining that the location of the third object corresponds to the location on the touchscreen display.