Image data for enhanced user interactions

• US 10,444,963 B2
• Filed: 07/13/2018
• Issued: 10/15/2019
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bereza, Marek, Berenson, Adi, Bernstein, Jeffrey Traer, Girling, Lukas Robert Tom, Hauenstein, Mark, Hoffnung, Amir, Lindmeier, William D., Malia, Joseph A., Missig, Julian
• Primary Examiner(s): Sun, Li P
• Application Number:
  US16/035,419
• Publication Number:
  US 20180321826A1
• Time in Patent Office: 459 Days
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

The present disclosure generally relates to using avatars and image data for enhanced user interactions. In some examples, user status dependent avatars are generated and displayed with a message associated with the user status. In some examples, a device captures image information to scan an object to create a 3D model of the object. The device determines an algorithm for the 3D model based on the capture image information and provides visual feedback on additional image data that is needed for the algorithm to build the 3D model. In some examples, an application'"'"'s operation on a device is restricted based on whether an authorized user is identified as using the device based on captured image data. In some examples, depth data is used to combine two sets of image data.

Claims

1. An electronic device, comprising:

a display;

a camera;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying, on the display, a user interface that includes a representation of content;

while displaying the user interface that includes the representation of content, receiving first user input on the electronic device corresponding to a request for performing a predefined action on the content; and

in response to receiving the first user input;

in accordance with a determination, based on captured image data captured by the camera, that a set of authorization criteria is met, wherein the authorization criteria include a criterion that is met when the captured image data indicates that the face of an authorized user was present in front of the camera and was looking at the display at the time that the user input was received, performing the predefined action; and

in accordance with a determination, based on the captured image data, that the set of authorization criteria is not met because the captured image data indicates that the face of an authorized user was present in front of the camera and was not looking at the display, forgoing performance of the predefined action and displaying a confirmation interface providing an affordance that, when activated, performs the predefined action.

2. The electronic device of claim 1, the one or more programs further including instructions for:

capturing the captured image data with the camera at a time that corresponds to a time at which the first user input was received.

3. The electronic device of claim 1, wherein the predefined action is a destructive action.

4. The electronic device of claim 1, wherein the request for performing the predefined action is a request to erase or delete the content.

5. The electronic device of claim 1, wherein the predefined action includes navigating to a particular view of the user interface associated with the content.

6. The electronic device of claim 1, the one or more programs further including instructions for:

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, switching the electronic device into a locked state.

7. The electronic device of claim 1, the one or more programs further including instructions for:

in response to receiving the first user input;

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, prompting the user to authenticate by looking at the camera.

8. The electronic device of claim 1, wherein the user interface includes a list of graphical elements representing a plurality of data objects, wherein the representation of the content is one of the graphical elements and the content is an associated data object of the plurality of data objects.

9. The electronic device of claim 1, the one or more programs further including instructions for:

receiving second user input on the representation of the content that exceeds an intensity threshold; and

subsequent to receiving the second user input, restricting access to the content, including requiring that captured image data indicates that the face of an authorized user is present in front of the camera and is looking at the display in order for the content to be accessed in response to the second first user input.

10. The electronic device of claim 9, the one or more programs further including instructions for:

in response to receiving the second user input, displaying an affordance, wherein, in response to selection of the affordance, the electronic device stores an indication on the electronic device that accessing the content requires that a set of access criteria be met.

11. The electronic device of claim 10, the one or more programs further including instructions for:

in response to receiving the first user input, determining whether the indication is present for the content.

12. The electronic device of claim 1, wherein the set of authorization criteria further includes a criterion that a face in the captured image data corresponds to one or more authorized faces previously registered with the device.

13. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display and a camera, the one or more programs including instructions for:

displaying, on the display, a user interface that includes a representation of content;

while displaying the user interface that includes the representation of content, receiving first user input on the electronic device corresponding to a request for performing a predefined action on the content; and

in response to receiving the first user input;

in accordance with a determination, based on captured image data captured by the camera, that a set of authorization criteria is met, wherein the authorization criteria include a criterion that is met when the captured image data indicates that the face of an authorized user was present in front of the camera and was looking at the display at the time that the first user input was received, performing the predefined action; and

in accordance with a determination, based on the captured image data, that the set of authorization criteria is not met because the captured image data indicates that the face of an authorized user was present in front of the camera and was not looking at the display, forgoing performance of the predefined action and displaying a confirmation interface providing an affordance that, when activated, performs the predefined action.

14. The non-transitory computer-readable storage medium of claim 13, the one or more programs further including instructions for:

capturing the captured image data with the camera at a time that corresponds to a time at which the first user input was received.

15. The non-transitory computer-readable storage medium of claim 13, wherein the predefined action is a destructive action.

16. The non-transitory computer-readable storage medium of claim 13, wherein the request for performing the predefined action is a request to erase or delete the content.

17. The non-transitory computer-readable storage medium of claim 13, wherein the predefined action includes navigating to a particular view of the user interface associated with the content.

18. The non-transitory computer-readable storage medium of claim 13, the one or more programs further including instructions for:

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, switching the electronic device into a locked state.

19. The non-transitory computer-readable storage medium of claim 13, the one or more programs further including instructions for:

in response to receiving the first user input;

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, prompting the user to authenticate by looking at the camera.

20. The non-transitory computer-readable storage medium of claim 13, wherein the user interface includes a list of graphical elements representing a plurality of data objects, wherein the representation of the content is one of the graphical elements and the content is an associated data object of the plurality of data objects.

21. The non-transitory computer-readable storage medium of claim 13, the one or more programs further including instructions for:

receiving second user input on the representation of the content that exceeds an intensity threshold; and

subsequent to receiving the second user input, restricting access to the content, including requiring that captured image data indicates that the face of an authorized user is present in front of the camera and is looking at the display in order for the content to be accessed in response to the first user input.

22. The non-transitory computer-readable storage medium of claim 21, the one or more programs further including instructions for:

in response to receiving the second user input, displaying an affordance, wherein, in response to selection of the affordance, the electronic device stores an indication on the electronic device that accessing the content requires that a set of access criteria be met.

23. The non-transitory computer-readable storage medium of claim 22, the one or more programs further including instructions for:

in response to receiving the first user input, determining whether the indication is present for the content.

24. The non-transitory computer-readable storage medium of claim 13, wherein the set of authorization criteria further includes a criterion that a face in the captured image data corresponds to one or more authorized faces previously registered with the device.

25. The method of claim 14, further comprising:

capturing the captured image data with the camera at a time that corresponds to a time at which the first user input was received.

26. A method comprising:

at an electronic device having a display and a camera;

displaying, on the display, a user interface that includes a representation of content;

while displaying the user interface that includes the representation of content, receiving first user input on the electronic device corresponding to a request for performing a predefined action on the content; and

in response to receiving the first user input;

in accordance with a determination, based on captured image data captured by the camera, that a set of authorization criteria is met, wherein the authorization criteria include a criterion that is met when the captured image data indicates that the face of an authorized user was present in front of the camera and was looking at the display at the time that the first user input was received, performing the predefined action; and

in accordance with a determination, based on the captured image data, that the set of authorization criteria is not met because the captured image data indicates that the face of an authorized user was present in front of the camera and was not looking at the display, forgoing performance of the predefined action and displaying a confirmation interface providing an affordance that, when activated, performs the predefined action.

27. The method of claim 26, wherein the predefined action is a destructive action.

28. The method of claim 26, wherein the request for performing the predefined action is a request to erase or delete the content.

29. The method of claim 26, wherein the predefined action includes navigating to a particular view of the user interface associated with the content.

30. The method of claim 26, further comprising:

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, switching the electronic device into a locked state.

31. The method of claim 26, further comprising:

in response to receiving the first user input;

in accordance with the determination, based on the captured image data, that the set of authorization criteria is not met, prompting the user to authenticate by looking at the camera.

32. The method of claim 26, wherein the user interface includes a list of graphical elements representing a plurality of data objects, wherein the representation of the content is one of the graphical elements and the content is an associated data object of the plurality of data objects.

33. The method of claim 26, further comprising:

receiving second user input on the representation of the content that exceeds an intensity threshold; and

subsequent to receiving the second user input, restricting access to the content, including requiring that captured image data indicates that the face of an authorized user is present in front of the camera and is looking at the display in order for the content to be accessed in response to the first user input.

34. The method of claim 33, further comprising:

in response to receiving the second user input, displaying an affordance, wherein, in response to selection of the affordance, the electronic device stores an indication on the electronic device that accessing the content requires that a set of access criteria be met.

35. The method of claim 34, further comprising:

in response to receiving the first user input, determining whether the indication is present for the content.

36. The method of claim 26, wherein the set of authorization criteria further includes a criterion that a face in the captured image data corresponds to one or more authorized faces previously registered with the device.