Devices, methods, and graphical user interfaces for system-wide behavior for 3D models

• US 10,475,253 B2
• Filed: 09/27/2018
• Issued: 11/12/2019
• Est. Priority Date: 01/24/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Yerkes, Giancarlo, Lemay, Stephen O., Faulkner, Jeffrey M., Paul, Grant R., Forssell, Lisa K., Griffin, Bradley W.
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US16/145,045
• Publication Number:
  US 20190228582A1
• Time in Patent Office: 411 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1643 the display being associated to a digitizer, e.g. laptops that can be used as penpads details related to the relative motion of the display enclosure with respect to the body enclosure, e.g. to move between laptop and tablet PC configuration G06F1/1615
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 : Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 : Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A computer system having a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors receives a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras. In response to receiving the request, in accordance with a determination that calibration criteria are not met, a calibration user interface object is displayed. In response to detecting a change in attitude of the one or more cameras in the physical environment, at least one display parameter of the calibration user interface object is adjusted in accordance with the detected change. In response to detecting that the calibration criteria are met, the calibration user interface object ceases to be displayed.

Claims

1. A method, comprising:

at a device having a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors for detecting changes in attitude of the device including the one or more cameras;

receiving a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes;

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detecting that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, ceasing to display the calibration user interface object.

2. The method of claim 1, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

3. The method of claim 2, including:

displaying the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

4. The method of claim 2, including:

displaying the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

5. The method of claim 1, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

6. The method of claim 1, including:

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

7. The method of claim 1, including:

displaying a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

8. The method of claim 1, including:

in response to detecting that the calibration criteria are met, displaying a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

9. The method of claim 1, including:

in response to receiving the request to display the augmented reality view of the physical environment;

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, displaying an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

10. The method of claim 1, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

11. The method of claim 1, including:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgoing moving the calibration user interface object based on the second type of movement.

12. The method of claim 1, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

13. The method of claim 1, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

14. The method of claim 1, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.

15. A computer system, comprising:

a display generation component;

one or more input devices;

one or more cameras;

one or more attitude sensors;

one or more processors; and

memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes;

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detecting that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, ceasing to display the calibration user interface object.

16. The computer system of claim 15, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

17. The computer system of claim 16, wherein the one or more programs include instructions for:

displaying the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

18. The computer system of claim 16, wherein the one or more programs include instructions for:

displaying the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

19. The computer system of claim 15, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

20. The computer system of claim 15, wherein the one or more programs include instructions for:

in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

21. The computer system of claim 15, wherein the one or more programs include instructions for:

displaying a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

22. The computer system of claim 15, wherein the one or more programs include instructions for:

in response to detecting that the calibration criteria are met, displaying a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

23. The computer system of claim 15, wherein the one or more programs include instructions for:

in response to receiving the request to display the augmented reality view of the physical environment;

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, displaying an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

24. The computer system of claim 15, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

25. The computer system of claim 15, wherein the one or more programs include instructions for:

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgoing moving the calibration user interface object based on the second type of movement.

26. The computer system of claim 15, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

27. The computer system of claim 15, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

28. The computer system of claim 15, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.

29. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computer system with a display generation component, one or more input devices, one or more cameras, and one or more attitude sensors, cause the computer system to:

receive a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of a field of view of the one or more cameras;

in response to receiving the request to display the augmented reality view of the physical environment, display the representation of the field of view of the one or more cameras and, in accordance with a determination that calibration criteria are not met for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes;

while displaying the calibration user interface object, detecting, via the one or more attitude sensors, a first change in attitude of the one or more cameras in the physical environment, wherein the first change in attitude includes lateral movement of the one or more cameras in the physical environment; and

in response to detecting the first change in attitude of the one or more cameras in the physical environment that includes the lateral movement of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment, including rotating the calibration user interface object about an axis that is perpendicular to the lateral movement of the one or more cameras in the physical environment;

while displaying the calibration user interface object that moves on the display in accordance with the detected first change in attitude of the one or more cameras in the physical environment, detect that the calibration criteria are met; and

in response to detecting that the calibration criteria are met, cease to display the calibration user interface object.

30. The non-transitory computer readable storage medium of claim 29, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display a representation of a virtual three-dimensional object in the augmented reality view of the physical environment.

31. The non-transitory computer readable storage medium of claim 30, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display the representation of the virtual three-dimensional object in the first user interface region that includes the representation of the field of view of the one or more cameras after ceasing to display the calibration user interface object.

32. The non-transitory computer readable storage medium of claim 30, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display the representation of the virtual three-dimensional object in the first user interface region concurrently with the calibration user interface object, wherein the representation of the virtual three-dimensional object remains at a fixed location in the first user interface region during the movement of the one or more cameras in the physical environment.

33. The non-transitory computer readable storage medium of claim 29, wherein the request to display the augmented reality view of the physical environment in the first user interface region that includes the representation of the field of view of the one or more cameras includes a request to display the representation of the field of view of the one or more cameras without requesting display of a representation of any virtual three-dimensional object in the physical environment captured in the field of view of the one or more cameras.

34. The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to receiving the request to display the augmented reality view of the physical environment, display the representation of the field of view of the one or more cameras and, in accordance with a determination that the calibration criteria are met for the augmented reality view of the physical environment, forgoing display of the calibration user interface object.

35. The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

display a textual object in the first user interface region concurrently with the calibration user interface object that provides information about actions that can be taken by the user to improve calibration of the augmented reality view.

36. The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to detecting that the calibration criteria are met, display a visual indication of a plane detected in the physical environment captured in the field of view of the one or more cameras.

37. The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

in response to receiving the request to display the augmented reality view of the physical environment;

in accordance with the determination that the calibration criteria are not met and before displaying the calibration user interface object, display an animated prompt object that includes a representation of the device moving relative to a representation of a plane.

38. The non-transitory computer readable storage medium of claim 29, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object by a first amount in accordance with a first magnitude of movement of the one or more cameras in the physical environment; and

moving the calibration user interface object by a second amount in accordance with a second magnitude of movement of the one or more cameras in the physical environment, wherein the first amount is distinct from the second amount, and the first magnitude of movement is distinct from the second magnitude of movement.

39. The non-transitory computer readable storage medium of claim 29, wherein the one or more programs include instructions, which, when executed by the computer system, cause the computer system to:

while displaying the calibration user interface object, detect, via the one or more attitude sensors, a second change in attitude of the one or more cameras in the physical environment, wherein the second change in attitude corresponds to a second type of movement that does not include the lateral movement of the one or more cameras in the physical environment; and

in response to detecting the second change in attitude of the one or more cameras in the physical environment that does not include the lateral movement of the one or more cameras in the physical environment, forgo moving the calibration user interface object based on the second type of movement.

40. The non-transitory computer readable storage medium of claim 29, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment without altering a characteristic display location of the calibration user interface object over the first user interface region.

41. The non-transitory computer readable storage medium of claim 29, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object at a speed that is determined in accordance with a rate of change detected in the field of view of the one or more cameras.

42. The non-transitory computer readable storage medium of claim 29, wherein adjusting at least one display parameter of the calibration user interface object in accordance with the detected first change in attitude of the one or more cameras in the physical environment includes:

moving the calibration user interface object in a direction that is determined in accordance with a direction of change detected in the field of view of the one or more cameras.