METHOD AND SYSTEM FOR PROVIDING INFORMATION ASSOCIATED WITH A VIEW OF A REAL ENVIRONMENT SUPERIMPOSED WITH A VIRTUAL OBJECT

• US 20160307374A1
• Filed: 12/19/2013
• Published: 10/20/2016
• Est. Priority Date: 12/19/2013
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Kurz, Daniel, Wang, Lejing
• Application Number:
  US15/100,355
• Publication Number:
  US 20160307374A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

Providing information associated with a view of a real environment superimposed with a virtual object, includes obtaining a model of a real object located in a real environment and a model of a virtual object, determining a spatial relationship between the real object and the virtual object, determining a pose of a device displaying the view relative to the real object, determining a visibility of the virtual object in the view according to the pose of the device, the spatial relationship and the models of the real object and the virtual object, and if the virtual object is not visible in the view, determining a movement for at least one of the device and at least part of the real object such that the virtual object is visible in the view in response to the at least one movement, and providing information indicative of the at least one movement.

Claims

1. A method of providing information associated with a view of a real environment superimposed with a virtual object, the method comprising

obtaining a model of a real object located in a real environment and a model of a virtual object superimposed in a view of the real environment;

determining a spatial relationship between the real object and the virtual object;

determining a pose of a device displaying the view of the real environment relative to the real object;

determining a visibility of the virtual object in the view according to the pose of the device, the spatial relationship between the real object and the virtual object, and the models of the real object and the virtual object;

and

if the virtual object is not visible in the view, determining at least one movement for moving at least one of the device and at least part of the real object such that the virtual object is visible in the view in response to the at least one movement, and

providing, by the device, information indicative of the at least one movement.

3. The method according to claim 1, wherein the visibility of the virtual object is defined according to a fraction of the virtual object which is visible in the view.

4. The method according to claim 1, wherein the visibility of the virtual object is defined according to a visibility of at least a part of the virtual object.

6. The method according to claim 1, wherein the virtual object or a part of the virtual object is determined to be not visible when at least part of the virtual object is located behind at least a part of the real object with respect to the viewer or when the virtual object or the part of the virtual object is out of a field of a view of the device.

7. The method according to claim 1, wherein the at least one movement is determined according to the pose of the device, the spatial relationship between the real object and the virtual object, the models of the real object and the virtual object, and the visibility of the virtual object.

8. The method according to claim 1, further comprising calculating a cost of the at least one movement, wherein the at least one movement is determined according to the cost.

9. The method according to claim 8, wherein

the cost of the at least one movement is determined depending on at least one selected from a group consisting of a distance of at least one translation, a direction of at least one translation, an angle of at least one rotation, an axis of at least one rotation, a pre-defined range for moving the device, a pre-defined range for moving the at least part of the real object, and the physical energy needed to perform the least one movement.

11. The method according to claim 1, wherein providing information indicative of the at least one movement comprises providing information for giving at least one instruction about at least part of the at least one movement in a manner that is at least one selected from a group consisting of visual, acoustic, and tangible.

12. The method according to claim 1, further comprising:

creating a view of the virtual object by a virtual camera using one selected from a group consisting of rasterization, ray casting and ray tracing,

wherein the visibility of the virtual object is further determined according to the view of the virtual camera onto the virtual object.

13. The method according to claim 1, wherein the model of the real object is selected from a group consisting of a CAD model, a polygon model, a point cloud, a volumetric dataset, and an edge model.

14. The method according to claim 1, wherein the device comprises a projector, and the visibility of the virtual object is further determined according to a position of a projection surface onto which visual content could be projected by the projector, wherein the position of the projection surface is relative to the projector and the projection surface is at least part of the real environment.

15. The method according to claim 14, wherein the position of the projection surface is determined according to a tracking system that tracks the projector and the projection surface.

2. (canceled)

5. (canceled)

10. (canceled)

16. (canceled)

17. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. (canceled)

22. A non-transitory computer readable medium comprising computer readable code which, when executed by one or more processors, causes the one or more processors to:

obtain a model of a real object located in a real environment and a model of a virtual object superimposed in a view of the real environment;

determine a spatial relationship between the real object and the virtual object;

determine a pose of a device displaying the view of the real environment relative to the real object;

determine a visibility of the virtual object in the view according to the pose of the device, the spatial relationship between the real object and the virtual object, and the models of the real object and the virtual object;

if the virtual object is not visible in the view, determine at least one movement for moving at least one of the device and at least part of the real object such that the virtual object is visible in the view in response to the at least one movement, and

provide, by the device, information indicative of the at least one movement.

23. The non-transitory computer readable medium of claim 22, wherein the visibility of the virtual object is defined according to a visibility of at least a part of the virtual object.

24. The non-transitory computer readable medium of claim 22, wherein the at least one movement is determined according to the pose of the device, the spatial relationship between the real object and the virtual object, the models of the real object and the virtual object, and the visibility of the virtual object.

25. The non-transitory computer readable medium of claim 22, wherein the virtual object or a part of the virtual object is determined to be not visible at least part of the virtual object is located behind at least a part of the real object with respect to the viewer or when the virtual object or the part of the virtual object is out of a field of a view of the device.

26. A system for providing information associated with a view of a real environment superimposed with a virtual object, comprising:

a camera;

a display;

one or more processors; and

a memory coupled to the one or more processors and comprising computer code which, when executed by the one or more processors, causes the one or more processors to;

obtain a model of a real object located in a real environment and a model of a virtual object superimposed in a view of the real environment;

determine a spatial relationship between the real object and the virtual object;

determine a pose of the system relative to the real object;

determine a visibility of the virtual object in the display according to the pose, the spatial relationship between the real object and the virtual object, and the models of the real object and the virtual object;

if the virtual object is not visible in the display, determine at least one movement for moving at least one of the system and at least part of the real object such that the virtual object is visible in the display in response to the at least one movement, and

provide information indicative of the at least one movement.

27. The system of claim 26, wherein the visibility of the virtual object is defined according to a visibility of at least a part of the virtual object.

28. The system of claim 26, wherein the at least one movement is determined according to the pose of the system, the spatial relationship between the real object and the virtual object, the models of the real object and the virtual object, and the visibility of the virtual object.

29. The system of claim 26, wherein the virtual object or a part of the virtual object is determined to be not visible at least part of the virtual object is located behind at least a part of the real object with respect to the viewer or when the virtual object or the part of the virtual object is out of a field of view of the display.