Synchronized, interactive augmented reality displays for multifunction devices

• US 10,176,637 B2
• Filed: 03/25/2016
• Issued: 01/08/2019
• Est. Priority Date: 01/05/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bilbrey, Brett C., King, Nicholas V., Pance, Aleksandar
• Primary Examiner(s): Hannett, James
• Application Number:
  US15/081,145
• Publication Number:
  US 20160203648A1
• Time in Patent Office: 1,019 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 23/632 : for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/222 : Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A device can receive live video of a real-world, physical environment on a touch sensitive surface. One or more objects can be identified in the live video. An information layer can be generated related to the objects. In some implementations, the information layer can include annotations made by a user through the touch sensitive surface. The information layer and live video can be combined in a display of the device. Data can be received from one or more onboard sensors indicating that the device is in motion. The sensor data can be used to synchronize the live video and the information layer as the perspective of video camera view changes due to the motion. The live video and information layer can be shared with other devices over a communication link.

Claims

1. An augmented reality device, comprising:

one or more sensors configured to sense motion of the augmented reality device;

a display;

a camera configured to capture a video;

a processor coupled to the one or more sensors; and

a memory storing instructions that, when executed by the processor, cause the processor to;

identify an object based on pattern recognition of the object as depicted in the video;

generate computer-generated imagery depicting the identified object based on a model of the identified object depicted in the video;

generate an overlay including information corresponding to the identified object and the computer-generated imagery depicting the identified object;

combine the overlay and the video into a composite video for presentation on the display;

display the composite video on the display;

receive sensor data from the one or more sensors indicating that the augmented reality device is in motion; and

synchronize the video and the overlay in response to receiving the sensor data to update the composite video on the display.

2. The augmented reality device of claim 1, wherein the model includes:

a three-dimensional model of the identified object depicted in the video captured by the camera.

3. The augmented reality device of claim 1, comprising:

a communication interface, coupled to the processor, the communication interface configured to share the composite video over a communication link.

4. The augmented reality device of claim 1, wherein the information includes:

one or more annotations related to the object in the video captured by the camera.

5. The augmented reality device of claim 4, wherein the one or more annotations includes a user annotation received at an input device of the augmented reality device.

6. The augmented reality device of claim 5, wherein the input device includes a touch screen.

7. The augmented reality device of claim 1, wherein the one or more sensors include at least one of:

a gyroscope, a magnetometer, an accelerometer, a motion sensor, a light sensor, a proximity sensor, or a positioning system.

8. A non-transitory machine-readable storage medium comprising instructions that, in response to being executed on a computing device, cause the computing device to:

capture a video with a camera of the computing device;

identify an object based on pattern recognition of the object as depicted in the video;

generate computer-generated imagery depicting the identified object based on a model of the identified object depicted in the video;

generate an overlay including information corresponding to the identified object and the computer-generated imagery corresponding to the identified object depicting the identified object;

combine the overlay and the video into a composite video for presentation on a display of the computing device;

display the composite video on the display;

receive sensor data from one or more sensors of the computing device indicating that the computing device is in motion; and

synchronize the video and the overlay using the sensor data to update the composite video for presentation on the display.

9. The non-transitory machine-readable storage medium of claim 8, wherein the model includes:

a three-dimensional model of the object depicted in the video captured by the camera.

10. The non-transitory machine-readable storage medium of claim 8, comprising instructions that cause the computing device to:

share the composite video over a communication link established by a communication interface.

11. The non-transitory machine-readable storage medium of claim 8, wherein the information includes:

one or more annotations related to the object in the video captured by the camera.

12. The non-transitory machine-readable storage medium of claim 11, wherein the one or more annotations includes a user annotation received at an input device of the computing device.

13. The non-transitory machine-readable storage medium of claim 12, wherein the input device includes a touch screen.

14. The non-transitory machine-readable storage medium of claim 8, wherein the one or more sensors include at least one of:

a gyroscope, a magnetometer, an accelerometer, a motion sensor, a light sensor, a proximity sensor, or a positioning system.

15. A method comprising:

capturing a video with a camera;

identifying an object based on pattern recognition of the object as depicted in the video;

generating computer-generated imagery depicting the identified object based on a model of the identified object depicted in the video;

generating an overlay including information corresponding to the identified object and the computer-generated imagery depicting the identified object;

combining the overlay and the video into a composite video for presentation on a display;

displaying the composite video on the display;

receiving sensor data from one or more onboard sensors indicating that the camera is in motion; and

synchronizing the video and the overlay using the sensor data to update the composite video.

16. The method of claim 15, wherein the model comprises:

a three-dimensional model of the object depicted in the video captured by the camera.

17. The method of claim 15, comprising:

sharing the composite video over a communication link established by a communication interface.

18. The method of claim 15, wherein the information includes:

one or more annotations related to the object in the video captured by the camera.

19. The method of claim 18, wherein the one or more annotations includes a user annotation received at an input device.

20. The method of claim 19, wherein the input device includes a touch screen.

21. The method of claim 15, wherein the one or more onboard sensors include at least one of:

a gyroscope, a magnetometer, an accelerometer, a motion sensor, a light sensor, a proximity sensor, or a positioning system.