Image processing method and apparatus for interactive augmented reality

• US 10,133,534 B2
• Filed: 09/26/2017
• Issued: 11/20/2018
• Est. Priority Date: 11/25/2015
• Status: Active Grant

• Current Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Inventors: Wang, Xiaoting
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/715,661
• Publication Number:
  US 20180018141A1
• Time in Patent Office: 420 Days
• Field of Search: 345633
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/1462 with means for detecting differences between the image stored in the host and the images displayed on the remote displays
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 21/41 Structure of client; Structure of client peripherals
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43 Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware real-time communication protocols in data switching networks H04L65/00
• CPC Class Codes: H04N 21/4363 Adapting the video stream to a specific local network, e.g. a Bluetooth® network
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104

Abstract

The present disclosure relates to methods and systems for generating augmented reality. The method may include capturing a first image in a viewing range of a first augmented reality device in real time via a camera of the first augmented reality device, and synchronizing the first image to a second augmented reality device, the first augmented reality device being a device for providing an augmented reality image to a first user in the first display, and the second augmented reality device being a device for providing an augmented reality image to a second user in a second display. The method further includes receiving a second image from the second augmented reality device, the second image being an image returned in real time by the second augmented reality device after augmenting the first image by the second augmented reality device. The method further includes generating the augmented reality image to the first display at the first augmented reality device based on the first image and the second image, and displaying the augmented reality image in the first display at the first augmented reality device.

Claims

1. A method for generating augmented reality, comprising:

capturing a first image in a viewing range of a first augmented reality device in real time via a camera of the first augmented reality device, and synchronizing the first image to a second augmented reality device, the first augmented reality device comprising a first memory having stored therein a first set of instructions and a first processor for executing the first set of instructions to provide an augmented reality image to a first user via a first display, and the second augmented reality device comprising a second memory having stored therein a second set of instructions and a second processor for executing the second set of instructions to provide an augmented reality image to a second user via a second display;

receiving a second image from the second augmented reality device, the second image being an image returned in real time by the second augmented reality device after augmenting the first image by the second augmented reality device;

determining whether the second image and the first image are similar;

generating the augmented reality image to the first display at the first augmented reality device based on the first image alone when determining that the second image and the first image are similar;

when determining that the second image and the first image are not similar;

identifying a first difference image of the second image relative to the first image;

converting the first difference image to a second difference image based on a viewing angle of the first augmented reality device; and

superimposing the second difference image onto the first image; and

displaying the augmented reality image in the first user in the first display of the first augmented reality device.

2. The method according to claim 1, wherein capturing the first image in the viewing range of the first augmented reality device in real time via the camera of the first augmented reality device comprises:

detecting a first retinal viewing angle of the first user; and

capturing the first image in real time via the camera according to the first retinal viewing angle.

3. The method according to claim 1, wherein the viewing range of the first augmented reality device corresponds to a viewing range of the first user.

4. The method according to claim 1, after synchronizing the first image to the second augmented reality device, and before receiving the second image from the second augmented reality device, further comprising:

recording, by the second augmented reality device, a virtual image and an operation action by second user based on viewing the virtual image, the virtual image being an image provided by the second augmented reality device to the second user, and the virtual image comprising at least the first image; and

superimposing, by the second augmented reality device, an image that represents the operation action onto the virtual image to obtain the second image.

5. The method according to claim 4, wherein recording the virtual image and the operation action comprises:

detecting a second retinal viewing angle of the second user; and

recording the virtual image and the operation action according to the second retinal viewing angle.

6. The method according to claim 5, wherein recording the virtual image and the operation action according to the second retinal viewing angle comprises:

recording the virtual image and the operation action in a viewing range of the second augmented reality device according to the second retinal viewing angle.

7. The method according to claim 6, wherein the viewing range of the second augmented reality device corresponds to a viewing range of the second user.

8. A system comprising a first augmented reality device for providing an augmented reality image to a first user via a first display, the first augmented reality device comprising a camera, a memory having instructions stored therein, and a processor in communication with the camera and the memory, wherein the processor, when executing the instructions, is configured to cause the first augmented reality device to:

capture a first image via the camera in a viewing range of the first augmented reality device in real time, and synchronize the first image to a second augmented reality device, the second augmented reality device comprising a device providing an augmented reality image to a second user via a second display;

receive a second image from the second augmented reality device, the second image being an image returned in real time by the second augmented reality device after augmenting the first image by the second augmented reality device;

determine whether the second image and the first image are similar;

generate the augmented reality image to the first display at the first augmented reality device based on the first image alone when the second image and the first image are similar;

when determining that the second image and the first image are not similar;

identify a first difference image of the second image relative to the first image;

convert the first difference image to a second difference image based on a viewing angle of the first augmented reality device; and

superimpose the second difference image onto the first image; and

display the augmented reality image in the first user in the first display.

9. The system according to claim 8, wherein the processor, when executing the instructions to cause the first augmented reality device to capture the first image via the camera in the viewing range of the first augmented reality device in real time, is configured to cause the first augmented reality device to:

detect a first retinal viewing angle of the first user; and

capture the first image in real time via the camera according to the first retinal viewing angle.

10. The system according to claim 8, wherein the viewing range of the first augmented reality device corresponds to a viewing range of the first user.

11. The system according to claim 8, further comprising the second augmented reality device configured to:

record a virtual image and an operation action by second user based on viewing the virtual image, the virtual image being an image provided by the second augmented reality device to the second user, and the virtual image comprising at least the first image; and

superimpose an image that represents the operation action onto the virtual image to obtain the second image.

12. The system according to claim 11, wherein the second augmented reality device, to record the virtual image and the operation action, is configured to:

detect a second retinal viewing angle of the second user; and

record the virtual image and the operation action according to the second retinal viewing angle.

13. The system according to claim 12, wherein the second augmented reality device, to record the virtual image and the operation action according to the second retinal viewing angle, is configured to:

record the virtual image and the operation action in a viewing range of the second augmented reality device according to the second retinal viewing angle.

14. The system according to claim 13, wherein the viewing range of the second augmented reality device corresponds to a viewing range of the second user.

15. A computer-readable and non-transitory storage medium having storing instructions stored therein, the instructions, when executed by a processor of a first augmented reality device for providing an augmented reality image to a first user via a first display, causes the first augmented reality device to:

capture a first image in a viewing range of the first augmented reality device in real time, and synchronizing the first image to a second augmented reality device, the second augmented reality device being a device providing an augmented reality image to a second user via a second display;

receive a second image from the second augmented reality device, the second image being an image returned in real time by the second augmented reality device after augmenting the first image by the second augmented reality device;

determine whether the second image and the first image are similar;

generate the augmented reality image to the first display at the first augmented reality device based on the first image alone when the second image and the first image are similar;

when determining that the second image and the first image are not similar;

identify a first difference image of the second image relative to the first image;

convert the first difference image to a second difference image based on a viewing angle of the first augmented reality device; and

superimpose the second difference image onto the first image; and

display the augmented reality image in the first display at the first augmented reality device.