Method for processing local information

• US 9,940,716 B2
• Filed: 05/16/2015
• Issued: 04/10/2018
• Est. Priority Date: 05/21/2014
• Status: Active Grant

• Current Assignee: Airbus Group SAS (Airbus SE)
• Original Assignee: Airbus Group SAS (Airbus SE)
• Inventors: Chevassus, Nicolas, Marraud, Denis, Tarault, Antoine, Perrotton, Xavier
• Primary Examiner(s): Wu, Sing-Wai
• Application Number:
  US14/714,273
• Publication Number:
  US 20150339819A1
• Time in Patent Office: 1,060 Days
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules

Abstract

A method for processing local information acquired by a virtual representation and a device having an inertial unit and an image sensor. At least one image of a real environment of the device is captured. The localization of the device in the virtual representation, corresponding to the localization of the device in the real environment, is obtained by correlating the portions of the captured image and portions of the virtual representation, The inertial unit determines the displacement of the device. The localization of the device in the virtual representation is modified as a function of the displacement so that the real position of the device corresponds, during the displacement, to the localization of the device in the virtual representation.

Claims

1. A method for processing local information acquired by a virtual representation and a device comprising an inertial unit and an image sensor, comprising the steps of:

capturing at least one image of a real environment of the device;

localizing the device by correlating portions of said at least one captured image and portions of the virtual representation to obtain localization of the device in the virtual representation, corresponding to the localization of the device in the real environment, by automatic recognition, in said at least one captured image, of visual, multimodal and non-ambiguous seamarks that are predefined in the virtual representation, wherein the localization initializes the device that is uninitialized;

determining a displacement of the device at least by the inertial unit;

modifying the localization of the device in the virtual representation as a function of the displacement determined by the inertial unit so that a real position of the device corresponds, during the displacement, to the localization of the device in the virtual representation;

jointly displaying said at least one captured image and a portion of the virtual representation corresponding to the displayed captured image; and

modifying the virtual representation by editing localized information related to the real environment, such as an indication of nonconformity, a measurement of pressure, temperature or lighting level, on the virtual representation.

2. The method for processing local information according to claim 1, wherein the step of determining the displacement comprises the steps of:

estimating a movement by the inertial unit;

estimating a relative movement between images captured at one instant and images captured at a later instant; and

combining the movement estimation and the relative movement estimation.

3. The method for processing local information according to claim 1, further comprising the step of decreasing the displacement determined by the inertial unit.

4. The method for processing local information according to claim 1, wherein the displaying step is a superposition in which the virtual representation is transparent.

5. The method for processing local information according to claim 1, further comprising a step of recording the localized information in the virtual representation in order to take account of a change in a local situation, such as a nonconformity.

6. A device for processing local information, comprising:

an image sensor configured to provide at least one image of a real environment of the device;

a microprocessor to access a virtual representation;

a localization processor to initialize the device that is uninitialized and to localize the device in the virtual representation, corresponding to the localization of the device in the real environment, by correlating portions of said at least one captured image and portions of the virtual representation, the processor performs correlation by automatic recognition, in said at least one captured image, of visual, multimodal and non-ambiguous seamarks that are predefined in the virtual representation;

an inertial unit to determine a displacement of the device;

a modifying processor to modify the localization of the device in the virtual representation as a function of the displacement determined by the inertial unit so that a real position of the device corresponds, during the displacement, to the localization of the device in the virtual representation;

a display device configured to jointly display a captured image and a portion of the virtual representation corresponding to the displayed captured image;

a display screen to modify the virtual representation by editing localized information related to the real environment, such as an indication of nonconformity, a measurement of pressure, temperature or lighting level, on the virtual representation.

7. A communicating portable terminal comprising the device for processing local information according to claim 6.

8. The device for processing local information according to claim 6, wherein the display device is configured to record the localized information in the virtual representation in order to take account of a change in a local situation, such as a nonconformity.