Method and system for recalibrating sensing devices without familiar targets

• US 9,681,118 B2
• Filed: 10/01/2015
• Issued: 06/13/2017
• Est. Priority Date: 10/01/2015
• Status: Active Grant

• Current Assignee: Alibaba DAMO Technology Company Limited
• Original Assignee: Infinity Augmented Reality Israel, Ltd. (Alibaba Group Holding Ltd.)
• Inventors: Protter, Matan, Kushnir, Motti
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Zohrabyan, Narek
• Application Number:
  US14/872,158
• Publication Number:
  US 20170099477A1
• Time in Patent Office: 621 Days
• Field of Search: 382154, 345178, 345419
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

Methods and systems for recalibrating sensing devices without using familiar targets are provided herein. The method may include: capturing a scene using a sensing device; determining whether or not the device is calibrated; in a case that said sensing device is calibrated, adding at least one new landmark to the known landmarks; in a case that said sensing device is not calibrated, determining at least some of the captured objects as objects stored on a database as known landmarks at the scene whose position is known; and calibrating the sensing device based on the known landmarks. The system may have various architectures that include a sensing device which captures images of the scene and further derive 3D data on the scene of some form.

Claims

1. A method comprising:

capturing a scene using a sensing device comprising one or more sensors;

determining whether or not the device is calibrated;

in a case that said sensing device is calibrated, adding at least one new landmark to the known landmarks;

in a case that said sensing device is not calibrated, determining at least some objects in the captured scene as objects stored on a database as known landmarks at the scene whose position is known; and

calibrating the sensing device based on the known landmarks.

2. The method according to claim 1, wherein in a case that said sensing device is not calibrated, checking whether the captured scene contains a minimal number of known landmarks sufficient for performing calibration;

and in a case that said scene contains at least the minimal number of known landmarks, calibrating the sensing device based on the known landmarks.

3. The method of claim 1, wherein the check of calibration is performed using the known landmarks.

4. The method according to claim 1, wherein the at least one new landmark is selected based on predefined criteria including at least one of:

textured area;

uniqueness of appearance;

low-complexity structure;

size and distribution of points in the captured scene; and

distance from the sensing device.

5. The method according to claim 1, further comprising maintaining a landmarks database configured to store known and newly added landmarks, wherein the database is searchable.

6. The method according to claim 5, wherein the database is configured to further store low resolution versions of the landmarks.

7. The method according to claim 5, wherein searching the database for a known landmark takes into account various points of views.

8. The method according to claim 5, wherein the new landmarks are stored together with metadata used to perform a contextual search of said landmarks.

9. The method according to claim 5, wherein the at least one landmark is stored in a way such that landmarks from the same environment are clustered.

10. The method according to claim 1, wherein the calibration is solved by minimizing an objective function measuring the projection errors.

11. The method according to claim 10, wherein the minimizing is using gradient descent.

12. The method according to claim 10, wherein the minimizing is using the Levenberg-Marquardt optimization technique.

13. A method comprising:

capturing a scene using a sensing device comprising one or more sensors;

determining whether the captured scene contains a minimal number of known landmarks sufficient for determining whether the sensing device is calibrated;

in a case that the captured scene contains said minimal number of landmarks, checking whether said sensing device is calibrated,

in a case that said sensing device is calibrated, adding at least one new landmark to the known landmarks;

in a case that said sensing device is not calibrated, calibrating the sensing device based on the known landmarks; and

in a case that the captured scene does not contain said minimal number of landmarks, checking calibration of the sensing device without using known landmarks, and in a case that the sensing device is calibrated, adding at least one new landmark to the known landmarks.

14. The method according to claim 13, wherein in a case that said sensing device is not calibrated, checking whether the captured scene contains a minimal number of known landmarks sufficient for performing calibration;

and in a case that said scene contains at least the minimal number of known landmarks, calibrating the sensing device based on the known landmarks.

15. The method of claim 13, wherein the check of calibration is performed using the known landmarks.

16. The method according to claim 13, wherein the at least one new landmark is selected based on predefined criteria including at least one of:

textured area;

uniqueness of appearance;

low-complexity structure;

size and distribution of points in the captured scene; and

distance from the sensing device.

17. The method according to claim 13, further comprising maintaining a landmarks database configured to store known and newly added landmarks, wherein the database is searchable.

18. The method according to claim 17, wherein the database is configured to further store low resolution versions of the landmarks.

19. The method according to claim 17, wherein searching the database for a known landmark takes into account various points of views.

20. The method according to claim 17, wherein the new landmarks are stored together with metadata used to perform a contextual search of said landmarks.

21. The method according to claim 17, wherein the at least one new landmark is stored in a way such that landmarks from the same environment are clustered.

22. The method according to claim 13, wherein the calibration is solved by minimizing an objective function measuring the projection errors.

23. The method according to claim 22, wherein the minimizing is using gradient descent.

24. The method according to claim 22, wherein the minimizing is using the Levenberg-Marquardt optimization technique.

25. A system comprising:

a sensing device comprising one or more sensors configured to capture an image scene;

a database configured to store landmarks; and

a computer processor configured to determine whether the device is calibrated or not, based on the captured image,

wherein in a case that said sensing device is calibrated, the computer processor is configured to add at least one new landmark to the known landmarks, and

wherein in a case that said sensing device is not calibrated, the computer processor is configured to calibrate the sensing device based on the known landmarks.

26. The system according to claim 25, wherein in a case that said sensing device is not calibrated, the computer processor is configured to check whether the captured image contains a minimal number of known landmarks sufficient for performing calibration;

and in a case that said scene contains at least the minimal number of known landmarks, the computer processor is configured to calibrate the sensing device based on the known landmarks.

27. The system according to claim 25, wherein the at least one new landmark is selected based on predefined criteria including at least one of:

textured area;

uniqueness of appearance;

low-complexity structure;

size and distribution of points in the captured scene; and

distance from the sensing device.

28. The system according to claim 25, further comprising maintaining a landmarks database configured to store known and newly added landmarks, wherein the database is searchable.

29. The system according to claim 28, wherein the database is configured to further store low resolution versions of the landmarks.

30. The system according to claim 28, wherein searching the database for a known landmark takes into account various points of views.

31. The system according to claim 28, wherein the new landmarks are stored together with metadata used to perform a contextual search of said landmarks.

32. The system according to claim 28, wherein the at least one new landmark is stored in a way such that landmarks from the same environment are clustered.

33. A system comprising:

a sensing device comprising one or more sensors configured to capture a scene;

a database configured to store landmarks; and

a computer processor configured to determine whether the captured scene contains a minimal number of known landmarks sufficient for determining whether the sensing device is calibrated,

wherein in a case that the captured scene contains said minimal number of landmarks, the computer processor is configured to check whether said sensing device is calibrated,

wherein in a case that said sensing device is calibrated, the computer processor is configured to add at least one new landmark to the landmarks database,

wherein in a case that said sensing device is not calibrated, the computer processor is configured to calibrate the sensing device based on the known landmarks, and

wherein in a case that the captured scene does not contain said minimal number of landmarks, the computer processor is configured to check calibration of the sensing device without using known landmarks, and in a case that the sensing device is calibrated, the computer processor is configured to add at least one new landmark to the landmarks database.