Method and system for determining a model of at least part of a real object

• US 10,297,083 B2
• Filed: 09/16/2013
• Issued: 05/21/2019
• Est. Priority Date: 09/16/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kurz, Daniel, Wang, Lejing
• Primary Examiner(s): Sams, Michelle L
• Application Number:
  US15/022,184
• Publication Number:
  US 20160232678A1
• Time in Patent Office: 2,073 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models

Abstract

The invention is related to a method of determining a model which describes at least part of a real object, which comprises the steps of: providing a first image of at least part of the real object to a first processing device, wherein the first image is captured by a camera, sending at least part of the first image to a second processing device which is physically separate from the first processing device, determining on the second processing device a registration between the first image and a first model provided on the second processing device according to the at least part of the first image received at the second processing device, determining geometric information for at least one pixel in the first image according to the registration, sending the geometric information from the second processing device to the first processing device, and using at least part of the geometric information received at the first processing device together with at least part of the first image to determine on the first processing device a second model comprising geometric and material information which describes at least part of the real object.

Claims

1. A computer-readable medium comprising computer-readable code which, when executed by a processor, causes the processor to:

obtain, by a first processing device, a first image of at least part of a real object captured by a camera,

send, by the first processing device, at least part of the first image to a second processing device that is physically separate from the first processing device, wherein a first model of the real object is stored on the second processing device, and wherein the first model comprises 3D geometric information associated with the real object;

receive, by the first processing device, the 3D geometric information and a registration between the first image and the first model of the real object from the second processing device,

wherein the registration indicates a spatial relationship between the first image and the real object; and

generate, by the first processing device, a second model of at least part of the real object based on the 3D geometric information received from the second processing device, the registration, and material information obtained from the first image at the first processing device.

2. The computer-readable medium of claim 1, wherein the 3D geometric information is in one of a coordinate system of the first model and a coordinate system of the camera.

3. The computer-readable medium of claim 1, wherein the computer-readable code further comprises computer-readable code to cause the processor to:

obtain, by the first processing device, a second image of at least part of the real object captured by the camera; and

determine, by the first processing device, a pose of the camera while capturing the second image relative to the real object according to at least part of the second image and at least part of the second model.

4. The computer-readable medium of claim 1, wherein the computer-readable code further comprises computer readable code to cause the processor to:

obtain, by the first processing device, a second image of at least part of the real object captured by the camera; and

determine, by the first processing device, an image position of at least part of the real object in the second image according to at least part of the second image and at least part of the second model.

5. The computer-readable medium of claim 4, wherein the computer readable code further comprises computer-readable code to cause the processor to:

use the image position in an Augmented Reality application to overlay a virtual object with the second image.

6. The computer-readable medium of claim 1, wherein the real object is a planar object and wherein the 3D geometric information comprises 3D positions of a plurality of pixels in the first image.

7. The computer-readable medium of claim 1, wherein the real object is a three-dimensional object and wherein the 3D geometric information comprises 3D positions in a coordinate system of the first model corresponding to 2D pixels in the first image.

8. The computer-readable medium of claim 1, wherein the computer-readable code further comprises computer-readable code to cause the processor to:

send at least one reading from at least one sensor associated with the first processing device to the second processing device, wherein the at least one sensor comprises at least one of a GPS sensor, a magnetometer, an accelerometer, and a gyroscope.

9. The computer-readable medium of claim 1, wherein sending at least part of the first image includes sending a processed version of the at least part of the first image to the second processing device, wherein the processed version of the at least part of the first image includes data derived from at least one of compression, encryption, cropping, scaling, blurring, sharpening, color conversion, and conversion into a sparse representation based on feature descriptors of any representation that allows for registration.

10. The computer-readable medium of claim 1, wherein sending at least part of the first image to the second processing device uses at least one of the following techniques:

Bluetooth radio, infrared light, ZigBee, Wi-Fi, orthogonal frequency-division multiplexing (OFDM), WiMAX, cellular network, mobile network, Global System for Mobile Communications, Personal Communications Service, and Digital Advanced Mobile Phone Service.

11. The computer-readable medium of claim 1, wherein the computer-readable code further comprises computer-readable code to cause the processor to:

use, by the first processing device, at least part of the second model in vision-based localization for visual camera localization and tracking relative to the real object.

12. The computer-readable medium of claim 1, wherein the 3D geometric information comprises a distance between at least part of the real object and the camera.

13. The computer-readable medium of claim 1, wherein the first processing device and the second processing device are components of a client-server architecture, wherein the first processing device is, or is comprised in, a client device and the second processing device is, or is comprised, in a server.

14. The computer-readable medium of claim 1, wherein the first processing device is comprised in a mobile device, particularly a mobile phone, a tablet computer, a handheld device, a head-worn device, a smart watch, or a wearable computer with a video-see-through display or an optical-see-through head-mounted display.

15. A system for determining a model which describes at least part of a real object, comprising:

a first processing device; and

a memory, coupled to the first processing device and comprising computer code that, when executed by the first processing device, causes the first processing device to;

obtain a first image of at least part of the real object captured by a camera,

send at least part of the first image to a second processing device that is physically separate from the first processing device, wherein a first model of the real object is stored on the second processing device, and wherein the first model comprises 3D geometric information associated with the real object,

receive, by the first processing device, the 3D geometric information and a registration between the first image and the first model of the real object from the second processing device

wherein the registration indicates a spatial relationship between the first image and the real object; and

determine, by the first processing device, a second model of at least part of the real object comprising the 3D geometric information received from the second processing device, the registration, and material information obtained from the first image at the first processing device.

16. The system of claim 15, wherein the 3D geometric information is in one of a coordinate system of the first model and a coordinate system of the camera.

17. The system of claim 15, wherein the memory further comprises computer code to cause the processor to:

obtain a second image of at least part of the real object captured by the camera; and

determine a pose of the camera while capturing the second image relative to the real object according to at least part of the second image and at least part of the second model.

18. The system of claim 15, wherein the memory further comprises computer code to cause the processor to:

obtain a second image of at least part of the real object captured by the camera; and

determine an image position of at least part of the real object in the second image according to at least part of the second image and at least part of the second model.

19. The system of claim 18, wherein the memory further comprises computer code to:

use the image position in an Augmented Reality application to overlay a virtual object with the second image.

20. A method for determining a model that describes at least part of a real object, comprising:

obtaining, by a first processing device, a first image of at least part of the real object captured by a camera,

sending, by the first processing device, at least part of the first image to a second processing device that is physically separate from the first processing device, wherein a first model of the real object is stored on the second processing device, and wherein the first model comprises 3D geometric information associated with the real object,

receiving, by the first processing device, the 3D geometric information and a registration between the first image and the first model of the real object from the second processing device

wherein the registration indicates a spatial relationship between the first image and the real object; and

obtaining, by the first processing device, a second model of at least part of the real object based on the 3D geometric information received from the second processing device, the registration, and material information obtained from the first image at the first processing device.