Generating a three-dimensional model using a portable electronic device recording

• US 9,842,429 B2
• Filed: 01/06/2014
• Issued: 12/12/2017
• Est. Priority Date: 01/28/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tsai, Richard, Just, Andrew, Harris, Brandon
• Primary Examiner(s): Chang, Jungwon
• Application Number:
  US14/147,706
• Publication Number:
  US 20140118346A1
• Time in Patent Office: 1,436 Days
• Field of Search: 348148, 348135, 348113, 345419, 345420, 382103, 382107
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Systems and methods are provided for navigating a three-dimensional model using deterministic movement of an electronic device. An electronic device can load and provide an initial display of a three dimensional model (e.g., of an environment or of an object). As the user moves the electronic device, motion sensing components can detect the device movement and adjust the displayed portion of the three-dimensional model to reflect the movement of the device. By walking with the device in the user'"'"'s real environment, a user can virtually navigate a representation of a three-dimensional environment. In some embodiments, a user can record an object or environment using an electronic device, and tag the recorded images or video with movement information describing the movement of the device during the recording. The recorded information can then be processed with the movement information to generate a three-dimensional model of the recorded environment or object.

Claims

1. A non-transitory machine readable medium on which are stored instructions, comprising instructions that when executed by a programmable device cause the programmable device to:

determine a first position and orientation of a programmable device relative to a three-dimensional object;

detect, at a first time, a movement of the programmable device relative to the first position and orientation;

associate the movement with a second image captured by the programmable device at the first time at a second position and orientation relative to the three-dimensional object; and

generate a three-dimensional model of a portion of the three-dimensional object from at least a first image captured by the programmable device at the first position and orientation, the second image captured by the programmable device at the first time, and the movement associated with the second image,

wherein generating a three-dimensional model comprises determining a three-dimensional representation of the first position and orientation and the second position and orientation onto which the first image and the second image can be applied.

2. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to determine an absolute position and orientation of the programmable device using positioning circuitry and a magnetometer.

3. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to determine a position and orientation of the programmable device relative to the three-dimensional object using a sensor.

4. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to measure radio waves reflected from the three-dimensional object.

5. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to detect motion of the programmable device using a gyroscope or an accelerometer.

6. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to detect a movement based on an output of a positioning circuitry of the programmable device.

7. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to detect a change in orientation of the programmable device using a magnetometer.

8. The machine readable medium of claim 1, wherein the instructions that when executed cause a programmable device to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to define movement information defining one or move vectors indicating movement of the programmable device relative to an initial or previous position and orientation.

9. A method of modeling a three-dimensional object, comprising:

determining a first position and orientation of a programmable device relative to the three-dimensional object at a first time;

detecting movement of the programmable device relative to the first position and orientation at a second time;

capturing a first image of the three-dimensional object at the first time;

capturing a second image of the three-dimensional object at the first time at a second position and orientation relative to the three-dimensional object;

associating the movement with the second image; and

generating a three-dimensional model of a portion of the three-dimensional object using the first image and the second image, wherein the three-dimensional model is based in part on the associated movement, and wherein generating a three-dimensional model comprises determining a three-dimensional representation of the first position and orientation and the second position and orientation onto which the first image and the second image can be applied.

10. The method of claim 9, wherein determining a first position and orientation comprises:

determining an absolute position and orientation of the programmable device.

11. The method of claim 9, further comprising:

associating the first position and orientation with the first image; and

associating the second position and orientation with the second image.

12. The method of claim 11, wherein associating the movement with the second image comprises embedding the second position and orientation in the second image.

13. A system comprising:

one or more processors; and

a memory coupled to the one or more processors and comprising instructions to cause the system to;

determine a first position and orientation of a programmable device relative to a three-dimensional object;

detect, at a first time, a movement of the programmable device relative to the first position and orientation;

associate the movement with a second image captured by the programmable device at the first time at a second position and orientation relative to the three-dimensional object; and

generate a three-dimensional model of a portion of the three-dimensional object from at least a first image captured by the programmable device at the first position and orientation, the second image captured by the programmable device at the first time, and the movement associated with the second image, and wherein generating a three-dimensional model comprises determining a three-dimensional representation of the first position and orientation and the second position and orientation onto which the first image and the second image can be applied.

14. The system of claim 13, wherein the instructions that when executed cause the system to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to determine an absolute position and orientation of the programmable device using positioning circuitry and a magnetometer.

15. The system of claim 13, wherein the instructions that when executed cause the system to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to determine a position and orientation of the programmable device relative to the three-dimensional object using a sensor.

16. The system of claim 13, wherein the instructions that when executed cause the system to determine a first position and orientation of a programmable device relative to a three-dimensional object comprise instructions that when executed cause the programmable device to measure radio waves reflected from the three-dimensional object.

17. The system of claim 13, wherein the instructions that when executed cause the system to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to detect motion of the programmable device using a gyroscope or an accelerometer.

18. The system of claim 13, wherein the instructions that when executed cause the system to detect a movement of the programmable device relative to the first position and orientation at a first time comprise instructions that when executed cause the programmable device to detect a movement based on an output of a positioning circuitry of the programmable device.