GENERATING A THREE-DIMENSIONAL MODEL USING A PORTABLE ELECTRONIC DEVICE RECORDING

• US 20100188503A1
• Filed: 01/28/2009
• Published: 07/29/2010
• Est. Priority Date: 01/28/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tsai, Richard, Harris, Brandon, Just, Andrew
• Granted Patent: US 8,294,766 B2
• Time in Patent Office: Days
• US Class Current: 348/142
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Systems and methods are providing for navigating a three-dimensional model using deterministic movement of an electronic device. An electronic device can load and provide an initial display of a three dimensional model (e.g., of an environment or of an object). As the user moves the electronic device, motion sensing components, positioning circuitry, and other components can detect the device movement and adjust the displayed portion of the three-dimensional model to reflect the movement of the device. By walking with the device in the user'"'"'s real environment, a user can virtually navigate a representation of a three-dimensional environment. In some embodiments, a user can record an object or environment using an electronic device, and tag the recorded images or video with movement information describing the movement of the device during the recording. The recorded information can then be processed with the movement information to generate a three-dimensional model of the recorded environment or object.

Claims

1. A method for generating a three-dimensional model, comprising:

determining an initial position and orientation of an electronic device;

recording images using a lens of the electronic device;

detecting the movement of the electronic device during the recording; and

associating, for particular moments in time, the images recorded and movement detected at the particular moments in time.

2. The method of claim 1, further comprising:

processing the recorded images and associated detected movement; and

generating a three-dimensional model in response to processing.

3. The method of claim 1, wherein detecting further comprises:

receiving an output of a movement detection component.

4. The method of claim 3, wherein associating further comprises:

identifying a particular output of the movement detection component;

identifying an image recorded at the time the particular output was received; and

associating the identified output with the identified image.

5. The method of claim 4, further comprising:

processing the particular output to determine at least one of the change in orientation and position of the electronic device relative to the initial position and orientation of the electronic device.

6. The method of claim 2, wherein processing further comprises converting the output from a coordinate system moving with the device to a coordinate system independent of the device location.

7. The method of claim 6, wherein the coordinate system independent of the device location is static relative to an environment recorded by the lens of the electronic device.

8. An electronic device operative to capture information for a three-dimensional model, comprising:

a lens operative to capture a recording;

a motion-sensing component operative to provide an output reflecting the movement of the electronic device; and

a processor operative to;

direct the lens to capture a recording;

receive a plurality of motion-sensing component outputs during the recording as the user moves the electronic device; and

associate each of the received motion sensing component outputs with the portion of the recording captured at the same time as the outputs were received.

9. The electronic device of claim 8, wherein the processor is further operative to:

process the plurality of motion sensing component outputs; and

define at least one mathematical quantity describing the electronic device movement relative to at least one previous electronic device position and orientation.

10. The electronic device of claim 9, wherein the processor is further operative to:

determine an initial position and orientation of the electronic device; and

define at least one mathematical quantity describing the electronic device movement relative to the initial position and orientation.

11. The electronic device of claim 8, wherein the processor is further operative to:

define a three-dimensional representation of the electronic device movement based on the plurality of motion-sensing component outputs;

associate the portions of the recordings with the appropriate portions of the three-dimensional representation; and

provide the three-dimensional representation and associated portions of the recording as a three-dimensional model.

12. The electronic device of claim 8, wherein the processor is further operative to provide the recording and the associated each of the motions sensing component outputs to a host device for generating a three-dimensional model.

13. A host device operative to generate a three-dimensional model of an object, comprising a processor operative to:

receive a recording of an object, and movement information associated with the recording from an electronic device, wherein a plurality of the images of the recording are associated with movement information describing the movement of the electronic device during the recording;

process the associated movement information to quantify the movement of the electronic device;

define a three-dimensional grid representing the object in a static coordinate system relative to the object in response to processing;

apply the images of the recording to the defined three-dimensional grid; and

create a three-dimensional model of the object in response to applying.

14. The host device of claim 13, wherein the processor is further operative to:

receive a second recording of the object, and movement information associated with the second recording;

generate a second three-dimensional model of the object based on the second recording; and

combine the second three-dimensional model and the created three-dimensional model to form a combined three-dimensional model.

15. The host device of claim 13, wherein the processor is further operative to:

receive a second recording of the object, and movement information associated with the second recording;

identify portions of the second recording that do not overlap with the recording; and

apply the identified portions of the second recording to the defined three-dimensional grid.

16. The host device of claim 13, wherein the processor is further operative to:

identify at least one absolute position of the electronic device relative to the object based on the received recording and associated movement information; and

define the three-dimensional grid relative to the identified at least one absolute position.

17. A method for generating a three-dimensional model, comprising:

capturing a plurality of images as an electronic device moves;

receiving an output from a motion sensing component characterizing the movement of the electronic device;

simultaneously associating the received output and the captured images as they are received and captured, respectively;

processing the received output to define a three-dimensional grid depicting the electronic device movement in a static coordinate system relative to the electronic device; and

applying the plurality of images to the grid in a manner reflecting the associating.

18. The method of claim 17, wherein associating further comprises associating the motion sensing component output with captured images as metadata.

19. The method of claim 17, wherein associating further comprises linking the motion sensing component output to captured images from a distinct file.

20. The method of claim 17, wherein the grid can be navigated to display the plurality of sequential images in an order other than the sequential order of the images.

21. The method of claim 17, wherein capturing further comprises recording a video.

22. The method of claim 21, wherein the associating further comprises associating captured index frames of the recorded video with the received output.

23. Computer readable media for generating a three-dimensional model, comprising computer readable code recorded thereon for:

determining an initial position and orientation of an electronic device;

recording images using a lens of the electronic device;

detecting the movement of the electronic device during the recording; and

associating, for particular moments in time, the images recorded and movement detected at the particular moments in time.