Portable augmented reality device and method

• US 8,547,401 B2
• Filed: 08/19/2004
• Issued: 10/01/2013
• Est. Priority Date: 08/19/2004
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Mallinson, Dominic S., Marks, Richard L.
• Primary Examiner(s): CHOW, JEFFREY J
• Application Number:
  US10/922,514
• Publication Number:
  US 20060038833A1
• Time in Patent Office: 3,330 Days
• Field of Search: 345/632, 345/633
• US Class Current: 345/633
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 1/00323 : with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings

Abstract

A portable device configured to provide an augmented reality experience is provided. The portable device has a display screen configured to display a real world scene. The device includes an image capture device associated with the display screen. The image capture device is configured to capture image data representing the real world scene. The device includes image recognition logic configured to analyze the image data representing the real world scene. Image generation logic responsive to the image recognition logic is included. The image generation logic is configured to incorporate an additional image into the real world scene. A computer readable medium and a system providing an augmented reality environment are also provided.

Claims

1. A portable gaming device configured to provide an augmented reality experience on a display screen of the portable device, the portable device being a handheld device, comprising:

the display screen configured to display a real world scene, the display screen disposed on a front side of the portable gaming device;

an image capture device associated with the display screen, the image capture device configured to capture real video image data representing the real world scene, the image capture device disposed on a back side of the portable gaming device opposite the front side;

a first input region defined on the front side for holding by a first hand of a user, the first input region disposed on a first adjacent side of the display screen, the first input region having one or more input buttons;

a second input region defined on the front side for holding by a second hand of the user, the second input region disposed on a second adjacent side of the display screen opposite the first adjacent side, the second input region having one or more input buttons;

image recognition logic configured to analyze the real video image data representing the real world scene to identify a marker in the real video image data representing the real world scene, the augmenting visual media being related to a gaming environment and response to the first input region and the second input region,

media generation logic responsive to the image recognition logic, the media generation logic configured to display at least part of the real world scene from the real video image data along with augmenting visual media on the display screen, the augmenting visual media being a moving animation superimposed in and around the real world scene, wherein the image recognition logic is configured to communicate identification of the marker in the real video image data to the media generation logic, and

in response to the identification of the marker in the image data, triggering integration of the augmented visual media into the real world scene as viewed through the display screen of the portable gaming device, wherein the portable gaming device provides a view of the moving animation through the display screen;

wherein the augmenting visual media and the at least part of the real world scene is presented on the display screen of the portable gaming device and wherein at least part of the processing by the image recognition logic and the media generation logic occur within the portable gaming device;

wherein the portable gaming device is configured to be handheld during operation, such that a location and view direction of the user'"'"'s eye viewing the display screen is independent of a location and view direction of the image capture device, to facilitate viewing of the augmented visual media integrated with the real world scene from a perspective defined by the location and view direction of the image capture device, independent of the location and view direction of the user'"'"'s eye.

2. The portable gaming device of claim 1, wherein the image capture device is removable.

3. The portable gaming device of claim 1, wherein the image capture device is affixed to one of a surface above the display screen or a surface behind the display screen.

4. The portable gaming device of claim 1, further comprising:

a memory card slot for inserting a memory card; and

wireless networking access capability.

5. The portable gaming device of claim 4, further comprising:

a memory card inserted into the memory card slot, the memory card configured to be shared with other portable devices.

6. The portable gaming device of claim 1,

wherein the display screen faces in a first direction;

and

wherein the image capture device is directed in a second direction opposite the first direction.

7. A method for augmenting real video display data presented to a viewer on a display screen of a portable gaming device having an image capture device, the portable gaming device being a handheld device, comprising method operations of:

capturing the real video display data with the image capture device, the image capture device disposed on a back side of the portable gaming device;

analyzing the captured display data through image recognition logic incorporated into the portable gaming device;

identifying a marker within the captured display data through the analyzing, obtaining audio/visual media in response to identifying the marker within the captured display data, and the identified marker dictating which audio/visual media to obtain from a database; and

upon identifying the marker, presenting on the display screen and through sound output of the portable gaming device, for a period of time, augmenting audio/visual media along with at least part of the captured real video display data, the display screen disposed on a front side of the portable gaming device opposite the back side;

receiving input from one or more input buttons defined on a first input region disposed on a first adjacent side of the display screen, the first input region defined on the front side of the portable gaming device for holding by a first hand of a user;

receiving input from one or more input buttons defined on a second input region disposed on a second adjacent side of the display screen opposite the first adjacent side, the second input region defined on the front side of the portable gaming device for holding by a first hand of a user;

wherein the audio/visual media is identified by the marker;

wherein a visual part of the augmenting audio/visual media is defined by moving animation superimposed in and around real world objects found in the captured display data, the augmenting visual media being related to a gaming environment and response to the received input;

wherein the portable gaming device provides a view of the moving animation through the display screen of the portable gaming device;

wherein the portable gaming device is configured to be handheld during operation, such that a location and view direction of a user'"'"'s eye viewing the display screen is independent of a location and view direction of the image capture device, to facilitate viewing of the augmented visual media integrated with the real world scene from a perspective defined by the location and view direction of the image capture device, independent of the location and view direction of the user'"'"'s eye.

8. The method of claim 7, wherein the portable gaming device is capable of executing a video game.

9. The method of claim 7, wherein the marker is hand drawn.

10. The method of claim 7, wherein the image capture device is a video capture device.

11. The method of claim 7, wherein the method operation of analyzing the captured display data includes,

comparing the captured display data with data stored in a memory of the image capture device, the memory being detachably linked to the image capture device.

12. The method of claim 7,

wherein the display screen faces in a first direction;

and

wherein the image capture device is directed in a second direction opposite the first direction.

13. A handheld device configured to provide an augmented reality experience on a display screen of the handheld device, comprising:

the display screen configured to display a real world scene, the display screen disposed on the front side of the handheld device;

and image capture device associated with the display screen, the image capture device configured to capture real video image data representing the real world scene, the image capture device disposed on a back side of the handheld device opposite the front side;

an input region defined on the handheld device;

image recognition logic configured to analyze the real video image data representing the real world scene to identify a marker in the real video image data representing the real world scene;

media generation logic responsive to the image recognition logic, the media generation logic configured to display at least part of the real world scene from the real video image data along with augmenting visual media on the display screen, the augmenting visual media being a moving animation superimposed in and around the real world scene, the augmenting visual media being related to a gaming environment and responsive to the input region;

wherein the image recognition logic is configured to communicate identification of the marker in the real video image data to the media generation logic, and, in response to the identification of the marker in the image data, trigger integration of the augmenting visual media into the real world scene as viewed through the display screen of the handheld device.