Interface object and motion controller for augmented reality

• US 9,990,029 B2
• Filed: 04/11/2016
• Issued: 06/05/2018
• Est. Priority Date: 02/06/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Kochi, Masami
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US15/096,251
• Publication Number:
  US 20160224103A1
• Time in Patent Office: 785 Days
• Field of Search: 345419, 345632, 345633
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method for interfacing with an interactive program is provided. Images of an interface object and a motion controller disposed in a gameplay environment are captured, a tag defined on a surface of the interface object. The captured images are analyzed to identify the tag on the surface of the interface object. Movement of the interface object is tracked by tracking the tag. Augmented images are generated by replacing, in the captured images, the tag with a virtual object, wherein generating the augmented images includes processing the tracked movement of the interface object to define movement of the virtual object. The captured images are analyzed to identify a pointing direction of the motion controller in the gameplay environment. The pointing direction of the motion controller is processed to identify a selection of a portion of the virtual object. The augmented images are presented on a display.

Claims

1. A method for interfacing with an interactive program, comprising:

capturing images of an interface object and a motion controller device disposed in a gameplay environment, the motion controller device being separate from the interface object, the interface object having a tag defined on a surface of the interface object;

analyzing the captured images to identify the tag on the surface of the interface object;

tracking movement of the interface object by tracking the tag on the surface of the interface object;

generating augmented images by replacing, in the captured images, the tag with a virtual object, wherein generating the augmented images includes processing the tracked movement of the interface object to define movement of the virtual object;

analyzing the captured images to identify a pointing direction of the motion controller device in the gameplay environment;

wherein generating the augmented images further includes rendering in the augmented images a selection tool that is maneuverable as controlled by the pointing direction of the motion controller device;

processing the pointing direction of the motion controller device to position the selection tool so as to point towards a selectable region of the virtual object, to further identify a selection of the selectable region of the virtual object; and

presenting the augmented images on a display.

2. The method of claim 1,

wherein the virtual object includes a virtual surface corresponding to the surface of the interface object;

wherein movement of the surface of the interface object controls movement of the virtual surface of the virtual object.

3. The method of claim 2, wherein generating the augmented images is configured to render a scene on the virtual surface, the scene defining an animation, wherein the animation defines movement of at least one three-dimensional object in proximity to the virtual surface on which scene defining the animation is defined.

4. The method of claim 1,

wherein the virtual object is a three-dimensional object;

wherein the movement of the virtual object is a rotational or translational movement of the virtual object.

5. The method of claim 4, wherein the rotational movement of the virtual object enables viewing into virtual spaces defined within the virtual object.

6. The method of claim 1, wherein the selection tool is configured to indicate the pointing direction of the motion controller.

7. The method of claim 6,

wherein processing the pointing direction of the motion controller defines a positioning of the selection tool over the selectable region of the virtual object to identify the selection of the selectable region;

wherein the selection of the selectable region of the virtual object triggers an action related to the selectable region.

8. A system for interfacing with an interactive program, comprising:

an image capture device for capturing images of an interface object and a motion controller device disposed in a gameplay environment, the motion controller device being separate from the interface object, the interface object having a tag defined on a surface of the interface object;

a computing device configured to receive the captured images from the image capture device, the computing device being configured to execute the interactive program to define,

an analyzer module for analyzing the captured images to identify the tag on the surface of the interface object;

a tracking module for tracking movement of the interface object by tracking the tag on the surface of the interface object;

an augmented image generator for generating augmented images by replacing, in the captured images, the tag with a virtual object, wherein generating the augmented images includes processing the tracked movement of the interface object to define movement of the virtual object;

wherein the analyzer module is configured to analyze the captured images to identify a pointing direction of the motion controller device in the gameplay environment;

wherein generating the augmented images further includes rendering in the augmented images a selection tool that is maneuverable as controlled by the pointing direction of the motion controller device;

a selection module for processing the pointing direction of the motion controller device to position the selection tool so as to point towards a selectable region of the virtual object, to further identify a selection of the selectable region of the virtual object; and

a presenter for presenting the augmented images on a display.

9. The system of claim 8,

wherein the virtual object includes a virtual surface corresponding to the surface of the interface object;

wherein movement of the surface of the interface object controls movement of the virtual surface of the virtual object.

10. The system of claim 9, wherein generating the augmented images is configured to render a scene on the virtual surface, the scene defining an animation, wherein the animation defines movement of at least one three-dimensional object in proximity to the virtual surface on which scene defining the animation is defined.

11. The system of claim 8,

wherein the virtual object is a three-dimensional object;

wherein the movement of the virtual object is a rotational or translational movement of the virtual object.

12. The system of claim 11, wherein the rotational movement of the virtual object enables viewing into virtual spaces defined within the virtual object.

13. The system of claim 8, wherein the selection tool is configured to indicate the pointing direction of the motion controller.

14. The system of claim 13,

wherein processing the pointing direction of the motion controller defines a positioning of the selection tool over the selectable region of the virtual object to identify the selection of the selectable region;

wherein the selection of the selectable region of the virtual object triggers an action related to the selectable region.

15. A system for interfacing with an interactive program, comprising:

a interface object having a tag defined on a surface of the interface object;

a motion controller device, the motion controller device being separate from the interface object;

a client device for capturing images of the interface object and the motion controller in a gameplay environment;

at least one server computer having a processor and memory, including,

an analyzer module for analyzing the captured images to identify the tag on the surface of the interface object;

a tracking module for tracking movement of the interface object by tracking the tag on the surface of the interface object;

an augmented image generator for generating augmented images by replacing, in the captured images, the tag with a virtual object, wherein generating the augmented images includes processing the tracked movement of the interface object to define movement of the virtual object;

wherein the analyzer module is configured to analyze the captured images to identify a pointing direction of the motion controller device in the gameplay environment;

wherein generating the augmented images further includes rendering in the augmented images a selection tool that is maneuverable as controlled by the pointing direction of the motion controller device;

a selection module for processing the pointing direction of the motion controller device to position the selection tool so as to point towards a selectable region of the virtual object, to further identify a selection of the selectable region of the virtual object; and

wherein the client device includes a presenter for presenting the augmented images on a display.

16. The system of claim 15,

wherein the virtual object includes a virtual surface corresponding to the surface of the interface object;

wherein movement of the surface of the interface object controls movement of the virtual surface of the virtual object.

17. The system of claim 16, wherein generating the augmented images is configured to render a scene on the virtual surface, the scene defining an animation, wherein the animation defines movement of at least one three-dimensional object in proximity to the virtual surface on which scene defining the animation is defined.

18. The system of claim 15,

wherein the virtual object is a three-dimensional object;

wherein the movement of the virtual object is a rotational or translational movement of the virtual object.

19. The system of claim 18, wherein the rotational movement of the virtual object enables viewing into virtual spaces defined within the virtual object.

20. The system of claim 15,

wherein the selection tool is configured to indicate the pointing direction of the motion controller;

wherein processing the pointing direction of the motion controller defines a positioning of the selection tool over the selectable region of the virtual object to identify the selection of the selectable region;

wherein the selection of the selectable region of the virtual object triggers an action related to the selectable region.