Orienting a user interface of a controller for operating a self-propelled device

• US 9,841,758 B2
• Filed: 02/10/2016
• Issued: 12/12/2017
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: Sphero Inc.
• Original Assignee: Sphero Inc.
• Inventors: Bernstein, Ian H., Wilson, Adam, Smith, Brian Keith
• Primary Examiner(s): CHEUNG, CALVIN K
• Application Number:
  US15/040,331
• Publication Number:
  US 20160202696A1
• Time in Patent Office: 671 Days
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: A63H 33/005 Motorised rolling toys
• CPC Class Codes: B62D 61/00 Motor vehicles or trailers, characterised by the arrangement or number of wheels, not otherwise provided for, e.g. four wheels in diamond pattern
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0259 using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G05D 1/0891 specially adapted for land vehicles
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A self-propelled device determines an orientation for its movement based on a pre-determined reference frame. A controller device is operable by a user to control the self-propelled device. The controller device includes a user interface for controlling at least a direction of movement of the self-propelled device. The self-propelled device is configured to signal the controller device information that indicates the orientation of the self-propelled device. The controller device is configured to orient the user interface, based on the information signaled from the self-propelled device, to reflect the orientation of the self-propelled device.

Claims

1. A controller device for operating a self-propelled device, the controller device comprising:

a touch-sensitive display;

one or more processors; and

one or more memory resources storing instructions that, when executed by the one or more processors, cause the one or more processors to;

generate a user interface on the touch-sensitive display, the user interface comprising a virtual controls to enable a user to remotely operate the self-propelled device;

using a camera, detect the self-propelled device in image data captured by the camera based on a location of the self-propelled device in a real-world environment, generate a virtual representation of the self-propelled device in a virtual environment on the user interface;

receive one or more user interactions with the virtual controls, the one or more user interactions to maneuver the self-propelled device; and

transmit one or more commands to the self-propelled device, the one or more commands to maneuver the self-propelled device in accordance with the one or more user interactions;

wherein the executed instructions cause the one or more processors to generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment, and wherein the executed instructions cause the one or more processors to generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment in accordance with execution of a gaming application that provides a gaming environment.

2. The controller device of claim 1, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:

directionally calibrate the user interface based on an orientation of the self-propelled device in relation to an initial frame of reference.

3. The controller device of claim 1, wherein the executed instructions cause the one or more processors to detect the self-propelled device in the image data as a fiducial marker for the virtual representation of the self-propelled device.

4. The controller device of claim 1, wherein the self-propelled device comprises a remotely operated aircraft.

5. The controller device of claim 1, wherein the executed instructions cause the one or more processors to detect the self-propelled device in the image data based on a pattern of light emitted by the self-propelled device.

6. The controller device of claim 1, wherein the gaming application corresponds to one of a plurality of gaming applications, executable by the controller device, and each providing a unique gaming environment.

7. A computer-implemented method for operating a self-propelled device, the method performed by one or more processors of a controller device and comprising:

generating a virtual user interface on a touch-sensitive display of the controller device, the user interface comprising virtual controls enabling a user to remotely operate the self-propelled device;

using a camera, detecting the self-propelled device in image data captured by the camera;

based on a location of the self-propelled device in a real-world environment, generating a virtual representation of the self-propelled device in a virtual environment on the user interface;

receiving one or more user interactions with the virtual controls, the one or more user interactions to maneuver the self-propelled device; and

transmitting one or more commands to the self-propelled device, the one or more commands to maneuver the self-propelled device in accordance with the one or more user interactions;

wherein the one or more processors generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment, and wherein the one or more processors generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment in accordance with execution of a gaming application that provides a gaming environment.

8. The method of claim 7, further comprising:

directionally calibrating the user interface based on an orientation of the self-propelled device in relation to an initial frame of reference.

9. The method of claim 7, wherein the one or more processors detect the self-propelled device in the image data as a fiducial marker for the virtual representation of the self-propelled device.

10. The method of claim 7, wherein the self-propelled device comprises a remotely operated aircraft.

11. The method of claim 7, wherein the one or more processors detect the self-propelled device in the image data based on a pattern of light emitted by the self-propelled device.

12. The method of claim 7, wherein the gaming application corresponds to one of a plurality of gaming applications, executable by the controller device, and each providing a unique gaming environment.

13. A non-transitory computer readable medium storing instructions for operating a self-propelled device, wherein the instructions, when executed by one or more processors of a controller device, cause the one or more processors to:

generate a user interface on a touch-sensitive display of the controller device, the user interface comprising virtual controls enabling a user to remotely operate the self-propelled device;

using a camera, detect the self-propelled device in image data captured by the camera;

based on a location of the self-propelled device in a real-world environment, generate a virtual representation of the self-propelled device in a virtual environment on the user interface;

one or more user interactions with the virtual controls, the one or more user interactions to maneuver the self-propelled device; and

transmit one or more commands to the self-propelled device, the one or more commands to maneuver the self-propelled device in accordance with the one or more user interactions;

wherein the executed instructions cause the one or more processors to generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment, and wherein the executed instructions cause the one or more processors to generate the virtual representation of the self-propelled device in the virtual environment to reflect user control of the self-propelled device in the real-world environment in accordance with execution of a gaming application that provides a gaming environment.

14. The non-transitory computer readable medium of claim 13, wherein the instructions when executed by the one or more processors further cause the one or more processors to:

directionally calibrate the user interface based on an orientation of the self-propelled device in relation to an initial frame of reference.

15. The non-transitory computer readable medium of claim 13, wherein the executed instructions cause the one or more processors to detect the self-propelled device in the image data as a fiducial marker for the virtual representation of the self-propelled device.

16. The non-transitory computer readable medium of claim 15, wherein the self-propelled device comprises a remotely operated aircraft.

17. The non-transitory computer readable medium of claim 13, wherein the executed instructions cause the one or more processors to detect the self-propelled device in the image data based on a pattern of light emitted by the self-propelled device.