Remotely controlling a self-propelled device in a virtualized environment

• US 10,248,118 B2
• Filed: 11/05/2015
• Issued: 04/02/2019
• Est. Priority Date: 01/05/2011
• Status: Active Grant

• Current Assignee: Sphero Inc.
• Original Assignee: Sphero Inc.
• Inventors: Bernstein, Ian, Wilson, Adam, Carroll, Jonathan, Polo, Fabrizio
• Primary Examiner(s): Cheung, Mary
• Application Number:
  US14/933,827
• Publication Number:
  US 20160054734A1
• Time in Patent Office: 1,244 Days
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: A63H 33/005 Motorised rolling toys
• CPC Class Codes: B62D 63/025 Modular vehicles superstructures for busses having modular sections B62D31/025
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0259 using magnetic or electromagnetic means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A computing device operating as a controller can obtain image data from a camera component. The computing device can determine a location of the self-propelled device relative to the camera based on the image data. A virtual content may be generated on the computing device based at least in part on the location of the self-propelled device.

Claims

1. A computer-implemented method for operating a computing device, the method comprising:

generating image data by a camera component of the computing device;

programmatically detecting, by the computing device, from the image data, a location of a self-propelled device relative to the computing device;

receiving, at the computing device, input from the self-propelled device, the input including sensor information that is obtained on the self-propelled device; and

generating, on the computing device, content based on the location of the self-propelled device as the self-propelled device moves, wherein the self-propelled device is remote from the computing device.

2. The computer-implemented method of claim 1, wherein programmatically detecting the location of the self-propelled device comprises image processing the image data.

3. The computer-implemented method of claim 2, wherein image processing the image data comprises utilizing an object-type specific detector to locate at least one of a predetermined shape, a predetermined color, a predetermined visual code, and a predetermined structural feature.

4. The computer-implemented method of claim 1, further comprising programmatically detecting, from the image data generated by the camera component of the computing device, a location of an object relative to at least one of the computing device and the self-propelled device.

5. The computer-implemented method of claim 1, wherein the sensor information comprises information obtained from at least one of a gyroscope, an inertial mass unit, a GPS, an accelerometer, a light sensor, and a proximity sensor located on the self-propelled device.

6. The computer-implemented method of claim 1, further comprising displaying the generated content on the computing device.

7. The computer-implemented method of claim 6, wherein the generated content comprises a map.

8. The computer-implemented method of claim 6, wherein the generated content comprises a depiction of a physical space in which the self-propelled device is located.

9. The computer-implemented method of claim 6, wherein the generated content corresponds to a virtual environment.

10. The computer-implemented method of claim 9, wherein the generated content comprises a graphic form of the self-propelled device.

11. The computer-implemented method of claim 1, further comprising:

identifying a landmark from the image data generated by the camera component of the computing device; and

translating the landmark into an object, wherein the generated content comprises a virtual environment and the object.

12. The computer-implemented method of claim 11, further comprising displaying the generated content on the computing device.