Home

Welcome!

In 2010, I designed a robot that can be remotely controlled over the internet. I named it Nestor, after the butler from the Belgian comic book “Tintin”.

The current version of my robot, which took around one year to design and build (in 2016), is 3D printed and fully reproducible. Based on the Raspberry Pi (RPi) single board computer, it is remotely controlled using a home made and open-sourced tool called NestoRPi Engine.

I wrote this tool in a generic way, making it easy to adapt to many kinds of use cases on the RPi (not only robots!). It can handle multiple devices through the GPIOs and provide and Web User Interface to control them. Finally, it supports the RPi GPU hardware encoder (using MMAL) as well as WebRTC to povide a very low latency video (below 200 ms) in high quality (H264 1080p 30 fps).

The robot is equipped with 8 motors, a speaker, a sound amplifier, and an LCD screen.

Other features include input devices, such as:

A camera and a light
A microphone
A speaker with a voice synthesizer
Ultrasound sensors (to determine the distance to objects in front of it)
A photoresistor sensor (to measure light intensity)
Rotary encoders (to control the speed of the motors)

In addition, the robot features a mechanical arm fixed on top, which can be raised and lowered, allowing the “hand” to pick up small objects. When the battery is low, the robot can easily and remotely plug in to a docking station.

Users can interact with the robot thanks to a web interface based on WebRTC technology. This technology, along with data channels, is used to provide a very low latency access for streaming audio, video, and commands, even when internet connection is slow.

Demo

Source code

A video of the robot is available (click on the thumbnail to play it):

Another video with a hacked RC car using the NestoRPi Engine:

A last one showing the Web User Interface:

A Powerpoint presentation:

Presentation link

An article from the Scality's online magazine:

Article link

The engine is mainly written in C++.

Strength:

10 000 LOC C/ C++ / JS
H264 hardware encoder / decoder support for very low latency (below 200 ms)
Command latency: < 20 ms between Paris and Bordeaux (fiber internet connection)
Engine: cross compilation from RPi0 to 4
Web UI: compatible with Chrome / Firefox / Safari. Tested on Windows / Linux / Android / Android TV

Doxygen link (develop branch)

Releases

Git branching model

Build history

TODO: The user manual

First Use

Consider the following example: you want to set up the NestoRPi Engine on a remotely-controlled car, based on a RPi, a L298N driver, and a camera. This quick tutorial will show you how to proceed.

After compiling and installing the engine on the RPi by following the README.md file from the repository, you have to edit a few config files in order to establish the communication between the engine and your hardware.

To do so, the bin/cfg/ dir contains the required files.

The "carpi" subdir is a perfect skeleton to start with.

Copy this dir to a new dir named after your robot's name.

WARNING: the robot's name should also be the hostname of your RPi. In this example, let's call the robot "Terminator". "terminator" has to be the hostname of your RPi, as well as the name of your config dir:

$ cp -r carpi terminator

Let's have a look at the files inside this config dir:

config.json

{
  "audio_dev": "default",
  "http_address": "0.0.0.0",
  "http_port": "8000s",
  "hw_video_encoder": true,
  "mbrola_params": "-a 100 -s 120 -p 20 -v mb/mb-fr1",
  "picture_height": 1080,
  "picture_width": 1920,
  "robot_name": "carpi",
  "role_minimal_audio": "full",
  "role_minimal_video": "full",
  "ssl_certificate": "nestorpi.pem",
  "stun_url": "stun.l.google.com:19302",
  "turn_url": "fmssoftware@hotmail.com:lolilol#numb.viagenie.ca",
  "v4l2_dev": [
    {
      "name": "/dev/video0"
    }
  ],
  "video_height": 480,
  "video_width": 640
}

"audio_dev" parameter: the "capture" audio used to connect a microphone and send the recorded audio to the web user interface.

To get the list of detected audio devices, run the NestoRPi Engine, and connect to it using a regular web browser: https://:8000/api/getAudioDeviceList

Keep the http_address as it is The "s" char at the end of the port indicates that the connection is done is a secured way using HTTPS
hw_video_encoder => use the GPU to encode the video and decrease the latency. Set it to false only if you are NOT using a RPi.
mbrola_params => Parameter used by the synthetic speech.
picture_height and picture_width is the default resolution used when a picture is taken from the robot.
robot_name: rpi's hostname (in our example, "terminator")
role_* => permission rules. Keep as it is.
ssl_certificate: "lets encrypt" is a good way to generate it: https://letsencrypt.org/ Only if HTTPS is enabled.
v4l2_dev: the webcam device
video_height / video_width: the video resolution. 640 * 480 is the highest resolution on RPi zero. On other RPis, you can use 1920 * 1080

devices.json

{
  "L298N": {
    "device": "L298N",
    "pins": {
      "ini1": 7,
      "ini2": 0,
      "ini3": 2,
      "ini4": 3,
      "pwm_enA": 5,
      "pwm_enB": 4
    },
    "pwm_hard": false,
    "type": "actuator"
  },
  "mbrola": {
    "device": "mbrola",
    "type": "actuator"
  },
  "temp_cpu": {
    "command": "echo $((`cat /sys/class/thermal/thermal_zone0/temp` / 1000))",
    "device": "command",
    "refresh_interval": 4,
    "type": "sensor"
  },
  "wifi_signal": {
    "command": "iwconfig wlan0 | grep -Po 'Quality=\\K([0-9]*)'",
    "device": "command",
    "refresh_interval": 1,
    "type": "sensor"
  }
}

This file specifies the devices connected to the GPIO pins of your RPi. The GPIOs numbers are the one used by the Wiring Pi lib, as described here: https://raw.githubusercontent.com/ppelleti/hs-wiringPi/master/pin-diagram.png

NestoRPi supports multiple devices, including L298N, HCSR04, MCP3008, servo.... Take a look at src/devices to get the list. You can add support to other devices by adding your own code.

mbrola is used for the synthetic voice. temp_cpu and wifi signal are sensors data that will be displayed in the web user interface.

tasks_automation.json

{
  "log": {
    "type": "log"
  },
  "monitoring": {
    "sensors_id": [
      "temp_cpu",
      "wifi_signal"
    ],
    "shm_path": "/dev/shm/nestor_",
    "type": "monitoring"
  }
}

tasks_controller.json

{
  "L298N": {
    "actuator_L298N_id": "L298N",
    "type": "L298N"
  },
  "audio_video": {
    "type": "audioVideo"
  },
  "voice": {
    "actuator_mbrola_id": "mbrola",
    "type": "voice"
  }
}

Copy this dir into the "cfg" dir of your RPi and launch the engine, as described in the README.md file. The web user interface is available at: https://:8000/