Robbie G2

Gen 2 AI Agent that uses OCR, Canny Composite, and Grid to navigate GUIs

Explore the docs »

View Demo · Report Bug · Request Feature

Meet Robbie, or Gen 2 agent.

Robbie navigates GUIs to solve tasks for you.

Unlike other bots, he doesn't just work on the web because he doesn't use Playwright. Robbie is a pure multimodal bot. He can navigate the web or a desktop.

That means he can navigate SaaS apps or he can work on a remote desktop and send emails, search for flights, check Slack, do research and more.

Robbie-g2, aka Gen 2, is a leap from our first gen agents, SurfPizza and SurfSlicer. He's very capable at navigating complex, never before seen GUIs via a remote virtual desktop which the AgentSea stack serves up as a device to him via DeviceBay. He connects to it via ToolFuse and AgentDesk, which lets him know what he can do with it, like move the mouse, send key commands, etc.

‣ Check out our community on Discord where we develop in the open, share research and connect with other developers who are building cutting edge agents or who just want to use them to get things done!

‣ Check out the deep dive video right here:

Quick Start

Prerequisites

• Install Docker - you need it to run a Tracker
• Install QEMU OR Configure GCP OR Configure AWS - you need one of these to host a Device

Setup

1. Setup your OpenAI API key:

export OPENAI_API_KEY=<your key>

2. Install/upgrade SurfKit:

pip install -U surfkit

3. Clone the repository and go to the root folder:

git clone git@github.com:agentsea/robbie-g2.git && cd robbie-g2

4. Install dependencies:

poetry install

Creating required entities

5. Create a tracker:

surfkit create tracker --name tracker01

6. Create a device:

• If you are using QEMU:

surfkit create device --provider qemu --name device01

• If you are using GCE:

surfkit create device --provider gce --name device01

• If you are using AWS:

surfkit create device --provider aws --name device01

7. Create an agent:

surfkit create agent --name agent01

Solving a task

surfkit solve "Search for common varieties of french ducks" \
  --tracker tracker01 \
  --device device01 \
  --agent agent01

Documentation

See our docs for more information on how to use Surfkit.