Merge remote-tracking branch 'origin/main'

Author: bcherry

README.md

@@ -1,15 +1,15 @@
 # LivePaint
 
-This is an example project demonstrating how to build a realtime data app using LiveKit.
+This LiveKit example project is a realtime drawing game where players compete to complete a drawing prompt as fast as possible, while being judged by a realtime AI agent that oversees the whole game. 
 
-In this example, we build a realtime drawing game where players compete to complete a drawing prompt as fast as possible, while being judged by an AI agent that oversees the whole game. 
+It demonstrates the use of LiveKit's [realtime data messages](https://docs.livekit.io/home/client/data/messages), [room metadata](https://docs.livekit.io/home/client/data/room-metadata/), [RPC](https://docs.livekit.io/home/client/data/rpc/), [participant management](https://docs.livekit.io/home/server/managing-participants/), [token generation](https://docs.livekit.io/home/server/generating-tokens/), and [realtime audio chat](https://docs.livekit.io/home/client/tracks/) in a real-world app built on the LiveKit [JS SDK](https://github.com/livekit/client-sdk-js), [React Components](https://github.com/livekit/components-js), [Python agents SDK](https://github.com/livekit/agents), and [Python Server API](https://github.com/livekit/python-sdks).
 
-This example demonstrates the use of [realtime data messages](https://docs.livekit.io/home/client/data/messages), [room metadata](https://docs.livekit.io/home/client/data/room-metadata/), [RPC](https://docs.livekit.io/home/client/data/rpc/), [participant management](https://docs.livekit.io/home/server/managing-participants/), [token generation](https://docs.livekit.io/home/server/generating-tokens/), and [realtime audio chat](https://docs.livekit.io/home/client/tracks/) in a real-world app built on the LiveKit [JS SDK](https://github.com/livekit/client-sdk-js), [React Components](https://github.com/livekit/components-js), [Python agents SDK](https://github.com/livekit/agents), and [Python Server API](https://github.com/livekit/python-sdks).
-
-Try it live at [https://live-paint.vercel.app](https://live-paint.vercel.app)!
+Play live at [https://paint.livekit.io](https://paint.livekit.io)!
 
 ## Architecture
 
+This is a short overview of how this game was built. The entire codebase is also annotated with comments that go into more detail. The `agent` directory contains the code for the realtime agent (built on [LiveKit Agents](https://docs.livekit.io/agents)). The `web` directory contains the code for the game frontend (built on [Next.js](https://nextjs.org/) with [LiveKit React Components](https://github.com/livekit/components-js)).
+
 ### Rooms & Participants
 
 Each game is hosted in a single [LiveKit room](https://docs.livekit.io/home/client/connect) where each player is a standard participant.  The room is reused between games, so the same group of players can complete multiple games back-to-back.
@@ -58,37 +58,56 @@ The agent is responsible for judging each player's drawing. It runs a single loo
 
 Realtime chat is enabled within each room by [publishing the local microphone](https://docs.livekit.io/home/client/tracks/publish/) and [rendering the room audio](https://docs.livekit.io/reference/components/react/component/roomaudiorenderer/).
 
-## Ideas / What's Next?
+## Ideas & What's Next?
 
-If you'd like to learn to build with LiveKit, try to implement the following feature ideas or invent your own:
+Learn to build with LiveKit by adding one of the following features, or come up with your own!
 
 - Add a scoreboard that shows how many wins each player has racked up
     - We think [participant attributes](https://docs.livekit.io/home/client/data/participant-attributes/) is a great place to keep track of this
 - Have the AI agent make its guesses and announce winners with realtime audio as well as text
-    - We'd try using a [Text-To-Speech plugin](https://docs.livekit.io/agents/plugins/#text-to-speech-tts)
-    - Consider having the agent publish a different track to each participant, so they don't need to hear the guesses for everyone else in realtime
-- Add a room list on the front page that shows open rooms and lets you join one
-    - Try the [List Rooms](https://docs.livekit.io/home/server/managing-rooms/#list-rooms) Server API
+    - We'd try using a LiveKit [Text-To-Speech (TTS) plugin](https://docs.livekit.io/agents/plugins/#text-to-speech-tts)
+    - To make it perfect, have the agent [publish](https://docs.livekit.io/home/client/tracks/publish/) a different audio track to each participant so they can hear the guesses for everyone else in realtime
+- Add a room list on the front page that shows open rooms and lets you join any of them
+    - We'd use the [List Rooms](https://docs.livekit.io/home/server/managing-rooms/#list-rooms) Server API in a Next.js API route
 - Add support for multiple brush sizes and colors
-    - You'll probably want to extend the data format for `Line` to record brush size and color
-
+    - You'll need to extend the data format for `Line` to record brush size and color
 
 ## Development & Running Locally
 
-Run the agent:
+You'll need a LiveKit instance to run this project, either from [LiveKit Cloud](https://cloud.livekit.io) or [Self-hosted](https://docs.livekit.io/home/self-hosting/local/).
 
-```
+### Running the Agent
+
+First add `agent/.env` with LIVEKIT_API_KEY, LIVEKIT_API_SECRET, LIVEKIT_URL, and OPENAI_API_KEY.
+
+Then run the following commands to install dependencies:
+
+```shell
 cd agent
 python -m venv venv
 source venv/bin/activate
 pip install -r requirements.txt
+``
+
+Finally, boot the agent:
+
+```shell
 python main.py dev
 ```
 
-Run the site:
+### Running the Site
 
-```
+First add `web/.env.local` with LIVEKIT_API_KEY, LIVEKIT_API_SECRET, and LIVEKIT_URL.
+
+Then run the following commands to install dependencies:
+
+```shell
 cd web
 pnpm install
+```
+
+Finally, start the site:
+
+```shell
 pnpm dev
 ```

agent/drawings.py

@@ -4,6 +4,8 @@
 from PIL import Image, ImageDraw
 
 
+# Points are stored as floats between 0 and 1 and assume a square canvas
+# This allows the actual display canvas size to differ between players and the host to fit their needs
 class Point:
     def __init__(self, x: float, y: float):
         self.x = x
@@ -15,6 +17,11 @@ def __init__(self, from_point: Point, to_point: Point):
         self.from_point = from_point
         self.to_point = to_point
 
+    # Lines are encoded as efficiently as possible as they are sent over the network in high frequency as data messages
+    # and also encoded in bulk as base64 for drawing restoration (the `player.get_drawing` RPC call)
+    # While points are normally stored as 32-bit floats, we can can save 50% of the space by using 16-bit integers instead when they are sent over the network
+    # The integers are thus in the range of 0 to 65535. This is more than enough for our purposes, as no player is likely to have a canvas larger than about 1024x1024 pixels anyways
+    # Also note that we have a parallel implementation in the client in `web/lib/drawings.ts` that performs the same operations in TypeScript
     def encode(self) -> bytes:
         return struct.pack(
             "<HHHH",
@@ -24,6 +31,7 @@ def encode(self) -> bytes:
             int(self.to_point.y * 65535),
         )
 
+    # We decode lines by reversing the packing operation performed above
     @staticmethod
     def decode(data: bytes) -> "Line":
         return Line(
@@ -44,6 +52,8 @@ def __init__(self, player_identity: str):
         self.lines = set()
         self._hash = None
 
+    # We use an MD5 hash of the line segments to identify duplicate drawings
+    # This allows us to make efficient keys for the `GuessCache` that will automatically change when the drawing is modified
     def hash(self) -> str:
         if self._hash:
             return self._hash
@@ -55,14 +65,19 @@ def hash(self) -> str:
         self._hash = hash_obj.hexdigest()
         return self._hash
 
+    # Adds a new line to the drawing
     def add_line(self, line: Line):
         self.lines.add(line)
         self._hash = None
 
+    # Clears the drawing (removes all lines)
     def clear(self):
         self.lines.clear()
         self._hash = None
 
+    # Generates an image representing the current state of the drawing
+    # We use a size of 512x512 by default, which is an efficient size for GPT-4o-mini in "low detail" mode
+    # See https://platform.openai.com/docs/guides/vision#low-or-high-fidelity-image-understanding for more information
     def get_image(self, size: int = 512, stroke_width: int = 4) -> Image:
         canvas = Image.new("1", (size, size), 1)
         draw = ImageDraw.Draw(canvas)
@@ -82,6 +97,4 @@ def get_image(self, size: int = 512, stroke_width: int = 4) -> Image:
                     width=stroke_width,
                 )
 
-        debug_path = f"/tmp/drawing_{self.player_identity}.png"
-        canvas.save(debug_path)
         return canvas

agent/game.py

@@ -0,0 +1,113 @@
+from typing import Literal, List
+import json
+from collections import OrderedDict
+
+DifficultyLevel = Literal["easy", "medium", "hard"]
+
+PROMPTS = {
+    "easy": [
+        "cat",
+        "dog",
+        "elephant",
+        "giraffe",
+        "lion",
+        "monkey",
+        "penguin",
+        "rabbit",
+        "turtle",
+        "bed",
+        "door",
+        "fan",
+        "apple",
+        "banana",
+        "cake",
+        "cookie",
+        "car",
+        "boat",
+        "bus",
+    ],
+    "medium": [
+        "airplane",
+        "helicopter",
+        "rocket",
+        "castle",
+        "bridge",
+        "lighthouse",
+        "windmill",
+        "doctor",
+        "chef",
+        "pilot",
+        "dancer",
+        "baseball",
+        "basketball",
+        "soccer",
+        "tennis",
+        "robot",
+        "dragon",
+        "wizard",
+        "pirate",
+        "ghost",
+    ],
+    "hard": [
+        "thunderstorm",
+        "northern lights",
+        "coral reef",
+        "redwood forest",
+        "hot air balloon",
+        "vacuum cleaner",
+        "musical conductor",
+        "construction site",
+        "garden party",
+        "tug of war",
+        "arm wrestling",
+        "rock climbing",
+        "thumb wrestling",
+        "playing chess",
+        "building sandcastle",
+    ],
+}
+
+NO_GUESS = "NO_GUESS"
+CHEATER_CHEATER = "CHEATER_CHEATER"
+PARTICIPANT_LIMIT = 12
+
+
+class GameState:
+    def __init__(
+        self,
+        started: bool = False,
+        difficulty: DifficultyLevel = "easy",
+        prompt: str | None = None,
+        winners: List[str] = [],
+    ):
+        self.started = started
+        self.difficulty = difficulty
+        self.prompt = prompt
+        self.winners = winners
+
+    def to_json_string(self) -> str:
+        return json.dumps(self.__dict__)
+
+    @staticmethod
+    def from_json_string(json_string: str) -> "GameState":
+        return GameState(**json.loads(json_string))
+
+
+class GuessCache:
+    def __init__(self, max_size: int = 1000):
+        self._cache = OrderedDict()
+        self._max_size = max_size
+
+    def get(self, hash: str) -> str | None:
+        if hash in self._cache:
+            self._cache.move_to_end(hash)
+            return self._cache[hash]
+        return None
+
+    def set(self, hash: str, guess: str):
+        if hash in self._cache:
+            self._cache.move_to_end(hash)
+        else:
+            if len(self._cache) >= self._max_size:
+                self._cache.popitem(last=False)
+        self._cache[hash] = guess