# Godot Level Generator — RAG vs No-RAG Demo

Side-by-side comparison of code generated by the same LLM (minimax, via opencode) with and without lilbee RAG over Godot 4.x class reference XML docs.

## Setup

| | With lilbee | Without lilbee |
|---|---|---|
| LLM | minimax (opencode default) | minimax (opencode default) |
| Search tool | `search` (local RAG over 917 Godot XML class reference files) | `codesearch` ([Exa AI](https://exa.ai) code search, opencode built-in) |
| [AGENTS.md](../../demos/godot-with-lilbee/AGENTS.md) | Plan → `search` each class/method → write → verify | [Same workflow](../../demos/godot-without-lilbee/AGENTS.md) but uses `codesearch` instead |
| Prompt | *"make a procedural level generator that places wall and floor tiles and scatters collectibles using pathfinding"* | Same |

The "without lilbee" baseline is not unarmed — it has Exa code search, which can find Godot examples and documentation on the web. The difference is that lilbee provides pre-indexed, chunk-level access to the exact [Godot 4.4 class reference XML][godot-docs], while Exa returns web results that may mix 3.x and 4.x examples or miss specific method signatures.

## API Bugs Found

Both versions have bugs. The lilbee version has 1 structural issue; the without-lilbee version has 4 API errors.

### With lilbee (261 lines) — 1 bug

| Bug | Line | Issue | Correct API |
|-----|------|-------|-------------|
| `tile_map` never initialized | [21](../../demos/godot-with-lilbee/level_generator.gd#L21) | `tile_map` declared but never assigned — `set_tile_map()` setter exists but nothing calls it | Should create `TileMapLayer.new()` in `_ready()` or require it as `@export` |

All Godot API calls are correct: [`set_cell(coords, source_id, atlas_coords)`][tml-set-cell], [`AStarGrid2D.region`][asg2d], [`cell_shape`][asg2d], [`diagonal_mode`][asg2d], [`set_point_solid()`][asg2d], [`get_point_path()`][asg2d].

### Without lilbee (213 lines) — 4 bugs

| Bug | Line | Issue | Correct API |
|-----|------|-------|-------------|
| [`astar_grid.heuristic`](../../demos/godot-without-lilbee/level_generator.gd#L44) | 44 | Property doesn't exist | [`default_compute_heuristic`][asg2d] and [`default_estimate_heuristic`][asg2d] |
| [`existing_room.grow_individual(1)`](../../demos/godot-without-lilbee/level_generator.gd#L68) | 68 | `grow_individual` takes 4 args `(left, top, right, bottom)`, not 1 | [`grow_individual(1, 1, 1, 1)`][rect2i] or [`grow(1)`][rect2i] |
| [`rng.bool()`](../../demos/godot-without-lilbee/level_generator.gd#L80) | 80 | Method doesn't exist on [`RandomNumberGenerator`][rng] | `rng.randi() % 2 == 0` or `rng.randf() > 0.5` |
| [`get_id_path` → `PackedVector2Array`](../../demos/godot-without-lilbee/level_generator.gd#L205) | 205 | `get_id_path` returns `Vector2i[]`, not `PackedVector2Array` | Use [`get_point_path()`][asg2d] for `PackedVector2Array`, or type the var as `Array[Vector2i]` |

## Comparison

| Metric | With lilbee | Without lilbee |
|---|---|---|
| Lines of code | 261 | 213 |
| Pathfinding API | [`AStarGrid2D`][asg2d] | [`AStarGrid2D`][asg2d] (both found it) |
| API bugs | 1 (structural — `tile_map` uninitialized) | 4 (hallucinated property, wrong arity, nonexistent method, type mismatch) |
| `set_cell` signature | [Correct][tml-set-cell]: `(coords, source_id, atlas_coords)` | [Correct][tml-set-cell]: `(pos, 0, Vector2i, 0)` |
| Wall marking | [`set_point_solid()`][asg2d] ✓ | [`set_point_solid()`][asg2d] ✓ |
| Reachability check | Not used — places collectibles on open floor tiles | BFS flood-fill from center |
| Maze generation | Recursive backtracking + random rooms (inner class) | Room placement + corridors |
| Collectible placement | Shuffled floor tiles, adjacency check | Shuffled reachable tiles via BFS |
| External dependencies | Expects `TileMapLayer` passed in via `set_tile_map()` | Self-contained, creates own `TileMapLayer` |

## Did the Agents Follow Instructions?

Both [AGENTS.md](../../demos/godot-with-lilbee/AGENTS.md) files use the same 4-step workflow: plan → search each class/method → write using only confirmed APIs → verify every call before saving.

**The lilbee version followed instructions well.** It called `search` multiple times during planning and verification. All Godot API calls are correct — the only issue is a structural one (uninitialized `tile_map`), not an API hallucination.

**The without-lilbee version partially followed instructions.** It used `codesearch` but still produced 4 API errors — `heuristic` (should be `default_compute_heuristic`), `grow_individual(1)` (needs 4 args), `rng.bool()` (doesn't exist), and a type mismatch on `get_id_path`. These suggest the agent searched but didn't verify individual method signatures as step 4 requires.

## What RAG Changed

**API accuracy.** The lilbee version has zero incorrect Godot API calls. Every property, method, and enum maps exactly to the [Godot 4.4 class reference][godot-docs]. The without-lilbee version has 4 incorrect calls — `heuristic`, `grow_individual(1)`, `rng.bool()`, and a return type mismatch. Web search found the right classes but missed signature details.

**Both found `AStarGrid2D`.** Unlike the previous recording where the no-lilbee version fell back to `AStar2D`, this time both versions used `AStarGrid2D`. The stronger AGENTS.md (explicitly naming `codesearch`) helped the baseline agent search more thoroughly.

**Hallucination rate.** With lilbee: 0 hallucinated APIs out of ~15 distinct calls. Without lilbee: 4 hallucinated APIs out of ~18 distinct calls (~22% error rate). The indexed class reference XML gave the agent exact method signatures, while web search results were close but imprecise.

## Room for Improvement

These demos used a single prompt with no follow-ups and a minimal [AGENTS.md](../../demos/godot-with-lilbee/AGENTS.md). Several things could improve results:

- **Multi-turn iteration** — let the agent run the script in Godot, see errors, and fix them
- **Better model** — minimax-m2.5-free is a free-tier cloud model; a stronger model may follow the verify step more faithfully
- **More targeted indexing** — index only the ~20 classes relevant to 2D tile-based games instead of all 917, reducing noise in search results
- **Stronger AGENTS.md** — require the agent to print search results and confirm each method signature before writing code

[godot-docs]: https://github.com/godotengine/godot/tree/4.4-stable/doc/classes
[rect2i]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/Rect2i.xml
[rng]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/RandomNumberGenerator.xml
[tml-set-cell]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/TileMapLayer.xml
[ts-add-source]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/TileSet.xml
[asg2d]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/AStarGrid2D.xml
[as2d]: https://github.com/godotengine/godot/blob/4.4-stable/doc/classes/AStar2D.xml
