Database-Connected AI Chatbot with Evals

I built a conversational assistant in five stages, each one adding a single capability: a Streamlit chat screen, a first OpenAI call, conversation memory, a weather tool, and finally a tool that reads a live database.

Memory was the smallest change with the largest effect. Sending the running message history back with every request turns a stateless endpoint into something that follows a thread.

Tool calling is where the assistant stopped guessing. A weather tool calls a live API, and a second tool reads our cohort's table from Supabase Postgres. The model decides which one to call, and in what order, based on the question. A system prompt then sets how it speaks without touching any of the logic. Every commit deploys it to Fly automatically.

Later in the course I went back to the same chatbot and made it measurable. I added Langfuse tracing so every answer is recorded with its inputs, then set up two AI judges that score each live answer for relevance and for tone and safety. Finally I ran an offline experiment against a fixed test set of 23 questions with a correctness judge, so a change to the prompt can be checked against the same questions before it ships.