Fitness Buddy: turning conversations into activities

Fitness Buddy is now defunct. This is a retrospective on how the project worked and what made it an interesting experiment.

Fitness Buddy started as a short hackathon project: an AI companion with a personality, some knowledge of your activities, and the ability to help turn a conversation into something concrete. A buddy could talk about a run, help establish a goal, or create an activity from a message. Different personas made the same basic product feel surprisingly different.

The conversation was the visible part. Underneath it sat a fairly familiar web application: Next.js and TypeScript, Supabase authentication and PostgreSQL, Deno edge functions, an S3-backed media pipeline, and a Strava integration. The interesting question was how to let a conversational interface interact with that application without losing track of what had actually happened.

Personality sits on top of application state

A buddy persona supplied a name, personality traits, and a voice for responses. Conversations belonged to a user and a buddy, and the response path could include recent activity and active goals as context. The project also used generated character art and animated assets to give the buddies a more tangible presence.

That combination made the experiment fun, but the personality could only carry it so far. A friendly reply about an activity needs to agree with the activity that was actually saved. I wanted conversational warmth to coexist with ordinary, inspectable records: activities, goals, messages, and uploaded media.

A conversation becomes a proposed action

Take a message such as “I want to go for a 30 minute run at noon.” Producing an encouraging sentence is one task. Creating a run with a duration, date, owner, and relationship to the conversation is another. The implementation separated analysis from the functions that create, update, or delete application records.

User message + recent conversation + activities + timezone
  → structured analysis of proposed actions
  → validation and confidence checks
  → activity, goal, or media handler
  → database records and notification messages

Analysis returned fields describing whether an activity or goal should be created, whether an existing activity should change, or whether media should be associated with something already recorded. Each operation had its own details and confidence score. Conversational prompting and action processing used different thresholds, with the latter configured more conservatively.

The application then had to validate the proposal. Were required details present? Did the model suggest conflicting operations? Was something that looked like a reference to a run actually being treated as a database ID? The shared validator checked several of these cases and included a repair path for media associations incorrectly placed inside activity updates. A schema was useful, but domain rules still had to live in code.

“This was from my run” is a data-model problem

Media made the distinction especially clear. If someone uploaded a photo and said it was from yesterday's run, the useful operation was to connect the photo to the existing run. Creating another run would leave the conversation looking helpful while quietly making the activity history worse.

Images uploaded directly to S3 through signed URLs, with media records tracking upload state and ownership. An edge function could analyze the image and store structured metadata, including tags and a description influenced by the chosen buddy. Media-to-activity association remained a separate operation, with its own IDs and application logic.

Keeping those relationships explicit made it possible to inspect an individual media object, its analysis, and the activity it belonged to. The database migrations also added audit logs for activities and media. When a conversation can change persistent state, being able to reconstruct that change becomes part of the product.

Strava and timezones make the conversation less abstract

The Strava integration handled OAuth tokens, refreshed expired credentials, fetched activities, and mapped activity types into the application's own categories. It carried over data such as duration, distance, elevation, and available route information. Activities could also come from manual input or conversations, so their source was part of the record.

Time was just as important as activity type. “Noon” is incomplete without a date and a timezone. The application stored a profile timezone and passed it into analysis and activity handlers, with UTC as a fallback. Even a tiny conversational example quickly becomes an integration problem once another service and a real person's local time are involved.

There were limits to the experiment

This was a prototype, and the code reflected that. Some paths implemented real database operations, while a more general agent abstraction still contained placeholder actions. The conversational workflow was much more specific than a universally capable assistant. Keeping that distinction visible matters when looking back at what the project actually did.

Confidence scores also had a practical limit. They helped route model output, but they were model-generated values, not a guarantee that an operation was correct. The useful engineering was in the surrounding checks: ownership, context, record lookup, validation, and the notifications that told the user what changed.

Fitness Buddy is no longer running, but it was a good project for exploring how language interfaces meet ordinary application design. The parts I still find interesting are the boundaries: a personality and a user record, a sentence and a scheduled activity, a photo and the event it belongs to. Each one gives the application something specific to verify.