How could I integrate the single-call Qwen3.5-9B inference flow through Together AI into Vee's main workflow?
So here was the plan:
1. Update the assemble_interaction_state function to also include the calendar periods for this week, last week, and next week.
2. Build the inference API code to send the System Prompt + Interaction State to Together AI, using Qwen3.5-9B, and receive the model's response.
3. Integrate the inference logic into Vee's core backend workflow.
So I started with the first step: updating assemble_interaction_state.
I'd already refactored the code into a cleaner architecture, which made this part easier.
I added the calendar periods and updated how the Interaction State was assembled.
Here's a snippet of the Interaction State, ready to be concatenated with the System Prompt before invoking Qwen.
Next, I moved on to the inference API.
1
42
Now that I'd finished testing Vee's system prompt,
it was time to move on to Phase 4 of Milestone 4:
integrating the single-call Qwen3.5-9B inference flow into Vee's main workflow.
Outcome = Vee's structured decisions in JSON, ready to be parsed and validated.
So, first, for the inference, I needed to choose a provider.
I'd already chosen @togethercompute as my inference provider.
So far, after thousands of API tests, they'd been reliable enough to become my leading candidate for Vee's production.
But I wasn't ready to make that decision just yet.
I thought, "Let me keep using them for the initial development."
And honestly, I didn't want to worry about hardware at this early stage of my company.
I could make millions from software first, then eventually own my hardware.
Besides, the real power wasn't the model or the infrastructure.
It was the harness I was building around it.
As long as that remained model-agnostic,
I could plug in another small model if anything happened to my provider.
So I had my model, a working system prompt, and an inference provider.
Now I needed to bring everything together.
So I went back to the code editor.
Replying to @MoTheAgent
What Was I Actually Building With Each Milestone?
While rethinking M4's goal, I realised my perspective on the milestones had changed.
I was no longer seeing them as just technical deliverables, but as missions to build different parts of Vee, the AI witness.
Looking back:
M1 - Foundation: Vee and the company's memory. A relational database storing work, messages, and history.
M2 - Telegram Bot: The component allowing Vee to receive and reply to my messages.
M3 - Interaction State Assembly: The component preparing the context Vee needed to make decisions: my message, recent conversations, and active work items.
And then there was M4.
Originally, I'd viewed it as implementing the system prompt, invoking the LLM, and validating its output.
But now I saw it as Vee's central decision-making component.
So I refined its goal to:
Build Vee's central decision-making component, capable of making judgements, generating a tool execution plan, and deciding what to say, all in a SINGLE LLM invocation.
And since my perspective on the milestones had changed, I went back and refined the entire development plan around the different components of Vee.
Here's what the updated milestones looked like: 👇
Now that I had M4's goal, next, i had to figure out how was I going to break it into phases?
1
67
1/2 Backstories
👇 The story of how I came up with the phases.
So How Did I Plan to Build Vee's Central Decision-Making Component?
This was the part responsible for making a single structured decision for every interaction:
Judge → Plan → Say
All in ONE LLM invocation.
So I broke M4 into four phases:
Phase 1 - Decision Output Contract & Validation
Define the Judge → Plan → Say decision output schema, parsing, and validation.
Phase 2 - Dialogue Test Cases
Design test conversations covering Vee's brain-dumping and remembering behaviours.
Phase 3 - System Prompt & Manual Testing
Write the system prompt and manually test it against those cases using Qwen3.5-9B.
Phase 4 - Integration
Integrate the single-call Qwen3.5-9B inference flow into Vee's main workflow, using the assembled Interaction State, then parse and validate its output.
Now that I had the plan, it was time to start building.
Starting with Phase 1: What should Vee's decision output actually look like?
1
19
2/2
👇 How I came up with Vee's core workflow architecture.
Replying to @MoTheAgent
Here's the architecture I came up with.
First, the mechanism stores my message.
Then it builds the state it needs to understand what I'm talking about.
That includes our last 10 conversation turns and the company's existing items.
It combines that state with the system prompt and sends everything to the LLM.
But that's only the beginning.
What happens when the LLM decides something needs to change?
7
1/2
So apparently, Vee's little Qwen 9B brain was interpreting "last week" in my question:
"What did I get done last week?"
As if "last" meant the FINAL week relative to the current date.
That's why it kept selecting THIS week instead of LAST week.
I spent hours and hours thinking, refining the Thinking Loop, and rewriting the Facts section of the system prompt.
And Qwen STILL struggled to select the correct period.
Then it HIT ME.
"WHY was I forcing the LLM to do something I could easily do in code????"
I was already passing the current date and time to Vee through code.
So why couldn't I use that same code to calculate the dates for last week, this week, and next week BEFORE invoking the model?
That would mean invoking:
My question + Interaction State (current date and time + Items + Item History + calendar periods) + System Prompt.
Just like a human having a calendar in front of them.
So I made the change.
And guess what?
10/10 PASSED. I ran another 10 tests. 10/10 again.
And another 10. 10/10 AGAIN.
30/30 successful runs on the same scenario that had been driving me crazy for hours.
And the funny thing?
This was completely different from how I'd originally planned to solve the problem.
All I did was give Vee a calendar.
Here's the test:
Back to finalizing Vee’s system prompt on Qwen’s little 9B brain.
Scenario 2: Remembering - Reflection.
This was the LAST scenario I needed to test before moving on to the next phase of the project.
It was basically me asking Vee:
“What did I get done last week?”
Then invoking my question + Interaction State (current time + Items + Item History) + System Prompt.
The Items included work with different states, like todo, completed, and cancelled, along with the history of how each Item got there.
The current date and time in the test was:
Friday, Nov 13, 2026 - 4:45 PM.
I expected Vee to:
- Identify last week: Nov 2–8 (Monday–Sunday).
- Return the exact IDs of Items completed during that period.
Well, after 10 test runs, Vee kept selecting Nov 9–15 instead.
That’s THIS week. Not LAST week.
And the funny thing?
It was correctly identifying completed Items based on the period it selected.
But it was selecting the WRONG period.
I kept refining the prompt( the facts and thinking loop sections ) and testing again.
Same problem.
I genuinely couldn’t figure out why it kept doing this.
And this was the ONE remaining scenario standing between me and the next phase.
Man, I had no idea a simple question about last week was going to take up so much of my time. 😂
2
3
150
2/2
What's crazy about all of this is that @Wv3EcACHtizRjZK actually commented on my last post suggesting something very similar.
But it completely went over my head.
I even replied, explaining why I wanted the LLM to handle the whole interpretation.
Only for me to spend HOURS trying to fix the same problem.
And then it finally hit me.
OMG. THIS IS WHAT HE MEANT.
Man, sometimes the answer is literally right in front of you, and you still have to suffer for hours before it clicks.
But this whole experience taught me something important.
I'd spent hours trying to make the model reason its way through something I could simply give it as a fact.
It wasn't about making the prompt longer or adding more instructions.
It was about understanding what judgement the LLM actually needed to make, and what I could take off its plate.
There were still other scenarios I wanted to test, but I decided to leave those until I integrated the backend with Telegram.
And with that, the final scenario was done.
Phase 3 of Milestone 4 was officially complete.
Next, I moved on to the final phase of M4: Phase 4.
Replying to @MoTheAgent
Since it picks the right items once it has a range, I'd calculate Nov 2–8 in code and pass those boundaries in. Then test the week resolver separately with Sundays, Mondays and year-end dates.
1
50
Back to finalizing Vee’s system prompt on Qwen’s little 9B brain.
Scenario 2: Remembering - Reflection.
This was the LAST scenario I needed to test before moving on to the next phase of the project.
It was basically me asking Vee:
“What did I get done last week?”
Then invoking my question + Interaction State (current time + Items + Item History) + System Prompt.
The Items included work with different states, like todo, completed, and cancelled, along with the history of how each Item got there.
The current date and time in the test was:
Friday, Nov 13, 2026 - 4:45 PM.
I expected Vee to:
- Identify last week: Nov 2–8 (Monday–Sunday).
- Return the exact IDs of Items completed during that period.
Well, after 10 test runs, Vee kept selecting Nov 9–15 instead.
That’s THIS week. Not LAST week.
And the funny thing?
It was correctly identifying completed Items based on the period it selected.
But it was selecting the WRONG period.
I kept refining the prompt( the facts and thinking loop sections ) and testing again.
Same problem.
I genuinely couldn’t figure out why it kept doing this.
And this was the ONE remaining scenario standing between me and the next phase.
Man, I had no idea a simple question about last week was going to take up so much of my time. 😂
1/2
Back to testing Vee's system prompt on Qwen's little 9B brain.
So far, the same prompt had consistently passed my brain-dumping tests.
Now it was time to test REMEMBERING.
As a solo founder trying to compete with much larger companies, I needed to context-switch faster.
And to switch faster, I needed to remember faster.
Instead of spending time reconstructing where I left things off every time I returned to a different part of my company, I wanted to simply ask Vee.
But here's the technical challenge.
I wanted the SAME system prompt to:
- Judge whether I was brain-dumping or asking to recall progress.
- Plan the appropriate actions.
- Decide what to say.
All in ONE invocation.
No separate if/else instructions for each use case.
👉 Just a Thinking Loop + State + Output Schema.
It finally handled both kinds of requests.
Now I wanted to know how reliably.
So I ran Remembering Scenario 01 through Vee's Lab.
Me asking: "What do i got today"?,
and passing a sample tasks & events.
10 runs, measuring Judge → Plan → Say accuracy.
Here's what happened. 👇
1
84
2/2
Scenario 01: Today's Alignment.
I asked Vee:
"What do I got today?"
Progress Memory contained 7 items, including unfinished tasks, completed work, and scheduled events.
I expected Vee to select ONLY the 2 open tasks.
First 10 runs?
0/10 strict passes. 😂
But here's where things got interesting.
Vee selected both open tasks in ALL 10 runs.
It also included the 11 AM standup, and sometimes tomorrow's family meeting.
At first, I considered that a failure.
But then I questioned my own expectations.
When I'm trying to orient myself, sometimes knowing what's already happened or what's coming next actually helps.
So why was I forcing Vee to show ONLY unfinished tasks?
I ended up loosening the evaluator to allow reasonable interpretations.
Man, measuring an autonomous agent's reliability was becoming a challenge of its own.
1
40
1/2
The last question in my previous post had me going back to WHY I was building Vee as a solo founder in the first place.
For now, I wanted it to help me answer one question:
"Where did I leave things off?"
Every time I returned to a different part of my company.
But I discovered that answering this required reconstructing an entire working state.
And Vee's first job was to help me with just ONE part of that state.
My progress.
Before answering "Where did I leave things off?", I needed to know:
What's completed?
What's still unresolved?
So far, I'd built:
My brain dumps → Vee ✅
But in my last post, I was figuring out the other direction:
Vee → ME.
So I went back to asking myself:
What was my intention when i asked "What's completed?" or "What's still unresolved?".
Replying to @MoTheAgent
3/3
And here's where things got interesting. 😂
0/10 strict passes.
But when I inspected the actual decisions, Vee had selected the correct 2 Item IDs in ALL 10 runs.
No extra items. No missing completions.
So why was it failing?
Vee kept classifying my intention as Orientation instead of Reflection.
even though it understood that I was asking what I'd completed today.
it failed to identify why was it asking ?
That got me questioning something.
"Did I even need this classification in the first place?"
1
55
2/2
I found that when I asked "What's completed?", I was reflecting.
And when I asked "What's still unresolved?", I was trying to orient myself to start work.
But then I realised something.
Sometimes, when orienting, I wanted to reflect first, THEN orient.
It just got a bit complicated, not gonna lie.
But I didn't have time to overthink it.
I could keep refining while shipping.
For now, I just wanted to test whether Vee could reliably recall:
- What I'd completed during a specific period.
- What still remained unresolved.
So I went back to improving the system prompt and the benchmarks.
I needed to improve Vee's Judge → Plan → Say reliability in answering these questions so I could move on to the next phase of the project.
So the next question was:
How reliable could I actually make Vee at answering these two questions?
29
1/3
Back to testing Vee's system prompt on Qwen's little 9B brain.
First of all, this experience taught me something.
I seriously underestimated the importance of language and philosophy in prompt engineering.
Especially when trying to get reliable outputs while giving the AI agent enough autonomy to judge, plan, and decide what to say.
And enough agency to grow with you.
Anyway, this day's challenge was Reflection.
I wanted Vee to reliably answer a question I'd normally ask myself every day or week:
"What did I actually get done today/this week?"
FYI - purpose was:
Before Vee: [Me sitting there trying to remember everything. ]
After Vee: An evidence-based account of my progress in seconds.
Well, after hours of testing, refining definitions, and questioning my own expectations...
Here was the reliability benchmark from Vee's Lab.
2
2
75
2/3
So what exactly was I testing?
Scenario 06: Remembering - Reflection.
the question was:
"what did i get done today"
Progress Memory in input context contained 7 Items (shortened IDs):
001 — Hit the gym | Cancelled (Oct 30)
002 — Landing copy | Todo (Carried Oct 30)
003 — Call doctor | Done (Oct 30)
004 — Family meeting | Todo (Oct 31)
005 — Meet a friend | Done (Oct 30)
006 — Send proposal | In progress (Carried Oct 30)
007 — Team standup | Todo (Oct 29)
I expected Vee to:
- Recognize intention(Reflection, not Orientation).
- Identify what actually happened today.
- Select IDs 003 and 005.
- Generate the right structured testimony.
And here's what I got after multiple rounds of 10-run reliability testing. 👇
1
52
3/3
And here's where things got interesting. 😂
0/10 strict passes.
But when I inspected the actual decisions, Vee had selected the correct 2 Item IDs in ALL 10 runs.
No extra items. No missing completions.
So why was it failing?
Vee kept classifying my intention as Orientation instead of Reflection.
even though it understood that I was asking what I'd completed today.
it failed to identify why was it asking ?
That got me questioning something.
"Did I even need this classification in the first place?"
1/2
Back to testing Vee's little 9B brain.
Scenario 05: Remembering — Orientation.
This time, I asked Vee:
"What do I have today?"
Progress Memory contained completed, cancelled, and unfinished work, plus scheduled Events.
I expected Vee to select the relevant Item IDs and generate a structured testimony.
First 10 runs?
5/10 PASSED. 😂
But the interesting part wasn't the score.
It was what Vee decided NOT to include. 👇
1
45
2/2
Here's what Progress Memory contained:
- 2 unfinished tasks.
- Completed work.
- A cancelled task.
- A family meeting scheduled for tomorrow.
- A Team standup scheduled for 11 AM today.
The current time was 4:30 PM.
Vee excluded the standup, even though its status was still todo.
At first, I questioned why.
Then it hit me. 😂
An Event isn't a Task.
A meeting scheduled for 11 AM didn't remain actionable at 4:30 PM just because its status was todo.
Vee was right.
And I learned something:
Remembering wasn't just retrieving records.
It required understanding my intention, the current reality, and which Item IDs actually mattered.
Another lesson learned from testing a 9B model.
1
34
1/3
Back to testing Vee's 9B brain.
Scenario 05: Multiple work-state changes.
This time, I reported FOUR changes in ONE message:
"Angel investor meeting was done.
Login bug was fixed.
Gym was cancelled.
Family meeting moved to tomorrow at 5 PM."
I expected Vee to find all four existing items and generate the correct update_item calls.
Result: 10/10 PASSED. 🤯
What surprised me was how consistently it distinguished completion, cancellation, and rescheduling.
FOUR different changes. FOUR correct operations.
All in ONE LLM invocation.
But what if I reported information that was actually conflicting? 👇
1
34
2/3
Scenario 06: Conflicting facts.
Progress Memory had my family meeting recorded for tomorrow at 5 PM.
Then I told Vee:
"My brother said it's at 3, but my sister said it's at 6. I haven't confirmed which one is right yet."
I expected Vee to recognize the conflict and ask for clarification WITHOUT changing the record.
First result: 3/10 PASSED. 😂
Here's what surprised me.
Vee sometimes decided that since the recorded time was now uncertain, it should REMOVE that time from Progress Memory.
And philosophically, I could see where it was coming from.
Was that actually a wrong decision, though?
Or was my evaluation too rigid?
So I refined the evaluation criteria to account for reasonable decisions that preserved the uncertainty rather than pretending it was resolved.
Another reminder that measuring an autonomous agent's reliability wasn't as straightforward as I expected.
1
29
3/3
Scenario 07: Outside Vee's responsibility.
After that unresolved family meeting discussion, I simply asked:
"How are you?"
That's it. 😂
I expected Vee to recognize that the message had NOTHING to do with witnessing or testifying.
No actions. No tools. No response.
But the first tests kept failing.
Vee would answer the greeting or bring up the unresolved family meeting AGAIN.
Even though I hadn't asked about it.
So I went back to the Facts section and defined what a MESSAGE was, and what it meant for something to fall outside Vee's responsibility.
Result: 10/10 PASSED. 🤯
Vee had learned to use conversational context.
Now it was learning when NOT to use it.
And with that, I'd finished testing the brain-dumping use cases.
Next up?
REMEMBERING use cases.
7
1/3
Back to testing Vee's little 9B brain.
Scenario 02: Missing information.
I told Vee about a family meeting happening today, but didn't specify the time.
I expected it to:
- Recognize an Event.
- Identify the missing start time.
- Ask for clarification.
- Generate ZERO tool calls.
Result: 10/10 PASSED. 🤯
But here's what surprised me.
I never gave Vee a specific rule in the prompt about requiring time for events.
It connected what an Event WAS, what the tool REQUIRED, and what information was MISSING.
And decided it couldn't record the meeting yet.
All in ONE invocation.
That was interesting. 👇
1
1
39
2/3
Scenario 03: Conversational continuity.
Remember Scenario 02?
Vee had asked me what time the family meeting was because I hadn't specified one.
So this time, I simply replied:
"7 PM"
That's it.
I expected Vee to:
- Understand that I was answering its previous question.
- Connect the time to the family meeting.
- Recognize that the missing information was now resolved.
- Generate ONE add_item call to record the Event at 7 PM.
Result: 10/10 PASSED. ✅
What surprised me was that Vee didn't treat "7 PM" as an isolated message.
It understood WHY it had asked for that information and what needed to happen once it received it.
All in ONE invocation.
Now, what if I mentioned NEW work alongside something already recorded in Progress Memory? 👇
1
28
3/3
Scenario 04: New work vs existing work.
This time, I told Vee about an Angel investor meeting today at 3 PM and mentioned the login bug again.
But the login bug was ALREADY in Progress Memory.
I expected Vee to:
- Recognize the investor meeting as a NEW Event.
- Schedule it for 3 PM.
- Recognize the existing login bug.
- Leave the login bug unchanged.
- And generate just ONE add_item call.
But the first tests kept failing.
Vee was also updating the login bug's due date to today.
At first, I thought it was making an incorrect assumption.
But then I realised something.
The way I phrased the message COULD actually mean I wanted to work on the login bug today.
So was Vee wrong? Or was my benchmark too rigid?
Man, measuring reliability for an autonomous domain agent was turning out to be a challenge of its own.
After refining the evaluation:
Result: 10/10 PASSED. ✅
Vee wasn't just deciding what needed recording.
It was also judging what was already known, what had changed, and what could reasonably be inferred.
And that was 4 out of 16 use cases tested.
12 more to go.
11






