Notebook

Defending Apex: hard, mostly good, and fair criticism

Notes from my MSc defence on 23 October 2026: what was hard, what went well, and what the jury was right to push on.

Read in Português

Contents

I defended my MSc dissertation, Governed Human-AI Collaboration Across the Software Development Lifecycle, at Instituto Superior Técnico on 23 October 2026, sixteen days after submitting it. I had twenty minutes to present six months of work: a framework for bounding what AI does at each phase of the lifecycle, and Apex, the working tool that implements it. The dissertation was graded 17/20.

It was harder than I expected and went well overall, and I agree with most of the criticism I got.

Why it was hard

Twenty minutes is not much for a framework with six phases, five quality gates, three governance metrics, a hat model and a reference implementation. Most of the preparation went into deciding what to leave out and walking through the phases one at a time, so the evaluation got less room than it deserved.

The moment I did not expect came from the jury’s president. I saw the practitioner and expert interviews as one of the stronger parts of the evaluation. The president saw semi-structured conversations, used too casually to count as a rigorous evaluation instrument. That was fair, and the evaluation should have been more rigorous.

What went well

The framework held up. Its central idea is that every phase states what AI may contribute, what it must produce, who is accountable and what gate closes it. That came through, and so did the honesty of the results. Apex scored below average on usability (SUS 56.73, grade D) and the thesis says so. I put those numbers on the slides, and I think showing them helped more than it hurt.

What the jury pushed on

The DSRM format

The dissertation follows Design Science Research Methodology: identify the problem, define objectives, design, demonstrate, evaluate, communicate. The jury’s president questioned how I applied that structure. In places the chapters follow DSRM’s activities more closely than the research needed, so the method shaped the document more than the decisions.

An evaluation that was too casual

Thirteen participants, one unmoderated session each, one seeded project, a sample drawn from my own network, and instruments that measure perception and say nothing about delivered quality. I listed all of these as threats to validity in the dissertation, but listing a limitation does not remove it. The evaluation describes how thirteen people experienced the tool; it does not show that the framework works for teams.

Consistency of roles

The framework assigns work to functions it calls hats. Some roles mentioned elsewhere in the text were missing from the main role table, so the model looked less closed than it is. I fixed this in the final version I submitted after the defence.

What I would do differently

  • Design the evaluation before the tool. The usability study was planned around a tool that already existed. Deciding first what would prove the framework wrong, and building toward that, would have made the study sharper.
  • Get a real team involved early. The partner-organisation demonstration never happened. One team that did not design the framework, using it on their own project, would have told me more than thirteen solo sessions.
  • Use the methodology lightly. DSRM is useful for thinking, but it should not dictate the table of contents.

The thesis also came up somewhere I did not expect: it was the main topic of my final job interview.

The case study, the dissertation PDF and the code at github.com/thomastabs/apex are public, weaknesses included.