AI can fix the symptom and miss the problem

As we have adapted to AI assisted development at Philo Software, we have been exploring how directly customers can help move their software forward. One customer wanted to take that further: they would drive feature requests and feedback, and our AI agent would work on the changes automatically. Our involvement would focus on technical assistance and security validation.

We both learned something useful from the experience. An AI can make the change someone asks for while missing the reason they asked for it.

A confusing screen can look like broken software

Some of the reported problems turned out to involve valid data presented poorly. The customer was right that the experience was confusing. But the explanation for what they were seeing was different from the one behind the requested fix.

There was not enough challenge in the process to establish that difference. The AI accepted the description of the problem and started building a solution. We ended up with changes intended to fix behaviour that was already correct, while the underlying confusion remained.

Each addition gave the customer more to interpret. Instead of making the application easier to understand, the fixes made the user experience more convoluted.

When the fixes become circular

We also saw feedback become circular. One issue would be addressed, another would appear, and the next change would try to accommodate that. Looking at each report separately made it easy to keep treating the latest symptom.

When we stepped back and investigated the system as a whole, we found an architectural issue. The underlying data model needed larger changes. The individual reports were pointing to a shared cause that the sequence of smaller fixes had not resolved.

Once we identified that cause and directed the AI to address it, the cascading issues went away. The useful change came from giving the AI a better diagnosis and a coherent direction for the system.

Asking for root causes is only part of the work

It is tempting to think this can be solved by telling the AI to investigate more thoroughly. That instruction helps, but in our experience it did not hold up on its own.

The customer naturally kept bringing the conversation back to what they could see and what they needed changed. They understood their workflow and the disruption they were experiencing. They did not have the technical context to judge why a change to the data model might be necessary.

The AI followed that direction back towards the visible symptom. A request to find the root cause could still end with another local workaround.

Customers should be able to describe a problem in terms of their experience. It is our responsibility to connect that experience to how the software actually works, explain what we find, and challenge a proposed solution when the evidence points elsewhere.

Where developers add value

For us, this clarified an important part of the developer's role in AI assisted development. We need to help the AI test the assumptions behind a request, trace the behaviour through the application and its data, and recognise when several reports share the same cause.

That includes distinguishing between incorrect data, correct data that is difficult to understand, and a data model that no longer represents the business properly. Those problems can look similar on a screen, but they need different responses.

A coherent data model gives the user experience a sound foundation. The screens can explain a consistent system, rather than accumulating exceptions to hide inconsistencies underneath.

The lesson we took from this experiment is that diagnosis needs to remain part of the development work. Customer knowledge, technical judgement and AI implementation are most useful when they work together. Building the requested change quickly only helps when we understand which problem it needs to solve.