Like most others, I was amazed in early 2023 when Microsoft released Bing Chat (now Copilot), whose unique selling point above ChatGPT was that it could search the web instead of being fully isolated. I remember watching this clip on The WAN Show where one of the hosts asks it to estimate how many of a certain backpack can fit inside the boot of a certain car. And, like them, I was amazed that the technology had gotten so advanced that we can now do this. It's funny to look back at clips like this from three years ago and see the wonder and amazement at the fact that answering this type of vague question was possible at all.
Three-and-a-half years later, and one can't help but shake the skepticism around AI, its accuracy, impact on business and society and so on, compared to those halcyon days of wonder and amazement. The term "AI slop" didn't exist back then. If the aforementioned clip were recorded today, the hosts might question the accuracy, or seek to independently confirm the dimensions of the backpack and/or the car's boot to verify these claims. This is because we've now all been burned one too many times by our token prediction engine of choice simply predicting things out of thin air and us having to play whack-a-mole to figure out what's fabricated and what's real.
Fast forward to earlier this year, where our organisation had procured Claude Code licenses for the engineering team. I'm not really a user of AI in normal life; the only exposure I get to it is Google's un-disable-able (is that a word?) "AI Overview", or when I notice something AI-slopped in the real world.
So when the chance came to use AI as part of my daily work, I was intrigued - as a non-user of LLMs - to approach it with an open mind and see how it could help me (or not) go about my daily work, and to document the journey.
Easy mode: repetitive tasks
Two years ago I wrote about our move to Java 17, for which I had tried to use AI tooling at the time using JetBrains AI Assistant (which I believe is now called Junie). This turned out to be a nightmare since I couldn't find a way to easily share context between projects (to run the same set of changes against multiple projects). Also it was doing a horrendous job of doing the actual upgrade because it was not possible to provide a feedback loop: only tell it to make some change and it would do it. So I swiftly gave up on that for Java 17.
Around the same time we acquired the Claude Code licenses, we were due for our next upgrade to Java 25. So what better way to get a feel for the progress of coding assistants than to try it with another Java upgrade!
One thing it was amazing for was doing highly repetitive tasks. I wrapped this in a Java 25 skill. The skill was something along the lines of:
-
Clone if project doesn't exist, stash all changes (if any), checkout development branch, do a pull
-
Change Jenkinsfile to build with a different JDK
-
Change pom.xml to refer to new parent and update project's major version
-
Absorb the Spring musical chairs of imports accompanied with a major Spring Boot upgrade (3 → 4), and absorb any removals and deprecations
-
Other stuff I added as I discovered as I went along
Claude Code was great for this. I pointed it at our graph of projects, and it understood the dependencies between them. Where there was parallelism to be had, it automatically engaged a kind of swarm mode (or whatever it's actually called) to build things in parallel. This was objectively much faster than what I could have possibly done myself, and the added understanding of what's actually going on (over a shell script that I'd have written myself) was a genuine increase in speed.
What took us two months last time now only took around two weeks. Admittedly, there were fewer major changes between Java 11 → 17 and 17 → 25, and Spring 2 → 3 and 3 → 4, but there's no way that we'd have been able to do it in two weeks without the aid of automated tooling that could fix basic problems by itself.
Verdict: coding assistants are a game changer for this type of work.
Intermediate mode: repetitive task with a spanner in the works
When things didn't go to plan, for example if there was some hitherto unseen incompatibility between Spring and some other dependency we had but needed updating, I found that it started going around in circles: making increasingly specific Google searches, using javap to disassemble a bunch of classfiles, and the one that I particularly enjoyed: requesting individual source files from GitHub repos instead of cloning the entire project and inspecting it locally. I quite enjoyed the last one because I would see it asking for permission to request some file from GitHub, then another, and another.
On quite a few occasions I had to ask it "what are you trying to do?" - sometimes with one or two unprofessional words interspersed. To which it would give me an extremely convoluted novella of what it is trying to do, when the actual fix was quite straightforward, like updating a dependency version or something.
I understand that this may be unfair because I have done this for a long time and so have seen lots of different failure modes, especially in our codebase. But I got the feeling that since this is a tool that predicts tokens based on a massive context of patterns that it has seen, it should have performed better at this task. I wasn't asking it to do any domain-specific things, just update some dependencies and resolve well-known types of errors relating to version incompatibility.
When it started tying itself into knots like this, I found that I was spending more time trying to get the tool to behave rather than if I were to fix the problem myself, and so I gave up on it for a while. I also tried to keep updating the skill definition with any repeating errors that I encountered
Verdict: terrible experience; it was more efficient to navigate the uncharted territory myself.
Nightmare mode: brownfield development (existing codebase, new code)
This is where I think great care has to be taken. I tried using it to add a new feature to an existing codebase. I think the use case is fairly common: the new feature references a preexisting shared library, and there is a similar feature in a different codebase which uses the same shared library which can be used as a kind of template.
So I gave all of this to the coding assistant as a prompt. The output was like something that a junior engineer would produce, with a twist; and that's not a disparaging comment towards junior engineers. It was of acceptable quality albeit a little bit shabby in places which would require tidying up manually (or prompting it to tidy up, depending on whether you think the tool will get it right). The twist, though, is that it was evident that it's trying to cut corners in some places.
I don't think a junior engineer would do this. Most junior engineers would be unsure of what to do next, and would ask a domain expert or the team lead what the next steps would be. A great junior engineer would know how to ask a good question, including what they've already tried, what they expect to happen, what is actually happening, etc. Instead the coding assistant was doing the classic LLM thing of just ploughing ahead and never stopping to check what it's actually doing.
Verdict: TDD would be a great approach here: write all the tests upfront (using AI or not, but preferably not) and let the tool iteratively do the implementation.
Conclusion
I can already hear the calls of "ok boomer" and "you're holding it wrong". I know the world has gone agentic, and I'm simply riding on all of your coattails. Yes, coding assistants like Claude have been a great help for elementary tasks, but I worry about the confidently incorrect behaviour it can exhibit at times. This can set a bad example to junior engineers and so I think it needs some caveats or guardrails need to be applied when using these tools.
It does make me wonder about the best way to teach junior engineers to engage with this new world of software development. I found success with TDD as I discussed earlier, but are junior engineers interested in spec-driven development like that? I guess I will have to find one and ask them. Creating interfaces and writing tests manually before asking the tool to do the implementation sounds to me like an ideal middle ground: the engineer is still responsible for defining the correct behaviour but can still accelerate their productivity.
Finally, I didn't get a chance to try it with any greenfield type projects, but I suspect it would do exceptionally well considering it's great at predicting the next token that fits best, and doing so without a preexisting context it needs to be aware of is probably right in an LLM's wheelhouse. I do, however, wonder about the maintainability of a project conceived this way as I suspect it would soon become spaghetti-like and not human understandable.
As for how the future looks, I think the AI boom will probably settle back into a phase of realistic adoption. As much as the initialisms "SOA", "IoT" or "5G" were bandied around when those respective technologies were at their peak of popularity, I suppose AI will soon follow that trend and we will look past it at the next shiny thing, whatever that may be.
