AI coding assistants are some of the most impactful generative AI applications, where the return on investment can be more quickly seen by companies. Earlier this year, Amazon Web Services (AWS) introduced its own AI coding assistant, Amazon Q Developer. The tool has led to $260 million of annual savings at the company and is helping AWS clients become more efficient and productive while reducing costs.
Leading the software engineering team for Amazon Q Developer is Srini Iragavarapu, AWS’s director of generative AI applications and developer experiences. In this exclusive interview with The AI Innovator, Srini delves into the capabilities of Amazon Q Developer and gives us a peek into what’s coming next.
The following is an edited transcript of our conversation.
The AI Innovator: What is Amazon Q Developer and how much of a game-changer is it?
Srini Iragavarapu: Amazon Q Developer is the most capable generative AI assistant for software development that actually leverages company data, internal data for enterprises, if they choose to use it. The tool overall is something that we designed, and we are building continuously, iterating on top of it to help with all stages of the software development lifecycle, from designing your applications to coding your applications, testing your applications and deploying and troubleshooting as well. Amazon Q Developer is available to users in the IDE (Integrated Development Environment) while you’re coding, it’s available in the AWS console when you’re actually managing resources and deploying to AWS cloud compute as well.
The tasks that it performs is what really differentiates it (from other coding assistants). We launched the General Availability (GA) of Amazon Q Developer in April. Since then, we’ve been iterating and (recently launched) inline chat. … The idea is to keep every developer state within the IDE, not have to context-switch all the time, provide as much information as possible within the IDE, and then, as they start coding, as they start debugging, to help them – say, ‘I’m within my file, I’m typing my code and I would like to stay within the same file and start getting more comments and stuff like that.’
That’s when we realized inline chat is going to be a great user experience. With inline chat, developers can select a piece of code and ask for comments, and it will just magically add comments, docstrings, if you will. You can ask for unit tests. It will look at the function, and it will just add the unit test. We are using the latest Claude Sonnet 3.5 model on (Amazon’s gen AI platform) Bedrock.
Can Q make developers more efficient and more productive? Do you have any performance metrics you can share?
I’ll break this down into multiple chunks of metrics. … There are two agents in the Q Developer world: There is the transformation agent, and then there’s the software development agent. We used the Q transformation agent to migrate and upgrade old Java 8 and 11 versions to Java 17 and deploy them to production. Thirty thousand production applications have been migrated that were deployed.
(In the past), you would have assigned migration to a group of engineers. They would do the upgrades, test it and spend weeks in sprints of sorts, and then deploy and test it. (With Q), we were able to do this within hours. We did the math and realized we saved 4,500 years of work where the developers, instead of actually having to migrate code and upgrade the APIs, now they can do a lot more creative tasks.
Also, because we upgraded to later versions of Java, from both the performance standpoint and from the savings standpoint, we anticipated around $260 million of savings on a yearly basis. That is purely for the transformation agent.
Within Amazon, we are using it for production services. A bunch of other customers have used Q, and they’ve shared publicly the improvements that they’ve been seeing.
As an example, National Australia Bank is one of our customers that have been using Q Developer, and they reported up to 50% acceptance rate for inline code suggestions. British Telecom is another one. They have generated over 100,000 lines of code in the first four months of them adopting Q Developer, (automating 12% of tedious, time-consuming tasks). Other companies, like Persistent Systems, have also been using Q Developer for their tech debt reduction and productivity improvement (up 83%).
Internally, the Amazon Prime Video team has been using Q Developer, which has a customization feature. Enterprises might say, ‘Along with using Q Developer, I want my code also to be used to give me better recommendations.’ So an enterprise can choose to customize Q Developer through using their code base. The Prime Video team actually has been using a version of that, and they have seen a 30% or 40% higher acceptance rate than the generic version of Q Developer.
What other developments are coming in coding assistance?
We have a multistep reasoning agent where you, as a developer, can provide instructions to Q to complete complex tasks. One of the funny examples that somebody showed me was they told Q to write a program for the Hangman (word game) application and then write a bunch of tests too. It took me back to when I first learned about Hangman and the logic around Hangman. I still remember the code, because back then I was memorizing all of the code as well. In a couple of minutes, (Q) generated the entire function snippet, the code, the test cases and the data that was needed.
Another example comes from one of my own developers, who told me he was reading a PDF file on a Mac. He thought that the Mac PDF reader needed bookmarks, but he couldn’t find them. So he said, ‘Why don’t I code it up using Q Developer?’ He was able to provide instructions to Q Developer to create a PDF reader with bookmarks that can run on a Mac. In 15 minutes, he actually got a PDF reader running. …
That is what the agent can do; it reasons back and forth. Think of it as having a peer software engineer helping you through the journey. A few weeks ago, we updated the Amazon Q Developer agent. We are on the leaderboard for SWE-Bench, (which tests an AI model’s ability to understand, generate, and modify code effectively). … Since launch, we’ve improved it by 50%.
Can you describe your agentic AI hierarchy?
There are multiple layers, so I’ll kind of go backwards. There are models that we have through Bedrock, multiple models that we use depending on the task itself. For example, the transformation agent is one and the software agent is another. Then there is this middle layer, which I call the scaffolding layer. In fact, with the software agent refresh that happened a few weeks ago, we were able to convert code into a text format. … This format is a unique one, which actually helps the models to understand better.
The scaffolding I’ll split into probably three chunks. There is an actual reasoning agent that we have, which does all the processing, talking to the model and being able to rank and figure out which ones are better and then talks to the model. Then there is the responsible AI layer that we have where both inputs and outputs go through different guardrails.
So these two cluster models are how we differentiate ourselves, and the reasoning agent is a multistep exercise. Now, in the case of transformation, we also use open rewrite libraries, public libraries as well, to be able to help us with Java upgrades. So as you can tell, with the scaffolding, along with leveraging large language models, we are also using things like program analysis, automated reasoning in some cases, and that’s the third one.
… So there are the models, there is the text code conversion that I talked about, responsible AI layer, and then there’s the analysis and automated reasoning layer, which goes above and beyond what these models can do and how they interact with each other.
You’re making engineers more efficient. But are they worried that they’re going to be replaced?
I always go back to how I had to learn a little bit of punch card programming, and how I had to learn ASCII codes when I did graphics, when I essentially was building graphics on a very monochrome (monitor). I would actually have to figure out where pixels go. My focus was all on pixel positioning. But my focus was not on the art itself. I still remember for us to get a little logo to move on the screen required a lot of logic. So the focus was less on the art and more on the logic of movement.
But then came CGI, then came all the big front-end heavy applications – the anglers and the reacts and all of those things. When graphics came naturally, now the focus shifted to creativity. … The focus and the time spent went from ‘how do you put pixel positioning’ to the Avatar (movies) of the world. The movie happened because of this innovation. I see this as the same progression happening here too.
AI is going to do all the routine work, freeing engineers to be more creative. How you innovate within your team?
There are two parts to this: the team that is building all of this to innovate for customers, and the team that are developers innovating for themselves. I’ll first answer the first part. Because technology is moving really fast, and these models are evolving and becoming more powerful, (staying at the forefront means) being able to experiment internally. We experiment with a lot of features. Sometimes they work, sometimes they don’t work.
This is classic, going back to Amazon’s traditional customer focus. When I joined Amazon, there were meetings where they would leave a seat empty for customers, assuming that there is a customer sitting and telling me what to do. Now I realize how important customer feedback is. As we talk to customers, we get feedback fast and iterate. So being able to experiment, being able to understand what technology is coming, are the two pieces in conjunction with customer problems that we are solving. That’s how we focus on (what to develop next), whether it’s inline chat, the transformation agent, or software development agent.
Internally, my team uses web developer – 80% to 90% of them use it. They fix each other’s issues, give feedback to each other and iterate. That’s one that gets them all excited. … They are finding use cases in their workflow, and that is very important to us. We want to go to where the developers are, fit in their workflow, and help them improve.
In fact, there was one developer who said, ‘I’m drawing sequence diagrams using Amazon Q.’ … The way she did this is she has a JSON extension running in VS Code that draws sequence diagrams, and so she’s converting sequence diagrams to JSON and then using Q Developer to update the JSON and then convert it back into sequence diagram.’ I thought, ‘great! I’ve never thought of it. That is not something that we sell to anybody, or talk to anybody about.’ A developer, on their own, came up with this use case and started using it.
