PCG logo
Article

Amazon CodeGuru Reviewer: already time for retirement?

Introduction

Amazon CodeGuru ReviewerExternal Link is a relatively recent AWS service (2020) which promises to improve code quality and find security vulnerabilities. Just like its partner released around the same time, CodeGuru ProfilerExternal Link, CodeGuru Reviewer is based on Machine Learning models built using AWS’ many years of coding experience.

The recent release of a new service in the same series, CodeGuru SecurityExternal Link, brought the two older services back to my attention. I decided it was time to look into what they offered, and whether it would be worth including in the deployment services we offer to our clients. I was also wondering why it seems to have completely disappeared less than a year after its release, at least from most community blog posts, despite regular feature releases and a new language addition (Javascript).

This article focuses on exploring and demonstrating with real examples CodeGuru Reviewer features, as well as determining its cost and potential usefulness. While reading this article, you should quickly realize that the features are not that great in its current state, and why I think that this service might altogether disappear or be merged into, or replaced by CodeGuru security.

image-a86968dad56c

What is CodeGuru Reviewer?

CodeGuru Reviewer is a “static” analysis tool that will connect to Git repositories, and analyze the code of the main branch to figure out if there are critical or hard-to-find bugs or issues, which could have an impact on:

  • performance: deep loops, heavy logging, inefficient code…
  • maintainability: bad practices, redundant code…
  • concurrency: thread-safety, race conditions…
  • security: hard coded secrets, un-sanitized inputs…

The analysis is performed using a set of detectorsExternal Link, built based on machine learning, and classified by:

Once it finds such issues, it will highlight and describe them in a human-friendly way in one of the following places:

  • in the CodeGuru AWS console (always visible there for all analysis)
  • as comments in the Pull Request analyzed (for incremental analysis)
  • in the Github Security tab (when integrating via CI/CD)
image-0011607aa35e

Supported languages and repositories

CodeGuru Reviewer supports the following languages:

  • Java (8?)
  • Python (3.8?)
  • JavaScript (ECMAScript 2021?)

Note: the versions are educated guesses, based on the experiences of myself and others with the tool, since the docs don’t mention any language version (and doesn’t answer community related questionsExternal Link).

As for Git hosting services, it supports AWS CodeCommit, Bitbucket, GitHub and Amazon S3 (not Gitlab, cloud or on-premise).

How to use it?

CodeGuru Reviewer can be used in one of two ways:

  • directly as a reviewer on your repository
  • as part of your CI/CD pipeline

As a reviewer

Using that approach, the configuration is simple:

  1. configure CodeGuru as a reviewer in your repository
  2. let CodeGuru automatically run a full repository scan (which can also be triggered manually) on the main branch
  3. check the findings in the AWS console
image-12c2f2e65bdb
image-5b0bfeebbe98

Once CodeGuru has access to your repo, any Pull Request will trigger an incremental review, focused on the code changes you made. The findings will be shown as:

  • PR comments and
  • browsable items in the AWS console (just like full scans)
image-99310743bd95

As part of the CI/CD

As an alternative, CodeGuru Reviewer can run reviews triggered by Github ActionsExternal Link. The findings will then be available in the Github “Security” tab.

image-a68633ae7e97

You might want to use this approach if:

  • you want to control when PR reviews should happen (you prefer to avoid new comments at every commits)
  • you want to integrate CodeGuru as part of a pipeline

However this approach has a few drawbacks:

  • you can only use it on Github, or on other Github actions supporting pipelines (AWS recently made GH actions usable on CodeCommitExternal Link)
  • the code will be downloaded to an S3 bucket that you have to cleanup yourself, which can end up being pricey if the codebase is very large

Pricing

Now that we know what we can do with CodeGuru Reviewer, let’s see how much it would cost us to use the pricing described in the docsExternal Link.

The pricing model is based almost solely on the size of the codebase. The only other criteria is the number of full scan run, which shouldn’t be more than twice a month. The number of incremental scan (in PRs) does not affect the price.

Let’s take an example, and say that we have a project with:

  • 3 repositories containing each 150k lines of code in their largest branch
  • 3 full scans performed per month on 1 of the repositories instead of the free maximum of 2
image-02c9d6632ed7

The result is a monthly cost of $150. A good way to see if this price is expensive or not is to compare it with other static analysis tools offering similar code quality and security checks.

The most popular competitor among those is probably SonarQubeExternal Link, which offers for its cloud hosting option and 500k lines of code, a similar price of $150. It is worth noting that SonarQube offers a much larger set of features and languages supported, so the comparison only by price is not exactly fair.

Where it falls short

Documentation

The documentationExternal Link is not consistent and often out-of-date:

  • some partsExternal Link mention Javascript while the rest of the docs only mention Java and PythonExternal Link
  • no language version is mentioned anywhere
  • on the main page of CodeGuru, CodeGuru Reviewer is not listed directly anymore, suggesting changes in progress behind the scenes

Languages

Only 3 languages are supported, and not their latest versions. For Java this becomes quickly a blocker, since the supported version is only 8 (while the most recent LTS is 17), and unsupported code breaks the analysis.

Another point worth mentioning is that CodeGuru can only review one languageExternal Link at a time. If your project has multiple languages, you will not be able to select which you want to scan. It will be determined automatically by the number of files. From the documentation:

If a repository contains Java and Python files, then CodeGuru Reviewer generates recommendations for the language for which there are more files. […] If the number of Java and Python files is the same, then only Java recommendations are generated.

Findings

While the security findings can be pretty elaborate and helpful, the code quality and performance focused findings are not that impressive and can often be detected by more basic or powerful tools like SonarQubeExternal Link (paying) or PMDExternal Link (free). To see what I mean, you can have a look at the list of Java code quality detectorsExternal Link, which is pretty short and contains a lot of simple findings like:

There are also a few detectors capable of finding more elusive bugsExternal Link, but they are not many.

Conclusion

CodeGuru Reviewer offers several interesting features with useful recommendations, especially in the security domainExternal Link. The findings it provides for that area are not easy to identify with standard static analysis. The fact that it can integrate as a normal reviewer or via CI/CD is also convenient and makes it easy to quickly generate results.

However, when it comes to code quality, there are many alternative tools that provide similar insights, for more languages and at a lower cost, or even for free (if you are willing to plug and configure together multiple open source tools). Furthermore, the poor quality of the documentation in some areas (which is not common for most AWS services), and the recent documentation updates removing mentions of CodeGuru Reviewer, give the impression that this tool is not the focus of the AWS team.

The final hint that something will probably happen soon was the announcement of the CodeGuru Security serviceExternal Link, and the modification of the main CodeGuru pageExternal Link to point instead towards this new service. CodeGuru Security at first glance seems to be a modified version of the Reviewer, with a focus on security. This is pure speculation at this point, but I suspect that CodeGuru Reviewer will soon be either dropped or maybe merged into the newer and more focused CodeGuru Security.

So as a final advice, I would recommend not investing your time in CodeGuru Reviewer since its current state does not make it a very attractive tool, and even suggest that the time invested might be wasted if the tool is not maintained or discreetly disappears. However, I would keep an eye out for the future evolutions of CodeGuru SecurityExternal Link, which seem much more promising.


Services Used

Continue Reading

Article
Automation
Automated Control Rollout in AWS Control Tower

Control Tower Controls help you to set up guardrails making your environment more secure and helping you ensuring governance across all OUs and accounts.

Learn more
News
Above the Clouds: PCG's Stellar Performance at the AWS LeadMaster Challenge 2024

Wow, what a triumph! Public Cloud Group has just swept the AWS Summit 2024 Lead Master Challenge.

Learn more
Article
AWS Events 2025: The Future is Cloud

As a leading AWS Premier Partner, we're thrilled to present the exciting lineup of AWS events for 2025.

Learn more
Article
Protecting Lambda URLs with Cognito, IAM, Lambda@Edge and CDK

In this article, we’ll look at how to secure Lambda URLs using IAM access control. With complete code to try yourself!

Learn more
See all

Let's work together

United Kingdom
Arrow Down