Writing Contract Tests with Pact in Spring Boot

This blog will explain how consumer and provider tests are written with Pact JVM in a Spring Boot environment.

Pact was initially written for Ruby but is now available for many different languages e.g. JavaScript, Python and Java. In this post we will use Java 8, Junit 4, Maven and Spring Boot because those were the primary technologies used at our client.

In Pact, the contract itself is called a pact. It is a JSON file that contains interactions. An interaction consists of the request and the expected response. The expected response contains the expected JSON but only the parts that are actually relevant for the consumer.

Example participants

We will use the following participants - or pacticipants as they are called in Pact - throughout this blog post series:The provider is a user service which has the endpoint GET /users/{userId} which returns a user’s data in the following form:

Code Copied!

{
  "id": "1",
  "legacyId": 123456,
  "name": "Beth Miller",
  "role": "ADMIN",
  "lastLogin": "2018-10-16T14:34:12.000Z",
  "friends": [
    {
      "id": 2,
      "name": "Ronald Smith "
    },
    {
      "id": 736,
      "name": "Matt Spencer"
    }
  ]
}

The consumer is a messaging app that shows the user’s name. It expects that a call to GET users/{userId} returns

• a 200 success code,
• content type JSON with UTF-8 encoding and
• a JSON body that contains the field name of type string

All the code can be found in this GitHub repository.

Consumer: Creating the contract

The easiest way to create the Pact file is via a unit test. The test goes in the same directory as all the other unit tests of the consumer.

The unit test will do two things: It verifies that our code can handle the expected provider responses and - as a nice side effect - it creates the Pact file.

One of the first questions is: Which code should be tested in order to verify that the expected provider response can be handled? A good starting point is the class that directly interacts with the provider. In our example, we have a UserServiceClient that provides a getUser method. This method calls the user service via a RestTemplate, parses the response into a User object and returns it:

java

Code Copied!

public class UserServiceClient {

  private final RestTemplate restTemplate;

  public UserServiceClient(@Value("${user-service.base-url}") String baseUrl) {
    this.restTemplate = new RestTemplateBuilder().rootUri(baseUrl).build();
  }

  public User getUser(String id) {
    return restTemplate.getForObject("/users/" + id, User.class);
  }

}

public class User {
  private String name;
  // Getter + setter if needed
}

The unit test will perform the following steps:

1. Start a server that mocks the provider with the given interactions.
2. Call the getUser method which will call the mocked provider.
3. Assert the returned User object.
4. Write the Pact file based on the given interactions.

The interactions will be defined in a separate method, annotated with @Pact. Steps 1. and 4. will be handled by the Pact framework’s PactProviderRuleMk2 Junit rule and steps 2. and 3. will be in a regular @Test method.

As a first step, we create the unit test class and add the test method:

Code Copied!

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE,
  properties = "user-service.base-url:http://localhost:8080",
  classes = UserServiceClient.class)
public class UserServiceContractTest {

  @Autowired
  private UserServiceClient userServiceClient;

  @Test
  public void userExists() {
    User user = userServiceClient.getUser("1");
    assertThat(user.getName()).isEqualTo("user name for CDC");
  }

}

Note: One of the most challenging parts is which user ID to use. For now, we just assume that a user with ID 1 exists in the provider. We will come back to this question later on.

As a next step, we will define the interactions by creating a method annotated with @Pact and the consumer name. The method returns the description of the contract using the pact-jvm Lambda DSL.

Code Copied!

@Pact(consumer = "messaging-app")
public RequestResponsePact pactUserExists(PactDslWithProvider builder) {
  return builder.given("User 1 exists")
    .uponReceiving("A request to /users/1")
    .path("/users/1")
    .method("GET")
    .willRespondWith()
    .status(200)
    .body(LambdaDsl.newJsonBody((o) -> o
      .stringType("name", “user name for CDC”)
     ).build())
    .toPact();
}

Note: The meaning of given(…) and uponReceiving(…) was quite confusing for us. uponReceiving(…) is just the description of the contract. given(…) can be used to prepare the provider i.e. bring it into a certain state. More on this later.

Note: It is very important to use stringType instead of stringValue even though stringValue is used in the pact-jvm documentation. stringType generates a matcher that just checks the type, whereas stringValue puts the concrete value into the contract. It might be tempting to use the real user name since we know what it is. However, this again leads to a tight coupling between the consumer and provider. (If the name changes, the tests will fail.) Note that user name for CDC is just an example value that is returned by the mocked server. It’s not strictly necessary to set it. We just use it to assert that the client parses the response correctly.

The last step is setting up the mock server. This is done by adding the PactProviderRuleMk2 rule to the class and annotating the test method with PactVerification and the name of the previous method. This annotation is important because it tells the Pact provider rule to start the mock server with the interaction defined in the given method (which is called fragment here for some reason).

Note: The rule was quite confusing for us as well. It does many things and has a weird name. Most importantly, it starts a mock server that will return the expected response and writes the Pact JSON file at the end.

The complete class now looks like this:

Code Copied!

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.NONE,
  properties = "user-service.base-url:http://localhost:8080",
  classes = UserServiceClient.class)
public class UserServiceContractTest {

  @Rule
  public PactProviderRuleMk2 provider = new PactProviderRuleMk2("user-service", null,     
    8080, this);

  @Autowired
  private UserServiceClient userServiceClient;

  @Pact(consumer = "messaging-app")
  public RequestResponsePact pactUserExists(PactDslWithProvider builder) {
    return builder.given("User 1 exists")
      .uponReceiving("A request to /users/1")
      .path("/users/1")
      .method("GET")
      .willRespondWith()
      .status(200)
      .body(LambdaDsl.newJsonBody((o) -> o
        .stringType("name", “user name for CDC”)
       ).build())
      .toPact();
  }

  @PactVerification(fragment = "pactUserExists")
  @Test
  public void userExists() {
    final User user = userServiceClient.getUser("1");
    assertThat(user.getName()).isEqualTo("user name for CDC");
  }

}

Note: In our real setup, we obviously don’t use port 8080 for the mocked server because we want to run tests in a CI environment where this port might be in use. Thus, we created our own JUnit rule that finds an open port and stores it into an environment variable so that it can be used in the Spring properties.

Now if we run the test it will generate the following file <consumer-name>-<provider-name>.json in the folder target/pacts:

Code Copied!

{
    "provider": {
        "name": "user-service"
    },
    "consumer": {
        "name": "messaging-app"
    },
    "interactions": [
        {
            "description": "A request to /users/1",
            "request": {
                "method": "GET",
                "path": "/users/1"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json; charset=UTF-8"
                },
                "body": {
                    "name": "user name for CDC"
                },
                "matchingRules": {
                    "body": {
                        "$.name": {
                            "matchers": [
                                {
                                    "match": "type"
                                }
                            ],
                            "combine": "AND"
                        }
                    },
                    "header": {
                        "Content-Type": {
                            "matchers": [
                                {
                                    "match": "regex",
                                    "regex": "application/json;\\s?charset=(utf|UTF)-8"
                                }
                            ],
                            "combine": "AND"
                        }
                    }
                }
            },
            "providerStates": [
                {
                    "name": "User 1 exists"
                }
            ]
        }
    ],
    "metadata": {
        "pactSpecification": {
            "version": "3.0.0"
        },
        "pact-jvm": {
            "version": "3.5.24"
        }
    }
}

The metadata section and the Content-Type are added automatically by the Pact framework.

It’s easy to extend the contract by including more attributes in the unit test:

• lastLogin in a specific date format
• role has to be either ADMIN or USER (we will come back later to the question if this is a good idea)
• friends has to be an array with a minimum size of 0. The array is expected to contain object entries with string attributes ID and name. Note that the 2 is not used in the contract. It just tells the mocked server how many array elements to return to the mocked response.

Code Copied!

LambdaDsl.newJsonBody((o) -> o
 .stringType("name", "user name for CDC")
 .timestamp("lastLogin", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", new Date(1539693252000L))
 .stringMatcher("role", "ADMIN|USER", "ADMIN")
 .minArrayLike("friends", 0, 2, friend -> friend
   .stringType("id", "2")
   .stringType("name", "a friend")
 )).build();

This will result in a Pact file with rules for all the given fields. The above example shows only some of the available methods to specify the expected body. More methods can be found in the pact-jvm documentation.

Now we can move on to verifying this test on the provider side.

Provider: Verifying the contracts

We will now show how to create the tests on the provider site that verify that the contracts are fulfilled. We use Spring Boot integration tests for this because it allows us to

• easily mock away any downstream systems we don’t want to depend on e.g. a database and
• to run the tests as part of our normal build.

The controller for GET /Users/{userId} looks like this:

Code Copied!

@RestController
public class UserController {

  private final UserService userService;

  public UserController(UserService userService) {
    this.userService = userService;
  }

  @GetMapping("/users/{userId}")
  public User getUser(@PathVariable String userId) {
    return userService.findUser(userId);
  }

}

The userService just returns a dummy user for now but usually it loads the user from somewhere e.g. a database.

Code Copied!

@Service
public class UserService {

  public User findUser(String userId) {
    return User.builder()
      .id(userId)
      .legacyId(UUID.randomUUID().toString())
      .name("Beth")
      .role(UserRole.ADMIN)
      .lastLogin(new Date())
      .friend(Friend.builder().id("2").name("Ronald Smith").build())
      .friend(Friend.builder().id("3").name("Matt Spencer").build())
      .build();
  }
}

For the test, we create a regular Spring Boot web integration test and use the SpringRestPactRunner Junit runner. Additionally, we need to add the following annotations to the class:

• Which Pact files to load by specifying the @Provider annotation with the provider name.
• Where to load the Pact files from by specifying one of @PactBroker or @PactFolder. We will use @PactFolder for now to load the files from the file system because it’s the easiest way to get started. Thus, we create a pacts directory and copy the Pact file created by the consumer to it. (We will show how the Pact Broker can be used instead in the next part of this blog post series.)

Inside the class, we specify:

• The target: Where to run the interactions against and verify the responses. The SpringBootHttpTarget is for the Spring Boot integration tests. The tests are executed against the application started by the integration test on the random port. There are other targets e.g. MockMvcTarget which we have successfully used in a plain spring application where we run the test with just the controller.
• A method for each provider state given in the contract. The method can be used to set up the desired provider state, e.g. creating the user in the database or mocking the service provides the user.

Code Copied!

@RunWith(SpringRestPactRunner.class)
@Provider("user-service")
@PactFolder("pacts")
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class ContractTest {

  @TestTarget
  public final Target target = new SpringBootHttpTarget();

  @State("User 1 exists")
  public void user1Exists() {
    // nothing to do, real service is used
  }

}

If we run the test now, it passes and outputs the following:

Working with the contract

In this section, we will go through several scenarios that can happen in the lifecycle of an API and show how Pact will behave.

Provider adds a field to the API

The provider adds a new field to the API. Nothing will happen, everything will still pass because the consumer only cares about the attributes in its contract.

Provider deletes a field from the API

Removal of an unused field: The provider decides to remove the legacyId field from the API. They can just do it because the consumer does not expect it to be part of the response. The contract test still passes.

Removal of a used field: The provider decides to rename the name field to fullname (which is the same as removing the name field from the contract’s perspective):

The provider test fails because the consumer’s contract is violated since it expects the name field.

Provider changes date format

The provider changes the date format from an ISO-8601 date to a timestamp. (This actually happens in reverse when [upgrading from Spring Boot 1 to 2].)

The provider test fails because the consumer’s contract is violated since it expects a different format.

Provider introduces new enumAnother consumer requests that the provider adds a new user role SUPER_ADMIN. The provider adds it and the user with ID 1 (used in the consumer test) gets this new role assigned. The test now fails with:

Whether this is good or bad depends on the consumer. If the consumer cannot handle anything else except USER and ADMIN it’s good that the provider is now aware. On the other hand, it somewhat happened accidentally that the provider even noticed. If the user with ID 1 had kept its role, the test would have passed. In general, it’s the same as with fields: the consumer should be able to handle additional information. If the consumer really needs to distinguish between USER and ADMIN, it should have created two interactions with two different states, one for ADMIN role and one for USER role.

Consumer breaks own implementation

Someone new in the consumer team wants to rename name to fullname. They refactor the class accordingly. This assertion in the unit test will now fail:

Code Copied!

assertThat(user.getFullname()).isEqualTo("user name for CDC");

Because the response from the mocked server does not contain the fullname field (just the name field).

Consumer adds a new interaction (and new provider state)

The consumer wants to ensure that a 404 is returned if the user does not exist. A new Pact definition is added to the consumer unit test.

Code Copied!

 @Pact(consumer = "messaging-app")
  public RequestResponsePact pactUserDoesNotExist(PactDslWithProvider builder) {

    return builder.given("User 2 does not exist")
      .uponReceiving("A request to /users/2")
      .path("/users/2")
      .method("GET")
      .willRespondWith()
      .status(404)
      .toPact();
    }

  @PactVerification(fragment = "pactUserDoesNotExist")
  @Test
  public void userDoesNotExist() {
    expandException.expect(HttpClientErrorException.class);
    expandException.expectMessage("404 Not Found");

    userServiceClient.getUser("2");
  }

After running the test the generated Pact file contains two interactions:

The provider test will now execute and verify both interactions against the Spring Boot application sequentially. The test for the new interaction fails because the requested provider state does not exist. But only adding the state will still result in a failure because we did not yet add any real state setup - we always return user with ID 1.

This is a real drawback of using the Spring Boot integration tests. There is still a dependency between the consumer and the provider: Whenever the consumer needs a new state, the provider code needs to be updated first.A good solution is to provide more generic provider states that accept parameters e.g. in this case we can provide a parameter userExists. If it’s set to true, the user service (or the underlying source) will be prepared/mocked so that a user is always returned. If it’s set to false, the user service will be set up in such a way that the controller will return a 404.

The first step is to define a generic provider state instead of concrete states in the provider test:

Code Copied!

@State("default")
public void toDefaultState(Map<String, Object> params) {
  final boolean userExists = (boolean) params.get("userExists");
    if (userExists) {
    // set up user service to return a user
  } else {
    // set up user service to return no user
  }

The consumer tests can now be rewritten to use the new state and pass true respectively false:

Code Copied!

@Pact(consumer = "messaging-app")
public RequestResponsePact pactUserExists(PactDslWithProvider builder) {
  return builder.given("default", Collections.singletonMap("userExists", true))
    .uponReceiving("A request for an existing user")
    ...
}

and

Code Copied!

@Pact(consumer = "messaging-app")
public RequestResponsePact pactUserDoesNotExist(PactDslWithProvider builder) {
  return builder.given("default", Collections.singletonMap("userExists", false))
    .uponReceiving("A request for a non-existing user")
    ...
}

Consumer needs a new field

The consumer would like to have a new field nickname in the response. They change their unit test, generate the new Pact file and pass it to the provider. The provider test will fail until they add the new field. We will show in the next part of this blog post series what a workflow for changes can look like and how it can be integrated into the build pipeline.

Lessons

In the previous sections we showed how to create tests for the consumer and provider and how they help in detecting breaking changes in an API. However, the more you use those tests the more you realize that the devil is in the detail. Here are three of our most important lessons:

Mock as little as possible on the provider site

Have you noticed how we skimmed over the user creation in the previous section? It turns out that it’s actually not that obvious where those users come from. In one provider we initially mocked the service layer and returned a mocked user. However, this mocked user always had all possible fields set. So the contract tests kept passing even when one of the fields that the consumer expected was never set by the real service anymore (because the users in the database never had that field anymore).

That’s why we later chose to mock as far back as possible - the database layer (using the real database was not possible for various reasons).

Be strict on the consumer site

We have a consumer in a legacy system where the communication with the provider is similar to the UserServiceClient in the example above - a Rest Template that just parses the response.We have had several discussions about whether it’s sufficient to use this class in the contract test or not. Why? If the provider stops sending a field, the class just parses it as null value and throws no error. However, a lot of the fields are mandatory and the service layer throws a runtime exception if they are missing.

Since the service layer is not part of the contract test, these exceptions are never detected. In the end we agreed that it would be better to include some parts of the service layer in the contract test.For a new consumer we wrote the client in a way that it additionally validates the response. In this way exceptions are thrown by the client itself and the service layer can rely on mandatory fields being present.

Echo input back instead of testing functionality

We only covered a simple GET request so far where the consumer only cared about the endpoint name and the response object. It’s not that straightforward with non-simple GET operations.Let’s say we have an endpoint which returns a list with a limited number of users:GET /users?limit={limit}The consumer wants to ensure that the provider accepts the limit parameter. It’s not sufficient to just write a contract that passes the parameter and expects a successful response. Why?

Because if the provider removes the parameter from the controller, it will still return a successful response. Spring Boot by default just ignores any unknown query parameters.Of course the consumer could just write a test that passing a limit of 5 will result in 5 users being returned. But the consumer actually does not want to test the internal logic of the provider. This should be covered by the provider’s own functional tests. You could argue that it’s no big deal to ensure that exactly 5 users are returned.

But image a sorting parameter instead with a complicated algorithm behind it - do you really want to re-implement that sorting logic in every single consumer? The approach recommended by Pact is to simply echo the query parameters back to the consumer and only fall back to functional testing if that’s really not possible.For example the provider could respond like this for the request GET/users?limit=5&foo=bar:

Code Copied!

{
  “query”: {
    “limit”: 5
  },
  “users:” [
   // users
  ]
}

The unknown query parameter foo is not echoed back.The same approach can be used for methods that pass a payload: return the parsed payload and - only if that’s not possible - check the desired outcome.

Conclusion

Setting up Pact and writing the initial tests is very easy. Solving all the nitty-gritty details so that everyone really trusts those tests is difficult. But once you have, you can replace all those awful, brittle, end-to-end tests and never look back.

In the next part of this blog post series we will show how contract verification can be integrated into the build pipeline so that any breaking change is automatically detected before it gets deployed.