Mandatory 3: Microservice Tests. NoSQL

Learning objective

Using TestContainers to make integration/consumer-driven tests. Deploying NoSQL database (Redis) instance(s) and using it as storage tier for SkyCave.

Deadline

December 4th at 23.59

Exercise 'consumer-driven-test-hello-spark'

This is a warm-up exercise - most/all of the code is provided below. And is a good foundation for the later CDT exercises!

Create a Consumer-Driven Test/Contract Test using TestContainers to validate your 'hello-spark' image from the previous 'docker-hello-spark' exercise. (Or, if you did not finish it, you may pull 'henrikbaerbak/hellospark'.)

To help you out, here is a working 'build.gradle', that retrieves the libraries for TestContainers and the Unirest client HTTP library (you may of course substitute any other HTTP library that you may prefer):

plugins {
    id 'java'
}
repositories {
    mavenCentral()
}

dependencies {
    testImplementation group: 'com.konghq', name: 'unirest-java',
      version: '3.14.5'

    // Need JUnit
    testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'
    testImplementation group: 'org.hamcrest',
            name: 'hamcrest', version: '2.2'

    // Need TestContainers
    testImplementation "org.testcontainers:junit-jupiter:1.19.1"
    testImplementation 'org.testcontainers:testcontainers:1.19.1'
}

tasks.named('test') {
    // Use JUnit Platform for unit tests.
    useJUnitPlatform()
}

And a almost-complete template for the JUnit code in 'src/test/java/example' is

package example;

import kong.unirest.HttpResponse;
import kong.unirest.Unirest;
import org.junit.jupiter.api.*;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.MatcherAssert.assertThat;

import org.testcontainers.containers.GenericContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

@Testcontainers
public class TestHelloSpark {

  public static final int SERVER_PORT = 4567;
  @Container
  public GenericContainer helloSpark =
          new GenericContainer(DockerImageName.parse("(your image here)"))
                  .withExposedPorts(SERVER_PORT);
  private String serverRootUrl;
        
  @BeforeEach
  public void setup()
  {
    String address = helloSpark.getHost();
    Integer port = helloSpark.getMappedPort(SERVER_PORT);
    serverRootUrl = "http://" + address + ":" + port + "/hello/";
  }
        
  @Test
  public void shouldGETonPathHello() {
    // change to sharp brackets
    HttpResponse(String) reply =
            Unirest.get(serverRootUrl + "Henrik").asString();
    System.out.println("** ROOT: " + reply.getBody());
    assertThat(reply.getStatus(), is(200));

    assertThat(reply.getBody(),
            containsString("Hello to you Henrik"));
  }
}

Or - download this zip: consumer-driven-test-hello-spark.zip .

Exercise 'cdt-quote-service' [M 40]

The official quote service stems from the docker hub image

 
henrikbaerbak/quote:msdo_2_3

In this exercise you should make Consumer Driven Tests (CDT)/Contract tests for the quote service - based upon the REST API described earlier in exercise 'quote-service'.

Requirements:

Create CDT/Contract tests using your choosen HTTP/REST client library and TestContainers, that validate the REST API of the quote service. Ensure you cover all endpoints as well as all potential return values (read: all the HTTP status codes).
Ensure that the test case lies in the 'integration' subproject of your SkyCave project.
(Of course!) All your out-of-process tests pass, when running 'gradle itest'.

Update 21/11: Ensure that the failing test case in 'TestCaveStorage.java' is silenced (or fixed). I will assess your handin by A) pulling your skycave-image and extract the code, B) run your integrations test and review them. And all tests must pass.

Hand-in:

Provide the FULL PATH of your CDT (ala 'cave/integration/src/test/.../MyFullCDTForQuoteService.java')

Evaluation:

I will review your code from your image; and I will run your tests in the 'integration' subproject. I will use the grade sheet to evaluate your submission.

Learning Goal	Assessment parameters
Submission	Required artifacts are all present.
Test Type	The test code is CDT code, not integration test code (exercise the REST API, not the QuoteService implementation).
Test Code	The test code is simple (no complex constructs, basically only assignments, simple private method calls, very basic for loop). The test code reflects using the TDD principles (Isolated Test, Evident Test, Evident Data, etc.). Robert Martin 'Clean Code' properties are generally kept.
Test Completeness	All central aspects of the API are well covered by tests. Irrelevant test aspects are not present.
Functionality	The tests execute correctly, all tests pass.

Exercise 'integration-quote-service' [M 40]

In this exercise, you should make Integration Tests (in the Fowler sense) (or Connector tests in the Bærbak sense) of your QuoteService implementation, that is the connector, developed earlier in the 'quote-service' exercise.

Requirements:

Create Integration Tests using TestContainers, that validate your implementation of your QuoteService connector that contacts a real quote service. Again, ensure you cover all possible return values.
The quote service must be started from the 'henrikbaerbak/quote:msdo_2_3' image in the TestContainer JUnit code itself, not by contacting the production server at 'quote.baerbak.com'.

Hand-in:

Provide the FULL PATH of your integration test (ala 'cave/integration/src/test/.../MyConnectorTestForQuoteService.java).

Evaluation:

I will review your code on the Crunch machine; and I will run your tests in the 'integration' subproject. I will use the grade sheet to evaluate your submission.

Learning Goal	Assessment parameters
Submission	Required artifacts are all present.
Test Type	The test code is integration test code (it exercises the 'RealQuoteService' implementation), not CDT test code (it is not exercising raw REST calls) nor service tests (testing indirectly through exercising the PlayerServant code).
Test Code	The test code is simple (no complex constructs, basically only assignments, simple private method calls, very basic for loop). The test code reflects using the TDD principles (Isolated Test, Evident Test, Evident Data, etc.). Robert Martin 'Clean Code' properties are generally kept.
Test Completeness	All central aspects of the integration are well covered by tests. Irrelevant test aspects are not present.
Functionality	The tests execute correctly, all tests pass.

Exercise 'no-cdt-for-redis'

Argue why it does not make sense to create Consumer-Driven Tests for the Redis database.

Exercise 'redis-datatype-model'

In this exercise, the learning focus is on using the Redis shell, and explore the key-value paradigm of a NoSQL db.

In mandatory exercise 'integration-redis-connector', you will develop a Redis 'CaveStorage' connector/driver, and in order to do that, you of course have to reflect upon which Redis datatypes (and potentially 'secondary indices') that suits the domain - just as Entity-Relation models are deveveloped for 'good old SQL'.

Requirements:

Briefly study the datatypes provided by Redis: Keys, Sets, Sorted Sets, Hashes, ...
Study the 'CaveStorage' interface, and propose datastructures model handling its data. Note that CaveStorage actually covers diverse 'domains' like "players", "rooms", and "wall postings" which are rather disjoint and can be considered in isolation.
Consider some of the 'not straight forward key-value' queries/commands like
1. computeListOfPlayersAt()
2. addMessage() + updateMessage()
What kind of datatypes are involved? Do you need secondary indices? How are primary and secondary datastructures linked (the 'foreign key' so to speak)?

Select one or two of your proposed datatypes, and hand-craft some contents and try out the 'redis-cli' in a running Redis. Ala:

csdev@m1:~/proj/cave$ docker run -d -p 6379:6379 --name redis redis:7.2.1-alpine
5a2802c17c21cd550b068b11f6071c64f669fb31c0ca0cd215c1be190b3e99ed
csdev@m1:~/proj/cave$ docker exec -ti redis redis-cli

and

127.0.0.1:6379> set room(0,0,0) "You are standing"
OK
127.0.0.1:6379> get room(0,0,0)
"You are standing"
127.0.0.1:6379> set room(0,1,0) "You are in an open forest"
OK

Try another datatype than the first one and redo some of the queries. Ala

127.0.0.1:6379> hmset room(0,0,0) description "You are standing" creatorId 0
OK
127.0.0.1:6379> hmset room(0,1,0) description "You are in an open forest" creatorId 27
OK
127.0.0.1:6379> hget room(0,1,0) description
"You are in an open forest"

Study the 'Transaction' concept in Redis and argue to your fellow students why it is essential that you use that in certain contexts when solving the later mandatory exercises.

Exercise 'architectural-prototyping-redis-connector'

In my 'Software Architecture in Practice' course, I teach about Architectural Prototyping: Small codebases that explore/experiment with an architectural issue or an architectural tradeoff.

Prototyping work is often too cumbersome in the original codebase context, therefore often a minimal codebase is harvested and used for quick experiments.

This exercise is basically a warm up to the 'integration-redis-connector' exercise later; and shows some of the setup you need.

The code base uses 'Jedis' as java driver, see some examples at How to use Redis in Java using Jedis. Beware, though, that the Jedis version in that tutorial is pretty old (2.4.2 versus currently 5.0)

Exercise: Implement (in partial) a Redis backed CaveStorage implementation using TestContainers as tool for an Integration Test (in the Fowler sense) suite.

You will find the initial steps for a solution in the gradle project: ap-redis-connector.zip

Hint:

JavaTPoint's description of Hashes states that "they are the perfect data type to represent objects." You can see that I instead use Gson to marshall/demarshall to JSON. You are of course free to pick what you find most readable/easy to code.
JUnit tests for the FakeCaveStorage interface already exists ('server' project, package: cloud.cave.service.TestCaveStorage), so it makes good sense to reuse them in a TDD and 'small-steps' fashion: Copy them one by one to your integration tests, ensuring your implementation makes it pass, before proceeding to "copying" the next - essentially growing your RedisCaveStorageConnector implementation.
Getting the connection code right is another story. Find inspiration at Jedis tutorial full version.

Exercise 'integration-redis-connector' (*) [M 50]

Presently, the CaveStorage is an Fake Object test double, and our SkyCave cannot handle restarts at all. We need a real storage tier for production, of course. Therefore, implement the CaveStorage interface so it interfaces a running Redis database. That is, make a Redis connector implementation.

In this exercise you should make Integration Tests (in the Fowler sense / Connector Tests in Bærbak sense) of your CaveStorage implementation that connects to a real Redis container, and use these tests to make a (test-driven?) implementation.

I advise to 'take small steps' by considering solving the 'architectural-prototyping-redis-connector' exercise first! This exercise then more-or-less only deals with the integration into SkyCave aspect.

Requirements:

Create Integration Tests using TestContainers, that validate your implementation of the CaveStorage interface which contacts an external Redis. (Crunch uses image 'redis:7.2.1-alpine', so it is a bit safer to use that.)
It is are not required to implement the Wall behaviour! That is, you can get full points for this exercise, while just implementing the three '(add/update/get)Message()' methods as in-memory storage. (Simply copy from the FakeCaveStorage; these methods may be overridden in a later exercise).
Your initialization of the cave storage must create the same four rooms in the cave with the same layout as in the fake storage, in case the bound Redis does NOT already have any contents! I.e. no separate bootstrapping script.
Your tests must reside in the 'integration' gradle subproject (run by gradle itest), as they are out-of-process tests that are slow to execute.

Hand-in:

A small text outlining the datastructure(s) used to implement a correct computeListOfPlayersAt()
Provide the FULL PATH of your integration test (ala 'cave/integration/src/test/.../TestRedisCaveStorage.java)

Evaluation:

I will review your code on the Crunch machine; and I will run your tests in the 'integration' subproject. I will use the grade sheet to evaluate your submission.

Learning Goal	Assessment parameters
Submission	Required artifacts are all present.
Test Type	The test code is integration (connector) test code, not CDT or Service test code.
Test Code	The test code is simple (no complex constructs, basically only assignments, simple private method calls, very basic for loop). The test code reflects using the TDD principles (Isolated Test, Evident Test, Evident Data, etc.). Robert Martin 'Clean Code' properties are generally kept.
Test Completeness	All central aspects of the integration are well covered by tests. Irrelevant test aspects are not present. Wall message methods are optional to include.
Production Code	The CaveStorage implementation that connects Redis is 'Clean Code' and handles connections correctly. You may ignore handling JedisExceptions for now.
Functionality	The tests execute correctly, all tests pass.

Hints and issues:

You need to update your build.gradle to retrieve the a Jedis Java driver - find the driver on 'mvnrepository.com' (or in the AP code base).
Remember Take Small Steps! Learning Redis, learning the Jedis Java API, learning TestContainers, and integrating in SkyCave in a single step equal stepping into an abyss of agony and with 'big ball of mud' code as result!!! What is the smallest step you can make? What separate learning tests can you make and later integrate? Solve the architectural prototype version of the exercise first. Keep detailed records - may be they are useful at the exam!
Even more hint at Take small steps: The TestCaveStorage test cases of the FakeCaveStorage actually contains all the test code you need. As in the previous exercise, copy them one by one, and make them drive your real Redis code. And the FakeCaveStorage actually contains a lot of code you can copy directly, like the initialization of the initial four rooms, computing room exit lists, etc.
We do not yet 'design for failure' which will make your code easier to write, as JedisExceptions just propagate up the call stack. However, note that this is actually wrong, but permitted for this exercise! Why wrong? Because allowing them to propagate does not respect the role boundaries: What is CaveStorage responsible for handling and what is PlayerServant responsible for handling? If JedisExceptions travel up the chain then PlayerServant and the Invoker code suddenly gets tied up to Jedis specific implementation details. And they should not, they should operate at the level of the CaveException. Morale: Once we start designing for failure, all methods must catch Jedis exceptions and rethrow them as CaveExceptions. We will return to it in the next course...
This said... If you just create a 'jedis' instance in the 'initialize()' and close it in the 'disconnect()' method, then your code is extremely fragile to temporary disconnects. It is better is 'try (Jedis jedis = pool.getResource())' for each CaveStorage method call.
There are quite a few cool ORM frameworks out there which can 'autogenerate' the storage layout for you. Do not to use them! The learning goal is the Redis API, not integrating Hibernate to avoid doing raw datastructure manipulations. Second (and not least) it will make solving a potential Redis based exam exercises harder!

Exercise 'redis-storage-journey' (*) [A 80]

Crunch will execute a couple of test journeys on your SkyCave connected to a Redis storage, validating that it works. Wall behaviour will not be tested. Crunch will test against a Redis v 7.2.1.

Requirements:

Add a CPF file in the server subproject, redis-storage-journey.cpf, defining the following configuration:

Subscription: Real CaveStorage: **Redis** Quote: Real PlayerNameService: InMemory
Wall behaviour is not required to be in functional, it will be tested in the next (non-mandatory) exercise.
Be sure that your production code uses hostname and port of the Redis container as defined in the CPF file (Crunch will overwrite it and bind your server to its own container on a local network).

Hand-in By Crunch.

Evaluation: Crunch will bind your daemon to its own, empty, redis database through overriding the SKYCAVE_CAVESTORAGE_SERVER_ADDRESS value in the CPF. It will start your daemon, go to room (0,0,0) and validate that the cave has its initial configuration. Then Crunch will dig some random rooms, restart the daemon, log in a random new user, and make certain these rooms are present. It will also try to update a room's description, both as creator and non-creator.

Hints and issues:

In production, you must ensure that your DB is not wiped clean after each restart! Review the Docker Hub description for the redis image, it contains valuable information regarding how to keep database files on the host.

Exercise 'redis-storage-journey-wall' [A 60]

Crunch will execute wall behaviour test journeys.

Requirements:

Wall behaviour is functional.

Hand-in By Crunch. Crunch will start your daemon using 'redis-storage-journey.cpf'.

Evaluation: Crunch will create 11 postings in one room and 2 in another, and test the pagination and ordering of wall postings. It will update a message, both as author and as non-author, and validate that sematics is as expected.

Exercise 'operations-redis' [M 30]

Production Checkpoint! Update your production environment, so your production server in the cloud uses a Redis as persistent storage.

Requirements:

None. I may run your Cmd using the 'operations.cpf' to connect to your production server.

Hand-in: Submit screenshot(s) with shells on your production server that shows

The IP address of your server ('ifconfig eth0' or 'ifconfig ens32' or ...) so I can unambigously determine that all screenshots are actually from the production machine (match the DNS/IP in your 'operations.cpf').
Output of 'docker ps' showing the daemon and the database containers.
Output from 'docker exec -ti (your-db-container) redis-cli' shell where you find a user created room in your collection of rooms. Both the 'docker exec' and the room data contents must be visible in that shell.
Evidence that Redis stores data on the production/host machine (volume mounts, directory contents, ...)

Evaluation: I will review your output. I will log into your production server using your Cmd and verify that the room you have created exits. I may DIG a room, and ask you to find that room in the database at any time during the course.