Mandatory 5: Horizontal Scaling. Continuous Integration

Learning objective:

To explore horizontal scaling, by making multiple replicas of the daemon; and to use (BitBucket) build pipelines for CI.

Deadline

December 12th at 23.59

Exercise 'horizontal-scaling-failing'

Learning Goal: To demonstrate the inability of SkyCave to do scale horizontally.

Note: This exercise may strain your laptop quite a bit. If you can spare it, increase Memory (6GB) in the VM, as I had to do.

We aim to make a stack running 2-3 skycave daemons as replicas, but configured to accept any login credentials; and next generate some load on it.

Experiment Part I: Setting up

Experiment Part II: Scaling

Exercise 'session-database' [M 20]

Prerequisite: You have done the experiment above, as it is required as part of the hand-in.

The previous exercise should establish that our SkyCave daemon does not presently scale horizontally, as it is statefull, a state which is not shared across daemon replicas.

You should update SkyCave so it handles horizontal scaling of the 'daemon' using the 'session database' pattern.

Requirements:

Hand-in:

Evaluation: I see your screenshots, I see your code, I like it, I award 20 points. And by now, you know what I like...

Hint: The solution to this exercise is extremely simple, once you 'see the light', and SkyCave is already prepared for it through the configurable 'PlayerNameService'. So - if you find yourself wanting to code a lot of Java, starting new databases, adding new service points to the CPF, then you are on the wrong track! You should notice that current SkyCave design already has a database which stores current session data...

Exercise 'horizontal-scaling' [M 40]

Update 'daemon' so it runs two or three SkyCave daemon instances in a horizontal scaled / load-balanced setup in Docker Swarm.

Note: Solve the previous exercises first, as they contain 95% of the solution ;-).

Requirements:

Hand-in:

Evaluation: I will evaluate analyzability and correctness of the compose file, with (0,4,7,10) points for each learning goal, as outlined below. I will then multiply by two to get the final score.

Learning Goal Assessment parameters
Analyzability The compose file is easy to analyze (read, understand, reason correctly about). The compose file contains a preamble outlining how to use it (build instruction, stack deploy command, etc.)
Correctness I will (as best possible) evaluate correctness of the composefile from your screenshots, the file contents, and potential execution. The screenshots must be detailed and comprehensive enough that I can convince myself that everything works correctly.

Exercise 'horizontal-scaling-haproxy'

This exercise is "very optional" and only works on Linux!

In this exercise we will run two daemons and use HAProxy as load balancer. HAProxy defaults to true 'round-robin' load balancing which makes it much easier to detect failures when horizontally scaling the daemon. Thanks to former students Mark and Finn for defining the ha proxy configuration file.

Use the HAProxy load balancer for a manually configured, local, setup: Your 'cmd' should contact the IP of the load balancer (localhost:7777), which in turn routes all calls to two instances of daemon (e.g. localhost:6677 and localhost:6688). Naturally, you have to run the daemons with the Redis storage attached (but all other services may be test doubled). And create CPF files that match a suitable configuration for each of the two daemons (you need to set the SKYCAVE_APPSERVER to make the two daemons run on individual ports.)

You can find Finn and Mark's (slightly modified) haproxy configuration which should more or less work out of the box, if you run all services on localhost.

The (as always) by far easiest way to run the HAProxy service, is using docker but beware that you need to run it using --network host so the load balancer is on the host machines network (otherwise it cannot forward requests to the localhost based daemons.)

Note: you have to copy the haproxy configuration file into the haproxy container. Either build a small Dockerfile which creates a local image you can run (it is described at docker hub), or use volume mounting using -v to make the configuration file available from within the container. I did the former, and it boils down to: 

# Build file for the HAProxy server alone

FROM haproxy:2.0
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

Process:

Exercise 'horizontal-scaling-mq'

As an alternative to swarm's ingress network or a load balancer, you can use a Message Broker instead. We may return to MQ systems in course two, as they have a lot of nice Availability Quality aspects.

In RabbitMQ's RPC Tutorial for Java you can find both client side and server side example code, which is rather easily modified to fill the 'ClientRequestHandler' and 'ServerRequestHandler' roles of the FRDS.Broker pattern.

So, start a RabbitMQ in a docker container, code the CRH and SRH roles, and let the Broker pattern use the MQ as IPC layer instead of HTTP or Sockets.

Exercise 'bitbucket-pipeline-ci' (*) [M 80]

We have a lot of trickery going to move from a code change, until it provides value for our customers, and while gradle and docker IaC helps a lot, we still do a lot by hand. In this exercise, we start automation in a build pipeline, but focuses on the production of a 'release candidate deployment unit'.

In this exercise, you should write a BitBucket IaC pipeline, which does Continuous Integration, to produce a (Docker) Hub image release candidate.

Requirements: Your pipeline must (at least) consist of four stages:

Hand-in

If you are used to another pipeline system, like GitLab, you can do the exercise using that particular system, but please provide some good comments as to what is going on, so I can get an understanding :-).

Evaluation: The 80 points are split into (20/60). I use the (0, 4, 7, 10) scale to evaluate your pipeline IaC file's analyzability - and then multiply by two. The remaining 60 points are rewarded in case your screenshots leaves no doubt in my mind, that you have actually made a working pipeline.

Learning Goal Assessment parameters
Submission Required artifacts (pipeline file, screenshots) are all present. All four required steps in the pipeline has been made.
Analyzability The pipeline file is easy to analyze (read, understand, reason correctly about). It contains author/group information. It contains concise comments on cental aspects of each step.

Hint: You have to log into docker to push your image but do NOT put your credentials in the script! Instead you can enter your credentials securely in the pipeline account variables which are then exposed as normal Linux environment variables in your pipeline scripts. To set a variable, you have to open the BitBucket repository for your project, next click Repository settings (in the menu on the left), then scrolle down to Pipelines/Repository variables where you can add an environmental variable and its value which is stored encrypted. Alternatively you can set a 'workspace variable' from your root bitbucket account page.

Example: I have set the workspace variable DOCKERHUB_PASSWORD to be my hub password and can then do stuff like 

          - echo ${DOCKERHUB_PASSWORD} | docker login --username "$DOCKERHUB_USERNAME" --password-stdin

If you use hub.baerbak.com, remember that in your login!

Exercise 'alternative-ci'

Redo the above exercise but with Jenkins, or GitLab CD, or Concourse, or Hudson, or ... instead.