Purpose

This is a temp document for transferring information before my PTO. The purpose is help others participate in our Winterlight containerization effort which in short involves:

• containerizing this Winterlight project so that it can serve as our first internally developed containerized application.
• setting up Gitlab CI to automate the application build.
• (re)provisioning some AWS EKS clusters for test/stage/prod application hosting
• deploying Winterlight to EKS as our first internally developed EKS application.
• Winterlight AuthN/AuthZ integration testing.

Status

Summary

EKS Provisioning

Summary

At present we have three EKS clusters in our gred-ace-prod AWS account, only one of which relates to this project. I’ll summarize all three below for context. Spoiler alert: #3 is what matters here.

1. ecdi-prod - A first pass production EKS cluster that was spin up by Will for hosting our Prometheus monitoring solution. We can technically use this, but Adam and I have decided to start anew using a different provisioning strategy.
2. wandb-prod - EKS cluster dedicated (for now) to our Wandb (vendor) production deployment. Completely irrelevant to this project.
3. ace-test - Our new test EKS cluster, provisioned on June 27. The rest of this section relates to this cluster.

TFCloud Workspace

• Terraform Cloud workspace.

Note that it’s in an ERRORED state, but the cluster is successfully built because I provisioned it via a workstation run of terraform apply. After getting it working as desired I checked the code in and moved to TFCloud in preparation for PTO. I’m sure I can resolve this issue when I spend some time on it, but it’s not the most important thing before I leave so I’ll deal with it later.

If anyone needs to make cluster changes in my absence, just go into the workspace and change it from an agent run to a local run and proceed accordingly. If that doesn’t work and you’re blocked (perhaps due to perms which I’ll cover later) just Slack me and I’ll fix next time I check in.

Cluster Access

At present, cluster access is integrated with AWS IAM. Some highlights.

• All cluster access is configured in Terraform code for simplified administration.
• I’m employing what I consider to be a better access pattern that grants cluster access via assumeRole rather than hardcoding users in the aws-auth Kubernetes config map, as is commonly done.
• Today, anyone in the AWS infra-team user group has cluster administrator access.
• Access requires assuming the ace-test-k8s-admin role. I’ll provide an example on how to do this below.
• I also have a ace-test-k8s-reader role, but I have not mapped it to any user groups because we don’t yet know who needs such access. It’s unimportant now.

Access Howto

As noted above, access requires that you’re a member of the infra-team AWS IAM user group. If you are, you can simply assume the ace-test-k8s-admin role to interact with the cluster.

Thankfully, the assumeRole workflow can be baked into your Kubernetes configuration so that you don’t need to prefix your kubectl commands with aws sts assume-role commands.

To access our `ace-test cluster please follow these steps:

1. Run the following command to add the ace-test cluster to your KUBECONFIG:

aws eks update-kubeconfig --name ace-test --role-arn arn:aws:iam::712649426017:role/ace-test-k8s-admin

2. (Optional) Run the following command to shorten the name of your newly added K8s context from the default ARN to ace-test:

kubectl config rename-context arn:aws:eks:us-west-2:712649426017:cluster/ace-test ace-test

3. Test access to the cluster with a few common kubectl commands

# get cluster info
kubectl cluster-info
 
# list k8s nodes
kubectl get nodes
 
# get pods in kube-system namespace
kubectl get pods -n kube-system

Auth Integration Testing

As noted, this app will eventually make its way to the EKS test cluster detailed above. In the meantime, we can proceed with integration testing by simply running the app anywhere that Docker is installed. I’ll provide instructions here for doing that.

Container Image Access

• The Winterlight project code is here.
• The Winterlight container registry is here
• Gitlab Container Registry documentation is here.

Create Gitlab Personal Access Token

To get and run a Docker image from Gitlab, you need to first authenticate via docker login. You will use your UNIX id as your username, but you must use a Gitlab token as your password. Your LDAP password will not work. If you don’t already have a Gitlab access token you can create one easily like so:

1. Login to the Gitlab UI.
2. Click your avatar in the top right corner and select Edit profile.
3. Go to Access tokens. Here is a direct link to that page.
4. Name your access token. I named mine after my workstation name: stompy.
5. Optionally set an expiration if you want. It’s your token.
6. Set the permissions you want for this token. You need read_registry to download a container image.
7. Click Create personal access token and protect the token like a password.

Login to Gitlab Container Registry

1. Run docker login.
2. Enter your UNIX ID as your username.
3. Enter your personal access token as your password.

For example:

$ docker login registry.code.roche.com -u bushnet1
Password: *****your personal access token*****
Login Succeeded

Run Winterlight

1. Visit the Winterlight container registry.

2. Click ace/Winterlight/Root image.

3. Find the latest image tag, by date. We’ll eventually support semver and latest, but today we don’t.

4. Click the icon to copy the container image path <!— image: image (from original wiki uploads) —>

5. Run the following command to pull and run the application on port TCP/8000.

$ docker run -p 8000:8000 registry.code.roche.com/ace/winterlight:caf986b3
Unable to find image 'registry.code.roche.com/ace/winterlight:caf986b3' locally
caf986b3: Pulling from ace/winterlight
6c33745f49b4: Pull complete
ef072fc32a84: Pull complete
c0afb8e68e0b: Pull complete
d599c07d28e6: Pull complete
f2ecc74db11a: Pull complete
26856d31ce86: Pull complete
d4db05d8da44: Pull complete
a63ae5575b2c: Pull complete
a8c72204fafe: Pull complete
588d4fc87a50: Pull complete
0d7a4a6073cb: Pull complete
3a7e0859d882: Pull complete
644da687b92a: Pull complete
f6e61dbee877: Pull complete
3e19947ae254: Pull complete
4a4150ad76a8: Pull complete
331fe1ea8d9a: Pull complete
edff73282333: Pull complete
4f6de0459686: Pull complete
bedd5cb502e6: Pull complete
Digest: sha256:8f50b5f418ce3b554dcbf65e5fd0826b4beaf407ccde18456a7951f588995081
Status: Downloaded newer image for registry.code.roche.com/ace/winterlight:caf986b3
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

6. Confirm

Note that system firewall changes will dictate what works for you and from where. Assuming you have TCP/8000 open you will be able to hit the application like I depict below on host tmbtest-svcs.ecd-ai.com.

$ curl -Ls http://tmbtest-svcs1.ecd-ai.com:8000
{"msg":"Hello World"}

Once you’ve confirmed connetivity, you can proceed with testing per the Winterlight project README.

FAQ

When can we expect a Winterlight EKS dev deployment?

This is done. ✅

$ kubectl get deploy winterlight -o wide
NAME          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS    IMAGES                                             SELECTOR
winterlight   1/1     1            1           79m   winterlight   registry.code.roche.com/ace/winterlight:caf986b3   app=winterlight

Winterlight is accessible via internal network load balancer service

$ kubectl get service winterlight -o wide
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)          AGE   SELECTOR
winterlight   LoadBalancer   192.168.183.7   k8s-default-winterli-d2d80ef815-af2db454066ec246.elb.us-west-2.amazonaws.com   8000:31049/TCP   55m   app=winterlight

And is accessible via friendly name like so:

$ curl -s http://winterlight-test.gred.ai:8000
{"msg":"Hello World"}

When can we expect a Winterlight EKS prod deployment?

Assuming the above Dev timeline and there are no application delays, I can get a production EKS cluster up and the application deployed to it by Monday, 19 July.