ArgoCD Administrator Guide

This document is intended for ArgoCD administrators and troubleshooting purposes. For information on how to use this tool, please refer to the ArgoCD user guide.

Summary

Starting in 2023, the infrastructure team decided to implement ArgoCD to facilitate application deployment on EKS and improve overall efficiency.

Argo CD enables the seamless deployment of applications on Kubernetes. It offers an intuitive WebUI that enables developers to gain a visual representation of their applications on the EKS cluster. This can be achieved by defining your application utilizing Kubernetes manifests in YAML format, Kustomize, or Helm charts.

ArgoCD is famous for implementing GitOps approach. However, in our organizations, we have decided to utilize Helm charts instead of following the GitOps approach. This decision was made to simplify the process of providing and deploying the repository for our specific needs. As a result, we have chosen to use Argo CD as a provider for our continuous delivery pipeline for Helm chart applications.

Quick Start

We have set up Argocd for each of our EKS clusters:

ArgoCD URLs

ACE EKS Test: https://argocd.eks.test.gred.ai/
ACE EKS Prod: https://argocd.eks.prod.gred.ai/

Please note that these URLs are only accessible from the office or over the enterprise VPN.

ArgoCD Terraform Code

ArgoCD was deployed via a Helm chart under the namespace argocd on both EKS clusters.

GitHub Repositories:

Kubectl, WebUI, and CLI Access

There are three ways you can access ArgoCD:

WebUI: Most of the work can be done via the WebUI mentioned above, which is recommended for both admins and users. But for access management and authentication parts you need to use two below options.
Kubectl: Since you are part of the infra-team, you have access to argocd components (k8s resources like configmaps, ingress, etc.) via the kubectl command.
- All the ArgoCD components are under the namespace argocd.
Argocd Terminal Application: ArgoCD has an application for running commands from your terminal. You can perform all the tasks you need with it, as it has enriched command lists.
- You can install ArgoCLI via this link.
- You can authenticate yourself via sso with this command. Here is an example for ArgoCD Test:
```
argocd login argocd.eks.test.gred.ai --grpc-web --sso
```
For more commands, please check the Troubleshooting section.

Architecture

Terms

Here are some terms related to ArgoCD that will help you better understand the content of this document:

ArgoCD AppProject: Each ArgoCD application must be part of an AppProject. You can configure application access and roles at this level and allow access to specific repositories, clusters, namespaces, and k8s resources for applications.
- You can find the AppProject template we are using here.

We control user access at the AppProject level. More details can be found in the Access Management section.

ArgoCD Application: This is the main component where we define our application, its repository location, custom helm values, and more.
- When you create an ArgoCD application via WebUI an ArgoCD application will be created in the background.
- Here is an example of a helm-based ArgoCD application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <application_name>
spec:
  destination:
    namespace: <namespace_name>
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: <registry_URL>
    targetRevision: <helm_version>
    chart: <repo_URL>
    helm:
      values: |
        replicaCount: 1
  project: <name_of_AppPorject>

Access Management

Authentication

ArgoCD authentication is done via Okta SAML. The SAML configuration is located in the argocd-cm configmap at argocd namespace.

It is important to note that this configmap is a critical file and should not be directly modified. Instead, any changes should be made via Terraform.

Authorization

ArgoCD uses a configmap called argocd-rbac-cm to manage user access. All user access has been configured through this file.

All gred_ecdi teams that have been added to Okta have access to ArgoCD and their access is limited based on the associated AppProject.
Each team has a dedicated AppProject based on their team name in the format <team_name>-default, and they have full access to the Argo applications they create under this project.
However, they are explicitly not allowed to create or delete these <team_name>-default AppProjects.

Their access to ArgoCD is as follows:

g, gred_ecdi_<team-name>, role:<team-name>                                                     # Assigne okta group to argocd role
p, role:<team-name>, repositories, *, *, allow                                                 # Allow users to add/modify/delete repositories
p, role:<team-name>, applications, *, <team-name>-default/*, allow                             # Allow users to create/modify/delete only applications under their own appproject
p, role:<team-name>, projects, get, <team-name>-default , allow                                # Allow users to see their team appproject
p, role:<team-name>, exec, create, <team-name>-default/*, allow                                # This privilege allows a user to exec into Pods via the Argo CD UI. The functionality is similar to kubectl exec.
p, role:<team-name>, logs, get, <team-name>-default/*, allow                                   # Allow users to see their logs of their own appprojects
p, role:<team-name>, repositories, delete, 712649426017.dkr.ecr.us-west-2.amazonaws.com, deny  # Prevent all users to delete AWS ECR repository

Please note that the ace-infra team has administrative privileges and full access to everything. It is important to avoid directly modifying the argocd-rbac-cm configmap because it is a critical file. Instead, any changes should be made via Terraform.

Monitoring & Logging

Num	Dashboards	Description
1	EKS Ace Test - Applications	Select the `argocd` namespace in the namespace field
2	EKS Ace Prod - Applications	Select the `argocd` namespace in the namespace field

Troubleshooting

1. Check User Access to Specific Resources

If a user complains about not having access to specific resources, follow these steps:

1.1 Check the user’s Okta group name in sso.gred.ai.
1.2 Verify if the argocd-rbac-cm configmap contains the Okta group name.
1.3 To verify user or role access to a specific resource, execute argocli, get the argocd-rbac-cm manifest, and run commands against it. This ensures that users or roles can access specific resources. For more information, see this document.

Here is an example of how to check if a role with name <role_name> can get all applications:

kubectl get confimap argocd-rbac-cm -n argocd -oyaml > argocd-rbac-cm.yaml
argocd admin settings rbac can role:<role_name> get applications --policy-file argocd-rbac-cm.yaml

By following these steps, you can quickly troubleshoot user access issues related to specific resources.

2. Useful ArgoCLI Commands

Below are some useful ArgoCLI commands:

2.1 Use the following command to create an application and authenticate using an auth token.

--upsert: This option will allow you to override the existing application with the same name in case the supplied application specification is different from the existing one.

argocd app create <application_name> --file <application_file>.yaml \
	--auth-token <auth_token> \
	--server <argocd_server_address> --grpc-web --upsert

TODO

Martin reported some issues related to project tokens. It appears that although they do not expire, they result in errors after some time.
We are currently unable to enable multiple-team access for a single project. We are working towards resolving this issue.
Due to the unique use case of the Eyenotate app, the project is currently shared between the Data-Eng and Air teams using manual codes. However, we need to address this issue. For more information, check this link.
Enable monitoring for ArgoCD.

Overview ArgoCD User Guide