ECDI Logging architecture

Shown in the following diagram, how the flow of the logs from eks clusters ec2 instances into Elasticsearch.

There are three main tools shown in the diagram which are:

1. Kibana
2. Elasticsearch
3. FluentD

<!— image: ecdi-logging-architecture diagram (from original wiki uploads) —>

What is Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine-tuned relevancy, and powerful analytics that scale with ease.

What is FluentD

fluentd is an open source data collector for unified logging layer. it allows you to unify data collection and consumption for a better use and understanding of data

What is Kibana

Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.

Kibana Tenants

Tenants in Kibana are spaces for saving index patterns, visualizations, dashboards and other Kibana objects.

By default, all Kibana users have access to two tenants: Private and Global. The global tenant is shared between every kibana user. The private tenant is exclusive to each user and can’t be shared.

You might use the private tenant for exploratory work, create detailed visualizations with your team in analysts tenant, maintain a summary dashboard for corporate leadership in an executive tenant.

To switch tenants, click on your profile icon on the top right and choose Switch tenants.

How to access kibana

Kibana could be accesses using the following link <https://vpc-estack-gcore-prod-5c5762c2hudzfzg5p7veppwtri.us-west-2.es.amazonaws.com/_plugin/kibana/>.

Kibana uses roche credentials (unixID, password) to get you logged in.

note To access the link, you need to be connected to pulse secure VPN.

Applications Indices

each application logs will be indexed Daily in Elasticsearch according to the following format <app_name>-logs-<YYYY.MM.DD>, so each day logs will have its own index in elasticsearch.

To search and visualize the data in these indices, you will need to create an index pattern. index pattern selects the data to use and allows the definition of properties of the fields.

e.g. Spell logs are indexed daily in Elasticsearch so you should find indices named spell-logs-2021.11.1 and spell-logs-2021.11.2 in Elasticsearch. to search these indices and visualize the data inside them, we will need to create index pattern called spell-logs-*. * is a wild card character.