khi

Language: English | 日本語

Kubernetes History Inspector

Kubernetes History Inspector (KHI) is a rich log visualization tool for Kubernetes clusters. KHI transforms vast quantities of logs into an interactive, comprehensive timeline view. This makes it an invaluable tool for troubleshooting complex issues that span multiple components within your Kubernetes clusters. Also, KHI is agentless, allowing anyone to access its features without a complicated process.

Timeline view	Cluster diagram view
Timeline view visualizes resource status change timings with timeline charts and manifest diffs from Kubernetes audit logs.	Cluster diagram visualizes relationships among Kubernetes resources, solely from kube-apiserver audit logs.

Why use KHI?

Insightful Log Visualization

The key strength of KHI is its ability to visualize logs of numerous activities associated with each Kubernetes resource as timeline-based graphs, moving beyond traditional text-based log analysis. You do not need to manually filter logs by a single resource and chronologically reading through individual activity logs in text data anymore. Instead, you can grasp what happened at a glance directly from the timeline visualization. Also, in addition to log visualization, KHI allows you to review the raw log data for that specific moment in its familiar log format in text, and even examine the YAML manifests at the time the specific event took place. This significantly simplifies the process of pinpointing the root cause of an event. KHI can also generate diagrams that depict the state of your Kubernetes cluster's resources and their relationships at a specific point in time. This is invaluable for understanding the status of resources and topology of your cluster at a specific time during an incident.

Agentless and User friendly

KHI is very easy to set up. It is agentless and allows anyone to easily begin using it without any complicated prior setup on target clusters. Also, KHI enables you to visualize Kubernetes logs through GUI operations. You do not need to write complex queries or commands for log retrieval.

Developed from real Log Troubleshooting Experience

KHI is originally developed by the Google Cloud Support team before it became open sourced. It emerged from the practical experience of support engineers, who developed it while analyzing Kubernetes logs in their daily operations. KHI is a tool that takes in their deep expertise in Kubernetes log troubleshooting.

Supported Products

Kubernetes cluster

• Google Cloud

  • Google Kubernetes Engine
  • Cloud Composer
  • GKE on AWS
  • GKE on Azure
  • GDCV for Baremetal
  • GDCV for VMWare

• Other

  • kube-apiserver audit logs as JSONlines (Tutorial)

Logging backend

• Google Cloud

  • Cloud Logging (For all clusters on Google Cloud)

• Other

  • Log file upload (Tutorial)

Getting started

Run from a docker image

Supported environment

• Latest Google Chrome
• docker command

[!IMPORTANT] We only test KHI with on the latest version of Google Chrome. KHI may work with other browsers, but we do not provide support if it does not.

Run KHI

1. Open Cloud Shell
2. Run docker run -p 127.0.0.1:8080:8080 gcr.io/kubernetes-history-inspector/release:latest
3. Click the link http://localhost:8080 on the terminal and start working with KHI!

[!WARNING] The container image repository has been moved from asia.gcr.io to gcr.io. While the old repository is still available, we recommend switching to the new one as the old one will be deprecated in the future.

[!TIP] If you want to run KHI in an environment where the metadata server is not available, you can use Application Default Credentials (ADC) by mounting your ADC file from your host filesystem to the container.

For Linux, MacOS or WSL

gcloud auth application-default login
docker run \
 -p 127.0.0.1:8080:8080 \
 -v ~/.config/gcloud/application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json:ro \
 gcr.io/kubernetes-history-inspector/release:latest

For Windows PowerShell

gcloud auth application-default login
docker run `
-p 127.0.0.1:8080:8080 `
-v $env:APPDATA\gcloud\application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json:ro `
gcr.io/kubernetes-history-inspector/release:latest

For more details, try Getting started.

Run from source code

[!IMPORTANT] Do not expose KHI port on the internet. KHI itself is not providing any authentication or authorization features and KHI is intended to be accessed from its local user.

Authentication settings

Settings for Managed Environments

Google Cloud

Permissions

The following permissions are required or recommended.

• Required
  • logging.logEntries.list
• Recommended
  • These permissions are used to fetch autocomplete candidates in the New Inspection dialog. KHI works without these permissions, but cluster name suggestions will not be displayed.
    • monitoring.timeSeries.list
    • container.clusters.list (Only when using Cloud Composer features)
• Setting
  • Running KHI on environments with a service account attached, such as Google Cloud Compute Engine Instance: Apply the permissions above to the attached service account.
  • Running KHI locally or on Cloud Shell with a user account: Apply the permissions above to your user account.

Audit Logging

• No required configuration KHI fully works with the default audit logging configuration.
• Recommended
  • Kubernetes Engine API Data access audit logs for DATA_WRITE

[!TIP] Enabling these will log every patch requests on Pod or Node .status field. KHI will use this to display detailed container status. KHI will still guess the last container status from the audited Pod deletion log even without these logs, however it requires the Pod to be deleted within the queried timeframe.

• Setup
  1. In the Google Cloud Console, go to the Audit Logs page.
  2. In the Data Access audit logs configuration table, select Kubernetes Engine API from the Service column.
  3. In the Log Types tab, select the Data write Data Access audit log type
  4. Click "SAVE".

OSS Kubernetes

Read Using KHI with OSS Kubernetes Clusters - Example with Loki.

User Guide

Read user guide.

Development Contribution Guide

If you'd like to contribute to the project KHI, read Contribution Guide and then follow Development Guide

Disclaimer

Please note that this tool is not an officially supported Google Cloud product. If you find any issues and have a feature request, file a Github issue on this repository and we are happy to check them on best-effort basis.