Setting up Loki to fetch logs from VM's

To set up Loki to fetch logs from applications running on AWS EC2 instances, you need to add a k8s_resource to your Blueprint to deploy an internal load balancer. After that, install and configure Promtail on your VM.

Note: This guide assumes Loki is already deployed and running in your environment. If not, please follow the Enable Loki runbook to deploy Loki.

How to Add a K8s Resource to the Blueprint?

To deploy an internal load balancer, you need a K8s resource.

Navigate to the Blueprint and click Add Resource. Search for and add K8s Resource.
Now, select the resource and click Configure.
In the JSON mode, replace the current contents with the contents given below.

{
	"flavor": "default",
	"metadata": {
		"name": "log-collector-lb-svc",
		"namespace": "facets"
	},
	"advanced": {
		"k8s_resource": {}
	},
	"kind": "k8s_resource",
	"disabled": true,
	"version": "latest",
	"spec": {
		"resource": {
			"apiVersion": "v1",
			"kind": "Service",
			"metadata": {
				"name": "log-collector-lb-svc",
				"namespace": "facets",
				"annotations": {
					"service.beta.kubernetes.io/aws-load-balancer-type": "elb",
					"service.beta.kubernetes.io/aws-load-balancer-internal": "true",
					"service.beta.kubernetes.io/aws-load-balancer-scheme": "internal",
					"service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "ip"
				}
			},
			"spec": {
				"type": "LoadBalancer",
				"ports": [
					{
						"name": "http",
						"protocol": "TCP",
						"targetPort": "http",
						"port": 80
					}
				],
				"selector": {
					"app.kubernetes.io/component": "gateway",
					"app.kubernetes.io/instance": "<loki-instance-name>",
					"app.kubernetes.io/name": "loki-distributed"
				}
			}
		}
	}
}

Replace <loki-instance-name> with the instance name of the Loki provisioned via the Blueprint.

Installing Promtail on Your VM

To forward logs to your Loki setup, you need to install Promtail on your VM.
To download and Install Promtail, run the following commands on your VM:

mkdir /opt/promtail && cd /opt/promtail
curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/promtail-linux-amd64.zip"
unzip "promtail-linux-amd64.zip"
chmod a+x "promtail-linux-amd64"

Note: The commands may vary depending on the base image of your virtual machine. Please refer to the official documentation for deploying Promtail.

Creating the configuration file

To set up Promtail on the VM, create the configuration file and run Promtail with those configurations.

Create the following configuration file, replacing with the DNS name of the load balancer created by deploying the k8s_resource.

Modify other parameters according to your service configuration, especially the scrape_configs section to specify the targets and paths to the log files.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: http://<loki-internal-dns>/api/prom/push

scrape_configs:
 - job_name: system
   pipeline_stages:
   - docker:
   static_configs:
   - targets:
      - localhost
     labels:
      job: varlogs
      host: yourhost
      __path__: /var/log/*.log

 - job_name: someone_service
   pipeline_stages:
   - docker:
   static_configs:
   - targets:
      - localhost
     labels:
      job: someone_service
      host: yourhost
      __path__: /srv/log/someone_service/*.log

Save this file on your VM as ./ec2-promtail.yaml. This file will be required to run Promtail.

Running Promtail on Your VM as a Daemon

Create a new service to run Promtail as a daemon by using the following steps:

Create the Service Definition:

Create a new service using vim /etc/systemd/system/promtail.service.
Copy and paste the following service definition:

[Unit]
Description=Promtail

[Service]
User=root
WorkingDirectory=/opt/promtail/
ExecStartPre=/bin/sleep 30
ExecStart=/opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Reload systemd, enable, and start the Promtail service:

systemctl daemon-reload
systemctl enable promtail.service
systemctl start promtail.service

Verify the Service:
1. Ensure the service is running correctly by using the following command:
```
systemctl status promtail.service -l
```

By following these steps, you should start seeing application logs available on Loki.