How I fell in love with logs thanks to Grafana Loki (2022)

As part of my job as a Senior Solutions Engineer here at Grafana Labs, I tend to pretty easily find ways out of technical troubles. However, I was recently having some Wi-Fi issues at home and needed to do some troubleshooting. My experience changed my whole opinion on logs, and I wanted to share my story in hopes that I could open up some other people’s eyes as well. (I originally posted a version of this story on my personal blog in January.)

First, some background info:

I’ve always been a metrics person. I love charts and graphs through and through. My limited exposure to logging was mostly back in my enterprise IT days grepping through a fair share of application logs across hundreds of endpoints. Do you remember the days of creating shared NAS exports and just writing out logs until they filled up the filesystems? (Yeah, me neither . . . ahem . . .)

My home computing footprint is filled with 10Gb switches, multiple wireless access points, trays of render-farm servers, and terabytes of storage. I was having a hard time figuring out some new issues with my network, and it was difficult to see what my gear was doing all the time.

My challenge was to sort out why my wireless devices were having intermittent connection instabilities and which (if any) of my wireless access points were having the most number of issues. But all I had to work with was syslog, a standard network-based logging protocol that devices use to send messages and log events. Metrics weren’t going to save me. I needed a way to get at all of those messages!

(Video) Meet Grafana LOKI, a Log Aggregation System for EVERYTHING

An Internet search for “Syslog Collector” presented me with 342,000 results to start my effort. Most of the attention-grabbing “6 Free Syslog Servers” links led to a fair number of Windows utilities, and they were each pretty limited to just a few hosts at a time. The problem? I needed to collect data from more than a dozen systems and I’m running on Linux and macOS. I was looking for something easy, and I knew that finding some open source goodness might be useful.

This now becomes a tale of how I came to love logs. And Loki. <3

My initial exposure to Grafana Loki came during my first days here at Grafana Labs. Loki is an amazing solution when you want to discover and consume logs alongside Prometheus and Kubernetes for microservices, and it provides a great file and application endpoint logging aggregation system.

When I was learning to utilize Loki in some of its core use cases, I didn’t immediately make the connection that I could also use it to capture standalone network logs. However, once I understood that, I jumped right in with a new deployment of Loki.

Even though Loki’s roots come from Prometheus and Kubernetes, my goal was to build out a quick-start standalone syslog ingester. Discovering just how easy Loki was to deploy as a single binary, either via the command line or in Docker, meant I was able to get started on my project right away. After deploying Promtail (which ingests logs into Loki) I felt I was close to unlocking the mysteries of my network with just a few minutes of effort.

A look through Loki’s documentation on configuring Promtail with Syslog made me realize that Promtail by itself only works with IETF Syslog (RFC5424) — which is how I also found out my devices were limited to only RFC3164. Thankfully, I also discovered how best to solve my syslog dilemma: syslog-ng.

(Video) Grafana Loki: Like Prometheus, But for logs. - Tom Wilkie, Grafana Labs

What’s useful about syslog-ng in my situation is that it can be spun up to listen for RFC3164 (UDP port 514) and then forward to Promtail RFC5424 on port 1514, since most of my devices only output the older style of syslog. A few quick default configuration changes were all I needed to bring together syslog-ng and Promtail and have them happily talking to each other.

syslog-ng configuration

#syslog-ng.conf

source s_local {internal();};source s_network {default-network-drivers();};destination d_loki {syslog("promtail" transport("tcp") port("1514"));};log {source(s_local);source(s_network);destination(d_loki);};

Promtail configuration

#promtail-config.yml

server: http_listen_port: 9080grpc_listen_port: 0positions:filename: /tmp/positions.yamlclients: - url: http://loki:3100/loki/api/v1/pushscrape_configs:- job_name: syslogsyslog:listen_address: 0.0.0.0:1514idle_timeout: 60slabel_structured_data: yeslabels:job: "syslog"relabel_configs: - source_labels: ['__syslog_message_hostname']target_label: 'host'

The relabeling in Promtail takes the hostname of the device sending messages into syslog-ng and turns it into a host label for Loki to index. Within a few minutes, I had all of my hosts streaming syslog from my network into Loki, and it was explorable within Grafana!

An example dashboard showing Loki ingested syslogs forwarded from syslog-ng.

(Video) The Explore Workflow and Troubleshooting with Loki

A dashboard of my dreams

Around the same time as all of my network troubleshooting was happening, one of Grafana Labs’ Lead Solutions Engineers, Ward Bekker gave a presentation to our team on the upcoming launch efforts for the Loki 2.0 release. He showed us some of the new dashboarding examples and without hesitation said to me, “Dave, look how easy it is to turn logs into metrics.” He had my attention!

I returned to my effort to build out a dashboard, taking all of my (now easily gathered) device logs and applied the “logs to metrics” magic recently released with Loki. The net result was group summary counts over time by wireless access points!

This was my first LogQL query that started my logs to metrics journey:

count_over_time({host=~"$hostname", job="syslog"}[$__interval] |="$filter”)

And this example below is showing the number of logs over time filtered by hostname ($hostname) coming from my syslog Promtail job (syslog) with a freeform search query string from my Grafana variable ($filter) auth_failures.

(Video) GrafanaCONline: Loki future

With a bit more dashboard tweaking, I was able to visualize other types of syslog messages from some additional network devices, such as my network gateway, my server IPMI stats, and NAS details. I could now scroll back through my log history, which up until that point was invisible to me. I also had a way of visually understanding the frequency and type of messages being collected, as well as an easy way to free-text filter all of my logs. Truly, logs AND metrics!

Bringing together Grafana, Loki, and Syslog into an all-in-one project

How I got started with Loki and kicked off my logging journey is pretty simple, and I believe it represents how quick and easy it is to connect open source solutions to solve immediate problems, even in a home lab situation.

I wanted to share the configurations so other log-averse people could see the light, so I created an “all-in-one” docker-compose project that I call Loki Syslog AIO.

The quick example project allows you to run all of the mentioned services with docker-compose on a Linux server. Point your network devices at (hostname:514) and log into Grafana (hostname:3000) and you’ll be presented with the “Loki Syslog AIO – Overview” dashboard.

For those of you that want to see some of the behind-the-scenes details, I’ve included some prebuilt performance overview dashboards for each of the main services (Grafana, Loki, MinIO, Docker, and host metrics). You’ll see dropdown links to the “Performance Overview” at the top of the Loki Syslog AIO – Overview dashboard, including links to get you back to the starting dashboard. If you don’t have syslog devices immediately available but want to try the dashboard out, I also built an optional syslog generator container.

(Video) Grafana Loki The road to higher scales and lower latencies for logs

For more setup details and downloads, check out my Grafana Loki Syslog AIO Github repository. My Loki Dashboard example is also available in Grafana’s Community Dashboards.

And in case you’re wondering if Loki helped me figure out what was causing the dropped connections on my home lab servers, the answer is yes! They were related to high DHCP retries and too-aggressive settings on my minimum data rate controls. Thanks, Loki!

The easiest way to get started with Grafana, Prometheus, Loki, and Tempo for tracing is Grafana Cloud, and we’ve recently added a new free plan and upgraded our paid plans. If you’re not already using Grafana Cloud, sign up today for free and see which plan meets your use case.

FAQs

How do you get Loki logs in grafana? ›

To see the logs, click Explore on the sidebar, select the Loki datasource in the top-left dropdown, and then choose a log stream using the Log labels button.

What is the difference between Loki and Prometheus? ›

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

What is grafana and Loki? ›

Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure. The easiest way to get started is with Grafana Cloud, our fully composable observability stack.

How do I get logs in grafana? ›

To use Grafana Logs, you have three options:
  1. Grafana Loki. An open source, horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. ...
  2. Easiest way to get started. Cloud Logs. ...
  3. Enterprise Logs.

How do you query Loki logs? ›

Querying Loki Logs - YouTube

How do I ship logs to Loki? ›

Quickly let's start the installation steps:
  1. Step 1 – Install Grafana Monitoring Tool. In this section we will cover installation of Grafana on Ubuntu. ...
  2. Step 2 – Install Grafana Loki Log aggregation System. ...
  3. Step 3 – Install Promtail Agent. ...
  4. Step 4 – Configure Loki Data Source. ...
  5. Step 5 – Visualize Logs on Grafana with Loki.
17 Mar 2022

Who is the Greek equivalent of Loki? ›

Loki was bound to a rock (by the entrails of one or more of his sons, according to some sources) as punishment, thus in many ways resembling the Greek figures Prometheus and Tantalus.

How is Loki like Prometheus? ›

About Loki

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

How are Prometheus and Loki similar? ›

Through a plot structure similar in both myths, the crafty characters Loki and Prometheus use trickery to publicly challenge authority and upset the sanctity of the hierarchal pantheon in favor of their own ambitions.

What is Grafana used for? ›

Grafana is an open source interactive data-visualization platform, developed by Grafana Labs, which allows users to see their data via charts and graphs that are unified into one dashboard (or multiple dashboards!) for easier interpretation and understanding.

Is Grafana free to use? ›

A forever free plan

The new Grafana Cloud free plan is a composable observability solution, with metrics and logs integrated within Grafana, and it's managed by us completely for free.

What is Loki and Promtail? ›

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored. It primarily: Discovers targets. Attaches labels to log streams.

Where are Grafana logs stored? ›

Usually located at /var/log/grafana/grafana.

Can we see logs in Grafana? ›

View log usage

Use the data sources dropdown located at the top of the page to select the data source corresponding to your Grafana Cloud Logs endpoint. The data source name should be similar to grafanacloud-<yourstackname>-logs .

What is the difference between Grafana and Splunk? ›

Customers use Splunk to search, monitor, analyze and visualize machine data. Grafana belongs to "Monitoring Tools" category of the tech stack, while Splunk can be primarily classified under "Log Management". Grafana is an open source tool with 29.7K GitHub stars and 5.64K GitHub forks.

How do I run a Grafana query? ›

How to build a Prometheus query in Grafana - YouTube

Does Loki have a UI? ›

Grafana's user interface for Loki is part of a new Grafana feature called Explore. Explore got started with a more query-oriented workflow around troubleshooting with Prometheus metrics.

How do you deploy Loki on Kubernetes? ›

Deploy a Loki cluster
  1. Update the chart repository: helm repo update. Bash.
  2. Deploy the Loki cluster using one of these commands. Deploy with the default configuration: helm upgrade --install loki grafana/loki-distributed. Bash. Deploy with the default configuration in a custom Kubernetes cluster namespace:

Is Loki a syslog server? ›

This Loki Syslog All-In-One example is geared to help you get up and running quickly with a Syslog ingestor and visualization of logs.

How do I install Loki on Windows? ›

Install
  1. Navigate to the release page.
  2. Scroll down to the Assets section under the version that you want to install.
  3. Download the Loki and Promtail . zip files that correspond to your system. ...
  4. Unzip the package contents into the same directory. ...
  5. Enter the following command to start Loki:

What is Loki in Kubernetes? ›

Open sourced by Grafana Labs during KubeCon Seattle 2018, Loki is a logging backend optimized for users running Prometheus and Kubernetes with great logs search and visualization in Grafana 6.0. Grafana Loki is a set of components, that can be composed into a fully featured logging stack.

Who is Loki's wife? ›

Loki is married to Sigyn and they have two sons, Narfi and Nari or Váli. By the jötunn Angrboða, Loki is the father of Hel, the wolf Fenrir, and the world serpent Jörmungandr. In the form of a mare, Loki was impregnated by the stallion Svaðilfari and gave birth to the eight-legged horse Sleipnir.

Who is the most powerful trickster god? ›

Prometheus
  • In Greek mythology, Prometheus is one of the Titans, the supreme trickster, and a god of fire. ...
  • Zeus, the chief god, who had been tricked by Prometheus into accepting the bones and fat of sacrifice instead of the meat, hid fire from mortals.
16 Aug 2022

Who is older Thor or Loki? ›

Respective Ages in The Avengers Movies

If you're looking for a quick answer: Thor is older than Loki in the Marvel movies.

What is the difference between Grafana and Prometheus? ›

Prometheus and Grafana are both built for time-series data. Prometheus excels in metric data collection, whereas Grafana champions metric visualizations. Both tools are open source, free and have vibrant communities of open source developers supporting their development.

What is BoltDB shipper? ›

BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index.

What is LogQL? ›

LogQL is Grafana Loki's PromQL-inspired query language. Queries act as if they are a distributed grep to aggregate log sources. LogQL uses labels and operators for filtering. There are two types of LogQL queries: Log queries return the contents of log lines.

Who is Prometheus in Norse mythology? ›

Prometheus is a Titan God of Forethought, knowledge, crafty counsel, fire and creator of mankind. He is the son of Iapetus and Themis. His brothers include Atlas, Epimetheus, and Menoetius.

Does Prometheus have cunning intelligence? ›

Prometheus was known above all for his shrewd intelligence and cunning. During the Titanomachy, which pitted the Titans against the Olympian deities, Prometheus betrayed his brethren and sided with Zeus.

What was the story of Prometheus? ›

In Greek mythology, Prometheus (/prəˈmiːθiəs/; Ancient Greek: Προμηθεύς, [promɛːtʰéu̯s], possibly meaning "forethought") is a Titan god of fire. Prometheus is best known for defying the gods by stealing fire from them and giving it to humanity in the form of technology, knowledge, and more generally, civilization.

What is the advantage of Grafana? ›

Grafana has pluggable data source model and comes bundled with support for popular time series databases like Graphite. It also has built-in support for cloud monitoring vendors like Amazon Cloudwatch, Microsoft Azure and SQL databases like MySQL. Grafana can combine data from many places into a single dashboard.

Is Grafana a DevOps tool? ›

Grafana has become the industry standard because it gathers data from many different data sources, collects the data in one place, and displays it in a unified way. For these reasons, Grafana's usage has increased, and DevOps teams prefer it over the available individual monitoring and logging user interfaces.

How many people use Grafana? ›

With more than 950,000 active installations, Grafana has inspired many different use cases and applications. Find out how customers and community members are leveraging the Grafana stack.

Who owns Grafana? ›

It was the only word Grafana Labs CEO and Co-founder Raj Dutt could use to describe how it felt to look out at the sea of more than 600 Grafanistas gathered together in Whistler, British Columbia, for the first company-wide employee event in two years.

What company owns Grafana? ›

Grafana Labs

Is Grafana better than Kibana? ›

Logs vs.

Grafana's design for caters to analyzing and visualizing metrics such as system CPU, memory, disk and I/O utilization. The platform does not allow full-text data querying. Kibana, on the other hand, runs on top of Elasticsearch and is used primarily for analyzing log messages.

What is Loki stack? ›

Loki is the heart of the PLG stack. It's a data store optimized for logs. In contrast to other log aggregation systems, Loki does not index log messages itself. Instead, it indexes labels assigned with every log. We can query logs stored in Loki using LogQL, a query language inspired by PromQL.

How do you start a Promtail? ›

  1. Download and Install Promtail Binary.
  2. Create the Promtail config.
  3. Configure Promtail as a Service.
  4. Configure Firewall.
  5. Troubleshooting. Permission Denied. Spaces versus Tabs. Tail Promtail.
  6. Useful Links.
  7. Grafana 9 and Ubuntu 22.04 Notes. Promtail v2.6.1. Daemons using outdated libraries.

How do you set up Loki? ›

In order to run Loki, you must:
  1. Download and install both Loki and Promtail.
  2. Download config files for both programs.
  3. Start Loki.
  4. Update the Promtail config file to get your logs into Loki.
  5. Start Promtail.

What database does Loki use? ›

Cassandra is a popular database and one of Loki's possible chunk stores and is production safe.

Is Loki a database? ›

Loki is a log database developed by Grafana Labs. It's similar to Elasticsearch, with some major conceptual differences: Loki has a very simple indexing system, that only indexes a few labels for every log line; Loki will compress the actual log lines before they are saved to disk.

Does Grafana store data? ›

Grafana doesn't store any data, it only visualize data coming from integrated datasources. If you're referring to a cache there's none in Grafana.

What is metrics in Grafana? ›

Metrics tell you how much of something exists, such as how much memory a computer system has available or how many centimeters long a desktop is. In the case of Grafana, metrics are most useful when they are recorded repeatedly over time.

What are traces in Grafana? ›

Grafana Cloud Traces is based on Tempo, an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki.

Does Grafana have an API? ›

The Grafana Cloud API is sometimes referred to as the Grafana.com API or the Gcom API. It is a work in progress, however the following calls and paths on this page are static and approved for general use.

Which is better Grafana or Splunk? ›

Summary and Conclusion. To conclude, Grafana and Splunk are completely two different tools serving completely different purposes. Grafana would be great to visualize KPIs whereas Splunk would be great to search and query data among large volumes of logs.

Why elk is better than Splunk? ›

Both tools support search capabilities in their own distinct languages. Splunk uses the SPL language for querying whereas ELK uses the query DSL (Domain Specific Language). If we look at compression, Splunk is able to support compression whereas ELK does not.

What is difference between Grafana and Tableau? ›

Key differences between Grafana and Tableau. The concept of Grafana is very broad, while Tableau is more focused on business intelligence. Grafana is well suited for working with time-series, application monitoring, and server monitoring. Working with servers can be a bit less comfortable with Tableau.

How do you set up Loki Grafana? ›

Create an account to get started.
...
In order to run Loki, you must:
  1. Download and install both Loki and Promtail.
  2. Download config files for both programs.
  3. Start Loki.
  4. Update the Promtail config file to get your logs into Loki.
  5. Start Promtail.

How can I see Prometheus metrics in Grafana? ›

Click the graph title, then click "Edit". Under the "Metrics" tab, select your Prometheus data source (bottom right). Enter any Prometheus expression into the "Query" field, while using the "Metric" field to lookup metrics via autocompletion.

What is Loki and Promtail? ›

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored. It primarily: Discovers targets. Attaches labels to log streams.

How do I run a Grafana Query? ›

How to build a Prometheus query in Grafana - YouTube

What is Grafana used for? ›

Grafana is an open source interactive data-visualization platform, developed by Grafana Labs, which allows users to see their data via charts and graphs that are unified into one dashboard (or multiple dashboards!) for easier interpretation and understanding.

What is Loki query? ›

Grafana Loki (or Loki) is a horizontally-scalable, multi-tenant log aggregation system that is extremely easy to operate. Loki does not index the contents of the logs, but rather a set of labels for each log stream. Nobl9 users can leverage Loki to query and build metrics on top of their logs.

How do you deploy Loki on Kubernetes? ›

Deploy a Loki cluster
  1. Update the chart repository: helm repo update. Bash.
  2. Deploy the Loki cluster using one of these commands. Deploy with the default configuration: helm upgrade --install loki grafana/loki-distributed. Bash. Deploy with the default configuration in a custom Kubernetes cluster namespace:

Which is better Prometheus or Grafana? ›

Prometheus & Grafana: Better Together

Prometheus collects rich metrics and provides a powerful querying language; Grafana transforms metrics into meaningful visualizations. Both are compatible with many, if not most, data source types. In fact, it is very common for DevOps teams to run Grafana on top of Prometheus.

Is Grafana better than Kibana? ›

Logs vs.

Grafana's design for caters to analyzing and visualizing metrics such as system CPU, memory, disk and I/O utilization. The platform does not allow full-text data querying. Kibana, on the other hand, runs on top of Elasticsearch and is used primarily for analyzing log messages.

Is Prometheus the same as Grafana? ›

Prometheus and Grafana are both built for time-series data. Prometheus excels in metric data collection, whereas Grafana champions metric visualizations. Both tools are open source, free and have vibrant communities of open source developers supporting their development.

What is Loki stack? ›

Loki is the heart of the PLG stack. It's a data store optimized for logs. In contrast to other log aggregation systems, Loki does not index log messages itself. Instead, it indexes labels assigned with every log. We can query logs stored in Loki using LogQL, a query language inspired by PromQL.

Does Loki have a UI? ›

Grafana's user interface for Loki is part of a new Grafana feature called Explore. Explore got started with a more query-oriented workflow around troubleshooting with Prometheus metrics.

How do you start a Promtail? ›

  1. Download and Install Promtail Binary.
  2. Create the Promtail config.
  3. Configure Promtail as a Service.
  4. Configure Firewall.
  5. Troubleshooting. Permission Denied. Spaces versus Tabs. Tail Promtail.
  6. Useful Links.
  7. Grafana 9 and Ubuntu 22.04 Notes. Promtail v2.6.1. Daemons using outdated libraries.

Which query language is used in Grafana? ›

You will have to learn the Prometheus query language (PromQL). The query editor in Grafana for Prometheus is just a textbox for writing PromQL. (Other Time Series dbs have a query editor with dropdowns and intellisense but the Prometheus community generally prefers to use PromQL).

How do I edit a data source in Grafana? ›

The following trick worked for me with Grafana 8. As preparation you have to create a datasource variable "DS_PROMETHEUS".
...
Then inside the dashboard:
  1. Click "Share Dashboard or Panel"
  2. Switch to "Export" Tab.
  3. Activate "Export for sharing externally"
  4. Click "View JSON" and copy the json to the clipboard.
19 Feb 2019

Videos

1. Loki - Prometheus for logs
(FOSDEM)
2. Monitoring, Logging, And Alerting In Kubernetes
(DevOps Toolkit)
3. For billion series scale or home IoT projects, get started in minutes with Grafana Mimir
(Grafana)
4. Dashboards for DAYS! - How we use Grafana in our #homelab!
(2GuysTek)
5. Full Tutorial: Monitoring and Troubleshooting stack with Prometheus, Grafana, Loki and Komodor
(Anais Urlichs)
6. Introduction to Grafana Loki - Julien Pivotto
(TheCentOSProject)

Top Articles

You might also like

Latest Posts

Article information

Author: Greg O'Connell

Last Updated: 01/29/2023

Views: 5229

Rating: 4.1 / 5 (42 voted)

Reviews: 81% of readers found this page helpful

Author information

Name: Greg O'Connell

Birthday: 1992-01-10

Address: Suite 517 2436 Jefferey Pass, Shanitaside, UT 27519

Phone: +2614651609714

Job: Education Developer

Hobby: Cooking, Gambling, Pottery, Shooting, Baseball, Singing, Snowboarding

Introduction: My name is Greg O'Connell, I am a delightful, colorful, talented, kind, lively, modern, tender person who loves writing and wants to share my knowledge and understanding with you.