Skip to content

Releases: netdata/netdata

v1.45.3

12 Apr 13:07
Compare
Choose a tag to compare

Warning

Important Security Update

Netdata v1.45.3 is a patch release to fix a local privilege escalation vulnerability discovered in v1.45.x releases. Users are advised to upgrade any systems running v1.45.0, v1.45.1, or v1.45.2 immediately. Stable releases before v1.45.0 are unaffected by this vulnerability. Full details on the vulnerability can be found in the associated security advisory on GitHub. A big thank you to mia-0 for identifying and reporting this issue!

This patch release also addresses other issues discovered since v1.45.2.

This patch release provides the following bug fixes and updates:

  • Mitigated a security issue in ndsudo by restricting its search paths to a predefined set of directories (#17377, @ilyam8)
  • Resolved an issue that prevented the "percentage" option from functioning correctly in alert lookups (#17391, @ktsaou)
  • Enhanced macOS uninstallation by enabling removal of the associated LaunchDaemons plist file (#17357, @ilyam8)
  • Increased the default minimum thread stack size to 1 MB to address potential stability issues caused by the musl libc's smaller default (128kB) (#17317, @ilyam8)

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!

v1.45.2

01 Apr 17:15
Compare
Choose a tag to compare

Netdata v1.45.2 is a patch release to address issues discovered since v1.45.1.

This patch release provides the following bug fixes and updates:

  • Improved PostgreSQL/MySQL local listener discovery to automatically check for connections using both TCP and Unix sockets, enabling support for passwordless Unix socket connections (#17304 #17305, @ilyam8)
  • Fixed an issue that prevented negative matching of host/chart labels in alert configurations (#17290 #17292, @ktsaou)
  • Improved go.d.plugin stability by preventing Netdata from shutting down the entire plugin due to an issue with registering jobs for unregistered modules (#17289, @ilyam8)
  • Improved go.d.plugin HTTP requests now include a UserAgent string, enhancing identification in server log (#17286, @ilyam8)
  • Improved Nginx discovery in go.d.plugin by automatically trying multiple status endpoints when discovering Nginx containers (#17285, @ilyam8)
  • Fixed a go.d.plugin panic that could occur when using the Unbound collector with TLS (#17283, @ilyam8)
  • Fixed a libyaml linking issue (#17276, @Ferroin)
  • Improved go.d.plugin configuration validation, preventing unexpected or invalid options through dynamic configurations (#17269, @ilyam8)

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!

v1.45.1

27 Mar 15:24
Compare
Choose a tag to compare

Netdata v1.45.1 is a patch release to address issues discovered since v1.45.0.

This patch release provides the following bug fixes and updates:

  • Ensured proper handling of default values for data collection jobs submitted via dynamic configuration. (#17255, @ilyam8)
  • Optimized go.d.plugin service discovery by filtering out irrelevant docker-proxy listeners. (#17254, @ilyam8)
  • Improved go.d.plugin's ability to find applications, including those using IPv6, and identify Apache processes more reliably. (#17252, @ilyam8)
  • Improved OpenSSL discovery on macOS for Homebrew builds. (#17250, @Ferroin)
  • Obsolete references to saving the internal database using the USR1 signal, reflecting the removal of save/map memory modes. (#17249, @ilyam8)
  • Added ZSTD compression support for dbengine (disabled by default for now). This improves storage efficiency when available, automatically falling back to uncompressed pages for compatibility. (#17244, @ktsaou)
  • Fixed a bug that caused metric reference count errors during release. (#17239, @ktsaou)
  • Code cleanup. (#17237, @ktsaou)
  • Enabled Gorilla compression by default for dbengine, reducing memory usage. (#17234, @ktsaou)
  • Improved dbengine unit tests for better code coverage and maintainability. (#17232, @ktsaou)
  • Fixed a database engine cache bug that could cause queries to stop prematurely under pressure. (#17231, @ktsaou)
  • Implemented caching optimization to reduce the number of cache flushes following journal file v2 creation. (#17220, @stelfrag)
  • Reduced clutter in MySQL/MariaDB query logs by disabling session query logging for the go.d/mysql collector. (#17219, @ilyam8)
  • Improved go.d.plugin to correctly identify MariaDB databases. (#17218, @ilyam8)
  • Enhanced macOS build stability by using native libraries and optimizing checks for dependencies. (#17216, @Ferroin)
  • Suppressed unnecessary compiler warnings about redefined macros, improving build cleanliness and compatibility with stricter build flags. (#17209, @Ferroin)

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 2000 engineers are already using it!

v1.45.0

21 Mar 13:48
Compare
Choose a tag to compare

Table of Contents

Netdata Growth

  • 67.5k GitHub stars!
  • 626M Docker Hub pulls!

Thanks to your love ❤️, Netdata is leading the observability category in CNCF, having significantly more stars than Elasticsearch, Grafana, Prometheus and all other observability solutions listed in CNCF landscape.

We are committed to provide the most advanced and innovative observability solution, to help us minimize monitoring costs while providing AI-powered high-fidelity monitoring!

You like Netdata? Give Netdata a ⭐ too, on GitHub!

Release Summary

3 months have passed since the previous Netdata release. A lot has changed since then! Netdata now has a mobile app for alert notifications, new drag-and-drop custom dashboards, network connections monitoring, dynamic configuration for data collection jobs and alerts, and many more...

To see how Netdata stacks up against the most advanced commercial offerings available today, we did an analysis on how Dynatrace, Datadog, Instana, Grafana and Netdata commercial offerings compare.

It is nice to see that Netdata stands out for its:

  1. Excellent technology coverage

    Netdata's monitoring coverage is significantly higher compared to others, in all areas!

  2. High-fidelity, real-time insights

    Netdata is the only monitoring solution offering this kind of fidelity (per-second for all metrics), at this extend!

  3. Real AI and Machine-Learning.

    Netdata is the only monitoring system that offers real machine learning, running at the edge.

  4. Lightweight

    Netdata is among the lightest agents, despite the fact that it does a lot more than the others.

  5. Best cost efficiency

    Netdata's cost efficiency is unbeatable, making Netdata the most cost-efficient monitoring solution available today!

Read the full blog here.

Release Highlights

Netdata Mobile App

You can now receive Netdata alerts directly on your mobile phone!

Choose your space and see all the available notifications since you last signed in!

Check the full demo here.

The Mobile App is available for Homelab and Business plan users.

Custom Dashboards

You can now create advanced custom dashboards with Netdata!

  • Drag-and-Drop

    Easily move charts from Metrics or Single Node views straight to your dashboards. It's intuitive and fun.

  • New Chart Types

    Discover your data in new ways with Bar, Circle, Gauge, Pie, Value, and Group boxes.

  • Quick Dashboard Creation

    Hit the plus button, drag, and you've got a new dashboard. Simple as that.

  • Rename Charts

    Customize your dashboard by renaming charts to whatever makes sense to you.

  • Refreshed Text Cards

    We've upgraded text cards for better clarity and aesthetics.

On the Agent UI and the Community plan of Netdata Cloud 1 custom dashboard is allowed. The Homelab and Business plans of Netdata Cloud support an unlimited number of custom dashboards.

Network Viewer

Explore the network connections of your servers and processes!

302690196-5f71c102-9146-463e-acba-329094b136a5

Netdata got a network viewer (select network-connections from the Top tab inside the dashboard).

The tool reports all IPv4 and IPv6, TCP and UDP sockets a system and all its processes have. Also, it automatically and reliably classifies them as inbound, outbound, local (i.e. within the host itself), or listen (for daemons).

The visualization graph has 4 sides:

  • public (i.e. public IPs),
  • private (i.e. private and reserved IPs),
  • servers (i.e. listening and inbound sockets),
  • clients (i.e. sockets towards other servers).

The position of each application on the chart is determined by the classification of the sockets it has. To the top are clients, to the bottom are servers, to the right are internet facing applications, to the left is internal network applications.

The size of each application in the chart is determined by the number of sockets it has, and each application is a pie chart representing the percentage of each kind of sockets it has.

For servers with dozens of thousands of sockets, the tool provides an aggregated view, grouping similar sockets together and reporting the total. Users can switch to a detailed view from the UI.

User Settings

We've improved immensely the customization capabilities of Netdata with the introduction of User Settings.

Our first release on this front is focused on the customization of charts, either on the Metrics tab or the Single Node view tab. You can now create for any chart:

  • Personal views
  • Room specific views
  • Space dedicated views

With this, you can define what is best for your team to visualise a given chart but still allow each teammate to define their own. Users will be presented with the view they should see, based on setting hierarchy, but they
can interchangeably select which of the select views they want.

chrome_UPtwnDlNyd

More areas of customization will come soon, Filters saved views, Dashboards Table of Content (TOC) ordering, etc.

Dynamic Configuration (beta)

Netdata agents are now deployed with the ability to dynamically accept configuration from the UI, for data collection jobs and alerts. The feature is released in beta.

Alerts Configuration Manager

The Alerts Configuration Manager is transforming the way users configure and manage alerts in their Netdata environment. This powerful tool integrates directly into the Netdata Dashboard, offering a streamlined and intuitive interface for both novice and experienced users.

Check the full demo here

Alerts Silencing Rules

g6oHMtz

🔕Improvements done to make it easier to interact and see Alert Silencing Rules.
With this release, you will be able to:

  • See silencing rules status directly on entities like Alerts, Rooms, and Nodes
  • Immediately create a silencing rule for an Alert, a Room, or a Node

We hope this makes it easier for you to interact with the Alert Silencing Rule Manager.
Stay tuned for more improvements!

MacOS Processes Monitoring

Netdata's apps.plugin has been ported to macOS, allowing users to view processes information on Linux, FreeBSD and macOS!

Just install the latest Netdata on your macOS and enjoy full processes monitoring!

Homelab Plan

For non-professional use, get the whole and the latest of Netdata! For the cost of a beer per month, you can get access to all Business features of Netdata, for your home lab or personal project!

Our Homelab plan is available to technology enthusiasts and students, for non-professional user, offering the entire Netdata suite, for a small flat fee, under a fair usage policy.

  • Unlimited Access: Enjoy the freedom of unlimited usage, with no caps on nodes or custom dashboards.

  • Premium Features: Get your hands on business-level features, including enhanced alert integrations and access to our mobile app, all tailored for your personal projects.

  • Support Netdata: Support the open-source Netdata, to ensure it will be there for you, when you need it!

New Build Infrastructure

Starting with Netdata 1.45, we have completely removed our GNU Autotools based build system and replaced it with
CMake. The new CMake build system has a number of significant benefits for developers, package maintainers, and
...

Read more

v1.44.3

12 Feb 19:09
Compare
Choose a tag to compare

Netdata v1.44.3 is a patch release to address issues discovered since v1.44.2.

This patch release provides the following bug fixes and updates:

  • Improved handling of slow queries and CPU usage of the ACLKSYNC thread. (#16838, @stelfrag)
  • Improved error handling for listen bind failures. Instead of terminating fatally, Netdata now exits gracefully. (#16937, @stelfrag)
  • Fixed invalid alert durations in health log entries. (#16931, @stelfrag)
  • Fixed a race condition during analytics data setup, preventing potential Netdata crashes. (#16929, @stelfrag)
  • The Netdata base image includes Debian backports for comprehensive security and stability. (netdata/helper-images#271, @tkatsoulas)

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1900 engineers are already using it!

v1.44.2

06 Feb 17:50
Compare
Choose a tag to compare

Netdata v1.44.2 is a patch release to address issues discovered since v1.44.1.

This patch release provides the following bug fixes and updates:

  • Fixed an inconsistency where the NETDATA_LOG_LEVEL environment variable did not affect log level in Docker containers. (#16943, @ilyam8)
  • Fixed inconsistent log severity across sources: log severity level setting now work for all Netdata log sources (daemon, collector, health, access, aclk). (#16922, @ilyam8)
  • Fixed a bug in chartd.d.plugin that prevented loading of its modules configuration files. (#16939, @ilyam8)
  • Fixed inaccurate server type identification in Netdata Cloud for FreeBSD jails. Jails are now recognized correctly. (#16858, @ilyam8)
  • Fixed a bug that prevented the edit-config script from running correctly in Podman containers. The script now accurately identifies container environments. (#16825, @Ferroin)
  • Fixed a bug that caused excessive logging of "Using host prefix directory" messages. (#16814, @ilyam8)
  • Fixed incorrect label source for apps.plugin charts, ensuring they are now accessible when querying Prometheus metrics. (#16810, @boxjan)
  • Fixed a bug in the cgroups.plugin that could lead to crashes. Additionally, addressed incorrect thread name during fatal Agent exits. (#16771, @ktsaou)
  • Fixed a race condition related to pthread_detach() calls, preventing potential Netdata crashes during thread creation. (#16760, @ktsaou)
  • Fixed a bug that caused "maximum number of cgroups reached" messages to spam logs. (#16730, @ilyam8)
  • Fixed incorrect service file location during MacOS installation: now, launchctl commands can reliably start and stop Netdata. (#16693, @ilyam8)
  • Fixed a bug that caused the Netdata claiming process to fail on macOS due to an inaccessible netdata-claim.sh script. (#16686, @ilyam8)
  • Fixed missing host label streaming from child nodes: host labels are now transmitted reliably to parent nodes. (#16821, @stelfrag)
  • Fixes a bug in clock resolution calculation that prevented some data collection plugins from working correctly. (#16720, @ktsaou)
  • Fixed a bug that caused Netdata to crash when calculating database size due to missing or single datafiles. (#16699, @ktsaou)
  • Fixed a bug that caused the cups.plugin to not terminate upon receiving a SIGPIPE (Broken Pipe) signal. (#16691, @ilyam8)
  • Fixed a reference counting issue that could lead to Netdata crashes. (#16687, @ktsaou)
  • Fixed charts context and family definitions of exporting engine. (#16683, @ilyam8)
  • Fixed a bug that could cause crashes when processing web requests. (#16664, @ktsaou)
  • Fixed improper handling of the dbengine event loop during shutdown. (#16658, @stelfrag)
  • Fixed a potential memory corruption issue in database code. (#16654, @stelfrag)
  • Fixed "response too big" error for Systemd-journal: addressed limitations by raising the maximum web response size. (#16649, @ktsaou)
  • Fixed compilation issues with --disable-dbengine: addressed errors that prevented successful builds when this flag was used. (#16611, @stelfrag)
  • Fixed labels corruption due to duplicate key/value pairs. Additionally, addressed logging errors that occurred during fatal Agent exits. (commit, @ktsaou)
  • Update go.d.plugin to v0.58.0. (#16725, @ilyam8).

Acknowledgements

We would like to thank our dedicated, talented contributors who make up this amazing community. The time and expertise that you volunteer are essential to our success. We thank you and look forward to continuing to grow together to build a remarkable product.

  • @boxjan for fixing incorrect label source for apps.plugin charts.

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1900 engineers are already using it!

v1.44.1

12 Dec 18:54
Compare
Choose a tag to compare

Netdata v1.44.1 is a patch release to address issues discovered since v1.44.0.

This patch release provides the following bug fixes and updates:

  • Fixed an issue in the uninstall script that prevented log2journal and systemd-cat-native from being removed (#16585, @ilyam8).
  • Fixed a bug that caused the debugfs.plugin to not terminate upon receiving a SIGPIPE (Broken Pipe) signal (#16569, @ilyam8).
  • Fixed memory leak during host chart label cleanup (#16568, @stelfrag).
  • Fixed incorrect cpu architecture/ram/disk values in build info (#16567, @ilyam8).
  • Fixed a bug that prevented the parent from accepting streaming connections on systems with one CPU core (#16565, @stelfrag).
  • Make the systemd-journal mandatory package on Centos 7 and Amazon linux 2 (#16562, @tkatsoulas).
  • Fixed crash on reading memory clock speed of an AMD graphics card (#16561, @MrZammler).
  • Fixed an unhandled error that occurred when setting file capabilities in the Debian postinst script of the perf.plugin (#16558, @tkatsoulas).
  • Fixed an issue where the user's netdata home directory was set to an incorrect value (#16548, @ilyam8).
  • Added the lightweight text editor to the Docker image (#254, @tkatsoulas).

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1700 engineers are already using it!

v1.44.0

06 Dec 18:15
Compare
Choose a tag to compare

Table of Contents

Steady to our schedule, this is another great Netdata release!

Important

Stay informed about upcoming changes and potential deprecations by reviewing the deprecation notice sections. This will help you plan for any necessary adjustments to ensure a smooth transition.

Netdata Growth

  • 66k+ GitHub Stars ⭐
    Since October 2023, Netdata is leading the observability category in the CNCF landscape, surpassing Elasticsearch. Thank you for your love ❤️! Give Netdata a ⭐ too, on GitHub!

  • 600M+ docker hub pulls
    Netdata runs with about 200k docker hub downloads per day. Since June 2023 we are a Verified Publisher, so that Netdata pulls don't count against docker hub pull limits for our users, allowing all our users to integrate Netdata to their CI/CD toolchains.

Release Summary

  • Netdata beats Prometheus in all aspects: this version of Netdata includes significant improvement allowing Netdata to be a lot more performant than Prometheus, at scale. Full performance analysis included.
  • Netdata Journal Logs: Netdata can now deal with huge systemd-journal databases and is available for the host logs when Netdata runs in a container.
  • First beta version of Netdata's log2journal: a utility to extract, convert, transform and send to systemd-journal any kind of structured logs (including JSON and logfmt logs), similar to what promtail does for Loki.
  • More Netdata Functions: monitor containers and VMs, network interfaces, mount points, block devices, systemd units, systemd services, and more!
  • Netdata now logs to journal instead of log files and the results are amazing!

Release Highlights

Netdata beats Prometheus in all aspects

image

We tested Netdata and Prometheus at scale, both ingesting 2.7 million metrics per second. On the same workload, Netdata vs Prometheus needs:

  • 35% less CPU
  • 49% less RAM
  • 12% less bandwidth
  • 75% less disk space
  • 98% less disk I/O

Read the full performance comparison between Netdata and Prometheus.

To achieve these astonishing results, we made the following changes to Netdata since the previous release:

New SLOTS streaming protocol

A new streaming protocol, allows Netdata children and parents to share a common index of the metrics streamed, allowing the parents to receive metrics without consulting hashtables, reducing the overall overhead on parents by about 30%, without increasing the overhead on children (the children just number each metric).

The new protocol, called SLOTS, is automatically selected when both the child and the parent support it.

Streaming compression algorithms

Streaming now supports multiple compression algorithms. Previous Netdata releases supported only LZ4, which is known for its speed and average compression ratio. This release adds support for ZSTD, GZIP, and BROTLI.

ZSTD provides the best balance between compression ratio and CPU consumption, and therefore it is now the default.

The compression algorithms selection order can be configured on parents, in stream.conf, at the [API] section (parents), by setting compression algorithms order = zstd lz4 brotli gzip.

If you need to save most bandwidth at the expense of CPU utilization set this so that brotli or gzip appear first in the list, before zstd and lz4.

This also means that parents can now have a different compression order for each API key, allowing the use of different API keys based on the location of the child (i.e. children that are on billable egress bandwidth can use an API key that prefers the best compression, like brotli and gzip, while children on non-billable egress bandwidth can use an API key that prefers the best CPU utilization, like zstd or lz4).

Gorilla compression beta

Gorilla compression is a time series data compression technique, developed by Facebook for their time series database, Gorilla. It's particularly efficient for compressing data that changes incrementally over time, which is a common characteristic of time series data.

This release of Netdata includes an adaptation of Gorilla compression, which once enabled, provides 30% additional memory reduction to Netdata.

This was not ready when we compared Netdata and Prometheus, so the Gorilla compression benefits weren't accounted in the comparison. By enabling Gorilla compression, Netdata memory reduction is 70%+ compared to Prometheus.

To try Gorilla compression, edit netdata.conf and set at the [db] section, dbengine page type = gorilla.

Keep in mind that enabling Gorilla compression changes the dbegnine file format to Gorilla compressed metrics. This version of Netdata can read Gorilla-compressed data from dbengine even if Gorilla compression is not enabled, but previous versions of Netdata cannot read it. So, enable Gorilla, only if you don't plan to switch back to a previous version of Netdata.

Our plan is to have Gorilla compression enabled by default at the next release of Netdata.

systemd-journal logs

Our systemd-journal.plugin was already quite faster (10x) than journalctl, but still it was slow when the journal databases is huge (e.g. at journals centralization points where hundreds or thousands of nodes push their logs).

In this release, we introduce several changes to allow the plugin to work promptly in such environments.

Sampling and estimations

The biggest performance issue with systemd-journal logs is the query performance when dealing with huge logs databases.

To overcome this performance issue and provide prompt responses to queries, Netdata now uses the following strategy:

  1. The latest 500k log entries read from journal files work like before: we read all of them and all the values for all their fields, so that we can have accurate histograms and counters per field value at the filters.
  2. Once we hit the 500k log entries limit on a single query, we turn on sampling and estimations.
  3. Sampling distributes 500k more log entries to all the journal files to be read, so that the total log entries queried for their field values will be 1M. This means that if we have to read 100 files, 10k log entries per file will be sampled and 10k log entries more will be unsampled. Since files are usually spread over time, this provides a good sample across time.
  4. When the sampling threshold is hit, Netdata continues reading more log entries without querying the values of the fields. These log entries appear as [unsampled] at the histogram. We know these log entries are there, but the value counters on the field filters do not include them.
  5. When the [unsampled] threshold is hit, and we have read more than 1% of each file, Netdata estimates the number of entries that will be read from the file and skips the rest of it. This estimation appears as [estimated] in the histogram.

The above process allows Netdata to provide a histogram of the logs in a timely manner, even when the number of log entries in the visible timeframe is several dozen million.

A similar process is usually used by log management systems, including Grafana Loki and Elasticsearch. However, Netdata takes a much bigger sample of the data (other systems usually sample only a few thousand log entries, while Netdata usually samples more than a million) and the visualization allows exposing the exact sampling and estimations made at the histogram.

Image showing [unsampled] and [estimated] on a systemd journal system that collects about 10k nginx log entries per second:
image

Read more about journals query performance.

journals scan

On busy logs centralization servers, the number of journal files available in /var/log/journal/remote can grow significantly, slowing down directory listing (even ls -l is very slow on them).

To overcome this issue, Netdata now uses inotify events and sorts the files to be scanned from the latest to the oldest.

These change...

Read more

v1.43.2

30 Oct 15:49
Compare
Choose a tag to compare

Netdata v1.43.2 is a patch release to address issues discovered since v1.43.1.

This patch release provides the following bug fixes and updates:

  • Fix rrdlabels type (1676de2, @stelfrag).
  • Fix label copy to allow new keys with different values (6179213, @stelfrag).
  • Fix internal label source propagation when streaming metrics (60cd86d, @ktsaou).
  • Speed up queries when sending alerts to Cloud on parents with a large number of alerts per child (f80f0fc, @MrZammler).
  • Fix filtering when selecting multiple fields in systemd-journal plugin (750ca8e, @stelfrag).
  • Fix an issue where parents were missing chart labels of child instances (240f9e7, @ktsaou).
  • Fix an issue where updated labels were not propagated to parents (644d432, @stelfrag).

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1700 engineers are already using it!

v1.43.1

26 Oct 14:17
Compare
Choose a tag to compare

Netdata v1.43.1 is a patch release to address issues discovered since v1.43.0.

This patch release provides the following bug fixes and updates:

  • Prevent wrong optimization armv7l static build (#16274, @stelfrag).
  • Fixed pattern matching in Functions Search (#16264, @ktsaou).
  • Fixed an issue where the query planner was using the wrong dbengine tier that had no data for the selected time period (#16263, @ktsaou).
  • Fixed invalid payload in Discord notifications (#16257, @luchaos).
  • Fixed possible deadlock on discovery thread shutdown in cgroups plugin (#16246, @stelfrag).
  • Fixed duplicate chart labels (#16249, @stelfrag).
  • Fixed dimension HETEROGENEOUS check (#16234, @stelfrag).
  • Updated go.d plugin version to v0.56.3 (#16228, @ilyam8).
  • Fixed calculation of dbengine statistics on 32bit systems (#16222, @stelfrag).
  • Improved handling of duplicate labels (#16172, @stelfrag).
  • Improved cleanup on shutdown of collectors (#16023, @ktsaou)

Acknowledgements

We would like to thank our dedicated, talented contributors that make up this amazing community. The time and expertise
that you volunteer are essential to our success. We thank you and look forward to continuing to grow together to build a
remarkable product.

  • @luchaos for fixing Discord notifications.

Support options

As we grow, we stay committed to providing the best support ever seen from an open-source solution. Should you encounter an issue with any of the changes made in this release or any feature in the Netdata Agent, feel free to contact us through one of the following channels:

  • Netdata Learn: Find documentation, guides, and reference material for monitoring and troubleshooting your systems with Netdata.
  • GitHub Issues: Make use of the Netdata repository to report bugs or open a new feature request.
  • GitHub Discussions: Join the conversation around the Netdata development process and be a part of it.
  • Community Forums: Visit the Community Forums and contribute to the collaborative knowledge base.
  • Discord Server: Jump into the Netdata Discord and hang out with like-minded sysadmins, DevOps, SREs, and other troubleshooters. More than 1700 engineers are already using it!