Logging operator 4.0 launch | Cisco Tech Weblog

We’re blissful to announce a brand new main launch of the Logging operator.

It’s been some time since we wrote in regards to the Logging operator. Within the background, we have been actively engaged on superior new options to make Kubernetes logging as handy and easy as doable. Amongst its new options, one stands out because it brings basically new performance to the Logging operator: the choice to make use of syslog-ng because the central processing aspect of the Logging operator.

The preliminary plan for the Logging operator was all the time to be a collector-independent answer for Kubernetes. We selected Fluent Bit and Fluentd as our first strategy as a result of at the moment these have been probably the most extensively used instruments and the staff had a whole lot of expertise with them. We’re blissful that the Logging operator is prospering and reaching increasingly more customers. Naturally, this brings new challenges and we have to re-evaluate the way in which how the operator works.

We received an attention-grabbing case from a Logging operator person. They’ve comparatively massive clusters with an software set-up that requires round 100-150 flows to deal with the totally different sorts of purposes. Our preliminary routing implementation for Fluentd has its limitations:

• It’s a single-threaded Ruby plugin.
• As a result of nature of the operate, you may’t do batch processing: you want to make choices on a per-event foundation.
• Consequently, computation-intensive regex match filters of the logging flows are evaluated a number of instances for each incoming message.

Now we have labored intensively with this buyer to optimize their configuration and we managed to triple the efficiency of the log processing, nevertheless it turned clear that massive clusters produce a whole lot of log messages, and our optimizations round Fluentd gained’t have the ability to sustain with large-scale manufacturing site visitors. So, we needed to suppose exterior the field of Logging operator, and add a brand new software that may route and course of log site visitors at scale.

Why syslog-ng?

There’s an amazing variety of totally different instruments on the market, so we gathered a listing of necessities that we anticipated from the brand new software:

• Small footprint: Assets on the sting may be very restricted. We don’t need to affect the endpoint’s efficiency considerably.
• Intensive configuration choices: Logging operator solves complicated issues with a simplified configuration layer. That implies that a easy rule may be compiled into a number of small, associated configuration items. To realize that, the underlying language should present an excellent diploma of freedom. 
• Sizzling reload: It might appear insignificant to have the ability to reload the configuration at runtime, however in bigger clusters with frequent configuration adjustments sizzling reloading saves you from an enormous overhead. It is advisable to persist/restore information from the disk.
• Plugins: It’s nearly inconceivable to embed the whole lot into one software. Utilizing plugins permits us to simply lengthen the options with out having to recompile or rebuild the entire toolset. And we will even exclude pointless elements.
• Responsive and useful builders: A vibrant neighborhood of builders is without doubt one of the key values we seek for in a Third-party part. Fast fixes and up-to-date safety patches is a should to maintain the stack wholesome.
• Open-source: It goes with out saying that open-source software program has many advantages in relation to embedding in your software program stack.

Syslog-ng is a mature open-source log administration software that has been utilized by giant enterprises for over twenty years, with a broad characteristic set and glorious efficiency. It’s actively maintained and developed, has a big, worldwide person base, and is thought to work on virtually the whole lot, from e-readers to vehicles and airplanes.

We labored carefully with the syslog-ng builders and determined to roll collectively. Extending syslog-ng capabilities to run easily on prime of Kubernetes is a a lot smaller scope than evaluating brand-new instruments and doing thorough testing of the fundamental capabilities. We’re not saying that different instruments like Vector, Tremor, or OpenTelemetry Collector gained’t do the job however we’ve belief in syslog-ng.

Nonetheless, syslog-ng didn’t assist cloud-native Kubernetes use instances out of the field and confirmed some bizarre habits if its configuration was modified ceaselessly. Additionally, the format of its configuration wants some getting used to. To beat these issues, we’ve:

• Added new customized sources to the Logging operator which can be particular to syslog-ng. That approach you configure these CRs utilizing the acquainted Logging operator objects in YAML format, and Logging operator generates the syslog-ng configuration from these CRs routinely.
• Labored with the syslog-ng builders and the neighborhood so as to add Kubernetes assist to syslog-ng.
• Filed and glued bugs and points that got here up when working syslog-ng in Kubernetes, and improved its JSON assist.

Outcomes

Now we have labored with our clients and examined the syslog-ng assist in Logging operator of their setting to ensure it meets their necessities. Proper now, they’re already utilizing it in manufacturing, and the answer is steady and has solved their efficiency points.

Fluentd and syslog-ng, assist, and options

Fluentd stays totally supported in Logging operator, and Logging operator 4.0 is totally suitable with the older 3.x releases.

Logging operator can use Fluentd and syslog-ng as log forwarders to obtain, filter, and remodel the incoming logs, and to move them to a number of vacation spot outputs. Fluentd and syslog-ng have their separate capabilities they usually assist their very own vacation spot outputs (in fact, there are overlaps). At the moment, solely a restricted variety of syslog-ng outputs can be found in Logging operator, however we’re actively engaged on extending the record. Nonetheless, when you’re sending logs to Sumo Logic, or a generic HTTP output, you may need to attempt syslog-ng.

Superior routing instance

Routing with syslog-ng is way more versatile than with Fluentd. You possibly can match strings or common expressions to any a part of the message, for instance, to route a precise kind of message from a container. Like match solely the 4XX codes from an Nginx entry log.

match:
    and:
    - regexp:
        worth: json.kubernetes.labels.app.kubernetes.io/occasion
        sample: one-eye-log-generator
        kind: string
    - regexp:
        worth: json.code
        Sample: 4*
        kind: glob

We’re planning to make extra syslog-ng options obtainable from Logging operator sooner or later, and in addition so as to add new options to Logging operator based mostly on syslog-ng. For particulars, see the Logging operator documentation and out fast begin guides.

Different adjustments in Logging operator 4.0

A listing with all of the adjustments since 3.0.0 can be a bit overwhelming: we launched 3.0.0 nearly three years in the past, even the final minor launch (3.17.0) is over a yr outdated now. So to maintain issues quick and manageable, let’s see the notable adjustments and bugfixes for the reason that final patch launch (3.17.10):

Contributor acknowledgements

We’d wish to thank the syslog-ng builders for his or her time and help make syslog-ng prepared for the Kubernetes world. (For instance, making ARM builds of syslog-ng obtainable.)

After all, aside from the larger options, there have been many smaller additions and fine-tunings within the varied Logging operator plugins and outputs, and we’d like to specific our honest and utmost gratitude to all our contributors for actively supporting the challenge.

Logging operator strikes to a brand new place

In case you are watching the Logging operator challenge, you might need observed some main adjustments. The explanation for that is that the staff behind the Logging operator needed to spend extra time on the challenge. In consequence, Logging operator received its personal group, and the previous staff (now Axoflow) and Cisco will work collectively on the success of the challenge.

What has modified: