How to use FluentBit multiline

One day, my friend asked a question about how to use fluentBit (It’s popular in k8s ¹) to collect Java application logs. I had no idea how to do this at first, but finally the result seems good, so I want to give this tale to introduce the way I walk pasted.

The config visualizes

Usually, the log file pattern seems unified, it looks like

2022-04-17 03:10:42.381 INFO 28420 --- [restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...

You can easily use regex to collect each line into a structure. But when the app throws exceptions, you need to use multiline parser ²³. Briefly, it’s caught raw text read line by line, so each line needs to match rule, like this:

# rules   |   state name   | regex pattern                   | next state
# --------|----------------|---------------------------------------------
rule         "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"   "cont"
rule         "cont"          "/^\s+at.*/"                      "cont"

Each line pipeline

2022-04-17 03:10:43.578 ERROR 28420 --- [scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task.

java.lang.RuntimeException: cs
	at guru.sfg.brewery.config.TaskConfig.throwError(TaskConfig.java:25) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2022-04-17 03:10:43.591  INFO 28420 --- [restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2022-04-17 03:10:43.593  INFO 28420 --- [restartedMain] g.sfg.brewery.SfgBreweryUiApplication    : Started SfgBreweryUiApplication in 2.717 seconds (JVM running for 3.998)
2022-04-17 03:10:43.596  INFO 28420 --- [scheduling-1] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory

But how to? The official Doc is good, but in this case, in the default spring logging pattern, log message with break-line, and, in Windows, it’s \r\n, Me using windows, so be an attention

So, collusion below (remember when the up satisfied that one can be execute):

1. The start line is simple (\d{4}-\d{2}-\d{2}\s\d+\:\d+\:\d+.\d+)(.*) , see
2. next is tricky, see

• \r\n
• \s + name: default

3. and the next, loop 3, see

• \s + ...]
• xxx.xxx.xxException
• \s + at XXXX

In the end, the exception snippet will produce a single line, you can use filter to exclude this.

gist: https://gist.github.com/bxb100/de46e5f708d03d509430d4767806fb14

Other thing

Using docker needs packing by yourself ⁴, don’t forget it should with ES under same network ⁵

1. build an image

FROM fluent/fluent-bit:1.9.0
ADD parsers_multiline.conf /fluent-bit/etc/
ADD fluent-bit.conf /fluent-bit/etc/

docker build -t fluentbit-cs:sim .

docker run -id --name fluent -v /e/docker/fluentBit/log:/var/log --network 8x_default fluentbit-cs:sim

2. mount config to /fluent-bit/etc

docker run -id --name fluent -v /e/docker/fluentBit/log:/var/log -v /e/docker/fluentBit/config:/fluent-bit/etc --network 8x_default fluent/fluent-bit:1.9.0

If you are missing cont2 rule, the ES log will trigger exception like single document, it’s not combined with log message

Obviously It's not my wanted type