Target
I want to use log4j SocketAppender to log into logstash. It is pretty straight since Logstash has log4j input type – supports internally log4j.
Additionally, I want that log4j message will be send in JSON format and in that form comes into Elasticsearch. That is pretty important for me since Elasticsearch can be easily queried afterwards.
So, let’s start.
Solution
Download Logstash
Download logstash from http://logstash.net
into the project directory called „logstash”. You can download Logstash in anywhere you want but then please use a config file from „logstash” directory. In the other case example will not work for you.
Start logstash
here is example for the version 1.3.3:
java -jar logstash-1.3.3-flatjar.jar agent -f logstash-log4j.conf -- web
Download project
you can clone in as git repo:
https://github.com/agorski/logstash-log4j-example.git
or download as ZIP file:
https://github.com/agorski/logstash-log4j-example/archive/master.zip
Execute Unit Test
just start terminal, go to project directory and execute following command:
mvn clean test
What the test do?
There are 2 Unit Tests, each creates 1 Log entry:
- simple with a map content
"msg" : { "first-name" : "Albert", "last-name" : "Gorski", "city" :"Berlin" }
- complex with a map content which includes a sub-map:
"msg": {"customer" : { "first-name" : "Albert", "last-name" : "Gorski", "city" : "Berlin" } }
Log entries, Maps, are converted to JSON String on the Java side and logged normally with log4j. Then send to Logstash by SocketAppender. Logstash gets field message, converts String to JSON format and saves it as msg field. It also mutate entry – all fields like message, path, priority, etc. are stripped – they are only a noise in my case. All stripped fields are not visible in Elsaticsearch / Kibana.
You can configure all this stuff in the logstash-log4j.conf config file.
Look at Kibana frontend to see log events
start browser you like and go to Kibana dashboard:
http://localhost:9292/index.html#/dashboard/file/logstash.json
Fun: Query Elasticsearch in Kibana frontend
Now you can query Elasticserach for JSON messages in Kibana like:
- query simple entry with:
msg.city:Berlin
- query complex entry with:
msg.customer.city:Berlin
Logstash Hero
wanna be Logstash hero? Read a fantastic book about Logstash: The Logstash Book by James Turnbull
Happy logging with Logstash!