Logstash example with log4j input and JSON message content

Target

I want to use log4j SocketAppender to log into logstash. It is pretty straight since Logstash has log4j input type – supports internally log4j.
Additionally, I want that log4j message will be send in JSON format and in that form comes into Elasticsearch. That is pretty important for me since Elasticsearch can be easily queried afterwards.
So, let’s start.

Solution

Download Logstash

Download logstash from http://logstash.net
into the project directory called „logstash”. You can download Logstash in anywhere you want but then please use a config file from „logstash” directory. In the other case example will not work for you.

Start logstash

here is example for the version 1.3.3:
java -jar logstash-1.3.3-flatjar.jar agent -f logstash-log4j.conf -- web

Download project

you can clone in as git repo:
https://github.com/agorski/logstash-log4j-example.git
or download as ZIP file:
https://github.com/agorski/logstash-log4j-example/archive/master.zip

Execute Unit Test

just start terminal, go to project directory and execute following command:
mvn clean test

What the test do?
There are 2 Unit Tests, each creates 1 Log entry:

simple with a map content "msg" : { "first-name" : "Albert", "last-name" : "Gorski", "city" :"Berlin" }
complex with a map content which includes a sub-map: "msg": {"customer" : { "first-name" : "Albert", "last-name" : "Gorski", "city" : "Berlin" } }

Log entries, Maps, are converted to JSON String on the Java side and logged normally with log4j. Then send to Logstash by SocketAppender. Logstash gets field message, converts String to JSON format and saves it as msg field. It also mutate entry – all fields like message, path, priority, etc. are stripped – they are only a noise in my case. All stripped fields are not visible in Elsaticsearch / Kibana.
You can configure all this stuff in the logstash-log4j.conf config file.

Look at Kibana frontend to see log events

start browser you like and go to Kibana dashboard:
http://localhost:9292/index.html#/dashboard/file/logstash.json

Fun: Query Elasticsearch in Kibana frontend

Now you can query Elasticserach for JSON messages in Kibana like:

query simple entry with: msg.city:Berlin
query complex entry with: msg.customer.city:Berlin

Logstash Hero

wanna be Logstash hero? Read a fantastic book about Logstash: The Logstash Book by James Turnbull

Happy logging with Logstash!