Using ELK's JMX Plugin to gather JBoss EAP metrics

In this short blog post I outline how to use the ELK (ElasticSearch/LogStash/Kibana) stack to gather data via JMX from JBoss EAP 6.

Ingredients:

- ElasticSearch 1.4.0
- Logstash 1.4.2
- Kibana 3.1.2
- JBoss EAP 6.2

After having gone throught the downloading and installation process first we need to extract some libs out of the modules for JBoss EAP 6.2 that are needed for getting remote JMX calls to work.

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib create a file jboss_remoting.rb

module JMX
  module JBoss
    java_import java.lang.System

    class JBossRemoting

      sep = System.get_property 'file.separator'
      home = System.get_property 'jboss.home'

      if home != nil
        modulebase = [home, 'modules','system','layers','base','org','jboss'].join(sep)

        libs = [
          ['remoting-jmx','main'],
          ['remoting3','main'],
          ['logging','main'],
          ['xnio','main'],
          ['xnio','nio','main'],
          ['sasl','main'],
          ['marshalling','main'],
          ['marshalling','river','main'],
          ['as','cli','main'],
          ['staxmapper','main'],
          ['as','protocol','main'],
          ['dmr','main'],
          ['as','controller-client','main'],
          ['threads','main']
        ]

        for index in 0 ... libs.size
          Dir[modulebase + sep + libs[index].join(sep) + sep + "*.jar"].each {|file| require file }
        end
      end
    end
  end
end

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb after

require 'jruby'

        add

require 'jboss_remoting'

In logstash-1.4.2/config/logstash.conf I created the following configuration

input {
jmx{
    path => "logstash-1.4.2/jmx"
    polling_frequency => 30
    type => "jmx"
    nb_thread => 4
  }
}

output {
  elasticsearch { host => localhost }
}

and in logstash-1.4.2/jmx/jmx.conf I configured

{
  // JBoss Remoting JMX URL
  "url" : "service:jmx:remoting-jmx://localhost:9999",
  //username to connect to jmx
  "username" : "<your-user>",
  //password to connect to jmx
  "password": "<your-password>",
  "alias" : "jmx.instance1.elasticsearch",
  //List of JMX metrics to retrieve
  "queries" : [
    {
      "object_name" : "java.lang:type=Memory",
      "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "object_alias" : "Memory"
    }, {
      "object_name" : "java.lang:type=Runtime",
      "attributes" : [ "Uptime", "StartTime" ],
      "object_alias" : "Runtime"
    }, {
      "object_name" : "java.lang:type=Threading",
      "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
      "object_alias" : "Threading"
    }, {
      "object_name" : "java.lang:type=OperatingSystem",
      "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
      "object_alias" : "OperatingSystem"
    } ]
}

As the original JMX input plugin only supports configuration via host and port you need to apply the pull request 141 (https://github.com/elasticsearch/logstash-contrib/pull/141) to your local installation, i.e. by downloading https://raw.githubusercontent.com/jcordes73/logstash-contrib/master/lib/logstash/inputs/jmx.rb and save it to logstash-1.4.2/lib/logstash/inputs

Now you need to setup a JBoss management user via jboss-eap-6.2/bin/add-user.sh to reflect the username and password defined in jmx.conf above.

Because of a change in behaviour in regards to security you need to add the following configuration to elasticsearch-1.4.0/config/elasticsearch.yml

http.cors.enabled: true
http.cors.allow-origin: "*"

and also change http.enabled to true.

We need to set some environment variables so that the JBoss Remoting libraries can be found from an existing JBoss EAP installation:

JBOSS_HOME=/opt/jboss-eap-6.2
export JBOSS_HOME
JAVA_OPTS="-Djboss.home=$JBOSS_HOME"
export JAVA_OPTS

Now you can start ElasticSearch via elasticsearch-1.4.0/bin/elasticsearch and logstash via logstash-1.4.2/bin/logstash -f logstash-1.4.2/config/logstash.conf.

To finally see your JMX data via Kibana, open kibana-3.1.2/index.html with your favourite web-browser.