Google+ Followers

Tuesday, November 25, 2014

Using ELK's JMX Plugin to gather JBoss EAP metrics

In this short blog post I outline how to use the ELK (ElasticSearch/LogStash/Kibana) stack to gather data via JMX from JBoss EAP 6.

Ingredients:

- ElasticSearch 1.4.0
- Logstash 1.4.2
- Kibana 3.1.2
- JBoss EAP 6.2

After having gone throught the downloading and installation process first we need to extract some libs out of the modules for JBoss EAP 6.2 that are needed for getting remote JMX calls to work.

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib create a file jboss_remoting.rb

module JMX
  module JBoss
    java_import java.lang.System

    class JBossRemoting

      sep = System.get_property 'file.separator'
      home = System.get_property 'jboss.home'

      if home != nil
        modulebase = [home, 'modules','system','layers','base','org','jboss'].join(sep)

        libs = [
          ['remoting-jmx','main'],
          ['remoting3','main'],
          ['logging','main'],
          ['xnio','main'],
          ['xnio','nio','main'],
          ['sasl','main'],
          ['marshalling','main'],
          ['marshalling','river','main'],
          ['as','cli','main'],
          ['staxmapper','main'],
          ['as','protocol','main'],
          ['dmr','main'],
          ['as','controller-client','main'],
          ['threads','main']
        ]

        for index in 0 ... libs.size
          Dir[modulebase + sep + libs[index].join(sep) + sep + "*.jar"].each {|file| require file }
        end
      end
    end
  end
end

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb after

    require 'jruby'

            add
    require 'jboss_remoting'
    In logstash-1.4.2/config/logstash.conf I created the following configuration

    input {
    jmx{
        path => "logstash-1.4.2/jmx"
        polling_frequency => 30
        type => "jmx"
        nb_thread => 4
      }
    }

    output {
      elasticsearch { host => localhost }
    }

    and in logstash-1.4.2/jmx/jmx.conf I configured

    {
      // JBoss Remoting JMX URL
      "url" : "service:jmx:remoting-jmx://localhost:9999",
      //username to connect to jmx
      "username" : "<your-user>",
      //password to connect to jmx
      "password": "<your-password>",
      "alias" : "jmx.instance1.elasticsearch",
      //List of JMX metrics to retrieve
      "queries" : [
        {
          "object_name" : "java.lang:type=Memory",
          "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
          "object_alias" : "Memory"
        }, {
          "object_name" : "java.lang:type=Runtime",
          "attributes" : [ "Uptime", "StartTime" ],
          "object_alias" : "Runtime"
        }, {
          "object_name" : "java.lang:type=Threading",
          "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
          "object_alias" : "Threading"
        }, {
          "object_name" : "java.lang:type=OperatingSystem",
          "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
          "object_alias" : "OperatingSystem"
        } ]
    }

    As the original JMX input plugin only supports configuration via host and port you need to apply the pull request 141 (https://github.com/elasticsearch/logstash-contrib/pull/141) to your local installation, i.e. by downloading https://raw.githubusercontent.com/jcordes73/logstash-contrib/master/lib/logstash/inputs/jmx.rb and save it to logstash-1.4.2/lib/logstash/inputs

    Now you need to setup a JBoss management user via jboss-eap-6.2/bin/add-user.sh to reflect the username and password defined in jmx.conf above.

    Because of a change in behaviour in regards to security you need to add the following configuration to elasticsearch-1.4.0/config/elasticsearch.yml

    http.cors.enabled: true
    http.cors.allow-origin: "*"
    and also change http.enabled to true.

    We need to set some environment variables so that the JBoss Remoting libraries can be found from an existing JBoss EAP installation:

    JBOSS_HOME=/opt/jboss-eap-6.2
    export JBOSS_HOME
    JAVA_OPTS="-Djboss.home=$JBOSS_HOME"
    export JAVA_OPTS
    Now you can start ElasticSearch via elasticsearch-1.4.0/bin/elasticsearch and logstash via logstash-1.4.2/bin/logstash -f logstash-1.4.2/config/logstash.conf.

    To finally see your JMX data via Kibana, open kibana-3.1.2/index.html with your favourite web-browser.