Sunday, May 19, 2013

Storm Monitoring using JMX-JMXTrans-Ganglia

Though Storm supports a full-fledged UI, some applications where Ganglia is being used as a kind of universal tool for displaying the metrics of the nodes in clusters of various technologies present in the application, it's essential to have Storm cluster nodes also enabled to report their metrics to Ganglia. 

Since as of yet, there is no in-built support in Storm like we have in Hadoop, HBase etc that might enable its monitoring using Ganglia, we need to do that using JMXTrans. This post is about how to setup a JMX cluster and configure Storm Cluster nodes so that the target can be achieved.

Setting up JMXTrans: 

Follow the steps below to setup a JMXTrans cluster that would act as a bridge between   the Storm Cluster nodes and Ganglia and form the reporting channel for Ganglia.
  • Obtain the setup jmxtrans_20121016-175251-ab6cfd36e3-1_all.deb and extract it on all the machines of the Storm cluster
  • Copy /path_to_extracted_jmx_setup/jmxtrans_20121016-175251-ab6cfd36e3-01_all/data/usr/share/jmxtrans to /usr/share/jmxtrans 
  • Now any .json can be run using the following command 
/usr/share/jmxtrans/jmxtrans.sh start /path_to_json/example.json       
  • And jmxtrans can be stopped using 
/usr/share/jmxtrans/jmxtrans.sh stop                                                  

Configure the storm daemons to report to JMXTrans, add the following to ~/.storm/storm.yaml


“storm.yaml”- Supervisor Nodes


Add the following to the "conf/storm.yaml" on all the supervisor nodes of the cluster


worker.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID%"

Now, json file can send the metrics from ports 16700, 16701, 16702, 16703. Also add the following to report the metrics of the jvm running the supervisor


supervisor.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

“storm.yaml”- Nimbus


Specify just the following, however the presence of the above entries will not affect its performance.


nimbus.childopts: " -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=any_open_port_number"

Rest needs to be done in the json files. Storm cluster machines do not need ganglia monitoring daemons(gmond) to be running on all nodes, they can also report to a remote gmond. Finally, run the json files as mentioned above on each of the storm cluster nodes you want to monitor.


Sample JSON- Storm Workers



{
  "servers" : [ {
    "port" : "16700", <--- Defined Storm JMX Port 
    "host" : "127.0.0.1",  <--- Storm Worker
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",
        "settings" : {
            "groupName" : "workerMemory",
          "host" : "ip_of_gmond_server",
          "port" : 8649,
            "v3.1" : false
        }
      } ],
      "obj" : "java.lang:type=ClassLoading",
      "attr" : [ "LoadedClassCount", "UnloadedClassCount" ]
    } ],
    "numQueryThreads" : 2
  } ]

Sample JSON- Storm Supervisors



{
  "servers" : [ {
    "port" : "assigned_port_no for eg. 10000", <---Defined Storm JMX Port

  "host" : "127.0.0.1", <--- Storm Supervisor
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",
        "settings" : {
            "groupName" : "SuperVisorMemory",
          "host" : "ip_of_gmond_server",
          "port" : 8649,
            "v3.1" : false
        }
      } ],
      "obj" : "java.lang:type=Memory",
      "resultAlias": "supervisor",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
    }],
    "numQueryThreads" : 2
  } ]

Sample JSON- Storm Nimbus



{
  "servers" : [ {
    "port" : "assigned_port_no", <---Defined Storm JMX Port 
    "host" : "127.0.0.1",  <--- Storm Nimbus
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",
        "settings" : {
            "groupName" : "NimbusMemory",
          "host" : "ip_of_gmond_server",
          "port" : 8649,
            "v3.1" : false
        }
      } ],
      "obj" : "java.lang:type=Memory",
      "resultAlias": "nimbus",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
    }],
    "numQueryThreads" : 2
  } ]
}                                                                                                                                                           

With the help of the above JSON sample files your Storm Cluster nodes can start reporting their metrics onto the Ganglia Web UI. 
All the best !!!

6 comments:

  1. Hi,

    How can I report the configs visible in Storm UI (e.g nimbus config,topology summary,spout/bolt stats,etc.) to Ganglia?

    ReplyDelete
  2. Hi

    I am new to jmxtrans.

    As explained above configured storm.yaml along with Json configuration.
    Now ran jmxtrans.sh. However found that jmxtrans.sh refers to "jmxtrans-all.jar".

    So by running this jar will it be monitoring Storm services.

    I dont have any idea about JMX and Java. Trying to understand how it works.
    Also it would be great if you could give me some insight. Thanks in advance.

    ReplyDelete
  3. Thank you for sharing. If you are not a JMXTrans fan, there is also Storm monitoring in SPM - http://sematext.com/spm/ . It's easier to set up, I think, and shows all Storm metrics.

    ReplyDelete
    Replies
    1. Thanks for the link.

      :) No I am not a JMXTrans fan, but the time I had to work on this, JMX was seemingly the only option I was aware of. I guess now Storm itself also provides a monitoring API.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete