Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Monitoring HDInsight Spark cluster on Azure with OMS

Today I would like to share the information with you on how to monitor an HDInsight Spark cluster on Azure with OMS.

HDInsightOMS01HDInsightOMS02

For this I just created an HDInsight Spark cluster with default settings and no further customization in my Azure subscription. After the successful creation, I looked into the solution gallery of my OMS workspace and had to recognize that only HDInsight HBase Monitoring (Preview) showed up.

HDInsightOMS03

A quick search on the Internet pointed me to the following GitHub repository.

-> https://github.com/hdinsight/HDInsightOMS

You can download there the Spark view and import it through the view designer into the OMS workspace.

Nevertheless there is an HDInsight Spark Monitoring solution available in the solution gallery, but you do not see it in the portal. You have to use the following PowerShell script to enable the solution for your OMS workspace.

Login-AzureRmAccount
$subscription=Get-AzureRmSubscription|Out-GridView -PassThru -Title "Select Azure subscription"
Select-AzureRmSubscription -SubscriptionId $subscription.Id -Verbose
$OMSworkspace=Get-AzureRmOperationalInsightsWorkspace|Out-GridView -PassThru -Title "Select OMS workspace"
$tempIP=Get-AzureRmOperationalInsightsIntelligencePacks -ResourceGroupName $OMSworkspace.ResourceGroupName -WorkspaceName $OMSworkspace.Name|Out-GridView -PassThru -Title "Select OMS solution"
Set-AzureRmOperationalInsightsIntelligencePack -ResourceGroupName $OMSworkspace.ResourceGroupName -WorkspaceName $OMSworkspace.Name -IntelligencePackName $tempIP.Name -Enabled $true -Verbose

HDInsightOMS04HDInsightOMS05

The next step is to connect to the HDInsight Spark cluster via SSH and run the following commands to install the OMS agent onto it.

HDInsightOMS07HDInsightOMS08

wget https://raw.githubusercontent.com/hdinsight/HDInsightOMS/master/monitoring/scriptspark.sh
chmod 777 scriptspark.sh
./scriptspark.sh workspaceid workspacekey

Do not forget to provide the OMS workspace id and the primary or secondary key, when you execute the shell script.

HDInsightOMS09

Afterwards you have to wait a couple of minutes, so the OMS agent can send the first data sets to the OMS workspace. But then the data should show up in the HDInsight Spark Monitoring solution view.

HDInsightOMS10HDInsightOMS11HDInsightOMS12


Posted

in

WordPress Cookie Notice by Real Cookie Banner