Today I would like to share the information with you on how to monitor an HDInsight Spark cluster on Azure with OMS.
For this I just created an HDInsight Spark cluster with default settings and no further customization in my Azure subscription. After the successful creation, I looked into the solution gallery of my OMS workspace and had to recognize that only HDInsight HBase Monitoring (Preview) showed up.
A quick search on the Internet pointed me to the following GitHub repository.
-> https://github.com/hdinsight/HDInsightOMS
You can download there the Spark view and import it through the view designer into the OMS workspace.
Nevertheless there is an HDInsight Spark Monitoring solution available in the solution gallery, but you do not see it in the portal. You have to use the following PowerShell script to enable the solution for your OMS workspace.
Login-AzureRmAccount $subscription=Get-AzureRmSubscription|Out-GridView -PassThru -Title "Select Azure subscription" Select-AzureRmSubscription -SubscriptionId $subscription.Id -Verbose $OMSworkspace=Get-AzureRmOperationalInsightsWorkspace|Out-GridView -PassThru -Title "Select OMS workspace" $tempIP=Get-AzureRmOperationalInsightsIntelligencePacks -ResourceGroupName $OMSworkspace.ResourceGroupName -WorkspaceName $OMSworkspace.Name|Out-GridView -PassThru -Title "Select OMS solution" Set-AzureRmOperationalInsightsIntelligencePack -ResourceGroupName $OMSworkspace.ResourceGroupName -WorkspaceName $OMSworkspace.Name -IntelligencePackName $tempIP.Name -Enabled $true -Verbose
The next step is to connect to the HDInsight Spark cluster via SSH and run the following commands to install the OMS agent onto it.
wget https://raw.githubusercontent.com/hdinsight/HDInsightOMS/master/monitoring/scriptspark.sh chmod 777 scriptspark.sh ./scriptspark.sh workspaceid workspacekey
Do not forget to provide the OMS workspace id and the primary or secondary key, when you execute the shell script.
Afterwards you have to wait a couple of minutes, so the OMS agent can send the first data sets to the OMS workspace. But then the data should show up in the HDInsight Spark Monitoring solution view.
