<p”>In Part 1 of this series, Using Kibana and Elasticsearch to Examine Twitter Trends, we learned several things, including:</p”>
- <p”>Setting up an Elasticsearch cluster on qbox.io</p”>
- <p”>Installation of the Twitter river plugin to stream the data of our topic of interest, represented by a hashtag (in our case, #Marvel), to your Qbox Elasticsearch instance from Twitter.</p”>
- <p”>Familiarizing ourselves with the anatomy and structure of our data.</p”>
<p”>At this point, we are ready to learn the Kibana landscape. In this post, we will learn how to a</p”>ccess Kibana from the cluster, a<li”>ccess the Twitter data index we created in the previous post while familiarizing ourselves with the Kibana dashboard, and a</li”>nalyze the trending hashtags of our indexed Tweets using differrent forms of visualization such as pie charts, histograms, and tables
1. Accessing Kibana from the Qbox Cluster
In the Qbox dashboard, assuming you have enabled Kibana as instructed in the previous post, you will see an option to “View data with Kibana” (the highlighted option in the screen shot below).
Click on the “View data with Kibana” link, and a new tab will be opened like the one below.
After reviewing the screen shot above that shows some of the great features that we’ve already discussed, click on the “Blank Dashboard” link in the bottom-right of the page. From here, we can configure and create everything from the scratch. Of course, you can certainly play around with the other two options.
2. Introduction to the Dashboard
The starting point for your simple dashboard creation project will look like this:
We have highlighted with red rectangular boxes the 4 most important areas worth exploring, which are:
- The Settings button : As the name suggests, this is where we will configure basic settings like the index name, the name of the dashboard, access settings, etc.
- The Query Panel : This panel is used for querying into the indexed database.
- The Filter Panel : To narrow down the search further, we will apply filters from the queries, which will, in turn, trim the results to the topic we seek.
- The Add a row button : The name is self-explanatory. We will discuss in detail below.
- The Save Button : Remember to press this after every important step to avoid the loss of your progress.
3. Configuring our Kibana Dashboard
Let’s start with configuring the dashboard.
Naming and Theme Selection
First, we need to name our Kibana Dashboard. You can see from the previous screenshot that we named it “New Dashboard.” So click the settings button on the top right of the page, and a window will appear as below.
Under the general tab, which is also the default, we have options to choose both name and style. We named it “River Comics,” and we selected the dark theme.
Now that we have named the dashboard and given it a theme, the next job is to tell Kibana where our data of interest is located. We wish to operate on the “comics” index that we configured in the last tutorial. Just drop it down and select. Don’t forget to check the Preload Fields checkbox, which will allow Kibana to take all the field names in our index into its cache and will act as an autofill process in the upcoming selections.
The resulting operations on the indexed data are shown in default templates that can be placed as rows. We’ll use a simple pie chart to represent our data manipulations, so select the pie chart template from the rows tab in the dashboard settings.
Your first row will look like the picture above after adding “title” to the Row Column. Since we are going do to a pie chart analysis first, let’s name our row “Pie chart row,” as shown. Then, press the “Create Row” button in the right bottom corner. Presto! “Pie chart row” has been added. (Be sure to save!)
4. Viewing Trends in Pie Chart
Upon save, the Kibana page will now look like this screenshot:
Now, just below the green filtering tab, you will see a tab called “Add panel to empty row.” After we select that, we will be shown options to add various other panels.
Here we select “terms” patterns from the dropdown under the “Select Panel Type.” This will enable us with many more selectors and specifiers as shown in the following screenshot.
- “Title” : Here we are asked to name the panel of the row. In this example we called it “Pie Chart.”
- “Span” : The next column is for the span of the panel we are going to input. Since we are having only one panel, it would look better if it occupies the entire row. So the value has been set to 12. You can change the values according to the number of panels you intend to place in one row.
- “Terms Mode” : Here we have it as “terms” for simple analytics purposes.
- “Field” : Denotes the field type in which the operation has to be done. We can use any field that contains terms or text. I have chosen “hashtag.text” as the field here since it will give us the exact trending hashtags in the index we have created.
- “Length” : This indicates how many terms should be shown or taken in to account while creating the pie charts. If the value of length is 10, for example, the pie chart will have 10 divisions.
- “Order” : This field denotes which order should the sorting be done. Here we prefer the “count” order because it gives us the counts of the hashtag terms done. Reverse_count, terms, and reverse_terms are also there, and we will learn about them in subsequent tutorials.
View Options Row
- “Style” : The method of visualization is to be specified here. There are options such as pie chart, bar chart, line chart, table, etc. Here we chose pie chart.
- “Legend”: The names of the differrent sections of the chart are displayed if we enable this option. We have given it a top-of-panel position.
- “Legend Format” : A vertical value would place all the legends listed vertically as a column. Since that would consume space, I chose the horizontal format so we can display the legend names horizontally in a row.
Now we click “save” and have the pie chart ready as shown in the screenshot below.
What this pie chart says:
- We have the terms analysis on the index, Comics.
- The field on which we did the analysis is the “hashtag.text” field of all documents. (You can refer to the our preceding post if you have any doubt on the data structure.)
- We now see our pie chart displays our top 10 trending hashtags, all indicated by different colors.
5. Creating a Tweet Histogram
Histograms make sense in many particular cases, especially in time-series datam wo we once again click the “Add a row” button. You will see a now-familiar window that we first encountered in Step 3 above.
We will need to set a title for our new row, as we did before, and we’ll call it “Date Histogram.” You can see our previous row named “Pie Chart,” which is shown lying in the first row in the adjacent left column. Next, click the “Create Row” and then the “Save” buttons. The result will be an added empty row below your “Pie Chart” row, with an “Add panel to the row” button. Click the “Add Panel to row” button, and a window like the one below will pop up.
Select the option “Histogram” from the “Select Panel Type” dropdown box. The window will transform into a bigger one with a lot more configuration options, as in the screenshot below.
As with the Pie Chart creation scenario, a number of rows now need to be filled with required information.
- “Title” : Because we are doing date histogram analysis on the Tweets, we can name it as ”Date histogram of tweets.”
- “Span” : Just as with the pie chart creation, we set the span value as 12.
- “Chart Value” : We will set this field as count because we need to get the count of Tweets against a time field.
- “Transform Series” : Differrent types of transforms like normalizing the intervals to per-second, plotting the change per interval, and filling the gaps with zero are available here. For our purpose, we need to only check the zero fill for now.
Time Options Row
- “Time Field” : We need to apply the field of our indexed data which deals with a time stamp. In our “comics” Twitter data index ,the time stamp field is “created_at.” The “created_at” field gives us the exact time of Tweet creation.
- The rest of the fields can keep their default values because they are quite self-explanatory.
- These can be left to their default values and can be manipulated once you get the basics firm.
Now, click the “save” button. We have the date-histogram ready, as shown in the screenshot below.
This is a pretty clumsy graph to understand because the default interval period is set as to 1 year. In order to change that, we need to click the “Views” button marked by the red rectangle. A small panel will appear just below the views with many more options. Change the interval value from 1 year to 30 minutes from the interval dropdown hightlighted. Now the graph shows how many Tweets were fired on an interval of 30 minutes per day.
You can see the date and time details by simply hovering the mouse over the green bars in the resulting graph.
6. Field-Wise Analysis using Tables
Apart from the pie chart and the histogram, we have another powerful tool from Kibana that will be of immense help in analyzing the fields of an index.
The table tool allows us to do a comparative study of the fields of the same index. In this example, from our “comics” index, we wish to know the usernames of the persons who Tweeted, what they Tweeted, at what time they Tweeted, and in what language.
In order to start, we need to create a new row. This process should be familiar by now. Just click the “Add a row” button beneath the “Date Histogram” row. Name the row “Table View,” then click the “Create Row” and the “Save” buttons.
In the new row, which has appeared below the date histogram row, click on the “Add a panel to empty row.” The “Select Panel type” dropdown window will appear, from which we select the option “table”.
- “Title” : We can name the panel “Table comparison of fields.”
- “Span” : As in the case of pie chart creation and date histogram, we set the span value as 12 for better viewing.
- Leave the next rows as they are. You can see a “_score” option in the “sort” field of the “Options” row. It indicates that we are seeking all fields for our listing. For the time being, leave all the other fields as they are for now because we will be covering them in detail in future posts.
Hit the save button, and the following table will appear in the selected row.
In the above table there are 2 important areas numbered in red:
1. Fields: This column shows the entire number of fields available in a single document, and this is where we can select all the fields we want to show up in the list. There is a total of 41 fields in a single document, and all of them are listed in the fields column. Our goal for this exercise is do a comparison of username, his/her Tweet, the Tweeted time, and the language. These fields are “user.name”, ”text”, ”created_at” and “language,” respectievely. Select these four in the fields column.
2. Hide arrow: The arrow button is next to the field, marked as 2 in red. Clicking this would make the fields column hide towards the left and our desired rows remain.
Now we select the required fields and then press the arrow button (2). The table will be now rearranged like the one below.
Rearrangement of columns is also possible by just clicking on the left/right arrows on the corresponding column titles. This will push the columns to the required position.
Also clicking on any row element will show us the complete data of that user with all the field information.
In this tutorial we covered in detail the accessing and configuring of the Kibana dashboard and also three types of simple analytics methods, pie-chart representation of hashtags, the date-histogram of the Tweets, and the table comparison of the various fields. I have neglected a few details in the interest of simplicity, but we will examine them in more detail in future posts.