In the previous article, we covered “Painless” and provided details about its syntax and usage. The article also covered some best practices, like why to use "params", when to use "doc" values versus  “_source” when accessing the document fields, and how to create fields on the fly etc.  

We also covered topics like using Painless scripting in a query context, filter context and topics like using conditionals in scripting, deleting fields/nested fields, accessing nested objects and usage of scripting in scoring etc. In this final "Painless" post, we explore how to use Painless scripting in Kibana.

Previous Painless Posts

Kibana

One of the powerful component of the ELK stack is Kibana. Kibana is an analytics and visualization platform designed to work with Elasticsearch.  Kibana makes it easy to understand large volumes of data. It is a simple, browser-based interface that enables you to quickly create and share dynamic dashboards that display changes in data using Elasticsearch queries in real time.

Kibana looks for fields defined in Elasticsearch mappings and presents them as options to the user building a chart. Sometimes, a person doing analysis might need to create new fields either by combining existing fields, or by extracting a part of a field and using it for analysis. In these cases, do we ask the developer to reimport the data creating those fields needed for analysis, or is there any other way to achieve this?

Scripted Fields in Kibana

Kibana “Scripted Fields” comes to the rescue here. Using scripted fields, you can create new fields for each documents and use them like other existing fields in the index. Support for scripted fields in Kibana was added since version 4. With the introduction of Painless in Elasticsearch 5, it allows operating on a variety of data types thus making scripted fields in Kibana 5.0 much more powerful and safe at the same time.

Painless in Kibana has few restrictions when working with fields. Just like Lucene expressions, it can only work on fields which have “doc_values” enabled. By default, all non-text data types are “doc_values” enabled in ES. To work with text data, make sure those are of the type “keyword”. Painless cannot access the fields using “_source,” rather it should use the syntax “doc[‘field’]” when working with fields.

As scripted fields are computed on the fly, and are not stored in the ES index, you cannot search on them. However, you can make use of the scripted fields when filtering search results. These scripted fields are stored in “.kibana” index and they wouldn’t be available when directly working on the index using ES API’s outside of Kibana.

Data

Before we see how to create and use scripted fields in Kibana, let’s import some data to ES:

curl -X POST   http://localhost:9200/_bulk   -d '{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"KRAIG","LastName":"NOWLAN","Designation":"Trainee","Salary":28000,"DateOfJoining":"2011-02-22","Address":"944 Tower Lane Nicholasville, KY 40356","Gender":"Male","Age":24,"MaritalStatus":"Married","Interests":"Parkour,Gyotaku"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"LACIE","LastName":"MALANDER","Designation":"Trainee","Salary":32000,"DateOfJoining":"2016-01-06","Address":"7822 Creek Ave. Mount Holly, NJ 08060","Gender":"Female","Age":25,"MaritalStatus":"Unmarried","Interests":"Computer activities,Storytelling,BoardGames,Legos,Planking,Tool Collecting"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"MADELEINE","LastName":"ELLEBRACHT","Designation":"Trainee","Salary":33000,"DateOfJoining":"2014-02-21","Address":"68 NE. Oklahoma Dr. Kennesaw, GA 30144","Gender":"Female","Age":24,"MaritalStatus":"Married","Interests":"Crochet,Scuba Diving,Mountain Climbing,Ziplining"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"BLANCA","LastName":"AMENTA","Designation":"Trainee","Salary":27000,"DateOfJoining":"2013-01-10","Address":"7438 Stillwater Drive West Chester, PA 19380","Gender":"Female","Age":23,"MaritalStatus":"Unmarried","Interests":"Airbrushing,Self Defense,Tatting,Collecting Antiques,Boomerangs,Hula Hooping"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"STEPHANE","LastName":"HARNIST","Designation":"Trainee","Salary":27000,"DateOfJoining":"2016-12-16","Address":"1 Mill Road Whitestone, NY 11357","Gender":"Female","Age":23,"MaritalStatus":"Married","Interests":"Yoga,Horse riding,Fast cars,Bowling"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"ELVA","LastName":"RECHKEMMER","Designation":"CEO","Salary":154000,"DateOfJoining":"1993-01-11","Address":"8417 Blue Spring St. Port Orange, FL 32127","Gender":"Female","Age":62,"MaritalStatus":"Unmarried","Interests":"Body Building,Illusion,Protesting,Taxidermy,TV watching,Cartooning,Skateboarding"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"JENNEFER","LastName":"WENIG","Designation":"President","Salary":110000,"DateOfJoining":"2013-02-07","Address":"16 Manor Station Court Huntsville, AL 35803","Gender":"Female","Age":45,"MaritalStatus":"Unmarried","Interests":"String Figures,Working on cars,Button Collecting,Surf Fishing"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"LAUREN","LastName":"RIDENS","Designation":"President","Salary":123000,"DateOfJoining":"2010-01-14","Address":"287 SE. Schoolhouse Street Clifton, NJ 07011","Gender":"Female","Age":63,"MaritalStatus":"Married","Interests":"Saltwater Aquariums"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"VANIA","LastName":"HAMMING","Designation":"President","Salary":127000,"DateOfJoining":"1970-01-05","Address":"871 Leatherwood Street North Canton, OH 44720","Gender":"Female","Age":65,"MaritalStatus":"Unmarried","Interests":"Golf,Weather Watcher,Fencing,Leathercrafting,Tutoring Children,Blogging,Building Dollhouses"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"GENARO","LastName":"HARDNETT","Designation":"Vice President","Salary":108000,"DateOfJoining":"2008-01-14","Address":"20 Sage Dr. El Paso, TX 79930","Gender":"Male","Age":59,"MaritalStatus":"Married","Interests":"Collecting Antiques,Hula Hooping,Airbrushing"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"CIERRA","LastName":"BRIGGERMAN","Designation":"Vice President","Salary":106000,"DateOfJoining":"1991-09-16","Address":"974 Boston Ave. Huntington, NY 11743","Gender":"Female","Age":47,"MaritalStatus":"Unmarried","Interests":"Diecast Collectibles,Tool Collecting,Stamp Collecting"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"IRA","LastName":"MOUTON","Designation":"Senior Software Engineer","Salary":64000,"DateOfJoining":"2013-01-17","Address":"25 County Rd. Benton Harbor, MI 49022","Gender":"Male","Age":32,"MaritalStatus":"Unmarried","Interests":"Rock Balancing,Railfans"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"HELENE","LastName":"ROCKWOOD","Designation":"Senior Software Engineer","Salary":61000,"DateOfJoining":"2007-01-10","Address":"313 NE. Sugar Street Mahopac, NY 10541","Gender":"Female","Age":31,"MaritalStatus":"Unmarried","Interests":"Floorball,Ziplining,Church/church activities"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"CECILE","LastName":"ROBLOW","Designation":"Senior Software Engineer","Salary":62000,"DateOfJoining":"2006-01-04","Address":"899 Acacia Ave. Smithtown, NY 11787","Gender":"Female","Age":35,"MaritalStatus":"Married","Interests":"Lasers,Collecting Swords,Ziplining,Texting,Tea Tasting,Casino Gambling,Calligraphy"}
'

Create Scripted Fields

  • Open Kibana from browser using the URL http://localhost:5601 or any other URL you use to access Kibana. This article assumes Kibana is running with default settings.

  • Navigate to the “Management” tab and select the index pattern for which new “scripted fields” need to be defined.

  • For this tutorial, create a new index pattern “company” as shown below:

painless1.png#asset:1517

After creating the index, navigate to “scripted fields” tab:

painless2.png#asset:1518

Click “Add Scripted Field”:

painless3.png#asset:1519

Define Field:

  • Give a name to the field.

  • Choose “Painless” scripting language.

  • Select data type in the “Type” drop down. Depending on the type selected, “Format” dropdown provides options to change the display format.

  • Provide the Painless script in “Script” field which defines how this new field needs to be formed.

  • Click “Create”.

painless4.png#asset:1520

The above steps can be repeated to create new scripted fields. All the scripted fields that are created can be later viewed and can be modified/deleted from the “Management” > “Index Patterns” screen.

painless5.png#asset:1521

Note: A few things to be aware of when creating “Scripted Fields”. As these scripted fields are created on the fly, they can be resource intensive have a direct impact on Kibana’s performance. There is no validation during the field creation, and scripts with issues throw exception during their usage in visualizations.

Validate the newly created field and see how it appears in “Discover” interface. Navigate to the ‘Discover’ interface and expand the document in the “Documents Table”. You should be able to see the newly created field with the computed value:

painless6.png#asset:1522

One More Example

Below is another example. Label the employees by “Age” group as “young”, “seniors” and “old”.  Create a new scripted field named “AgeGroup”:

painless7.png#asset:1523

Next, validate it in the “Discover” tab:

painless8.png#asset:1524

Filter the documents with “AgeGroup” as “young”.

painless9.png#asset:1525

Visualize the employee distrubition by “AgeGroup”:

painless10.png#asset:1526

Conclusion

In this article, we have seen how to make use of “Painless” scripting in Kibana using “Scripted Fields”. With Painless, you can combine various data fields to produce valuable aggregations of your data. These fields will be, however, available only at search time and won't be a part of your indexed documents. 

Give it a Whirl!

It's easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, or Amazon data centers. And you can now provision your own AWS Credits on Qbox Private Hosted Elasticsearch

Questions? Drop us a note, and we'll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.