In the previous article, we covered “painless” and provided details about its syntax and its usage. It also covered some best practices, like why to use params, when to use “doc” values versus  “_source” when accessing the document fields, and how to create fields on the fly, etc.  

We also covered topics like using painless scripting in a query context, filter context and topics like using conditionals in scripting, deleting fields/nested fields, accessing nested objects and usage of scripting in scoring etc. In this final "Painless" post, we explore how to use painless scripting in Kibana.

Previous Painless Posts

Kibana

One of the powerful component of the “ELK” stack is “Kibana”. Kibana is an analytics and visualization platform designed to work with Elasticsearch.  Kibana makes it easy to understand large volumes of data. It is a simple, browser-based interface that enables you to quickly create and share dynamic dashboards that display changes in data using Elasticsearch queries in real time.

Kibana looks for fields defined in Elasticsearch mappings and presents them as options to the user building a chart. Sometimes, a person doing analysis might need to create new fields either by combining existing fields, or by extracting a part of a field and use it for analysis. In these cases, do we ask the developer to reimport the data creating those fields needed for analysis, or is there any other way to achieve this?

Scripted Fields in Kibana

Kibana “Scripted Fields” comes to the rescue here. Using scripted fields, someone can create new fields for each documents and use them like other existing fields in the index. Support for scripted fields in Kibana is present since version 4. With the introduction of painless in Elasticsearch 5, it allows operating on a variety of data types thus making scripted fields in Kibana 5.0 much more powerful at the same time safe and secure.

Painless in Kibana has few restrictions when working with fields. Just like lucene expressions, it can only work on fields which has “doc_values” enabled. By default, all non-text data types are “doc_values” enabled in ES. To work with text data, make sure those are the type “keyword”. Painless cannot access the fields using “_source,” rather it should use the syntax “doc[‘field’]” when working with fields.

As scripted fields are computed on the fly, and are not stored in the ES index, you cannot search on them. However, you can make use of the scripted fields when filtering search results. These scripted fields are stored in “.kibana” index and it wouldn’t be available when directly working on the index using ES API’s outside of Kibana.

Data

Before we see how to create scripted fields in Kibana and find its usages, let’s import some data to ES:

curl -X POST   http://localhost:9200/_bulk   -d '{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"KRAIG","LastName":"NOWLAN","Designation":"Trainee","Salary":28000,"DateOfJoining":"2011-02-22","Address":"944 Tower Lane Nicholasville, KY 40356","Gender":"Male","Age":24,"MaritalStatus":"Married","Interests":"Parkour,Gyotaku"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"LACIE","LastName":"MALANDER","Designation":"Trainee","Salary":32000,"DateOfJoining":"2016-01-06","Address":"7822 Creek Ave. Mount Holly, NJ 08060","Gender":"Female","Age":25,"MaritalStatus":"Unmarried","Interests":"Computer activities,Storytelling,BoardGames,Legos,Planking,Tool Collecting"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"MADELEINE","LastName":"ELLEBRACHT","Designation":"Trainee","Salary":33000,"DateOfJoining":"2014-02-21","Address":"68 NE. Oklahoma Dr. Kennesaw, GA 30144","Gender":"Female","Age":24,"MaritalStatus":"Married","Interests":"Crochet,Scuba Diving,Mountain Climbing,Ziplining"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"BLANCA","LastName":"AMENTA","Designation":"Trainee","Salary":27000,"DateOfJoining":"2013-01-10","Address":"7438 Stillwater Drive West Chester, PA 19380","Gender":"Female","Age":23,"MaritalStatus":"Unmarried","Interests":"Airbrushing,Self Defense,Tatting,Collecting Antiques,Boomerangs,Hula Hooping"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"STEPHANE","LastName":"HARNIST","Designation":"Trainee","Salary":27000,"DateOfJoining":"2016-12-16","Address":"1 Mill Road Whitestone, NY 11357","Gender":"Female","Age":23,"MaritalStatus":"Married","Interests":"Yoga,Horse riding,Fast cars,Bowling"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"ELVA","LastName":"RECHKEMMER","Designation":"CEO","Salary":154000,"DateOfJoining":"1993-01-11","Address":"8417 Blue Spring St. Port Orange, FL 32127","Gender":"Female","Age":62,"MaritalStatus":"Unmarried","Interests":"Body Building,Illusion,Protesting,Taxidermy,TV watching,Cartooning,Skateboarding"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"JENNEFER","LastName":"WENIG","Designation":"President","Salary":110000,"DateOfJoining":"2013-02-07","Address":"16 Manor Station Court Huntsville, AL 35803","Gender":"Female","Age":45,"MaritalStatus":"Unmarried","Interests":"String Figures,Working on cars,Button Collecting,Surf Fishing"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"LAUREN","LastName":"RIDENS","Designation":"President","Salary":123000,"DateOfJoining":"2010-01-14","Address":"287 SE. Schoolhouse Street Clifton, NJ 07011","Gender":"Female","Age":63,"MaritalStatus":"Married","Interests":"Saltwater Aquariums"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"VANIA","LastName":"HAMMING","Designation":"President","Salary":127000,"DateOfJoining":"1970-01-05","Address":"871 Leatherwood Street North Canton, OH 44720","Gender":"Female","Age":65,"MaritalStatus":"Unmarried","Interests":"Golf,Weather Watcher,Fencing,Leathercrafting,Tutoring Children,Blogging,Building Dollhouses"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"GENARO","LastName":"HARDNETT","Designation":"Vice President","Salary":108000,"DateOfJoining":"2008-01-14","Address":"20 Sage Dr. El Paso, TX 79930","Gender":"Male","Age":59,"MaritalStatus":"Married","Interests":"Collecting Antiques,Hula Hooping,Airbrushing"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"CIERRA","LastName":"BRIGGERMAN","Designation":"Vice President","Salary":106000,"DateOfJoining":"1991-09-16","Address":"974 Boston Ave. Huntington, NY 11743","Gender":"Female","Age":47,"MaritalStatus":"Unmarried","Interests":"Diecast Collectibles,Tool Collecting,Stamp Collecting"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"IRA","LastName":"MOUTON","Designation":"Senior Software Engineer","Salary":64000,"DateOfJoining":"2013-01-17","Address":"25 County Rd. Benton Harbor, MI 49022","Gender":"Male","Age":32,"MaritalStatus":"Unmarried","Interests":"Rock Balancing,Railfans"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"HELENE","LastName":"ROCKWOOD","Designation":"Senior Software Engineer","Salary":61000,"DateOfJoining":"2007-01-10","Address":"313 NE. Sugar Street Mahopac, NY 10541","Gender":"Female","Age":31,"MaritalStatus":"Unmarried","Interests":"Floorball,Ziplining,Church/church activities"}
{"index":{"_index":"company","_type":"employees"}}
{"FirstName":"CECILE","LastName":"ROBLOW","Designation":"Senior Software Engineer","Salary":62000,"DateOfJoining":"2006-01-04","Address":"899 Acacia Ave. Smithtown, NY 11787","Gender":"Female","Age":35,"MaritalStatus":"Married","Interests":"Lasers,Collecting Swords,Ziplining,Texting,Tea Tasting,Casino Gambling,Calligraphy"}
'

Create Scripted Fields

  • Open Kibana from browser using the url http://localhost:5601. This article assumes kibana is running with default settings.

  • Navigate to the “Management” tab and select the index pattern for which new “scripted fields” need to be defined.

  • For this tutorial, create a new index pattern “company” as shown below:

painless1.png#asset:1517

Navigate to “scripted fields” tab:

painless2.png#asset:1518

Click “Add Scripted Field”:

painless3.png#asset:1519

Define Field:

  • Give a name for the field.

  • Choose the Language as “Painless”.

  • Select the data type in “Type” drop down. Depending on the type selected, “Format” dropdown provides options to change the display format.

  • Provide the painless script in “Script” field which defines how this new field needs to be formed.

  • Click “Create”.

painless4.png#asset:1520

The above steps can be repeated to create new scripted fields. All the scripted fields that are created can be later viewed and can be modified/deleted from the “Management” > “Index Patterns” screen.

painless5.png#asset:1521

Note: A few things to be aware of when creating “Scripted Fields”. As these scripted fields are created on the fly, they can be resource intensive have a direct impact on Kibana’s performance. There is no validation during the field creation, and scripts with issues throws exception during their usage in visualizations.

Learn about our Enterprise Kubernetes Development Support

Validate the newly created field and see how it appears in “Discover” interface. Navigate to the ‘Discover’ interface and expand the document in the “Documents Table”. You should be able to see the newly created field with the values created:

painless6.png#asset:1522

Other Example

Below is another example. Label the employees by “Age” group as “young”, “seniors” and “old”.  Create a new scripted field as “AgeGroup”:

painless7.png#asset:1523

Validate it in the “Discover” tab:

painless8.png#asset:1524

Filter the documents with “AgeGroup” as “young”.

painless9.png#asset:1525

Visualize the employee distrubition by “AgeGroup”:

painless10.png#asset:1526

Conclusion:

In this article we have seen how to make use of “Painless” in Kibana using “Scripted Fields”. This article concludes the “Deep Dive into Painless Series”. Questions/Comments? Drop us a line below. 

Give it a Whirl!

It's easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, or Amazon data centers. And you can now provision your own AWS Credits on Qbox Private Hosted Elasticsearch

Questions? Drop us a note, and we'll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.

comments powered by Disqus