Summary: Introduction to various concepts related to using the Code Engine, including common scenarios for using the Code Engine, an overview of the functionality (lightweight Python IDE, importance of the
transform() function). Also includes information on working with multiple inputs and how to specify events from a certain input.
As Alooma receives events from your inputs, you can use the Code Engine to transform whichever events you choose before they are propagated to your target data warehouse.
There are several scenarios where transforming events on the fly with the Code Engine can be useful:
Cleansing values that are represented differently in a database; for example changing values of 0 or 1 to false or true, or rounding floats to integers.
Enriching data with the results of lookups from other sources, such as Geo-IP resolution or currency conversion.
Discarding undesired events.
Splitting up a single event into multiple events across different database tables, in order to make the data more modular and easily queryable.
Gathering metrics on incoming data.
- Scrubbing PII (Personally Identifiable Information).
The Code Engine is actually an in-browser Python (version 3) environment. This environment is a lightweight IDE for you to provide data transformation code for received events. By default, the transform function simply returns the received
def transform(event): return event
event object is a standard Python dictionary, whose keys and values are populated by the input source that sent the event. Fields in the object can be accessed with
event object can be modified to your liking - fields can be added, modified, or removed.
Because the environment is Python, you have access to the standard libraries included in Python 3. If you need access to a library or package not included in the Python defaults, please contact Support.
The basics of using the Code Engine are simple. You enter code (transformations, usually) in the editor and then that code is run against incoming events. As you change the code (add new transformations, change or remove existing ones, create modules, etc.) you will probably want to test the functionality.
You can use the Get Sample button at the bottom of the interface to get a random event from the input, and then use the Run Test button to run your current code against the random event you selected.
You can also specify which kind of event you get for use when you run a test via the drop down menu next to Get Sample.
Remember when writing transforms in the Code Engine that you need to make transforms specific to your various inputs. In other words, you need a way to specify which input your transform should modify.
A good approach is to use a conditional that looks at one of the metadata attributes such as
input_type. You can use any field you want if it can specify an input. Keep in mind, though, that some inputs have fewer or different fields. For example, webhook events do not contain an
event_type field; however, you do know which
input_label is associated.
In this code snippet we look at the event metadata for the
input_label and if it matches our
mysql_stage input, then we run the
run_mysql_staging_transform function and pass it the event.
if event['_metadata']['input_label'] == 'mysql_stage': run_staging_transform(event)
Those are the basics - read on to learn about the Code Engine's features and how to perform specific common tasks.