Summary: Introduction to various concepts related to using the Code Engine, including common scenarios for using the Code Engine, an overview of the functionality (lightweight Python IDE, importance of the
transform() function). Also includes information on working with multiple inputs and how to specify events from a certain input.
As Alooma receives events from your inputs, you can use the Code Engine to transform whichever events you choose before they are propagated to your target data warehouse.
There are several scenarios where transforming events on the fly with the Code Engine can be useful:
Cleansing values that are represented differently in a database; for example changing values of 0 or 1 to false or true, or rounding floats to integers.
Enriching data with the results of lookups from other sources, such as Geo-IP resolution.
Discarding undesired events.
Splitting up a single event into multiple events across different database tables, in order to make the data more modular and easily queryable.
Gathering metrics on incoming data.
The Code Engine is actually an in-browser Python (version 3) environment. This environment is a lightweight IDE for you to
provide data transformation code for received events. By default,
the transform function simply returns the received
def transform(event): return event
object is a standard Python dictionary, whose keys and values are
populated by the input source that sent the event. Fields in the
object can be accessed with
can be modified to your liking - fields can be added, modified, or
Because the environment is Python, you have access to the standard libraries included in Python 3. If you need access to a library or package not included in the Python defaults, please contact Support.
Remember when writing transforms in the Code Engine that you need to make transforms specific to your various inputs. In other words, you need a way to specify which input your transform should modify.
A good approach is to use a conditional that looks at one of the metadata attributes such as
input_type. You can use any field you want if it can specify an input. Keep in mind, though, that some inputs have fewer or different fields. For example, webhook events do not contain an
event_type field; however, you do know which
input_label is associated.
In this code snippet we look at the event metadata for the
input_label and if it matches our
mysql_stage input, then we run the
run_mysql_staging_transform function and pass it the event.
if event['_metadata']['input_label'] == 'mysql_stage': run_staging_transform(event)
Those are the basics - read on to learn about the Code Engine's features mentioned above.