The Restream Queue is your safety net for ensuring zero data loss. The Restream Queue collects all the events that were not loaded to your target database, for whichever reason, making them easy to fix and enabling you to “restream” them into your target database.
Each event remains in the Restream Queue until:
You address the error which resulted in it being there.
You restream them to your target database.
The event is successfully loaded to your target database.
Learn how to utilize the Restream Queue.
The Restream Queue is at your disposal - but how do you best utilize it?
Identify that a problem exists - You may realize that events are going to the restream queue in a number of ways.
You may realize that events you expect to be in your target data warehouse don’t arrive. The Restream Queue should be the first place to check.
You may notice an increment in the Restream count on the plumbing page or in the event list of the Restream Queue page.
You may receive a digest email (configurable) which indicates events are erroring out.
Understand and address the problem(s)
The Restream Queue page will show you the events that have errors along with the error reason. You can sort and filter by the input the event came from, the event type, the type of error, timeframe, or do a free text search to understand the events.
Make the change in the Mapper, Code Engine, or your target data warehouse to address the source of the error. You can click to navigate directly to the relevant area in the Mapper or view the event as a sample in the Code Engine from the right-hand pane of an expanded event row.
Restream - Restream the events into your target database by clicking the Restream button in the Restream Queue screen.
Any event that still generates an error is simply stored in the Restream Queue again.
You can restream events as many times as you like without affecting the processing of newly arriving events. A metadata field
[metadata][restream_count]will indicate the number of times this event has been restreamed.
We recommend resolving the Restream Queue iteratively, meaning that you should first handle one or a small set of problems that caused a significant set of events to be sent to the Restream Queue, then move on to the next problem(s).
If your events load into your target data warehouse, but you decide you'd like to change the way that loading occurred, Alooma also has an S3 Retention feature that enables you to redo or change the loading of events that have already loaded into your target data warehouse.
Those are the basics of restreaming - read on about some common event error causes and solutions.
There are common event error causes and straightforward solutions for them.
The Notifications pane in the Alooma Dashboard and the email digests clearly describe the reason for each event’s error. Clicking on a notification expands it to show additional details.
Some event issues in the Restream Queue can be handled inside Alooma, such as changing the code in the Code Engine, changing your mapping in the Mapper, or updating the target data warehouse access credentials. Other event errors must be handled outside of Alooma, such as a table having been dropped from your target data warehouse or a data warehouse that is offline.
Here are some common reasons your events may be sent to the Restream Queue and their solutions:
New Event Type Needs Mapping – Occurs when a new event type has reached the mapper, either because you connected a new input or a new event type was created in the Code Engine. You'll need to go to the Mapper and map this event type.
Copy Failure Error – Loading error to the target data warehouse. For example, temporary database connection failure or changed access credentials. You may need to reconfigure Alooma’s data warehouse access configuration.
Type Conversion Error – The type of data in the event doesn’t match the column type to which the event is mapped. For example, when an event field arrives with a string value in a field that is mapped to an integer in the target database or when an event field value is too long. You may need to change the mapping in the Mapper.
Out-of-Range Error – This type of error generally occurs when the value of a field is out of range of the data type that field is mapped to, such as a small number appearing in a field which is mapped to timestamp, and that number is too small to be considered a timestamp as seconds from epoch. Fix this by modifying the value or discarding the event in the Code Engine.
Varchar Value Too Long – This occurs when the value of a field that is mapped to a varchar type of a particular length arrives as longer than the allowed length of the varchar. This is another type of "out-of-range", but specifically for varchar lengths. You can fix this by modifying or truncating the field value in the Code Engine, or choosing the "Truncate" option in the data type drop-down in the Mapper.
Redshift users: you can change the length of a varchar field directly in the Mapper UI. See Altering a Column via the Mapper.
Transform Function Error – Events that trigger an error in the code that you wrote in Alooma’s Code Engine. For example, if your code assumes that a specific field exists in an event, and it doesn’t. You may need to sample an event of this error type and modify your code accordingly in the Code Engine.
Unmapped Event Fields – Alooma lets you configure how strictly to handle new incoming event fields (meaning schema changes). According to your configuration, Alooma can be:
Strict – Automatically sends each event that contains an unmapped field to the Restream Queue.
Flexible – Each unmapped field of an event is discarded (ignored) and the rest of the event is handled according to the defined mapping.
Auto-mapping – Automatically maps event field changes and loads these events to the target database.
A sudden increase of events in the Restream Queue may indicate the source of the error, such as a Code Engine change, inputting a bad batch of events and so on. Keeping the Restream Queue less than 50% full may prevent it from overflowing, and keeps restreaming times short.
If you have questions or need help resolving any events in the Restream Queue, feel free to contact us anytime.
The Restream Queue page (accessible via the Restream Queue button at the top right of the Plumbing page, or via the left navigation pane) shows a breakdown of all the events in the Restream Queue.
Sortable and filterable by input, event type, error type, timeframe, or free text search.
Breakdowns of the contribution to the entire queue from different sources or errors, with a timeline chart.
The ability to see the error message and the event itself.
The ability to go directly to the relevant area in the Mapper or bring the event as a sample to the Code Engine.
First of all, don’t worry, your Restream Queue is monitored to ensure you are notified if it is starting to fill up beyond a certain threshold. This could be caused by importing a new source with a high volume of data and not mapping these fields to a table in your data warehouse, for example. Another potential reason events are sent to the Restream Queue is if a data warehouse table was deleted or Alooma cannot connect.
Although unlikely, if you still do not have a chance to address any of our emails, phone calls, IMs, or smoke signals and the Restream Queue does fill to capacity, any further events will be discarded. However, if S3 Retention is enabled, you can still recover your raw events and there will be no data loss.
We get it, not all events are created equal and not all must reach your data warehouse. You have a few options:
If there's a set of events which you want to clear out of the Restream Queue, add code to the Code Engine that identifies them by some attribute and then discards them. Then, restream. Note that newly arriving events with that attribute will also be discarded.
If you don't care about any of the events currently in the Restream Queue, because you've addressed the events in there that you do care about, you can click the restream button and then choose the Purge Restream option. Note this is a destructive operation and will permanently delete all events from the restream queue.