This version is all about improving stability and accuracy.
- A couple of performance tweaks have been implemented. The first is data compression which saves disk space and network throughput and allows our engine to run just a bit faster. The second is a small enhancement to the library used to upload files to S3 (before COPYing to redshift).
- One critical issue of data duplication in the Restream queue has been introduced in an intermediate version (due to Kafka compression) and immediately fixed.
- A couple of safety mechanisms have been added for better error reporting failure recovery.
- Elasticsearch now supports authentication. All you need to do is just add a user and password.
- ODBC-based Inputs (MSSQL, Oracle, etc) received a few enhancements:
- Composite primary keys are now supported seamlessly. When you choose incremental replication, our input uses the primary keys along with the update indicator to track changes to the table. If a replicated table has a composite primary key, it will be detected and used automatically. If a table does not have primary keys defined at all, but you can identify a column which acts as a primary key - contact us. We will configure a custom primary key for your special table.
- Two bugs have been fixed: An ODBC based input will now retry after network connection errors, and a syntax error has been fixed in one the queries used for Oracle databases.
- Google Adwords now reports more descriptive error messages to the Dashboard, and a minor UI bug in the input configuration form has been fixed.
- A Salesforce bug has been fixed. In scenarios where a user defined a large number of objects to replicate (>30), requests issued to the Salesforce API would time out. You may now configure as many Salesforce Objects as you'd like.
- Several Inputs Platform improvements have been applied. The first took care of a unicode decoding bug, which affected non-unicode events retrieved by any type of input. The second solved a situation where some inputs failed to report notifications to the Dashboard. The third bug fix solved a case where an ODBC based input would freeze when trying to transmit an overly large batch of events.
- A rare yet critical bug was fixed in inputs.alooma.com, preventing a data loss situation.
- We've calibrated our auto-mapper to map the metadata fields of events more accurately. Till now the auto-mapper would sample values of those fields and make a mapping decision based on those fields. Now the auto-mapper has a strict schema for mapping every metadata field, as well as discard the ones which are not needed. This change also fixed a rare case where a metadata field was mapped to a column with a "not null" constraint.
- The auto-mapper now supports mapping event types to a table with non SQL-safe characters (which require quoting).
- The auto-mapper now also supports some uncommon PostgreSQL data types (hstore, tsvector), as well as one very common one (bool).
- The Mapper will now show the final destination table for event types which undergo consolidation. Till now the name of the change log table (usually denoted with a _log suffix) was displayed.
- We fixed two problems causing data discrepancies. The first was a race condition in clearing the change-log table after consolidating it to the destination table. The second was a logical bug in tracking the cycles of fully dumped/loaded tables.
- A minor UI issue causing spontaneous navigation to the Mapper view. You might have encountered this, and probably felt confused. We hope this makes it right.
- Consecutive Dashboard notifications are now more tightly grouped.
- Navigating out of the Code Engine will no longer ask for confirmation when no changes have been made to the code.
As a growing company, balancing fast progress and stable scaling, we strive to offer our users the best and most reliable data platform as a service out there. Your success in driving intelligent, data-backed decisions in your company, is our success. Let us know how we can help.