Postgres RDS and MongoDB Improvements
- Last release we had big improvements to report for MySQL - this release we show that Postgres RDS and MongoDB get the same kind of love!
- Cleaner Schema Baselines - We've cleaned up the schema for both Postgres RDS and MongoDB by pulling the table columns/collection fields out of the data dictionary and putting them in the root of the event itself.
- Postgres RDS'
_metadata.timestampfield has been renamed to
_metadata.consolidationto reflect its usage as the key for consolidation, and also to disambiguate from the timestamp of the event.
- MongoDB got a new metadata field called
_metadata.ordinal, which we are using to disambiguate events with the same timestamp, to make the consolidation process more reliable.
mongoprefixes from the metadata fields have been removed.
- Postgres RDS'
- MongoDB auto-detection of new collections - Your data is best served fresh! Not only can Alooma pull in all collections, but it can also detect when new collection are added to your MongoDB database and pull them in automatically. They'll show up as a new event type based on the name of the collection - just map the collection and go! Note that until the collection is mapped its events will go to restream, so make sure to map it as soon as you can to avoid filling up the Restream Queue.
Flexible Schema - Your schema is likely to expand with new columns or fields in your Postgres RDS tables or MongoDB collections. From this release you'll see that new data show up in the matching table or collection's event type as new fields, unmapped and waiting for action. Once mapped, the consolidation query will automatically adjust to keep business as usual. Note that while partially mapped, events will go to Restream, and may cause discrepancies when consolidated out of order. So make sure to map it as soon as you can!
- Storm 0.10.0 Upgrade - After a bit of sweat and elbow grease, we've upgraded Storm from 0.9.2 to 0.10.0 (Check out the Storm Release Notes). That will mean better security, performance as we scale up, and stability, which will be passed directly to you!
- Mapper Filter Behavior - We've gotten some feedback (and felt ourselves!) that our mapper page's filtering behavior was a bit off. Now, when you click a filter to view only the mapped, unmapped, discarded, or invalid fields, you'll see all fields that match that filter (instead of filtering those fields away). You give us feedback and we listen - keep it coming and hope you enjoy the change.
- Invalid null bytes causing COPY errors - Redshift can't accept data that has a null byte at the beginning of a string. While this is a Redshift limitation (which, of course, can be addressed in our Code Engine), it was an issue on our side that this error was failing an entire batch of valid events. Now, we only fail the events with the null byte issue, which you'll find waiting in the restream for your changes.
- We found that if you took the action to discard unmapped fields, the UI stopped you from applying the mapping - no more!
- We fixed an issue where the type selection dropdown in the Mapper was empty if you tabbed into it or clicked on it after being on the column name.
- Since the LSF/Filebeat input allows you to set the input name, we were having issues with illegal filenames (anything that included text other than alphanumeric character or underscores). Now any illegal character will be replaced with an underscore by the stystem.
- Our metrics reported that the Generic Input was flowing when in fact it was other inputs that were flowing. This resulted in seeing the Generic Input box as blue when it shouldn't have been - that has been resolved.