Append Example Sets of Varying Structure/Schema
I am reading Tweets from S3 using the "Loop Amazon S3" operator. I have it working well (thanks, Marcel!) but the output of that process are several example sets. I could use the Append operator to combine them all, but due to the nature of Twitter's API, the Tweets can be of various structure/schema.
Question is: can I use an RM operator (like "Collect") to understand the distinct structures of my source data (which is JSON coerced into a kind of CSV format, by the way) then treat them differently? Ideally I would understand their structures, manipulate them and them combine the example sets based on a subset of common attributes in the data.
Thanks for the help!