🎉 🎉 RAPIDMINER 9.10 IS OUT!!! 🎉🎉

Download the latest version helping analytics teams accelerate time-to-value for streaming and IIOT use cases.


Real-time consumer monitoring

HemsiHemsi Member Posts: 1 Newbie
I am currently researching various ways of setting up a real-time consumer monitoring tool which should entail the following: 
- using publicly available (daily or weekly) data from social media, forums, blogs, search engines, etc. 
- analyzing current and upcoming trends (eg. are there any topics which are growing in relevance)
- integrating macroeconomic data
- integrating demographic data 
- visualizing the data through dashboards and ideally integrate the analysis / reports into Salesforce 

Any thoughts on how to best approach that? Social media data of Facebook, Instagram has been restricted more and more after the API change, any workarounds? 
Would be great if you've have some ideas on how to approach that, what tools to use, etc. 


  • BalazsBaranyBalazsBarany Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert Posts: 692   Unicorn

    seems like you're setting up a new department in your company ;-)

    Yes, Facebook stopped giving out too much information after many data scandals. 

    But there are some basic questions you need to answer. Are you prepared to properly process personally identifying information according to the different laws? GDPR for EU residents, the Californian law etc.?
    Are you able to run the necessary infrastructure (scheduled process execution, in parallel if possible, a powerful database, ...)?
    Do you have a good way of matching your customers to external identities (Facebook etc.)? A method for breaking down statistical/macroeconomic/demographic data to the customers? Which problem are you trying to solve? Can you quantify it or find a success criterion?

    With external data sources you can forget real-time anyway. Most API interfaces are poll based, so you need to periodically check them. There are some services that have event based interfaces, though, mostly implemented through webhooks. This might be a use case for AI Hub, or you implement it with some other solution. 

    There are companies that do this and offer the customer's properties (if they know them) in an API, identified by the email address or other criteria. I've seen projects trying these but stopping because the data weren't useful.

    Feel free to ask when you have more focused questions I can help with.


Sign In or Register to comment.