Yes, exciting news today with the release of RapidMiner 7.6. Here are some highlights:
Studio: Sending notification emails now support all modern connection security and authentication mechanisms like TLS 1.2 + PFS, help panel text for the most used operators has been fully reviewed and explanations are now clearer and more useful, update Java for Windows and Mac to 8u141, lots of improvements in missing / error data handling
Server: Allow admin to set recursive folder permissions, improved performance of web services under heavy load, improved logging of installation process, implementation of secure email notifications (like Studio), update for Windows and Mac to 8u141, lots of improvements in missing / error data handling
Radoop: Support for standard and premium Microsoft Azure HDInsight, container re-use support for Hive-on-Tez as well as Hive-on-Spark, support for HiveServer2 High Availability, upgrade to Hadoop 2.8.1
So will there be 30 years of pain? Maybe. Will automation kill a lot of jobs, probably. Will the smart companies keep highly skilled workers in the next 30 years? Yes. In this new knowledge economy, brain power is key.
Today I read an interesting article about a speech that Jack Ma, founder of Alibaba, gave. In this speech he warned that the Internet and Robots will continue to disrupt the world we know. This disruption will cause social conflicts and we should get ready for 30 years of pain. He also said that if companies don't get on board with new technologies (i.e. cloud computing), they will die.
The most interesting comment he made was this:
He also warned that longer lifespans and better artificial intelligence were likely to lead to both aging labor forces and fewer jobs. “Machines should only do what humans cannot,” he said. “Only in this way can we have the opportunities to keep machines as working partners with humans, rather than as replacements.”
My colleagues and I do talk about how machine learning and this new startup space is disruptive. It's usually the "water cooler" banter but we talk about a living wage and how automation, in theory, will make society better.
I take a similar view, based on my experiences working with customers. For example, one customer I had managed a small group of R and Python programers. They did a lot of data science for their company. These data scienctists, programmers, and engineers all were busy but it got hard managing things.
They created one off scripts and tried to write systems to do ETL and modeling work, but there never was enough time to get all the projects done. They wanted a better way to automate, encapuslate R and Python scripts, and collaborate across the team.
How did they solve this problem? They ended up buying a few seats of Studio and a Server.
It took a few months to migrate a few of their processes to Studio and Server. They ended up dropping a lot of Python and R code into Building Blocks and automated a lot of ETL work. The teams started tossing models and processes back and forth to each other on the Server. REST API's were exposedquickly and put into use on their website.
In the end, they start clearing out the backlog of data science projects they had. The got more productive and did more with the same staff they had.
If I took Jack Ma's view of the world, I would say the company should automate more with RapidMiner and "let go" a few R and Python programmers. Fortunately I do not and neither did that company.
They kept the same anount of R/Python programmers but crosstrained a few to be RapidMiner Rock Stars. Then they tackled new projects and got back to doing more data science. Something they really enjoyed doing.
So will there be 30 years of pain? Maybe. Will automation kill a lot of jobs, probably. Will the smart companies keep highly skilled workers in the next 30 years? Yes. In this new knowledge economy, brain power is key. You would remiss to waste your talent on things that can be automated. This way you can free up their most valuable resource, time.