The RapidMiner community is on read-only mode until further notice. Technical support via cases will continue to work as is. For any urgent licensing related requests from Students/Faculty members, please use the Altair academic forum here.
Server 9.2: Writing in parallel Loops can cause defect Repository structure
![land](https://us.v-cdn.net/6030995/uploads/userpics/457/nMGDU908VYP1G.jpg)
![](https://s3.amazonaws.com/rapidminer.community/vanilla-rank-images/unicorn-16x16.png )
Hi,
if you put a Store operator inside a parallized Loop, it can happen that directories and entries are duplicated with the same name multiple times. Therefore the Store operator needs to write into a non-existent repository directory. Depending on the timing the directory is created TWICE or even more (depending on number of threads/timing). In this case you have a directory with the same name multiple times. This creates very funny effects when accessing the data...
It seems that the creation of entries is not working synchronized, so that two threads are checking the existence in parallel, both getting a non-existend back and then both create the same entries.
This problem limits the parallizability drastically as we have to switch of parallel execution for all loops writing data!
Tagged:
1
Comments
thanks for the report. I've created an internal ticket and we will check for potential solutions.
A potential workaround in the meantime might be that you create the required folder structure before the parallelized loop operator - so the store operator only needs to persist the object itself.
Hope this helps,
Marcel
It would also be good to know which operating system and setting you are using for parallel execution:
Settings > Preferences > General > Parallel Execution
Best,
Marc
Regarding the settings. I don't know what the settings were, it was executed on the server in some queue. OS should be Linux, but I doubt it has anything to do with that.
Hi Sebastian,
thanks for the detailed response!
Sadly, we cannot provide a fix for this right now as a proper synchronization would require a major refactoring. In general, we are currently investigating other options to cover these kind of use cases.
Cheers,
Marcel
can you give me some details about the reasons and your plans to resolve that? If you meet an unhappy customer, who pays 60.000 € a year for a system that just lost his trust as the repository containing work artifacts for half a million euro project budget crumbled to pieces under a regular operation, it's better to have a real good explation at hand. "Too difficult" is astonishingly not a well accepted explanation in such situations. Even more if the customer is an IT affine person, it's hard to argue if the simple Studio repository is handling that without a problem as every single commonly used file system is thread safe.
Greetings,
Sebastian
Dortmund, Germany
How often does the development ask you for ideas. May it would be worthwhile.
Dortmund, Germany
as we also want to support the high availability setup of RapidMiner Server the synchronization/locking of the folders or entries would need to be performed outside of the JVM, e.g. within the shared DB. The locking of frequently used entries will decrease the performance of the repository operations which are used by the external executors. As we are doing some changes to the repository anyway, we will incorporate some new techniques which should solve the concurrency issue without decreasing the performance. But this kind of stuff cannot be delivered with a patch release and requires a new major version.
Cheers,
Marcel