GRNET presents OMTD workflow stack in the Athens Docker meetup
It was a great honour and opportunity to interact with the Docker community during the meetup in Athens on November 29th. More than 30 people attended our talk ‘A scalable, virtual, flexible workflow infrastructure in OpenMinTeD stack’. The talk covered the software stack responsible for executing Text and Data Mining (TDM) workflows on a distributed cloud environment. The workflow setup greatly overlaps with (but is not limited to) modern containerization technologies and especially Docker.
About the Athens Docker meetup
The Athens Docker meetup is the Greek community of Dockers. Meetups (www.meetup.com) are very active and open social platforms for sharing knowledge, meeting people with common interests, and getting valuable feedback.
OpenMinTeD platform architecture and operations explored
Among the main points explored and focused on the OpenMinTeD platform architecture and operations, were:
- the fascinating challenges on setting up the workflow architecture of OpenMinTeD and the used open source Text Data Mining technologies incorporated in the project (i.e. Galaxy TDM workflow engine)
- the design choices of the architecture and the enhancements we applied after trials
- the competing technologies considered and tried out,
- compute and storage scaling that Text Data Mining content and services required
- scheduling of large amount of datasets and TDM requests of OpenMinTeD platform to the GRNET ~okeanos Cloud
- interoperability of services among TDM components and services running on the ~okeanos Cloud and OpenMinTeD platform
- monitoring and accounting of the system services
- security of authentication and authorization process and guarantee of secure and undisturbed service run
- performance of highly demanding OpenMinTeD Text and Data Mining services
Representing OpenMinTeD
Stavros and Nick were the main presenters, but also Byron (GRNET), Androniki and Dimitris (ARC) were there to talk about the project in general, as well as TDM tools and the forthcoming Open Call 2 for service providers (TDM tools/components/apps providers), and give each attendee useful OpenMinTeD memorabilia (notebooks, stickers, leaflets, etc).
Good responses from participants
The participants posed some very interesting questions, a sign that they understood the talk and identified common problems and issues relating to their own research and company work. More networking and Q&A happened during the beer session that followed afterwards. In conclusion, we took home some insightful comments and ideas about Kubernetes, NFS, horizontal scaling and parallelization of execution, that will overall help us optimize the architecture.
More information
Information about the meetup along with photos can be found here: https://events.docker.com/events/details/docker-athens-presents-a-scalable-virtual-flexible-workflow-infrastructure-in-openminted-stack-dockercon-recap
By: Byron Georgantopoulos for the GRNET team