Kedro
Reproducible data pipelines solved
IvaraX Analysis
Kedro is a well-established open-source Python framework under Linux Foundation governance that standardizes data pipeline development using software engineering best practices. It provides essential tooling for reproducible, modular data science and engineering workflows with strong cloud platform integrations and visualization capabilities.
Key Strengths
- +Strong governance under Linux Foundation (LF AI & Data) providing credibility and community stability
- +Proven production use at scale, evidenced by case studies from Telkomsel handling tens of TBs of data
- +Comprehensive ecosystem with visualization, data catalog, and extensive third-party integrations
- +Framework-agnostic deployment supporting multiple orchestrators and cloud platforms
- +Focus on software engineering best practices including testing, documentation, and code linting standards
Ideal For
- →Data science teams transitioning from exploratory notebooks to production-ready code
- →Organizations requiring standardized, reproducible ML pipelines across multiple team members
- →Enterprises with complex data engineering workflows needing flexible cloud deployment options
- →Teams already working with Python-based data tools seeking better pipeline management
Things to Consider
- !As an open-source framework, commercial support options may be limited compared to enterprise vendors
- !Requires adoption of Kedro's project structure and conventions, which may involve a learning curve for existing workflows
- !Best suited for Python-centric environments; teams using other languages may need alternative solutions
About Kedro
Kedro is an open-source Python framework hosted by the Linux Foundation (LF AI & Data) that enables data scientists and engineers to build production-ready data pipelines. The framework applies software engineering best practices to data engineering and data science code, making projects reproducible, maintainable, and modular. By providing standardized scaffolding for complex data and machine-learning pipelines, Kedro allows teams to focus on solving problems rather than managing tedious infrastructure concerns. The framework offers a comprehensive toolbox that includes pipeline visualization through Kedro-Viz, a lightweight Data Catalog supporting multiple file formats and cloud storage systems, and seamless integrations with popular platforms like Apache Airflow, Databricks, AWS SageMaker, and MLflow. Kedro supports flexible deployment strategies across single or distributed machines and provides dedicated IDE support for Visual Studio Code. With its dataset-driven workflow and automatic dependency resolution, Kedro has been adopted by major organizations including Telkomsel and Beamery for production-scale data operations.
Why Choose Kedro
- Open-source framework backed by the Linux Foundation (LF AI & Data) ensuring long-term community support and governance
- Comprehensive Data Catalog with lightweight connectors supporting major cloud providers (S3, GCP, Azure) and file formats (Pandas, Spark, Dask)
- Pipeline visualization tool (Kedro-Viz) that provides data lineage and facilitates stakeholder collaboration
- Standardized project templates that ensure consistent code organization across teams
- Flexible deployment options with integrations for Airflow, Kubeflow, Databricks, AWS Batch, and more
Services
Data Pipeline DevelopmentData EngineeringData ScienceMachine Learning Pipeline Development
Technologies
PythonApache SparkApache AirflowAmazon SageMakerAzure MLDaskDatabricksDockerJupyter NotebookKubeflowMLflowPandasVertexAIAWSGCPAzure
Tech Stack(detected from website)
PythonApache AirflowApache SparkDaskPandasJupyter NotebookpytestSphinxruffDockerAWS (S3, SageMaker, Batch)Azure MLGCP (VertexAI)DatabricksKubeflowMLflowfsspec
Industries Served
Notable Clients
TelkomselBeamery
Categories
Company Info
- Headquarters
- US
