User interface web solution to automate data ingestion process/pattern for different source system to different target system.
Solution
- Designed and built 5 screens (Monitor, Ingest, Patterns, Connections, Settings) using React,NextJs, FastAPI and Python
- Built pre and custom connector to export/import data from different type of system (file, Database)
- Built code using python to generate dynamic Ingestion DAGS (such as SQL queries ,connectors and pattern) in Airflow based on user inputs
- Implemented OIDC for multiple repositories
- Implemented dynamic rendering with configuration
- Scheduled Spark job to transfer data between different data sources using airflow
- Implemented Audit, Balance and Control to log each of the tasks in airflow to MongoDb
Key Features
- Created standardized framework for all repositories including SSO and UI elements
- Transferring data between databases and file systems.
- Monitoring the logs of ingestion schedules.
Key Metrics
70% reduction of Ingestion developed cost and time
90% of Ingestion process is automated
50% Reduction in maintenance and resource cost
Single platform for 5+ databases and 10+ Ingestion patterns
Technology Stack
