pal.tech logo

User interface web solution to automate data ingestion process/pattern for different source system to different target system.

Solution

  • Designed and built 5 screens (Monitor, Ingest, Patterns, Connections, Settings) using React,NextJs, FastAPI and Python
  • Built pre and custom connector to export/import data from different type of system (file, Database)
  • Built code using python to generate dynamic Ingestion DAGS (such as SQL queries ,connectors and pattern) in Airflow based on user inputs
  • Implemented OIDC for multiple repositories
  • Implemented dynamic rendering with configuration
  • Scheduled Spark job to transfer data between different data sources using airflow
  • Implemented Audit, Balance and Control to log each of the tasks in airflow to MongoDb

Key Features

  • Created standardized framework for all repositories including SSO and UI elements
  • Transferring data between databases and file systems.
  • Monitoring the logs of ingestion schedules.
Key Metrics

70% reduction of Ingestion developed cost and time

90% of Ingestion process is automated

50% Reduction in maintenance and resource cost

Single platform for 5+ databases and 10+ Ingestion patterns

Technology Stack