Infometry Inc.
|Data Engineer
Bangalore, Karnataka, India
Summary
Led the development and implementation of enterprise data warehouse test automation, ETL/ELT migration, and metadata extraction tools, driving significant efficiency gains and data integrity improvements.
Highlights
Architected and implemented a comprehensive data warehouse automation workflow using Python and Snowflake SQL, enhancing data integrity through automated schema comparison, row/column validation, and consistency checks across enterprise platforms.
Developed an intuitive PyQt-based GUI with a zero-code interface, empowering non-technical users to manage and trigger test cases, which significantly reduced manual coding requirements and improved operational efficiency.
Orchestrated a complete ETL-to-ELT transformation by scripting and automating essential tasks, which improved data pipeline reliability and boosted operational efficiency by 40% with zero data loss.
Automated critical data ingestion processes into Google Cloud Storage (GCS) and BigQuery through the development and implementation of robust Shell and Bash scripts, significantly streamlining file transfers.
Developed Python-based tools for automated metadata extraction and parsing from Talend and DataStage XML files, enabling comprehensive data lineage tracking and discovery across multiple platforms.
Mentored team members in advanced Python, PyQt, and Snowflake SQL, establishing best practices for query writing and contributing to enhanced code quality and maintainability across the development lifecycle.
Engineered secure integrations with Snowflake SQL, enabling efficient execution of large-scale validation queries and reliably processing millions of records for critical data platforms.
Designed and implemented multiple UI modules within Snowflake for real-time data updates, validations, and robust user/access management, enhancing security with role-based features.