Location: Bangalore
Employment Type: Full-Time
Experience: 10+ Years
Department: Data & Analytics

About the Role

We are seeking a highly skilled and experienced Senior Data Engineer to design, develop, optimize, and maintain scalable data platforms and high-performance ETL/ELT pipelines across Microsoft Azure and Snowflake.

The ideal candidate will have strong expertise in Python, SQL, PySpark, Azure Databricks, Azure Data Factory, Snowflake, Apache Airflow, Azure Synapse, and modern data architecture. This role requires a hands-on technical leader who can work closely with architects, business stakeholders, DevOps teams, and data consumers to deliver reliable, scalable, secure, and cost-effective data solutions.

Key Responsibilities

• Lead the design, development, testing, deployment, and maintenance of high-performance ETL/ELT pipelines using Azure and Snowflake.
• Design and optimize modern Azure Lakehouse architectures using ADLS Gen2, Delta Lake, Azure Databricks, and Azure Synapse Analytics.
• Build scalable data ingestion frameworks supporting:
  • Batch data processing
  • Real-time/streaming data
  • API-based ingestion
  • External data sources
• Develop and optimize complex SQL queries, stored procedures, analytical functions, and data transformation logic.
• Perform performance tuning and cost optimization across Spark jobs, Snowflake warehouses, SQL pools, and data pipelines.
• Develop scalable data models supporting BI, analytics, reporting, and machine learning use cases.
• Implement robust data quality, validation, governance, metadata, and lineage processes.
• Work with Microsoft Purview and Collibra for data cataloging, classification, governance, and lineage.
• Integrate APIs, streaming platforms, IoT sources, and other external data systems.
• Implement automated testing using unit tests, integration tests, Great Expectations, and dbt tests.
• Design and maintain CI/CD pipelines using GitHub Actions, Azure DevOps, and Jenkins.
• Implement Infrastructure as Code using Terraform or Bicep.
• Monitor and troubleshoot production data systems using Azure Monitor, Log Analytics, and other observability tools.
• Collaborate with DevOps and Platform Engineering teams to automate infrastructure and application deployments.
• Troubleshoot complex issues involving distributed systems, cloud infrastructure, networking, data processing, and data platforms.
• Establish and improve engineering standards, development practices, automation, CI/CD, and IaC processes.
• Conduct code reviews and mentor junior and mid-level data engineers.
• Maintain comprehensive architectural, technical, and operational documentation.

Required Skills & Qualifications

• 5+ years of professional experience in Data Engineering or a related field.
• Strong programming skills in Python, SQL, and PySpark.
• Advanced hands-on experience building and optimizing ETL/ELT pipelines using:
  • Azure Data Factory (ADF)
  • Azure Databricks
  • Apache Spark
  • Delta Lake
  • Apache Airflow
  • Azure Functions or Azure Synapse Pipelines
• Experience integrating APIs, streaming data, and external data sources.
• Strong understanding of data governance and data quality frameworks.
• Hands-on experience with Microsoft Purview and/or Collibra.
• Experience implementing CI/CD pipelines using GitHub Actions, Azure DevOps, or Jenkins.
• Experience with Infrastructure as Code using Terraform or Bicep.

Good to Have

• Experience with Azure Kubernetes Service (AKS) and/or Dockerized workloads.
• Experience developing high-performance APIs using FastAPI, Flask, or Django.
  • Azure Event Hub
  • Azure Stream Analytics
  • Apache Kafka
  • OneLake
  • Lakehouse
  • Data Pipelines
  • Warehouses

Senior-Level Behavioral Expectations

• Demonstrate a strong ownership mindset and proactively resolve technical challenges.
• Translate complex business and technical requirements into clear, actionable engineering tasks.
• Promote engineering excellence, continuous learning, experimentation, and innovation.
• Mentor team members and contribute to the development of engineering best practices.
• Take responsibility for the reliability, scalability, security, and performance of data platforms.

What We Offer

• Opportunity to work on modern Azure, Snowflake, Lakehouse, and Data Engineering technologies.
• Exposure to large-scale data platforms, analytics, cloud, streaming, and AI/ML workloads.
• A collaborative and technically driven work environment.
• Opportunities to take ownership of challenging data engineering initiatives.
• Career growth and continuous learning opportunities.

Apply now and join our team to build the next generation of scalable and intelligent data solutions.