Azure Data Engineer - Senior/Lead

Job Description

Job Description

Location: Alpharetta, Georgia

Experience:8 - 10 Years

Mandatory Requirement:

8+ years of hands-on data engineering experience in enterprise environments.
Strong expertise in Azure services, especially Azure Databricks, Functions, and Azure Data Factory (preferred).
Advanced proficiency in Apache Spark with Python (PySpark).
Strong command over SQL, query optimization, and performance tuning.
Deep understanding of ETL/ELT methodologies, data pipelines, and scheduling/orchestration.
Hands-on experience with Delta Lake (ACID transactions, optimization, schema evolution).
Strong experience in data modelling (normalized, dimensional, lakehouse modelling).
Experience in both batch processing and real-time/streaming data (Kafka, Event Hub, or similar).
Solid understanding of data architecture principles, distributed systems, and cloud-native design patterns.
Ability to design end-to-end solutions, evaluate trade-offs, and recommend best-fit architectures.
Strong analytical, problem-solving, and communication skills.
Ability to collaborate with cross-functional teams and lead technical discussions.

Skills: Java,Python,Spark

You Are:

We are seeking a highly skilled Senior Data Engineer to architect, design, and implement scalable data platforms and pipelines on Azure and Databricks . The ideal candidate will have extensive experience in large-scale data engineering, data modelling, and performance optimization, with a strong focus on building reliable, high-quality data solutions for analytics and operational workloads.

• Architect, design, and implement scalable data platforms and pipelines using Azure and Databricks .

• Build and optimize data ingestion, transformation, and processing workflows for both batch and real-time data streams.

• Work extensively with Azure Data Lake Storage (ADLS) , Delta Lake , and Apache Spark (Python/PySpark) for large-scale data engineering.

• Lead the development of complex ETL/ELT pipelines , ensuring high performance, reliability, and maintainability.

• Design and implement data models conceptual, logical, and physicalfor both analytical and operational use cases.

• Collaborate on data integration and migration projects, working with PostgreSQL, Delta Lake , and other relational or lakehouse systems.

• Define and enforce best practices in data governance, quality, and security.

• Ensure compliance with enterprise data standards and architecture frameworks.

• Troubleshoot production issues, optimize performance, and continuously improve platform efficiency.

• Collaborate with data scientists, analysts, architects, and business teams to translate requirements into scalable data solutions.

• Mentor junior engineers, promote coding and design best practices, and contribute to engineering standards and reusable components .

• Participate in code reviews, design discussions , and technical documentation .

• 8+ years of hands-on data engineering experience in enterprise environments.

• Strong expertise in Azure cloud services , including Azure Databricks , Azure Functions , and Azure Data Factory (preferred).

• Advanced proficiency in Apache Spark with Python (PySpark) .

• Deep understanding of ETL/ELT methodologies , data pipelines , and workflow orchestration .

• Proven ability to work with Delta Lake (ACID transactions, schema evolution, optimization).

• Solid background in data modelling (normalized, dimensional, and lakehouse models).

• Experience with both batch processing and real-time streaming (Kafka, Event Hub, or equivalent).

• Strong command of SQL , query optimization, and performance tuning.

• In-depth understanding of data architecture principles , distributed systems , and cloud-native design patterns .

• Ability to design end-to-end data solutions , evaluate trade-offs, and recommend best-fit architectures.

• Strong analytical, problem-solving, and communication skills.

• Proven ability to collaborate across cross-functional teams and lead technical discussions.

• Experience with CI/CD tools such as Azure DevOps and Git .

• Familiarity with Infrastructure as Code (IaC) tools Terraform , ARM templates , etc.

• Exposure to data governance and cataloging tools such as Azure Purview .

• Experience supporting Machine Learning (ML) or Business Intelligence (BI) workloads on Databricks.