Senior Data Engineer (Azure & Databricks)
Senior Data Engineer – Azure & Databricks
Industry: Technology Consulting / Digital Transformation
Role Level: Lead II – Software Engineering
Location: Alpharetta, Georgia (USA)
Employment Type: Full-Time
About the Role:
We are seeking an experienced Senior Data Engineer – Azure & Databricks with a proven track record (8+ years) of designing, building, and scaling modern data platforms within the Azure ecosystem.
This role is ideal for someone who thrives in complex enterprise environments, is deeply hands-on with Databricks and Spark, and can lead end-to-end data engineering initiatives from architecture to implementation.
You will be responsible for building robust, high-performance data pipelines, establishing engineering best practices, and collaborating with cross-functional teams to deliver reliable, scalable data solutions across both batch and real-time streams.
Key Responsibilities:
- Architect, design, and implement scalable data platforms and pipelines using Azure and Databricks.
- Build and optimise ingestion, transformation, and processing workflows for batch and real-time (streaming) use cases.
- Work extensively with ADLS, Delta Lake, and Spark (Python/PySpark) to enable large-scale data engineering capabilities.
- Lead the development of complex ETL/ELT pipelines, ensuring performance, reliability, and code quality.
- Design conceptual, logical, and physical data models to support analytics and operational workloads.
- Work with relational and lakehouse systems, including PostgreSQL and Delta Lake.
- Define, implement, and enforce best practices for data governance, security, quality, and architecture standards.
- Partner closely with architects, data scientists, analysts, and business teams to translate requirements into scalable technical solutions.
- Troubleshoot production issues, drive performance optimisation, and support continuous platform improvements.
- Mentor junior engineers and contribute to the creation of reusable components and engineering standards.
Required Qualifications:
- 8+ years of hands-on data engineering experience in enterprise environments.
- Strong expertise in Azure services, particularly Azure Databricks, Azure Functions, and (preferably) Azure Data Factory.
- Advanced proficiency in Apache Spark with Python (PySpark).
- Strong SQL skills, including query optimisation and performance tuning.
- Deep experience with ETL/ELT methodologies, scheduling, and data orchestration.
- Hands-on expertise with Delta Lake (ACID, schema evolution, performance tuning).
- Strong understanding of data modelling (normalised, dimensional, lakehouse).
- Proven experience with batch and streaming technologies such as Kafka or Event Hub.
- Solid grasp of data architecture, distributed systems, and cloud-native design patterns.
- Ability to design and evaluate end-to-end technical solutions and recommend best-fit architectures.
- Excellent analytical, problem-solving, and communication skills.
- Ability to collaborate across teams and lead technical discussions.
Preferred Skills:
- Experience with CI/CD tools such as Azure DevOps and Git.
- Familiarity with Infrastructure-as-Code (Terraform, ARM).
- Exposure to data governance and metadata cataloguing tools (e.g., Azure Purview).
- Experience supporting machine learning or BI workloads on Databricks.
Benefits:
For Full-Time, Regular Employees
- Minimum 10 days paid vacation annually
- 6 days paid sick leave (prorated for new hires)
- 10 paid holidays
- Paid bereavement and jury duty leave
- Eligibility for 401(k) Retirement Plan with employer match
- Medical, dental, and vision insurance eligibility (employee + dependents)
- Company-paid:
- Basic life insurance
- Accidental death & disability coverage
- Short- and long-term disability
- Access to HSA and FSA programs (healthcare, dependent care, commuting)
Recommended Jobs
Office Representative (Sales and Customer Service)
Zan McBride - State Farm Agency, located in Fayetteville, GA has an immediate opening for an Office Representative (Sales and Customer Service). Account Manager. Insurance experience is not required …
Optometrist - Peachtree City, GA
We are on the lookout for a passionate and licensed Optometrist to join this dynamic team, whether you're a recent graduate eager to embark on your professional journey or a seasoned expert looking f…
Infant and/or Toddler Lead Teacher
Summary Isle of Hope Baptist Child Development Center is seeking a committed Early Childhood professional to work in a faith based Childcare Center. Applicant must organize and lead faith-based …
Residential Electrician with Mr. Electric of Atlanta
It’s an exciting time at Mr. Electric® of Atlanta with new, positive and supportive owners as of May 2021. Our culture is one where employees are well-trained, competitively paid and made to feel lik…
Localisation Project Manager
Are you interested in being a part of an international environment and making a difference at LanguageWire? Then you might be the one we are looking for! The role you’ll play You will be the …
Now Hiring: Passionate Toddler Educator
Summary We are searching for a nurturing toddler teacher to ensure that our students' basic and educational needs are met. You must prepare and deliver classroom activities, communicate with par…
Sales Account Rep II
: Job Family: Sales Support Type: Full time Date Posted: Nov 14, 2025 Req #: JR95113 Description Sales Account Rep II Location: Midland or Atlanta, GA; this is a hybrid position w…
Lieutenant
Valor Protection Safety Agency is seeking a confident, service-driven leader to serve as Lieutenant. This role plays a key part in leading daily security operations, supporting officers in the field,…
Class A Experienced CDL Driver Job
Class A Experienced CDL Driver Job Job Title: CDL Truck Driver Location: Savannah, GA Job Type:Part-Time or Full-time Company: Strategic Logistic Transportation Services, LLC Job Title: Experien…
Human Resources Generalist (HR)
Job Opening: Human Resources Generalist (HR) Company: Disinfectant Solutions of Atlanta Job Description Disinfectant Solutions of Atlanta, a leader in sanitation and disinfectant services, is e…