Senior Data Enginner bei Swinerton

Compensation Range:

Job Description Summary:

Job Description:

POSITION RESPONSIBILITIES AND DUTIES

• Architect, optimize, and evolve data storage solutions and data models, including medallion architecture (bronze, silver, gold layers), for scalable and cost-efficient lakehouse platforms on Azure and Databricks.
• Integrate data from diverse internal and external sources, ensuring interoperability and consistency across platforms.
• Apply data governance, security, privacy, and compliance standards within engineering solutions, following organizational and regulatory guidelines.
• Design, build, and maintain scalable ETL/ELT pipelines for ingesting, transforming, and delivering data from diverse sources (structured, semi-structured, unstructured) using Azure Data Factory, Databricks, and related tools.
• Implement workflow orchestration tools (e.g., Airflow, dbt) for managing complex data workflows.
• Ensure data quality, consistency, and reliability through robust validation, monitoring, and error-handling processes.
• Implement privacy and security best practices in data pipelines (e.g., data masking, encryption, role-based access control).
• Engage with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that drive business value.
• Communicate complex technical concepts through clear, actionable insights and documentation tailored to both technical and non-technical audiences.
• Foster alignment and adoption of data engineering solutions by building strong relationships and ensuring business relevance.
• Mentor and support junior engineers, promoting a culture of learning and excellence.
• Optimize data engineering capabilities by leveraging existing and emerging tools and technologies, with a focus on performance, cost efficiency, scalability, data workflow management, and reliable deployment.
• Build and manage real-time / streaming data pipelines using Azure Event Hubs, Kafka, or Spark Structured Streaming.
• Collaborate with data scientists to enable seamless integration of data pipelines with analytics and machine learning workflows, including ML model deployment and monitoring.
• Complete other responsibilities as assigned.

MINIMUM SKILLS AND EXPERIENCE

• Bachelor’s degree in Computer Science, Engineering, or a related quantitative field required; Master’s degree preferred. Equivalent work experience will also be considered.
• 5+ years in data engineering or related roles, with a track record of delivering production-ready data solutions.
• Deep expertise in Azure data services (e.g., Azure Databricks, Azure Data Lake).
• Advanced experience with Databricks, including Spark, Delta Lake, and medallion architecture.
• Ability to connect business needs with technical capabilities, ensuring solutions are scalable and value-driven.
• Strong experience with data modeling, data warehousing, lakehouse design patterns, and data governance best practices.
• Proficiency in SQL and Python (including pandas, PySpark, or similar frameworks).
• Experience with real-time/streaming data solutions (e.g., Azure Event Hubs, Kafka, Spark Structured Streaming).
• Skilled in handling structured, semi-structured (e.g., JSON, XML), and unstructured data.
• Experience with CI/CD for data pipelines and infrastructure as code (e.g., Azure DevOps, GitHub Actions, Terraform).
• Experience with workflow orchestration tools (e.g., Airflow, dbt).
• Experience with containerization (Docker, Kubernetes) and/or serverless compute (Azure Functions) is a plus.
• Experience with GenAI/LLM integration (e.g., vector databases, RAG pipelines) is a plus.
• Excellent problem-solving, critical thinking, and communication skills; ability to work independently and collaboratively with cross-functional teams.
• Construction industry experience is preferred but not required.
• Proven ability to work independently in a remote or hybrid environment with minimal supervision, while adapting to shifting priorities and evolving business needs. 

Anticipated Job Application Deadline: