Senior Systems Developer (UKG) bei Corgan

At Corgan we are passionate about great design but the best thing we build is each other. Here, you’ll find camaraderie and collaboration as abundantly as curiosity and creativity. Corgan is a welcoming and supportive environment that provides significant career advantages. You’ll find room to grow, freedom to explore — and the safety to fail. Thank you for your interest in joining our curious, passionate, hardworking team.

Corgan is actively recruiting a dynamic Senior Data Engineer (Developer) to join the firm as part of our Shared Services MediaLab team. Tackle these responsibilities alongside a team of creative, highly motivated individuals who are passionate about their work and ready to collaborate to achieve outstanding results. Our ideal team member is passionate and progressively seeking more responsibilities to expand their expertise. This role will focus on designing, building, and maintaining scalable data infrastructure, real-time analytics systems, and AI-powered data pipelines that support our corporate strategic objectives. 

As a Senior Data Engineer, you will architect and implement comprehensive data solutions spanning cloud-native data lakes, real-time streaming platforms, and modern data warehousing systems. You'll work at the intersection of traditional data engineering and emerging AI/ML technologies, ensuring our data infrastructure can support both current analytical needs and future artificial intelligence initiatives. 

Development and data operations will require close collaboration with IT staff, data scientists, business analysts, and internal stakeholders across multiple departments. The role requires expertise in the entire Software Development Lifecycle (SDLC), encompassing requirements gathering, system design, implementation, testing, and production deployment. 

Data Architecture & Infrastructure 

• Design and build enterprise-scale data architectures supporting structured, semi-structured, and unstructured data across multi-cloud environments (Azure, AWS) 

• Implement scalable data lakes and data warehouses optimized for both batch and real-time analytics workloads 

• Develop and maintain data mesh architectures that enable self-service analytics while ensuring data governance and security 

• Architect cloud-native solutions leveraging serverless computing, containerization, and microservices patterns 

Data Pipeline Development 

• Build robust, fault-tolerant data pipelines using modern ELT methodologies and orchestration tools 

• Implement real-time data streaming solutions using Apache Kafka, Apache Pulsar, and cloud-native streaming services 

• Design and maintain automated data quality frameworks with comprehensive monitoring, alerting, and auto-remediation capabilities 

• Develop CI/CD pipelines for data engineering workflows, including automated testing, deployment, and rollback procedures 

AI/ML Integration & Advanced Analytics 

• Integrate machine learning workflows into data pipelines, supporting feature engineering, model training, and inference at scale 

• Implement MLOps practices including model versioning, A/B testing frameworks, and automated retraining pipelines 

• Build data infrastructure to support generative AI applications, including vector databases and retrieval-augmented generation (RAG) systems 

• Collaborate with developers, engineers, and data scientists to produce machine learning models and ensure scalable inference capabilities 

Data Governance & Security 

• Implement comprehensive data governance frameworks including data lineage tracking, metadata management, and data cataloging 

• Ensure compliance with data privacy regulations (GDPR, CCPA) and implement data masking, encryption, and access controls 

• Establish data quality standards and automated validation rules across all data assets 

• Design and maintain audit trails for data processing activities and model predictions 

Performance Optimization & Monitoring 

• Optimize data processing performance through query tuning, indexing strategies, and cost-effective resource allocation 

• Implement comprehensive observability solutions for data pipelines, including metrics, logging, and distributed tracing 

• Conduct root cause analysis for data quality issues and system performance bottlenecks 

• Establish SLAs for data freshness, accuracy, and system availability 

Collaboration & Knowledge Sharing 

• Collaborate with cross-functional teams, including developers, analysts, and business stakeholders, to understand requirements and deliver solutions 

• Provide technical leadership and mentoring to junior developers and analysts 

• Develop and maintain technical documentation, data architecture diagrams, and best practices guidelines 

• Lead technical design reviews and contribute to technology strategy decisions 

Qualified candidates should exhibit strong problem-solving, written, and verbal communication skills. 7+ years of experience in data engineering required. The candidate must have experience in dealing with internal clients. This candidate must have extensive experience with database development and maintenance. A bachelor’s degree in computer science, information technology, or another related field is preferred. In addition, qualified candidates will have experience with: 

• This role is an in-office position. 
• 7+ years of hands-on data engineering experience with a proven track record of building production-scale data systems5+ years of hands-on experience with Microsoft Dynamics 365 (CRM)  

• 5+ years of experience with cloud platforms (Azure, AWS), including data services and infrastructure management 

• 5+ years of advanced SQL experience including query optimization, performance tuning, and complex analytical queries  

• 3+ years of experience with big data frameworks (Apache Spark, Hadoop ecosystem, Databricks)  

• 3+ years of experience with real-time data processing and streaming technologies  

• Strong programming skills in Python, C#, and/or Scala with a focus on data processing and automation 

• Expert-level proficiency in SQL, NoSQL, and NewSQL databases (PostgreSQL, MongoDB, Cassandra, Snowflake)  

• Advanced experience with ETL/ELT tools and orchestration platforms (Apache Airflow, Azure Data Factory, Fabric Dataflow Gen2 and Data Pipelines, AWS Glue, dbt)  

• Deep understanding of data modeling techniques for both transactional and analytical workloads  

• Experience with data warehousing concepts including dimensional modeling, star/snowflake schemas, and slowly changing dimensions 

• Hands-on experience with cloud-native data services (Azure Synapse, AWS Redshift/Athena)  

• Proficiency with Infrastructure as Code (Bicep, Terraform, CloudFormation) and containerization (Docker, Kubernetes)  

• Experience with serverless computing architectures and event-driven data processing  

• Understanding of cloud security, networking, and cost optimization strategies 

• Expert-level Apache Spark development using PySpark, Scala, or Java 

  Zusätzliche Nebenleistungen