Remote Sr Data Engineer (Python, Airflow) (5600 USD/Mes) [Remote]

About The PositionOur ClientOur client leverage advanced AI to revolutionize patient recruitment in clinical trials. They utilize a modern, LLM-powered data stack to develop cutting-edge AI solutions, including helping patients find clinical trials they are a match for.Why our client is hiring for this position/missionAs a Data Engineer, you will be responsible for designing, developing, and maintaining robust data pipelines to support the AI-driven clinical trial solutions. You will work with modern data engineering tools such as dbt, dlt, Apache Airflow, Docker, and Kubernetes, ensuring scalable and reliable data workflows. You will integrate data pipelines with machine learning models and web application backends. Ideally, you've built a data product and you understand the resiliency and flexibility that is required for data pipelines that sit behind an evolving product, as opposed to those used only for analytics. You're excited about the possibilities that LLMs bring and have interest in multi-agent LLM systems.TeamYou will be part of a team of ~5 people, and will be reporting to the Manager.Responsibilities:

• Design, develop, and maintain robust data pipelines
• Ensure data pipelines are scalable, reliable, and efficient
• Monitor and optimize the performance of data workflows
• Work with dlt and dbt for data ingestion and transformation
• Use Apache Airflow or Cloud Composer for orchestrating workflows
• Implement containerization with Docker and orchestration with Kubernetes
• Manage code via GitHub and deploy solutions on Google Cloud Platform (GCP)
• Implement Continuous Integration/Continuous Deployment (CI/CD) practices
• Utilize Infrastructure-as-Code (IaC) tools to manage and provision infrastructure
• Collaborate with cross-functional teams including data scientists, software engineers, and clinical experts
• Integrate data pipelines with machine learning models, LLMs, and NLP frameworks
• Propose and implement improvements to existing data pipelines and infrastructure