Hashlist is a platform for projects within the automotive industry.
We establish supplier relationships with automotive companies and provide a one-stop shop for candidates interested in working in the industry.
Are you ready to embark on this journey?
Our client is seeking a Senior Data Engineer. By applying, you gain access to the Hashlist network, unlocking opportunities in automotive projects across the industry.
Responsibilities:
- Design and build a scalable data platform that can efficiently ingest, store, manage, and stream large volumes of data, simplifying analysis and enabling rapid development of high-quality data products and services.
- Implement and test low-latency, real-time data pipelines using cutting-edge big data technologies to support data-as-a-service products.
- Collaborate with cross-functional teams to clean, curate, and build end-to-end data pipelines based on requirements.
- Design and implement privacy and security controls into the CI/CD of each data pipeline.
- Ensure operational excellence by incorporating error handling, restartability, logging, monitoring, and alerting into the pipelines to maintain data consistency.
- Drive continuous improvements in reliability, performance, scalability, and quality, owning the associated KPIs.
- Provide on-call support for the platform and troubleshoot any issues that arise.
Qualifications:
- 5+ years of experience in developing and operating cloud-native data streaming systems, with a focus on scalability, low latency, data privacy, and quality control.
- Experience with cloud-native real-time streaming data technologies such as Spark, Flink, Kinesis, Lambda, Kafka, EMR/EKS platforms, and Lakehouse platforms (e.g., Delta.io).
- 5+ years of experience in optimizing AWS data system architecture, including Infrastructure as Code, deployment automation, and security.
- Proven experience in deploying and monitoring AWS infrastructure, particularly with MSK and EKS clusters in production environments.
- Proficiency in programming languages such as Spark-Scala, Py-Spark, Python, or Java.
- Strong knowledge of relational databases, SQL, SQL tuning, and NoSQL big data technologies.
- Experience with geospatial data processing is a plus.
- Experience in developing and deploying containerized applications using Docker, Kubernetes, and Helm.
- Proven ability to build systems that monitor data quality and prevent data losses.
- Strong understanding of data structures, algorithms, and operational excellence principles.
- Curiosity and passion for learning, with a strong bias for action.
Next Steps:
- Click "Apply."
- We will review your application.
- If qualified, you’ll be accepted into the Hashlist network and considered for this and other relevant projects.
