Devops-automation-tools Hybrid Jobs in Indien

Hybrid Excelra hiring Data Engineers - Remote Excelra hiring Data Engineers - Remote

Excelra · India · Hybrid

About the job

Excelra hiring Data Engineer : Hiring @ Excelra

Remote Role - www.excelra.com
  • strong on Python - Ready to take assessment on Python


    • Job Summary

    At Excelra ,We are seeking a highly skilled Data Engineer with expertise in Databricks, SQL, Python, and data flow pipelines. The ideal candidate will have experience in creating Azure Data Factory (ADF) pipelines, deploying code in Azure DevOps using Git Repos and YAML files, and exposure to Snowflake. This role involves designing, implementing, and maintaining robust data solutions to support various business needs.

    Key Responsibilities

    Databricks Expertise:

    Develop and maintain data processing workflows using Databricks.

    Optimize Databricks jobs for performance and cost efficiency.

    Implement data transformations and aggregations within the Databricks environment.

    Collaborate with data scientists and analysts to integrate and process large datasets.

    SQL Proficiency

    Write complex SQL queries to retrieve and manipulate data from various databases.

    Ensure data accuracy and consistency through effective querying and data validation techniques.

    Optimize SQL queries for performance and scalability.

    Python Programming

    Develop data pipelines and ETL processes using Python.

    Create reusable code modules and libraries for data processing tasks.

    Perform data cleaning, transformation, and analysis using Python-based tools and frameworks.

    Data Flow Pipelines

    Design and implement data flow pipelines to move and transform data across different systems.

    Ensure data integrity and reliability in data pipelines.

    Monitor and troubleshoot data pipelines to resolve issues promptly.

    Azure Data Factory (ADF) Pipelines

    Create and manage ADF pipelines to orchestrate data movement and transformation activities.

    Develop custom ADF activities using Azure Functions or other services.

    Implement logging, monitoring, and alerting mechanisms for ADF pipelines.

    Azure DevOps Deployment Pipelines

    Set up and maintain deployment pipelines in Azure DevOps.

    Use Git Repos for version control of data flow related code and Databricks notebooks.

    Create and manage YAML files for continuous integration and deployment (CI/CD) processes.

    Ensure seamless deployment of Databricks and other data-related code through Azure DevOps.

    Snowflake Exposure

    Work with Snowflake data warehouse to store and manage large volumes of data.

    Write and optimize SQL queries for Snowflake.

    Integrate Snowflake with other data processing tools and platforms.

    Qualifications

    Education:

    Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field. A master’s degree is a plus.

    Experience

    Proven experience as a Data Engineer or similar role.

    Hands-on experience with Databricks and Spark.

    Strong SQL Skills With Experience In Query Optimization.

    Proficiency in Python for data processing and automation.

    Experience in building and managing ADF pipelines.

    Familiarity with Azure DevOps, Git, and YAML for CI/CD processes.

    Exposure to Snowflake and understanding of its architecture and capabilities.

    Skills

    Strong analytical and problem-solving skills.

    Excellent communication and collaboration abilities.

    Ability to work in a fast-paced, dynamic environment.

    Attention to detail and commitment to data quality.

    Additional Desirable Skills

    Experience with other cloud platforms (AWS, GCP).

    Knowledge of data governance and security best practices.

    Familiarity with data visualization tools (e.g., Power BI, Tableau).

    Benefits

    Competitive salary and benefits package.

    Opportunities for professional growth and development.

    Collaborative and innovative work environment.

    Access to the latest technologies and tools.