AI Test Automation Engineer bei IBM

Introduction

At IBM Infrastructure & Technology, we design and operate the systems that keep the world running. From high-resiliency mainframes and hybrid cloud platforms to networking, automation, and site reliability. Our teams ensure the performance, security, and scalability that clients and industries depend on every day. Working in Infrastructure & Technology means tackling complex challenges with curiosity and collaboration. You’ll work with diverse technologies and colleagues worldwide to deliver resilient, future-ready solutions that power innovation. With continuous learning, career growth, and a supportive culture, IBM provides the opportunities to build expertise and shape the infrastructure that drives progress.

Your role and responsibilities

We are looking for a Senior QA Engineer with 5+ years of overall QA experience, a strong foundation in traditional QA automation, and hands-on exposure to Machine Learning and Generative AI systems.

This role combines classical test automation (UI, API, regression, CI/CD) with modern AI quality engineering, focusing on ensuring accuracy, reliability, safety, performance, and trust of AI-driven features in production—especially in enterprise-scale platforms.

Key Responsibilities

Traditional QA Automation & Test Engineering

• Design, develop, and maintain robust automated test suites for web, API, and backend systems

• Automate UI tests using Selenium (mandatory) with maintainable page-object or screen-play patterns

• Build and maintain API automation for REST services ( integration, and regression tests using REST Assured)

• Own regression, smoke, and sanity suites with clear quality gates

• Integrate automated tests into CI/CD pipelines with reliable pass/fail signals

• Drive best practices in test pyramid, test data management, and test stability

• Analyze flaky tests and improve automation reliability and execution time

AI / ML Quality Engineering

Test LLM and GenAI features, including prompt behavior, response quality, hallucinations, edge cases, and failure modes

Validate RAG pipelines for retrieval accuracy, relevance, grounding, and citation correctness

Perform ML regression testing across model versions, prompt changes, and data updates

Design test strategies for non-deterministic outputs, including scoring thresholds, tolerance bands, and golden datasets

• Validate agentic or multi-step AI workflows where applicable

Non-Functional & Responsible AI Testing

• Validate performance characteristics: latency, throughput, concurrency, and scalability

• Test reliability and resilience under load and failure scenarios

• Validate security, privacy, and Responsible AI risks, including:

PII leakage

Prompt injection and jailbreak attempts

Bias, toxicity, and unsafe outputs

• Collaborate closely with backend, ML, SRE, platform, and product teamsRequired

Required technical and professional expertise

Skills & Experience

• 3-5 years of experience in software QA / test engineering

• Strong hands-on experience in traditional QA automation

• Proven experience with UI automation using Selenium (mandatory)

• Solid experience in API automation (REST Assured / JSON)

• Strong understanding of test pyramid, regression strategies, and CI/CD integration

• Core Java, Python for test automation (mandatory)

• Hands-on exposure to ML or GenAI systems testing

• Solid understanding of:

LLMs and prompt engineering

RAG architectures

AI regression and model drift conceptsPreferred Skills (Added Enterprise Storage Focus)

• Experience testing enterprise storage platforms (block, file, object storage)

• Familiarity with storage concepts such as:

IOPS, latency, throughput, queue depth

Replication, snapshots, backups, disaster recovery

High availability (HA), failover, and resiliency testing

• Experience testing storage management, observability, or infrastructure platforms

• Exposure to distributed systems and data consistency concepts

• Familiarity with on-prem, hybrid, or cloud storage environments

• Experience validating scale, endurance, and long-running workloads

• Experience with agentic AI frameworks (LangChain, LangGraph, etc.)

• Familiarity with vector databases, embeddings, and semantic search

• Exposure to MLOps / LLMOps practices

• Experience with BDD frameworks (Cucumber / Behave)

• Familiarity with performance testing tools (JMeter, Locust, k6)

• Experience testing enterprise-grade or regulated systemsWhat Sets This Role Apart

Bridge traditional automation excellence with AI-driven quality engineering

Apply AI testing rigor to enterprise storage and infrastructure platforms

Define automation standards for non-deterministic AI systems

Work on production-grade GenAI platforms, not prototypes

• Influence Responsible AI quality practices at scale

Preferred technical and professional experience

• Data Analytics Knowledge: Exposure to data analytics concepts and techniques, with ability to apply data-driven insights to inform testing strategies and optimize test automation. • Advanced Scripting Skills: Experience with advanced scripting languages, including the ability to develop and maintain complex scripts for test automation and data analysis. • Industry Leading Practices: Familiarity with industry-leading testing practices and methodologies, including experience with testing frameworks and tools.

IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.