Automation Specialist, AI Data Service Operations

Job Overview
Your core objective is to maximize the response speed and performance efficiency of AI Agent prompts. You will manage a dynamic, multi-project workflow queue and leverage advanced AI-assisted diagnostic tools to continuously iterate on complex prompt architectures. This role goes beyond simply “chatting” with AI. It focuses on engineering robust, scalable, and precise logical structures that consistently drive desired model behavior.

Key Responsibilities

• Prompt Engineering: Design and deploy advanced prompting strategies, including Chain-of-Thought (CoT), Meta-Prompting, and Least-to-Most Prompting, to precisely guide model behavior.
• AI Tool Integration: Maintain and leverage automated Root Cause Analysis (Auto-RCA) tools and automated Prompt Agents to quickly identify the causes of model failures and generate potential solutions within minutes.
• Logic Diagnosis: Conduct deep-dive analysis into model hallucinations and classification errors. You will identify whether failures are caused by tokenization issues, logical gaps, or variance driven by temperature settings.
• High-Frequency Experimentation: Conduct rapid A/B testing across prompt variants. You should be able to quickly adjust your approach based on performance data, decisively move away from overly complex or ineffective solutions, and prioritize strategies that are robust and efficient.
• Tool Maintenance: Provide feedback and prompt updates for internal Agent tools, such as Auto-RCA, to ensure the team’s diagnostic toolchain remains at the forefront of industry practices.

Minimum Qualifications

• Education: Bachelor’s degree or above. Candidates with backgrounds in STEM, Linguistics, Philosophy, or related disciplines are preferred.
• Relevant Experience: Work or internship experience related to Large Language Models (LLMs), such as Prompt Engineering, AI Data Operations, AI Product Operations, or related areas.
• Data Analysis & Logical Thinking: Strong analytical skills with the ability to identify patterns in data and clearly define problems. Demonstrates structured thinking, attention to detail, and the ability to follow standardized workflows.
• Operational Agility: Ability to quickly switch between different project contexts within a dynamic workflow queue while maintaining a high level of attention to logic and quality.

Preferred Qualifications

• Domain Experience: Experience in Trust & Safety, content moderation, content review, or classification and annotation.
• Technical Understanding: Familiarity with fundamental NLP and LLM concepts, such as Attention mechanisms, context windows, and tokenization, with a good understanding of model capabilities and limitations.
• Technical Skills: Basic proficiency in Python or SQL, with the ability to write simple scripts for prompt testing or supporting data analysis.