This is a remote position. AI Quality Analyst Reporting to Manager, Quality Engineering & AI Validation, focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. Builds benchmark scenarios, defines scoring rubrics, evaluates business usefulness, and identifies failure patterns that conventional pass or fail software testing would not catch. Key Responsibilities AI Output Evaluation · Design and execute structured evaluations for AI-enabled features and workflows. · Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk. · Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations. Benchmark & Rubric Development · Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards. · Develop rubrics that allow quality to be measured consistently across releases and changes. Workflow & Model Change Validation · Compare performance across prompt versions, workflow revisions, tools, and models. · Support release decisions with evidence on quality regression or improvement. Business & Domain Partnership · Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts. · Help define human-review thresholds and escalation patterns for higher-risk use cases. Production Feedback · Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time. Requirements Required Qualifications · 4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles. · Strong writing, analysis, and pattern-recognition skills. · Experience evaluating outputs against nuanced criteria rather than only binary correctness. · Ability to work with structured rubrics, scenario libraries, and evidence-based reviews. · Comfort collaborating across Engineering and business teams. · Experience with finance, accounting, FP&A, transaction services, or business process design preferred. · Bachelor's degree preferred. You Are · Thoughtful, precise, and highly discerning. · Strong at spotting subtle output problems others miss. · Comfortable with ambiguity but disciplined in scoring and documentation. · Focused on trust, usefulness, and business reality. Benefits Salary plus performance-based bonus. Actual compensation packages are determined by evaluating a wide array of factors unique to each candidate, including but not limited to skill set, years and depth of experience, education, certifications, cost of labor, and internal equity.
...Position Title: Foreman - Renewable Energy Location: St. Michael, MN Salary Interval: Union Scale Pay_Range: N/A Application Instructions: All applicants must apply online. Please use the Apply Now button above to begin the application process. Follow...
...Job Description Job Description Experienced commercial electricians Helpers (no experience required)\nCompany Description We are a commercial electrical contractor that has been in business since 1981. We specialize in restaurants, gas stations, convenience...
...Pursuit Aerospace is seeking a Tool Setter in Stuart, FL to perform basic machining, set up tools for CNC/EDM, and learn new tool setter technology with RFID communications. You will load programs, adjust feeds and speeds, maintain tolerances, inspect parts, and contribute...
If you are seeking a new opportunity or would simply like to learn more about locum tenens, give Weatherby a call today for details. ~ Monday - Friday, 8 am - 5 pm with limited weekend call ~1 - 3 weeks of coverage per month ~ Clinic volume: 16 - 18 patients per day...
...Job Summary and Specifications As a Facility Support Janitor/Custodian , employee should reflect a general to advanced knowledge of Housekeeping and Janitorial services. Service will be performed in an office environment, common walk areas, lobbies, and restrooms. General...