← Work
AI Content Systems

Collecting user feedback on AI-generated output in Zendesk

User feedback helps train LLMs, especially when it identifies outputs that are irrelevant, inaccurate, or unhelpful. But you don't want to constantly bother users by asking for feedback. Zendesk was implementing many new features powered by generative AI, and needed a framework and pattern for collecting user feedback appropriately, consistently, and usefully.

Collecting user feedback on AI-generated output in Zendesk

Problem

  • Product designers were reinventing the wheel whenever they needed to include a user feedback flow in a generative AI experience
  • The ML Science team was developing an internal taxonomy for categorizing AI-generated errors and needed a way to align it with user feedback collection

Opportunity

Create clear, simple guidelines to help product designers determine when and how to ask users for feedback on AI-generated output. Provide a basic flow pattern to use and adapt for various use cases.

Collaborators

  • Product designers working on AI features
  • ML Science product manager
  • UX researcher for AI features

Approach

  1. Audit. Audited existing user feedback flows in the product to understand what patterns already existed and where they fell short
  2. Competitive analysis. Performed a competitive analysis to identify emerging patterns and conventions for AI feedback collection across the industry
  3. ML Science alignment. Met with the ML Science team to understand their needs for feedback collection and how their taxonomy could map to the user-facing feedback flow
  4. UX Research collaboration. Collaborated with UX Research to gather insights on the best ways to collect user feedback without disrupting the experience
  5. Pattern adaptation. Adapted existing feedback flows into a bare-bones pattern that could serve as a starting point for any AI feature
  6. Designer feedback. Collected feedback from product designers on the proposed flow pattern
  7. Revision. Revised the flow pattern based on feedback
  8. Publication. Published the flow pattern along with guidance on collecting user feedback
AI feedback collection guide, part 1

The guide: when to collect feedback, how to choose the right collection method, and how to align with the ML Science error taxonomy.

AI feedback collection guide, part 2

Guidance on finding the right form for the feedback UI — covering thumbs, ratings, freeform text, and when each is appropriate.

AI feedback flow pattern

The flow pattern: a reusable starting point any product designer can adapt for their specific AI feature and use case.

Results

  • Reduced dependency on explicit feedback collection, encouraging designers to carefully consider when to request feedback without disrupting the user experience
  • Improved consistency in how feedback is requested across AI features in the product
  • Designers are able to work faster because they don't need to design feedback flows from scratch
← Work

Contact