User Interview Protocol and Insights

Rating System

Purpose and Goal

1. to find out how people use the rating system, how they perceive it and what they expect to receive from it

2. the key theme of the interview was to find the factors that clients look into when hiring a freelancer, how do they evaluate the freelancer, how do those ratings and feedback affect the freelancer, and learn about the experience of freelancing (from both client and freelancer perspective), both on the platform and otherwise. 

Protocol for Clients

Starting the interview

• We’ll introduce ourselves

  • Names

  • We are a group of  graduate students in a UX course at Purdue University.

  • Expectation:

    • This will be a 30-45 minute interview

    • It will be semi-structured interview so we have general topic and questions we want to cover, but others will come up based on the conversation.

• Brief about the project: 

  • Wish to learn how freelancers are evaluated 

  • In particular, we know you have hired freelancers in the past and are curious about your perspective as a client.

• Ask for permission to record (and then get a recording of them saying they are ok with it being recorded)

Demographics

Tell us about your occupation and professional experience, in particular how it relates to working with freelancers.

Topic Domain: Hiring freelancers

Lead-off Question: When’s the last time you hired a freelancer?

Follow-up Questions:

• How do you find freelancers for the job ? Have you used any of the freelancing platforms like Fiverr, Upwork or 99Designs ?

• How do you interpret the current rating system? When hiring a freelancer, what does it mean to you to for a freelancer to have a particular rating and what is the average rating of freelancers you normally hire?

• What information do you want to know before hiring a freelancer?

• What factors influences you when you make hiring decisions? 

• Which one is more important to you?

• How do you figure out if a freelancer is a best match?(portfolio, rating, comments)

Topic Domain: Evaluating Freelancers

Lead-off Question: What goes into your thought process when evaluating freelancers?

Follow-up Questions:

• How do you interpret the current evaluation system? When evaluating a freelancer, what does it mean to you to give someone a particular rating?e.g.  when would you give someone a 4-star vs a 5-star rating? What is the average rating you have given freelancers?

• What qualities/metrics are important to you when choosing a freelancer?

• How much time do you prefer spending when evaluating the freelancers work and providing your feedback?

• Has the freelancers rating been reflective of your experience of working with them?

• Do you like the current feedback and rating mechanism you can leave for the freelancer? Are there other measures you would like to rate them on or be able to view ratings for when hiring a freelancer?

Topic Domain: Evaluation behaviors

Lead-off Question: How would you evaluate a freelancer?

Follow-up Questions:

• How do previous evaluation of a freelancer affect your hiring process?

• What are the top things you would like to see freelancers improve upon?

• How does the pricing affect your overall experience?

Topic Domain: Evaluation Timeline

• Is there any difference when working on a shorter project vs a longer one in terms of interaction with a freelancer? How so?

• When working on a long term project, do you give feedback at different intervals or only upon completion of the contract?

Closing the interview

• Anything else which you want to add?

• Thanking them