System Usability Scale (SUS) Survey - Q2-FY2024 - Research Execution

SaaS (run every quarter)

Recruitment SaaS

  1. Duplicate the SUS survey template in the shared UX research Qualtrics account
  2. Complete the Promotional Games workflow to generate the abbreviated rules for the drawing.
    1. Publish the survey and update the Abbreviated Rules link in Q12 with the newly generated rules.
  3. Find SaaS users from data warehouse
    1. Request access to Sisense as an editor to be able to query, if you don't already have access.
    2. Use these queries to generate a list of user IDs.
      • SUS - Paid Segment - All Users - With Email
      • SUS - Free Segment - All Users
    3. Import mailing list into Qualtrics
      • Refresh Sisense charts that contain Free Segment - All Users with emails and Paid Segment - All Users with emails.
      • Download data, save it to your local file system, and rename it with the appropriate wave number
      • In Qualtrics, create a new email list with the appropriate name (ex. Q2FY23 SUS Free Wave 1) and upload the correct list from your local machine.
    Old Instructions - [ ] Import mailing list into Qualtrics via Google Sheets following handbook guidance - [ ] Generate your list of GitLab.com user IDs and put them into the first column of a Google Sheet. The first entry in the first column (A1) should be id, as this confirms that this is a list of ids that you want to use. - [ ] The name of the specific sheet (or tab) should be what you want to name the Qualtrics mailing list, for example, `05/20 - CI Pipeline Prototype Testing`. **This name cannot contain the following characters: `. < >`. Doing so will case an error in the transfer to Qualtrics.** - [ ] The filename of the entire spreadsheet should be `qualtrics_mailing_list`. followed by the name of the worksheet/mailing list that you used in the step above. So keeping with our previous example, `qualtrics_mailing_list.05/20 - CI Pipeline Prototype Testing` - [ ] Once everything is ready, move the file into the `QualtricsRequest` Google Drive folder. - If you do not see the `QualtricsRequest` Google Drive folder, ask another researcher to share access. - [ ] If everything was done correctly, within about 15 minutes, the `id` entry in A1 should turn into `processing`. That means the data pipeline is working on your request. - This can take up to 1 hour for lists of 6000. - [ ] A1 will show `processed`. Your mailing list should now be in Qualtrics at this point. - [ ] Add `processed_` to the filename in order to make it clear which files have been completed, as well as to keep the process executing as quickly as possible in the future. - [ ] The list will show up as a shared list for the _UX Research & Product_ directory. You can access the list itself from the Contacts section of Qualtrics, or you can select the list as part of the process of sending an email distribution. - The GitLab.com user ID you specified in your Google Sheet will be included in the Qualtrics mailing list as embedded data. You can use this to associate any responses you get with your original user list. - https://about.gitlab.com/handbook/engineering/ux/ux-research-training/finding-saas-users/ - https://about.gitlab.com/handbook/business-technology/data-team/#-data-warehouse
  4. Distribute survey to a test sample of users.
    1. Within the Distributions tab in Qualtrics select Email and then click "Compose Email"
    2. In the "To" dropdown, hover over Organization Library: UX Research & Product and search for the name of the list you generated in Sisense.
    3. In the "Message dropdown, hover over Organization Library: UX Research & Product and search and select SUS Survey Outreach Template. Make sure the following information within the email is updated:
    • Review that from address is listed as research@gitlab.com
    • Review that reply-to email is listed as research@gitlab.com
    • Review that subject line states GitLab Survey - We'd Love Your Feedback! (Chance to Win 1 of 3 $200 Gift Cards)
    • Update rules link within email to include the latest prize draw rules
    • Note: Filling the cohorts could take several weeks so it’s good to start recruiting at the beginning of the last month in the quarter.
    • If the target N = 800 (200 Paid/Mature, 200 Paid/New, 200 Free/Mature, 200 Free/New) and response rate is 2-3%, you will need to sample between 26,666 and 40,000. Start with waves of 6000 every couple days from Free and Paid users and adjust subsequent waves according to response rate.
  5. Review answers and make any necessary amendments to the survey. Distribute survey to remaining users.
  6. Close survey and cleanse data
  7. Conduct prize drawing and pay participants (or request Research Coordinator’s assistance)

Analysis

  1. Download R and R Studio
  2. Download the SUS Scripts Repo from GitLab
    • You can do this by clicking the Download icon to the left of the Clone CTA in the repo and select zip. It will download the files in the repo to the Downloads folder of your computer. You can move this folder within your local file system to wherever makes sense to you.
    • This is a copy specific to you and has no connection to the repo any longer. You cannot impact the repo, so do not worry. You can always redownload the repo if you have made changes.
  3. Make sure the Wave CSVs you generated in step 3.3 in the Recruitment SaaS section are in a folder called Distribution Output within the Repo folder you just downloaded.
  4. Update the Qualtrics Survey ID value on line 61 of 1-Read and Transform Qualtrics data.R
  5. Run 1-Read and Transform Qualtrics data.R to ensure that you have met the sampling thresholds. You can use this script to monitor data collection.
  6. Run RUN THIS FIRST.R to begin the transfer process to Google Slides
Old Instructions> 1. [ ] Export raw numeric data from Qualtrics 1. [ ] Transfer raw data into the corresponding sheets in the ‘Results scoring template’ sheet. 1. [ ] Move previous quarter’s data into the sheets for previous quarters. - **Note: This sheet will be shared with the entire company so contact information should be removed.**
  1. Analyze open-text responses to the following question: “Is there anything else you’d like to share with us about GitLab’s usability?”
    • Data validation can be updated to cover new themes.
  2. Create an issue for categorization by stage for the PMs and PDMs, use this issue as a template.
  3. Update the ‘SaaS SUS Survey Scores’ dashboard in Sisense (Wait until release plan is determined)
    1. Export survey data from Qualtrics (text and numeric)
    2. Import CSVs into Sisense following the guidance in gitlab-com/www-gitlab-com#10068 (comment 517593830)
  4. Download Verbatim data (after Stage categorization) and intermediate_current as .csv's. Add those .csv's to the SUS Sisense Upload folder of the System Usability Scale folder in the UX Research google drive.

Reporting & Sharing

  1. Create a slide deck to share key findings (Sample quarterly slide deck)
  2. Document survey results as an epic and add the epic to the following parent epic: &1455
  3. Create issue to accurately categorize SUS verbatim by stages (Sample Issue)
  4. Create a separate sheet that contains participants who are interested in a follow-up conversation (sample follow-up sheet; handbook overview of process)
  5. Add verbatims to the SUS Themes by Usability Issues sheet

Associated Resources

Prior Survey Data

Edited by Nicholas Hertz