User Feedback

The mockups created to illustrate my assumptions were tested with users. Although my initial goal entailed testing the Contributor page, early research indicated that various aspects throughout the website could better support newcomers. For this reason, solely soliciting feedback about the Contributor page during testing would’ve felt like a missed opportunity. For example, if a newcomer lands on the website they may overlook the Contribute link in the footer. Therefore, I wanted to see the path users took to navigate to the Contribute page when links to it were included in the top navigation and the body of the homepage.


Four representative users similar to the personas were recruited to review the mockups. A script outlining the process and scenario was read to each participant. They were then observed while completing 6 tasks and answering follow-up questions related to each activity.

Activities and questions were structured around important goals for newcomers, which include:

  • Learn about contribution opportunities
  • Contact the project to ask a question
  • Identify members behind the project
  • Locate documentation
  • Join the mailing list

Initial Impressions

Overall, first impressions were favorable. Several participants commented about the simplicity and clarity of the website.

When asked if they thought their questions would receive a quick reply, a participant commented about the conciseness of the website.

“I think so. The site looks put together, looks clean and succinct.”

Another liked the warmth of the site’s colors as well as the overall simplicity.

“It’s very clean and to the point—how it works, 1,2, and 3—there’s not a whole lot of verbiage, so it gives me exactly what it is that I need. The objective is laid out in a succinct way.”

When first looking through the site and asked to give their overall impression, one participant commented on a revised heading.

“So you can help them improve the web…so pretty much anyone can do this. Pretty simple.”


Career Advancement

As mentioned in a previous post, many newcomers join an open source community to advance their career. When asked if they felt joining this community would help their career, several participants answered that it would, if they were a software tester.

“My career? Maybe not. But if people wanted to start a career as software testers, I’m sure it would.”

On one hand, this could mean further clarification is needed in explaining that contributions to develop and tools are welcome. However, it’s also an indication that those currently working in QA or perhaps those who aspire to be SDETs would be good people to recruit. Perhaps guest blogging on or outreaching to testing blogs would be a good way to spread word about the community.

Social Media Icons

When asked if they thought their questions would be answered quickly, another participant decided to explore the site’s Twitter profile. Although the Twitter link in the footer was quickly located, the link directed to a profile landing page instead of the full Twitter profile. I hadn’t initially thought much of it, but the user assumed the account was less active than it actually is, as only a few older tweets were displayed. This was a quick fix, that’s already been made.

Screen Shot 2017-03-06 at 8.45.06 AM.png

“Last post was 6:35 am Jan 30—so no, I would say they would not respond quickly. And then I’d have to take my chances with email—I would probably not even try to ask them on Twitter.”

Contribution Page

Opinions about the newly drafted content for the contributor page were mixed. Two participants liked the clearly defined roles.

“What I do like is that it clearly states what it is—bug reporter, issue triager…and in it, which I really love is that there are links which take me directly to it, it doesn’t just explain it.”

However, one user thought the page was too text heavy–a concern that’s come up before.

Screen Shot 2017-03-06 at 8.47.41 AM.png

“I think it’s so much text—like there’s so much you have to read.”

Also, it may still be unclear that the site accepts contributions outside of the bug reporting process–particularly when quickly scanning the page. Two users commented that there was some initial confusion around how to contribute.

“It’s not straightforward on what various opportunities I could be engaged in, other than finding bugs.”

“Contribute [in the nav] is a bit misleading—it seems like there’s an opportunity to contribute to this particular website. Like to do some coding, or to help with this particular website. But I guess this just means like reporting bugs. I’m not sure it should be telling you about bug reporter. Like if it’s the part of contributing. It should tell you only about contributing—I’m not sure.”

With this in mind, an alternative contribution page was mocked-up, with the goal of visually separating the types of contributor roles and breaking up the text.

Screen Shot 2017-03-06 at 8.49.34 AM.png

Two participants expected a concrete way to sign-up for each role. While looking under the Site Contributor role, a participant selected one of the links and expected to find more information about the role.

“At first it seems like a way to contribute to this website–but then again there’s a link to filing an issue.”

Directly linking to GitHub on contributor pages is common in open source communities. However, it might be worth giving newcomers additional support. This could entail detailed pages for each role, or call to action buttons that allow interested users to “sign-up” with their email. Users could then be sent a welcome email with more information and be directly connected with a mentor.

Success Story

When asked if they felt they’d be appreciated if they were to join the community, one participant suggested highlighting community members on the homepage.

“Maybe [add to] the front page—the best contributor of the month. Maybe some incentive to be the best contributor of the month—even if it’s just 10 dollars.”

Most of the participants seemed to skip over or simply not notice the success story on the contributor page. Changes to the mockup were made with the idea that it would better stand-out if it took up more space.

Screen Shot 2017-03-06 at 8.52.04 AM.png

Get Started Guide

We questioned whether or not it would be ideal to add the how-to guides to separate pages or keep them in tabs underneath the contributor roles. Two of the participants interacted with mockups that kept the guides in tabs and the other two interacted with guides that were placed on separate pages.

When asked to locate documentation or instructions, both groups easily found the guides. Therefore, I don’t believe adding the individual pages will frustrate the users or cause them to drop off.

Screen Shot 2017-03-06 at 8.55.53 AM.png

Overall, the layout of the how-to guides was well received.

“I like the graphics and the bullet points that walk you through the process. It’s very visually appealing. I’m not overwhelmed. Easy steps laid out.”

One user suggested videos before being asked if they’d like them. And the other three agreed that a video would be helpful.

“It’d be nice to have a video on exactly what to do—cause I’m more of a visual learner—visual, hands on.”

I think the feedback gathered from users will prove helpful in deciding what content to include in the video.

Contact Page

“Seems like there’s not a real way to contact people.”

“I’m not sure if it’s ways of contact, it seems like ways of getting information or news updates.”

Screen Shot 2017-03-06 at 8.58.56 AM.png

A contact page was mocked-up with several options for reaching out to the community. The overall impression from users was that it wasn’t really a contact page. When tasked with finding a way to ask a question, two participants were unsure about the use of IRC.

“Kind of have no idea what an IRC is—what I would like though is a live chat, even though it might not be 24/7, but a certain time of the day you can talk to someone.”

“Most people I would suspect don’t use IRC, so I don’t see a way to ask a question here.”

One user felt that the given options of contacting the community lacked privacy.

“Twitter is quite public.”

When asked their preferred method of communication, two users suggested email. A contact form or the email of a single point of contact would likely meet users’ needs and expectations of the page.

Contribute Links

Two additional links to the contribute page were added to the mockups, as it was assumed the one link in the footer wasn’t enough. Participants most often selected the Contribute link in the top navigation, and one selected the link in the ‘Help Us Improve the Web’ copy.

Report a Bug and Issues Links

On the homepage, I added more context to the Report Bugs, Diagnose Bugs, and Reach Out to Sites links. My initial assumption was that the current links, without context, may confuse newcomers. Three users read the text and one user expected more information after clicking Diagnose Bugs.  One user didn’t read the added text and had no expectations for where the links would take them.

“I’m the kind of user who doesn’t like to read the text—I just like to click and see what’s in there.”

About or Team Page

Three of the four users were able to quickly locate members of the community by navigating to the about page. Initially, I was unsure that this was where team photos belonged. One user, with a similar thought, had a difficult time locating the members behind the project.

Screen Shot 2017-03-06 at 10.48.52 AM.png

“I would’ve gone to ‘About’ to find out more about the community itself—not so much the members of the team.”

Clarifying this could be as simple as changing the page’s name from “About” to “About Us” or presenting the community as a completely separate team page.