Skip to content

HPX Google Season of Docs 2021 case study

Jump to bottom Edit New page
Mikael Simberg edited this page Nov 30, 2021 · 1 revision

Organization or Project: HPX (STE||AR Group)

Organization Description: The goal of the STE||AR Group is to promote the development of scalable parallel applications by providing a community for ideas, a framework for collaboration, and a platform for communicating these concepts to the broader community.

Problem Statement

What problem were you trying to solve with new or improved documentation?

The HPX documentation (https://hpx-docs.stellar-group.org/latest/html/index.html) has evolved over the life of the project to be extensive, but hard to navigate. It contains information about many parts of the project, but users regularly struggle to find relevant information, as evidenced by the questions and requests we see on our IRC channel. In addition to content being hard to find, some information, especially build instructions, is also outdated. The central idea behind this GSoD run was to rearrange and reformat the documentation pages on the whole in order to provide a more flexible and cohesive interface to the end user.

Proposal Abstract

A brief summary of your original organization proposal. Link to the proposal page on your project site, if possible.

The aim of the project was to collect feedback from existing developers and users about the current documentation, and apply that information to restructure and better present the current documentation. The aim of the project was explicitly not to create new content, but to take what already exists and present it better.

The full proposal is available here: https://github.com/STEllAR-GROUP/hpx/wiki/GSoD-2021-Project-Proposal.

Project Description

Creating the proposal

How did you come up with your Season of Docs proposal? What process did your organization use to decide on an idea? How did you solicit and incorporate feedback?

Having participated in Season of Docs in 2019 we already had a preliminary list of potential topics for this year's Season of Docs. The basic idea was to make our documentation more accessible to the users. Google Summer of Code students were the best source of feedback for that cause. Based on feedback we hear frequently from them we decided to focus on the content that is already available in the documentation. We combined the most critical parts from a few project ideas from 2019 into one project. For example, a commonly recurring difficulty for new Summer of Code students is how to build HPX.

We had not yet found a technical writer during the application period. This led us to leave the project proposal relatively open to fit various experience levels, so that we could adapt the topic to the hired writer.

Budget

Include a short section on your budget. How did you estimate the work? Were there any unexpected expenses? Did you end up spending less than the grant award? Did you have other funds outside of Season of Docs that you were able to use?

The budget, like the project scope, was left relatively open to be able to adapt to the experience level of the technical writer that we hired. We estimated a rough budget based on average technical writer salaries and a 3-5 month project of part-time work. We knew ahead of time that there would likely be enough work in our documentation for far longer than a 5-month period and thus the project would have to focus on only a subset of the documentation. Likewise, if the writer would make faster than expected progress we knew there would be enough work for them towards the end of the project. Based on this we estimated a budget of USD 5000.

At the time of writing the case study we have paid out the full 40% (USD 2000) given at the beginning of the project to our second technical writer. Another 20% of the budget is earmarked for our first technical writer. The remaining 40% of the budget is left for ongoing and potential future work by the current technical writer. All in all, we spent (or rather allocated) the budget slower than expected due to the project making slower than expected progress. However, we intend to pay out the remaining grant soon after the official end of the project. We are also planning to transfer some additional external funds to our Open Collective account that we intend to use for similar projects in the future.

Participants

Who worked on this project (use usernames if requested by participants)? How did you find and hire your technical writer? How did you find other volunteers or paid participants? What roles did they have? Did anyone drop out? What did you learn about recruiting, communication, and project management?

The core Season of Docs team consisted of:

  • Technical writer: Dimitra Karatza

  • Support: Ioannis Gonidelis, Nikunj Gupta, Hartmut Kaiser, Auriane Reverdell, Mikael Simberg (org admin)

We had two technical writers during the course of the project. Dimitra Karatza was the second one, while the first one will remain anonymous.

We advertised the acceptance of HPX to Season of Docs on our blog (https://stellar-group.org/2021/04/hpx-accepted-for-google-season-of-docs-2021/). We asked writers to fill in a simple application form. Although the official project proposal was already written by us, we asked writers to include which parts of the proposal they would like to focus on, both in terms of interest and previous experience. Additionally, we asked writers for a brief overview of their background, and motivation to work on HPX in general and specifically as a writer for our documentation.

We hired our first technical writer at the beginning of the project. Due to mismatched expectations on the time commitment and experience we unfortunately decided to end the official relationship with our first technical writer after three months. Despite this decision, we very much appreciate our first writer's motivation and contribution to the project.

We found a new technical writer through one of our previous Summer of Code students, and we proceeded to continue with the project.

With both technical writers we set up weekly calls, as we've done previously with Season of Docs and Summer of Code. This has been essential in keeping track of progress and making sure things are going in the right direction. Additionally, quick questions were typically handled on IRC, and more in-depth questions were handled either in the weekly calls or on GitHub issues. With the core team already spread out across the US and Europe, we are already used to the mostly asynchronous forms of communication. We found that the technical writers adapted quickly to this setting.

Having participated in Summer of Code previous years, as well as this year, we already had volunteers in the form of Summer of Code students. They helped both by supporting the technical writers editing documentation and building HPX and by providing feedback on which parts of the documentation had been particularly difficult or lacking for them. Previous students included Ioannis Gonidelis and Nikunj Gupta. New Summer of Code students were Akhil Nair and Srinivas Yadav. In addition to the support from the students, the writers were supported by three core HPX developers: Hartmut Kaiser, Auriane Reverdell, and Mikael Simberg.

Timeline

Give a short overview of the timeline of your project (indicate estimated end date or intermediate milestones if project is ongoing).

The project had the following timeline:

  • June: Familiarization with documentation. Gathering inspiration and materials. Identifying concrete pain points. Created and submitted a survey about the current state of the documentation. Focused on one of the most commonly mentioned pain points: API documentation. Decided to first try to address this using different themes. Research on Sphinx themes.

  • July: Research on and modifying themes continued.

  • Beginning of August: Research on doxygen.

  • End of August: After a couple of weeks of a break, work continued with a new technical writer. Switched focus to content editing. Focused on making the quickstart as "quick" as possible.

  • September: Continued with content editing. Focused on build instructions. Removed outdated information. Updated and rearranged content to have the most important information up-front.

  • October: Continued cleaning up the build instructions. Started work on an additional HPX matrix multiplication example section in the documentation.

  • November: Break due to exams. Work on finalizing the new example.

We intend to finalize currently ongoing topics (the new example listed in the following section, as well as some minor changes) in December. However, there is interest from both the HPX developers and the current technical writer to continue work on improving the documentation even after Season of Docs.

Note that our initial project plan explicitly stated that no new content should be created, and yet we decided to add a new example. We made this decision based on the following circumstances:

  • We have been asked several times to provide more examples.

  • The technical writer was at the moment working on something where HPX would be useful.

  • Writing the example (mostly) on their own was a good test case to find holes in our documentation.

Results

What was created, updated, or otherwise changed? Include links to published documentation if available. Were there any deliverables in the proposal that did not get created? List those as well.

The changes made during the project are listed below with links to the corresponding pull requests. The changes are currently visible in the documentation for our master branch (the development branch) and will be part of the documentation for the 1.8.0 release of HPX.

Metrics

What metrics did you choose to measure the success of the project? Were you able to collect those metrics? Did the metrics correlate well or poorly with the outcomes you wanted for the project? Did your metrics change since your proposal?

We had previously made a survey in general about the usage of HPX and received a fair number of responses. Thus we decided to do a survey at the beginning of the documentation project to collect feedback on what parts of the documentation needed the most attention. We posted the survey on our mailing list and IRC channels. We received zero responses. We had planned to use the survey to make focused changes, and then do a follow-up survey to ask users if they felt that the changes had improved the documentation. This would have been our metric. However, given that we had no "official" responses we instead turned to talking directly to the users that we knew. We received concrete feedback from Summer of Code students, who commented that getting started with HPX can be difficult and that the API documentation is difficult to navigate. Hence we decided to focus on these things during the project. We also asked for feedback directly from an external user who commented that more examples are always helpful.

To follow up on these changes, we will see how many build system-related issues we receive on GitHub in the next 6-12 months.

Analysis

What went well? What was unexpected? What hurdles or setbacks did you face? Do you consider your project successful? Why or why not? (If it's too early to tell, explain when you expect to be able to judge the success of your project.)

We received several significant changes to our documentation aimed at newcomers to HPX: the quickstart guide, examples, and the build instructions. These in themselves already make this year's Season of Docs a success from our perspective.

Both our technical writers had only some previous technical writing experience prior to the project but they still showed great enthusiasm for the project which helped us get to work quickly. Our second technical writer has even shown interest in continuing to work with us beyond Season of Docs. Given the time she has invested into learning about our documentation, we believe this will be very valuable as she'll be able to continue focusing on improving the content instead of dealing with the inevitable technical difficulties that come up at the beginning of a project.

We decided to stop working with our first technical writer about halfway through the project. This was a difficult decision. However, there was clearly a mismatch in expectations, both in terms of experience and time commitment that eventually made it the correct decision to stop. Despite the decision, we think both parties learned a great deal about both expectation management and documentation tools. We are grateful for the work our first technical writer did while with us.

Summary

In 2-4 paragraphs, summarize your project experience. Highlight what you learned, and what you would choose to do differently in the future. What advice would you give to other projects trying to solve a similar problem with documentation?

In terms of plain project management we think our prior experience with Summer of Code and Season of Docs has been valuable. The lessons we have learned about setting short term goals, requiring weekly meetings, and encouraging students to interact with others in the community on IRC has paid off previously and did so this time as well.

What we did not do well was the selection of our first technical writer. First, finding experienced technical writers has been difficult and if we decided to participate in the future we would definitely have to consider searching and advertising far outside the regular HPX communication channels. Second, we took a chance on a writer that we believed may work well for us but did not work out as well as we thought it would. While the result could have been the opposite and we would have been happy, we should in the future have a stricter application process with in-depth interviews.

Finally, we think the documentation changes themselves have been a success. A lot of work remains to be done given the size of the HPX project itself and the amount of existing documentation we have. However, we hope that by focusing on the beginners in this project we'll be able to motivate more people to try HPX out and hopefully expand both the user and developer community of HPX. We highly recommend projects just starting out with their documentation to invest time in thinking about their documentation from the perspective of newcomers to the project. Best is to talk to people who have just gone through the process of using your project. We've found that people tend to be quite open to talking about these things if you ask them.

Add a custom footer

Content

Clone this wiki locally