Skip to content

Accessibility Test Guideline

Jump to bottom Edit New page
David H. edited this page Dec 29, 2024 · 38 revisions

If you love image and slide structure: A Get started in Google slide

Get started

For team member:

  1. Read the task list
  2. Find something that is not assigned to anyone
  3. Click on title to read the detailed instructions
  4. Set Assignees to yourself and change Status to Todo
  5. Change Status to In Progress and start working on it
  6. Finish or Pending

If you've finished the task, change the Status to Done

  • Share things you have done with us on Tuesday Hacknight

Or, if you've run into issues, change the Status to Pending

  • Raise the issue at the Tuesday meeting

For tester:

Pages requiring testing are:

The testing is completed in three phrases:

  1. Test using the automated devtool
  2. Test manually using screen reader and keyboard navigator and try to find the behavior that does not meet Web Content Accessibility Guidelines (WCAG) standard
  3. Browse and review the user experience

Phase 1 Test:

  1. Go to Google sheet
  2. Find a page that does not have a test report (In testing phases we also list pages that need to be tested in task list so people can assign themselves)
  3. Follow the tutorial to generate test results
  4. Follow the tutorial to add the test results into the Google sheet

Phase 2 Test:

  1. Go to Google sheet
  2. Find a page that does not have a test report (In testing phases we also list pages that need to be tested in task list so people can assign themselves)
  3. Follow the tutorial to manually verify some detailed tests that the automated test tool may not notice
  4. Follow the tutorial to add the test result into the Google sheet

Phase 3 Test:

  1. Visit Figjam for Whiteboarding
  2. Edit whiteboard for feedback in UX from ordinary users, and experts besides of the screenshots.
  3. Follow the guideline to check the web page behavior does not jam with perception
  4. Follow the tutorial to add the test result into the Google sheet

Phase 4:

  1. Go to Google sheet
  2. Follow tutorial to convert all issues recorded into GitHub issues on issue page so developers can follow up

Troubleshoot 🛠️

Mistake in this Guideline

If you can find the edit button on this page

  • Refine the guideline for performing Accessibility Testing

If you cannot edit the page

Run into trouble

  1. You are working on a task
  2. Things do not happen as expected
  3. You are confused, looking for help
  4. Raise the problem in Slack or create GitHub issue

Test with Automated Accessibility Tool

Axe DevTools and WAVE are two browser extension tools we use for free automated accessibility testing.

Axe DevTools

The Axe highlights issues (broken down by categories) that do not meet WCAG guidelines, and for each issue, has a link to a standalone article dedicated to the specific issue.

The detailed run sheet.

Installation

To install, visit the respective link for the browser of your choice:

To run a Test

To test the home page of pol.is for example

  1. Open browser and go to pol.is
  2. Open your browser's developer tools (Press ctrl + shift + i).
  3. Navigate to the axe DevTools tab from the more tools (>>) drop-down.
  4. Fill out the necessary details to get started.
  5. From the extension start screen, click Full Page Scan to run an automated scan of the entire page.
  6. 🕙 Wait

Results

The issues overview contains the total number of issues found along with a breakdown, which act as "filters" to view specific issue types or categories:

  • "Automatic Issues": Issues found during the automated scan
  • "Needs Review": Potential issues in which axe-core was not able to determine with absolute certainty what were true issues. This can often occur with color contrast calculations for text over background images or complex gradients.
  • "Best Practices": Issues that do not necessarily conform to WCAG success criterion but are industry accepted practices that improve the user experience

Issues are grouped by the rules that have been "violated". Each violated rule is expandable, and contains:

  • the rule description
  • a "Highlight" button (that toggles a highlight around the affected element)
  • a "more information" link (link to the dedicated help page for the given rule)
  • "Element location" (the generated CSS selector for the element with the accessibility issue)
  • Remediation guidance (information on how to fix the issue)

To report a problem

  1. Go to Access issue log Google spreadsheet
  2. See if the problem already exists
  3. In case a problem is not recorded, refer to the guide to add the issue to the spreadsheet
  4. 🎉 Thank you!

WAVE

The WAVE browser extension is a free accessibility tool by WebAIM. WAVE provides visual feedback about accessibility by injecting icons onto the page. WAVE facilitates human evaluation and educates about accessibility issues.

The detailed run sheet.

Installation

To install, visit the respective link for the browser of your choice:

To run a Test

To test the home page of pol.is for example

  1. Open browser and go to pol.is
  2. Click on the WAVE icon to the right of your browser address bar, or select "WAVE this page" from the context menu.
  3. 🕙 Wait

Results

WAVE evaluates a page by 6 categories:

  • errors (failures to meet WCAG that will impact certain users with disabilities)
  • contrast errors (text that does not meet WCAG contrast requirements)
  • alerts (potential accessibility issues that requires further human evaluation)
  • features (elements that improve accessibility when implemented correctly)
  • structural elements (identified elements on the page that give it its structure)
  • ARIA (use of ARIA on the page: ensure that labels are used correctly)

The Details tab gives a further breakdown of each category, allowing you to toggle the appearance of the icon types on the page. Each icon type has a reference link (i button) that provides useful context about the rule/guideline and recommended ways to comply. Greyed-out icons refer to elements that are in the HTML but are hidden by CSS.

To report a problem

  1. Go to Access issue log Google spreadsheet
  2. See if the problem already exists
  3. In case the problem is not recorded, refer to guide to add the issue to the spreadsheet
  4. 🎉 Thank you!

A Guide to add test results

Axe

  1. In the extension, click on the Highlight button, it will toggle a highlight around the affected element(s).
  2. Browse through the affected elements using the > icon button.
  3. Go to Google sheet
  4. Add the nature of the issue under the "Issue title" column.
  5. Under the "Component/Area" section, add the title of the affected area. This will depend on the nature of the scan you perform- Full page or Partial Scan.
  6. Use the remediation guidance to describe the issue details under the Comments column.
  7. Select from the drop-down the "source" of the test.
  8. Enter the name of the affected component under the "Affected element" column.
  9. Lastly, enter your name under the "Identified by" column.
  10. And wait to get credited 🎉

Guide to convert test results

  1. Go to Access issue log Google spreadsheet and download the document.
  2. Remove problems that are already listed in the GitHub issues
  3. Share the document with Gemini (Google's powerful AI) and ask it to write issue statement for each of the problem for our GitHub repository with a title and a Statement.
  4. Copy each of the issues
  5. 🎉 Thank you!
Example prompt for Gemini 
The attached file contains a list of accessibility problem we found in our test. Now we need to convert it into GitHub issues and report to developers. Your job is to format the issue statement according to following example.
---
**Description:**

The group and statement selection buttons on the participation page should have the `tablist` role to ensure keyboard navigability.  Currently, they are not properly structured for keyboard users.

**Steps to Reproduce:**

1. Navigate to the participation page (e.g. `https://pol.is/4cvkai2ctw`)
2. Attempt to navigate the button groups using only the keyboard.

**Expected Behavior:**

The button groups should be keyboard navigable.

**Actual Behavior:**

The button groups are not properly structured for keyboard navigation.

**Component/Area:** Participation page

**Source:** Other

**Affected Elements:** button groups

screen reader & keyboards

ℹ️ If you are using Mac, try your screen reader tutorial and then use a screen reader to access a website (e.g. Canadian Tire) to understand how the software is used. This will be helpful for you to identify accessibility issues using screen read and keyboard navigation tools.

Tools:

  • NVDA (recommended) - Free
  • JAWS (if available) - USD$85/home license
  • Browser Developer Tools (for inspecting elements and focus)
  • Automated Accessibility Testing Tools (Axe, WAVE, Lighthouse - Note limitations)

Testing Focus: Keyboard Navigation and Screen Reader Compatibility

The Test Cases.

1. Keyboard Navigation:

  • Tab Navigation:
    • Verify all interactive elements (buttons, links, form fields, etc.) are reachable using the Tab key. Follow visual sequence where possible. Note any discrepancies.
    • Confirm the presence of a clear focus indicator (WCAG 2.4.7) on each element as it receives focus. The indicator should have sufficient contrast (3:1 or higher). Note any missing or insufficient focus indicators.
    • Check for any elements skipped during tab navigation (e.g., <a> tags without href attributes).
    • Verify tabbing order aligns with logical reading order.
  • Spacebar and Arrow Keys: Test functionality of Spacebar and Arrow keys where appropriate (e.g., navigating within menus, selecting radio buttons/checkboxes).
  • Enter Key: Verify Enter key activates interactive elements (buttons, links, form submissions).

2. Screen Reader Compatibility (using NVDA primarily):

  • Headings:
    • Ensure all headings (<h1> to <h6>) are used semantically and reflect the page structure.
    • Verify screen reader announces headings correctly and in a logical order.
    • Check for any non-heading elements incorrectly identified as headings by the screen reader.
  • Links and Buttons:
    • Confirm screen reader announces links and buttons with descriptive text.
    • Verify <a> tags used for navigation have href attributes; use <button> elements for interactive actions within the page. Confirm consistent usage.
  • Form Fields:
    • Ensure all form fields have associated <label> elements for clear identification.
    • Verify screen reader announces labels and input field types correctly.
    • Test error handling with screen reader to ensure error messages are conveyed accessibly.
    • Confirm character limits on text inputs are announced to screen reader users. Test character limit warnings are accessible.
    • Placeholder text should not be used as a replacement for labels.
  • Images and Graphics:
    • Verify all informative images have appropriate alternative text (alt attribute) conveying their meaning. Decorative images should have empty alt attributes (alt="").
    • Test complex graphics (charts, diagrams) for screen reader accessibility (refer to previous meeting notes with Solomon).
  • Dynamic Content:
    • Ensure all changes to the page content (e.g., loading new information, form validation errors) are announced to screen reader users.
    • Verify screen reader announces relevant information when focus changes (e.g., input field descriptions).
  • Voice Recognition: Test page functionality using voice recognition software to ensure compatibility and identify potential breakpoints.

3. Visual Checks:

  • Color Contrast: Check for sufficient color contrast between text and background elements, especially for interactive elements and important information. Consider users with low vision.
  • Visual Cues: Ensure interactive elements have clear visual cues (e.g., hover effects, focus indicators) to indicate their interactivity.

4. General Usability:

  • Error Handling: Test error messages for clarity and accessibility.
  • Cognitive Load: Consider users who may be easily distracted. Ensure clear and concise language, consistent navigation, and helpful cues to prevent users from getting lost.

To report a problem

  1. Go to Access issue log Google spreadsheet
  2. See if the problem already exists
  3. In case the problem is not recorded, refer to guide to add the issue to the spreadsheet
  4. 🎉 Thank you!

UX Accessibility

ℹ️ All content should be easy to understand.

To enable testers of different level in participating the UX Accessibility evaluation, we uses a whiteboard to collect opinions from users and experts.

  1. A whiteboard containing screen shots of the pages of our latest polis version should be provided. Figjam for Whiteboarding
  2. Edit whiteboard for feedback in UX from ordinary users, and experts besides of the screenshots.
  3. Analysis and organize the notes in the whiteboard and build tickets/GitHub issues for development. Analysis will occur as a group and the notes will be synthesized.

🔑 If you are not familier with UX Accessibility evaluation, here are some key criteria

Resources

Introduction to Web Accessibility:

This section provides links to valuable resources for web accessibility:

Resources for how to... UX review:

Add a custom footer
Add a custom sidebar
Clone this wiki locally