[Vue] ais-index ignore/obey specific filters

[indykoning]

I'm using vue-instantsearch to display a list of products with their filters.
But i also want to show a list of categories matching the search query.
However due to the product filters once one is applied all categories disappear.
Is there a way to only listen to a specific filter for this categories index?

Here's an example of my situation

<ais-instant-search 
    ... 
    index-name="products"
>
  <div class="sidebar">
    <ais-search-box>
      <template v-slot="{ currentRefinement, refine }">
        <input
          type="search"
          v-bind:value="currentRefinement"
          v-on:input="refine($event.currentTarget.value)"
          placeholder="Search within the results"
        />
      </template>
    </ais-search-box>

    <ais-refinement-list operator="multiselect" attribute="color" limit="6">
      <template v-slot="{ items, refine }">
        <ul>
          <li v-for="item in items">
            <label>
              <input
                type="checkbox"
                v-bind:checked="item.isRefined"
                v-on:change="refine(item.value)"
              />
              @{{ item.label }} (@{{ item.count }})
            </label>
          </li>
        </ul>
      </template>
    </ais-refinement-list>
  </div>
  <main>
    <ais-index index-name="categories">
      <ais-hits v-slot="{ items }">
        <div v-if="items && items.length">
          Categories:
          <ul>
            <li v-for="(item, count) in items">
              <a v-bind:href="item.url">
                <ais-highlight
                  v-bind:hit="item"
                  attribute="name"
                  highlighted-tag-name="mark"
                ></ais-highlight>
              </a>
            </li>
          </ul>
        </div>
      </ais-hits>
    </ais-index>
    <ais-hits v-slot="{ items }">
      <div v-if="items && items.length">
        Products:
        <ul>
          <li v-for="(item, count) in items">
            <a v-bind:href="item.url">
              <ais-highlight
                v-bind:hit="item"
                attribute="name"
                highlighted-tag-name="mark"
              ></ais-highlight>
            </a>
          </li>
        </ul>
      </div>
    </ais-hits>
  </main>
</ais-instant-search>

When i use the search box, both show relevant results.
The moment i use the color filter, categories will never show results unless i deselect the color.