InvertedIndices don't work with CategoricalArrays

[greimel]

MWE

using CategoricalArrays, InvertedIndices
categorical(1:10)[Not(3)]

yields

ERROR: ArgumentError: unable to check bounds for indices of type InvertedIndex{Int64}
Stacktrace:
 [1] checkindex(::Type{Bool}, ::Base.OneTo{Int64}, ::InvertedIndex{Int64}) at ./abstractarray.jl:561
 [2] checkbounds at ./abstractarray.jl:491 [inlined]
 [3] checkbounds at ./abstractarray.jl:506 [inlined]
 [4] getindex(::CategoricalArray{Int64,1,UInt32,Int64,CategoricalValue{Int64,UInt32},Union{}}, ::InvertedIndex{Int64}) at /Users/fabiangreimel/.julia/dev/CategoricalArrays/src/array.jl:665
 [5] top-level scope at REPL[6]:1