API to specify max number of elements

[AdamWRichardson]

Thanks all for the great project, I can't wait until this is made stable!

To help with the portability I thought it would be useful to have some way of asking for a SIMD struct with the maximum number of elements supported by the target, such that you could write something like:

let a = f32::with_max_elements();

And that would give you Simd<f32, 16> for a target which supports 512 bits but Simd<f32, 4> on targets which only support 128 bits.

I thought this would be a fairly easy addition so I tried a couple of different implementations (including the one above) but each on so far has had problems. I'm very keen to contribute to the project and would happily write this up but wanted to check a couple of things:

• Is this something that would be useful and worth implementing?
• What do you think is the best way to go about this?

[calebzulawski]

I think something like this would be nice, but difficult to implement. The difficult part is accounting for detected target features. My target-features and multiversion crates support this, this may help: https://calebzulawski.github.io/rust-simd-book/4.2-native-vector-width.html

[programmerjake]

other things that make this harder is that on some architectures you can't get the maximum length at compile time, e.g. on RISC-V V, the maximum length is decided by the cpu manufacturer and may be all the way up to 16384 but can be much shorter. Some architectures you can't get a reasonable maximum length, e.g. SVP64 on PowerISA has a maximum length of 64, but you can only have 3 u64x64 vectors in registers, but you can instead easily have dozens of u64x4 vectors in registers which may even be faster.

[calebzulawski]

That's another good point--you could conceivably have two versions of such a function, one that's const and one that isn't. In target-features I simply return 128-bit vectors for the v extension, even though they can of course be much larger. If rust adds zvl*b target features, that would at least make it possible to statically detect a larger minimum.

[AdamWRichardson]

Thanks for the responses, I hadn't even considered non-mainstream architectures yet! Your target-features crate seems to do similar things to my plan which was just having a load of #[cfg(target-feature = "...")] within a function although the crate is a much nicer way to do this.

One potential annoyance one could have with the suggested_vector_width API is that a user could conceivably do something like:
const N: usize = selected_target!().suggested_vector_width::<f32>().unwrap_or(1);
let vector = Simd<f64, N>;


Which is why I thought an associated method on the Simd struct might have been nicer because it just has to care about one instance of T.

To implement this functionality within portable_simd it would be good to use the target-features and multiversion crates but is that allowed when thinking about up-streaming to stable?

[calebzulawski]

It would be difficult to make it an associated function because the return type needs to have the number of elements in it, and that number of elements is contextual, at least on the statically enabled features but also potentially from the #[target_feature] attributes. This is definitely something I'd like to be easier to do, but unfortunately it's a bit difficult in Rust today.

[kleinesfilmroellchen]

This seems to me like C++’s std::experimental::native_simd, which on various AVX extensions usually outperforms Rust in my testing (and is more succinct if I start using cfg(target_features) in Rust). The C++ API also only provides a static size here, as the Abi template parameter to simd<T, Abi> really only contains a size() constexpr method.

For dynamically-sized SIMD vectors, a lot more additional design is probably necessary. For example, you cannot use to/from_array methods on such vectors, as arrays are statically sized. Also, things like casting to differently-sized element types may pose problems for codegen, though I’m not familiar how the dynamically-sized vector types in LLVM work exactly. (Since they seem to exist at least for AArch64 SVE, we might as well make use of them :). As a first step, however, supporting statically-sized extensions through a native_simd-like API (I quite like the name here as well) would be an important improvement.

I consider this a crucial feature for portable_simd -- using a natively-sized vector is essential for performance.

[bjorn3]

So the size of std::experimental::native_simd changes depending on which target features are enabled for a function? That would make it impossible to pass it between two functions with differing target features. It would also mean that it can't implement Sized (as all Sized types have a fixed layout for a given compilation, independent of which function uses them), making usage a lot harder.

[kleinesfilmroellchen]

So the size of std::experimental::native_simd changes depending on which target features are enabled for a function?

Yes.

That would make it impossible to pass it between two functions with differing target features.

What do you mean with “differing target features” (across crates?)? If I understand it correctly, you’re correct and C++ should have the same problem. I do not consider this a problem at all, seeing how portable_simd does not have proper SIMD ABI support anyways at the moment. Such specialized SIMD vectors are really intended for one algorithm and its self-contained collection of functions (possibly all inlined).

It would also mean that it can't implement Sized (as all Sized types have a fixed layout for a given compilation, independent of which function uses them), making usage a lot harder.

I think this is more complicated than you portray it here, and it’s not the same as normal unsized types (while it does not implement Sized). I think you’re overly concerned with ABI boundaries and such, which are really not a big concern for SIMD in general. We can limit the use of such types across ABI boundaries if need be. For example, Linux on RISC-V mostly uses an ABI (LP64D) which cannot pass vector arguments in registers at all.

[calebzulawski]

That would make it impossible to pass it between two functions with differing target features.

What do you mean with “differing target features” (across crates?)? If I understand it correctly, you’re correct and C++ should have the same problem.

This is referring to the target_feature attribute, different functions in the same crate do not always have the same features. As far as I know, C++ doesn't worry about ABI and can have peculiar bugs and strange error messages.

It would also mean that it can't implement Sized (as all Sized types have a fixed layout for a given compilation, independent of which function uses them), making usage a lot harder.

I think this is more complicated than you portray it here, and it’s not the same as normal unsized types (while it does not implement Sized).

Not implementing Sized means not implementing Copy, which is already a huge limitation in my opinion.

[calebzulawski]

If these issues are predicated on performance vs the C++ implementation, I'd like to see a comparison so we can address codegen issues if they exist.

[kleinesfilmroellchen]

I think we’re getting a bit off-track here and trying to solve too many problems. Statically-sized native vector types are already an improvement in my testing, and if you want to I can provide some more detailed measurements based on my research when I find the time. What to do with these types for non-SIMD platforms or SVE/RISC-V V is an open question that shouldn’t derail these improvements.

Do note that GCC and LLVM are both not always very good at generating code from std::experimental::simd at present, even in comparison to Rust on platforms that Rust doesn’t properly support (e.g. PowerPC AltiVec, where I observed several 5x performance differences between almost identical Rust and C++ code).

[kleinesfilmroellchen]

From what I quickly measured, AVX512 on Cascade Lake is 30% faster with 512-bit vectors versus 256-bit vectors.

[kleinesfilmroellchen]

Excuse the posting of much code in this comment. I have two microbenchmarks, one with aligned accesses (rice-golomb), one with unaligned accesses (stereo). Recent nightly compilers. 16-bit numbers means 16 lanes on AVX2 and 32 lanes on AVX512.

pub fn decorrelate_stereo_simd_flexible<const N: usize>(
    left: &Samples,
    right: &Samples,
    mid: &mut Samples,
    side: &mut Samples,
) where
    LaneCount<N>: SupportedLaneCount,
{
    let (left_vecs, left_end) = left.as_chunks::<{ N }>();
    let (right_vecs, right_end) = right.as_chunks::<{ N }>();
    let (mid_vecs, mid_end) = mid.as_chunks_mut::<{ N }>();
    let (side_vecs, side_end) = side.as_chunks_mut::<{ N }>();

    for (((l, r), m), s) in left_vecs.iter().zip(right_vecs).zip(mid_vecs).zip(side_vecs)
    {
        let l = Simd::<i16, { N }>::from_array(*l);
        let r = Simd::<i16, { N }>::from_array(*r);
        *m = ((l.cast::<i32>() + r.cast::<i32>()) >> 1)
            .cast::<i16>()
            .to_array();
        *s = (l - r).to_array();
    }
    decorrelate_stereo_normal(left_end, right_end, mid_end, side_end);
}

#[inline(never)]
pub fn bits_required_for_rice_golomb_simd_flexible<const N: usize>(
    residuals: &Samples,
    order: u8,
) -> u64
where
    LaneCount<N>: SupportedLaneCount,
{
    let mut partition_bits = 4 + (1 + order as u64) * residuals.len() as u64;

    let (residuals_start, residuals_vec, residuals_end): (_, &[Simd<i16, { N }>], _) =
        residuals.as_simd();
    for residual in residuals_start.iter().chain(residuals_end) {
        partition_bits += (((residual << 1) ^ (residual >> 15)) as u32 >> order) as u64;
    }

    for residual_vec in residuals_vec {
        partition_bits += (((residual_vec << 1) ^ (residual_vec >> 15)).cast::<u32>()
            >> (order as u32))
            .reduce_sum() as u64;
    }

    return partition_bits;
}

Varying the lane count N, we see the following result for AVX2 (Alder Lake, pinned on P-core):

flexstereo 2            time:   [89.328 µs 90.473 µs 91.709 µs]
Found 17 outliers among 100 measurements (17.00%)
  4 (4.00%) high mild
  13 (13.00%) high severe

flexstereo 4            time:   [60.985 µs 61.103 µs 61.230 µs]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

flexstereo 8            time:   [50.219 µs 50.325 µs 50.442 µs]
Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) high mild
  9 (9.00%) high severe

flexstereo 16           time:   [44.345 µs 44.418 µs 44.496 µs]
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe

flexstereo 32           time:   [44.543 µs 44.601 µs 44.662 µs]
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  5 (5.00%) high severe

flexstereo 64           time:   [42.261 µs 42.329 µs 42.405 µs]
Found 11 outliers among 100 measurements (11.00%)
  9 (9.00%) high mild
  2 (2.00%) high severe

flex-rice-golomb 2      time:   [65.664 µs 65.777 µs 65.911 µs]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

flex-rice-golomb 4      time:   [39.530 µs 39.583 µs 39.640 µs]
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

flex-rice-golomb 8      time:   [26.267 µs 26.325 µs 26.408 µs]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe

flex-rice-golomb 16     time:   [18.927 µs 18.986 µs 19.066 µs]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

flex-rice-golomb 32     time:   [16.447 µs 16.480 µs 16.522 µs]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe

flex-rice-golomb 64     time:   [18.765 µs 18.787 µs 18.811 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

Same for AVX512 (Cascade Lake, pinned to non-hyperthreaded core):

flexstereo 2            time:   [80.846 _s 81.115 _s 81.491 _s]
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild                      
  4 (4.00%) high severe     
                                                                                            
flexstereo 4            time:   [66.286 _s 66.324 _s 66.365 _s]
Found 14 outliers among 100 measurements (14.00%)  
  7 (7.00%) low mild        
  4 (4.00%) high mild
  3 (3.00%) high severe                       
                                              
flexstereo 8            time:   [65.325 _s 65.342 _s 65.364 _s]                             
Found 10 outliers among 100 measurements (10.00%)                                           
  5 (5.00%) high mild                                                                       
  5 (5.00%) high severe 
                                              
flexstereo 16           time:   [66.854 _s 66.870 _s 66.889 _s]
Found 8 outliers among 100 measurements (8.00%)                                            
  4 (4.00%) high mild                                                                       
  4 (4.00%) high severe                       
                                                                                            
flexstereo 32           time:   [79.470 _s 79.528 _s 79.589 _s]
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low severe                        
  1 (1.00%) low mild                                                                        
  5 (5.00%) high mild                                                                       
  2 (2.00%) high severe                                                                     
                                                                                            
flexstereo 64           time:   [76.967 _s 77.025 _s 77.091 _s]
Found 5 outliers among 100 measurements (5.00%)                                             
  3 (3.00%) low mild 
  1 (1.00%) high mild                                                                       
  1 (1.00%) high severe 

flex-rice-golomb 2      time:   [58.688 _s 58.741 _s 58.802 _s]
Found 17 outliers among 100 measurements (17.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  5 (5.00%) high mild
  10 (10.00%) high severe

flex-rice-golomb 4      time:   [39.077 _s 39.126 _s 39.187 _s]
Found 16 outliers among 100 measurements (16.00%)
  7 (7.00%) high mild
  9 (9.00%) high severe

flex-rice-golomb 8      time:   [26.466 _s 26.486 _s 26.511 _s]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

flex-rice-golomb 16     time:   [24.627 _s 24.655 _s 24.690 _s]
Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) high mild
  5 (5.00%) high severe

flex-rice-golomb 32     time:   [18.454 _s 18.478 _s 18.508 _s]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

flex-rice-golomb 64     time:   [20.327 _s 20.351 _s 20.381 _s]
Found 15 outliers among 100 measurements (15.00%)
  8 (8.00%) high mild
  7 (7.00%) high severe

Of course this tells us that alignment is important, but if we account for it, AVX512 has some very significant benefits from its native vector size.