WIP - Use ArcArray to remove copy

[bharrisau]

This isn't that well thought through - but it worked in my testing. Still missing the .as_standard_layout call - I assume calling .to_shared on the CowArray resolves that, but I haven't checked.

let data1 = vec![0_u8; 16*1536*2048];
let mut inp1 = ArcArray::from_shape_vec((16, 1, 1536, 2048), data1).unwrap().into_dyn();
let start_ptr = inp1.as_mut_ptr();
c.bench_function(name, |b| {
    b.iter(|| {
        assert_eq!(start_ptr, inp1.as_mut_ptr());
        let input_tensor = Value::from_array(None, &mut inp1).unwrap();
        bind.bind_input("images", input_tensor).unwrap();
        let _ = bind.run().unwrap();
    })
});

[decahedron1]

Removing the lifetime here means SessionOutputs is not bound to the lifetime of the session, meaning it could be accessed after the session is dropped, causing a segfault. Maybe we could add something like session_bind: PhantomData<&'session ()> here?

[bharrisau]

If the Value all have an Arc<Session> then the session will be kept alive until all the Values are dropped?

[bharrisau]

Updated the WIP - still don't totally like the ergonomics. And like you said, the SessionOutputs lifetime issue exists. And it probably makes sense to have IOBinder track bind_output and return the RustOwned instead of a CppOwned.

I'll see if I can find some time to tackle them soon.

[bharrisau]

So, I'm fixing things as I'm going through. But it's getting to the point where I think it's better to just split this in to a larger refactoring where we split out the crates at least in to an ort_sys with all the FFI, memory management, unsafe code, etc. And a ort_array with the ndarray bindings. Then you can keep the main ort exporting these subcrates.

Lots of unnecessary allocations, memory leaks and incorrect FFI. It would be easier to maintain to shift that to a smaller crate that is easier to do properly.

e.g.

ort/src/io_binding.rs

Line 67 in ba9df19

let mut output_values_ptr: *mut *mut sys::OrtValue = vec![ptr::null_mut(); count].as_mut_ptr();

This allocates a vec and extracts the pointer. But GetBoundOutputValues overwrites that pointer anyway, then calling to_vec on the slice makes a copy of everything, but then the original FFI pointer is never freed. So 2 extra heap allocations and a memory leak.

Let me know your thoughts?