Move usm_ndarray into dpctl_ext.tensor

[vlad-perevezentsev]

This PR proposes to migrate the tensor interface (usm_ndarray, dlpack, flags) into dpctl_ext/tensor making dpnp independent of dpctl's tensor module.

Updates:

• Introduce dpctl_ext_capi.h
• Implement a clean CMake interface library DpctlExtCAPI to properly propagate generated headers to consumers
• Update remaining imports from dpctl.tensor to dpctl_ext.tensor
• Link all backend extensions against DpctlExtCAPI to ensure consistent access to the C-API

• Have you provided a meaningful PR description?
• Have you added a test, reproducer or referred to an issue with a reproducer?
• Have you tested your changes locally for CPU and GPU devices?
• Have you made sure that new changes do not introduce compiler warnings?
• Have you checked performance impact of proposed changes?
• Have you added documentation for your changes, if necessary?
• Have you added your changes to the changelog?

[antonwolfy]

I guess we have to remove the warning suppress, since it's the last PR removing migration of the source code

[ndgrigorian]

@antonwolfy @vlad-perevezentsev
A thought but should we consider flattening the directory structure? Since currently it's
dpnp -> dpctl_ext -> tensor (actual files...)

Maybe could we flatten to something like dpnp -> tensor (or dpctl_ext)

[antonwolfy]

@antonwolfy @vlad-perevezentsev A thought but should we consider flattening the directory structure? Since currently it's dpnp -> dpctl_ext -> tensor (actual files...)

Maybe could we flatten to something like dpnp -> tensor (or dpctl_ext)

dpnp.tensor sounds reasonable as for me.

[antonwolfy]

Based on warning report seems we still using import dpnp.tensor somewhere in code or tests.
Is it intentional?

[antonwolfy]

Why do we need that and how it worked previously without linking with dpctl lib?

[vlad-perevezentsev]

Previously dpnp4pybind11.hpp included dpctl_capi.h from the external dpctl package which was found via ${Dpctl_INCLUDE_DIRS} set by find_package(Dpctl).

Now dpnp4pybind11.hpp includes dpctl_ext_capi.h which is an internal header,and without linking to DpctlExtCAPI I got an error

dpnp4pybind11.hpp:41:10: fatal error: 'dpctl_ext_capi.h' file not found
   41 | #include "dpctl_ext_capi.h"

[antonwolfy]

But the error is about include, not about linkage.

[antonwolfy]

Is it a temporary approach, or do we need to define interface lib even when it will be exposed by dpctl when the migration is completed?

[ndgrigorian]

It's a design decision either way: dpctl exposed PyUSMArrayObject for numba-dpex's sake. So this way, usm_ndarray would be exposed to C-API and could be interacted with from C code conveniently.

It could be removed, in which case, the api methods like i.e., cdef api char* UsmNDArray_GetData could also be removed (barring causing any problems in the dpctl_ext).

[antonwolfy]

I don't think we need to migrate the code, which will not be used by someone, I'd propose to drop that.
In any case we will able to port that in the future if someone will be interesting in.