Support for long integers (aka int64, s64, long)?

[blueberry]

Summary

Many LLM models take tensors of 64-bit integers at input/output (GPT2, etc). If one uses them from ONNX Runtime or CUDA, both natively support long integers. OneDNN doesn't, even in the most basic way of creating a memory descriptor, or data reordering.

Problem statement

The users of oneDNN (usually higher-level libraries) typically rely on DNNL as a base to build tensors on CPUs. Since OneDNN doesn't suport 64-bit integers, such implementations can't ingest data coming form other sources that work with them, such as cuDNN (supports long integers) or ONNX Runtime (also support long tensors).

Preferred solution

The solution doesn't even to cover most of oneDNN routines, since these would probably be complicated to implement. Here's what I think is not very complicated, and yet will increase the usefulness of oneDNN:

1. Support 64-bits integers as data types in dnnl_memory_desc
2. Support reorder primitive, and other simple-ish primitives, even with sub-optimal performance

Nobody is going to use 64-bit integers as their first choice, but sometimes there's data, and at leas enable oneDNN to recognize this data, and easily convert it into a more optimal data-type (int32, float, or whatever).

[vpirogov]

@blueberry, thank you for the suggestion. Couple questions:

• Could you please share a pointer to cuDNN documentation that covers int64/long data types support?
• Conversion from int64 to int32 or fp32 is fairly straightforward operation in C. What advantages you expect to get from using reorder primitive instead?

[blueberry]

1. There's no documentation because there's no current support. I'm suggesting barebones support.
2. The main benefit is consistent interface. Supose there's general code that works with a oneDNN tensor, and the code needs to support various data-types that depend on a parameter (say, to accept an onednn tensor and create a cudnn tensor of the same data-type and move the data). Such code would work well for almost all types, but then for int64, it would have to create a completely new branch with custom logic, because there can't be an onednnl tensor of long integers, sice I can't create a dnnl_memory_desc with long integer type.
   2a. For the reorder operation: yes, it is straightforward in any language, but then the client of the code has to write custom logic for that case, check what is the type, and call different operation for the reorder, possibly having to implement non-trivial logic if strides are not straightforward, instead of calling the reorder primitive in a general way, for all meaningful data types (64-bit integer is a staple in many languages, such as Java, C#, etc.).
3. Please note that onednnl is typically used by developers of higher level language bindings (I develop one in Clojure/Java). I can call onednn via automatically generated JNI bindings, so consistent API that works with all common types would be very useful (at least for a set of most basic operations is very useful)

[vpirogov]

Thanks for clarifications. Now I understand where you are coming from.

1. There's no documentation because there's no current support. I'm suggesting barebones support.

My question was about int64 support in cuDNN. Your initial report says mentions that "cuDNN (supports long integers)". Is this a typo?

[blueberry]

As a non-hypothetical use case, many LLM models use long integer for input-outputs. I guess that no-one would need a thing such as long integer convolution, but almost everyone would benefit from being able to treat that output as a onednn tensor, and reorder its data to diferent type/format, without having to write custom logic in C (or other language of choice).

[blueberry]

Yes, you're right about cuDNN. It doesn't support int64; I mixed that up with ONNX runtime tensors who do support int64 (unfortunately :)), where I used a combination of onnx runtime and vanilla cuda to support that.

But cuDNN not having int64 is not that big of a deal, because cuDNN is used as an execution provider there, but then the model output is usually accessible from main memory. There, being able to handle it consistently as a onednn tensor would be very useful.

As for reorder, it would still be useful too. One reason is consistency, another one is that often we need just a relevant subtensor of an output tensor. In such cases, reorder can be used to copy just a relevant part of the larger tensor for further processing, without writing custom logic for that (I already have general code for creating subtensor descriptors etc.).