Does cutlass_profiler support f8 gemm?

[liligwu]

As 3.2 supports f8 gemm, does the profiler cutlass_profiler support it as well? Thanks.
I tested like
./cutlass_profiler --operation=Gemm --accum=f32 --m=16 --n=16 --k=1024 --A=f16:* --B=f8:* --C=f16:* --providers=cutlass --output=gemm_out.csv. It's not working. And I couldn't find any f8-related code in the profiler.

[thakkarV]

Yes it does. There are two f8 types, e4m3 and e5m2. Look for those specifically and you should be able to find them

[liligwu]

Hi @thakkarV, thanks for your quick reply. Would you please give an example of running command using f8 in cutlass_profiler?

[thakkarV]

cutlass_profiler --kernels=cutlass3x_sm90_tensorop_s64x128x32gemm_e5m2_e4m3_f32_f32_f32_128x128x128_2x1x1_0_tnn_align16_stream_k_warpspecialized_cooperative_epi_nosmem --m=2048 --n=8448 --k=8192

[thakkarV]

or

./cutlass_profiler --operation=Gemm --accum=f32 --m=16 --n=16 --k=1024 --A=e4m3:* --B=em52:* --C=f32:* --providers=cutlass --output=gemm_out.csv

[liligwu]

Hi @thakkarV, Thanks for your examples. Currently, I have a trouble of building cutlass_profiler on H100. The following dockerfile crashes the system.

ARG BASE_DOCKER=nvcr.io/nvidia/pytorch:23.07-py3
FROM $BASE_DOCKER
USER root

ENV WORKSPACE_DIR=/workspace
RUN mkdir -p $WORKSPACE_DIR
WORKDIR $WORKSPACE_DIR

RUN git clone https://github.com/NVIDIA/cutlass.git \
    && cd cutlass \
    && mkdir build && cd build \
    && cmake .. -DCUTLASS_NVCC_ARCHS=90a \
    && make cutlass_profiler -j \
    && cd examples \
    && make -j

The system has two H100-PCIe. I'm attaching the system info and nvidia-smi. I also attempted to cross-compile it on a V100, it crashed the V100 system as well.
nvidia-smi.txt
config.txt

Please let me know if you can reproduce the issue. Thank you.

[thakkarV]

do you have any relevant logs or error messages? I don't see anything indicating a crash in the config or the nvidia-smi output that you have included. the command list does not look out of the ordinary for me. Are you able to build without docker ?

[liligwu]

The build stops at around 93% after make cutlass_profiler -j, and I simply lose the connection to the system. I have to power cycle it. I can try it without docker, but it would be good if it works in a docker for us. Thanks.

[liligwu]

I tried nvcr.io/nvidia/pytorch:23.11-py3. The same outcome.

[thakkarV]

its hard for me to help without knowing what the failure cause is, and since I do not have a docker setup, its not possible for me to repro this either. Adding @jackkosaian since he has a pytorch docker setup though I believe.

[d-k-b]

Can you try with `-j 8` or smaller set of parallel processes. Likely the image is hanging due to resource contention.

…

On Fri, Dec 8, 2023 at 12:18 PM Vijay Thakkar ***@***.***> wrote: its hard for me to help without knowing what the failure cause is, and since I do not have a docker setup, its not possible for me to repro this either. Adding @jackkosaian <https://github.com/jackkosaian> since he has a pytorch docker container though I believe. — Reply to this email directly, view it on GitHub <#1217 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADYUBHQQ62Z6E73AZFEG2UDYINKVRAVCNFSM6AAAAAA77XEP76VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBSGQ3DM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[liligwu]

Reducing the MAX_JOB to 8 works. Thanks @d-k-b , @thakkarV

[liligwu]

Hi @thakkarV , sorry to bother you again. I built the docker successfully, but still have problems running the profiler on f8. specifically,
./cutlass_profiler --operation=Gemm --accum=f32 --m=16 --n=16 --k=1024 --A=f16:* --B=f16:* --C=f16:* --providers=cutlass --output=gemm_out.csv
returns a correct csv file. but
./cutlass_profiler --operation=Gemm --accum=f32 --m=16 --n=16 --k=1024 --A=f16:* --B=e5m2:* --C=f16:* --providers=cutlass --output=gemm_out.csv returns a csv file with header only.
Did I miss anything? Thank you.

[liligwu]

Hi @thakkarV , do you have any examples or documents available for ./cutlass_profiler --operation=Gemm format? Thank you.

[thakkarV]

I don't think we have any OOTB f16 output kernels in the profiler for fp8. you can add them to the generator yourself and then set C to fp16, or switch to using fp32 output. The profiler documentation is here: https://github.com/NVIDIA/cutlass/blob/main/media/docs/profiler.md

[liligwu]

Yeah, it doesn't matter about the output precision. Would you please give a legitimate example of fp8 profiling command in ./cutlass_profiler --operation=Gemm format? Thanks.

[thakkarV]

./tools/profiler/cutlass_profiler --m=16 --n=16 --k=1024 --A=fe5m2:\* --B=fe5m2:\*

works for me just fine, or any other combination of fp8 types, layouts etc. I also noticed that your A type if fp16 but B is fp8. Mixed input gemms are supported in CUTLASS but not plumbed into the profiler yet.

[liligwu]

@thakkarV , this command finally works. Thank you very much!