ASPLOS'24 paper on EVT (Epilogue Visitor Tree)

[hwu36]

@apuaaChen is going to present his CUTLASS paper - EVT: Accelerating Deep Learning Training with Epilogue Visitor Tree in ASPLOS'24 on May 1.

EVT is a framework to fuse almost any combination in the epilogue (elementwise, reduction, broadcast, etc.). If you attend ASPLOS and use CUTLASS, highly recommend to talk with him.

[ginowu]

On page 13 of the slides(CUTLASS: Python API, Enhancements, and CUTLASS 3.0 Preview) in GTC22, it lists some advanced fusion patterns that supported by cutlass-python, do you recommend using EVT to realize these patterns? If so, could you elaborate on the details to achieve it?

Mainloop	Elementwise compute	Conversion, scaling, masking
Reduction of A or B	GEMM + bias grad
Broadcast	Load vector and broadcast over channels
Epilogue	Multiplicity of tensor operands	GEMM + GELU with Aux tensor; GEMM + RELU with output bitmask GEMM + dRELUloading bitmask
Multiplicity of vector operands	Alpha and beta scaling as vector
Broadcast over column	Bias add
Reduce over column	GEMM + bias Grad Layer norm
Reduce over row	GEMM + Softmax
Elementwise compute	Arithmetic, conversion, activation functions