Session: Sailing Multi-Host Inference with LWS

[Slides]

Inference workloads are becoming increasingly prevalent and vital in Cloud Native world. However, it’s not easy, one of the biggest challenges is large foundation model can not fit into a single node, which brings out the distributed inference with model parallelism, again, make serving inference workloads more complicated.

Emirates Stadium.

Anfield Stadium.

Raining in Old Trafford.

Etihad Stadium.

Tottenham Hotspur Stadium.

As a follower and active contributor for inference platform, I created the llmaz project to provide an unified inference platform for LLMs and also joined the AIBrix community to build the next-gen GenAI infrastructure.

I just read the article about the business mode with open source commercialization in fit2cloud, which is really impressive, so I wrote down some highlights here for future reference. Note that, the original article was written in Chinese.

Due to DeepSeek-V3 technical report, it says:

In addition, both dispatching and combining kernels overlap with the computation stream,
so we also consider their impact on other SM computation kernels.
Specifically, we employ customized PTX (Parallel Thread Execution) instructions and
auto-tune the communication chunk size, which significantly reduces the use of the
L2 cache and the interference to other SMs.

then people are saying like DeepSeek is breaking the Nvidia core moat - CUDA by employing the PTX directly, but is that true?

Phi Phi Islands.