Phi 3.5 MOE and supporting MOE in general

[niceblue88]

What does it take to get a new MOE model running in exl2 format? Obviously there are a few in the past, notably Mixtral 8x7b and DBRX. What configuration or functions needed to be added to support, say, Phi 3.5 MOE? And can this process be generalized for any future MOE LLMs released. I'm happy to try to do it, but not sure how to start,

[niceblue88]

realised there was another thread important for this: #512