Appearance
Source: nn/FFN.luau
nn/FFN.luau
SwiGLU feed-forward block: FFN(x) = W3(silu(W1 x) .* (W2 x)).
new(name: string, C: number, hidden: number, rng: any)
Create a SwiGLU FFN block [C -> hidden -> C].
forward(: any, x: any) -> any
SwiGLU forward: W3(silu(W1 x) .* (W2 x)); flattens >2-D leading dims.