SWAP gate for qutrits

The formula is actually well-known and the general case is such that the Pauli operators on the two systems are inverses of each other.

To see this, let us define the $Z$ and $X$ operators on the computational basis $\{|x\rangle\}_{x\in\mathbb{Z}_d}$ of $\mathbb{C}^d$ as $X|x\rangle = |x\oplus 1\rangle$ and $Z|x\rangle = \omega^x |x\rangle$ where $\oplus$ is addition modulo $d$ and $\omega = e^{2\pi i/d}$ is a primitive $d$-th root of unity. The operators $Z^z X^x$ form an orthogonal basis of the space of linear operators $L(\mathbb{C}^d)$ and $Z^z X^x = \omega^{zx} X^x Z^z$.

Computing the expansion of SWAP in this basis can be obtained by evaluation Hilbert-Schmidt inner products, using $\mathrm{SWAP}^\dagger=\mathrm{SWAP}$ and the SWAP trick $\mathrm{tr}(\mathrm{SWAP} A \otimes B) = \mathrm{tr}(AB)$: $$ (\mathrm{SWAP}, Z^z X^x\otimes Z^u X^v) = \mathrm{tr}(\mathrm{SWAP}^\dagger Z^z X^x\otimes Z^u X^v) = \mathrm{tr}(\mathrm{SWAP} Z^z X^x\otimes Z^u X^v) = \mathrm{tr}(Z^z X^x Z^u X^v) = \omega^{ux}\mathrm{tr}(Z^{z\oplus u} X^{x\oplus v}) = d\,\omega^{ux} \delta_{z,-u} \delta_{x,-v} \,. $$ In the last line, we used that the $Z$ and $X$ operators are traceless, thus the only way the trace does not vanish is if the operators are raised to the power of zero (and thus $Z^z X^x$ and $Z^u X^v$ are inverses of each other).

Hence the formula is $$ \mathrm{SWAP} = \frac{1}{d}\sum_{z,x} \omega^{-zx} Z^z X^x\otimes Z^{-z} X^{-x} \,. $$ However, we typically choose differently normalized operators, namely so-called Weyl operators, defined as $$ W(z,x) = \tau^{-zx} Z^z X^x \, $$ where $\tau = (-1)^d e^{i\pi/d}$ is such that $\tau^2 = \omega$ for all $d$ (note that $\tau=i$ for $d=2$ and we obtain the standard Pauli operators). With this convention, $$ \mathrm{SWAP} = \frac{1}{d}\sum_{z,x} W(z,x) \otimes W(-z,-x) \,. $$