SOLUTION : $(A B)^{\top}=A^{\top} B^{\top} \Rightarrow(A B)^{\top}=(B A)^{\top} \Rightarrow A B=B A$
so $(A, B) \in R \Rightarrow A B=B A$
(i) reflexive $\Rightarrow(A, A) \in R \quad \Rightarrow A A=A A$ true
(ii) $(A, B) \in R \Rightarrow A B=B A \Rightarrow B A=A B \Rightarrow(B, A) \in R \Rightarrow$ symmetric
(iii) Take $A=\left[\begin{array}{ll}2 & -3 \\ 1 & -2\end{array}\right], B=\left[\begin{array}{ll}1 & 0 \\ 0 & 1\end{array}\right], C=\left[\begin{array}{cc}1 & 0 \\ 0 & -1\end{array}\right]$, then $A B=B A$ and $B C=C B$ but $A C \neq C A$
$A C=\left[\begin{array}{ll}2 & 3 \\ 1 & 2\end{array}\right], C A=\left[\begin{array}{cc}2 & -3 \\ -1 & 2\end{array}\right]$
so not transitive.