Let $Z$ be the Sierpinski 2-point space. Then the underlyying set of $C(X,Z)$ is naturally identified with the collection of open sets of $X$, and the specialization order from the compact-open topology is the inclusion order. Thus we can recover the poset of open sets of $X$ from $C(X,Z)$. So if you restrict your question to sober spaces, the answer is yes.
However, for arbitrary spaces the question seems more subtle. For instance, let's consider the case where $X$ and $Y$ are indiscrete. If $X$ is indiscrete, then for any $Z$, it is easy to see that $C(X,Z)$ looks like $Z$ except each maximal indiscrete subspace $A\subseteq Z$ has been replaced by $A^X$. In particular, if $X$ and $Y$ are sets such that $|A^X|=|A^Y|$ for all sets $A$, then $C(X,Z)\cong C(Y,Z)$ for all $Z$, where $X$ and $Y$ are equipped with the indiscrete topology. So the question reduces to the following: if $\kappa$ and $\lambda$ are cardinals such that $\mu^\kappa=\mu^\lambda$ for all cardinals $\mu$, must $\kappa=\lambda$? The answer is yes: if $\kappa<\lambda$ and $\mu$ is a strong limit cardinal of cofinality $\kappa^+$, then $\mu^\kappa=\mu$ but $\mu^\lambda>\mu$.
So the answer is also yes if $X$ and $Y$ are both indiscrete. However, it is noteworthy that this argument might require you to take a very large $Z$ to distinguish $X$ and $Y$. For instance, when $|X|=\aleph_0$ and $|Y|=\aleph_1$, if $2^{\aleph_0}=2^{\aleph_1}=\aleph_2$ and GCH holds above $\aleph_1$, then $C(X,Z)\cong C(Y,Z)$ for all spaces $Z$ of cardinality less than $\aleph_{\omega_1}$. So if the answer is yes in general, this is a sign that proving it probably isn't going to be very easy.