The Photos app normally uses photos that are stored in the Pictures folder inside of your home directory, although you can change the location that the app looks for and stores photos in.
rm -rf * means delete all files and folders in the current directory without warning me that I might be deleting something important or asking if I'm sure.
So this would have deleted the photos in the current folder. It would also delete things like your user Library and all the settings for your user account, which can make the account unusable.
Using disk utility to reformat the hard drive erases the entire computer, operating system and all.
To answer your question, unless someone used some kind of forensic tools to take apart your hard drive, your photos are gone.