There are four types of Fallen Empire, and each one has different things that anger them, based on their ethics.
Keepers of Knowledge (Fanatic Materialists): The Fallen Empire dedicates itself to the hoarding of technologies that it believes should be kept out of the hands of young and irresponsible races. Pursuing dangerous lines of research will draw their ire. (Technologies considered "dangerous" are marked in red)
Notably, AI and Jump Drives
Enigmatic Observers (Fanatic Xenophiles): Empires that use slaves, or purge populations may suffer the wrath of the Observers.
Holy Guardians (Fanatic Spiritualists): This fallen empire dedicates itself to the preservation and defense of its holy sites. Colonizing systems that they consider sacred is likely to incur their ire.
Militant Isolationists (Fanatic Xenophobe): The fallen empire dedicates itself to the defense of its borders against any intrusions. Colonizing systems that border them is likely to draw their ire.
Other things, such as insulting, rivalling them or declaring war (obviously) will anger them as well.
Fallen empires may also make requests or demands of you, and refusing to comply can reduce their opinion, whereas complying will increase it.
If you have the Synthetic Dawn DLC, there is a fifth kind of Fallen Empire: Ancient Caretakers (Machine Intelligence). The Caretakers are normally quite passive and won't object to anything you do short of actually declaring war on them. They will occasionally request permission to inoculate your population against an unspecified disease (this has random effects), or ask you to change your policies in various ways, but they don't mind if you refuse.
The Ancient Caretakers are the guardians against the AI Contingency disaster. If this happens, there is a 2/3 chance the Caretakers will awaken to help fight it; but a 1/3 chance that their firewalls fail and they start attacking everything instead. Obviously, depending on your own handling of the Contingency, this can bring the Caretakers into conflict or alliance with your nation.
Source: Stellaris Wiki