If young hackers can detect them, why can't security analysts get and patch them?
Also, I dream of finding a vulnerability in a major software, like IE or Chrome. Is it easy? Is it systematic or is it random? What books do I need to read to detect them in due time?
I now know how to program in C/C++/Java. I can easily learn Assembly.
It's a race between the people who want to fix the vulnerabilities and the people who want to exploit them. Finding vulnerabilities is difficult — writing correct code is a lot more difficult than writing almost correct code.
Suppose (numbers pulled out of thin air) that 99% of security defects in released products are found and fixed by the authors themselves, or reported privately to the authors. That means 1% of security defects are made public through an exploit (a zero-day). You only hear about the publicized exploits, not about the ones that are quietly fixed — so that 1% is all you hear about. The authors and analysts have to fix every single vulnerability in order to make a completely secure product; the “hackers” only need to find one unfixed vulnerability.
But it's actually worse than that: even in the 99% case, when the authors release a new version, not everyone will upgrade immediately. There's another race: the race between exploitation and patching. Once the upgrade is available, crackers can study the differences with the previous version and build exploits for older versions that people are still running. This leaves a lot of exposed machines.
There are a number of techniques to look for vulnerabilities, used by both sides alike. Fuzzing is one of them: try feeding random data to a program, and see if it reacts in an interesting way. For example, if the program crashes (which in itself is often a denial of service), study the way in which it crashes carefully; often crafting the input data then lets you escalate to executing arbitrary code. Another approach is static analysis: run automatic analyses on source code or compiled code, to look for suspicious patterns (e.g. look for unchecked array accesses) or verify the absence of certain kinds of bad behavior (e.g. prove that all array accesses are unchecked). Static analysis can be complemented by manual reviews, to look for aspects that automated tools aren't good enough to catch (e.g. review that all temporary files are created securely). Experience helps, of course — sometimes someone will think “hey, this is a difficult problem, I wonder if the developers got that case right”.
I recommend reading the presentation of OpenBSD's audit process and looking further at the OpenBSD project. This is an open source project, with open discussions, and they make security a priority. I don't know of a more comprehensive presentation of their security auditing process; you can find out more about it by scouring their mailing lists.
More generally, look at security announcements for open source software (e.g. a Linux distribution). When a security patch is announced, look at the description of the vulnerability, at the original source, and at the patch. Try to get a feeling for what patterns can be problematic, how they were found (the vulnerability announcement often has a link to a more detailed write-up) and how they were fixed.
Let me try to tackle the "patching" part of your question: Finding problems - at least in software - is an order of magitude easier than fixing them. Consider the shortest possible list of obstacles to overcome:
0) Is it a bug or a feature?
1) Is this the actual defect, or just a manifestation of a deeper problem?
2) Is the problem severe enough to warrant a fix?
3) Is this even fixable? (e.g. if your program was written under the assumption that supporting dates beyond the year 2038 is pointless or that nobody would ever need a file larger than 4 GB, a complete rewrite might be necessary)
4) If we fix this, whatever depended on the behavior will break - is that acceptable, do we need to devise workarounds, what are the odds of those workarounds having their own bugs?
5) "Fixing one bug always introduces at least two new bugs"
6) Is there a (time and money) budget for fixing the bug, testing and deploying the fix?
And this is all before company politics kick in - after all, some companies have been known to refuse to acknowledge the bug reports and play dead (for whatever reasons).
A few more thoughts:
If young hackers can detect them, why can't security analysts get and patch them?
Keep in mind
for any given product there are a fixed number of people creating and fixing the product and an almost limitless number of people available to break or exploit the product. The number of people trying to break the product can be roughly correlated to the popularity of the product, it's prevalance, and the nature of the information that it handles or protects. So - lots of people are trying to break IE (very popular), online banking software (high value) and not many people are trying to break a niche product created for an obscure purpose.
"young hacker discovers big bug in product" is a great headline. Consider your sources when it seems like only the young guys are finding bugs. "Crusty old guy breaks product in the course of doing business" isn't going to sell any online papers. The young hackers are finding bugs and making headlines. Old hackers are typically called "security analysts" and plenty of them have jobs that involve quietly finding exploits (just a little big tongue in cheek).
Fixing a bug isn't all that hard. Fixing a bug and not breaking anything else is often quite difficult. Worse yet are the bugs that arise from misuse that simply cannot be fixed.
Also, I dream of finding a vulnerability in a major software, like IE or Chrome. Is it easy? Is it systematic or is it random? What books do I need to read to detect them in due time?
If you are specifically interested in web browsers, I suggest you get very familiar with the technology surrouding the Web - HTML rendering, Java Script, Active X, AJAX, SSL, DNS, CSS - anything and everything that is passed to and processed by the web browser. An intimate understanding of these technologies and how the browsers work is the key to this area.
There is no one book. There is not even a reading list. This area changes FAST - new fixes to major browsers are published extremely frequently, new technologies and new browsers come out annually. The development cycle in this area is actually faster than the publication cycle for anything but an e-Book. You can get some decent background reading on the technologies I mentioned, but pretty quickly your going to want to readup on the actual WWW standards (like www.w3c.org) so that you know the up to date details on these technologies.
Hacking is both systematic and random. You need to know enough about the domain in which you are hacking to be able to make reasonably good guesses about what to try in order to find an exploit. Almost any modern product is likely to be too complicated for you try absolutely every input and output, so some degree of good guessing will be required to take a stab at a likely area of vulnerability. The best analysis I know are those who see the system as a whole, and are able to quickly drill into most areas and see how the conjoined enviromental factors can contribute making a product vulnerable.
I now know how to program in C/C++/Java. I can easily learn Assembly.
Quite honestly, I don't see how Assembly will help you. On occasion, it helps to know the language of the code base of the product - IF you have access to it. But more important is knowing what the product is supposed to do, and to be able to analyze it effectively to figure out where it's failing to do what it's supposed to do, or where what it's doing is producing undesirable results.
The answers here are great. Here is another way to think about it.
All software has bugs. A standard defect rate is 2-10 defects per thousand lines of code; some fraction of these will be security-relevant defects (of varying severity). Browsers have millions of lines of code. So for all we know, a typical browser might well have hundreds or thousands of security defects lurking, just waiting to be discovered.
Let's assume there are 500 unknown security vulnerabilities in the browser, which have not yet been discovered. Let's say that if you put in 500 hours of work (fuzzing, design review, code review, etc.), you will discover one vulnerability -- one drawn randomly from the pool of 500 vulnerabilities.
How long does it take for an attacker to find his first previously-unknown vulnerability? 500 hours = 3 person-months.
How long does it take for a defender to find all of the vulnerabilities, so they can be fixed? At least 500*500 = 250,000 hours = 125 person-years. In fact, that's an underestimate. Due to a mathematical phenomenom known as the coupon-collector's problem, the actual number is more about 500*log(500)*500 = 1.5 million hours = 776 person-years. This doesn't even take into account the amount of work to fix the problems once found, to test to fixes, to push them out to users -- or the likelihood that some fraction of the fixes might introduce new problems of their own. In practice, the defender's plight is even worse than this might suggest.
So you can see that the attacker has a tremendous advantage over the defender. With a little bit of effort, the attacker can find one vulnerability (since the attacker doesn't care which one he finds; any will do); whereas it takes a tremendous amount of effort for the defender to find and fix them all. Because the defender must find all vulnerabilities, while the attacker only needs to find one, the attacker has a tremendous advantage.
This is one reason why folks say "you can't bolt on security after the fact" or "you can't test security in". If you've built the software with subpar security practices, no reasonable amount of testing after-the-fact is likely to be enough.
The above mentioned answers are really great and I'd like to add something from the perspective of computational complexity, specifically to address the issue of finding vulnerabilities in any general case (or in any given code). The idea is similar to finding obstructions, which are in some sense "proof" that a computation can't be completed in "realistic" time (polynomial time). Now finding a polynomial time algorithm that can provide the existence of an obstruction family seems unfeasible.
An obstruction family is simply a set of obstruction labels where each label points to a well defined and explicitly constructed obstruction. So imagine a scenario where we know all the members of this set, then we can easily patch them but otherwise for finding members of this set, one would have to try every possible combination which isn't realistic. In practice, finding them would thus seem random however there can be clues to a particular procedure where vulnerabilities may arise so we can definitely find some members, but not all.
Finding zero-day vulnerabilities is even more difficult, however as I see in other answers, there some checklists that can be used to assist you in the process. Overall it is still a hit or miss case, some people take years to find them while others find them relatively quickly.
Another connection I see is especially with browsers, it may be easier in some sense to find vulnerabilities since there are so many components to the browser itself, as you may already know, metasploit database keeps on piling up with those. So that maybe a good place to start and if I may suggest, track down one specific vulnerability and read up on how it was found, then try to find it yourself, that'd give you good practice. Have fun :)
