Supposing that in a rocket failure engineers don't find what the root cause was, what will happen next?
In general if the root cause will never be fully understood is there a law, rule, or standard procedure in spaceflight that tells what to do next for moving forward and starting to operate again? If not and these are situations where the procedure depends by the case, then how will it be managed?
Its hard to answer this in the "typical case" let alone the general case because it will depend upon the manufacturer and customer relationships. Failure review board terms of reference can quite broad and "simply" say that the board must determine the cause and prognosis without reference cost and schedule concerns.
One concept that often arises in commercial, and perhaps wider, circumstances is the "most probable root cause". At this point Organic Marble's description is a good start:
Many times the exact root cause of an incident is never known. A fault tree is generated and all the branches are run down to closure if possible, but often it is not possible. Any open branches (possible causes) will be addressed, so more than one fix could be applied. Sometimes this process is executed and all the answers are wrong (like the initial try of fixing the foam shedding problem after the Columbia accident, and then there was a giant chunk of foam shed on the next flight).
It is quite common in both satellite and launch failure analysis to see many branches of a fault tree given a "not credible" assessment and then to find several attributed to as either "credible" or "credible but unlikely". It may well seem uncomfortable to the outsider but it is not impossible to see lot of money spent on producing evidence against every branch of a fault tree through ground tests and generation of a range of corrective actions and yet the failure review board has only narrowed the range of possibilities for the root cause. Then the "most probable" root cause simply becomes a coin flip between a number of scenarios, all of which happen to be plausible but unlikely. It can be quite unsettling.
If anyone is wondering what I'm adding to this answer myself, it would be that I have taken part in plenty of failure review meetings at equipment/subsystem/system level (mainly for satellites) and I have come to the view that this is a far softer-edged and human process than many people are comfortable with. Satellites and launch vehicles aren't naturally occurring phenomena, every failure comes down eventually a human failing in design or workmanship. Correspondingly every decision to go ahead with a design change also has to be a human made compromise.
If no root cause for the failure is found it should be discussed what to do: another ground test, repeat the launch without any change, add more telemetry and repeat the launch or do more tests during manufacture and assembly of the rocket. The problem is what extra sensors for telemetry might help to identify the cause. Additional high speed cameras inside the rocket may help to understand what is going wrong. But even if the next starts are a full success the team does not know if the problem still exists.
