One approach, mentioned by Pace Nielsen in the comments, is to start with what I call strict formalism. The only substantive assumption required for strict formalism is that you are capable of recognizing and manipulating finite strings of symbols in certain simple ways, and can understand what a syntactic rule is at the level of being able to confirm or disconfirm whether a particular syntactic rule is being obeyed. So for example, it is assumed that you do not balk at assertions such as, "$\phi$ is the same symbol as $\varphi$" or "appending the symbol 0 to the symbol 1 yields the string 10". If you are bothered by the concept of a symbol, or of a syntactic rule, then I don't think that you will be able to develop anything remotely resembling mathematics.
In strict formalism, the symbols do not mean anything. It is assumed that you have the capability of verifying that if you start with certain strings and apply certain rules then you will arrive at a certain result. But there is no assumption that you are able to reason mathematically. For example, consider the rule "append a 0" and consider applying that rule some finite number of times to the string "0". As a strict formalist, you will be able to confirm that at some point, the string 0000 will emerge. However, consider the following claim: "At no point will this process ever produce a string with the symbol 1 in it." While this claim may seem ridiculously obvious, it is not a claim that a strict formalist can deduce. Arriving at such a conclusion requires reasoning about symbols, and a strict formalist is assumed only to be able to carry out symbolic manipulations, not to be able to reason about them.
A strict formalist can verify any formal proof produced by a mathematician, and so in that sense can reproduce all the formal content of mathematics, no matter how arcane an axiom is invoked. But a strict formalist will not, for example, be able to tell us something like this: "If you write a computer program to search for strictly positive integers $a$ and $b$ such that $a^2 = 2b^2$, then the program will never halt." The strict formalist will be able to execute such a computer program, and can even verify a formal proof that $\sqrt{2}$ is irrational, but as far as the strict formalist is concerned, the formal proof that $\sqrt{2}$ is irrational is simply a meaningless sequence of symbolic manipulations, and tells us nothing about "reality."
If you want to recover the ability to claim with confidence that computer programs that search for positive integer solutions to $a^2 = 2b^2$ are doomed to fail, then you need to take an additional step beyond strict formalism. Namely, you need to claim that there is such a thing as sound mathematical reasoning, and you need to lay down the principles that you think are trustworthy. At this point, it's basically up to you to decide what principles you trust. Most mathematicians seem to be happy with ZFC, but others are uneasy with it and prefer to back off to some set of more modest principles. For almost every mathematician $M$, it is possible to write down a set of formal axioms with the property that anything the strict formalist deduces from those axioms will be accepted by $M$ as a true mathematical statement. So again, in that sense, the strict formalist can reproduce all of mathematics. But the set of formal axioms accepted by $M$ will vary as $M$ varies.