Machavity's answer is right to the point, the rights how the territority under the ground is national law and different for countries. The international border extends into the depth of the ground and needless to say, any attempt to cross the border is considered an aggressive act.
Some examples for the better understanding, starting with my own.
While I live in Hamburg, my native region is the Ruhr region in Germany which is severely undertunneled by former coal mining activity. German mining law is quite
unique because ownership of land and mining right is separated. Any
natural resources like oil, gas, metals, salt, even geothermal activity etc. (with the exception of sand, gravel, clay) do not belong the owner, but are considered ownerless until mined which needs a separate admission from the government (Some specific natural resources can even be only mined by the goverment which conveniently can transfer the rights). The owner cannot prevent mining under his ground, but has the right for compensation, especially if the mining activity cause damage.
Image by Ziko C, 2008-05-23, CC BY-SA 3.0
You see those cracks right in the front? These are caused when the mine shafts are collapsing and cause a depression on the surface.
The owner could also get mining rights, but the good old German bureaucracy makes this a real pleasure (This was irony). What is now interesting that once the right of mining is acquired, the right goes to the ewige Teufe in German mining language, right down to the Earth core. So the depth you are exploiting is only limited by technical limitations.
Much more owner-friendly is the law in the Anglosphere, Commonwealth nations only reserve the rights for gold and silver, but give mining rights for a limited time. The US laws mostly acknowledge that the owner owns what is on his land, but reserve some exploitation rights for specific natural resources. Also all rights here are unlimited in depth.
French rights from the Code Napoleon/Code civil are a middle thing which shortened gives the owner all rights for surface resources, mining deeper resources must have the permission of the government.
One interesting situation concerning sovereignty:
CERN:
The very large particle acclerators and most of the structures are built underground and are crossing borders between Switzerland and France. The legal position is quite unique: The foundation members gave CERN as scientific organisation acess as UN observer status and the area is considered extraterritorial area.
Violations of soveignty also regularly occur because underground structures are much harder to detect. The political situation of the Iron Curtain showed many situations where tunneling was used to cross borders without consent.
The former GDR/FRG situation was a hotbed for such situations. Apart from the Cold war conflict between secret services by building tunnels into enemy territorities, most tunnels were attempted escape tunnels from the GDR into the FRG in Berlin.
The most notable one was Tunnel 57, a secret tunnel in 12 m depth which allowed 57 persons to escape.
The GDR in return also had secretly build some tunnels under the border to easily smuggle material and agents over the border.
Image by Global Fish, 2012-04-28,CC BY-SA 3.0
The situation is still a current one between North and South Korea because
North Korea build several well-hidden big attack tunnels in comparatively extreme depth (down to 140 m) penetrating South Korean territority. It is not known how many more are existing, but the existing ones show an impressive level of sophistication.