This issue comes down to ownership, boundaries and precisely where the wall sits on the boundary. Planning permission has nothing to do with whether it is permissible to knock a wall in the context of a boundary issue between neighbours. The planning affects the building itself.
If the wall sits entirely on their property, your permission is not needed for the wall to be knocked.
If the wall sits entirely on your property, the wall can’t be touched.
If the wall abuts or straddles both adjoining properties (ie a “party” wall), then your permission is required.
The builders cannot enter or pass through your property to do the build without your permission.
In any event, you would need to take a look at the planning documents pertaining to the extension (usually available for free online on your local authority’s website). And your own title documents to see the legal position concerning the wall. If it’s a terraced house I suspect it’s a party wall.