Is it possible to configure Amazon app so that all the
amazon.co.uk links open in the App (or at least ask if I prefer the browser or the app)?
It's up to the app to register an intent filter for URLs it can understand. When the app does this, and you click an appropriate link, the chooser asks you which you want to open.
If you've previously had a chooser for Amazon links, and accidentally selected to always use a web browser, you can undo this by going into the device Settings, then Apps. Select the browser from the list, and then click Clear defaults.
If you've never had a chooser for Amazon links, then the app can't respond to them. It's possible to use a third-party app to act as an intermediary, so that you open the Amazon links in that app, and it forwards them to the Amazon app, but if the Amazon app doesn't understand the URL you're using, the results are unpredictable: it might crash, or simply open to the main screen. You'll have to contact the developer through their official support channels to ask them to add this feature.
There is no way to force open URLs through an app if it doesn't show up in the popup menu to choose an app. AFAIK there is a filter in applications called the intent filter that is hard coded into the applications code by the developer. The filter is triggered when certain URLs are pressed.
If Amazon hasn't done this you cannot get it to open amazon.co.uk URLs without manually adding it to the apps code.