Validate Basic Screen Pop Failing

I have recently set up an integration between a Salesforce sandbox and a test instance of Amazon Connect. After installing the CTI and creating a new AC CTI Adapter as per instructions from Amazon the last step of validating the "basic screen pop" fails.

I am using v5.7 of the AC Salesforce package (because this is what we have in Production right now) and I have not set up any CTI Flows, Attributes, Presence Synch Rules or Features on the CTI Adapter as according to the instructions the screen-pop should work already. However when I call in from a phone number that is noted in one of the Contacts in the org (using E.164 notation, e.g. +44xxxxxxxxxx with no spaces) the contact record does not pop up. I have validated that no other contact in the sandbox has that phone number so it is unique.

Do I need to set up the CTI flows etc. for this to work? Are the instructions wrong in indicating that screen pop should work after a basic setup? Or maybe there is a browser issue (It is the latest version of Chrome I think)?