Access internal web application from on-premises IP or local machine

AWS VPC allows the use of Site to Site VPN and DirectConnect for connections between locations.
This allows you to access AWS internal ALBs, etc. from your office network.
https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html

Another option would be Client VPN, but this requires the local machine to be configured with an SSL certificate or SAML authentication.
Perhaps the VPN you are looking for is a VPN between locations, so the above Site to Site VPN is a good choice.
https://docs.aws.amazon.com/mwaa/latest/userguide/tutorials-private-network-vpn-client.html