Specify outbound caller ID using Connect Streams?

We want to give our Streams application the ability to specify the caller ID for outbound calls based on the destination (based on value in our system, not based on country or area code). How can we initiate an outbound call for a specific agent that will use a specific caller ID number? I would think that this should be done by passing an attribute to the outbound call (we are currently using agent.connect() to initiate the call) and to then use a Lambda function called from the outbound whisper flow to specify the caller ID, but it doesn't seem possible to pass any attributes when initiating a call. We are open to any possible solutions.