Storage Gateway VTL experiencing slow upload

We have a storage gateway VTL VM appliance. We have a 1.6 TB Cache drive and 1.4 TB upload buffer.
I have increased the CPU's on the VTL to 8 and increased the memory to 32GB.
I have backups setup to use multiple tape drives.

We are in a co-lo datacenter with plenty of internet bandwidth.

Backups start off good with about 70-80 MB/s, but after several hours it all drops to 12-15 MB/s. There is no bandwidth throttling or QoS rules that would limit speed.

We are a small shop with less than 70 servers backing up and it would seem to me that the cache/upload buffer are of sufficient size.

Any ideas where to start troubleshooting this issue?