Handle Cosmos DB 429 Errors Within Cosmos DB Connector
In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.
The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.
Ideally some retry logic should be built into the Cosmos DB connector so it handles the error and uses the x-ms-retry-after-ms header that is returned to determine how long to wait before trying again.