When a custom query is used for a copy activity, the query gets cut off and does not display the entire query. There are actually three issues that occur currently.

1) The entire query should be visible or placed in a text area with read only properties.

2) There should be a copy button on the query so that we can execute the exact query that was used. This is for troubleshooting errors that occur.

3) Make the initial textbox that is used for the input of the query larger and add in formatting functions so viewing potential syntax errors are easier.

In our use case we are bulk loading data to Cosmos DB and have a requirement to scale each collection up at the beginning of a load and down at the end.

The scaling is performed by an Azure Function and we have seen issues where Cosmos DB returns a 429 error when performing metadata requests against Cosmos DB within the copy activity that comes after the Azure Function. This occurs frequently when running multiple pipelines in parallel. When a 429 error is received on a metadata request the error bubbles up and causes the pipeline to fail completely.

Ideally some retry logic should be built into the Cosmos DB connector so it handles the error and uses the x-ms-retry-after-ms header that is returned to determine how long to wait before trying again.

My use case is this:

I have a "Do Until" loop that checks a datasource for a particular set of files. If the files exist (based on the expression field of the do-until activity itself, then I need for the activity to continue.

What I expect of the output:

If the files do NOT exist, and a timeout is reached for what I have configured for the timeout of the "do-until" I would expect the outgoing result be something like "Skip" or "Timeout" or "Fail".

What actually happens:

However, if the Do-Until loop activity times-out because the expression (or condition) is never met, then, the output of the Do-Until loop is success.

Requested Change:

If an activity times out, it should NOT be a success, rather something like: Timout, Fail, or Skipped.

Thanks,
Matt

I created a master pipeline that execute other child pipelines. If there is an error in one of the child pipelines, and I want to fix the issue and rerun (resume is not available) failed child pipeline the parent pipeline doesn't resume. I have to rerun the the parent from the very beginning. Which forces me to reload all the data from source to stage and then from stage to EDW.
This is really ridiculous. At least show the activities in the monitor that were scheduled but didn't run due to a child pipeline failure and allow us to manually run the activities that were in a waiting state. Why do you guys always have to design something that doesn't serve real-time scenarios. This is a very fundamental requirement for any production system

Example on the pictures show the Data Copy Activity in this case I am copying data from a source (OData Feed) to a destination database (Azure SQL Database), then specified the query for filtering on the ModifiedOn columns of the tables in my source data.

I want to be able to also provide a query that will filter different tables along with different columns instead of providing a query for only ModifiedOn column.

E.g table 1 has a column named ModifiedOn (datetimeoffset)
table 2 has a column named CS_ModifiedOn (datetimeoffset)

I want to provide a query that will filter accommodate all the different columns.

So I have tried this query below but I am getting an error also below.

$filter= ((ModifiedOn ge @{adddays(pipeline().TriggerTime,-1, 'yyyy-MM-dd' )}T00:00:00Z and ModifiedOn le @{adddays(pipeline().TriggerTime,-1, 'yyyy-MM-dd' )}T23:59:59Z) OR ( CS_ModifiedOn ge @{adddays(pipeline().TriggerTime,-1, 'yyyy-MM-dd' )}T00:00:00Z and CS_ModifiedOn le @{adddays(pipeline().TriggerTime,-1, 'yyyy-MM-dd' )}T23:59:59Z))

Error
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorODataRequestNotSucceeded,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Requesting response from path AP_ActionPlans and query $filter= ModifiedOn ge 2018-06-28T00:00:00Z AND ModifiedOn le 2018-06-28T23:59:59Z failed with status code BadRequest.,Source=Microsoft.DataTransfer.Runtime.ODataConnector,'",
"failureType": "UserError",
"target": "Copy_hpz"
}

Currently my single pipeline consist of 20 pipelines each of them represented as 20 activities in a pipeline. Each of 20 activities perform
combination of Copy and Transform functionality and some are common input to next activities in same pipeline.
If any such activity in between fails, say my second last activity failed I do not have an option to rerun only my second last and then last activity out of 20 . But I need to trigger this big pipeline which re-runs complete 20 activities which is duplication and also additional usage of resources like analytics platform which may not be required.