Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

  1. ADLS Gen2 Backup and Point-in-time restore

    Can you add an option to backup data stored in the ADLS Gen 2 'out of the box' i.e. at specified intervals and retention periods, similarly to how we can configure backup options for Azure SQL?
    The current workarounds (e.g. using ADF to copy data from one ADLS account to another ADLS account) are insufficient. E.g. how are we supposed to recover from accidental file deletions or from malicious damages (for example files being damaged or encrypted maliciously e.g. ransomware)?
    I would like to be able to specify a backup option for each ADLS Gen2 account e.g. daily backup with…

    92 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Create new permission to allow read metadata/properties

    A key scenario for us on the Microsoft Azure Data Lake is the ability to grant permissions to a set of users (mostly developers and support) to inspect the current state of files. E.g.

    1- Check if a file has been uploaded, when
    2- Check the size and other properties of a file to compare to a known source (when uploading files from other locations).

    At the moment it does not seem possible to grant those permissions by any combination of RBAC or ACL (we have tried, including involving Microsoft Support).

    It seems like a suitable option is to create…

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  3. Add Power BI service as a "Trusted service" to Azure data lake gen2 firewall

    Power BI dataflows are an excellent way to leverage the power of Azure data lake gen2. The feature for data flows to store to data lake is currently in preview, but I thought I would get this idea in as I believe it will be a blocker for many organizations.

    Data lake best practice (and security center default policy) is to activate a firewall on data lake. However, if we do this, dataflows configuration event and data storage fails, unless the power BI service IPs are added as firewall whitelist rules. Even if they are added, they change, making the…

    10 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Azure Role “Data Lake Analytics Developer”

    Data Lake Analytics Developer is one of the Azure RBAC roles. This role does not describe the level of rights or where it is used. Can you please give us some scenarios of how this is being used? I

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Add support for ADLS Gen2 to ADLA

    Would like the ability to use the ADL Analytics job-as-a-service to work with data in ADLS Gen2.

    160 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    23 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Add a BlobModifed to ADLS Gen2 events

    Currently ADLS gen2 sends out BlobCreated and BlobDeleted events to an event grid topic. There seems to be no way of getting notifications when blobs are changed in the data lake. It would be very useful to also have a BlobModified event.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Data Lake Gen 2 - Discovery on Portal & Pricing Calculator

    Azure Data Lake Gen2 is not easily found on the Azure portal. It's available as an advanced option under Azure Storage. If you search "Azure Storage" or "Data Lake", you see "Azure Data Lake Gen1" appears on top of the list. On the Azure Pricing Calculator web page, you don't see ADLS Gen2 under search terms of "Storage" and "Data Lake". Not very intuitive to find and use such an important Azure resource.

    Azure Data Lake Gen 2 should be as easily discoverable on the Azure portal or Azure Pricing Calculator web page as Azure Data Lake Gen1 is.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  8. Assigning ACLs for existing child items

    We need a way to recursively apply ACLs to subdirectories or folders within a Data Lake Storage Gen2 resource, meaning, for them to be inherited for existing items at the time of the ACL creation operation is executed

    6 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  9. Rest API Method For Setting File Expiry on ADLS Gen1

    The existing REST API methods do not have an operation defined for setting file expiry date for a file stored in ADLS Gen1. It would be great to have this feature enabled so that it can be utilized from standalone Azure services like Logic Apps.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Introduce distinct "readPermissions" and "updatePermissions" Action

    Our tools need to read the ACLs assigned to users/applications, without having read access to data in storage.

    Creating a custom role would grant the ability to modify ACLs which is a more problematic alternative.

    Is there a plan to split the current "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/modifyPermissions/action" permission action to two read/write ones.
    Could we get an ETA on the requirement.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  11. Update the R extensions in Data Lake Analytics

    The Data Lake Analytics is a great product and I really like the functionality I've seen so far, especially the integration of R and Python with the USQL extensions. However, the R version is extremely outdated (over 2 years old, I believe) and severely limits the benefits of having this integration. Many R packages which include relatively new and novel methods cannot be used. It would be great to see the extensions for R upgraded to the newest version. Thanks!!

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  12. Add Power BI Dataflows to Government Cloud

    This is a great feature, particularly the connection to Azure Data Lake Storage Gen 2. We would love to use it in US Gov Cloud. Please add it as soon as possible! Thanks.

    19 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  13. Create Permission Set to Restrict Delete

    he way our team uses data lake is append only. This allows us to have a historical view of our data. However it seems we cannot create a permission set that fits this model. In order for a user to be able to create objects they must be able to delete them.

    Having all of our users not be able to delete would ease some concerns.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  14. 158

    Upgrade stroge 200GB

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  15. Out Of Memory Error Data Lake Analytics

    The fact that we had to develop this workaround is disappointing to me. This should be a built-in feature. The ability to process a 1 GB file should be an afterthought by modern standards. I cannot recommend this platform to our clients at this time. Please let me know if this feature gets developed, so I can re-evaluate this platform as a potential replacement for the big data systems currently used by our global clients.

    To be clear, this case was closed by using our workaround (which we are happy to share our code with you) per case 119112224002060. I…

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  16. RBAC + ACL Security for ADLS Gen2 is backward

    After reading the docs and speaking to a MS storage rep, I've learned that in ADLS Gen2 RBAC overrides ACL, and you must have at least a Reader role assigned via RBAC in order to even see the storage account. This is backwards and broken IMHO - it makes it impossible to allow someone access to only a single folder somewhere in a Filesystem. This is a show stopper for storing any kind of sensitive data (HR, Finance, etc) in ADLS Gen2 where you have a need to totally block access to some data for a person while simultaneously allowing…

    9 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  17. Read Delta Lake files with Data Lake Analytics

    Call to Arms for being the better cloud for Databricks!

    Realizing the increased usage of Delta Lake (https://delta.io/) for big data it would be very helpful to have the ability to query Delta files using Azure Data Lake Analytics on ADL. Today Azure is not up to competition with AWS Athena that is able to do just this on s3, albeit not perfectly.

    Best,
    Andreas

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  18. 7 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  19. Enable Out of the box Data Quality Capability for Azure Data Lake

    Enable Out of the box Data Quality Capability for Azure Data Lake

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  20. Support copy of Empty folders in data lake

    Currently while copying multiple folders in data lake gen1/gen2 empty folders are skipped.
    This causes big issue in partitioned data like Parquet/Orc where the corresponding hive tables are corrupted because of missing partitions.
    Apart from this the folders in the datalake are created for a reason even if empty and shouldn't be skipped anyway as it has huge implication on backup-restore approaches.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3 4 5 17 18
  • Don't see your idea?

Data Lake

Categories

Feedback and Knowledge Base