How can we improve Azure Storage?

Provide Time to live feature for Blobs

If I need to provide a user (or external system) some data (blob) which might be outcome of some processing (or other) and it has some expiration time I'd like to just put a new blob and set TTL property with TimeSpan (or set absolute DateTime). When the period is over my blob is deleted. So I don't have to pay for it and don't need to spin up some service for doing it myself.

959 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    I agree to the terms of service
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    AlexLF shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    We are pleased to announce that we have made an Azure Logic Apps template available to expire old blobs. To set up this automated solution in your environment: Create a new Logic Apps instance, select the “Delete old Azure blobs” template, customize and run. We will release a blog post detailing instructions and providing more templates in the coming weeks.

    Allowing users to define expiration policies on blobs natively from storage is still planned for the coming year. As soon as we have progress to share, we will do so. We will continue to provide updates at least once per quarter. 

    For any further questions, or to discuss your specific scenario, send us an email at azurestoragefeedback@microsoft.com.

    24 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      Submitting...
      • Ramon de Klein commented  ·   ·  Flag as inappropriate

        We implemented this on our BLOB storage and in two days the logic app resulted in a bill of $233 (logic app only). I cannot even consider this logic app a solution. It's far more efficient to create a Powershell/C# program that lists the blobs and removes the old ones instead.

        The only proper solution is to provide a TTL like AWS S3 does, so the deletion actually transparent.

      • Ramon de Klein commented  ·   ·  Flag as inappropriate

        I have implemented a logic app (see https://sameeraman.wordpress.com/2017/08/25/logic-apps-delete-files-older-than-x-days-from-a-blob-storage), but I don't like this solutions for a variety of reasons:
        1. I need to maintain a logic app.
        2. I need to pay for the logic app when it triggers.
        3. It doesn't seem to scale well. Listing all blobs to an array is madness with a container with >1M of objects.
        4. I get "Rate limit is exceeded. Try again in 4 seconds." when deleting the blobs.

        The AWS S3 lifecycle method (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) is much more convenient and user-friendly.

        It was confirmed to be implemented in 2017, but the only thing we got was a louse logical app, that doesn't work well. Please come up with something better.

      • Oliver Tomlinson commented  ·   ·  Flag as inappropriate

        Just received this MS after chasing :

        "We actually will have something published shortly that will enable this with Logic Apps (some final testing underway on this), while the full API/policy based version of it will be shipped in CY18 per the current plan."

      • Oliver Tomlinson commented  ·   ·  Flag as inappropriate

        Please allow us to define a TTL/expiry on a container that applies to all blobs in the container, not just individual blobs! Thank you.

        P.s. where is the update from the Azure Storage Team on this?

        Your last response has been over 7 months, and not the "at least once per quarter" as promised.

        Why doesn't the Azure Storage Team expose a public road map like other Azure Teams?!?

      • Guy commented  ·   ·  Flag as inappropriate

        Please add support to be able to maintain the blob store capacity
        we need an eviction policy to delete lets say x days of unused blobs
        also, would be great if this can be integrated with the alerts system so when an alert for say a capacity is breached this eviction policy also triggers with a param for days set by the alert.

        as for now there is a powershell script and not an integrated solution, and the script literally itrates over all the blobs in the storage

      • Darian Miller commented  ·   ·  Flag as inappropriate

        Please add the ability to set the original 'creation date' so that when data is initially migrated to these containers, it can expire as expected. (Say you import 3 years of transactions and the data expires in 5 years...you don't want to have some of this data on file for a total of 8 years just because the import date becomes the creation date.)

      • Shahiddev commented  ·   ·  Flag as inappropriate

        I think with the introduction of the cool storage, the native support for TTL on blobs is even more useful/important.
        Using Azure functions we can definitely achieve this reasonably easily however nothing beats native support

      • Žygimantas Beržiūnas commented  ·   ·  Flag as inappropriate

        I would like to extend the suggestion - expiration should be supported not only on blobs, but on queues as well. That would enable creating temporary queues.

      • Žygimantas Beržiūnas commented  ·   ·  Flag as inappropriate

        I would like to extend the suggestion - expiration should be supported not only on blobs, but on queues as well. That would enable creating temporary queues.

      • Phil commented  ·   ·  Flag as inappropriate

        This should actually be possible using Microsoft's azure event functions with triggers on blob creation to schedule their deletion

      • Santanu Dutta commented  ·   ·  Flag as inappropriate

        I am using Azure Storage BLOB in my project to store images and those images used to display in my webpage. The problem is in every 2-3 days I can see one or two images not displaying in my webpage. Its getting 404 for those images. Using Storage explorer also those blobs are not available in the container. I have also updated the retention days for Storage account to 0 , still its happening. I could not reproduce this issue its happening just suddenly. It would be great if you can give some idea on this.

      • Andrew commented  ·   ·  Flag as inappropriate

        AWS S3 has lifecycle rules that function in a similar way. You can declare path prefixes that are auto expired after a certain period of time. It would make our life a lot easier to have this!

      • John commented  ·   ·  Flag as inappropriate

        For this you can actually try using "Cloudbacko pro" for taking backup. This software not only gives backup but helps in recovery also. I am basically using this software from long time and really satisfied using this software. If you want you can actually use many software that are available in the market but i will only suggest you to take backup and recovery from this software only. Thanks.

      ← Previous 1

      Feedback and Knowledge Base