How can we improve Azure Storage?

Provide Time to live feature for Blobs

If I need to provide a user (or external system) some data (blob) which might be outcome of some processing (or other) and it has some expiration time I'd like to just put a new blob and set TTL property with TimeSpan (or set absolute DateTime). When the period is over my blob is deleted. So I don't have to pay for it and don't need to spin up some service for doing it myself.

1,205 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    AlexLF shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    Thank you for your feedback. We are currently in public preview of blob storage lifecycle management. The feature set offers a rich, rule-based policy engine which you can use to transition your data to the best access tier and to expire data at the end of its lifecycle. See our post on the Azure Blog to get started: https://azure.microsoft.com/en-us/blog/azure-blob-storage-lifecycle-management-public-preview/preview/.

    For any further questions, or to discuss your specific scenario, send us an email at DLMFeedback@microsoft.com.

    34 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      Submitting...
      • Christopher Warrington commented  ·   ·  Flag as inappropriate

        The storage lifecycle management preview is fine, but doesn't address the issue I have. I want to be able to create a blob and set it to expire in, say, 3 hours. When 3 hours have elapsed, the blob should be deleted (perhaps subject to the container's soft delete policy). If I create a different blob, I want to set the expiration to say 15 minutes. I also want to be able to extend and shrink a blob's TTL after it has been created. The once-a-day, policy based approach doesn't let me do this any of this.

      • Jasper Siegmund commented  ·   ·  Flag as inappropriate

        "We will continue to provide updates at least once per quarter."

        We would like one of those updates, please :)

      • David Gard commented  ·   ·  Flag as inappropriate

        I find it worrying that the only way to delete 'expired' objects in a blob store currently is to iterate through them all within a Logic App, checking each against an arbitrary date to determine if it should be deleted.

        I've tried the "Delete Old Blobs" Logic App template, and somehow a single run against a blob store with ~1100 objects generated just short of 4500 billable events, and that's without actually deleting any of the blobs. For blob stores with a large number of objects, this using this Logic App template would likely get very expensive very quickly.

        One would hope that a proper mechanism for setting a TTL on an object in a blob store will arrive soon, particularly given the arrival of GDPR in the EU.

      • Aniruddha Diwakar commented  ·   ·  Flag as inappropriate

        Sorry for immediate second post. I forgot to mention that this will actually also improve the security as we are making sure that the blob is not available when it is already used by my consumer. Offcourse this setting should be something like On/Off per blob/container so that i can choose to use or not use this feature.

        Basically this is more like a "one time use and throw" kind of a feature

      • Aniruddha Diwakar commented  ·   ·  Flag as inappropriate

        Is there a way similar logic can be applied when blob is accessed for the first time. Basically as owner of the data I want to have my consumer access the data for next say X hours. But the moment they access the data and download of blob is successful for the first time, I dont want to wait for remaining hours to expire/delete the blob. I want to delete the blob immediately after the download is completed.
        Any ideas on how can this be done?? Or is there a plan of having this feature in Azure itself so that I dont have to worry about having and maintaining another system, component.

      • Ramon de Klein commented  ·   ·  Flag as inappropriate

        We implemented this on our BLOB storage and in two days the logic app resulted in a bill of $233 (logic app only). I cannot even consider this logic app a solution. It's far more efficient to create a Powershell/C# program that lists the blobs and removes the old ones instead.

        The only proper solution is to provide a TTL like AWS S3 does, so the deletion actually transparent.

      • Ramon de Klein commented  ·   ·  Flag as inappropriate

        I have implemented a logic app (see https://sameeraman.wordpress.com/2017/08/25/logic-apps-delete-files-older-than-x-days-from-a-blob-storage), but I don't like this solutions for a variety of reasons:
        1. I need to maintain a logic app.
        2. I need to pay for the logic app when it triggers.
        3. It doesn't seem to scale well. Listing all blobs to an array is madness with a container with >1M of objects.
        4. I get "Rate limit is exceeded. Try again in 4 seconds." when deleting the blobs.

        The AWS S3 lifecycle method (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) is much more convenient and user-friendly.

        It was confirmed to be implemented in 2017, but the only thing we got was a louse logical app, that doesn't work well. Please come up with something better.

      • Oliver Tomlinson commented  ·   ·  Flag as inappropriate

        Just received this MS after chasing :

        "We actually will have something published shortly that will enable this with Logic Apps (some final testing underway on this), while the full API/policy based version of it will be shipped in CY18 per the current plan."

      • Oliver Tomlinson commented  ·   ·  Flag as inappropriate

        Please allow us to define a TTL/expiry on a container that applies to all blobs in the container, not just individual blobs! Thank you.

        P.s. where is the update from the Azure Storage Team on this?

        Your last response has been over 7 months, and not the "at least once per quarter" as promised.

        Why doesn't the Azure Storage Team expose a public road map like other Azure Teams?!?

      • Guy commented  ·   ·  Flag as inappropriate

        Please add support to be able to maintain the blob store capacity
        we need an eviction policy to delete lets say x days of unused blobs
        also, would be great if this can be integrated with the alerts system so when an alert for say a capacity is breached this eviction policy also triggers with a param for days set by the alert.

        as for now there is a powershell script and not an integrated solution, and the script literally itrates over all the blobs in the storage

      • Darian Miller commented  ·   ·  Flag as inappropriate

        Please add the ability to set the original 'creation date' so that when data is initially migrated to these containers, it can expire as expected. (Say you import 3 years of transactions and the data expires in 5 years...you don't want to have some of this data on file for a total of 8 years just because the import date becomes the creation date.)

      • Shahiddev commented  ·   ·  Flag as inappropriate

        I think with the introduction of the cool storage, the native support for TTL on blobs is even more useful/important.
        Using Azure functions we can definitely achieve this reasonably easily however nothing beats native support

      ← Previous 1

      Feedback and Knowledge Base