How can we improve Azure Storage?

Support secondary Indexes

Need to be able to sort on something other than the rowkey

1,252 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    I agree to the terms of service
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    andybritcliffeandybritcliffe shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    80 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      Submitting...
      • Anonymous commented  ·   ·  Flag as inappropriate

        This would awesome and bring this service to the premier league. Having the secondary indexes managed by the service makes it incredibly useful for so many different use cases and also for developers with different skills.

      • Frank SzendzielarzFrank Szendzielarz commented  ·   ·  Flag as inappropriate

        Agree with @Anonymous below. Secondary indexes would really help, but in my experience really committing to ATS and thinking hard about the problem usually (again, in my experience) leads to solutions that make sense and do not need the additional indexes. I think the main benefit of the secondary indexes would be making it much more adoptable and requiring much less of a leap of faith.

      • Anonymous commented  ·   ·  Flag as inappropriate

        ATS is still a great solution that works well. It is quite unique compared to what AWS and Google offer.

      • Steffen GammelgårdSteffen Gammelgård commented  ·   ·  Flag as inappropriate

        Hope the secondary index is still coming.

        Azure Table Storage is an immensely powerful platform, and is already an exceptionally cost effective NoSQL database for certain kinds of applications.

        Being able to add a secondary index would drastically reduce the complexity and the amount of redundant data in certain scenarios and would make an already awesome platform relevant for even more projects.

        I too have used index tables to get secondary indicies and sometimes that would still be necessary to allow for efficient partitioning, but having a second index still seems like it would be very useful!

      • LokitothLokitoth commented  ·   ·  Flag as inappropriate

        +1 on Inattention and Neglect.

        And with the evolution of discussion on this, it has become clear that to base critical software on top of Azure Table Storage with any needs beyond simple distributed partially sorted Hashtable is pointless; indeed, I can no longer trust UV as a channel where roadmap about these features is actually accurately communicated.

        No, AIS is not a solution to this. I do not want an index that may, or may not be in an updated state. Heck, I am even willing to build my own index! Just let me grab multiple non-contiguous rows (by RowKey) in one request!

        But more importantly than even that, decide if UserVoice is actually a channel that you want to use to engage with your customers. If it is, announcing at your conference that a feature is coming, then ignoring it for 2 years, then another 3 years before 2 years after that saying it actually is not coming, more than likely, is incredibly insulting.

      • Hans Olav StjernholmHans Olav Stjernholm commented  ·   ·  Flag as inappropriate

        inattention and neglect is the perfect summary of ATS. Microsoft, up the price or do whatever, just do something.

        SSD-backed storage + secondary indexes is a no-brainer.

      • smithkl42smithkl42 commented  ·   ·  Flag as inappropriate

        Appreciate the workaround. But the consensus that ATS has become - through inattention and neglect - an also-ran is hard to shake. Not sure how I could recommend that anyone build a solution on it at this point. At most, I would recommend ATS as a read-only data store, where you put data that you hope to never see again but aren't quite willing to say goodbye to.

      • Hans Olav StjernholmHans Olav Stjernholm commented  ·   ·  Flag as inappropriate

        Aaron, DocumentDB is not a replacement, it is a totally different beast and not a key-value store such as Table Storage. Also DocumentDB is not portable (yet at least) and gives you a lock-in to the public Azure cloud. Table Storage is portable through Azure Stack.

        I dream and hope MS is picking up Table Storage again and gives us Premium (SSD-backed) and with secondary indexes. There's really no reason on this good earth they shouldn't do this!

      • Aaron LawrenceAaron Lawrence commented  ·   ·  Flag as inappropriate

        I would agree with Mike Olsen that it is a BAD IDEA to build a new application on Azure Table Storage, as Microsoft appear to be deprecating it (although it's not completely clear what they regard as a replacement, DocumentDB seems most likely)

      • Hans Olav StjernholmHans Olav Stjernholm commented  ·   ·  Flag as inappropriate

        Regarding table storage, I've also noticed that in the new portal when you look at diagnostics config in WebApps, there's only File Storage and Blob Storage as options, while in the classic portal you also have Table Storage as an option.

        Even more, I've noticed that the internal logging in WebJobs via WebJobs Dashboard seems to have moved from using Table Storage to using Blob Storage - but that may be for any number of reasons, of course.

        Anyways, I wouldn't recommend table storage as a long-term option for anything new for these reasons. And of course it's hard to use without any secondary indexing.

      • Mike OlsonMike Olson commented  ·   ·  Flag as inappropriate

        The writing's on the wall folks. It's been what, 3 years since this was supposedly added to their roadmap? When was the last time Microsoft made *any* real improvements to Table Storage, besides putting a ton of breaking changes in the SDK that required hundreds of hours of dev to support?

        Azure Table Storage is old news and is being ushered unceremoniously out the door. It's time for us to give in and move to DocumentDB, Azure SQL, or one of the dozen new storage services that Microsoft is actually putting effort into. Or just do what I've been doing and just start storing your data in flat files on Blob Storage, since for a lot of cases that's easier to code, more performant and more flexible.

        At least Microsoft isn't slowly moving us away from Cloud Services... oh wait.

      • Aaron LawrenceAaron Lawrence commented  ·   ·  Flag as inappropriate

        It's difficult to use ATS as it stands for anything real. We built our own indices in SQL, which of course introduces exciting new consistency issues. One index is just too little - basically that gives you the ability to move data in and out by an identifier, but not do anything else with it.

      • JasonJason commented  ·   ·  Flag as inappropriate

        We absolutely need this! Due to the high cost and throughput limitations of DocumentDB it is not a viable alternative to having secondary indexes in table storage.

      • smithkl42smithkl42 commented  ·   ·  Flag as inappropriate

        Let me give you an example of why this is critical. I have about 30 GB of event logs. I want to store these in ATS. The problem is that I need to be able to query them along multiple different axes - the company they belong to, the individual user they belong to, the date/time they came in, the event name, and so forth. And eventually, probably lots more. My current solution is to store the data "n" times, each one with a different partition-key/row-key schema, to enable querying along that particular dimension. So far so good - it's not a problem to write the data "n" times, given how well ATS performs, and how cheap it is.

        But the problem is maintenance. Right now, with about 30 GB of data, if I come up with a new dimension that I need to support, it takes me at least a day to write the scripts to export and then re-import the data to the new format (because it all has to be parallelized, and needs to track state for each portion I'm running in parallel, or it would take weeks); and even if I don't ***** up on the (very complex) import/export scripts somehow, it then takes at least 1-2 days to actually get all the data over to the new table.

        And that's simply not a scalable model. What happens when I don't have 30 GB of data I need to pivot, but 300 GB? Or 30 TB? The pivot scripts will take weeks to run, and maintaining any shred of consistency through the process gets very, very complicated.

        I get that this is a complex problem to solve. But it doesn't make it any less complicated by telling every ATS user to come up with their own (almost certainly unoptimal and buggy) solution.

      • Anonymous commented  ·   ·  Flag as inappropriate

        With the DocumentDB and Search services now in play, does Azure still plan to add secondary index support to Table Storage?

      • RyanLMRyanLM commented  ·   ·  Flag as inappropriate

        You can manage this by hand, I do it today and built some reusable code to maintain this. However, changing or adding indexes in the future is a pain, and with the lack of cross table transactions, I have had instances where an index did not get created. Which means if you want/need true integrity, you need to have other process/patterns in place.

        So while, yeah you can do this today, other systems, even Amazon's does this now for you. I would love to not have to deal with this.

      ← Previous 1 3 4

      Feedback and Knowledge Base