when processing queued messages after reconnecting, send messages with LIFO instead of FIFO as the latest data is the most important for rea
when processing queued messages after reconnecting, send messages with LIFO instead of FIFO as the latest data is the most important for real time monitoring. This is similar to the one posted already
Currently IoT Hub and Edge Hub guarantee that messages will be delivered in order. Would folks want this promise to be broken? How would people want to rationalize ordering with other messages.
It sounds like the primary scenario here is making sure that the highest priority messages get delivered first. For those looking for that capability, check out priority queues.
A configurable option would be great with FIFO as default. The extra mile would be the ability for the sender to overwrite the default TTL value so that important messages are prioritised as already mentioned here https://feedback.azure.com/forums/907045-azure-iot-edge/suggestions/36632884-low-bandwidth-partially-connected-message-deliv. It probably adds significant overhead to queue processing so perhaps just being able to "subscribe" to the connected status or queue state to implement some "back pressure" logic would be enough.
Lars Kemmann commented
The default needs to be FIFO, but I'd be fine with an explicit opt-in to a LIFO pattern on a per-route basis.
Fabien Grenier commented
I vote for a configurable option.
In our case, we embed a timestamp linked to the measure so order is not mandatory and we could prefer LIFO.
Christian Wuerdig commented
Please don't change it. If there is a real use-case then a configurable option would be nice but it should default to FIFO - it would undoubtedly break countless systems if this were to change
Frederik Gheysels commented
I don't think it is a good idea to change from FIFO to LIFO. As your sending data to IoTHub, IoT Hub has a stream of events which are garantueed to be in order.
Edwin Howard commented
I would like to have 2 queues for messages: one FIFO, and one keyed, replaceable item FIFO. The replaceable item FIFO would hold messages that I deemed dynamic and expendable, and are identified by a message key. In the replaceable FIFO, if the queue hasn't been serviced, and a message with the same key as an existing queued message arrives, then the most recent message should replace the old message in the queue. That way, if there is a communication interruption, the most recent of the dynamic, expendable messages are transmitted first when communications are restored.
For using IoT for remote services would mean that we need to have real-time messaging. We wouldn't want FIFO, and LIFO would not work either. What we have now is a replaceable queue of messages for real-time telemetry (e.g. vessel positioning) These messages are updates >= 1 Hz, so a VSAT outage of several seconds means we have old data. Surveyors on the beach need the most up-to-date position as soon as the VSAT re-establishes connection. Since we already handle this replaceable queue in software, we could immediately make use of a direct IoT connection that does no queuing at all.
For such a direct connection, we're also concerned about the amount of bandwidth used over a limited VSAT connection (typically 128kbps). We currently run ping/pong messages across the connection to actively monitor throughput and insure that we don't send more data than possible at any given time during a connection. We do this so our WebSocket connection doesn't have problems with retries due to data becoming backed up on the TCP connection. We've looked into reliable UDP / multicast, but these aren't supported with current managed IoT solutions. VSAT optimization sometimes acks packets before going over the VSAT link, so running an ICMP echo ping would give a false impression of available bandwidth. Hope this helps.
For our use to monitor a remote asset in real-time we always want to know the latest state of it as soon as possible. So when an asset is disconnected for a period (e.g. 24 hours) we want to know the current status as soon as it has reconnected. We would not want to know what it was yesterday and wait for everything to catch up until we got the latest status again.
A setting similar to the Edge blob storage sync would be good, that way we know we are signing up for our messages to be delivered out of order.