Change behavior when sandbox runs out of memory (1MB max to one stream limit)
There is currently a maximum limit of 1MB when outputting values to the any stream (Serialized size per object in pipeline).
I understand why this limit is there, but I don't think the current behavior is the right way to go.
Current behavior is that the runbook restarts, and runs 3 times until it is stopped with the following error:
Suspended
The job action 'Activate' cannot be run, because the process stopped unexpectedly. The job action was attempted 3 times.
If we run it on look at the eventlog on a hybrid worker (this error also happens in Azure workers), we have this message:
Terminating sandbox on saving job stream error. [subscriptionId={7b58a17c-0824-40c7-a792-6176c19f65a3}][accountId={79ac99fc-db38-4380-b921-4346f24ced9f}][sandboxId={6a6e53d5-3347-4ffb-9d09-cf0757bf2082}][jobId={ad439f08-9624-43ba-9830-dc9f9f8b7d68}][streamType=Output][exception=System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://tempuri.org/:streamRecord. The InnerException message was 'There was an error deserializing the object of type System.String. The maximum string content length quota (1048576) has been exceeded while reading XML data. This quota may be increased by changing the MaxStringContentLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader.'. Please see InnerException for more details.
The issue here is not the actual limit, but the issue is that
1. The runbook restarts without waning
2. No way to use try/catch to catch the error
3. No way to see if the runbook was restarted. (You could add a value in $PSPrivateMetadata when it happens and we can use that for logic).
I suggest that you change the behavior to output a normal PowerShell error instead of crashing the sandbox.
This would make it possible for us to control it using try/catch.
Or as a second choide, I suggest that you add a "IsRestarted" value in PSPrivateMetaData variable which would make us able to handle the restarts, right now there is no way.
Jakob Gottlieb Svendsen (MVP)
shared this idea
Thanks for posting this feedback! I’m checking with our team to see if there is anything that we can do about this in the short term. We’ll definitely take this into consideration in our longer term plans.
3 comments
-
LAUDWEIN ARNAUD
commented
Agreed, we need a way to control it using try/catch, users wait for an output to whatever process ran the runbook, but never receive anything.
-
Anonymous
commented
Any update on this issue? We are facing the same issue.
-
Jakob Gottlieb Svendsen (MVP)
commented
The issue can be tested using this simple code:
"a" * 1048577