Do you have a comment or suggestion to improve SQL Server? We’d love to hear it!

UTF-8 in SQL 2019: An error occurred while executing batch. Error message is: Internal connection fatal error. Error state: 15, Token : 0

In SQL Server 2019 CTP 2, the query below gets a fatal error when executing in a database that has a "_UTF8" default Collation. The fatal error is:

"An error occurred while executing batch. Error message is: Internal connection fatal error. Error state: 15, Token : 0"

and the connection is terminated.

Better yet, if you go to the "Results" tab and scroll down to the bottom, it should completely crash SSMS (I tested in 17.9 ; have not yet tried 18.0). Of course, this is when using "TOP (142)" or greater. If using a value of 128 - 141 (roughly) for the TOP, then it often spins forever, cancelling the query doesn't work, and you need to close the query tab. AND, if using a value of 127 or below for the TOP, then it works (you can even scroll down without crashing SSMS)!

I have no idea what about this query causes this since if ANY part is removed, then it works just fine. Just comment out any of the columns / expressions in the SELECT list and it will work no matter what the TOP value is.

Complete wackiness.

-----------------------------------------------------------------------
IF (DB_ID(N'UTF8') IS NULL)
BEGIN
CREATE DATABASE [UTF8] COLLATE Latin1_General_100_CI_AS_SC_UTF8;
ALTER DATABASE [UTF8] SET RECOVERY SIMPLE;
END;
GO

USE [UTF8];

;WITH nums AS
(
SELECT TOP (142) (ROW_NUMBER() OVER (ORDER BY (SELECT 0)) - 1) AS [CodePoint]
FROM [master].[sys].[columns] col
CROSS JOIN [master].[sys].[objects] obj
), chars AS
(
SELECT nums.[CodePoint],
CONVERT(VARCHAR(4), NCHAR(nums.[CodePoint]) COLLATE Latin1_General_100_CI_AS_SC_UTF8) AS [TheChar],
CONVERT(VARBINARY(4), CONVERT(VARCHAR(4), NCHAR(nums.[CodePoint]) COLLATE Latin1_General_100_CI_AS_SC_UTF8)) AS [UTF8]
FROM nums
)
SELECT chr.[CodePoint],
chr.[TheChar] AS [Character],
CHAR(chr.[CodePoint]) AS [CHAR],
NCHAR(chr.[CodePoint]) AS [NCHAR],
chr.[UTF8] AS [UTF8hex],
(chr.[UTF8]) AS [UTF8bytes],
CONVERT(VARBINARY(4), CONVERT(NVARCHAR(3), chr.[TheChar])) AS [UTF16(LE)_Hex],
DATALENGTH(CONVERT(NVARCHAR(3), chr.[TheChar])) AS [UTF16_Bytes],
((DATALENGTH(CONVERT(NVARCHAR(3), chr.[TheChar]))) - (DATALENGTH(chr.[TheChar]))) AS [UTF8savingsOverUTF16]
FROM chars chr

-- TOP 128 (or less) works
-- 129 - 141 spins forever (can sometimes cancel; usually must close query tab)
-- 142 (or greater) breaks
-----------------------------------------------------------------------

P.S. Please see "Native UTF-8 Support in SQL Server 2019: Savior or False Prophet?" ( https://sqlquantumleap.com/2018/09/28/native-utf-8-support-in-sql-server-2019-savior-false-prophet-or-both/ ) for a detailed analysis of the new UTF-8 feature.

1 vote
Sign in
(thinking…)
Sign in with: Microsoft
Signed in as (Sign out)

We’ll send you updates on this idea

Solomon Rutzky shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

8 comments

Sign in
(thinking…)
Sign in with: Microsoft
Signed in as (Sign out)
Submitting...
  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    I have re-run all of these tests in CTP 2.1: SSMS queries in both 17.9 and 18.0 Preview 4; all 3 SQLCMD versions, and the SQLCLR test. None of the tests failed this time.

    As far as I can tell, the problem appears to be fixed. This item can be closed. Thanks!

  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    I just tested this (the query from my comment directly below this one) in SSMS v18.0 Preview 4 (15.0.18040.0) and received the same error:

    An error occurred while executing batch. Error message is: Internal connection fatal error. Error state: 15, Token : 0

  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    One last thing: I just found that the default Collation of the database does _not_ need to be a "_UTF8" Collation for this error to occur. I thought I tried this scenario before and didn't get the error, but I tried again and most definitely do now, in databases using various non-"_UTF8" Collations:

    USE [master];

    SELECT TOP (100)
    CONVERT(BIT, 1) AS [1],
    CONVERT(int, 1) AS [2], -- changed to "int"
    CONVERT(BIT, 1) AS [3],
    CONVERT(int, 1) AS [4], -- changed to "int"
    CONVERT(CHAR(10), NULL) COLLATE Latin1_General_100_CI_AS_SC_UTF8 AS [NULL], -- added COLLATE
    CONVERT(BIT, 1) AS [5],
    NCHAR(50505) + NCHAR(20202) + N'gf' + NCHAR(10101) + NCHAR(66666) AS [6],
    'abcdefghijklmnoptuvwxyz' AS [7],
    CONVERT(BIGINT, 1) AS [8]
    FROM [master].[sys].[columns] col;
    -- Returns 13 rows of the following (though last row is different in 2 columns):
    -- 1 1 1 1 앉仪gf❵𐑪扡摣晥桧橩汫湭灯畴睶祸ࡺ NULL NULL

  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    I have posted the SQLCLR tests to PasteBin:

    * UTF8_Bug_35606671_SQLCLR-Tests ( https://pastebin.com/qjTGsSET )

    Unlike with testing in SSMS and SQLCMD, or even testing with SqlBulkCopy in a Windows app or console app, testing with SqlBulkCopy in SQLCLR allows for changing the driver / type of connection without changing the client app in any way (such as needing to use a different version of the client program). The result is we can see that the problem is with the connection / driver and not with SqlBulkCopy itself (given that it executes just fine with using the in-process Context Connection, yet fails when using a regular / external connection.

  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    I think I have this narrowed down to being a driver problem. The newest SQLCMD, which uses the newest ODBC driver, is fine. But older versions of SQLCMD, using either slightly older ODBC or even older SQL Native Client, have issues. I even reproduced the problem in .NET using SqlBulkCopy.

    The issue seems to be a problem reading the TDS results. The problem does seem to require a NULL UTF8 value, but that does not always trigger the problem. The row size of the results, as well as the number of fields in the result set, as well as the size of the data / datatypes, and even the position of the field containing the NULL, all impact whether or not you get an error, which error you get (I have now seen 3 of them), and how many rows you get back (in one test I got 25 out of 100 rows back, yet it reported "Success").

    Errors:
    1) An error occurred while executing batch. Error message is: Internal connection fatal error. Error state: 15, Token : 4

    2) An error occurred while executing batch. Error message is: Invalid array size.

    3) An error occurred while executing batch. Error message is: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.

    I would attach the scripts containing the repro queries / SQLCMD command-lines, but this highly inadequate UserVoice system, which is barely any better than the MS Connect system it replaced (seems more so equally, but differently, deficient), only lets you add a file when creating the issue, not when adding a comment, and won't let you update the issue to make corrections, or add content or files. So, I have placed them on PasteBin:

    * UTF8_Bug_35606671_SQLCMD-Tests ( https://pastebin.com/z9kWRxrG )

    * UTF8_Bug_35606671_SSMS-Tests ( https://pastebin.com/adSnZztD )

    I will add the SQLCLR script later.

    For now, here is a fun repro:
    ------------------------------------------------------------------------------
    USE [UTF8];

    SELECT TOP (100)
    CONVERT(BIT, 1) AS [1],
    CONVERT(int, 1) AS [2], -- changed to "int"
    CONVERT(BIT, 1) AS [3],
    CONVERT(BIT, 1) AS [4],
    CONVERT(CHAR(1), NULL) AS [NULL],
    CONVERT(BIT, 1) AS [5],
    NCHAR(50505) + NCHAR(20202) + N'gf' + NCHAR(10101) + NCHAR(66666) AS [6],
    'abcdefghijklmnoptuvwxyz' AS [7],
    CONVERT(BIGINT, 1) AS [8]
    FROM [master].[sys].[columns] col;
    -- 1 1 1 1 앉仪gf❵𐑪.abcdefghijklmnoptuvwxyzĈ 1 ก䤀李昀甀ħ櫘⻜愀戀挀搀攀昀最栀椀樀欀氀洀渀漀瀀琀甀瘀眀砀礀稀ࠀ āā앉仪gf❵𐑪.abcdefghijklmnoptuvwxyzĈ NULL
    ------------------------------------------------------------------------------

  • Solomon Rutzky commented  ·   ·  Flag as inappropriate

    P.S. Forgot to mention: when scrolling down in the "Results" tab after this error occurs, it always crashes as it attempts to display row # 128.

Feedback and Knowledge Base