Have you ever been in a situation where SQL Server shuts down by itself? It sounds strange but I met across a gentleman in flight who reported this issue to me. Since I was curious to know the cause, we exchanged our business cards. Once I came back home, I sent him an email asking more details about the SQL shutdown issue. Let us learn about how to fix error, An Error Occurred While Writing an Audit Trace.

I asked him to share ERRORLOG from the SQL Server instance and I could find below information just before the shutdown.

spid397 Error: 566, Severity: 21, State: 1.
An error occurred while writing an audit trace. SQL Server is shutting down. Check and correct error conditions such as insufficient disk space, and then restart SQL Server. If the problem persists, disable auditing by starting the server at the command prompt with the “-f” switch, and using SP_CONFIGURE.

The error is kind of self-explanatory. SQL was shut down because it was not able to write to trace file, but the real question is why?

One of the possible way this error would come up would be if we were doing C2 type auditing.

But in the above case we can see that it’s set to zero. Then I have asked for more details about various traces which were running and he provided me below script.

exec @rc = sp_trace_create @TraceID output, 6, N’\\IndiaFileServer\T$\MSSQL_Trace\ProdServers\FooBar_Trace’, @maxfilesize, NULL

As per books online, the 6 above is a combination of options 2 & 4.  Option 2 makes file rollover after they reach @maxfilesize.  Option 4 specifies that if SQL ever can’t write to the trace file, it will shut down the SQL Server.  This is by design for a security auditing feature.

To find if shutdown option is enabled or not, we can look at running traces using catalog view called as sys.traces.

Since we know that trace is configured to shutdown SQL server in case of failure to write. He agreed to change option to use 2 instead of 6.

At the end, I was happy that we could find cause and solve a mystery.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 566, Severity: 21 – An Error Occurred While Writing an Audit Trace. SQL Server is Shutting Down

I was trying to set up database mirroring in my freshly created Azure Virtual Machines and encountered an error. The error was talking about the connectivity problem, but it was something else. Let us learn about how to fix the database mirroring connection error 4.

Environment

• SRV_P -Principal- Microsoft SQL Server 2014 – 12.0.4422.0 (X64)
• SRV_M-Mirror- Microsoft SQL Server 2014 – 12.0.4422.0 (X64)
• SRV_W-witness- SQL SERVER 2008 R2

SYMPTOMS

As soon as I configure database mirroring and click on “Start Mirroring”, I was getting below error.

Alter failed for Database ‘SQLAUTHORITY’. (Microsoft.SqlServer.Smo)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
The ALTER DATABASE command could not be sent to the remote server instance ‘TCP://SRV_W.SQLAUTHORITY.net:5022’. The database mirroring configuration was not changed. Verify that the server is connected, and try again. (Microsoft SQL Server, Error: 1456)

I looked around in ERRORLOG files to see if there is anything interesting.

PRINCIPAL SERVER

2016-12-26 01:23:16.710 spid43s Database mirroring connection error 4 ‘An error occurred while receiving data: ’24(The program issued a command but the command length is incorrect.)’.’ for ‘TCP://SRV_W.sqlauthority.net:5022’.
2016-12-26 01:23:36.490 spid75 Error: 1456, Severity: 16, State: 3.
2016-12-26 01:23:36.490 spid75 The ALTER DATABASE command could not be sent to the remote server instance ‘TCP://SRV_W.sqlauthority.net:5022’. The database mirroring configuration was not changed. Verify that the server is connected, and try again.
2016-12-26 01:23:36.750 spid41s Database mirroring is active with database ‘SQLAUTH’ as the principal copy. This is an informational message only. No user action is required.

MIRROR

2016-01-01 22:28:14.370 Logon Database Mirroring login attempt by user ‘SQLAUTHORITY\SRV_P$.’ failed with error: ‘Connection handshake failed. The login ‘SQLAUTHORITY\SRV_P$’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: 10.0.40.34]

WORKAROUND/SOLUTION

I realized that this is a typical issue where service account is set to LocalSystem. Due to this mirroring, would use the machine name (having $ symbols at the end) to communicate. There are two workarounds for this.

1. Change the Service account to domain account
   Read more here: SQL SERVER – Best Practices About SQL Server Service Account and Password Management

1. If you don’t want to use domain account, then we need to create machine account as login and provide CONNECT permission to endpoint (as mentioned in error message)

On principal

use [master]
GO
CREATE LOGIN [SQLAUTHORITY\SRV_M$] FROM WINDOWS
GO
GRANT CONNECT ON ENDPOINT::[Mirroring] TO [SQLAUTHORITY\SRV_M$]
GO
use [master]
GO
CREATE LOGIN [SQLAUTHORITY\SRV_W$] FROM WINDOWS
GO
GRANT CONNECT ON ENDPOINT::[Mirroring] TO [SQLAUTHORITY\SRV_W$]

Notice that we are giving permission to machine account of mirror and witness. Same set of command to be run on Mirror and Witness as well with other two machines account.

On Mirror = SQLAUTHORITY\SRV_W$ and SQLAUTHORITY\SRV_P$

On Witness = SQLAUTHORITY\SRV_P$ and SQLAUTHORITY\SRV_M$

Same issue can happen in AlwaysOn availability group as well if account is set to LocalSystem.

Have you encountered such errors? What was the solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Mirroring Connection Error 4 – An Error Occurred While Receiving Data: 10054

Question: How to Enable Auto Update Statistics and Auto Create Statistics?

Answer: I have received this question after watching my free webinar on group by conference by one of the user. I really loved his question so I am re-posting the question over here.

“Pinal,

I love your energy in while you presented the session. I think I tried one of your suggestions of enabling auto update statistics and auto create statistics on our primary database. Trust me, after making the changes my database is running extremely fast and my customers are congratulating me. Now the real challenge is that we have over 100s of database and I need to enable these settings where it is off. If I go and do it manually everywhere it will take forever, is there any shortcut for it?

~ Nitin S”

Well, Nitin is actually referring my webinar over here. In this webinar, I have shared three very important tips for improving your SQL Server’s performance. During SQL Server Performance Tuning Practical Workshop, I also discuss about this in detail.

Here is the script which will generate T-SQL Script which you can use to enable Auto Update Statistics and Auto Create Statistics for all the database where it is set to off.

</p>
SELECT'ALTER DATABASE ' +name
+' SET AUTO_UPDATE_STATISTICS ON WITH NO_WAIT'
FROM sys.databases
WHERE is_auto_update_stats_on = 0;

SELECT
'ALTER DATABASE ' +name
+' SET AUTO_CREATE_STATISTICS ON WITH NO_WAIT'
FROM sys.databases
WHERE is_auto_create_stats_on = 0;

Once you run above script, it will generate output as displayed in the following screen. Run the script in SSMS and you will enable required settings for your database.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on How to Enable Auto Update Statistics and Auto Create Statistics with T-SQL – Interview Question of the Week #108

I am used to exploring what is available in the technology of love – SQL Server. As I was exploring into one of the simplest set of DMVs which have been available for ages. There are several ways to get the OS information inside SQL Server. One of the ways is to use the DMV as shown below.

SELECT * FROM sys.dm_os_windows_info

The other way to know the OS related information about SQL Server is to scan your Error Logs. During startup, SQL Server logs all this information which is a wealth of information. A sample output is shown below:

Microsoft SQL Server 2016 (SP1) (KB3182545) – 13.0.4001.0 (X64)
Oct 28 2016 18:17:30
Copyright (c) Microsoft Corporation
Developer Edition (64-bit) on Windows 10 Enterprise 6.3 <X64>

When I started to write about this blog is when I realized will it also work similar on SQL vNext. To my surprise, there has been a new DMV that is getting added for Linux in mind. The “windows” term is getting removed and it made a lot of sense.

SELECT * FROM sys.dm_os_host_info

If you installed the vNext version then this DMV will work. On a side note, I am interested in knowing how many of you have installed the CTP of SQL Server on Linux? What has been your experience? Let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Server: Getting OS information using DMV

A lot’s been said about last week’s Gitlab outage so I’m not going to go over the details here but I do want to talk about one point that was when I was watching the guys fix the issue live on youtube.

A lot of the people making comments kept asking if the person who made the original mistake was going to be fired (and I mean a lot of people were asking). Finally one of the guys on the channel responded by saying that this wasn’t a personal mistake, it was a process failure and if the person who had made the original error hadn’t taken a backup before he started this work, they would not have been able to recover anything at all.

I did wonder about who was asking the question though and I came to the conclusion that it couldn’t have been anyone who’s worked in the tech industry for any sustained period time.

Show me a senior technical person who has never made a mistake causing an outage and I’ll show you a fibber

People do not get fired for making one mistake, could you imagine? Everyone that I know in tech has made at least one error causing an outage, others (like me) have made more than one. OK, yes, if this was the latest in a line of mistakes that that person had made at GitLab then maybe, but I doubt that it was.

It comes down to the age old adage, learn from your mistakes. Bet that guy at Gitlab won’t make that mistake again

Have a good (mistake free) week.

There was a question from the op regarding adding a new column to a query output by generating the cyclic sequence numbers from 1 to 3.

Select A =identity(int,1,1),B,C from table_abs

1,A,41

2,B,13

3,C,90

4,D,91

5,E,98

6,F,12

7,G,54

8,H,16

For this output, the 4th column generates the Sequence of numbers from 1 to 3 which is shown below

1,A,41,1

2,B,13,2

3,C,90,3

4,D,91,1

5,E,98,2

6,F,12,3

7,G,54,1

8,H,16 ,2

If you are using SQL 2012 then Sequence would be natural choice for any such operations.

Solution 1

Using Sequence

CREATE SEQUENCE Seq
 AS INTEGER
 START WITH1
 INCREMENT BY1
 MINVALUE 1
 MAXVALUE 3
 CYCLE;SELECT table_name,NEXT VALUE FOR Seq New_column
FROM information_schema.tables

Solution 2

Using CTE and Modulus operator

;with q as(select row_number()over(orderby(selectnull)) A,*from sys.objects
)select A,1+A%3 B,*from q

Solution 3

Loops and Temp table

createtable dummyTest
(
id int, 
col1 char(1),
col2 int
)insertinto dummyTest values(1,'A',410),(2,'B',411),(3,'c',4111),(4,'d',421),(5,'e',441),(6,'f',451),(7,'g',481),(8,'h',401)createtable#dummy(
id int, 
col1 char(1),
col2 int,
NewColumn int
)declare@n int,@i int,@limit int
 set@limit=1set@i=1select@n=count(*)from dummyTest
 while@i<=@n
 beginset@limit=1while@limit<=3beginprint@limit
 insertinto#dummyselect*,NewColumn=@limit from dummyTest where id=@i
 set@i=@i+1set@limit=@limit+1endendselect*from#dummy

Conclusion

The same solution can be derived using the cursor and there may be other solutions as well. At many instances, we opt for any solutions without thinking of data volume that may degrade the performance. This is one of the prime examples of why we need to upgrade to newer version.

If you are coming to this page by searching, then I am sure that you are in big trouble. I have come across this error and my SQL Server was not getting started. When I looked into ERRORLOG, I found below errors about the PageAudit Property.

2016-12-30 15:45:36.36 spid9s Starting up database ‘model’.
2016-12-30 15:45:36.37 spid9s Error: 5172, Severity: 16, State: 15.
2016-12-30 15:45:36.37 spid9s The header for file ‘C:\Program Files\Microsoft SQL Server\MSSQL12.DAR_P11D\MSSQL\DATA\model.mdf’ is not a valid database file header. The PageAudit property is incorrect.
2016-12-30 15:45:36.39 spid9s Error: 945, Severity: 14, State: 2.
2016-12-30 15:45:36.39 spid9s Database ‘model’ cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details.
2016-12-30 15:45:36.39 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

So, since the model database was having serious corruption, SQL was not getting started. The error message essentially means that file header has been corrupted and SQL Server is not able to understand it.

WORKAROUND/SOLUTION

In my case since it was model database, there were two possible options.

1. Get MDF and LDF file from some other server which has “exactly” same version of SQL Server. This would cause to lose data from the previous model database and will get it from another server.
2. If we have a backup of the model and we wish to restore it first, then we need to rebuild system databases using the setup.exe command line. Before rebuilding, we need to keep a copy of MSDB and MASTER database files. As soon as we rebuild, all system database would be as fresh as new installations. After rebuilding, we can restore backups and bring SQL back to life.

After I followed option 1, SQL Server came back online, but SQL Agent was giving trouble. I looked into SQLAgent.out file and here was the information.

2016-12-30 17:04:25 – ? [100] Microsoft SQLServerAgent version 12.0.2269.0 (X64 unicode retail build) : Process ID 83880
2016-12-30 17:04:25 – ? [495] The SQL Server Agent startup service account is NT Service\SQLAgent$SAPSQL.
2016-12-30 17:04:25 – ? [393] Waiting for SQL Server to recover database ‘msdb’…
2016-12-30 17:04:25 – ? [101] SQL Server version 12.00.2269 (0 connection limit)
2016-12-30 17:04:25 – ? [102] SQL Server ODBC driver version 11.00.2100
2016-12-30 17:04:25 – ? [103] NetLib being used by driver is DBNETLIB; Local host server is CLD-SQL01\MIMECAST_MSO
2016-12-30 17:04:25 – ? [310] 8 processor(s) and 17970 MB RAM detected
2016-12-30 17:04:25 – ? [339] Local computer is CLD-SQL01 running Windows NT 6.2 (9200)
2016-12-30 17:04:25 – ! [000] SQLServerAgent cannot start because the msdb database is not available for normal access
2016-12-30 17:04:25 – ? [000] Configuration option ‘show advanced options’ changed from 0 to 1. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457) Configuration option ‘Agent XPs’ changed from 1 to 0. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457) Configuration option ‘show advanced options’ changed from 1 to 0. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457)
2016-12-30 17:04:25 – ? [098] SQLServerAgent terminated (normally)

If you read through above lines, it is easy to identify that below is the cause:

SQLServerAgent cannot start because the msdb database is not available for normal access

When I connected to SQL Server, I found that MSDB database was in suspect state. So, I need to restore that from the backup.

If you ever face error mentioned in a blog title, restore from backup is the only option left. If you find some other solution, please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – The header for file Path is not a valid database file header. The PageAudit property is incorrect

When you see something not behaving the way normal way, either it must be magic or in software terms it is an error or bug. This blog was inspired by someone who caught me unaware with this challenge that it took me by surprise. I didn’t know how to solve it and it took me close to 10 mins before realizing what was happening behind the scenes. Let us see the puzzle about Why does sp_spaceused Show No Values?

I have not done any sort of trick in the below screen shot. It is real table and the output is real too.

Did you see something strange here? Even though the table has values, the sp_spaceused is not recording any values. When I first saw this, it surely looked like some bug.

Now if I had to ask for your help to help me, what would be your response? Why is the output showing like this?

Have you ever encountered such strange behavior with SQL Server? What would you do when such things happen? If this is a bug, I highly recommend you to please go to the Connect site and do a search. On a side note, do let me know if you ever visited this site. Do read about enhancement requests on this site, this is the best way to know what doesn’t work and what are people working on.

Hint: Coming back to the puzzle above. Do you know the various types of tables you can create? I think I am giving away the answer already.

Here is the code to build this example.

USE MASTER
GO
DROP DATABASE IF EXISTS [InMem_OLTP]
GO
-- Create the database
CREATE DATABASE [InMem_OLTP] ON PRIMARY
( NAME = N'InMem_OLTP_data', FILENAME = N'C:\DATA\InMem_OLTP_data.mdf',size = 30MB)
LOG ON
( NAME = N'InMem_OLTP_log', FILENAME = N'C:\DATA\InMem_OLTP_log.ldf',size = 10MB)
GO
-- Add Inmemory Filegroup
USE InMem_OLTP
GO
ALTER DATABASE InMem_OLTP
ADD Filegroup [InMem_OLTP_InMemory] CONTAINS MEMORY_OPTIMIZED_DATA
GO
ALTER DATABASE InMem_OLTP
ADD FILE (NAME = InMem_OLTP_InMemory, FILENAME = N'C:\Data\InMem_OLTP_mopt')
TO FILEGROUP [InMem_OLTP_InMemory]
-- Let us create the table srtuctures
USE InMem_OLTP
GO
CREATE TABLE Products
(
ProductID INT CONSTRAINT pk_products_pid PRIMARY KEY NONCLUSTERED HASH (ProductID) WITH (BUCKET_COUNT = 10000),
ProductName VARCHAR(25)
) WITH ( MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
GO

INSERT INTO Products VALUES (2, 'SQLAuthority');
GO
SELECT * FROM Products
GO
sp_spaceused Products
GO
USE master
GO
DROP DATABASE [InMem_OLTP]
GO

Do drop your guesses and understanding via comments below.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Puzzle – Why does sp_spaceused Show No Values?

There can sometimes be confusion, in IoT scenarios, between IoT Hub and Event Hub, as they can both be valid choices for streaming data ingestion.  My Microsoft colleague, Steve Busby, an IoT Technology Specialist, is here to clear up the confusion:

The majority of the time, if the data is coming directly from the devices, either directly or via a field-based gateway, IoT Hub will be the more appropriate choice.  Event Hub will generally be the more appropriate choice if either the data will not be coming to Azure directly from the devices, but rather either cloud-to-cloud through another provider, intra-cloud, or if the data is already landing on-premise and needs to be streamed to the cloud from a small number of endpoints internally.  There are exceptions to both conditions, of course.

Both solutions offer very high throughput data ingestion and can handle tremendous streaming data volumes.  In fact, today, IoT Hub is primarily a set of additional services that wrap an underlying Event Hub.

Event Hub, however, has some limitations that may make it inappropriate for an IoT scenario when data is being pulled from devices in the field, specifically:

• Event Hub is limited to 5000 concurrent AMQP connections.  At “IoT scale”, there would likely be a lot more devices than that
• Event Hub does not authenticate individual device connections, but rather all the devices essentially share the same (or few) access keys.  You also cannot enable/disable communication from a single device.
• Event Hub is ingestion only and has no facility for sending traffic back to the devices (command and control or device management scenarios)
• Event Hub does not support MQTT.  For better or worse, MQTT is a very popular IoT protocol and is supported by competitors, particularly AWS

In addition to the features of Event Hub, IoT Hub also adds:

• Scale out connections.  A single IoT Hub has been tested to more than a million concurrent connections
• IoT Hub does individual device authentication (via either a device-specific key, or x.509 certificate) and you can enable/disable individual devices
• Bi-directional communication, allowing you to either do asynchronous or synchronous commands to the device
• Supports AMQP and HTTP, like Event Hub, but also adds support for MQTT as a transport protocol
• Device Management – the ability to orchestrate firmware updates and desired and reported configuration management with devices
• Device Twins – A queryable cloud-side representation of the state of a device, allowing customers to group and manage their devices “at IoT scale”
• File update – orchestrate file uploads from device through the hub

As you can see, both Event Hub and IoT Hub are valuable Azure services with distinct use cases.  Event Hub is great for high throughput ingestion (only) of data from a limited number of sources (like cloud to cloud, on-prem to cloud, or intra-cloud).  In those scenarios, Event Hub will generally be appropriate and significantly cheaper than IoT Hub.  However, for IoT scenarios where interacting with the individual devices directly are involved, IoT Hub will generally be the better technology and adds a lot of features we need to be competitive.

A few weeks ago Grant Fritchey and I had the chance to speak at five user group meetings in five days, in five cities all over Florida.

Clearly we’re both insane as we agreed to do this. Everything was scheduled by Karla Landrum and she got some big shout outs at the User Group meetings.

But one person that didn’t get enough credit was Rodney. He was kind enough to take a week off of work and drive up to Nashville SQL Saturday to pick me up, then drove Grant and I all over Florida to all these user group meetings. Driving for 6-8 hours a day while Grant and I worked away on our laptops couldn’t have been an easy task; and I just wanted to through out a shout out to Rodney for being willing to put up with me, Grant and Karla for 5 days of driving around the state.

So thanks Rodney.

Denny

The post Thanks Rodney for driving me and Grant all over Florida appeared first on SQL Server with Mr. Denny.

I was trying to help my client in generating a report large data set. After spending some time and understanding the schema, I could provide them the query to get the results. Now, he wanted to save the results in excel sheet. So, he ran the query in SQL Server Management Studio (SSMS), got a lot of rows as output and he hit Ctrl+C in the grid. Let us learn about System.OutOfMemoryException error.

Here is the text of the message (copied using the Copy icon at the bottom left of the message windows)

Exception of type ‘System.OutOfMemoryException’ was thrown. (mscorlib)

If we click on the technical detail icon, we can see below.

Program Location:
at System.Number.FormatInt32(Int32 value, String format, NumberFormatInfo info)
at System.Int32.ToString(String format, IFormatProvider provider)
at System.DateTimeFormat.FormatCustomized(DateTime dateTime, String format, DateTimeFormatInfo dtfi, TimeSpan offset)
at System.DateTimeFormat.Format(DateTime dateTime, String format, DateTimeFormatInfo dtfi, TimeSpan offset)
at System.DateTimeFormat.Format(DateTime dateTime, String format, DateTimeFormatInfo dtfi)
at Microsoft.SqlServer.Management.UI.Grid.StorageViewBase.GetCellDataAsString(Int64 iRow, Int32 iCol)
at Microsoft.SqlServer.Management.QueryExecution.QEResultSet.GetCellDataAsString(Int64 iRow, Int32 iCol)
at Microsoft.SqlServer.Management.UI.VSIntegration.Editors.GridResultsGrid.GetTextBasedColumnStringForClipboardText(Int64 rowIndex, Int32 colIndex)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetClipboardTextForCells(Int64 nStartRow, Int64 nEndRow, Int32 nStartCol, Int32 nEndCol)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetClipboardTextForSelectionBlock(Int32 nBlockNum)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetDataObjectInternal(Boolean bOnlyCurrentSelBlock)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetDataObject(Boolean bOnlyCurrentSelBlock)
at Microsoft.SqlServer.Management.UI.VSIntegration.Editors.GridResultsTabPageBase.OnCopyWithHeaders(Object sender, EventArgs a)

Based on my understanding, we read stack from bottom to top. So, if I build a stack by ignoring parameters, it would be like below.

FormatInt32
ToString
FormatCustomized
Format
Format
GetCellDataAsString
GetCellDataAsString
GetTextBasedColumnStringForClipboardText
GetClipboardTextForCells
GetClipboardTextForSelectionBlock
GetDataObjectInternal
GetDataObject
OnCopyWithHeaders

As we can see “Clipboard” – I would assume that its due to copy we are seeing out of memory because we are copying many rows from grid.

WORKAROUND/SOLUTION

As we discovered above, I explained to them that SQL Server Management Studio is not design to handle such kind of requirement. If we want to save the result set into the file, we should save the query output directly to file rather than grid or text in SSMS (and then doing Ctrl + C and Ctrl + V).

Other option would be to follow steps given in one of my earlier blogs

SQL SERVER – Automatically Store Results of Query to File with sqlcmd

SQL SERVER – SSMS Trick – Generating CSV file using Management Studio

I also found that if result is very large, even query execution can fill SSMS buffer and raise same error.

Hope you would be able to work around the issue by using this blog.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER Management Studio – Exception of type ‘System.OutOfMemoryException’ was thrown. (mscorlib)

While applying patches on SQL server standalone instance, in the cluster, it failed with error as below in Summary.txt file. Let us learn about how to fix Exception data is: System.IO.IOException.

Overall summary:
Final result: The patch installer has failed to update the following instance: RTCLOCAL. To determine the reason for failure, review the log files.
Exit code (Decimal): -2058616824
Start time: 2017-01-02 03:23:07
End time: 2017-01-02 03:39:34
Requested action: Patch

When I looked into Detail.txt and searched for error, here was the place of failure.

‘C:\Windows\system32\hadrres.dll’. Exception data is: System.IO.IOException: The process cannot access the file ‘C:\Windows\system32\hadrres.dll’ because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
at Microsoft.SqlServer.Configuration.Cluster.UpdateClusterResourceAction.VersionCopy(String source, String target).

Final Parameter Values

(01) 2017-01-02 03:35:41 Slp: Parameter 0 : SQL Server 2012@RTM@KB3072779
(01) 2017-01-02 03:35:41 Slp: Parameter 1 : 0xE349FEF6
(01) 2017-01-02 03:35:41 Slp: Parameter 2 : 0x60797DC7
(01) 2017-01-02 03:35:41 Slp: Parameter 3 : 0xE02C15DF@1356@8
(01) 2017-01-02 03:35:41 Slp: Parameter 4 : 0x24C2C4E7
(01) 2017-01-02 03:35:41 Slp: Parameter 5 : HADRResourceDLLConfigpatch_HADRResourceDLLWorkflow
(01) 2017-01-02 03:35:41 Slp: Parameter 6 : 0x871FC66E
(01) 2017-01-02 03:35:41 Slp: Parameter 7 : 0x27FE6E93
(01) 2017-01-02 03:35:41 Slp: Parameter 8 : 0x27FE6E93

If I understand the message correctly, SQL setup is trying to copy file “hadrres.dll” from “c:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn\” to “C:\Windows\system32”. Due to some reason, we are getting error “The process cannot access the file ‘C:\Windows\system32\hadrres.dll’ because it is being used by another process”

Since the patch was incompletely installed, I found that SQL was not starting.

WORKAROUND/SOLUTION

I had no idea as to why copy is not working so I asked to rename files in C:\Windows\system32 folder and stop any monitoring software related services. After renaming, we could apply the patch and found new file copied via setup. We verified and found that version of the file was higher after installation was complete.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – UpdateHADRResource – Failed to version-copy file – Exception data is: System.IO.IOException

Some trivia for your Friday. I recently got a question asking about what the attribute “monitorLoop” is in the blocked-process-report because it’s not really documented.

You won’t find too much explanation about that field in the official documentation, because the schema is considered undocumented. But I believe I know what it means.

The blocked process report is closely tied to deadlock detection and it’s generated by the same process as the deadlock monitor. If you remember, the deadlock monitor runs frequently looking for deadlocks (which are just blocking chains in a circle). It runs every couple seconds when there are no deadlocks, and if it detects any, it runs a bit more frequently. Each time it runs it’s called a monitor loop. The monitorLoop is just a number that starts at zero when the server restarts and increments by one each time the monitor runs.

For me, when processing a collection of blocked process reports, it’s useful way to group them together so that if several blocked process reports have the same monitor loop, then they were detected at the same time and can be part of the same blocking chain.

This attribute wasn’t always there. There are some old versions of SQL Server that don’t provide the monitorLoop attribute. This is a reminder that the schema of the blocked process report can change (has changed and will change) without notice.

As a part of my independent consulting, I also provide a high level overview of the migration strategy. As a part of this engagement, my client wanted me to provide steps to move AlwaysOn availability group from DomainOld to DomainNew. This was happening because they were acquired by another company and they need to migrate all assets to a new domain.

Topology:

Here are the steps which I told them.

• Remove replica Node-03 from availability group.
• Evict Node-03 from the windows cluster.
• Since Node-03 is completely out of AG, we can rebuild Node-03 in DomainNew
• Create a new windows cluster having just one node Node-03.
• Use scripts to migrate all server level objects – logins, jobs, operators from Node-01 to Node-03.
• Node-02 also needs same steps as Node-03 with one difference that it would be part of the same cluster as Node-03 rather than new cluster.
• Configure database mirroring for user databases from Node-01 to Node-03.
• OUTAGE: failover mirror databases to Node03, break the mirror and point application to Node-03.
• Perform test and test and test.
• Evict Node-01 and decommission the original cluster.
• Rebuild Node-01 in the new domain and add it to the cluster.
• Backup and restore the database from the Node-03 to the two new nodes and implement AG across all three new nodes.

After getting above high level guidance, they executed the project and it went fine.

Hope this step would help others as well about AlwaysOn Availability Group.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Migrating AlwaysOn Availability Group to New AD Domain

Recently, while deploying a hybrid AlwaysOn availability group for a client, I faced this error. Since it was something I was not able to find many hits on internet search, I thought of sharing this via this blog. I am sure it would help others.

Topology

The client was trying to deploy hybrid cluster with both on-premise instances and instances hosted in Microsoft’s Azure cloud. All machines were domain-joined and it they were part of multi-subnet network connected via Express-route.

Error message

Here was the error message when they were trying to add a replica in Azure VM.

Msg 19456, Level 16, State 1, Line 3
None of the IP addresses configured for the availability group listener can be hosted by the server ‘AZURESQL-1’. Either configure a public cluster network on which one of the specified IP addresses can be hosted, or add another listener IP address which can be hosted on a public cluster network for this server.

Msg 41158, Level 16, State 3, Line 3
Failed to join local availability replica to availability, group ‘HR_AG’. The operation encountered SQL Server error 19456 and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry the ALTER AVAILABILITY GROUP JOIN command.

Workaround / Solution

I looked at the IP addresses under the network name in the “Failover Cluster Manager” and found that there were two IPs: one in 10.150.xx.xx and one 10.160.xx.xx range. We saw that the replica that we were attempting to add was in 10.140.xx.xx. So, we then added an IP address in the appropriate subnet as a dependency of the network name.

In short, this error can be resolved by adding a right IP to the listener. The IP address for all subnets cannot be an IP address already in use, i.e. The IP address of one of the nodes. After adding above, we again attempted to join the replica. And as expected, the operation succeeded.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Error 19456: None of the IP Addresses Configured for the Availability Group Listener can be Hosted by the Server