THE SQL Server Blog Spot on the Web

Welcome to - The SQL Server blog spot on the web Sign in | |
in Search

Kalen Delaney

  • Geek City: More Hekaton Details

    I apologize to all you folks who attended my Hekaton precon at the SQLPASS Summit last month, who have been eagerly awaiting my answers to some of the unanswered questions. I had got most of the answers sorted out, but I was waiting for one last reply, and put the whole list on the back burner. I just realized today it was now December, and I still hadn’t published this.

    So here it is, six questions that I didn’t completely answer during my talk, although for some of them I did make a guess.

    1. Do SCHEMA_ONLY tables affect the upper limit of memory available for memory-optimized table?

    Remember, there actually is no hard upper limit for the amount of memory you can use for memory-optimized tables. The recommended maximum of 256 GB has to do with the number of CFPs (Checkpoint File Pairs) that can be supported. So in that sense you might be able to have more data in SCHEMA_ONLY tables because you don’t have to worry about the size of the CPFs, as the data in those tables is never written to the files. But you still need to be aware of the overall system memory limit and make sure your machine has enough memory for all the memory_optimized tables, both SCHEMA_AND_DATA and SCHEMA_ONLY) as well as memory for the buffer pool for your disk-based tables. Plus memory for other SQL Server and system needs.

    If you have bound a database to a resource pool with a fixed upper limit of memory, all your tables have fit within that memory limit.


    2.  Is Point-in-time RESTORE possible for databases that have performed transactions on memory-optimized tables?  (i.e. can we do a RESTORE LOG WITH STOPAT command?)

    Simple answer here: YES


    3. Can a database with memory-optimized tables be restored or attached onto a SQL Server 2014 instance using STANDARD Edition?

    No. Both restore and attach will require that all filegroups are available, including the filegroups for your memory-optimized tables. And once you have those special filegroups, the database cannot be restored or attached on STANDARD Edition and you get an error. So you really don’t even have to have any memory-optimized tables. As soon as you create a memory-optimized filegroup, the database will no longer be accepted.

    I wish I could tell you what the exact error would be, but I don’t have a SQL Server 2014 STANDARD Edition to test on. I have frequently requested that Microsoft provide a feature that allows us to ‘switch off’ Enterprise features if we are using EVALUATION or DEVELOPER Edition, so we can actually test using a base other than ENTERPRISE. But so far, Microsoft has not complied.  There are two Connect items that seem to address this this need. The first one at is marked “Closed as WON’T FIX’ but the other is still active at, so you might want to go give it a vote.


    4. Can views on memory-optimized tables be accessed from within a natively compiled procedure? (I think the question was sort of hinting at a workaround where you could create a view on a disk-based table and access that through the native procedure.)

    The answer is NO. You can create a view on a memory-optimized table, but you cannot access it from a native procedure. No views at all can be referenced in a native procedure.


    5. With REPEATABLE READ Isolation, with which SQL Server needs to guarantee READ STABILITY, what happens if a column that wasn’t read is changed? (i.e. your query read col1 and col2 from Table1, what happens if another query changes col3?)

    It’s the row that is important. If Tx1 reads col1 and col2 from Table1 in REPEATABLE READ and doesn’t commit, then Tx2 updates col3 in Table1, when  Tx1 commits it will then fail due to REPEATABLE READ violation.

    6. Can transactions on memory-optimized tables run in RCSI (READ COMMITTED SNAPSHOT Isolation)?

    RCSI is just a variant of READ COMMITTED Isolation. Operations on memory-optimized tables have to run in something higher than READ COMMITTED, so RCSI is moot.


    I hope this is useful!



  • Did You Know? My PASS Demo Scripts are Up (and other news)!

    And so another PASS Summit passes into history. It was an awesome week, filled with old friends and new, and lots of superlative technical content! My Hekaton book was released just in time, and it was great seeing the excitement. Red Gate gave away all the copies they had, the bookstore sold all they had, and I gave away 20 in my sessions.

    There's also a review of my book up already:  

    My PASS Demo scripts are now available on my website. Just go to and click on Resources in the left-hand menu, and then choose Conference Content.

    The winners of my book giveaway were announced on Twitter, and email was sent out. I thought I was going to have to say that one of the winners had not yet responded, but just as I started writing, the sixth winner’s  email arrived.  I’m still scrambling to get caught up on everything that got put off while I was at the MVP Summit and PASS Summit last week, but I hope to have the books shipped by the end of this week.

    Here’s the list:


    And now that PASS is over, my next project is updating my SQL Server Internals course to SQL Server 2014. The first public delivery will be in Stockholm in February.  Check my schedule for more details:

    We are having a glorious Autumn here in the Beautiful Pacific Northwest. I hope you are enjoying your Autumn (or Spring in the Southern Hemisphere) as much as I am enjoying it here!



  • Did You Know? It’s Anniversary Time Again

    And it coincides with a new book of mine being published. So I decided I would give away some presents!

    Today marks 27 years that I have been working with SQL Server. On this day in 1987 I started working for the Sybase Corporation in Berkeley, California, and SQL Server has consumed by working life, ever since….

    I still have quite a few copies of my SQL Server 2012 Internals book, plus all the copies of the new In-Memory OLTP book, so I’ll be giving away a couple of each. (Many more of the In-Memory OLTP book will be given away at the PASS Summit next week.)


    So what do you have to do?

    1. Send an INSERT statement to this email address

    2. Include "Happy Anniversary" in the subject line. The mail will be filed as junk if you don't.

    3. The body of the email should consist of a single T-SQL INSERT statement (shown below).  For the last column, choose which book you’d prefer. 
        A 1 will indicate the SQL Server 2012 Internals book, and a 2 will indicate the In-Memory OLTP book.

    4. Doublecheck the syntax! Make sure you have open and close quotes for all strings, and that the quotes are STRAIGHT quotes, as in the INSERT statement below. If you use Word to add a quote of your own or replace one that's there, it's usually a "smart" quote, which in this case is NOT smart. SQL Server will reject the INSERT if the quotes are not STRAIGHT. I suggest you use SSMS to write the insert.

    5. Send the email before 23:59:59, October 30, US Pacific Time.

    Here is the INSERT statement. Replace the strings with your specific information, and include it in your email:

    INSERT INTO Raffle(emailAddress, firstName, lastName, country, city_region, choice)
                   VALUES('Your email address', 'First Name', 'Last Name', 'Country', 'City, State or Region', [1|2] );

    Due to postage costs, I’ll only choose one winner from outside the US. I’ll choose 3 from within the US.


    If my friend John wanted to enter, and wanted the In-Memory OLTP book, his INSERT might look like this:

    INSERT INTO Raffle(emailAddress, firstName, lastName, country, city_region, choice)
                   VALUES('', 'John', 'Cook', 'US', 'Houston, TX', 2 );


    If my son-in-law in Germany wanted to enter, and he wanted the 2012 Internals book, his INSERT might look like this:

    INSERT INTO Raffle(emailAddress, firstName, lastName, country, city_region, choice)
                   VALUES('', 'Manuel', 'Meyer', 'Germany', 'Weil der Stadt', 1 );


    Thank you!


  • Did You Know? My Hekaton Book is at the Publishers!

    I am very excited to be able to report that my latest book is at the printers now, and should be ready in time for the PASS Summit!

    Front and back

    And since the last files were sent off to the publisher (Red Gate) last Friday, I could finally finish up the slides for my Pre-Conference Seminar, all about In-Memory OLTP.

    Check it out here:

    Although there will be some copies of the book at  the Red Gate booth, along with many of their other wonderful titles, Red Gate will also be giving me a box full of books to give away during the session. There won’t be enough for everyone (unfortunately) but your chances of snagging one will be good!

    See you in November!


  • Did You Know? I’ll be talking for a full day at the PASS Summit!


    I’ll be speaking on my new favorite topic… Hekaton! (aka In-memory OLTP)

    I was delighted to have the chance to give a taste of the presentation for the the 24 Hours of PASS preview on September 9th.  I was in Germany at my daughter's house that day, so although my US audience heard me in the morning, for me, it was almost supper time.  I am always impressed by the quality of the questions I get when giving online talks, and this was no exception. I was only able to address about a third of the questions that came in, but the organizers sent me a list of most of the submitted questions to address later. 

    Note that I won’t be giving really detailed answers to most of the questions. After all, this session was just a preview of my precon at the PASS Summit. In addition, I am just finishing up a book on In-memory OLTP for Red Gate, that hopefully will be released at the Summit. And that will have lots more details!

    Also keep in mind that this is a brand new technology, so the best practices and use cases are still being explored and those people out there actually implementing In-memory OLTP solutions are discovering new things all the time!


    Q:  Is the size of the table (or tables) that you want to use Hekaton on dependent on the amount of RAM you have available?

    Absolutely. You can’t have more data in memory than will fit in your memory. In addition, if not all your tables are in-memory tables, you’ll need to make sure there is enough memory for the buffer pool to hold needed pages from the disk-based tables. You also need to make sure you have enough memory for multiple versions of rows in your in-memory tables, based on how much updating you’re going to be doing.


    Q: How can I tell if a disk-based table has overflow rows, before I attempt to migrate it to Hekaton.  Also, how would I workaround this?

    You can look for any objects that have rows in sys.allocation_units with a type description of anything other than IN_ROW_DATA. This will give tables with either ROW_OVERFLOW or LOB data. If you just want to know the tables with ROW_OVERFLOW data,  look for where the type_desc is ROW_OVERFLOW_DATA. This query should get you started:

    SELECT OBJECT_NAME(object_id) FROM sys.partitions
    WHERE index_id < 2
      AND partition_id IN (
       SELECT container_id FROM sys.allocation_units
       WHERE type_desc != 'IN_ROW_DATA'
       AND total_pages > 0)

    Working around this might take a table redesign (which is a little beyond the scope of this short Q&A).


    Q:  Is it a requirement that the memory should be double the amount of data to efficiently use the Hekaton engine?

    As mentioned in the first answer, you need to have room for all the table data for tables you want in-memory, and you need to have enough room for the indexes on those tables, and for the versions of rows as the data changes. In some cases you might end up needing more than twice the data size, but it completely depends on the volume of your data changes, and the indexes you’ve created. If you have insufficient memory, it won’t just be a matter of not efficiently using the Hekaton engine; you will not be able to do any more data modifications once you are out of memory. But, there is no hard and fast rule. You need to  know your data and your operations, and you need to test thoroughly.


    Q: What happens as unique row counts exceed allocated bucket sizes?
    Q: Do buckets grow as data is inserted ?   Can you manually alter the number of buckets?

    I will assume you are asking what happens when the number of unique values exceeds the predefined bucket count. Here, this does become a matter of efficiency. If there are lots fewer buckets that unique values, one bucket will be used for multiple values and your search operations will be slower.  The number of buckets is fixed at the time the table is created and cannot be changed, either automatically or manually. You’ll have to drop and recreate the table.


    Q: When does the data write to disk, and do you have to compress the row history to free up memory space?
    Q: The data file name does not need an extension (like 'mdf')?

    New rows of data are written to disk using a mechanism called ‘continuous checkpoint’ which uses SQL Server’s  filestream technology to write to files on disk. There is no concept of a data file, so no .mdf is used for the in-memory data. The database containing the in-memory tables still needs a regular data file and a log file but the in-memory tables are not written there.  The filestream files where the in-memory data is written, to be used only in case needed for recovery, do not use a file extension. You might want to read about the filestream technology, which was introduced back in SQL Server 2008.


    Q: How do you calculate how much memory the table is going to need?

    The required space depends on the data in the rows, the number of rows, and the volume of updates. There is no simple formula for calculation because no one knows what your volume of changes will be. You can make a guess if you know your workload, and run some tests. There is good metadata available to tell you how much space you are using for your in-memory tables once you have created them. You can look at the memory used, run some of your workload, and then see how much more memory is used, to see how the memory requirements are growing.  You might start with the DMV: sys.dm_db_xtp_table_memory_stats.

    Q: I really like to use in memory OLTP but we frequently alter table . What is your recommendation on using In Memory OLTP for these table?

    As you’ve heard, once an in-memory table is created, it cannot be altered, not even to add more indexes, or to change the bucket count for any of the existing hash indexes. You’ll have to test whether dropping and recreating the table, or creating a new table and copying the data, can be done is a reasonable amount of time for your requirements. (Keep in mind that even with disk-based tables, some of the ALTER TABLE operations are not fast, efficient changes. Some ALTER TABLE options require that SQL Server completely recreate every row. )


    If you want even more details on Hekaton in SQL Server 2014, there are still seats in my precon (last I heard)!


    And check out the Red Gate booth in the vendor area to find my new In-Memory OLTP Internals book!


    Have fun!


  • Geek City: Lost Identity

    Way long ago, when Sybase first added the IDENTITY property, it took people quite a while to get the hang of using it. Along with being able to automatically generate sequential numbers for each row, we had to tune a configuration option with the geekiest of names: “Identity Burning Set Factor”.  And figuring out how to use this value seemed to be one of the dark arts, so almost everyone left it at the default. (If you really want to know how to use the burning set, you can read the Sybase docs here.) The question of WHY we needed to even think about a burning set was never even an issue for most people.

    So when Microsoft included IDENTITY columns with no requirement of defining a burning set, it was a big relief. This isn’t to say that there were no issues with using IDENTITY columns at all. I still had to explain to students over and over that even with IDENTITY there was no guarantee of no gaps and no guarantee that you would never get duplicates. Basically, gaps could happen if you rolled back INSERT operations (or explicitly deleted rows) and duplicates could occur if you set the table property IDENTITY_INSERT to ON.  But once people understood this, everything else was pretty straightforward. Until now, with SQL Server 2012.

    I had actually started hearing strange reports of unexpected gaps in IDENTITY values quite a while ago, but the cases seemed to be not reproducible and certainly not predictable. But just recently, I came across the explanation and realized the reason it seemed so unpredictable was because there is a combination of conditions that both have to occur to see this particular behavior.

    It is true that there is a change of IDENTITY behavior in SQL Server 2012, to allow for greater throughput in heavy insert environments. In earlier versions, as each identity value was generated, it had to be logged separately. This meant that even for minimally logged operations, like SELECT INTO, every row inserted had to be logged separately. I blogged about this behavior here. But in 2012, identity values are generated in batches and only the maximum value of the IDENTITY column in the batch is logged. So minimally logged operations on tables with IDENTITY columns could be more efficient, but I still haven’t got around to actually testing that yet. But a related change is that if there is a server failure, SQL Server has to be sure it won’t reuse an IDENTITY value. If SQL Server was stopped without a CHECKPOINT, on restart the engine will add to the maximum value of the IDENTITY that it is aware of to set a new starting point for each table, thus potentially leaving a gap in the sequence. The size of the gap is dependent on the data type of the column. The value will be increased by 10 for tinyint, 100 for smallint, 1000 for int, and 10000 for bigint. Numeric and decimal increase their values based on the precision, but it seems the maximum is 10,000.

    If SQL Server is stopped ‘gracefully’, with a CHECKPOINT performed, there should be no gap. Ideally, a checkpoint is always issued for a service shutdown, unless you use the T-SQL command SHUTDOWN WITH NOWAIT, or if there is a crash, but apparently, there is a bug. It seems that stopping your SQL Server using Configuration Manager, or even the service control in SQL Server Management Studio will not perform a checkpoint. To make sure you get the checkpoint you can use SQL Server’s SHUTDOWN command (without the WITH NOWAIT option!) or of course, manually issue a CHECKPOINT before shutting down. So only crashes could cause these gaps.


    The gaps on restart only happen because SQL Server assigns IDENTITY values in blocks. You can choose to revert to the old behavior that allocated IDENTITY values one at a time, and logged each row even with minimally logged operations. To do this, you need to add trace flag 272  to your SQL Server startup parameters, set through the Configuration Manager.  Unofficial reports state that you have to use a lower case t to specify the flag, i.e. –t272 and NOT –T272.

    enable traceflag



    If you want to observe this gappy behavior yourself, you can run the following code. I tested it on SQL Server 2012 SP1, running on Windows Server 2008R2 SP1. I noticed that I didn’t get any gaps with only a single row in the table, after I restarted. I needed at least two rows. I used a database called testdb, but you can use any testing database you have.


    USE testdb;

    IF OBJECT_ID('ident1') IS NOT NULL DROP TABLE ident1;
    IF OBJECT_ID('ident2') IS NOT NULL DROP TABLE ident2;
    IF OBJECT_ID('ident3') IS NOT NULL DROP TABLE ident3;
    IF OBJECT_ID('ident4') IS NOT NULL DROP TABLE ident4;
    IF OBJECT_ID('ident5') IS NOT NULL DROP TABLE ident5;

    -- Create 5 tables with different data types for the identity column
    CREATE TABLE ident1
    (col_tinyint tinyint IDENTITY);
    CREATE TABLE ident2
    (col_smallint smallint IDENTITY);
    CREATE TABLE ident3
    (col_int int IDENTITY);
    CREATE TABLE ident4
    (col_bigint bigint IDENTITY);
    CREATE TABLE ident5
    (col_numeric30 numeric (30,0) IDENTITY);

    -- Insert 2 rows into each table

    -- uncomment the checkpoint for the second test;

    --- Shutdown your SQL Server using the SHUTDOWN WITH NOWAIT command, the configuration manager, or SQL Server Management Studio.

    -- Restart your SQL Server


    SELECT * FROM ident1;
    SELECT * FROM ident2;
    SELECT * FROM ident3;
    SELECT * FROM ident4;
    SELECT * FROM ident5;

    And here are the results I got when I restarted after shutting down using SHUTDOWN WITH NOWAIT:



    You can run the test a second time, running a checkpoint right before shutdown to see that no gaps are left.

    Of course, the issue of WHY a checkpoint is automatically performed before a controlled service stop is another issue, and hopefully there will be a fix for that soon.

    But even once it’s fixed, uncontrolled shutdowns will still leave gaps and you need to be aware of this and not expect your IDENTITY values to always be perfectly contiguous.


    I hope this is useful!


  • Geek City: A Hint of Degrees

    This is just a quick post to describe a test I just ran to satisfy my own curiosity.

    I remember when Microsoft first introduced the query hint OPTION (MAXDOP N). We already had the configuration option ‘max degree of parallelism’, so there were lots of questions about how the hint interacted with the configuration option. Some people thought the configuration option set an absolute maximum, and the hint could only specify something less than that value to be meaningful. Other people thought differently, and I actually can’t remember what I thought at the time. All I  remember is that there was confusion. So I decided to test it recently.

    I have a machine with 8 logical processors, so I can have a degree of parallelism up to 8. And my first test showed that all 8 were used when I left the configuration option set to the default and didn’t use a hint.  I ran this test on both SQL Server 2012 SP1 and SQL Server 2014 RTM. The results were the same. I needed to look at the actual execution plan, which runs the query, because the decision of how what degree of parallelism to use is not determined until runtime. If I look at an estimated execution plan, I will see a parallelism operator, but not degree of parallelism will be reported, because it is not known.  I can look get the actual plan either by using the button on the Management Studio toolbar called “Include Actual Execution Plan”, or I can use SET STATISTICS XML ON.

    So, as mentioned, the first test showed a degree of parallelism of 8:

    USE AdventureWorks2012;
    ORDER BY UnitPrice DESC;

    Screenshot showing DOP

    I then changed the configuration option, and ran the same test.

    EXEC sp_configure 'max degree of parallelism', 4; RECONFIGURE;
    ORDER BY UnitPrice DESC;

    This time, the degree of parallelism was 4, as expected.

    I then added the MAXDOP hint, with a value smaller than 4 and then a value larger than 4.

    ORDER BY UnitPrice DESC
    ORDER BY UnitPrice DESC

    In both cases, the degree of parallelism was the same as the value specified in the hint. So it seems, with this basic test, that a MAXDOP hint will completely override anything set in the configuration option. Whether this was true in the version when the hint was first introduced, I cannot say. But it’s true for me now, and that’s what’s important.

    So I’ll clean up and then go back to playing with Hekaton.

    EXEC sp_configure 'max degree of parallelism', 0; RECONFIGURE;

    Have fun!


  • Geek City: The Hekaton Saga Continues, and some answers to questions

    My Hekaton (aka In-memory OLTP) books is with the publisher. And I have no doubt they will do a much better job than my previous publisher did in bringing my SQL Server 2012 Internals Book into the light of day. 

    My only regret is that my awesome Tech Editor, Benjamin Nevarez, who has been with me since my SQL Server 2008 Internals book, was not available for this book since he is now writing books of his own! But he recommended the person who tech edited his book and I’m pretty sure I can trust Ben’s judgment on this. I’m intentionally not mentioning his name in this post, but I’m sure it will come up in a future post.

    I gave my first full day Hekaton presentation as a precon in Atlanta on May 2. I was worried that I wouldn’t have enough content to fill an entire day, but it pretty much worked out perfectly. As expected, there were some questions I couldn’t answer on the spot, either because I was too jet lagged, or because I just hadn’t thought about it enough yet, but I have some of those answers now, thanks to Sunil Argawal and Jos de Bruijn at Microsoft.


    1. Are default constraints allowed on memory-optimized tables?

    YES! However, there is a bug in the memory-optimization advisor that shows them as unsupported. In addition, not every expression that would be supported for defaults on disk-based tables is supported for defaults on memory-optimized tables. Only expressions that are allowed in natively-compiled procedures are allowed in default constraints on memory-optimized tables, but this does include a lot of the most frequently used ones: sysdatetime(), newid() and constants.

    2.  Are the data values in the rows compressed in any way.

    NO! No compression is used to store the data values in the rows.

    3. If a transaction fails after it is has obtained its timestamp, but then fails during validation, will its timestamp be reused?

    NO! Timestamps are never reused once they have been assigned.

    4. How is the timestamp used for reading rows determined?

    Every transaction that reads rows from memory-optimized tables reads as of a certain ‘point in time’, i.e. a timestamp. It then can only read row versions that were inserted after the read before the read timestamp, and not deleted until after the read timestamp (or not deleted yet at all!).  SQL Server maintains a current timestamp used for all read operations, which is incremented when a read/write transaction enters validation and is assigned its own timestamp. It can also be incremented for internal system transactions that are never visible to users. So basically, the read timestamp is the timestamp of the most recent committed transaction in the system.  Multiple read operations can share the same timestamp used for their reads.

    5. The documentation says that cross-database transactions are not allowed with memory-optimized tables. What about transactions involving temp tables (so they are in tempdb)?

    Since there can be no memory optimized tables in tempdb, this restriction does not apply. Cross-database transactions are allowed with tempdb and model, and read-only cross-database transactions are allowed with master.

    6. Can an identity value in a memory-optimized table be reseeded?

    (Note: earlier versions of Hekaton did not support identity at all, and I still get questions from people thinking that is still true. Hekaton tables can have identity columns as long as the seed is 1 and the increment is 1.) No reseed is possible but you can use IDENTITY_INSERT to manually increase the seed.

    7.  Will DBCC CHECKDB check memory-optimized tables?

    NO! DBCC CHECKDB does not include any checks on any part of Hekaton. However, checksums are maintained for the checkpoint files, and explicit checksums are supported for database backups, even those containing memory-optimized tables.

    8.  IF a checkpoint file grows larger than 128K due to large transactions, are there special considerations on merging it with neighboring files? E.g. is SQL Server more aggressive in merging large files?

    A checkpoint file pair (CFP) can be self-merged if the data file is larger than 256 MB and over half of the rows are marked deleted. Otherwise no…

    9.  Can transaction logs containing operations on memory-optimized tables be used for recovery to a point in time or to a log mark?


    10. Can the checkpoint files be excluded from virus scanners?

    Right now, there is no easy way to do this since the checkpoint files have no suffix. However, the team at Microsoft will be looking into this.


    Thanks for all the great questions!


    And if you want more Hekaton details, check out the class I recorded for SSWUG a couple of weeks ago:  



  • Geek City: A row with no fixed length columns

    Today I will return to my roots and post about the bits and bytes of internal storage. I received a request a while back to help with deciphering the DBCC PAGE output for a row with no fixed length columns. The person had read the section in my book (SQL Server 2008 Internals) on row storage, but was still having trouble. He presented this table with one row:


    USE testdb;

    CREATE TABLE varchk(name varchar(10))
    INSERT INTO varchk VALUES ('kalen')

    He used DBCC IND to find the page number:

    DBCC IND (testdb, varchk, 1)

    and then then took the file and page number returned to pass to DBCC PAGE:

    DBCC TRACEON(3604)
    DBCC PAGE (testdb, 1, 188, 1)

    He saw these bytes for the row:

    30000400 01000001 0010006b 616c656e †0..........kalen

    These were his questions:

    How to find the variable column offset?
    How to find the end fixed length column and start of Null bitmap?
    How to find the offset of each variable-length column?


    And here was my reply:

    Here is your row:

    30000400 01000001 0010006b 616c656e, it is exactly 16 bytes long.

    3000 = Status Bits

    0400 = 2 byte integer for length of fixed length portion, reverse bytes because it is a single integer = 0004 = 4 bytes for everything up to last fixed length column

    0100  = 2 byte integer for number of columns, reverse bytes because it is a single integer = 0001 = 1 byte

    00 (shaded) = Null bitmap

    0100 = 2 byte integer for number of variable length columns, reverse bytes because it is a single integer = 0001 = 1 column

    1000 (shaded) = 2 byte integer for position where 1st variable length column ends, reverse bytes because it is a single integer = 0010 = 16, which is last column where variable length column ends

    6b616c656e = variable length data

    6b = k

    61 = a

    6c = l

    65 = e

    6e = n

    Your question said you were looking for the offset of the variable length column. Please reread the section (in the book) that describes the storage of rows with variable length columns.  We are storing the ending POSITION, not the offset, and I say this:

    A data row that has any variable-length columns has a column offset array in the data row with a 2-byte entry for each non-NULL variable-length column, indicating the position within the row where the column ends. (The terms offset and position aren’t exactly interchangeable. Offset is 0-based, and position is 1-based. A byte at an offset of 7 is in the eighth byte position in the row.)

    I know that’s pretty geeky, but for anyone else struggling with understanding row storage, maybe all you needed was just one more example!


    Have fun!


  • Geek City: Did You Know … that not everything you know is actually true!

    Software changes, new versions not only add new features, they also change internal behavior of old features,  and not all the changes are documented!  And how often, after a software upgrade, do you go through and test everything you knew was true, to make sure it is STILL true after the upgrade? I write books about the SQL Server software, and although I do verify all the code in my books against the new version, I admit that I don’t always check every single fact mentioned in the entire book to make sure it is still true.

    There are a number of such changes that I’ve discovered over the last year or so, that I’ll be telling you about in a couple of blog posts. Plus, there are things that I read on other peoples’ blogs, where the author states something is true that I know is no longer true. Here’s an example:

    It used to be true, a very long time ago, that you could not rollback a TRUNCATE TABLE operation. That changed so long ago, I can’t even remember what version it was in. But in all current versions, you can rollback a TRUNCATE TABLE, and it very easy to prove that. All you have to do is question whether or not the fact that “you cannot roll back a TRUNCATE TABLE” is actually true. But if you think it’s true, and never question it, you’ll never know.

    Here’s the example I use in my classes to show that you CAN rollback a TRUNCATE TABLE:

    USE testdb – or any test database of your choice

    IF object_id('smallrows') IS NOT NULL
        DROP TABLE smallrows;

    CREATE TABLE smallrows
        b char(10)

    INSERT INTO smallrows VALUES
            ('row 1'),
            ('row 2'),
            ('row 3'),
            ('row 4'),
            ('row 5');
    SELECT * FROM smallrows;

    -- TRUNCATE TABLE can be rolled back
    TRUNCATE TABLE smallrows;
    SELECT * FROM smallrows;
    SELECT * FROM smallrows;

    In my classes, right after I talk about DELETE and TRUNCATE operations, I talk about UPDATE. You might be aware that UPDATE can be performed in a couple of different ways, SQL Server can do something called an “update-in-place” where it just changes the old bytes to the new bytes, and logs one simple update operation. Or, the UPDATE can be performed as two separate operations: DELETE the entire old row (and update all the indexes and log the DELETE plus all the index changes) and then INSERT an entire new row (and update all the indexes and log the INSERT plus all the index changes.)  Obviously, update-in-place is preferred, but there are some conditions that must be met in order for SQL Server to perform an update-in-place. And one of the conditions I had on my list for years turned out not to be true any more when I tested it several months ago.

    The main reason that SQL Server might not do an update-in-place is because you are updating the clustered key value. And since the clustered key value determines where the row goes in the table, changing that value will change the row location. It cannot be done in-place (even when the row doesn’t actually have to move, as you’ll see in the example.) Another reason for not doing an update-in-place used to be because the table had an UPDATE TRIGGER. Prior to SQL Server 2005, the contents of the special ‘inserted’ and ‘deleted’ tables that were available inside a trigger were generated from the transaction log. So in order for these rows to be available, the entire new row and old row had to be logged. But in SQL Server 2005, with the introduction of the version store, mainly used to support row version for snapshot isolation, SQL Server started using the version store technology to get the old and new versions of updated rows if there was a trigger that needed them.

    So did having an update trigger still preclude update-in-place?

    I decided to test it.

    This first block creates a table with no clustered index and no trigger.

    -- no clustered index, no trigger
    IF object_id('test1') IS NOT NULL DROP TABLE test1;
    CREATE TABLE test1
    (a int,
      b char(200) );
    INSERT INTO test1 values (1, replicate('a',200));
    SELECT * FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test1%';

    You should see 5 rows in the log, with the last one being an operation LOP_INSERT_ROWS, which indicated the single row inserted.

    Now update the row:

    UPDATE test1 SET a=2;
    SELECT [Current LSN], Operation, Context, [Log Record Length] as Length, AllocUnitName
    FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test1%';

    You’ll see something like this:


    There is one additional row, with operation LOP_MODIFY_ROW, for the update-in-place. Notice the length of that log record is 100 bytes.

    In the INSERT log record, the length is 304 bytes.

    Now run the same test on a table with a clustered index on column a.

    --  clustered index, no trigger
    IF object_id('test2') IS NOT NULL DROP TABLE test2;
    CREATE TABLE test2
    (a int primary key clustered,
      b char(200) );
    INSERT INTO test2 values (1, replicate('a',200));
    SELECT * FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test2%';
    UPDATE test2 SET a=2;
    SELECT [Current LSN], Operation, Context, [Log Record Length] as Length, AllocUnitName
    FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test2%';

    This time, after the UPDATE,  you’ll see these log records:


    Notice there is no LOP_MODIFY_ROW for the clustered index. There is a LOP_DELETE_ROWS and LOP_INSERT_ROW and their lengths are much longer than the length of the LOP_MODIFY_ROW. The entire old row and entire new row are being logged. This may not seem like a big deal, but imagine instead of one row, that you are updating thousands or even millions of rows. Also note that there is just the single row in the table, so the row doesn’t have to move when updated. There’s only one place for it to be! But SQL Server only knows that the UPDATE is changing a clustered key value, so it is performed as the two step operation.

    Finally, I’ll do the test one more time, on a table with no clustered index, but with an UDPATE trigger.

    -- no clustered index, update trigger
    IF object_id('test3') IS NOT NULL DROP TABLE test3;
    CREATE TABLE test3
    (a int,
      b char(200) );
    CREATE TRIGGER upd_test3 on test3 for UPDATE
      SELECT * FROM inserted;
      SELECT * FROM deleted;
    INSERT INTO test3 values (1, replicate('a',200));
    SELECT * FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test3%';
    UPDATE test3 SET a=2;
    SELECT [Current LSN], Operation, Context, [Log Record Length] as Length, AllocUnitName
    FROM fn_dblog(null, null)
    WHERE AllocUnitName like '%test3%';

    Again, you’ll see only the one LOP_MODIFY_ROW, even though both the inserted and deleted rows were accessed in the trigger.


    So it was obvious that I had to update my course material.

    In my class two weeks ago, I found out another old ‘fact’ that is no longer true, but that will have to wait until next time. And of course, a whole new version is coming out in just a few months. I wonder what old facts will no longer be facts?

    Have fun!


    p.s. Right after I published this, I noticed I had already written up the update-in-place with triggers, just about a year ago. So if you missed it then, you know about it now!

  • Geek City: SQL Server 2014 In-Memory OLTP (“Hekaton”) Whitepaper for CTP2

    Last week at the PASS Summit in Charlotte, NC, the update of my whitepaper for CTP2 was released.

    The manager supervising the paper at Microsoft told me that David DeWitt himself said some very nice things about the technical quality of the paper, which was one of the most ego enhancing compliments I have ever gotten! Unfortunately, Dr. DeWitt said those things at his “After-the-keynote” session, not in the keynote that was recorded, so I only have my manager’s word for it. But I’ll take what I can get.

    If you’d like to read the update, which has information about the new “range” indexes, natively compiled procedures, checkpoint files, recovery and garbage collection in addition to all the content from the CTP1 paper, you can get it here:


    There will be more enhancements made for RTM, including some graphics for the index maintenance operations on range indexes, plus best practice suggestions.  These enhancements may be in the form of a third paper, but they might be in book form. I’ll let you know!



  • Did You Know? There is a PASS Conference in Stockholm!

    And I’ll be there!

    For the first time EVER, I will not be speaking at or attending the US PASS Summit. However, I will be speaking at the PASS SQL Rally in Stockholm November 4-6. I am so excited!

    I’ll be giving a pre-con you can read about here:

    And then I’ll speaking about SQL Server Hekaton (In-Memory Database) in a session on Tuesday.

    I hope to see many of you there.


  • Geek City: How old are my statistics?

    This post is basically to answer a question asked in class this week: How can we get the last statistics update date for ALL user tables in a database?

    After working on the query for a while, I realized that the new metadata function I posted about here can give you that info easily:

    SELECT object_name(sp.object_id) as object_name,name as stats_name, sp.stats_id, 
        last_updated, rows, rows_sampled, steps, unfiltered_rows, modification_counter
    FROM sys.stats AS s
    CROSS APPLY sys.dm_db_stats_properties(s.object_id, s.stats_id) AS sp
    WHERE sp.object_id > 100;

    But for those of you not yet running SQL Server 2008R2 Service Pack 2 or SQL Server 2012 Service Pack 1 (as long as it’s 2005 or later), I’ll show you the other query I had worked out:

    SELECT schema_name(schema_id) AS SchemaName,  object_name(o.object_id) AS ObjectName, AS IndexName, index_id, o.type,
        STATS_DATE(o.object_id, index_id) AS statistics_update_date
    FROM sys.indexes i join sys.objects o
           on i.object_id = o.object_id
    WHERE o.object_id > 100 AND index_id > 0
      AND is_ms_shipped = 0;

    There are a few slight differences in the output between the two queries, in addition to the fact that the query using sys.dm_db_stats_properties() adds the extra columns for number of rows sampled, the column modification counter, etc. The function does does  not return the schema_id or the type of object (e.g. is it a table or an indexed view). Both those pieces of information can be determined by joining with the sys.objects view, but I have left that as an exercise for the reader/student.


    I hope you find this useful!


  • Geek City: sp_cacheobjects for SQL Server 2012

    In a post about 4 1/2 years ago, I gave you my version of a replacement for the old pre-2005 pseudotable syscacheobjects. I called it sp_cacheobjects and created it as a view in the master database. With the sp_ prefix, the view can be accessed from any database.

    When testing this on SQL Server 2012, I noticed that I almost always got a lot more rows back than I was expecting. Even when I added a WHERE clause to limit the database to only the database I was working in, I STILL got way too many rows back. It turns out that in SQL Server 2012, SQL Server is frequently running background queries checking for both filetable and fulltext activites, and these background queries, that run in all databases, are cached like any other queries. To get the same kinds of results from this view that I got in SQL Server 2008, I needed to add a few extra filters, so I am including the modified version of my view here.

    -- Create a view to show most of the same information as SQL Server 2000's syscacheobjects
    -- This script has been updated for SQL Server 2012 to remove plans dealing with filetable and fulltext activities,
    -- and to also not return queries from system databases.

    -- by Kalen Delaney, 2012

    -- Feel free to remove those filters from the WHERE clause at the bottom
    USE master
    IF EXISTS (SELECT 1 FROM sys.views WHERE name = 'sp_cacheobjects')
        DROP VIEW sp_cacheobjects;
    CREATE VIEW sp_cacheobjects(bucketid, cacheobjtype, objtype, objid, dbid, dbidexec, uid, refcounts, 
                            usecounts, pagesused, setopts, langid, date_first, dateformat, status, lasttime, maxexectime, avgexectime, lastreads,
                            lastwrites, sqlbytes, sql)

                SELECT            pvt.bucketid, CONVERT(nvarchar(19), pvt.cacheobjtype) as cacheobjtype, pvt.objtype,
                                        CONVERT(int, pvt.objectid)as object_id, CONVERT(smallint, pvt.dbid) as dbid,
                                        CONVERT(smallint, pvt.dbid_execute) as execute_dbid, 
                                        CONVERT(smallint, pvt.user_id) as user_id,
                                        pvt.refcounts, pvt.usecounts, pvt.size_in_bytes / 8192 as size_in_bytes,
                                        CONVERT(int, pvt.set_options) as setopts, CONVERT(smallint, pvt.language_id) as langid,
                                        CONVERT(smallint, pvt.date_format) as date_format, CONVERT(int, pvt.status) as status,
                                        CONVERT(bigint, 0), CONVERT(bigint, 0), CONVERT(bigint, 0), 
                                        CONVERT(bigint, 0), CONVERT(bigint, 0),
                                        CONVERT(int, LEN(CONVERT(nvarchar(max), fgs.text)) * 2), CONVERT(nvarchar(3900), fgs.text)

                FROM (SELECT ecp.*, epa.attribute, epa.value
                            FROM sys.dm_exec_cached_plans ecp
                    OUTER APPLY sys.dm_exec_plan_attributes(ecp.plan_handle) epa) as ecpa
                       PIVOT (MAX(ecpa.value) for ecpa.attribute IN ([set_options],[objectid],[dbid],
                              [dbid_execute],[user_id],[language_id],[date_format],[status])) as pvt
                           OUTER APPLY sys.dm_exec_sql_text(pvt.plan_handle) fgs
             WHERE cacheobjtype like 'Compiled%'
             AND text NOT LIKE '%filetable%'
             AND text NOT LIKE '%fulltext%'
             AND pvt.dbid between 5 and 32766;


    I hope you find this useful!


  • Did You Know? What settings to always change


    A week ago, I taught my SQL Server 2012 Internals class to a great group of very interactive students. Even though a dozen of them were taking the class remotely, there were still lots of really great questions and and lots of discussion.

    One of the students asked if I could summarize all the settings that I recommended changing from the default, right out of the box. I said I’d try to put a list together by the end of the week, but I didn’t make it. So I said I would put it together and blog it.

    I think it sounded during the week like there were more changes than there really are. Going back through my notes, I only found three settings, all of them instance-wide configuration settings, that I recommend always changing. Of course, depending on your hardware, your workload and your data, you may make more changes. But the short list here contains the options that I always make sure are changed from the default on any system I am working on.  So here they are:

    1. Remote admin connections

    This option doesn’t apply to all remote administrative connections, despite the name, but only to connections made using the DAC (Dedicated Administrator Connection).  This option has a value of either 0 or 1, and 0 is the default. I recommend changing it to 1, which allow someone working at a machine other than the machine where your SQL Server is installed to make a DAC connection.  It might be too late to change it to 1 once you realize you need it! This value also needs to be set to 1 when you are connecting to a clustered SQL Server, which is always considered a remote connection.

    2.  Optimize for ad hoc workloads

    This also is a two-valued option, with a default of 0. I recommend changing it to 1.  There may be some very edge-case scenarios for leaving it at 0, but they’re pretty rare so you’re better off changing it to 1. In most cases, this can save you lots of memory, because single-use ad hoc query plans will now only use 300 bytes of memory instead of a minimum of 16K. Some plans use more, a lot more. I’ve seen SQL Server instances with tens of thousands of single-use ad hoc plans. I’ll let you do the math.

    3. Show advanced options

    By default, only about 16 of the configuration options are viewable and settable with sp_configure. If you want to see all your options, including ‘optimize for ad hoc workloads’, you need this option set to 1. The default is 0. 


    As a bonus, I’ll give you another list. These are options that usually should be left at the default value, so you should verify regularly that no one has changed them.  Two of them are instance-wide options, and two are database options.

    1. Auto create statistics and auto update statistics

    These are database options that have the default value of ON, and should be left that way. There may be cases you want to update statistics more often than auto statistics provides for, but that doesn’t mean you should turn the automatic updating off.  And there may be a few edge cases where you want to turn off the automatic update of statistics. Make sure you have a very good reason if you do so. But before you do, try turning on auto update statistics async to see if that helps whatever problems are leading you to consider turning automatic updates off. Also, take a look at the procedure sp_autostats. It allows to turn off automatic updates just for particular tables or indexes.  

    2.  Max degree of parallelism

    I’m not going to tell you what you should set this configuration option value to. There are a lot of recommendations out there, on other people’s blogs. But I’m just going to say make sure it is not set to 1 if you have more than one processor available to your SQL Server.  The default is 0, which means ALL SQL Server’s processors are available for parallel queries. In some (if not most?) cases, leaving it at the default is ok, but again, don’t set it to 1.  If you find particular queries do not perform well when run in parallel, you can use the MAXDOP hint for those queries. But don’t turn all parallelism off across the entire instance.

    3. C2 audit mode and common criteria compliance enabled

    Keep these configuration options set to 0 unless your business is mandated to have one of these options on. C2 audit mode has been deprecated but’s it still around for now. If you’re mandated to have these options enabled, you should know it. So otherwise keep these set to 0. You WILL notice performance degradation if you enable this.

    4. Autoshrink

    This is a database option that should never have been invented. Pretend it doesn’t exist, unless someone has set it to ON in one of your databases, then set it back to OFF. 


    So these are the options that I specifically call out in my class as having general best practice values. I talk about other options as well, but most of the others have an ‘it depends’ answer for what values you should use. And what ‘it depends’ on is what I spend a lot of my class talking about.

    Feel free to let me know if there are options on your list… that you always change, or always make sure are unchanged!




More Posts Next page »

This Blog


Favorite Non-technical Sites or Blogs

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement