Administrator's Guide to SQL Server 2005

Next Page

Book Cover
Administrator's Guide to SQL Server 2005
By Buck Woody
...............................................
Publisher: Addison Wesley Professional
Pub Date: June 08, 2006
Print ISBN-10: 0-321-39797-5
Print ISBN-13: 978-0-321-39797-3
Pages: 648

Table of Contents | Index
Overview

"Buck Woody's new book will help new and experienced users alike in getting productive with SQL Server 2005 quickly. His special attention to security and performance make this an exceptional reference that is sure to remain relevant for some time."

Chip Andrews, CISSP, MCDBA, Director of Research and Development Special Ops Security, Inc.

"SQL 2005 is the most exciting upgrade to come out of Redmond in years; this book will help you get up and running quickly to take full advantage of it! If you are looking for a book about upgrading to SQL 2005, look no further, this one is all you need. It gives you all you need to take advantage of everything SQL 2005 has to offer, and that is a lot!"

Doug Ellis, Area Technical Manager, Cognos

"Buck Woody provides insight into administering SQL Server 2005 that all developers and database administrators will find useful; the book assumes very little knowledge by providing 'DBA 101' sections which will not alienate the experienced reader while providing the necessary introduction to topics for those less experienced."

Doug Holland, Visual C# MVP, Precision Objects

"SQL Server 2005 is a landmark release and this title does an awesome job of hitting everything new in SQL 2005. It covers new and improved administration tools, and contains an overview of Notification Services, SQL Server Service Broker, Reporting Services, Integration Services, and Analysis Services. As a result, you'll be able to work on your own SQL Server 2005 solutions quickly and painlessly. Ultimately, this book provides you with a starting point, a roadmap, and then the tools and knowledge to exploit this powerful new database server to its fullest power. So, start your computer and get ready to learn SQL Server 2005!"

Alexzander N. Nepomnjashiy, SQL Server DBA, NeoSystems NorthWest

"A solid introduction to administering SQL 2005. The 'DBA 101' sections provide useful tips for the new DBA, good review for the experienced DBA, and the 101 label enables the seasoned DBA to skip basic material and jump ahead to where the REAL action is…. Buck's conversational writing style makes this an enjoyable and refreshing contrast to the majority of tech books written these days."

Brad Russell, MCDBA, Visionary Integration Professionals

The Complete, Practical, 100% Useful Guide for Every SQL Server 2005 DBA!

This book has one goal: to help database administrators and their managers run SQL Server 2005 with maximum efficiency, reliability, and performance. Renowned SQL Server expert Buck Woody covers every facet of database administration, from installation and configuration through maintaining enterprise-class business intelligence environments.

If you're a new SQL Server administrator, Buck will help you master core tasks rapidlyand avoid costly mistakes that only show up when it's too late. If you're familiar with previous versions of SQL Server, he'll get you up-to-speed fast on everything new that matters, from high availability improvements and Integration Services to the SQL Server Management Studio. Whatever your background, you'll find best practices, real-world scenarios, and easy-to-use automation scriptsall grounded in Buck's unsurpassed SQL Server knowledge.

• Timesaving tips, practical applications, and start-to-finish examples throughout

• Understand the key "decision points" at every stage of your SQL Server deployment

• Perform day-to-day maintenance more efficiently, from hotfixes to backup/recovery

• Automate maintenance tasks with the SQL Server Agent

• Establish and audit security on SQL Server 2005 and the underlying Windows Server platform

• Monitor and optimize SQL Server: baselining, profiling, index strategies, query refactoring, and more

• Maximize availability using replication, clustering, mirroring, and SQL Server's new online indexing

• Leverage important new improvements in Integration Services and Reporting Services

• Use Analysis Services to support enterprise-class Business Intelligence deployments

• Even set up a simple configuration of SQL Server 2005 for freelegally!

Next Page


Table of Contents

Previous Page
Next Page

Book Cover
Administrator's Guide to SQL Server 2005
By Buck Woody
...............................................
Publisher: Addison Wesley Professional
Pub Date: June 08, 2006
Print ISBN-10: 0-321-39797-5
Print ISBN-13: 978-0-321-39797-3
Pages: 648

Table of Contents | Index

Copyright
Praise for Administrator's Guide to SQL Server 2005
Microsoft Windows Server System Series
Acknowledgments
About the Author
Preface
Foreword
Chapter 1. Installation and Configuration
 | System Sizing
 | System Use
 | SQL Server Editions
 | Architectures
 | Landscapes
 | Hardware Configuration
 | Installation Process
 | Take Away
Chapter 2. Architecture and Tools
 | System Architecture
 | Tools
 | Take Away
Chapter 3. Maintenance and Automation
 | Maintenance
 | Restore
 | Consistency Checks and Shrinking the Database
 | Index Fragmentation
 | Automation
 | Take Away
Chapter 4. Security
 | Security and the Data Path
 | Platform Security
 | User Security (Principals)
 | Object Security (Securables)
 | Application-Based Security
 | Encryption, Certificates, and Key Management
 | Take Away
Chapter 5. Monitoring and Optimization
 | Monitoring Your System
 | System Optimizations
 | Take Away
Chapter 6. High Availability
 | Near-Real-Time Availability Mechanisms
 | Real-Time Availability Mechanisms
 | Take Away
Chapter 7. Notification Services and the Service Broker
 | Notification Services
 | Service Broker
 | Take Away
Chapter 8. Integration Services
 | Integration Services Architecture
 | Security
 | Monitoring and Performance Tuning
 | Take Away
Chapter 9. Reporting Services
 | Reporting Services Architecture
 | Report Design and Creation
 | Reporting System Management
 | Reporting Security
 | Reporting Delivery System
 | Monitoring and Performance Tuning
 | Take Away
Chapter 10. Analysis Services
 | Business Intelligence Scenarios
 | Implementing Business Intelligence in SQL Server 2005
 | Analysis Services Maintenance
 | Take Away
Appendix A. Upgrading to SQL Server 2005
 | General Guidelines
 | Upgrade Strategies
 | Post-Upgrade Steps
Index

Previous Page


Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales

(800) 382-3419

corpsales@pearsontechgroup.com

For sales outside the U.S., please contact:

International Sales

international@pearsoned.com

Library of Congress Cataloging-in-Publication Data
Woody, Buck.
Administrator's guide to SQL server 2005 / Buck Woody.
p. cm.
ISBN 0-321-39797-5 (pbk. : alk. paper) 1. Client/server computing. 2. SQL server. I. Title.

QA76.9.C55W664 2006
005.2'768dc22
2006007818

Copyright © 2006 Pearson Education, Inc.

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:

Pearson Education, Inc.

Rights and Contracts Department

One Lake Street

Upper Saddle River, NJ 07458

Text printed in the United States on recycled paper at R.R. Donnelley in Crawfordsville, Indiana. First printing, June 2006

Dedication

For Marjorie, my wife and my best friend. Your love and patience amazes me every day.


Praise for Administrator's Guide to SQL Server 2005

"Buck Woody's new book will help new and experienced users alike in getting productive with SQL Server 2005 quickly. His special attention to security and performance make this an exceptional reference that is sure to remain relevant for some time."

Chip Andrews, CISSP, MCDBA,
Director of Research and Development Special Ops Security, Inc.

"SQL 2005 is the most exciting upgrade to come out of Redmond in years; this book will help you get up and running quickly to take full advantage of it! If you are looking for a book about upgrading to SQL 2005, look no further, this one is all you need. It gives you all you need to take advantage of everything SQL 2005 has to offer, and that is a lot!"

Doug Ellis, Area Technical Manager, Cognos

"Buck Woody provides insight into administering SQL Server 2005 that all developers and database administrators will find useful; the book assumes very little knowledge by providing 'DBA 101' sections which will not alienate the experienced reader while providing the necessary introduction to topics for those less experienced."

Doug Holland, Visual C# MVP, Precision Objects

"SQL Server 2005 is a landmark release and this title does an awesome job of hitting everything new in SQL 2005. It covers new and improved administration tools, and contains an overview of Notification Services, SQL Server Service Broker, Reporting Services, Integration Services, and Analysis Services. As a result, you'll be able to work on your own SQL Server 2005 solutions quickly and painlessly. Ultimately, this book provides you with a starting point, a roadmap, and then the tools and knowledge to exploit this powerful new database server to its fullest power. So, start your computer and get ready to learn SQL Server 2005!"

Alexzander N. Nepomnjashiy, SQL Server DBA,
NeoSystems NorthWest

"A solid introduction to administering SQL 2005. The 'DBA 101' sections provide useful tips for the new DBA, good review for the experienced DBA, and the 101 label enables the seasoned DBA to skip basic material and jump ahead to where the REAL action is…. Buck's conversational writing style makes this an enjoyable and refreshing contrast to the majority of tech books written these days."

Brad Russell, MCDBA, Visionary Integration Professionals


Microsoft Windows Server System Series

Books in the Microsoft Windows Server System Series are written and reviewed by the world's leading technical authorities on Microsoft Windows technologies, including principal members of Microsoft's Windows and Server Development Teams. The goal of the series is to provide reliable information that enables administrators, developers, and IT professionals to architect, build, deploy, and manage solutions using the Microsoft Windows Server System. The contents and code of each book are tested against, and comply with, commercially available code. This series should be an invaluable resource for any IT professional or student working in today's Windows environment.

Titles in the Series

Paul Bertucci, Microsoft SQL Server High Availability, 0-672-32625-6 (Sams)

Peter Blackburn and William R. Vaughn, Hitchhiker's Guide to SQL Server 2000 Reporting Services, 0-321-26828-8 (Addison-Wesley)

William Boswell, Learning Exchange Server 2003, 0-321-22874-X (Addison-Wesley)

Roberta Bragg, Windows Server 2003 Security, 0-321-30501-9 (Addison-Wesley)

Eric L. Brown, SQL Server 2005 Distilled, 0-321-34979-2 (Addison-Wesley)

Bill English, Olga Londer, Shawn Shell, Todd Bleeker, and Stephen Cawood, Microsoft Content Management Server 2002: A Complete Guide, 0-321-19444-6 (Addison-Wesley)

Don Jones, Managing Windows® with VBScript and WMI, 0-321-21334-3 (Addison-Wesley)

Sakari Kouti and Mika Seitsonen, Inside Active Directory, Second Edition: A System Administrator's Guide, 0-321-22848-0 (Addison-Wesley)

Jason Nadrowski and Stacy Draper, SharePoint 2003 Advanced Concepts, 0-321-33661-5 (Addison-Wesley)

Shyam Pather, Microsoft SQL Server 2000 Notification Services, 0-672-32664-7 (Sams)

Jeffrey R. Shapiro and Marcin Policht, Building High Availability Windows Server™ 2003 Solutions, 0-321-22878-2 (Addison-Wesley)

For more information please go to www.awprofessional.com/msserverseries


Acknowledgments

So many people helped to create this book. I want to acknowledge all the friends and co-workers who helped me with corrections and comments as they reviewed my rough drafts. The technical editors caught so many little things, making this book just that much betterthank you!

I want to thank Tom Rizzo, who took time out of his busy schedule at Microsoft to help me with a few questions here and there (and with many of the graphics). You will find some in-depth flow diagrams in this book that just would not have been possible without him.

Thanks also to Jana Jones, who is a joy to work with as an editor. And special thanks to J. C. Armand from Microsoft, who put a lot of time and effort into steering me the right way with many crucial concepts and details. His help was invaluable.


About the Author

Buck Woody has been working with technology since he built his first computer (with 1 whole kilobyte of RAM) in 1980. He has worked every position from the help desk to software design, with stops along the way as an IT manager here and there. He has been a database administrator since 1995, when he managed an Oracle system on an HP 9000. Buck also served for five years as the president of the Tampa SQL Server User Group (TSUG). He is a Microsoft MVP (2006, SQL Server) and holds the MSCE and MCDBA certifications. He currently works at Jabil Circuit as a database architect and hosts a weekly "Guide" for SQL Server at InformIT.com.


Preface

SQL Server 2005 represents one of the most sweeping changes to the platform since Microsoft rewrote the engine from Sybase long ago. As with most Microsoft products, first attempts often turn into market standardsand that is definitely the case with this release.

So many things have changed from previous versions. If you just dive into the product after having used SQL Server for years, you will be able to find your way around, but you will miss the true power of new paradigms such as schema abstraction, nonmodal windows, a full management studio, and more. If you are completely new to the database administration world, you have a lot to learn.

This book is designed to aid the technical professional tasked with managing a SQL Server 2005 system. I have divided the chapters into concepts starting at the beginning (installation and configuration) all the way through to using the platform as a full-scale business intelligence landscape. In this version, Microsoft has it all.

If you need to learn to manage a system quickly, the chapters are laid out in a chronological order of what you need to understand first. I have included not only the technical information you need to set up and maintain a server (or set of servers), but also years of tricks and tips learned at the helm of several database environments. Using this information, you will be able to avoid some of the mistakes that only tend to show up after the system is in full use.

If you are already familiar with SQL Server in previous versions but need to come up to speed on the features and new paradigms, I have covered everything from Integration Services and business intelligence to all the things you need to know about upgrading your current servers.

Although the book is arranged chronologically, I have encapsulated each chapter as much as possible so that you can quickly locate what you are interested in. I present the material in a concept-to-practical use format, and you can follow along on a test system. I even show you how to set up a simple configuration of SQL Server 2005 for free (and legally)!

At the end of the day, database administrators are rated on what value they add to the organization. At the end of each chapter, I have included a section called "Take Away" that provides practical applications and examples from the chapter.

I hope you have as much fun reading this book as I had writing it. I kept you in mind the whole time.

Buck


Foreword

It was back in 2000 when I first met Buck. He was president of Tampa SQL Server User Group, book author, and a site personality on InformIT.com. Buck continues to be an exceptionally dedicated and passionate contributor to the SQL Server community. Needless to say, I was quite pleased when Buck informed me of his plans to write such an important book. I knew that his knowledge and enthusiasm would make this project a great success. Therefore, it is with great pleasure that I write the foreword for this book, Administrator's Guide to Microsoft SQL Server 2005, by Greg "Buck" Woody, and congratulate him on this impressive accomplishment.

Since the release of Microsoft SQL Server 2000, the SQL Server team has been hard at work on a number of key additional features for SQL Server 2000, as well as on SQL Server 2005. In between the two major releases, the team released a Web Services Toolkit, an Accelerator for Business Intelligence, SQL Server Notification Services, SQL Server CE 2.0, a 64-bit Itanium version of SQL Server 2000, and finally SQL Server Reporting Services.

Now with the release of SQL Server 2005, Microsoft has further expanded the functionality and range of the SQL Server product family. This release has key new features in the areas of high availability, such as Database Mirroring and Online Indexing. Programmability has been enhanced with the integration of the .NET CLR and the introduction of SQL Server Service Broker. Improvements were made in the areas of business intelligence by the complete rewrite of SQL Server Integration Services and enhancements of SQL Server Analysis Services. In addition, we introduced support for the x64 platform to the existing 32-bit and Itanium platforms. SQL Server is now truly capable of running on the world's most challenging applications.

The goal of this book is to provide database administrators and their managers with the understanding and scenarios for leveraging these new technologies, along with the best practices necessary to manage SQL Server 2005 applications.

This book begins from a data architect's perspective of the product. It then covers the decision points by technology and provides guidance on how to implement SQL Server 2005.

This version of the product was a labor of love for dedicated professionals that make up the SQL Server product team. We would like to thank Buck for his support in writing this book. I sincerely hope that you enjoy this book as much as you will enjoy using SQL Server 2005.

Jean-Claude Armand

Senior SQL Server Technology Specialist

Microsoft Corporation


Chapter 1. Installation and Configuration

Installing software is not a difficult task. Regardless of the platform, most modern software has an installation program, so simply performing a "Next, Next, Finish" procedure typically gets you up and running. The same holds true for SQL Server. For an enterprise production system, however, that process is not enough.

SQL Server 2005 represents the latest Relational Database Management software offering from Microsoft. This version has a wide array of features, capabilities, and functions. Creating the proper environment for your installation involves making the right choices even before you install the software. It is important to understand all the variables so that you create a system that will function, and function well, for the long term as well as the short term.

You need to gather a lot of information before you buy any software or hardware for your environment and it is often difficult to know where to start. The best process is to understand the variables involved in your decision, and then match the information from possible configurations that meet your needs. Using this method, you can avoid a guessing game that can end up costing you a great deal of money and time.

I explain this process starting in the "System Sizing" section. The "Take Away" section at the end of this chapter provides a few spreadsheets that you can use to document the variables and thus make intelligent decisions when deploying SQL Server in your environment.

Note that this chapter deals with planning a new installation of SQL Server 2005. Even if you have a previous version of SQL Server already installed, you should read on; however, this chapter does not cover the upgrade process. Appendix A, "Upgrading to SQL Server 2005," explains the upgrade process in detail.


System Sizing

System sizing is an exercise often done by budget. A "standard" server is selected; the licensing costs for the operating system, SQL Server edition, backup software, and other utility software is tallied; and then some additional storage and memory costs are added. Then various committees whittle down that number and request bids.

After the parts arrive, the systems team assembles and configures the hardware and installs the operating system, sometimes even creating and assigning the storageall without consulting the users or administrators of the system. When that is complete, the database administrators (DBAs) install the SQL Server software and the system is off and running.

Although this is a valid method, it has some inherent problems. The variables involved in the use of the system are not known, so they are not considered during the build and configuration of the hardware and software. After the system has run for a while, these design flaws begin to show; by that time, however, changes are difficult (if not impossible) to make. What happens next is an evaluation that should have been performed at the beginning, with the dawning realization that your current configuration just cannot get there from here.

A better way exists to determine the best configuration for your server, storage, and other components. You can create a set of checklists that detail the use of the server, the SQL Server 2005 editions, and the hardware components that work best for each. You can then document the environment and combine this information into three choices. The three choices represent standard, enhanced, and optimal solutions, each incrementing in price. The important part is to use the information you have gathered to explain what the three choices represent in terms of costs and benefits.


System Use

To begin, you need to understand how your enterprise will use SQL Server. For instance, using the system as a Web-based back end entails a different configuration than when using it in a two-tier environment.

If SQL Server will be used to house databases for multiple applications, you must consider all the functions it performs before you configure the system. Record everything you can think of that the system will be asked to do, and then ask the requirements team the same questions. If the information is not readily available, you must do some research on the Web to locate all the information you can about the products that will use the server to store data.

Contact the vendors of the products that will be used on the server and ask them to provide application-load specifications. Also ask for client references. Often vendors supply white papers and sizing tools that you can use as a guide.

If the software that will use SQL Server 2005 is going to be designed in-house, talk to the developers to get a feel for the application's profile. Get the functional and technical specifications and read them thoroughly. With any luck, the developers will have a testing environment that they are using from which you can obtain some hard metrics. The key is to gather as much information as possible regarding how the users will use the system.

Use this information to document the software that will access the server, including the user count, and categorize the users to estimate the load each application creates on the database, reporting, or analytical engines.


SQL Server Editions

After you have determined what your system is used for, you can begin to select the SQL Server edition and configuration to meet those needs.

Several releases of the SQL Server software, called editions, are available to meet different needs. Each edition has different features and costs. Some run on all current Microsoft operating systems; others require a particular platform. Microsoft may make changes to some editions with service packs. These changes may add functionality from the higher versions to a lower one. Make sure you check the release notes for the service packs before you make your final edition choice.

In addition to the production editions of SQL Server 2005, there is a Microsoft Developer Network (MSDN) Edition (also called the Developer Edition), which has the same features as the highest version of SQL Server. The difference in the MSDN Edition of SQL Server 2005 is the licensing. The Developer Edition is not licensed for production systems.

I do not cover the laundry list of features in SQL Server 2005 here because doing so would take away from the bigger picture of this overview. Instead, I cover the feature sets as I explain the various technologies in the products required to implement solutions throughout this book.

Mobile

SQL Server 2005 Mobile Edition is used on PocketPC and Smart Phones. You can write applications that use this back end directly, but the more common use is to participate in a replication scheme with a server. Replication schemes allow consultants, salespeople, doctors, and so forth to use applications that store a small slice of data and manipulate it in the field. It is limited due to the small form factor it lives on but does include much of the standard language SQL Server uses, called Transact SQL (T-SQL).

To run the Mobile Edition, your machine must meet at least these specifications:

  • Microsoft Windows CE 5.0, Microsoft Windows XP Tablet PC Edition, Microsoft Mobile Pocket PC 2003, Microsoft Mobile Version 5.0 Pocket PC, or Microsoft Mobile Version 5.0 Smart Phone

  • Between 1MB and 3MB of available storage space, depending on processor type and components installed

Express

The Express Edition represents Microsoft's answer to free database offerings such as MySQL and replaces a previous version called MSDE. Microsoft made this edition much more capable than the older version, increasing the size of available databases and usable RAM. It can participate in replication, serve as a witness machine in clustering (more on that later), contains the Common Language Runtime (CLR) enhancement, integrates with Visual Studio, and has the same database engine that the other editions have.

As far as what this edition is capable of, it is easier to talk about what it cannot do. Its main limitations are that it can use only 1GB of RAM and 4GB of disk space (per database) and that it runs on one 32-bit CPU (or in Windows on Windows [WOW] mode on x64 processors).

To run the Express Edition, your machine must meet at least these specifications:

  • Intel Pentium III or compatible 600MHz or higher processor (1 gigahertz or higher recommended)

  • 200MB of drive space

  • 128MB of RAM (512MB or more recommended)

You can run the Express Edition on the following operating systems (and higher):

  • Windows XP Home Edition

  • Windows Server 2003 Web Edition

  • Windows 2000 Professional

Workgroup

The Workgroup Edition is the next step up from the Express Edition. It has all the same features as Express and increases the available RAM to a maximum of 3GB, can run two CPUs, and has no database size limit. It also includes the full SQL Server Management Studio, the SQL Server Agent, and has Full-Text support. This edition also has all the security features and programmability found in the larger editions. In the integration and business intelligence arena, this edition lacks about half of the features of Standard or Enterprise, such as Oracle integration, HTTP endpoints, Analysis Services, and Data Warehousing. It is most useful for your local machine or for development tasks. It is also available for 32-bit architectures.

To run the Workgroup Edition, your machine must meet at least these specifications:

  • Intel Pentium III or compatible 600MHz or higher processor (1 gigahertz or higher recommended)

  • 300400MB of drive space to start

  • 128MB of RAM (512MB or more recommended)

You can run the Express Edition on the following operating systems (and higher):

  • Windows XP Professional

  • Windows XP Media Edition

  • Windows XP Tablet Edition

  • Windows 2000 Professional

Standard

Most department-sized shops will run the Standard Edition of SQL Server 2005. It can run up to four CPUs, has no memory limitations other than those imposed by the operating system, and supports 64-bit architectures. It has a litany of features, including the following:

Database Mirroring | Common Language Runtime and .NET Integration
Failover Clustering | User-Defined Types
Backup Log-Shipping | Native XML
Online System Changes | Notification Services
XQuery | Service Broker
Import/Export | Merge Replication
Web Services (HTTP Endpoints) | Transactional Replication
Express Manager | Auto Tuning
Management Studio | Database Tuning Advisor
Serviceability Enhancements | Full Text Search
SQL Agent Job Scheduling Service | Report Server
Advanced Auditing, Authentication, and Authorization | Report Builder
Data Encryption and Key Management | Reporting Data Sources
Best Practices Analyzer | Integration with Microsoft Baseline Security Analyzer
Stored Procedures, Triggers, and Views | Data Warehousing
T-SQL Enhancements | Star Query Optimization
SQL Analytical Functions | Integration with Microsoft Update
BI Development Studio | Enterprise Management Tools
Analysis Services | Unified Dimensional Model
Business Analytics | Data Mining
Native Support for Web Services (Service-Oriented Architectures) |

To run the Standard Edition, your machine must meet at least these specifications:

  • Intel Pentium III or compatible 600MHz or higher processor (1 gigahertz or higher recommended)

  • 300400MB of drive space to start

  • 128MB of RAM (512MB or more recommended)

You can run the Standard Edition on the following operating systems (and higher):

  • Windows Server 2003, Standard Edition

  • Windows Server 2003, Enterprise Edition

  • Windows Server 2003, Datacenter Edition

  • Windows Small Business Server 2003, Standard Edition

  • Windows Small Business Server 2003, Premium Edition

  • Windows 2000 Server

  • Windows 2000 Advanced Server

  • Windows 2000 Datacenter Server

Enterprise

The Enterprise Edition adds high-end features such as Online Indexing, Online Page and File Restore, Fast Redo, Integration Services Advanced Transforms, Oracle Replication, Scale Out Report Servers, full Data Mining, and greater text-search capabilities. In short, if you get more than a thousand users or so, or if you have an especially heavy load, this is the edition you want to use.

To run the Enterprise Edition, your machine must meet at least these specifications:

  • Intel Pentium III or compatible 600MHz or higher processor (1 gigahertz or higher recommended)

  • 300400MB of drive space to start

  • 128MB of RAM (512MB or more recommended)

You can run the Enterprise Edition on the following operating systems (and higher):

  • Windows Server 2003, Standard Edition

  • Windows Server 2003, Enterprise Edition

  • Windows Server 2003, Datacenter Edition

  • Windows Small Business Server 2003, Standard Edition

  • Windows Small Business Server 2003, Premium Edition

  • Windows 2000 Server

  • Windows 2000 Advanced Server

  • Windows 2000 Datacenter Server

Additional Requirements

In addition to the SQL Server software, you must install a few other software components (on editions other than Mobile), such as the following:

  • TCP/IP

  • Microsoft Windows Installer 2.0 (SQL Server will install if needed)

  • Microsoft Internet Explorer 6.0 Service Pack 1 or higher

  • Internet Information Services 5.0 or higher (for Reporting Services)

  • Microsoft .NET Framework 2.0 (SQL Server will install if needed)

You also need to meet certain service pack levels, depending on the operating system you are using. The installation program detects these requirements for you and prompts you to install them if you do not have them.

You can install SQL Server 2005 with SQL Server 2000 if you want, or it will automatically upgrade version 2000 to 2005. I explain the upgrade process in Appendix A.


Architectures

Multiple programming methods are in use (what I call architectures), and new paradigms crop up all the time. Understanding the architecture your system uses is a vital part of your planning strategy, because the architecture determines, to a large degree, the software and hardware you need to run it.

When you ask IT management, developers, and users about the architecture of the software they are planning to purchase or develop, you will often face a little resistance in the answer. This resistance is normal, because you are making the decision to implement a more complex, and possibly even more expensive, solution. You are also giving other people work to do in gathering that information. Therefore, you must explain why you are asking the questions, and it is also helpful to explain that you will be presenting three options that have three different sets of costs and benefits. That usually helps lower the resistance, by showing everyone involved that you want to make the right decision and that you are sensitive to cost.

Whether you face resistance or not, it is important to get as clear a picture as possible before you build your environment, and that is where understanding the various application architectures comes into play. Do not be surprised if the answers you get involve multiple architectures. Treat them separately, and then determine whether they can be combined onto one or more servers.

At the risk of generalizing, the various parts of an application can be grouped into a presentation layer, composed of a user interface and its logic; a business logic layer, where the business rules are enforced; and a data layer, which controls the movement, storage, and maintenance of the data that the other two layers generate. You can generally categorize the architecture of an application by where it puts these three parts.

Monolithic

The simplest architecture is monolithic. In this architecture, a single computer runs all three layers of the application. This architecture is common on handheld devices, where the Mobile Edition of SQL Server 2005 is used in replication.

You will also see this architecture used in disconnected-client environments, such as a salesperson's laptop. This type of architecture is well suited for the Express Edition of SQL Server 2005. In a monolithic architecture, there is usually only a single user of the application at any one time.

Another example of the monolithic architecture is a server-based application, such as a manufacturing-monitoring or data-collection application. These applications have no human interaction and are so intense that the application needs direct access to the database. Depending on the load, you can use Express or the Workgroup Edition, unless you require the feature set found in the higher editions.

You can also find a monolithic architecture in a small Web-based application. Even though the user is located at the end of a browser, the actual client software in this arrangement is coded to run on the server. Depending on the load, you can use Express or the Workgroup Edition for this type of application, too.

Client/Server

The client/server architecture separates the system, grouping the presentation and business logic layers onto a client program using a high-level programming language and the data layer on the server. You might also see the presentation layer coded on the client and the business logic and data layers coded using stored procedures on the server.

The client/server architecture was the most widely used type of architecture in the 1980s and is still in wide use today. In this architecture, an application makes direct calls to the database and often stays connected through the entire session. In the past, this caused quite a few performance issues, but modern coding practices and libraries are allowing a resurgence in this method of programming. This architecture has a few implicit advantages, such as ease of coding and security.

The editions that are useful for this architecture include Express through Standard, depending on the number of users and the size of the database.

Three-Tier

Widely used in the late 1980s through the mid-1990s, the three-tier architecture separated all three layers onto different systems. In this architecture, the presentation layer is separated out to a program that runs on the client machine, and the business logic runs on what is called the "middle-tier," which itself might be composed of more than one server. The middle-tier systems make requests from the data tier on another server using Transact-SQL and stored procedure calls, often combining and breaking apart the requests to service them quickly and efficiently.

The defining characteristic of this type of architecture is that the business logic and data layers might run on any number of servers. You might have one business logic machine talking to multiple databases, even multiple types of databases, or you might have multiple middle-tier machines talking to one database server. You might also have multiple servers of both types in the mix.

Although it is possible to use the Express or Workstation Editions in this architecture, it is more common to see the Standard or Enterprise Editions used. It is also common to see replication and clustering in this architecture, which mandates multiple SQL Servers.

Distributed Computing

The distributed computing architecture is one of the most difficult to prepare for. The reason for this is that distributed computing involves multiple resources.

There are actually two definitions for a distributed computing architecture. The first is tightly coupled, in which specific programs run on defined systems, communicating closely with each other, often assisted by operating system calls. The second is called loosely coupled, where programs communicate with each other periodically using standards such as Extensible Markup Language (XML). Loosely coupled systems often do not rely on an identical programming language or in some cases even an identical operating system or network. The World Wide Web is an example of a loosely coupled architecture.

Relational databases are most often the base storage components of these systems, and they often use multiple database platforms. For this type of environment, you need to break out each component that requires storage and decide what that part needs. You might even be able to use the Express Edition in some locations and Standard for others, depending on the activity and storage levels and features needed.

Service-Oriented Architecture

A service-oriented architecture does not fit within the previous architecture definitions, at least in that it does not provide a user interface layer. It is a set of service programs that reside on various systems that publish their capabilities on the network. Clients (which might actually be other servers) request responses from these services when they need them.

Services can interoperate but are normally kept stateless and independent. This allows the solution to be based on "best-of-breed" platforms rather than a single vendor attempting to provide an end-to-end solution. Often, you can take an older program and retrofit it to participate in a service-oriented architecture.

An example of this type of setup is a portal-driven enterprise. The portal usually has sections that are tailored to each user or classification of users. These "Web parts" might display manufacturing information, inventory or sales data, or even financial numbers. Although that kind of information can come from a single source, in the service-oriented architecture, several servers publish the type of information they can provide, and the portal, serving as the presentation layer, displays the proper data on demand. Nothing other than the primary interface code is actually stored on the portal server.

SQL Server 2005 is well positioned to provide a service-oriented architecture. In many cases, the service-oriented architecture requires a high level of features from the database platform, unless another server provides the data. For this reason, you will normally use at least the Standard Edition.


Landscapes

The landscape of a computer system refers to environmental factors, such as language and locale settings, the network the system uses, the number of users, and the number and type of servers. Because you might install SQL Server systems into multiple environments, treat them as separate exercises and then see where the systems might benefit from overlap or consolidation.

In many organizations, there are multiple landscapes, and sometimes they fit in-between categories. The purpose of getting landscape information is more concerned with the network, layout, budget, and size of the systems to help explain the level of user activity. The number of users in a landscape is less important than what activity they cause.

It is important to keep in mind that five heavy-activity users can generate more traffic than dozens of Web users. For that reason, the categories discussed here are more of a guide than an absolute part of your overall formula.

Farm

The simplest type of environment is the farm, which treats each application environment separately. You will see this quite often in Web hosting sites, where security, settings, and possibly even patch levels need to be kept independent based on the particular requirements for the application. Most of the time, the only thing these landscapes share is security information. You can think of even a single installation as a simple farm landscape.

The number of users in a farm for a given database can range from the single digits to several hundred logins, but normally have a small impact, generating no more than 50K per transaction.

You have several configuration options in this area. You can have an independent installation of SQL Server, often the Workgroup or Express Edition, to suit the needs of the application. Another option is to install a more powerful edition and set up instances for each environment. An instance is an additional installation of SQL Server on the same system and allows for a completely separate environment for security and configuration. The instances are named separately but are operated on the same hardware. The system shares much of the program files (also called binaries) but not the database and configuration files.

One advantage of using an instance is its ease of maintenance. You have less hardware to track because SQL Server will control the management of the instances from a single point of view (instead of having multiple copies of the software). The other advantage is licensing. You do not have to maintain a separate license for multiple instances; you do, however, if you install multiple copies of a licensed edition on separate hardware.

The most important thing to keep in mind when using instances is that you need to ensure that they are still considered independently from a hardware standpoint, so that you provide enough resources on the server to handle them all.

Department

The most common landscape most DBAs work with is the department. In this configuration, you move out of the Express Edition as your bottom line into at least the Workgroup Edition. More often than not, you will want to set the baseline for this level of landscape at the Standard Edition, especially if your application requires more features than the smaller editions provide. It is also at this level that you need to investigate using multiple processors on your server hardware. The licensing costs are higher when you get into multiple processors, but you need to evaluate how CPU heavy the application is and plan accordingly.

As of this writing, Microsoft does not charge more for hyper-threaded or dual-core processors.

The number of users in a department ranges from ten to fewer than a hundred, and users generate between 20K and 500K of data per transaction. They generate less if a middle tier is used and more if the application is designed in the client/server architecture.

In the department landscape, you want at least the Workgroup Edition for most applications. For larger footprints, use the Standard Edition. In this area, you will begin to consider options such as replication, perhaps with the organization or enterprise servers. At this level, you also have higher reporting loads, so ensure the hardware you are using can handle the additional traffic.

Organization

Depending on the size of the organization, you might group several departments' needs into a single landscape. In this configuration, consider using clustering technology, which enables you to group two or more SQL Servers for backup. I cover clustering in depth later in the book.

Another factor that comes into play in an organization landscape is that you will probably need at least one additional server for the reporting system. The reporting system can also be federated into multiple servers, but this is not always necessary unless the reporting load is heavy.

The number of users in an organization ranges from the low hundreds to fewer than a couple of thousand, and users generate between 20K to 500K of data per transaction, less if a middle tier is used and more if the application is designed in the client/server architecture.

The organization landscape often involves multiple facilities. Having more locations affects SQL Server in considerations regarding replication, failover, and multiple-use setups.

In an organization landscape, you definitely want to start with Standard Edition, and the reporting instance normally ranges from the Express Edition for small reporting loads. If you require Reporting Services (which comes bundled with SQL Server), you need the Standard Edition for that server, too. When you break out Reporting Services, you need to factor in Web traffic patterns found in the distributed architecture type (in addition to whatever other architectures you have).

Enterprise

The enterprise landscape involves one or more database back ends that supply larger, integrated applications' data. You will often see SQL Server servicing an SAP or Siebel ERP application. Enterprise databases are often located across a large geographical area, sometimes even across the globe. When I worked at the Space Center in Florida, our network included two ships at sea that relayed information when they picked up the booster rockets from the shuttle. It took quite a bit of planning to engineer that architecture.

Using the Unicode page set becomes a consideration in an enterprise, because sites might be located in multiple countries. Unicode takes more space than a single collation set, so double the space needs when you plan.

DBA 101

Unicode is a way of increasing the bits used to store a single character so that it can accept more than just English (or most any Latin-based language). It increases the size of your databases but allows other language characters to be stored in your database.

Unicode does not translate anything at allit just provides more room for different characters.

Because of its size, many teams in an enterprise are responsible for various regions, sites, or parts of a system, and you must consider standardization as one of your primary duties at headquarters. Standardization includes configuration, patch procedures, and monitoring. Enterprise systems also change more slowly than smaller systems and require an intense amount of coordination and planning. In this endeavor, ensure that formal system planning takes place.

In a large enterprise, or any system with high reporting activity, you need to add a separate reporting system, not just another server. You may also need to consider a reporting data server as part of that sub-landscape. The reporting data can be refreshed with many technologies, from database snapshots to full replication, and I cover those in Chapter 9, "Reporting Services." You can also use third-party solutions to create reporting databases. Either way, you will most likely need at least two more servers for the reporting function, one for data and another for report presentation.

An enterprise has thousands of users and potentially thousands of connections to any given application. Because of this mix of users, implementing and tracking security becomes more complex. You may have to implement SQL Server security in addition to Windows authentication so that you can allow clients, vendors, or outside partners to access data in the system. I cover security strategies in Chapter 4, "Security."

Depending on the application architecture, all users may need to access a central instance of the database. In this case, you need to consider 64-bit systems and clustering, so you will want to use the Enterprise Edition.

Another situation occurs when an application is broken into separate parts. Perhaps the application runs individually at each region or site. If your applications implement this method, you might need to use replication to consolidate the data. You will still want to consider 64 bit or clustering, because even separate installations will probably be large. Enterprise Edition is usually still the best edition to use.

With a picture of the application architecture in mind, you can measure the expected load the application creates. Load measures the number of users, but more important, it defines what effects they have on the hardware resources. For instance, in an enterprise accounting system, there are several types of users, and each type produces a different number of bytes for the operations they perform. They also perform those operations in differing frequencies. In short, it is not enough to know how many users there are to understand the load on a system. The vendor or developer should be able to help you understand these user profiles.


Hardware Configuration

The minimums for hardware configurations from Microsoft shown earlier are just that. Any system configured at those values will not run the software quickly and certainly will not support any serious workload. To determine the proper configuration for a given application, you must understand how each component within a server affects SQL Server's performance.

CPU

The central processing unit is one of the single most important components for SQL Server performance. The factors that affect CPU choice are speed, its cache, the number of processors, and the architecture of the processors, such as 32 or 64 bit.

Obviously, the faster the processor, the faster instructions pass through the system. The number of processors, however, can outweigh the raw speed. For instance, a fast single (non-dual-core) processor might actually run a given query more slowly than two slower processors. This is true because of the ability of SQL Server to parallelize, or spread out a query across multiple processors.

Hyper-threaded processors also help speed queries by doubling certain parts of the CPU architecture within a single chip, allowing multiple threads to operate on one CPU. This gives the appearance to the software of having multiple processors. Dual-core processors take this architecture a step further by doubling an entire processor on a single chip, and in some configurations, it is also used to provide 32- and 64-bit concurrent operations.

Windows 2000 handles hyper-threading differently than Windows 2003. In Windows 2000, a thread is placed on a processor based on its logical processor count, not by physical processor. That means that a thread might be scheduled to run on physical processor one and then logical processor two, which may be on the same physical chip. The second physical processor, with its own cache and pipeline, would go underutilized in this situation. Windows 2003 schedules threads by physical processor first and then logical processors second.

Another important factor is the second level (L2) cache memory on the CPU. More is better. A slower CPU with a higher L2 cache can outperform a faster one with a lower L2 cache amount.

The difference between 32- and 64-bit processors is startling. A 64-bit processor is not just double the size of a 32-bit CPU; 64-bit architectures allow an amazing increase in address space for instructions and memory. For serious enterprise configurations, consider 64-bit processors.

There are two types of 64-bit processors. The first is called x64, which means that it is completely backward compatible with 32-bit software. The processor will run 64-bit code, but will also run 32-bit software if that is what is installed. Some tests show that 32-bit software actually runs faster on an x64 processor.

The second type of 64-bit processor is an entirely new architecture. The Intel version of this chip is called the Itanium or IA64, and it runs only 64-bit software compiled to work on it. SQL Server is currently designed to work on IA64 as well as x64 architectures.

Memory

Random access memory (RAM) is the second most important factor in how quickly a SQL Server responds to queries. Several buffers inside SQL Server use memory to cache query plans and procedures. If memory is constrained, the server has to cycle recent plans and procedures down to the I/O subsystem, which is several orders of magnitude slower than the circuitry in memory.

The recommendations for operating SQL Server for any production application should start at 2 gigabytes and progress from there. SQL Server automatically allocates memory among its various components, so no tuning for locks and caching is needed. By default, the system takes a range of memory from a lower boundary to a maximum limit and gives this memory back to the operating system as needed. It is much more efficient to keep all other applications away from the SQL Server and set the memory to a fixed amount, leaving at least 512MB for the operating system. This setup keeps SQL Server from having to manage the memory, which saves a few CPU cycles. I explain more about memory configurations in Chapter 5, "Monitoring and Optimization."

I/O

With the I/O subsystem, speed is the primary factor, followed closely by the interface type. For local systems, IDE or on-board SATA are adequate; in production configurations, however, the system should use an SCSI interface. SCSI has a wider bus, so it can transfer more data at a single pass.

The best local technology is a redundant array of inexpensive disks (RAID). This is a scheme of a special controller hooked up to one or more drives. There are several levels of RAID, all having to do with how the data is stored on more than one drive at a time.

DBA 101

RAID level 0 stripes all the data across several drives (at least two) and is extremely fast. But because the entire data set is dependent on all the drives working properly, lose one and you lose all the data. For this reason, RAID 0 is not appropriate for SQL Server files.

The first real choice for SQL Server is RAID 1, where all data is duplicated onto a "mirror" set of drives. SQL Server 2005 log files (more on those in the next chapter) are best stored on this level of RAID, because RAID 1 works well with sequential reads and writes. For this setup, you need an even number of drives.

The next level normally seen in a SQL Server is RAID 5, where the data is "striped" across multiple drives (at least three) and a parity bit is used to rebuild a single drive in case of failure. You can think of an equation, such as 3 + 2 = 5. If you were to lose any part of that equation, say the 3, you could rebuild the equation from the other two parts, like this: X + 2 = 5. The data, combined with the parity bit, works the same way. RAID level 5 is best used with SQL Server data files, because they are accessed randomly.

Although there are many other RAID levels (check out www.raid.com), there is most often only one other that you will see in SQL Server installations. It is actually a combination of other RAID levels and is called RAID 0+1. In this setup, the data is striped with no safety across several drives for speed, and then the entire set is duplicated for safety. You need at least four drives for this setup.

All RAID levels should be handled by hardware devices only, and it is best if you use multiple controllers for each set of drives. Software RAID setups are available, but they should never be implemented in a SQL Server environment, not even in testing or development. The reason is that the CPU and memory requirements to run the software RAID are so prohibitive that SQL Server is starved for resources.

In clustering or in larger configurations where greater speed and safety is a necessity, the SQL Server is normally connected to a storage area network (SAN). In a SAN configuration, a special I/O adapter called a host bus adaptor (HBA) is installed in the server, and the server is connected to an external storage array via a switched network, often using Fibre Channel technology.

DBA 101

Fibre Channel is often confused with fiber-optic interfaces, which it sometimes employs. The Fibre in this case refers to the interface as well as the network that it uses, called the fabric of the SAN. Fibre Channels can use fiber-optic setups and copper wiring, too.

In Fibre Channel, an HBA connects to a Fibre switch, which in turn connects to storage arrays, which have enclosures, which contain physical drives. These drives are served up to the computer as logical unit numbers (LUNs). These LUNs form the lowest level of storage that the operating system can see, and the LUNs are then carved into partitions available for either mount points or drive letters on the operating system.

To optimize the SAN configuration, ensure that there are multiple paths to your storage arrays and balance the I/O where possible across LUNs, physical storage, and processors.

There is also network attached storage (NAS), which is a drive enclosure that is mounted externally on the network. If you are using gigabit speeds or fiber-optic technology, these are also fine for SQL Server installations. The danger here is that the drives in these systems are often used by other applications, and if they mix with the SQL Server data files, the system will take a performance impact.

Another popular technology for large-scale storage is serial attached storage (SAS). In this setup, the drives are stored remotely from the server. Having a networked version allows for robust disaster recovery scenarios.

One rule of thumb with I/O whether local storage or SANS are used is that more spindles are better. More spindles means that more heads are able to satisfy requests, so data is delivered quicker. With more spindles, a read operation is not blocked by a write operation, and queries do not block log updates. This depends on how you arrange the files that SQL Server uses, something I cover in more depth in Chapter 5. You can check with your hardware vendor to find out how many spindles their solution has.

To properly plan for the final drive configuration, create separate mount points or drive letters for the tempdb database files, data files, logs, and indexes. This allows sequential writes (the log files) to be separated from random reads (the data files) and indexing operations (the index files) to be separate from internal sorting and tracking (tempdb).

Even if you are using a SAN that hides the physical separation and controls the I/O, having separate drive letters allows you greater flexibility when creating filegroups. Filegroups allow you to perform backups, maintenance, and other tasks on individual chunks of the database. Having filegroups arranged this way also simplifies transitions to newer systems, which happens often in large companies.

The most often overlooked component of storage during the planning stage is the backup strategy. Most often, you will use a magnetic tape solution. Smaller setups can use DAT or DLT tapes, and larger organizations will require LTO-1, LTO-2, or LTO-3 tapes for larger storage. At the largest levels, you can begin to consider virtual tape drives, which SQL Server 2005 supports. I explain the backup and recovery strategies more completely in Chapter 3 "Maintenance and Automation," but you should plan for at least the capacity, recovery interval, and automation level that you want right at the outset.

Network

Although many administrators do not realize it, the network interface in a computer is often the slowest component in the system.

The network cards inside the server should be as fast as possible and, where applicable, set to full duplex (not to auto-negotiate). You should include a fast network card in the server even if the network topology is slower than the card in the server can perform. This lets you increase the speed of the network later without having to alter the server and allows the server to satisfy more requests at a time.

The server should have at least two network cards if it will participate in replication or you plan to locate it in a network demilitarized zone (DMZ). In addition, each network card should be connected directly to a different network switch. This provides redundancy and greater speed to handle multiple requests. In addition, a separate "backbone" or dedicated connection for the servers isolates traffic from the rest of the network and provides the best level of performance for both.


Installation Process

All Microsoft software includes a simple installation program. Gone are the days when the system administrator had to set aside an entire week to install and configure the software. What I cover here is not the entire installation process, but the overall decisions you have to make during the install.

DBA 101

You can install SQL Server multiple times, using a separate designator for each installation. Microsoft calls these multiple installations "named instances" or simply "instances." Actually, the entire software suite is not installed multiple times, just enough of the binaries to make each instance unique. This feature allows you to have multiple security and configuration options. You could keep one instance at Service Pack 1 and another at Service Pack 2, for instance. Instances also allow you to have separate administrators for the same physical hardware.

When you install SQL Server for the first time, you can choose to have a named instance or a "default" instance. Using the default instance, your applications will refer to the server name alone as SQL Server; when you use a named instance, the format to refer to the SQL Server is SERVERNAME\INSTANCENAME. So, assuming you have installed SQL Server 2005 on a server named SVR1, you refer to the SQL Server default instance as SVR1. If you install the software again, this time as a named instance called INST2, you refer to it as SVR1\INST2.

When you install another instance of SQL Server, make sure you account for the increase in resources necessary to run it. Do not spend the same hardware twice without realizing that you have not added enough I/O and memory to carry the increased load. Instances are a convenience, not a way to buy fewer servers for the same load.

After you have read this chapter and planned out your system landscape, you will begin the actual installation. When you begin the installation, the primary choices you make are drive layouts and service accounts.

Before you install SQL Server, make sure you have your operating system current with the latest patches and security fixes and have all the utilities you need. In previous versions of SQL Server, it was also useful to have a mail client installed, but that is no longer the case.

First, you need to decide where you want to keep the binaries of the program and where you want to store data files. The optimal solution is to keep at least these five things apart: the SQL Server program files, data files, log files, index files, and the tempdb database files.

Next, pick a Windows account to run the SQL Server services. This account does not need administrative privileges on the local server or domain or enterprise admin rights. The installation handles assigning the account the proper privileges.

At least two services will be installed. One service will operate the SQL Server engine, another will operate the SQL Server Agent, which automates tasks and provides alerts and notifications. Use a separate account for each function and ensure that each is a specific Windows user account for that service. Make sure you set the password to something complex and set it not to expire. Do not use those accounts for anything else.

You will get one set of services for each instance. Use a separate set of accounts for each instance. The reason is that certain stored procedures (Transact-SQL code that runs on the server) might be allowed to access the operating system. If these are enabled (not always a wise choice anyway), using the same account across all instances loses the ability to track who has made changes to the system.

As you can see in Figure 1-1, when the installation program starts, it first installs the Windows Installer 2.0, the .NET Framework 2.0, and several SQL Server Setup support files. When that completes, it runs a system consistency check, which is a series of checks to ensure there is enough drive space, that the operating system is correct, and other housekeeping. You can see the output on my test system in Figure 1-2.

Figure 1-1.

[View full size image]

Figure 1-2.

[View full size image]

After the check runs, the system presents you with a list of options for the feature set you are interested in.

Do not install everything just because you think you might need it. That takes up space and makes significant changes to the way your installation is automatically configured. You can always come back and add features to the software later using this same installation program. On the other hand, if you are installing a test system (not production!) now, you can select everything. If you do not, we will add it later when we come to that feature in the book. In Figure 1-3, I've selected everything on this test installation.

Figure 1-3.

[View full size image]

Next you are asked about installing the software as a default versus a named instance. You are also asked where the programs and data should live and which account to use to start all the services. Because you have planned your installation, you are better able to intelligently answer these questions.

Two options at the end of the process might give you pause. The first option involves sending error reports to Microsoft in the case of a problem. The second deals with the usage of the tools you are using in SQL Server. In Figure 1-4, I've made both of these selections.

Figure 1-4.

[View full size image]

Think long and hard about these choices. The first reaction most technical professionals have is to clear these choices, because they are afraid of a privacy compromise. If you feel uncomfortable with these options or your organization policy prohibits them being set, then by all means deselect them. If you are willing to tolerate a little Big Brother, however, the information Microsoft receives from these choices can go a really long way toward making SQL Server a better, more secure platform. As you will learn in Appendix A, these installation check boxes make everyone's upgrade a little better.

If you take the defaults during the installation, you will not have any of the sample databases and code. This is desirable on a production server, but you will want to install these on development systems. To do that, go through the Control Panel in Windows and then select Add or Remove Programs. Select the Microsoft SQL Server 2005 Tools item and then click the Change button.

I usually add all the sample databases and examples on a development system, which will install the AdventureWorks database and (if you pick everything as I do) the AdventureWorks OLAP database, too.

You might want to make other changes after the system is installed. To do that, you use the SQL Server Surface Area Configuration tool. I cover that tool in a few chapters throughout the book.

Operating System Variables

Depending on the version of Windows your server is running, you need to set various switches and operating system variables. Windows Server 2003 and higher provides significant performance and security advantages over the earlier versions of Windows operating systems.

Regardless of the operating system version, keep in mind a few base settings. For starters, ensure that the system runs only SQL Server. Unless the installation is quite small, such as an Express Edition installation, you do not want to put any other applications on the server other than management utilities and backup software. It is tempting to put lots of applications on the new, not-heavily-used-yet SQL Server, but it could eventually conflict with SQL Server, and at the very least it will cause performance issues. Leave the SQL Server box for SQL Server.

Carefully consider the drive layout. The binaries for the operating system and other software should get its own physical drive, preferably inside the server hardware itself. The paging file should go on its own physical drive, too, and both of these should be kept away from the data, indexes, and tempdb files used by SQL Server.

If you are running Windows 2000 Server Editions, you need to set a few switches and options to take advantage of higher RAM amounts. If you are using more than 3GB of memory on a 32-bit CPU, you may want to enable the /3GB switch in the BOOT.INI file of the operating system. If you are using more than 4GB of RAM, you may want to use the /PAE switch, too.

Several caveats apply to these switches. You can find these in Books Online under the topic of "Enabling Memory Support for Over 4 GB of Physical Memory." The various settings and processes to follow depend on the operating system, processor type (32 or 64 bit), and the specific edition you are using. The SQL Server Web site at Microsoft also has several white papers on this topic; because each service pack might change these settings, check those resources for the most up-to-date information.

Post-Installation Configuration

With the planning for the server landscape complete, the final step is to predict the growth of the system. There are lots of fancy formulas to help you with this, and most of them amount to nothing more than an educated guess.

The better option is to monitor the system when it is in place and then use those numbers to do predictive analysis in the future. I explain more about that process in a bit. For the short term, plan for the growth you expect and add 10 to 25 percent, whichever your budget will allow. Most important of all, make sure the system you purchase can accept more CPUs and memory and is generally expandable.

After installation, gather metrics as you use the system. I show you how to do that in Chapter 5. Evaluate the metrics at least monthly and provide feedback to the site IT managers or whoever makes budget decisions. Most managers do not mind spending a little overtime to keep the infrastructure current, as opposed to a huge bill to change out the entire system.

There are many other factors in this decision process, but these are the primary ones to keep in mind. The "Take Away" section that follows shows a more complete exercise.

I have explained the versions and their capabilities and detailed the hardware components. Now it is time to put it all together and build your environment. In the following section, I have developed some processes that will help you document your variables, create a decision matrix, and then create a monitoring system for future growth.


Take Away

To size your server and install the software, you need to understand the variables involved in the decision and then match up the options to create three possible choices: standard, enhanced, and optimal. To do that, you need to find out what the minimal possible configurations are and what to get if money is not an option. You can use the information in this section to develop your baseline, and then expand that with metrics to determine the second two options. If you make your baseline realistic and support it with the information that follows, you should be able to make a strong case for it.

The bad news is that no magic formulas tell you what to buy and how to configure it. The best way to make the hardware decisions is to detail the information in some meaningful way and examine it to come up with a solution. An additional benefit is that after you have written everything down, you can talk with vendors and other professionals intelligently about your needs.

I detail several processes here using simple spreadsheets, tables, and scripts. You can take these processes and create a programmatic system from them; remember, however, that at some level there is always an element of judgment that you will use from your own experiences and corporate environment. Every situation will differ, which is why no computer program can build your landscape for you. Some vendor programs perform this task, but they are for specific application servers, not for a generic use such as a database server. Unless your application maps to one of these calculators directly, you still need to do the work yourself.

Variables

First, build the variables from the information in this chapter. Use a separate table or spreadsheet for each and dedicate a separate grouping of these tables for each function the servers will perform.

Application

Begin with the application. The following spreadsheet is a representation of the kind of information you want to gather:

Application Title:

Application Overall Description:

Application Class (ERP, CRM, etc.):

Application Architecture Type, with description of each layer from data to presentation (remove those architectures not in use):

Monolithic

Client/Server

Three-Tier

Distributed Computing

Service-Oriented Architecture

Landscape

Next, describe the type of environment the software will use. Record details regarding the number of sites, physical locations, and the number of users at each site:

Farm

Department

Organization

Enterprise

User Information

Create a list of the number of users expected for the application. For each location, categorize the users into types. For instance, in an ERP application, you might have 100 users that enter orders requests, 10 buyers, and so forth.

Find out the hours the users will access the system, using Greenwich mean time if you are in multiple time zones. This information is important to know to develop your disaster recovery and maintenance windows. If you discover that you have no discernable maintenance windows, you must add a duplicate system to "fail over" to so that you can apply patches and so forth.

Develop a "load profile" that gives a metric to the amount of resources that type of user will take. In the case of a user entering purchase orders, for instance, you might evaluate the system used to determine that they take 1 percent of the CPU, 14K of bandwidth, and 2K of storage per transaction. You can use those numbers to classify the users into high, medium, and low, or you can use a numeric scheme such as 1 to 10.

Finally, record the functions within the application that this classification of user accesses, especially as they relate to reporting.

Location | Type | Number | Hours | Load Profile | Functions Accessed

Decision Matrix

The process to arrive at a good decision is to record what you know and use that information to answer what you do not know. What you know is explained in the variables section, and the decision matrix helps you develop answers to what you do not know. What you know is captured in the spreadsheets you just created, and what you do not know is which edition(s) of SQL Server to install and what to install them on.

With that information filled out, you can put the variables into a matrix to determine the factors that will help you set up your systems. Assign a numeric set of values to each factor, using whatever scale is appropriate. In this example, I have set the numbers to range from 1 to 10, with 1 being a "lighter" value than 10. To further enhance the reliability of the decision, weight the factors that make sense in your environment. For instance, if speed is more important than growth, weight the user and I/O calculations higher than the projected growth values.

After you put the measurements together in a matrix, you must figure what meets the need. You can create a single "atomic" value of a transaction and multiply that out over the variables or you can generalize the load across peak and base values. If you are unsure of creating the atomic transaction number, it is safer to estimate a peak load and put the base at half that value. Those numbers enable you to select a system that runs at a minimal level and one that can handle the peak load. In the middle of those systems is the "average" choice. I use the peak and base method in this example.

In this chart, I take the variables I know and place them on the left. Each variable is given a score from 0 (not applicable/advisable) to 100 (perfectly suited), indicating how well that particular variable works on the choices of hardware and software. The numeric figure I use is not scientific, because I will weight the score for the environment that I have.

The application I have is a Web-based architecture with a separate reporting instance, and the landscape at the organization is a "Business" type. The storage for the application is 100GB for the first year, with an estimated 10 percent growth pattern. My testing scenario against the application involved a Fibre Channel SAN with six LUNs presenting. I also used a 32-bit operating system and SQL Server edition.

For this example, I have already filled out the user profile and used that information to get the following breakdown:

  • Light (fewer than 10 transactions per day)25 users

  • Normal (25100 transactions per day)350 users

  • Heavy (150+ transactions or heavy reporting and analysis)100 users

  • Mobile (local subset of data from system with periodic sync)15 users

  • Remote (clients with limited access to main system)10 users

Each transaction (averaged) in my testing showed the following loads on these objects, per minute:

  • CPU .7%

  • RAM 100K (released after each call)

  • I/O .05%

  • NIC 25K

In some systems, each transaction generates a different load on these resources. To keep the example succinct, I averaged the transactions from the various user types based on testing. This is a valid method if your application will (over time) show an average load distribution. Your application might differ.

To find out how your real-world application loads the system, measure the various objects such as CPU and RAM over a day of average use (or test to simulate that) and get the averages. Also calculate the standard deviations for those same periods. If the standard deviations are low, you can use the averages.

If the standard deviations are high and you use the average method, you will end up with an underpowered system at critical times and an idle system at others, and a lot of unhappy users. In that situation, it is better to determine just how often the high values occur. If they occur often, optimize the system to handle the high values.

I will consider the reporting system separate from the application server, so the information I gather in this exercise is for the application server only. I will follow the same process to develop the reporting server later. Using the information above, I developed the following matrix for my example:

Loads | Values |
CPU load (500 users at .7% per transaction) | 350.00 | % of capacity
Memory (500 users at 100K per transaction) | 48.83 | MB
Network bandwidth (500 users at 25K per transaction) | 12 | MB
I/O load (at .05% per transaction) | 25 | % of capacity

With the raw numbers on this chart, I can see that I need at least three and a half CPUs, and that is just for the application. That brings me to four full processors for the application and the operating system. I have to make a decision here, however, regarding growth.

After checking with the developers, I discover that the application can federate (that is, break functions onto other servers). I decide that four processors in a single machine will be enough. If the system did not federate, I would recheck my numbers, and if they still held true, I would recommend more processors. Each decision affects others, so I might repeat the load testing with 64-bit servers and see whether I experience any gains there. Adding more than four processors kicks the SQL Server edition up a notch.

The memory for the application is not a high value, but I want to keep as much cache as I can in RAM. I decide that 4GB of RAM will provide enough room for caching, locks, the operating system, and other utilities.

Evaluating the network bandwidth, I see that I have plenty of network capability on the network interface card (NIC) that I will dedicate to the clients. I always include a separate NIC in a server for backups and replication of data, so I do not need to factor that traffic on the dedicated client network interface.

My matrix also shows lots of room for growth with what is already in use over on the testing system. It is also fast, so I recommend a duplicate of that storage system for my production landscape.

Based on all this information, I see that I need a SQL Server edition that can support four processors and has the capability for Reporting Services. Because the servers can federate and I do not need more than four processors on any one server, I can get by with Standard Edition. For the mobile users, I query the developers again and find that the Express Edition will work for the laptops they use. I now have enough information to begin researching the vendors for the best deal, all from fairly well-researched information.

Although this is a simple representation, it does demonstrate the process. You can extrapolate from the information that you see here to create a system for your production applications. Remember that if the system will house multiple applications, you need to go through these exercises for each.

Monitoring

Using this process, you can now make the decisions for your environment and develop the metrics you will use to monitor the solution. Make sure that you instrument the system so that you can track the values over an appropriate interval, such as weekly or monthly. Each metric will have its own periodicity, so make sure your collection method takes that into account. I explain the mechanics of monitoring SQL Server further along in the book, but make sure you bring up this topic with the vendor or in-house developers. You need to make sure you can monitor their software, too.

In Chapter 5, you will find that SQL Server 2005 provides a wealth of functions to monitor and track your SQL Server. You can use those features to monitor not only for growth, but also for performance.

With your system installed and configured, let's explore a little more about the tools you have to control SQL Server.


Chapter 2. Architecture and Tools

When the proper planning is complete and SQL Server 2005 is installed, it is time to configure the system for production use. To properly configure and maintain the landscape, you need a sound understanding of how SQL Server operates. In the first part of this chapter, I focus on the many architectures within SQL Server that affect the transfer of data into and out of the system, including the processing engines, memory, I/O, and the networking layer, which has changed since previous versions. Depending on the edition, SQL Server 2005 does not have the TCP/IP protocol enabled, making the system "secure by default." I show you how to enable that protocol (and others) in this chapter. I explain the path that a query takes and how the system deals with the results. Along the way, I also explain where the various parts of the software are stored.

In the second half of the chapter, I explain the tools you use to control and interface with SQL Server 2005, and I finish up by giving you some practical examples for using them to create your own monitoring environment. The information in this chapter forms the basis for the rest of the book.

Understanding the architecture of a database system is an involved process. Many components make up SQL Server 2005, and knowing how they interact is important for several reasons. Knowing how a query routes through the plumbing of SQL Server will help you pinpoint errors when they occur. You will also be able to find performance bottlenecks, something you just cannot do if you do not know the path the data takes. Understanding the architecture will also help you design systems correctly.


System Architecture

Microsoft has positioned SQL Server 2005 as the back-end data platform for the Microsoft Office System and most of its enterprise products, with the notable exception of Microsoft Exchange. The database engine serves up data; Integration Services (SSIS) provides extracts, transformations, and loading of data; Analysis Services provides business intelligence; and Reporting Services serves the online and analytical reporting needs of the organization. The Service Broker allows for an HTTP-based messaging system to enable service-oriented applications (SOAs), and Replication allows for disconnected recordsets and high availability along with Clustering and several other technologies. And these are just the delivered services and features, not what you can create with them. Each of these features has multiple components in their individual architectures, and keeping everything straight can be a challenge.

There are a couple of approaches to comprehending the architecture in a product as large as SQL Server. The first is to take each component and break down its parts, examining how it operates. The second is to trace what happens when an operation is performed. I do a little of both in this chapter. I start with the physical "footprint" of the system and then explain how the database engine behaves. I then cover the objects within a database and finish up with how an application communicates with the server. I use that process to tie the physical, engine, and objects concepts together.

Physical Footprint

As in all software, SQL Server is at its core a mixture of files and registry settings. Some files store the code to run the software (called the binaries), and other files store the data the system produces. For this discussion, I call the first set operational files and the second set data files.

Operational Files

You may recall from the first chapter that I mentioned that you want to keep the various SQL Server files apart for optimal performance. In this section, I explain the default locations for operational and data files, and you can extrapolate to the drive letters you selected during your specific installation.

Assuming you have taken the default file locations and a standard installation, the base directory is found at C:\Program Files\Microsoft SQL Server. Within that, the following table shows the rest of the directories and their primary uses.

Location | Primary Use
C:\Program Files\Microsoft SQL Server | Base directory
C:\Program Files\Microsoft SQL Server\MSSQL.1 | System database files, log files, and default backup location
C:\Program Files\Microsoft SQL Server\MSSQL.2 | OLAP database files, log files, and control files, if installed
C:\Program Files\Microsoft SQL Server\90\ MSSQL.3\Reporting Services | Reporting Services databases, binaries, and assemblies, if installed
C:\Program Files\Microsoft SQL Server\90 |
C:\Program Files\Microsoft SQL Server\80\COM | Legacy files
C:\Program Files\Microsoft SQL Server\90\COM | Replication control files
C:\Program Files\Microsoft SQL Server\90\DTS | SQL Server Integration Services files
C:\Program Files\Microsoft SQL Server\90\NotificationServices | Notification Services control files
C:\Program Files\Microsoft SQL Server\90\SDK | Software development files, if installed
C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap | Setup files
C:\Program Files\Microsoft SQL Server\90\Shared | Interface files and error dumps, and the Surface Area Configuration Tool
C:\Program Files\Microsoft SQL Server\90\Tools | SQL Server tools, with the Visual Studio hooks (By default, this is where all projects are stored, too.)
C:\Program Files\Microsoft SQL Server 2005 Sample | Samplesdatabase files, source files, and script files, if installed
C:\Documents and Settings\Username\My Documents\SQL Server Management Studio | User backup files, settings, and templates for projects
C:\Documents and Settings\Administrator\My Documents\Visual Studio 2005 | User templates for projects, if installed

The Samples directory is not off of the main SQL Server directory by default. When you install your instances, you can change that if you like by editing the Advanced settings in the installer panels.

Data Files

The data files are actually made up of two kinds of files. The first stores data, with the extension of .mdf, which stands for Microsoft Database Format. The second stores the log files, which have the extension .ldf, standing for Log Database File.

Having the log files separated from the database files provides many benefits. The first is greater speed. Because the data is written sequentially to the log file, it is not blocked by nonsequential writes to the data file. And, because data is read nonsequentially from the data file, those reads do not block the transaction writes in the log.

The separation of transaction and data files also provides safety. Depending on the recovery model, you can then back up the (smaller) log files frequently, even once per hour or more. Because the database backups contain all the data to a point in time, and the log backups contain all the changes, if the database has an issue, you can restore the database to the last backup and then restore the log file(s) to the database. Your database is then up-to-date as of the last log backup. I cover this process in great detail in Chapter 3, "Maintenance and Automation."

You can have more than one of each of these kinds of files. In fact, you can create as many database and log files as you want. Placing them on different physical devices will help with performance, because you can specify what database objects, such as tables and indexes, are placed on these files. That allows you to plan the layout of your database, separating the heavily used objects onto different physical devices. Because those devices can deliver the data without interfering with each other, you get better response. I deal more with performance tuning in Chapter 5, "Monitoring and Optimization." You can have up to 32,767 files per database, and they can be 16 terabytes in size.

You can logically group the data files into filegroups. The process is quite simpleyou create a filegroup and name it, and when you create files you can specify which of these filegroups you want them to use. You can then back up and restore filegroups independently of each other, which is useful in large databases. You always have one filegroup called Default, and you can make up to 32,766 more.

The Query Process

The two types of data files work in concert to store and retrieve data for applications. These applications either change data (which includes adding, deleting, and editing) or request data. All of these operations are generically called queries, although the word query is often used to describe a request for data. Queries are generated from various programming methods, which I explain a little further in this chapter.

What follows is a high-level description of the query process. It does not differ much from most any Relational Database Management System. In the next section, I explain the low-level details of the entire data flow through SQL Server 2005.

When an application enters, edits, or deletes data, it submits a transaction. That package of information is communicated across the network layer on the client machine to the server network layer and then on to the database engine.

DBA 101: Transactions

You can think of a transaction in terms of an ATM transaction at your bank. Suppose that you decide to move $100 from your savings account to your checking account. If the money comes out of savings, you want it to go into checking, or not happen at all. The bank is concerned that if the money goes into checking, that it also leaves savings. Otherwise, they do not want anything to happen either. The idea that a group of things happen as a unit is a transaction.

Once submitted, the transaction is entered into the transaction log. After it is recorded there, a logging process enters the data into the database. Depending on a configuration setting that I explain later, the transaction is kept in the log until a backup is taken, or it is erased from the log after it is committed to the database.

Unless you specify otherwise, all transactions go through this process. What this means is that if you enter a row of data and then immediately delete it, both transactions go through the log. The only time that this does not hold true is when you enable the database to bypass the log when a special import process is run, called a bulk load.

When an application retrieves data, the client submits a query through whatever connection method the application uses, once again using the networking layer on both systems, sending the request on to the database engine. The database engine uses the Query Processor to locate the data in the physical files. The data is returned as a dataset to the client through the networking layers of the server and then the workstation.

The Database Engine

The database engine is the core process within SQL Server 2005 that moves data in and out of the physical files (I/O). The physical files I mentioned earlier map to logical files that SQL Server works with. The name of these logical files, what objects they store, and their physical names are stored in the master database, which is SQL Server's primary control file.

SQL Server automatically recovers itself in case of a problem shutdown. On the rare occurrences where the server does not have a chance to commit all the entries in the transaction log to the database before it shuts down, the system either rolls forward, meaning that the log entries (transactions) are written to the database, or rolls back, meaning that the transactions are not deemed complete and are not entered in the database at all. When either of these things happens, an entry is made into the Windows Application Log and the SQL Server error logs. I explain how to read those logs in Chapter 4, "Security."

Another of the database engine's primary tasks is to run the Query Processor. The Query Processor intercepts all calls made to the database and interprets them to develop a query plan that details the most efficient path to the data stored in the database and log files. If it can, the Query Processor will reuse a plan to get the same data when called for, making those queries even faster. The biggest impacts on the Query Processor's performance are the system's CPUs and memory.

The Query Processor, along with several dozen other processes, lives in memory while SQL Server is running. Data is also cached in memory. That is why SQL Server, like most Relational Database Systems, needs a lot of RAM. In Chapter 5, I explain how to watch these items and determine when you need more memory on your server.

The database engine passes off the transaction results to the networking layer, which in turn talks to the Windows networking layer. The more protocols you have enabled on SQL Server, the more cycles are used to deal with them. For this reason, it is best not to enable protocols you do not need. By default, some SQL Server 2005 editions only enable the Shared Memory protocol, which allows the system to talk to itself on the same machine only. In the section on tools that follows, I show you how to change that.

Objects

The database engine, along with the other processes, is designed to move data back and forth to and from a client. To facilitate a fast, safe environment, SQL Server 2005 uses several objects to store and optimize the data. I classify everything in SQL Server that is not a process to be an object.

Database

The first object is the database. The database is the outermost collection object, which means that it contains other objects. Databases have logical files defined in it that point to the location of physical files on the storage subsystem. Databases also have user logins associated with them, which I explain further in Chapter 4.

There are two categories of databases within SQL Server 2005: system databases and user databases. Both have the same structure; it is just that they are used for different reasons.

System Databases

The first system database is called master. This database contains the data used by SQL Server 2005 for all its operations, including security, and mounting and configuring all the other databases and their objects.

DBA 101: The Master Database

You can query the tables in the master database directly, but Microsoft provides lots of customized views and functions to get at the data they contain. In previous versions, Microsoft also provided system views, but they were not complete, and most administrators worked around them by writing queries directly against the system tables. This is usually a bad idea because the system tables are apt to change. Microsoft does not guarantee that the master database structure will remain constant throughout the life cycle of the product.

Because you should not write code depending on the master tables, understanding the system views and functions is essential for managing your system. In Chapter 5, I explain what the system views and functions do and how you can use them. Get in the habit of using these functions; they are safer than using the system tables.

The model system database is just thata model for any other databases that you create. When you create a new database and you do not specify any options, the model database structure is used as a template. You can place any other objects that you want in every database in the model database, and new databases will "inherit" those objects. This does not apply to databases that you restore from a backup, just the ones you create directly in code or using the graphical tools.

The msdb system database contains all the scheduling information, jobs, alerts, and operators (more on those in a moment) that SQL Server 2005 uses for things such as maintenance and replication. It also stores history entries for maintenance plans and database backups, as well as the SQL Agent metadata. System views and functions query from this database for that information.

The final system database is tempdb. This is SQL Server's "scratch" area. Whenever a query has an ORDER BY clause that sorts data for a query, or SQL Server itself needs to sort the results of a query or do some other intermediate work, it is done in tempdb. This database is destroyed and re-created each time the server is started, so you do not want to put anything into this database that you need to keep. When your developers create an in-memory table (called a temporary table), it is stored in the tempdb database.

DBA 101: Modifying System Databases

It is best if you do not modify system databases at all, other than perhaps to add objects into the model database. Whenever a new service pack comes around, the system databases are fair game for changes by Microsoft, and during upgrades you are sure to have issues if you change them. You also risk the stability of your system when you add or change anything on the system databases.

User Databases

User databases are those you create. There are a few ways to create a new database, such as issuing a Transact-SQL (T-SQL) command, using a programming method, or using graphical tools. I show you a few of these in the "Tools" section of this chapter. Another way is to restore the database from a previous backup. You can use a backup from the same server or even a backup file from another system. I explain that process in Chapter 3.

Yet another way that you can create a database is to attach it. This involves disconnecting a database from a system, copying the files elsewhere, and then connecting the files. You can detach and attach a database on the same server or even another SQL Server 2005 system.

To detach a database using the graphical tools, right-click that database and select Detach from the menu. This shuts that database down gracefully, and then removes the logical entries for the files it uses from the master database. You can then copy the physical files to the new location and right-click the databases object; select Attach from the menu and enter the location for the files. The system makes entries for the new database in its master database and starts up the database.

Within each user database are system tables, views, and other objects. After you create the database, you can start putting other objects in it right away.

Databases have states, which indicate how they can or cannot be used. You will see these states when you first start the graphical interfaces or you can use the T-SQL statement of SELECT for the state_desc column in the sys.databases catalog view (more on that later).

You can also call the Status property in the DATABASEPROPERTYEX function to give you the same information.

State | Available | Meaning
Online | Yes | This is the normal state for a database and means that it is ready for use.
Offline | No | This state is something that you set, when you want to remove it from use but not from the server. Use this state to move files around.
Restoring | Yes, for those filegroups not being restored | This state indicates that the database is being restored from a backup.
Recovering | No | This state indicates that the transactions in the log are being checked and if needed are rolled forward or back.
Recovery Pending | No | This state requires you to intervene. Either system resources or files are unavailable to complete the recovery process.
Suspect | No | One or more filegroups is not consistent at the operating system level.

There are also various synchronizing states for a database involved in mirroring, which I discuss in Chapter 7, "Notification Services and the Service Broker."

As far as the technical details go, the database size depends largely on the edition you use (see Chapter 1, "Installation and Configuration") and the operating system file limitations, but the larger editions can scale database sizes up to 1,048,516 terabytes. In each instance of SQL Server, you can have up to 32,767 databases. You can have up to 50 instances on a single server.

Schema

SQL Server 2005 uses an abstraction object called a schema. A schema is simply a naming object within a particular database, which in turn "owns" other objects. Users are granted rights to the schema so that they can access the objects the schema owns. This allows the database to have multiple tables called "product," for instance, and by using a separate schema name you can keep them separate. It also allows for objects to remain independent of users, so that when user accounts change, the object does not have to.

You are limited to 2,147,483,647 objects in any database, and schemas come out of that limit. In other words, if you had only one table and one user, you could have roughly 2,147,483,645 schemas.

Table

The next object within a database is the table. A table is an entity in a relational system that provides columns and rows of data. You can think of a single spreadsheet as a table.

I do not cover table design here, but to design tables properly, you should be familiar with a little relational calculus.

DBA 101: Relational Calculus

All the relational calculus you need to know right now is that in a relational system, tables should contain discrete data that is stored only once. To allow multiple rows to show the same data, Sequential Query Language (SQL) selects from various tables, joining them on a common field. A field that uniquely identifies a row in the table is called a key field. You can use one field in a table to point to the key field anotheran object called a foreign key.

This means that you would store a customer's name one time, with a single field in that row designating that customer, say, customer_number. When you place multiple orders for that customer, a field in the orders table would contain this single number, one per order, to show that a group of orders belonged to a particular customer. In that way, you could change the customer's name once, and all the orders would point to the new name, provided the number stayed the same.

In this case, the customer_number field in the customers table is called the primary key or key, and the customer_number field in the orders table is called the foreign key. The customer_key in the customers table can appear only once, and the customer_key in the orders table can appear multiple times. Combining the two in a query is called a join.

If it helps to have an analogy, you can think of the SQL Server environment as a building. The databases are the floors on that building, and the offices are the schemas within the database. A file stored in an office could be considered a table. Thinking of the database this way will help give you a basis for understanding the security chapter a little later on.

Tables form the basis for all your data. You have the same object limitation (2,147,483,647 objects per database) I mentioned earlier to take the number of tables from. You can have up to 1,024 columns in a table, and as many rows as storage space allows. The rows can be 8,096 bytes long (each). There are 16 columns allowed for a primary or foreign key, and you can have 253 foreign keys referencing a table.

View

A view is a T-SQL set of code that selects data from one or more "base" tables, and the result looks like a new table. For instance, if you took the customers and orders tables I mentioned earlier, and joined them with a T-SQL statement as a view called cust_orders, you could query the cust_orders view as if it were a real table. I show you an example of this in the "Take Away" section that follows.

Views are used to partition data by selecting only certain columns or by selecting rows where they match a certain criteria. You can combine the two to limit or simplify the data your users will see.

You can also use views to hide the structure of base tables, because you can name and order the columns differently in the view than they are in the tables. You can also combine multiple tables in a view to present them as a single table.

Often your base tables and the columns will see their names or structures change over time. If your application's code is based on those tables, the code will also need to be changed. However, if your developers base their code on views, you can simply change the view to reference the new table or column names and the original code does not have to be touched.

Views have a few limitations. You can only update data into one of the base tables at a time through a view. If you want to change the value of three columns in our sample view in a single statement, they need to be in either the customers table or the orders table, but not both.

The same overall objects limitation number holds for creating views, and there are a couple of other numbers to know. You can reference up to 256 base tables in a SELECT statement (which forms the basis for a view), and you can have 4,096 columns referenced in a view. You can get around some of these limits by having one view select data from another.

Stored Procedure

A stored procedure is similar to a view, in that it is a bunch of code that runs on the server and returns a result. The difference is that a view is unchanging. Whatever you have entered as the code for a view is that way until you delete (or drop) and re-create it. A stored procedure can accept parameters from users or code. A stored procedure can return data, a status, or perform some other work in the database. Stored procedures can be used like a view to hide the complexity of a table or to slice and dice the data.

Stored procedures have another advantage over views. SQL Server 2005 will evaluate the stored procedure the first time it is run and compile it into a faster form. This makes the stored procedure perform better when it is called again. You might hear this referred to as "server-side code" in some documentation, because it is processed and cached on the server. This is another reason that the server is so memory intensive.

Microsoft provides several stored procedures right out of the box, and we will be working with a few of those throughout this book.

A stored procedure has the same limits as a view as far as object counts go, and the text that forms the code for a stored procedure must be smaller than 250 megabytes. If you have a single stored procedure anywhere over 2 megabytes, you probably need to examine the codemost stored procedure code is only a few thousand bytes.

Trigger

A trigger is another set of T-SQL code similar to a stored procedure, but rather than being called by the user, it executes on certain conditions. The two kinds of conditions are when users alter data, called Data Manipulation Language (DML) operations, and when the structure of an object changes, called Data Definition Language (DDL) operations. In other words, this is code that runs when the user issues a statement that updates, edits, or deletes data, or runs a statement that changes the structure of the database objects.

For DML, the trigger types are INSERT, UPDATE, or DELETE statements against a table or view, and for DDL statements you can "fire" triggers for CREATE, ALTER, DROP, GRANT, DENY, REVOKE, or UPDATE STATISTICS statements. Triggers normally fire directly on the statement's call, but you can specify that they fire after the code on the table or view completes.

Another type of trigger, called an INSTEAD OF TRigger, fires when the statement is run but does not run the statement. When the statement is called, the code in the trigger runs instead. For instance, if you make an INSTEAD OF trigger on a table for a DELETE operation, when the user tries to delete data from the table, the trigger could print text on the screen that says "Do not delete this record. Archive it instead."

Triggers can become highly complex, and a logic-flow diagram is essential to track them properly when they get a few levels deep or when they implement complex functions.

Developers often use triggers to enforce business logic. In SQL Server 2005, you can also use the Common Language Runtime (CLR) feature to have entire programs run as a trigger. Your developers might, for instance, have a C# program act as a trigger to gather metrics from a Web service when a new order is entered.

Special settings are available to your developers to prevent recursive triggers from firing. If the developers are not careful, an INSERT to table A might cause a DELETE in table B, which might have an DELETE TRigger to perform an INSERT to table A, which in turn causes another DELETE to table B. That is a recursive trigger, and you normally want to prevent that kind of situation.

In any case, you can only "nest" triggers 32 levels deep. Triggers are also part of the 2,147,483,647 objects per database limit.

Function

A function is yet another set of T-SQL or CLR code, similar to a stored procedure that returns a value to the user. It falls under the same 2,147,483,647 objects per database limit.

Functions can return a table, and can even be used recursively in a query. This makes them very powerful for developing server-side code. SQL Server 2005 contains several system functions, and you can create your own user-defined functions (UDFs).

Index

An index is really another table but contains ranges of data and the physical location where the data is found. It is like an index in a book, showing where the data is located. SQL Server uses B-Tree indexes, which sequentially branch off of the ranges of data in an upside-down tree fashion.

Indexes make reads fast, but cause a slight penalty when data is written. This performance tradeoff is one of the primary factors you will use when you begin to tune your system, something I will cover in Chapter 5. To create an index, you name it and specify the columns that will be indexed. These columns are then said to be "covered."

There are two types of indexes: clustered and nonclustered. In a clustered index, the table is physically rearranged to the order of the indexed columns. In our fictional customers table, we might create the clustered index on the primary key, and the table would be physically arranged from 1 to 100, for instance. Because the table is physically arranged by the clustered index, there can be only one clustered index per table. In fact, the clustered index is the table.

It is also a good idea to cluster those columns that are additive, because the entire table would have to be relocated on disk every time an earlier number is inserted.

The other type of index is called a nonclustered index, and is another table-like structure entirely. It stores the range of data and a pointer to where the data is stored. When query references columns found in the index, the Query Processor process will use the index to look up the physical location of the data. If you reference only the columns in the index, SQL Server does not have to access the table at allit can get the data right from the index itself. That is called a "covered query."

You can have 249 of these indexes, and each can reference up to 16 columns. Indexes come out of the same object number limitation as everything else.

Job

A job is a special set of entries into the msdb database. Jobs contain steps, which are made up of tasks such as T-SQL queries, ActiveX scripts, operating system calls, and more. Steps have conditions, such as success or fail, which you can string together to loop or jump over one step to another to form simple logic.

You can call a job from a stored procedure, directly within T-SQL code or from a programming library. Jobs can also be called by an alert (see the next section). Jobs also have schedules, which tells the job when and how often to run. Jobs are used extensively by SQL Server itself; replication even depends on them.

Jobs have a status, which can be logged. Jobs interact with operators, which I explain in a moment. Jobs can also record their history automatically, which you can query.

Jobs are run by the SQL Server Agent service. If that service is not started or does not have an account associated with it that is allowed to perform the tasks, they will not run. I explain jobs in greater detail in Chapter 5.

Alert

An alert is a condition that is raised based on an event, such as a full database file or other system and database-related action in SQL Server 2005. Developers can code them to receive system responses in their programs. Microsoft delivers several system alerts right out of the box, mostly dealing with system-level events, and you can make more alerts for just about anything that happens on your server. The system events are triggered by the server engine, and user-defined alerts can be called by a server condition or directly in code.

An alert is made up of an error number, a severity level, a database reference, event texts, and actions.

The error number is a reference, so that you can track the alert in the Application Log in Windows or in the SQL Server logs. The error number is the identifier used to call a particular alert from code. Developers can also watch for these error numbers in their own code.

The severity level indicates how big an issue the alert is. Microsoft reserves levels 19 through 25 for itself and sends a SQL Server message to the Microsoft Windows Application Log when they occur.

Events with severity levels less than 19 trigger alerts only if you use the stored procedures called sp_altermessage or xp_logevent, or if you use the T-SQL statement RAISERROR WITH LOG to force them to be written to the Windows Application Log.

The database reference sets which database the alert fires in, and the event text specifies the message the alert generates.

An alert can do two thingsit can send a message to an operator (see below) or it can run a stored procedure or script. Putting all of this together, you can code an alert that is called by a program, which then inserts an entry into the SQL Server logs and the Windows Application Log, sends a message to the application, and then runs a job. All with a few lines of T-SQL code.

Operator

An operator is a set of identification information such as a name, network login, e-mail addresses, and pager number. SQL Server can send messages to operators, alerts and jobs can both notify operators, and automated maintenance plans can send an operator a message containing the results of system maintenance.

It is important to note that anyone can be an operator, not just people in IT. You can define any user as an operator and have the system send them a message based on code, jobs, or alerts.

You will want to create at least one failsafe operator that receives messages when the system cannot reach anyone else. That operator is normally the administrator for the system.

Communication Processes

Let's expand the view of the path that a data request takes that I mentioned earlier. We start with a fairly complex graphic (shown in Figure 2-1), and I explain each part as we drill down through the levels.

Figure 2-1.

[View full size image]

Application

Although this book does not deal with writing programs for SQL Server 2005, a little development background is useful to help you properly size and design your systems. It can also help you with troubleshooting.

The possible applications for SQL Server 2005 include "rich" or "fat" clients, Web interfaces, and development and maintenance tools. Rich clients are developed using C#, Visual Basic, C++, Java, or some other high-level programming language. These programs are compiled and placed on a client machine. Often these types of applications make frequent calls to the database server, and as a result use more resources.

Web interface clients include those based on a Web site or those that use Web protocols. These interfaces, when coded properly, normally take fewer resources on the local system than a rich client.

When you use the SQL Server Management Studio or a third-party tool to manage or control SQL Server, you are still using an application as far as SQL Server is concerned.

Regardless of which type of client or interface is used, all programs use a connection method, a library, and a network protocol to access the server.

Connection Methods

An application that is coded for SQL Server 2005 can use several methods to connect to the server. These are not network connections; they are programmatic connections.

ODBC

One of the most popular and widely used connection methods is Open Database Connectivity (ODBC). ODBC is made up of a provider, a data source name, and options. The provider is a library supplied by a database or other software vendor that allows connections to their software. The Data Source Name (DSN) is a group of settings stored either in a user-specific registry entry, one that is generic to the whole system, or DSN information that is stored in a file. These settings provide the name of the server, the database name, and the security credentials. Each provider offers options for the connection such as tracing and language settings.

ODBC is mature, well understood, and supported by every major platform and database vendor. The disadvantages with an ODBC driver are its speed and the need in many cases to configure the client to use it.

Microsoft ships several vendors' ODBC drivers natively on the Windows platform, but most vendors supersede the installed drivers with their own. This does not cause any problems with SQL Server 2005 because it updates the Windows system when the client tools are installed, but some third-party drivers have been known to cause issues with stability on the client.

OLE DB

To address some of the shortcomings of ODBC and allow for a distributed computing environment, Microsoft created Object Linking and Embedding for Databases (OLE DB). OLE DB is basically a set of Component Object Model (COM) interfaces for applications to access database functionality and datasets. With an OLE DB connection, the client machine does not need to create a data source name, and the developer can create a connection to the server, and provide locations and credentials all within code.

Both ODBC and OLE DB use a set of components within the Windows operating system called Microsoft Data Access Components (MDAC). The MDAC is a series of software libraries, binaries, and utilities that are so spread into the operating system that it is difficult to tell which version is installed.

You can search for the msador15.dll file in the Program Files directory and then view the Properties tab to find what version of MDAC your system is using. Microsoft also has a tool you can use to check the version of MDAC, called the Component Checker. You can find that by searching the Microsoft Knowledge Base for item 301202.

The primary issues you will run into crop up when the MDAC version on the client or middle tier is different than on the server. It is best to pair these up, but the developers sometimes require a certain version for their applications to work properly.

JDBC

Microsoft wants to make SQL Server 2005 widely available to all coding platforms, so they developed a Java Database Connection (JDBC) driver to provide access to SQL Server 2000 and SQL Server 2005. You can use this driver from a Java application, application server, or Java-enabled applet. It is a Type 4 JDBC driver using the standard JDBC application program interfaces (APIs) available in Java2 Enterprise Edition (J2EE). This is updated on the Microsoft SQL Server site, so after you have completed your installation, you can download the latest one from there.

DB-Library

DB-Library, or DBLIB as it is sometimes called, actually belongs to both connection methods and APIs (see below). It is a proprietary library used by early versions of SQL Server, most commonly seen in C (and C-based) programs. Prior to the ODBC and then JDBC connectors, DB-Library was often used on UNIX-based client systems to program SQL Server applications.

Application Programming Interfaces

Within each connection method, there are various programming libraries called application programming interfaces (APIs). These interfaces represent functionality the developer uses to perform database-related tasks. The developers can use standard T-SQL statements to manipulate data and database objects or use the API functions to do the same thing.

The API functions work by instantiating an object, such as a database, making a connection to it, and then calling methods or properties to work with them. One type of method the developer can use is the "command," which accepts and passes T-SQL statements.

DBA 101: Methods and Properties

Methods are things that objects can do and can be enacted. Properties describe the object and can be set. For instance, a method for a dog might be "Bark," and a property for the dog might be "Color."

The other ways of using an API is to instantiate an object, such as a database or table, and then call methods against it, such as "backup" or "query." If the developer follows this process, the code they create is independent of the language extensions for a particular database platform. The code could just as easily run against SQL Server as Oracle.

RDO/DAO

Two of the earliest programming interfaces are Remote Data Objects (RDO) and Data Access Objects (DAO). They were the first attempts at providing a distributed environment for applications. Both had fairly significant drawbacks, and you will not see a great many applications using these libraries in newer applications.

ADO/ADO.Net

ActiveX Database Objects (ADO) and the newer ADO.Net provide far more functionality than that found in RDO or DAO. One of the primary enhancements was that the developer can address independent objects such as tables and rows without having to navigate a hierarchy as they did in RDO.

ADO is one of the most popular programming libraries in use, and to allow for a smooth transition to the .NET environment Microsoft updated the library to ADO.NET. This newer library is largely backward compatible with ADO and allows the developers to use what they already know and take advantage of the managed environment that the .NET suite provides.

SQL Native Client

The SQL Native Client is the newest programming library provided by Microsoft. It is basically one dynamic link library (DLL) that combines the functionality of the OLE DB and ODBC libraries and provides new features. The SQL Native Client can take advantage of SQL Server 2005 features such as Multiple Active Result Sets (MARS), User-Defined Types (UDT), and XML data type support. This is the preferred library for newer development efforts.

SMO

In earlier versions of SQL Server, a library called Distributed Management Objects (DMO) allowed developers to write management applications. The objects exposed in this library dealt more with maintenance and performance monitoring than the other programming interfaces, although with a little work both could be made to do the same things.

In SQL Server 2005, the DMO library is replaced with a more powerful set of interfaces, called Server Management Objects (SMO). As a maintenance DBA, you might not program often, but when you do the SMO library is the one to learn.

Client Network Library

After the connection method and programming library are coded, the application uses a connection method to talk with the network protocols. Networking protocols include TCP/IP, Named Pipes, and those installed by other network applications. These protocols have properties based on what they provide including encryption, compression, and so forth. If you enable those on the client, they need to match what is enabled on the server.

You will find that TCP/IP is often faster than Named Pipes, but that depends on the environment and the coding techniques in the application. TCP/IP is more secure than Named Pipes.

SQL Server 2005 also comes with the Shared Memory protocol. Shared Memory is a protocol that you can use when the client and the server are on the same machine. Because the packets do not have to travel on a physical network card, speed and security are increased. The MDAC mentioned earlier can only use Shared Memory in versions 2.8 and higher.

Another protocol in SQL Server 2005 is the Virtual Interface Adaptor (VIA), which works with special adaptors shipped with some systems.

To set the protocols and their options, you use the SQL Server Configuration Manager tool available from the Start menu under Microsoft SQL Server 2005 > Configuration Tools > SQL Server Configuration Manager. I show you that tool in a moment.

Server Network Library

Moving to the server side of the transaction, SQL Server 2005 listens for network calls using a similar set of network libraries to those installed on the client. This layer also handles encryption at the network level. You may have more protocols enabled on the server than on a particular client so that the server can answer the various protocols on each client. If you find that you have a protocol enabled for the server that none of your clients are using, disable it. It is safer and improves performance.

You use the SQL Server Configuration Manager tool available from the Start menu under Microsoft SQL Server 2005 > Configuration Tools > SQL Server Configuration Manager to set options on the server as well as the client.

In addition to network protocols, SQL Server 2005 uses two other "shims" between the engine and the networking layer. The first is called the tabular data stream (TDS), which is a streaming protocol for moving data in and out of the server. There is nothing for you to set or tune here, but you might see it mentioned in an error log or program return.

The second is a layer that may be called from Internet connections that make up the Simple Object Access Protocol (SOAP). This layer allows SQL Server 2005 to communicate using HTTP and XML and also provides the "endpoints" that SQL Server uses. I describe endpoints in more detail in Chapter 4 and Chapter 7.

Algebrizer

Most of the time, the path a query takes can be explained in simple terms. The query originates at the client, as we have seen, and then is passed to the network layer on the client, which talks to the network layer on the server. The server processes the request and then returns the result through the networking layers back to the client.

Although it is not as important to understand each and every part of what the server does with the request once the networking layer contacts the SQL Server engine, it is interesting to know what the query goes through to process the request. The first thing that SQL Server does with a query it receives from the networking layer is to implement the SQL Server Algebrizer. The Algebrizer dissects the T-SQL code, and then creates an operator tree, which uses further scalar and relational operators to determine how to split up the query. The Algebrizer also takes a part in determining how much parallelism the server can implement on the query, and also where to locate some objects in cache memory.

And that is not allthe Algebrizer also helps determine the best way to work with subqueries and aggregate functions, which are two of the thorniest problems in database theory.

As you can see in the flow diagram, there is also an XML version of the Algebrizer. This separate function is necessary to work with the hierarchical data in an XML document as opposed to the relational calculus used by the Algebrizer.

Parser

When the query is split up, the Parser breaks the smaller queries down even further. Just like the Algebrizer, there is also an XML twin for this process, called the XML Parser.

Full Text

If the query calls for an operation involving full text queries, the Full Text process calls out to an external engine for text string parsing.

Service Broker

Next in line is the Service Broker. This component allows the developers to build disconnected or federated, applications, and this process handles where those requests should be processed.

Expression Service

In some queries, particularly those involving stored procedures, SQL Server has to deal with variables. The Expression Service process handles variables and parameters.

SQL Manager

The SQL Manager process takes care of stored procedures and how they execute. Some T-SQL statements, even those not found in a stored procedure, are also handled by the SQL Manager, especially if they use auto-parametized queries.

T-SQL Exec Engine

The T-SQL Execution Engine and its XML counterpart, the XQuery Execution Engine, are responsible for T-SQL language processing. They interpret the T-SQL or XQuery (the XML querying specification) statements and break them down into parts, checking to ensure the syntax is correct.

Common Language Runtime

The CLR process provides communications to .NET libraries. This is a new feature in SQL Server 2005 and allows developers to extend the power of the database engine by writing high-level code that can be called inside T-SQL just like a stored procedure or a function. Developers compile the code and then instantiate it into the database, or they can use Visual Studio to develop and instantiate the assembly directly into the database.

This ability is both a blessing and a curse. Providing a high-level language processor gives an incredible amount of flexibility to the platform, but also allows for poorly written code to disappear into a black box. Great care should be taken in implementing this feature.

The CLR has proven to be very stable, so it will not easily crash your SQL Server. Because of the security implications, however, you have to enable the CLR on a SQL Server before the developers can instantiate any assemblies into the database. To do that, you can use the SQL Server Surface Area Configuration (SAC) tool, or run the following statements on SQL Server 2005:

EXEC sp_configure 'show advanced options', '1';
GO
RECONFIGURE;
GO
EXEC sp_configure 'clr enabled', '1'
GO
RECONFIGURE;

I show you how to use the SAC tool further in Chapter 4.

Query Optimizer

The Query Optimizer is the process that determines the indexes, statistics, and optimal paths to answer the query. Several performance counters and system views help you see what the Query Optimizer is doing.

Query Execution

After the query has been checked for syntax and optimized, the Query Execution process creates a final query plan, checks and implements any timeouts, and then creates the final execution plan. You can view these plans, both graphically and in text, in the SQL Server Management Studio. We make extensive use of this feature in Chapter 5.

Sort Manager

By design, data stored in a relational database system is not stored in any particular order. The only time that is not true is when you have applied a clustered index, as I mentioned earlier. When an application wants to receive sorted data, the Sort Manager process manages ORDER BY statements that sort the recordsets. To do that, the process must be able to store temporary results as it sorts them, so this is one of the processes that use the tempdb database mentioned earlier.

Lock Manager

The Lock Manager process handles lock determinations. It determines when to escalate a lock from one type to another and will also find and settle deadlocks.

DBA 101: Locks

Locks are one subject that interest both administrators and developers.

A lock occurs when one query depends on another. Consider an inventory-ordering system. It is important that when the sales staff looks up an item in inventory, they can be sure of the number of items on hand. However, another salesperson might be selling a number of those items out of that same inventory location. To ensure that the two have the access they need to the right number, database engines implement locks.

The effect is that while the first salesperson is looking up the items, the second is prevented from selling them, and if the second is selling them, the first is prevented from seeing the number until second salesperson is done.

There are many lock types in SQL Server 2005, indicating everything from an intent to take a lock, through the various levels of the lock, such as database, table, or row.

The three most important lock categories are shared, update, and exclusive. A shared lock will not prevent another read. The update and exclusive locks prevent reads and other updates.

SQL Server tries to take the most optimal lock that it can. If it makes more sense to lock an entire section of rows because it would be quicker to do that and get out than it would to determine the row lock needed, it will.

Deadlocks occur when two items depend on each other. Proper coding can usually prevent this situation, but if it does not, SQL Server will pick a "victim," kill its query, and then send a response code to the offending program. Developers can watch for this code and handle it in their applications.

Transaction Manager

The Transaction Manager process handles data logging operations and recovery of the databases. This is the process that takes care of moving data between the transaction log and the database, and also rolls it forward or back on startup.

Filestream Agent

During replication operations (more on that in Chapter 6, "High Availability"), SQL Server needs to move potentially large amounts of data internally and from system to system. The Filestream Agent process handles these file operations.

Version Store

In SQL Server 2005, developers have the ability to have multiple versions of data available to the users. In effect, two users could see different results from the same database. To do that, SQL Server manages row versioning by storing row identifiers in the tempdb database. The Version Store process handles those transactions.

Page and Text Manager

SQL Server 2005 database storage is divided into "pages," which are 8K units on the hard drive. Pages are combined into "extents," which are eight pages long, making an extent 64K in size. Physically positioning this storage and tracking how fragmented (far apart from each other) they become is the job of the Page and Text Manager process.

SOS and Meta Data

To facilitate easy access to the operating system and thread scheduling, Microsoft has replaced its previous OS layer interaction process (UMS) with the SQL OS (SOS) process. It provides not only a better set of tools for developers to work with the thread model, but sets up for future versions when virtual machines come into play at the operating system level. This process also provides for processor affinity, so that the server can spawn off threads to specific processors when directed.

The Meta Data process handles the metadata SQL Server needs for itself. This is where most of the system views and functions come from that I detail in Chapter 5.


Tools

You will use multiple tools to manage the server and interface with the databases. For the most part, you will use the SQL Server Management Studio to work with a graphical interface and sqlcmd for the command-line interface. Other tools are installation specific, such as the Business Intelligence Studio and the Reporting Server tools. I cover those separately when I explain those products in detail.

I explain a high-level view of the tools in this chapter so that you can get an overview of how they work. I use these tools extensively in the rest of the book, so I introduce advanced concepts along the way that deal with the process at hand. This section is just an introduction.

SQL Server Books Online

Books Online is the primary help source and documentation for SQL Server 2005. You can find it on the Start menu under Microsoft SQL Server 2005 > Documentation and Tutorials >SQL Server Books Online. When you first run Books Online, you are asked whether you want to favor Internet content or local content. I normally favor local content because I travel a great deal, and do not want to pay the time penalty for online detection. If your system is always connected to the Internet, you might want to consider favoring online methods because you will get the latest information.

Check the Microsoft SQL Server site for updates to Books Online. The updates are delivered as a simple installer file and will not harm or disturb any of the database files you have installed on your system.

Books Online is an HTML-based help system that includes three tabs. The first tab is the Index tab, where each topic is listed alphabetically. As you type, the tool locates the topics by that name. If the topic is too broad, you will see indented subtopics below it, as you can see in Figure 2-2.

Figure 2-2.

[View full size image]

The second tab is the Contents tab, shown in Figure 2-3. This section reads more like a book and draws you through the documentation in a topical, sequential manner. You will also find some tutorials here, and I highly advise that you explore each one. Some are quite short, others are more involved, but all have valuable information that will give you an overview of topics all the way from the tools through data mining. Because Books Online is a Web-base product, there are already forward and back controls for the page you are looking at. In the case of a tutorial, you need to add another set of controls to navigate through them. To do that, click the menu bar item called Tools and then Customize, and then click the Commands tab. Locate Help in the categories, drag the Next Topic button somewhere on the icon bar, and then click Close. Now you can move through the tutorial screens.

Figure 2-3.

[View full size image]

The last tab, shown in Figure 2-4, is the Help Favorites tab. When you are reading any topic, you can right-click in the reading pane and select Add to Help Favorites. The link that you can see in the top bar is stored with a friendly name in this tab.

Figure 2-4.

[View full size image]

You can search Books Online by clicking on the Search tab in the reading pane. Enter the search information and Books Online will search local sources as well as other systems on the Internet, if you are connected. As you can see in Figure 2-5, this system is not connected to the network and so cannot browse those sources.

You will discover much more to Books Online as you use it, but I want to point out two more features. The first is located in the icon bar and is called How Do I…. This button has two modes. If you click the button graphic, the results default to the SQL Server engine. If you click the pull-down arrow, you can select another feature, such as Notification Services. When you click the button, you are taken to a section of Books Online that lists references to common tasks and topics.

Figure 2-5.

[View full size image]

Another useful feature is called Ask a Question. If your system is connected to the Internet, you can post a question to the user forums on SQL Server 2005. This is a useful feature, and I have found the community to be responsive. Just remember when you become a guru to give a little information back to those starting out.

SQL Server Configuration Manager

Installing the tools on a server or client installs the SQL Server Configuration Manager. It performs two major tasks: It controls the state of the services running on your SQL Server and configures client and server protocols in the network layer of SQL Server.

When you first open the Configuration Manager tool, you are placed in the service-control area, where you can see which services are running on your system and control them. In Figure 2-6, you can see that I have the database engine with SSIS installed on this system, so the services I am showing are SSIS, the Microsoft Full-Text Search engine (msftesql), the default instance of the engine (MSSQLSERVER), the SQL Server Browser that allows server information enumeration, and the SQL Server Agent, which performs automated tasks. If I had more options installed on this system, I'd show various other services here, too.

Figure 2-6.

[View full size image]

Each instance will also get another service for its engine, in the format MSSQL$InstanceName. Notification Services also gets an instance name, with NS being the prefix for the default instance and NS$ for each instance after that. SQL Server Agent also follows this instance-naming convention, with SQLAgent prefixing the default and SQLAgent$ for the rest.

Right-clicking any object brings up a menu for that item that allows you to stop, start, or in some cases, pause, resume, or restart the service. In the case of the database engine, pausing the service has the effect of completing running tasks but refusing any new connections. You will not see this used very often in the field, especially on a production server.

Another option on the right-click menu of an object is Properties. You can see that in Figure 2-7.

Figure 2-7.

This menu sets the startup parameters for the service and allows you to set the logon account. You can use the System account, which has full machine privileges but not e-mail, share, or network permissions, or you can use a user account. My preference is to use a user account with appropriate permissions so that I can send mail, connect to shares for backups, and so forth.

I normally use the same account for the SQL Server 2005 Engine, Agent, and Notification Services, but I use a different one for each instance. That allows me to set different permissions for each instance, based on what I want them to be able to access. As I mentioned in Chapter 1, this account does not need to be an administrator.

Other tabs on this panel include Service, which displays the file paths and other information, as well as the service startup mode, and Advanced, which changes based on the object you are looking at. In the case of the SQL Server engine, it sets the startup parameters, which tells the server to start up in single-user mode, secure mode, or multi-user mode (the default). You will also find version numbers, clustering indicators, and more on this tab. On some objects, such as SSIS, there are no advanced parameters at all.

Once you close out that panel and return to the Configuration Manager and expand the tree, you can see the protocols for the server and client. If you have installed SQL Server and tried to connect to it from a client, you will find that the server does not answer. When you walk to the server and connect from there, it works fine.

The reason is that by default the only protocol SQL Server enables on this Edition is Shared Memory, which does not use the network. TCP/IP is not enabled, which Microsoft calls "secure by default." That varies, as I mentioned in the last chapter, based on the edition of SQL Server 2005 you've installed. Figure 2-8 shows the default settings for the Developer Edition.

Figure 2-8.

[View full size image]

You can enable any protocol from here by right-clicking it and selecting Enable. Remember: Do not enable any protocols you do not need.

SQL Server Management Studio

We will spend a lot of time in the SQL Server Management Studio, so I use this section to introduce the basics of this interface. In the "Take Away" section at the end of this chapter, I show you how to perform a few representative tasks that you can try on your own development or testing server.

Microsoft has made sure that there is a consistent interface look and feel between SQL Server Management Studio, the Business Intelligence Developer Studio, and Visual Studio. The idea is to be able to share the same code base for an integrated development environment (IDE) and the management tools for SQL Server.

You will find yourself working with Management Studio in a couple of ways. The first is to create a project, which is a connection, a group of T-SQL and other files used in a larger setting. The other is to type in a query or run an immediate command. I will be working in the both modes in the "Take Away" section, so in this section I will show you the basic parts of the tool that you will be working with.

The SQL Server Management Studio is installed by default when you install the server software, but you can also install it on a workstation using the same installer and selecting the Client Tools option when you get to that selection panel. Most administrators install all the client tools on their local workstation and then manage their servers from one location.

You can launch the SQL Server Management Studio from the Windows Start menu under the SQL Server 2005 item. When you launch it, you are asked to log in to a server. You do not have to log in to a server to use the tool, but none of the commands you type will execute until you do. In Figure 2-9, I am connecting to the default instance of SQL Server 2005 on my test system. I have used the word (local) to indicate the local system. I have also set the credentials to the sa SQL Server account and provided the password for that account.

Figure 2-9.

You can connect to older versions of SQL Server as well as SQL Server 2005. To connect to a server, you just type its name in the first box. I have entered the (local) address, which directs the connection to the machine I am working on. I have also expanded the options for this connection so that you can see the connection tab where you can set things such as timeout limits.

If you are working with something other than the default instance of SQL Server 2005 as I described earlier in this chapter, you type the name of the server where the instance is located and then the name of the instance, separated by a backslash, like this:

TheServerName\TheInstanceName

You can connect to as many servers as you like from this tool, and if you are going to connect to the same systems fairly often, you can register them. When you register a server, SQL Server remembers the connection information, and the security information can be saved, too. If you are in a large enterprise or are administering lots of servers, you can also group the connections into a logical name set. I cover the security options in Chapter 4, so for now just select Windows as the authentication method.

Before I explain the parts of the screen in SQL Server Management Studio, I will take a moment and explain how to register a server. In this example, I have not connected to a server, but instead I have clicked Cancel on the main login panel. That drops me into the Management Studio with no connections. To register a server, you can press Ctrl-Alt-G or click on the View > Registered Servers menu item. A small panel will open on the upper-left part of the Management Studio called Registered Servers. The top object within that area is used to connect to the database engine of a server instance. If you connect to a business intelligence instance, you will see another object below this one.

To register a new server, right-click the Database Engine object and select New and then Server Registration. Type in the appropriate connection information and any description text, and then click the Test and Save buttons. You can also right-click the server's registration after you create it and change its properties to have a friendly or descriptive name.

To group your servers, follow the same process and select Server Group rather than Server Registration on the New… menu item. After that, you can move server registrations into the group by right-clicking the registration name and selecting the Move to… item or by right-clicking that group name and adding the server from there. You can also nest groups within groups.

When you connect to a server, a few panels display. For those familiar with the visual tools found in other Microsoft development environments, this one works much the same way. Depending on whether you have registered a server, you will have two or three. The panels on my system are shown in Figure 2-10.

Figure 2-10.

[View full size image]

The environment is arranged in the typical menu bar, icon bar, and main window arrangement. You can access the menu by pressing the Alt key once or clicking it. You can hover over any icon to see its meaning, and clicking in any panel gives it the focus.

By default, you will see the Object Explorer on the left. This panel allows you to navigate through the various objects on the server you have connected to. At the upper-right side of this panel, and even on some others, you will see three small icons. The first is a down-triangle, which provides a drop-down menu that will set the location for that panel. You can have it float, meaning that it appears as a separate window on the Management Studio. If you do that, you can dock it again by using the Window menu item. Move the smaller window around until the shading indicates where it will dock. You can also set the window to use tabs, which is useful if you need more screen real estate.

Next to the position menu is a push pin, which sets the auto-hide feature. With the pin "pushed in," the panel is always displayed. With the pin "pulled out," the panel slides away when you type or click another part of the screen and leaves a small tab to get it back. If you do not have multiple monitors or a huge screen with great resolution, this is usually a good way to work. In the last graphic, I have two tabs on the right set up this way.

You can open subobjects in the Object Browser by clicking the plus sign next to them. To perform an action on an object, right-click it. The menu items that appear will either have three periods beside them (called ellipses) or not. If they do, there is another set of selections to answer. If not, the action displayed on the menu happens with no further dialog when you click it.

As you click any object in the Object Browser, you see the information about that object in the right side of the screen, starting with the Summary panel. This part of the screen is tabbed by default, so as you click objects and perform actions on them, tabs open here to show the latest result of what you are working on. Its display is context sensitive, so the view depends on what you clicked in the Object Browser.

Selecting the View menu item displays all the panels you can open. You have already seen the Object Browser, the Summary panel, and the Registered Servers views. The next view, Template Explorer, shows predefined T-SQL statements to perform common tasks that you can use and even add to.

To use a template, just double-click the item that matches what you want to do. If you are not already connected to a server, you will be prompted to connect. You can click Cancel and still view and edit the script, but you cannot run it until you connect to a SQL Server instance. The text shows up in a tab in the middle panel.

As you explore the template, you will notice that there are some items that have less-than and greater-than symbols (< >) around them. These are the variables that you need to replace to make the template work in your situation, such as a table name or other parameter. You can work through the script to find and replace them, or you can click the icon (in the icon bar) that looks like an A-to-B conversion, or press Ctrl-Shift-M. The last two options bring up a panel where you can enter all the variables at once and will replace all the parameters for you throughout the script.

After the template is filled out, nothing happens until you press F5 or click the green triangle in the icon bar. I explain a little more about that when I discuss queries in the "Take Away" section.

To create your own templates, right-click the SQL Server Templates object in this panel or any of the folders beneath them, name the template, and then type the T-SQL you want to save. These templates are local, so they will only exist on the system where you are typing them.

We will use templates shortly. For now, I will close the Template Explorer view on my system and bring up the Project Explorer view from the View menu. A project is a collection of connections, scripts (which are recorded T-SQL statements), and other files grouped together under a single name. I explain how to create a project in the "Take Away" section. What is important to note here is that when you invoke this view and open a project, you will see another object browser on the right side of the Management Studio. This panel has Connections, Queries, and Miscellaneous folders where you can store the objects associated with your project.

Closing that panel often leaves a Properties view. You can also invoke a Properties view from the View menu. Property views are panels that allow you to change various settings based on the object you are looking at. The Properties page has items on the left and values on the right. If an item is grayed-out, you cannot alter that setting.

The next item on the View menu is the Bookmark Window. You can also press Ctrl-W to access this panel.

When you are working through large code windows, it can sometimes get a little confusing. You can place a comment in the code either by using and in-line comment using two dashes () on a single line or by a block comment, where you enter a slash and then asterisk to start a comment and an asterisk and a slash to end it, like this:

/* This is a block of comments, and the text inside does not
execute.
In fact, it will turn a different color inside the
Management Studio. */

You can then search through those comments to locate notes about them.

Using block comments is usually the safer method, even for a small comment. Dashes only cover the current line, and when you ship your scripts around, it is fairly easy to have a line feed inserted where you did not expect. With a block comment, that is not a problem.

An alternate way of marking sections of your code is to use a bookmark. To create a bookmark in your code window, position the cursor anywhere on that line and press Ctrl-K. To view or traverse these bookmarks, open the view (Ctrl-W) and click any bookmark. You can even create folders for these bookmarks.

The last view I cover here is the Toolbox view. The Toolbox panel opens as a blank slate and allows you to put common maintenance tasks in icons so that you can arrange them as you want. To add an item to the Toolbox view, right-click inside the panel and select the Choose Items… menu option. You will get a list of items to add to the toolbox.

To close out our tour of the SQL Server Management Studio, I explain a few of the settings you can change to work within the environment. To view and change settings, access the Options item from the Tools menu.

The first panel changes general environmental options such as window behavior. You can also change the fonts and colors for the Management Studio. You normally will not have to do that because the Management Studio automatically uses a separate font and color of various text states. For instance, comments are shown in green and in the Courier font. You will find yourself relying on this color coding quite a bit; so if you are familiar with another color scheme, you can change these settings to match what you are used to.

Another useful set of options is in the Help section. You can have the help engine search the local files and even the Internet when you invoke it.

You can set many other options, from adding source control systems such as Visual SourceSafe to the default output for query results to grid, text, or file. I revisit these options in later chapters as we need to change them.

Command-Prompt Utilities

You can use quite a few command-prompt utilities in SQL Server 2005. I cover the two primary tools you will use: sqlcmd and bcp. I also list other utilities and cover their use and parameters as I demonstrate them throughout the book.

sqlcmd

The sqlcmd utility is a command that is similar to the Query window in SQL Server Management Studio. You can connect to one or more servers, run a query, and receive the results, all from the operating system command line. It is a bit more flexible than the Management Studio because you can create script files and pass them variables and call those scripts using a single statement.

The sqlcmd utility uses the SQL Native Client to connect to the server, so you have access to the CLR functions of SQL Server 2005, too. Other scripting languages, such as VBScript, can call the sqlcmd utility and even receive results from it.

There are two modes of working with sqlcmd: interactive and scripting. To use the interactive mode, enter the sqlcmd command at the operating system command line with the connection information for the server. For the scripting mode, call the same command with the switches that run a script. I show you both in a moment.

In the interactive mode, you can enter commands, separated by the word GO and press Enter until you are done. When you have finished, type the word EXIT and you will leave the utility. In the scripting mode, your code runs and then exits the process.

Throughout this chapter and the ones that follow, I will uppercase certain words, such as GO and EXIT. Unless your system is installed with the case-sensitive option, you don't have to do that. Even then, the only case sensitivity is in the data, not the commands. I capitalize all letters in a reserved word to make the scripts more understandable.

If you are using the utility on the system where SQL Server 2005 is installed, you can just enter sqlcmd from the command line and then start entering commands. I have done that on my test system in Figure 2-11.

Figure 2-11.

[View full size image]

There are quite a few switches with the sqlcmd utility. You can get a readout of them by typing the following command

sqlcmd -?

On my test system, you can see the output shown in Figure 2-12.

Figure 2-12.

[View full size image]

You will not always need all these parameters, so I show you the basics here. I return to this tool throughout the text and demonstrate a few of the other parameters. For now, let's take a look at a simple example that runs a query and shows the databases installed on this system:

C:\> sqlcmd -S (local) -E -d master -Q "SELECT name FROM
sys.databases"
name

master
tempdb
model
msdb

Let me explain these switches. Unlike most Microsoft Windows paradigms, case matters. A small letter in the switch does something differently than a capital letter. The -S specifies the server and instance name. I have set my parameter to (local), but when you are using sqlcmd on the server itself, this really is not necessary.

The -E parameter enables trusted security, meaning that the Windows account is used to access the server. Again, if you are using this command from the server where SQL Server 2005 is installed and your Windows account is allowed into the server, you do not need to use this parameter. If you are not using trusted authentication (more about that in Chapter 4), you need to pass both the -U (username) and -P (password) parameters.

The -d parameter sets the database that you want to run the query against. The -Q followed by quotation marks contains the query you want to run. As I mentioned, the case of the parameter is important because an uppercase Q runs the query and returns to the operating system, and the lowercase q runs the query and stays in the sqlcmd interactive mode. If you make a mistake and use the small q, just type EXIT to leave.

A script file, as explained earlier, is just a text file of T-SQL commands. Without delving into a great deal of programming, if you place a GO between blocks of T-SQL code, the block will run as a batch. A batch in code is similar to a transaction. Batches can also limit the amount of data that is read, inserted, updated, or deleted at one time. Keeping the batch size small allows best use of the transaction logging process.

There are two methods of running a script. The first is to use a sqlcmd command, which is a colon followed by a letter. Several commands are available (see Books Online for a complete list), but the command to read and execute a file is :r. In the interactive mode, you can read and run a file by entering the filename after the :r, shown in Figure 2-13.

Figure 2-13.

[View full size image]

In this example, I have created a simple script at the command line and then run sqlcmd in interactive mode. I then invoked the :r command to load and run the script.

You can run a script without having to use interactive mode, too. All you have to do is add a different switch instead of the -Q parameter I demonstrated earlier:

C:\> sqlcmd -S (local) -E -d master -i C:\test.sql

The -o switch outputs the results of the query to a file rather than the screen. To send the results of the preceding query to a file called results.txt, use this command:

C:\> sqlcmd -S (local) -E -d master -i C:\test.sql -o
C:\results.txt

bcp

The bcp (bulk copy program) command is used to copy data from a table, view, or query to a file or to insert data from a file. Along the way, you can specify various formats for the file, from comma-separated values to a native format, sort of like a backup. The text file can be used with just about any other program, but the native type only works with SQL Server 2005.

The bcp utility moves data very quickly. It is one of the fastest methods around to move data in and out of SQL Server.

Once again, there are many switches, and like the sqlcmd command, they are case sensitive. Although the switches do not match the sqlcmd utility, the -? works the same way, as you can see in Figure 2-14.

Figure 2-14.

[View full size image]

The primary switches you will use at first for this overview are in, out, queryout, -f, and -o.

The format of the command to copy data from the sys.databases table to a text file looks like this:

C:\>bcp master.sys.databases out "c:\test.txt" -T -c

In this example, I have specified the bcp command and then the database, owner, and table that I am after. Next I told the utility which direction the data should goin this case, out. I also specified the name of the file that will hold the data. The -T (notice the uppercase) sets a trusted connection, and the -c sets the column types to the character, or text format. If I you use -n, the file will be stored in native mode, which is a binary format only SQL Server can read.

To export data using a view, you just specify it as if it were a table. To export data using a query, use the queryout switch, which looks like this:

C:\>bcp "SELECT name FROM master.sys.databases" queryout
"c:\test.txt" -T -c

The -f switch tells the bcp utility to use a format file, which is just a text file that allows you to use other column names and do field mapping. Here are the first few lines from a sample format file I created:

9.0
4
1 SQLCHAR 0 15 "\t" 1 Name ""

The first line is the version of SQL Server (6.5 = Microsoft SQL Server 6.5, 7.0 = Microsoft SQL Server 7.0, 8.0 = SQL Server 2000, and 9.0 = SQL Server 2005), and the second line sets the number of columns.

The next line starts with the column number in the destination, then the data type, which in this case is SQLCHAR, or text. (For a list of all data types, see Books Online under the topic of "Specifying File Storage Type by Using bcp.")

Next comes the prefix length, which can be 0, 1, 2, 4, and 8. For the most part, you will use 0. The number 15 in this example represents the length of the field.

After than comes the field terminatorin this case, a tab (\t). The other legal values here are comma (,) and a newline (\r\n). The rest of the file contains more lines that specify each column in the same way.

The 1 that follows the "\t" is the number of the column in the source data, and the word Name in this case is what the column will be named in the destination.

The final quote signs indicate the collation. Values here are the same as in SQL Server, such as SQL_Latin1_General_CP1_CI_AS.

There is a lot of tweaking to be done when you use one of these files. Books Online has a full rundown on the options and also shows more sample files.

There is also an XML version of this format file that will "shred" XML documents into the database. I deal with that in another section of the book.

Along with sqlcmd and bcp, other useful command-line utilities include those listed in the following table.

Command | Used For
osql | This is an older command similar to sqlcmd. It uses an older library to access the server and does not have as many options as sqlcmd.
cidump | This command lists the contents of an indexing service catalog.
DTexec | This command configures and executes a Data Transformation Services (DTS) package.
dtutil | This is used to manage DTS packages.
dtswizard | This command starts the DTS Import/Export Wizard.
dta | This command is used to analyze a workload. It can also recommend another physical design.
lrtest | This command invokes the Full Text Search.
nscontrol | This command creates and controls Notification Services.
profiler90 | This command starts the SQL Profiler from a command prompt.
rs | This command runs scripts for Reporting Services.
rsactivate | This is another command for Reporting Services and initializes a report server.
rsconfig | This command configures a report server connection.
rskeymgmt | Another Reporting Services command tool, this one manages encryption keys on a report server.
sqlagent90 | This command starts SQL Server Agent.
sqlservr | If you need to start or stop the SQL Server database engine from the command prompt, use this command with options to specify how it will start. I explain this command further in Chapter 3.
sqlmaint | Maintenance plans are groups of jobs used to optimize and back up a database. I explain these further in Chapter 3. This command is used to run those maintenance plans created in previous versions of SQL Server.
tablediff | This is a welcome new command that can compare the data in two tables. It is used a great deal in troubleshooting replication.

SQL Server Profiler

SQL Profiler can capture and display any or all of the statements that are running on your server for a given period of time for the database engine (and the business intelligence engine). You can save this information out to a file or in a table or you can get immediate feedback as you use the tool.

SQL Profiler is useful for tracing problems in code, identifying long-running queries, or even just finding out how the statement is processed by the server. It lets you capture events that happen on a SQL Server and filter them by various constraints. You can run Profiler on the server or from your workstation against a server.

I follow the same process here as I did in the previous section on the SQL Server Management Studio, showing you a high-level view of the interface, with an application in the "Take Away" section. We use Profiler again in Chapter 5.

You do not have to be a system administrator to run Profiler. You have a special permission that you can grant to allow users to run the Profiler. It is important, however, to grant this permission with caution. Anything that is not encrypted is visible to Profiler. SQL Server 2005 passwords and integrated security passwords are encrypted, but a developer might choose to implement security by passing names and passwords to a table in SQL Server, and that would be visible to the Profiler.

You can find SQL Server Profiler on the Start menu under Microsoft SQL Server 2005 > Performance Tools > SQL Server Profiler.

When you first open the tool, you are presented with a large blank screen. Capturing a group of events for a period of time is called taking a Trace, and the output of the session is called the Trace. To try it out, click on File and then New Trace. Connect to a server, and you are presented with the screen shown in Figure 2-15.

Figure 2-15.

[View full size image]

In this tab, you name the trace and set where the output for it will go. If you do not choose a file or table, your trace will only capture data for display on the screen. You can save the data after it is captured if you want. You can also set the trace to stop at a certain time. This is useful for tracing for a long period of time. There are also ways of automating traces, which I explain later.

The first tab also contains the choice of Templates that the trace will work with. A template is a saved group of settings that perform common tasks. For this example, leave this set to the default template, which captures login and logout events as well as T-SQL statements.

The Events Selection tab, shown in Figure 2-16, groups the events you can capture into your trace. You can select the items you would like to capture, and you can apply various filters to restrict what you collect. As a rule, you should collect only what you need, so most filters remove things such as the Profiler itself, system calls, and so forth.

Figure 2-16.

[View full size image]

Leaving the defaults here, all you have to do is start the trace is to click the Run button. After I started this trace, I switched over to the SQL Management Studio and ran a simple query. I logged out of Management Studio and then stopped the trace.

You can see the results of my session, shown in Figure 2-17.

Figure 2-17.

[View full size image]

Even though I simply connected and ran one query, you can see there is quite a lot of activity captured. Without the filters built in to this template, there would be even more. What I have highlighted in this graphic is the connection environment information sent to SQL Server 2005 by the client (in this case, Management Studio). Even this connection information is valuable, because it determines how the server will process the query and return the results.

Clicking the SQL:Batch Starting statement that has my query shows what was sent to the server, as you can see in Figure 2-18.

Figure 2-18.

[View full size image]

Off to the right of this panel (not shown in this graphic) are other columns that show the start time and duration of an event, the name of the user, the workstation name, and so forth. Filtering for these columns allows you to see long-running queries, as I explain in the "Take Away" section.

If you did not set the data to be captured directly to a file or table, you can save the data on your screen to either of those or even an XML output. Saving the trace allows you to evaluate it later or even "play it back," by running the trace against the server. This makes Profiler a testing aid in addition to a monitoring tool. You can also save the settings you have adjusted as a new template, and reuse those settings later.


Take Away

The information you learned in the first part of this chapter will help you understand, diagnose, and design your system from a conceptual standpoint. In this section, I demonstrate how you can combine that knowledge with a few of the tools explained this chapter in a practical way.

I begin this exercise by showing you how to create a project, move on to creating a database, and then populate that database with a table. I populate that table with a few values using a stored procedure, and then I create a view to query the table. You can follow along on your test or development system in your environment. This is not just an exercise, however. You can actually use this setup to create a useful database for your systems. You can use the information you find throughout this book to extend what you capture and track using this database.

Every database project should start with a plan, and this project is no different. Because we are focusing on an administration view of SQL Server 2005, I do not spend time covering that plan now. Along the way, I make comments about the design, but the idea here is to learn more about what we are doing and how we are doing it than why.

This database is one I create in most of the systems I manage. It provides a historical view of metadata that you can use to monitor one or more servers. I demonstrate some code along the way, but it is not necessary to understand each line of the scripts just to learn to use the tools.

Most of the data you will store in this database derives from system functions that you can run any time. So why store that data if you can get it any time? The reason is that some of the data changes over time. Storing snapshots of this information allows you to watch the trends and predict growth patterns.

Other data, such as a server's name, does not change, at least not often. Storing the name of the server in a table anyway allows you to create a "master" version of this system to monitor lots of servers, and create system-wide metrics such as overall growth patterns, which databases have not been backed up recently, and so forth.

Even if you do not plan to use this database, read through the examples. Nothing helps you learn faster than actually doing the work.

SQL Server Management Studio Tasks

SQL Server Management Studio is the primary tool you will use to interface with your system. Just as in real life, real estate is king. Find a method of working with the screens that you like and are comfortable with. At first there are so many options, panels, and information displayed that it is all a bit overwhelming. Take your time and practice with the tools and you will be comfortable with it in no time.

You can group the things you are working on into a solution. Within Solutions are projects, which are groups of connections and files that form a unit of work. Our example begins with a solution consisting of a single project.

Creating a Project

Microsoft provides a sample database that you can install on your server called AdventureWorks. I make use of that database in several of the chapters, but I also create a database that is much smaller. Although this is not a database design book, it is useful to know a little about the objects on your system, so I am going to show you how to create a database for yourself. You can follow the examples I have here and create your own copy of this system.

You do not have to create a project in SQL Server to work with a database. If you are just running a backup or some other maintenance task, it is not necessary to create a project. You can use the Object Browser or run a query from sqlcmd to do what you want, and then close the connection when you are through.

Projects collect information about connections to a system and store scripts and other files in one place. SQL Server Management Studio stores the connections and files in a directory on your system so that you can group them and back them up, or transport them to another system. If you plan to work iteratively through a software development cycle, then projects are useful.

To create a project, open SQL Server Management Studio from the Start menu. Click File, then New, and then Project. You could also just press Ctrl-Shift-N once inside the Management Studio.

You are given a panel that sets the name, location, and type of project to create. Select SQL Server Scripts as the type and name the project DBAMDT (Short for Database Administrator's Meta-Data Tool). Enter the location to save the project and allow the system to create a directory for it.

You now have a project with three empty folders: Connections, Queries, and Miscellaneous. Right-click the Connections folder and select New Connection from the dialog box. Complete the panel that pops up with the name of the server where you will house the database and enter your security information.

That forms the beginning of your project. Next, you will run some scripts to create objects on the server using the connection you just defined.

Creating a Database

I break out this project example into three sections. In the first, you create the objects, and in the second, you populate it. In the third phase, you use the view you create to query the objects you have created.

Although you can quickly and easily create a database graphically, in this case you use a template to create the database. You can then transport this script easily from server to server, and update it and track it as the versions change.

Click View and then Template Explorer in the menu bar. Drill down to the Database, and then create a database template and double-click it. You will be asked to connect to a server again. Go ahead and do that.

I am going to ask you to change the text of this script a bit. As mentioned earlier, I like to use block comments rather than in-line comments, and I will add some specific code to get the result we want. Edit your script to look like this:

/*
Create_DBAMDT_DB.SQL
Use:   Creates the Meta-Data Tool database
Created:  11/22/05, Buck Woody
Last Edited:
Change Log:
*/

/* Set the database to begin working in */
USE master
GO

/* Drop the database if it already exists.
If you run this again, it will erase what you
have and create a blank database!
 */
IF EXISTS (
     SELECT name
           FROM sys.databases
           WHERE name = N'DBAMDT'
)
DROP DATABASE DBAMDT
GO

/* Create the Database using the default drives */
CREATE DATABASE DBAMDT
GO

On a new system, adding all these comments and checking for the existence of a database is not strictly necessary, but you should do it anyway. It develops good habits.

This script creates a database on the default drive with a minimal size. Later in the book, you will learn to move the files to another location and change the size.

To save this script into your project, click File, then click Move SQLQuery1.sql Into… and then click DBAMDT. Name the script Create_DBAMDT_DB.SQL. If you hide the Template Explorer pane and switch to the project view, you will see the file in the Queries folder.

Press F5 or click the Execute button to create the database.

Creating a Table

Now you will create a table to hold the server's information. We will be adding to this structure throughout the book, so for now focus on interacting with the tools.

Slide out the Template Explorer pane again and locate the Table and then Create table template. It is a little wordy, so edit it to look like this:

/*
Create_DBAMDT_Tables.SQL
Use:   Creates the Meta-Data Tool tables
Created:  11/22/05, Buck Woody
Last Edited:
Change Log:
*/

USE DBAMDT
GO

/* Create table to hold server-wide data */

CREATE TABLE ServerInfo
(
      ServerInfoKey INT IDENTITY(1,1) PRIMARY KEY
,     ServerName VARCHAR(50) NULL
,     DBAName VARCHAR(50) NULL
,   LastUpdated DATETIME DEFAULT GETDATE()
)
GO

To save this script into your project, click File, then Move SQLQuery2.sql Into… and then click DBAMDT. Name the script Create_DBAMDT_Tables.SQL.

Now press F5 or click the Execute button to create the table.

You might have noticed that I keep my commas at the beginning of the line. I do that so that I do not forget any. If you would rather place yours at the end, that is fine, too.

You are going to use the same script file to create all the tables, so I have indicated that in the header comments. I also took out the table check and DROP statements to make the code shorter.

All you are storing so far is a key to the table, the name of the server and administrator and a timestamp every time changes are made to the table. The DEFAULT statement on this field enters the date each time you change a row.

We do not have enough columns to store the data you will want to see. You will want to capture things that can change over time, so we need to add an operating system, service pack level, and server information (such as processor, memory, storage information, and network card type). We can continue the process of entering more fields by using T-SQL code, but there is another way to alter the table and produce a script at the same time.

Open the Object Browser, and then drill down to the table called dbo.ServerInfo. Right-click that table and select Modify from the menu. Now you can graphically add the columns you want to add, as you can see in Figure 2-19.

Figure 2-19.

[View full size image]

After you have made your changes, click the button on the far left, third line from the top, called Generate Change Script. It looks like a small scroll over a computer disk.

Select the part from ALTER TABLE to the next GO and copy it with Ctrl-C. Now you can paste that into the original script. You do not need to save it out to file because you are pasting what you care about in the main script. You can see that in Figure 2-20.

Figure 2-20.

Paste the results at the bottom of the script, which should be still available as a tab in the main panel. The table script should now look something like this:

/*
Create_DBAMDT_Tables.SQL
Use:   Creates the Meta-Data Tool tables
Created:   11/22/05, Buck Woody
Last Edited:
Change Log:
*/

USE DBAMDT
GO

/* Create table to hold server-wide data */

CREATE TABLE ServerInfo
(
      ServerInfoKey INT IDENTITY(1,1) PRIMARY KEY
,     ServerName VARCHAR(50) NULL
,     DBAName VARCHAR(50) NULL
, LastUpdated DATETIME DEFAULT GETDATE()
)
GO

ALTER TABLE dbo.ServerInfo ADD
      OperatingSystem varchar(25) NULL,
      ServicePack varchar(15) NULL,
      CPU varchar(30) NULL,
      Memory varchar(20) NULL,
      Storage varchar(55) NULL,
      Network varchar(50) NULL
GO

Use the mouse or keyboard to highlight the lines you just pasted, starting with the ALTER TABLE statement all the way through the last GO statement. When you highlight an area of a script and press F5, only the highlighted portion of the code runs. Do that now.

Press Ctrl-S to save the changes to the script.

Now you have seen three ways to create your scripts: using templates, typing them directly, and designing objects graphically and creating a script from there.

Creating a Stored Procedure

You can insert data into the table easily with a graphical tool or by using an INSERT statement. I am going to show you how to create a simple stored procedure to do the same thing, to demonstrate how stored procedures work.

First, let's take a look at a statement that will insert data into the table:

USE DBAMDT
GO

      INSERT INTO ServerInfo (
      ServerName
,     DBAName
,     OperatingSystem
,     ServicePack
,     CPU
,     Memory
,     Storage
,     Network
)
      VALUES (
      'BuckSoftTS1'
,     'Buck Woody - woodyb@hotmail.com'
,     'Windows 2003 Server, 64-bit'
,     'None'
,     'Athlon x64'
,     '2048'
,     '200GB SCSI Internal'
,     '1Genet'
);

Do not concern yourself with the actual syntax at this point, just that some of the information stays the same, such as INSERT INTO, GO, and so forth, and others change, such as the server name, the operating system, and the like.

To keep the things that will stay the same but allow different inputs, you can use a stored procedure. Using a stored procedure also allows you to change the table later and still access it with the same commands.

To create a stored procedure that does the same thing as the INSERT statements while allowing variables, right-click the Stored Procedures object under the database name and then Programmability. Click New Stored Procedure on the menu that appears, shown in Figure 2-21.

Figure 2-21.

[View full size image]

Edit the stored procedure to have this text:

/*
Create_DBAMDT_Stored Procedures.sql
Use:   Creates the Meta-Data Tool Stored Procedures
Created:   11/22/05, Buck Woody
Last Edited:
Change Log:
*/
USE DBAMDT
GO
CREATE PROCEDURE usp_Insert_Server_Info
       @ServerName VARCHAR(50)
,      @DBAName VARCHAR(50)
,      @OperatingSystem VARCHAR(25)
,      @ServicePack VARCHAR(15)
,      @CPU VARCHAR(30)
,      @Memory VARCHAR(20)
,      @Storage VARCHAR(55)
,      @Network VARCHAR(50)

AS
/*
usp_Insert_Server_Info
Use:    Inserts new data into the ServerInfo table
Created:   11/22/05, Buck Woody
Last Edited:
Change Log:
*/

BEGIN
       SET NOCOUNT ON;
       INSERT INTO ServerInfo (
       ServerName
,      DBAName
,      OperatingSystem
,      ServicePack
,      CPU
,      Memory
,      Storage
,      Network
)
       VALUES (
       @ServerName
,      @DBAName
,      @OperatingSystem
,      @ServicePack
,      @CPU
,      @Memory
,      @Storage
,      @Network
); v

END
GO

Placing the comments below the beginning of the stored procedure ensures that the text will stay within the text of the stored procedure. Later I show you how to display this text.

Move that script into the project, with the name Create_DBAMDT_Stored_Procedures.sql.

Create the stored procedure by pressing F5 or click the Execute button in the Query screen.

To use the stored procedure and enter some data, open another Query window (by pressing Ctrl-N) and type the following statement, using your own variables in between the ' and ' (called tick marks):

USE DBAMDT
GO
EXEC usp_Insert_Server_Info
      'BuckSoftTS1'
,     'Buck Woody - woodyb@hotmail.com'
,     'Windows 2003 Server, 64-bit'
,     'None'
,     'Athlon x64'
,     '2048'
,     '200GB SCSI Internal'
,     '1Genet'
GO

Now you have data in the table and can use this stored procedure to insert more when it changes. That will maintain a history of the server name, operating system parameters, and administrator information. The primary key and date stamps are inserted for you automatically.

Creating a View

You can get data out of a table in several ways. You can use a SELECT statement or a stored procedure. I normally advocate the use of views for a couple of good reasons. For one, you can change the underlying structure of the table and not have to recode, just like a stored procedure. A view looks similar to a table, and users are often more familiar with it. Even if the view shows the same data as the table, this abstraction layer is useful.

To create a view that shows the server information, slide out the Solution Explorer and right-click the Queries object. Select New Query from the menu and type this information:

/*
Create_DBMDT_Views.SQL
Use:   Creates the Meta-Data Tool Stored Procedures
Created:   11/22/05, Buck Woody
Last Edited:
Change Log:
*/
USE DBAMDT
GO
CREATE VIEW vw_ServerInfo
AS
SELECT
      ServerName
,     DBAName
,     LastUpdated
,     OperatingSystem
,     ServicePack
,     CPU
,     Memory
,     Storage
,     Network
FROM
      ServerInfo
GO

Press F5 or click the Execute button in the Query screen to create the view. Save this script into the project as Create_DBMDT_Views.SQL.

To use the view, you can treat it like a table. Here is a SELECT statement that you can run in a query to return all the rows and columns:

USE DBAMDT
GO
SELECT *
FROM vw_ServerInfo
GO

You will add to these scripts throughout the text. Go ahead and type them inmost people learn better when they create the scripts by hand and make a few mistakes along the way. This kind of code is typical of what you will be asked to do as a DBA. In the next few chapters, I show you how to fill these tables using system views.

SQL Server Profiler

As you saw earlier in this chapter, using the Profiler is not difficult at all. One of the easiest and most valuable uses for this tool is to check for the queries in an application that take a long time to run.

With new coding methods, it is sometimes difficult to check what each application component does, because it will often morph into various uses depending on how it is called. Another reason that developers cannot always tell how fast a query is running is that the server might have multiple applications running against it. You cannot always tell how one query might affect another.

The solution is to monitor what is happening from the back end. The Profiler can tell you exactly what is happening; and in this case, you can limit it to show just the queries that run for a period of time.

Finding a Long-Running Query

To find a long-running query, start the Profiler as before and create a new trace. Connect to your server, name the trace, and as a template select TSQL_Duration.

In the Events Selection tab, you will see that both stored procedures and T-SQL (called Dynamic SQL) have already been selected. All that is left to do is to restrict the length of the queries that the tool will capture.

Click the Column Filters button, and then click the Duration section. Expand the Greater Than or Equal To object and enter the duration you want to filter. This duration is in milliseconds, so calculate accordingly.

Your developers should be able to tell you what value they want to check. If not, measure for things that run longer than 30 seconds and present the results in a spreadsheet or table for them to evaluate.

Be sure and save this session as a template. You will find yourself coming back to it again.


Chapter 3. Maintenance and Automation

After you have installed and configured a server, even before you let the first user on it, you need to create a maintenance plan. Maintenance includes more than just backups and defragmenting a systemit includes audits and checking the system for operational status. In the first part of this chapter, I explain what you need to understand about maintenance and the audits and other documentation you need to create.

In the second half of this chapter, I explain how automation works in SQL Server 2005. Automation is getting the computer to do what it does best: repetitive tasks. Although you will use automation in lots of places, the first logical application is in automating the maintenance your system needs. I show you various methods to automate your system, and I show you a prepackaged wizard that creates a maintenance plan that you can tailor to your own needs. In the "Take Away" section, I provide a checklist for maintenance tasks and a method to automate tasks and monitor them from a central location.

Database administrators (DBAs) perform several functions in an organization: database design and development, creating data structures, and performing system maintenance. Above all, maintenance is at the core of what you are expected to do. System maintenance is part of the umbrella that the term administration covers.

Maintenance is not just backing up databases. That is certainly part of the job, but you will need to do other periodic tasks to guarantee the availability and responsiveness of your system. There are logs to monitor, capacity plans to create, user accounts to manage, and many other things to do to be prepared for the daily emergencies that inevitably arise.


Maintenance

To administer a system means to be in charge of its hardware, software, availability, and security. Administration also involves tracking and managing a system's growth. In this section, I break those responsibilities down into tasks. These tasks form the maintenance that you are required to perform.

As I introduce each concept, I explain the theory behind it, followed by practical examples. In the "Take Away" section, I include a sample checklist you can start with to manage your own systems. In the section that follows, I explain how automation can help you with these tasks.

It is important to remember, however, that automation is not a substitute for actively managing your system. The automation I show you will collect the information you need, but it will not necessarily act on it, nor should it. Blindly "correcting" issues can lead to real trouble. I explain more about that in a moment.

I begin with a discussion of the Recovery Model of the database, which affects your backup and recovery strategies. I then explain how to take a backup and perform a recovery, and then discuss consistency checks. I also cover re-indexing strategies.

Although not technically part of maintaining a particular database, recording an audit is another part of system maintenance. I explain what you should audit and how to do that. Finally, I explain how to check for service packs and hot fixes, and when you should apply them; I also cover planning for system growth.

The Recovery Model

In Chapter 1, "Installation and Configuration," I mentioned a setting on a database that determines how the log file is used, called the recovery model. Let's complete that discussion now.

At its most basic, the recovery model setting on a database determines when an entry is erased from its Transaction Log. Tracing a transaction from an application to the database, the rows of data are first written sequentially to the log file, and then on to the database. The recovery model setting determines what happens to the transaction after that.

Simple

If the database is set to the simple recovery model, then after the transaction is written to the database, the entry is deleted from the Transaction Log. In this situation, the Transaction Log grows larger and then shrinks as entries are written to it and then erased. No additional maintenance is needed on the log file. The simple recovery model is quick and efficient. The problem with it becomes apparent when you imagine a backup and restore scenario.

Let's assume that an application writes datawe will call that transaction Aat 11 in the morning. The entry is written first to the Transaction Log and then on to the database. After it arrives safely in into the database, the entry is removed from the log. Transaction A is now only in one location: the database.

Now let's assume that you back up the database at noon. You now have nothing in the log file, transaction A in the database, and transaction A on a backup device such as a tape or in a file system.

Now assume that between noon and 1 o'clock two more transactions are written to the log and then the database; we will call them B and C. After making it safely to the database, they are then erased from the log file. You now have transactions A, B, and C safely within the database, and transaction A still on tape. Nothing is in the log file.

At 1:30 the unthinkable happens; the hard drive dies. You replace the drive and restore the database from tape. Now what do you have in the database? Just transaction A. You have no way to get B and C back. The only place they were stored is gone.

As you can see, the simple recovery model is useful for seldom-changing databases or those that are read-only and completely backed up somewhere else. It is not safe for databases where a loss of data would be a problem, unless you take full backups frequently. Because full backups have a performance impact, you don't want to run them when the system is under heavy load.

Full

The full recovery model overcomes the limitations of the simple model but requires a little more maintenance work. In this model, the entries in the log file are retained in the log even after they are stored in the database and are only removed when you perform another backup operation, either on the database or just the log file itself. Let's assume the database is in the full recovery model and then replay our scenario from above.

Once again, transaction A is entered first to the log and then to the database. The entry now exists (because of the full recovery model setting) in both the database and the Transaction Log.

At noon, you take a complete database backup (called a full backup). This has the effect of copying transaction A to a backup device and erasing the entry from the log. In other words, the tape contains the entire data set as it exists at a point in timehence the name full backup.

Once again, two more transactions (B and C) are written to the log file and then on to the database. Because of the full recovery model setting, this time the entries are not erased from the log file. Now you need to set up log backup operations. Let's assume you have set them to run once an hour. This type of backup just copies the entries from the log file, but not the whole database. The log backup operation then releases the transactions to be deleted. Let's review where all the data is located now.

The database contains A, B, and C. The full backup tape contains transaction A. The log backup tape contains transactions B and C. The log file is empty.

Once again, your troublesome server loses a drive at 1:30. To recover the database, you restore the full backup tape, without "closing out" the restore operation. You can allow users to read, but not write, data during this time. Next, you restore the Transaction Log backup tapes to "replay" transactions B and C to the database, and then you close the restore operation. You have now recovered all the transactions as of the last Transaction Log backup to the database and users can continue working.

In fact, even if you did not have a log backup, as long as the database failed and not the device where the logs are, you could still recover the database and then apply the logs to it.

This is a simple example; if your database is quite large, you will not be able to do a full backup several times a day, because you will run out of time or adversely affect the users. Using the full recovery model setting, you can perform a full backup periodically, such as once a day or once a week, and you back up the log more frequently, such as hourly or twice a day. Because the log file contains only the new entries, the backups are smaller and faster and do not affect the users as much.

Of course, you may need even more safety than that. If you need a high-availability solution, you would implement clustering, database mirroring, or replication. I explain those concepts in Chapter 6, "High Availability."

There is one other advantage to keeping the transactions around until you back them up: You can use the transactions to restore your database to a particular point in time.

Every entry to the log is time encoded, and so SQL Server 2005 knows when they occurred. Using a special flag on the RESTORE command, you can restore those entries to a particular point in time. Because of the safety it provides and the point-in-time feature, the full recovery model is what all of your production databases should use.

Bulk-Logged

The bulk-logged recovery model is similar to the full recovery model, with the exception that not every operation is sent through the Transaction Log. This model allows you to bypass the log file for these operations, so they are not recorded in the log:

  • Bulk import operations by the bcp utility, INSERT ... SELECT * FROM OPENROWSET(BULK...) T-SQL commands, and the BULK INSERT T-SQL command.

  • Certain full-text operations such as Text, ntext, and other image operations are not logged if you use the WRITETEXT and UPDATETEXT statements.

  • Using the T-SQL SELECT INTO command.

  • All CREATE INDEX T-SQL statements.

The advantage with this model is that it makes the log backups smaller, and the operations described above run faster. I have used this model when I have a lot of data to import into a database on a frequent basis. In those situations, I first take a log backup, set the database to the bulk-logged recovery model, and then do the import. I take a full backup of the database after the operation and then put the model back to the full setting.

Setting the Recovery Model

There are a few ways to change this setting, but the two you will use most often are using the SQL Server Management Studio and T-SQL commands.

To change the setting using SQL Server Management Studio, open the tool, drill down to the database, and then right-click it. Select Properties from the menu that appears. Click the Options item on the left side of the panel that appears and change the recovery model setting from there. You can see that in Figure 3-1.

Figure 3-1.

[View full size image]

The T-SQL command to do the same operation is as follows:

USE [master]
GO
ALTER DATABASE [DBAMDT] SET RECOVERY FULL WITH NO_WAIT
GO

Change the [DBAMDT] enTRy for the name of your database and the RECOVERY FULL to either RECOVERY SIMPLE or RECOVERY BULK_LOGGED for the other models.

If you want to start out every database that you create with a certain recovery model, then change the model database setting. Remember that model is used as a template when you create a database on the server.

Restoring a database or attaching it does not change the recovery model setting or use the model database.

Backup

Whenever you perform any maintenance task, I advise that you always try to have an exit strategy defined first. The first part of any exit strategy is to ensure that your databases have adequate backups. If you are not checking the system on a daily basis, then open the SQL Server Management Studio and drill down to the properties of the database and note the Last Database Backup date. The database on my test system was last backed up quite some time ago, as you can see in Figure 3-2.

Figure 3-2.

[View full size image]

An alternate method is to use the Report capability on the right side of the Management Studio, in the Summary pane. There is a pull-down for the Report button that shows all backup activity for the database.

If the backup is current, you can proceed on any other operations on the database. If not, take a backup before you do anything, regardless of how small you think the change might be. I show you how backups work in a moment.

You can back up a SQL Server 2005 database or Transaction Log while the system is in use, and users can continue to access the database while the backup is running. The backup can be sent to disk or tape directly, or you can create a backup device, which is a prenamed location for the backup.

When the backup starts, it continues to run even though users are still editing the data. So if the users are changing data after the backup passes the area they are working on, what is in the backup when it completes? In other words, if the backup takes an hour, and 200 transactions change in that hour, what data do you have in the backup?

The answer is that the data is current as of the time the backup completes. While the backup runs, SQL Server 2005 does not empty the Transaction Log, regardless of the recovery model setting. It backs up the database and then applies the log entries to the backup at the end of the operation. That ensures that if the backup runs for hours, the backup is current at the end of that time.

Creating a Backup Strategy Using Backup Types

A backup is a copy of all the data in your database in a proprietary binary format and is not compressed. You can password or encrypt the backup (more on that later), but it is not protected by default. The backup creates one or more of these files on tape or on disk.

Your maintenance plan should include backing up all the data as often as possible. Your backup strategy should answer the question "How much data can we afford to lose?" This is called a risk interval. It is the amount of work you will lose between a catastrophic failure of the system and when it is available again for use.

Of course, the organization will tell you the answer is "none," and in that case you are looking for high-availability solutions. These involve a real-time duplication of the data, which I explain in Chapter 6. But even with a high-availability solution, you still need a backup strategy. Before you discuss that solution, find out what the risk interval is. That number will define the rest of your strategy for you.

There are three backup types: full, differential, and Transaction Log. Each contains different data, and using them correctly forms the core of your backup strategy. You can also back up just a part of the database if you are using filegroups.

Full

A full backup copies all the data in your database and log files. It is the most complete backup you can take, and works with all of the recovery models. If your database is in the simple recovery model, it has no other affect. If your database is in any other model, the backup copies all of the data and releases the committed entries in the log file to be erased.

You should take a full backup at least once a day if the size of your database permits. If your database is extremely large, you can take a full backup as time permits and log, differential, and filegroup backups after that. The risk interval in this situation is mitigated if you use filegroups.

Transaction Log

A Transaction Log or just log backup is a separate backup operation and creates its own backup files. This operation copies the data in the log file that has been committed to the database and then releases those entries to be deleted. To create a Transaction Log backup, the database must be in either the full recovery model or the bulk-logged recovery model.

Your strategy in these two recovery models is to take a full backup periodically, such as every day, and then take a log backup on a more frequent basis after that, perhaps hourly. It is this interval that answers the question "How much data can we stand to lose?" Let's look at an example using this strategy.

In a medium-sized firm running only one daily shift, it is decided that a two-hour loss of data is tolerable. The DBA sets a full backup to run nightly at midnight and a log backup every two hours starting at 0600 hours just before the first shift arrives, until midnight to cover any late work or traveling workers who connect to the system remotely.

Using this strategy, the DBA could restore the last full backup, bringing the system to last night's known good state. The DBA would then restore each Transaction Log backup, starting with the 0600 backup, repeating with each log backup file until the time of failure. The data in this case would never be more than two hours out. When the next full backup is taken, the previous ten Transaction Log backups can be erased, because the full backup includes all of the data they contain.

Differential

A differential backup gets all the data since the last full backup. The easiest way to think about a differential backup is that it is similar to a log backup, because it gets all the committed transactions in the log file, and is not useful without a full backup. The difference is that the differential backup is not smallit gets all of the transactions from the time of the last full backup, so each time you take one, you are getting a little more data. Let's look at our earlier example again, this time with the simple recovery model.

Your application once again writes data (transaction A) at 11 in the morning. The entry is written first to the Transaction Log and then on to the database. Using the simple recovery model, after it arrives safely in into the database, the entry is removed from the log. Transaction A is now only in one location: the database.

You take a full backup at noon. You now have transaction A in the database and transaction A on a backup device.

Now assume that between noon and 1 o'clock, two more transactions are written to the log and then the databasewe will call them B and C. You now have transactions A, B, and C safely within the database, and transaction A still on tape.

At 1 o'clock, you take a differential backup, capturing B and C to another file on the tape. More data is written (D and E) and at 2 o'clock you take another differential backup. Now you have A, B, C, D, and E in the database but only B, C, D, and E on the differential file on the tape. Notice how the file grows, whereas in the log file backup example, you only have the entries between the log backups. Because the log files only capture those changes, its files are much smaller. The differential backup files get larger the further you get away from the last full backup, regardless of how many differential backups you take.

To take a differential backup, you need to take a full backup first. To restore the database, you need only restore the full backup and then the last differential backup. That makes restoring even faster than with a log file operation, in which you need multiple backup files to restore.

Another advantage with differential backup is that you can use it with any recovery model. In effect, your get a kind of Transaction Log backup without using the full model.

Keep in mind that in your overall backup strategy, you might use all three methods. For instance, you could take a full backup at midnight, a differential at noon, and log backups every hour. That way you would only need to restore the full, the differential, and at most 12 hours of log backups, making recovery that much faster.

Backup Locations

You have two options for the location of your backup files: disk and tape. There are other drivers provided by vendors for different destinations, but these are the ones you get out of the box.

Disk

A common method of database backups is to back up the database or log to a file on disk and then use whatever enterprise backup solution you have to "sweep" the files to your normal backup location. If the database is small to medium sized, this is a great way to go, because you have the most immediate access to the last backup and can store the backups off-site.

It is important to note that SQL Server locks its physical files during use, so simply backing up the drives where SQL Server 2005 databases live will not back them up. Even if you use a "live backup" tool, SQL Server will not always allow you to recover your database properly. Some commercial packages do allow this operation.

You can back up to share locations, but I have found this method to be less than reliable. A better strategy is to back up to a drive that the SQL Server can see natively and then compress and copy the file across. You can also apply Pretty Good Privacy (PGP) encryption to the database at that time. You can encrypt the data within a database before a backup, a process described in more detail in Chapter 4, "Security."

Tape

On a single server, it is common to see a tape device connected directly to the SQL Server computer. SQL Server can see and use the MTF, or Microsoft Tape Format, so you can use these devices with other products that follow this standard.

In fact, you can point the backup to multiple tapes at the same time (called striping), making backup operation faster. This feature also allows you to back up a database that is larger than a single tape. When you back up a single database to multiple tapes simultaneously, the restore operation needs all the tapes available to restore it. You are not making multiple copies of the backup; you are just using multiple tapes for the same backup.

You can back up to multiple disk locations simultaneously, too, but the advantages you see in a tape environment do not really carry through to files on disk.

The advantage of using tapes is that the data can be stored separately from the server and put into a safe place in case of a disaster. In some cases, financial, medical, or legal requirements dictate that you have a tape backup stored in an off-site location.

If you would like to make multiple backups at the same time (as opposed to using multiple tapes for the same backup), you can use the MIRROR TO command. I show you that in the examples that follow.

Backup Device

A backup device is not really different from a disk or tape location. It is simply a name that points to one or more of these backup locations.

The reason you might use a backup device rather than specifying the file or tape location is that you can make backup scripts that are transportable to multiple systems. For instance, if you name your backup device "Tape Device One," you can create a script that you can use on any system as long as it has a backup device with that name on it, regardless of where it really points.

To create a backup device, open the SQL Server Management Studio and drill down to the Server Objects item. Right-click that and select New Backup Device… from the menu that appears, as shown in Figure 3-3.

Figure 3-3.

[View full size image]

You can also create a backup device using T-SQL code:

/* Create BackupDevice [Bucks Backup]
*/
EXEC master.dbo.sp_addumpdevice @devtype = N'disk'
, @logicalname = N'Bucks Backup'
, @physicalname = N'D:\TEMP\SQL2K5Backup.bak'
GO

In this example, the name I chose for the backup device is Bucks Backup, and it points to a physical file location on my hard drive.

Alternate Methods

There are other methods to secure your data in addition to the ones I have explained here. Some cost money, and others are outside the normal strategies you will see at larger organizations, but all have their place.

Using a Third-Party Product

In larger organizations, it is common not to use the SQL Server 2005 backup methods at all. The IT department at a firm will select a comprehensive backup strategy for all of their storage, including SQL Server. Often, these proprietary systems are paired with a hardware solution such as a tape library or Write-Once Read Many (WORM) drive jukeboxes or even large storage area network (SAN) hard drives.

In this situation, the backup software will often contain "agents" that can read SQL Server 2005 databases. These agents have their own terminology and methodology and can reach into an operational SQL Server database to back up the data. Some of them are faster than the SQL Server native backups, and others provide features based on the hardware they use.

If you are in one of these shops, you need to familiarize yourself with the backup methodology these tools use. It is still a good idea to know how the SQL Server 2005 native processes work, however, because many of these products explain their use in comparison the native methods.

Detaching a Database

A database is simply a collection of files that is controlled by SQL Server. The files have a logical name in the master database on the server, which in turn points to physical files.

SQL Server 2005 can "release" these files by closing the transactions and connections to them and removing the entry from the master database tables. This is called "detaching" a database. The files are then just like any other on the hard drive, and can be copied to tape or another location. In the section that follows, I show you how SQL Server can "attach" these files again to the same or another server.

It is a good idea to close out all database activity before you detach a database by having all the users complete their work, setting the database to Single in the Options panel on the Properties menu on that database, and running all the maintenance discussed in this chapter. This makes the database smaller and safer to transport. Running a backup on the database also clears the log, and you will not need to care about the Log Database Files (LDFs) when you copy the database files. SQL Server creates the log file for you on the other system.

You can detach a database using SQL Server Management Studio graphical methods or by using a T-SQL command. To use the graphical methods, open SQL Server Management Studio, drill down to the database you want to detach, and right-click it. Select Properties from the menu that appears and note the filenames and locations. Close that panel and then right-click the database name again. This time select Tasks from the menu, and then Detach from there. You are able to select whether you want to close connections and a couple of other options. Click OK and the files are ready for transport. You can compress them to make them smaller. Database files normally compress well.

Backup Security

You need to physically secure the backup files. You can also add a backup password, which I demonstrate in the examples that follow. If someone gets a copy of the unprotected database backup file, they can restore it to any other SQL Server 2005 system, even if he does not exist as a user in the database.


Restore

Restoring a database is a bit different from backing one up. For instance, restore operations are much slower than backup operations. Also, your users cannot access the database while you are restoring.

Users can access the database while you restore it if you use more than one filegroup in the database. After you restore the primary filegroup, users can access the database while you restore other filegroups.

After you have restored the database using the methods discussed here, there may be security implications. As I explain further in Chapter 4, security is stored in two placesin the master database and in the user database. Users are represented by unique numbers in both places and tied together to allow them to access a database schema. When you restore a database to a different server, these connections get severed, even if you have the same user name and other information on the receiving server. I show you how to fix that condition in Chapter 4.

You can restore a database that is not there yet. That is, you can create a database by restoring a backup. You can also restore a database over the top of an existing database, after you ensure there are no users accessing the current one.

That means that you can take a backup file from another server and restore it on your server. Although there are no technical limitations on this process other than having enough space, there are some security implications, which I explain in Chapter 4.

You can restore a backup from a previous version (7.0 and 2000), and the restoration process will upgrade the database for you. I explain this process more in Appendix A, "Upgrading to SQL Server 2005."

Restore Types

When you restore a database backup, you select where the backup file is located, how you want it restored, and a few options. I show an example of a few restoration types in a moment.

Complete Restore

The first type of restore is the complete database restoration, closing out the database restore operation and leaving the database ready for use.

Your backup plan should include the time it takes to recover the server. This includes the time it takes to replace the parts that failed, and also the time it takes to perform a restore operation. To do that, you need to perform a restore on the same machine or another just like it. That combined time is what it takes from the identification of the failure to resuming operations.

It is also not enough to try this just once. You need to have a recurring process to test your backup and restore procedures.

Differential Restore

As explained earlier, the differential backup file contains all the transactions that have occurred since the last full backup. To restore a differential backup, you first restore the full backup, this time setting the options to leave the database open for more restore operations. You then restore the latest differential backup file, this time closing the restore operations and allowing the users back in to the database.

Transaction Log Restore

To restore a database that has the recovery model set to full or bulk-logged, you first restore the latest full backup, and the latest differential backup if you have one. In both cases, you leave the database in recovery mode.

Starting with the earliest Transaction Log backup from the last full or differential restore, you restore the logs in earliest-to-latest order, leaving the database in recover mode until the last log restore. Restore the last Transaction Log and close out the restoration process, and you can then allow users back in to the database.

Point-in-Time and Other Special Restore Operations

If your database is set to the full or bulk-logged recovery model, you can perform a restore that stops at a point before the actual end of the backup set. For example, if the backup contains data containing the values 1 through 10, you can recover to the time that 6 was entered, leaving the database completely usable, as long as you have the data safely stored on a backup location.

If you are set to the bulk-logged recovery model and you have performed some of those operations I mentioned earlier that bypass the log, you cannot use a point-in-time recovery.

Point-in-Time

A point-in-time recovery allows you to set the entries to be restored by specifying a time, down to the second, of the last piece of data that you want restored.

Be advised that this is not always the easiest thing to do. In most OLTP applications, the data flows in an out of the system in ways that are not easily untangled. It is often best if the activity in the system is stopped before the backup operation. To find out whether this is true in your system, ask one of the developers this: "If I could restore the system to the state the data was in at exactly 7:38 a.m. today, would the application still work?"

To recover a database to a point in time, you use a special modifier to the RESTORE command. I demonstrate that feature in the examples that follow.

LSN

To address the concerns I mentioned previously, Microsoft has provided another method to restore the database to a certain point. In every transaction, there is both a time stamp and a log sequence number (LSN). As an added feature, you can stop the restore before or after a particular LSN.

The difficulty is that this number is not easily determined, so Microsoft says that this feature is primarily reserved for third-party vendors. To recover a database using an LSN, you use a special modifier to the RESTORE command. I demonstrate that feature in the examples that follow.

Mark

A more developer-friendly method for recovering to a point in time is using a recovery with a mark. Developers can name transactions, and a special entry into the msdb database will record a "mark" along with the LSN in the logmarkhistory table.

This is a different process than rolling back a transaction to a mark, which is covered in many T-SQL programming texts.

To recover a database using a mark, you use a special modifier to the RESTORE command. I demonstrate that feature in the examples that follow.

Alternate Methods

Just like the database backup process, you do not have to use these methods to restore your database. You can use a third-party product, you can attach a database to your server, or you can use a SQL Management Objects (SMO) script.

Using Another Product

Any third-party product that you might use in a larger shop will restore a database with its own methods, just as it backs them up. Once again, it is best to learn the processes SQL Server 2005 uses for restores, because much of the documentation for third-party products extends these concepts.

Attaching a Database

Attaching a database is the opposite effect of detaching it, although the process is a bit more complex. Once again, you can perform the process using SQL Server Management Studio or T-SQL commands.

To attach a database, copy and unzip the files you created and place them in the location where you want them to live permanently. Note the names and locations because you need this information to attach the database.

Right-click the Databases item in the Object Explorer pane and select Attach… from the menu that appears. Click the Add button in the panel that appears to add the database files form the location you noted earlier. Add all the Microsoft Database Files (MDFs) until you have them all. SQL Server will automatically create the Transaction Log files for you if you did not copy them. Click OK and the system will "adopt" the database by noting the logical and physical names and locations and mounting the database. It then presents the database to users, ready to use.

Attaching a database also has the advantage of upgrading the source database if it was transferred from version 7.0 or 2000.

Let's examine the various backup and restore methods using a few examples.

Backup Example, Simple Recovery Model

The least complicated backup method is when your database is set to the simple recovery model. In essence, you set the location for the backup, select full or differential as the type, and then run the command or click the OK button. As mentioned earlier, this can be done while users are still in the database. If there is activity during the backup, both the backup and the users will be slightly affected.

Backing Up the Database

You can back up the database to disk or tape, and if other devices are installed they may present different devices in the menus that you will see in the following screenshots. The examples I show are all directed to a hard drive file, although you can also use the backup device mentioned earlier.

Graphical

Open SQL Server Management Studio and locate the database name in the Object Browser. Right-click that object and select Tasks… from the menu that appears. From there, click the Back Up item. You are presented with a screen similar to Figure 3-4, showing the General tab.

Figure 3-4.

[View full size image]

This tab sets the type of backup you want to take, whether you are backing up the entire database or just files or filegroups and the name and description of the backup.

You can also expire the backup after a certain number of days or on a specified date. You would do this to prevent stale data from being used again. I have seen this used in medical and financial industries.

In the General tab, you also set the location for the backup files. You can add as many as you like of either tape or disk locations, but remember that all of these need to be available at restore time because the backup will be striped across all of the devices you specify here.

Moving to the next item on the left side of the panel, you will see the Options tab, shown in Figure 3-5.

Figure 3-5.

[View full size image]

The first section sets whether you will use the existing media set (a named group of files or tapes) or create a new one. In the case of a drive, you will most often use the existing media set.

The option that I have set here determines whether the backup gets written in addition to the backups that are already in the file (or tape) or the backup overwrites the data there. For a tape, you will often append the information, but in a disk file, it is usually better to select the overwrite option. The reason is that if you select the append option, the backups get a number within the device. That is no problem during the backup operation because this happens automatically; but when you restore the file (which I explain in a moment), you have to remember which number you want to restore. That is something you might think about with a tape, but for a file you tend to think of one file, one backup. What will happen when you forget or do not know that the file contains more than one backup is that you will restore an older (or other) backup, and you will be quite confused.

In this example, I have set the backup verification option because I want to be sure that the file is valid before I store it. If you are running low on time during the backup, you can turn this option off, but I do not recommend it. As a further check of safety, you can turn checksum verification on, but I have not found this to be necessary.

To run the backup with these options, click the OK button.

T-SQL Commands

To do the same backup that I have shown here graphically, you can run the following script:

/* This script backs up a database
 to disk, overwriting the backups
 that are already in the file if any.
*/
BACKUP DATABASE [DBAMDT]
TO DISK = N'd:\temp\DBAMDT.bak'
WITH DESCRIPTION = N'First Full Backup'
, INIT
, NAME = N'DBAMDT-Full Database Backup'
GO

In this first half of the script, the database name is encased with right and left brackets ([]) in case the name has a space in it. Using spaces in any database object name is a poor idea, and including brackets anyway is a good idea.

The line that begins TO DISK sets the drive and filename. It is prefaced with an N, which is another good habit to get into because it enables a proper Unicode rendering of the text that follows it, even if you are not using Unicode.

DBA 101: Unicode

Unicode is an idea similar to ASCIIit is just a way to encode the bits of data so that they can be rendered as text, but Unicode allows more bits to allow for more languages. Although your databases may not be set to Unicode now, if you get into the habit of using proper prefaces for Unicode objects and text by placing an N in front of them, you will serve yourself well.

The Options tab is represented as the WITH statements you see here. The INIT predicate sets the backups to be overwritten in the file, and the NAME predicate sets the name.

The next half of the script performs the verification:

/* This part verifies the database
 using the first set in the file
*/
RESTORE VERIFYONLY
FROM DISK = N'd:\temp\DBAMDT.bak'
WITH FILE = 1
GO

You are actually doing a restore here, but only a read of the backup file. The file is named, and then the backup number that you want to verify.

Mirroring the Backup

A mirror backup is a copy of the same backup at the same time to a different location. This provides a safety net or another copy that you can restore to another server for reporting purposes. There are no graphical methods to create a mirror backup, but you can use T-SQL commands to create one in code or in a script. Keep in mind that the backup is not complete until both files are written, so the speed of both locations should be the same. You can create a mirror backup to disk or tape, and you can mirror the backup to three additional devices, for a total of four backups of the same data.

T-SQL Commands

To create a mirror backup of the same database that we backed up a moment ago to two files in the same directory, you would use the following commands:

BACKUP DATABASE DBAMDT
TO DISK = 'd:\temp\DBAMDT1.bak'
MIRROR TO DISK = 'd:\temp\DBAMDT2.bak'
WITH
    FORMAT
GO

This time the WITH qualifier is FORMAT. This qualifier writes a new header over a media set, which is a grouping of backups. The tests on the machines I have available are very fastthere is almost no discernable speed difference on the smaller databases at all.

Adding a Backup Password

There are two ways to add a password to a backup. This is important to do, because as you will learn in Chapter 4, a backup contains all your data, and by default is not protected in any way. Anyone running SQL Server 2005 can restore your unprotected files. You can password a media set or a backup set.

T-SQL Commands

To back up the same database to a file location with a password, just add the PASSWORD qualifier to the WITH predicate:

BACKUP DATABASE DBAMDT
TO DISK = 'd:\temp\DBAMDT.bak'
WITH
    INIT
,   PASSWORD='NotAVeryHardPassword'
GO

When you restore this backup, you need to make sure you have that password handy. The backup file in this case is not encrypted, but it is protected from a simple restore operation.

Backup Example, Full or Bulk-Logged Recovery Model

There is no difference in backing up the database itself in the full or bulk-logged recovery model than the way I showed you in the simple model, but in the former two models you can back up the Transaction Log separately. You can do this using graphical means or T-SQL commands.

Backing Up the Log

Backing up the log uses the same processes as backing up the database. You can use the same graphical options and T-SQL commands, with the exception that you specify a log operation rather than the database for the backup process. Log backups should be taken frequently, so they are smaller and do not take a lot of time. Log backups are designed not to impact the users while they are running transactions against the database.

Graphical

The only difference in the graphical method for backing up the Transaction Log rather than the database is to select Transaction Log in the backup type on the first panel. Make sure you also change the location for the backup or you will overwrite your database backup with your Transaction Log.

On the Options panel, you have the same overwrite versus append options for the file or tape, but because you are backing up the log, you get two additional options.

The first option (which I have set here in Figure 3-6) removes the entries in the log. This process is called "truncating" the log. The second copies the log and sets the database to the "restoring" or recovery state. This second option is often used in log-shipping operations where you copy the log file to another server and restore it there.

Figure 3-6.

[View full size image]

T-SQL Commands

To create the same log backup as in the graphical method shown previously, you can use the following script:

BACKUP LOG [DBAMDT]
TO DISK = N'D:\Temp\DBAMDTTL-1.bak'
WITH
    INIT
, NAME = N'DBAMDT-Transaction Log Backup'
GO

To create the log backup but not truncate the log, the script changes to this:

BACKUP LOG [DBAMDT]
TO DISK = N'D:\Temp\DBAMDTTL-1.bak'
WITH
     NO_TRUNCATE
,    INIT
,    NAME = N'DBAMDT-Transaction Log Backup'
GO

In this example, it is the WITH NO_TRUNCATE option that leaves the entries in the log. There is a similar option that does the opposite: WITH TRUNCATE_ONLY. This option does not perform a backup at all; it just removes the entries from the log as if it had.

One reason to use a WITH TRUNCATE_ONLY qualifier is if the log is so large you have run out of room on the disk and you cannot operate the database properly. If this happens, run the log backup command with that statement and then immediately back up the entire database. At that point, you have broken the "log chain" and do not have an effective point in time recovery method until you take another set of log backups over time.

Making a "Copy" Backup

If you need to back up a database out of the normal cycle, you will break your normal backup strategy and potentially affect the ability to perform a point-in-time recovery. The reason is that you truncate the log when you perform a complete backup. You can, however, take a backup without harming the log chain with a script.

T-SQL Commands

The following script backs up the database we have been working with but does not disrupt the log chain:

BACKUP DATABASE DBAMDT
TO DISK = 'd:\temp\DBAMDTCopy.bak'
WITH
     INIT
,    COPY_ONLY
GO

You can also use this WITH COPY_ONLY qualifier in the log backup operation to copy the log but not affect the restore chain.

Restore Example, Simple Recovery Model

With the database and logs safely backed up, you can restore the database to the same or another system. Remember that users need to be out of the database for this operation and that if the database is restored to another system there may be security implications.

Graphical

To restore the database using graphical means, open the SQL Server Management Studio and right-click the word Databases in the Object Browser pane. The reason for not selecting the name of an existing database is that doing so places a connection to that databaseand now someone is in the database and you cannot restore it. After you right-click Databases, select Restore Database… from the menu that appears. You will see a panel similar to Figure 3-7.

Figure 3-7.

[View full size image]

You can pull down a list of database names or even type one in that does not exist yet. If you do that, make sure you pick the same name that the database had on the source system.

Next you select the restore location. The first option, called From database, lets you pick from the databases on the current system. It then reads from their backup histories, and places the recent backups in a list for you to choose from.

In this example, you will use the second method, and assume that the database does not currently exist on the system. That requires that you select the locations manually. Click the button next to the From Device line to bring up a panel similar to Figure 3-8.

Figure 3-8.

From here, you can add all of the backup files you created earlier. Remember that if you selected multiple files for the backup (not the copy operation), you need to have all of those available. After you have finished adding them, click the OK button.

In Figure 3-9, you can see that I have the list of backup sets within the files I selected. This allows me to pick the one I want to restore. Now let's move to the Options section by selecting it in the left part of the panel. There are quite a few options here to review, as you can see in Figure 3-10.

Figure 3-9.

[View full size image]

Figure 3-10.

[View full size image]

The first option overwrites a database if it already exists. The second option takes care of any replication settings, which I explain further in Chapter 6. The third option prompts you for each restore section, in case you are using multiple tapes in a single device. The fourth option leaves the database in a state such that no one else can access the database until you manually allow that user to.

The next box asks you where to restore the files, and it is important that you pay attention to this location. A database, as you recall, is made up of a logical name that SQL Server uses in the master database and a set of physical files that this logical name points to. The backup file contains a pointer to these original locations. You should leave the filename alone but ensure that the driver letters and the path match what you have on the system where you are restoring the database. Failure to do this will result in an error.

The final set of options is explained next to their radio buttons. The first option is used in a simple recovery model or when you are finished with the last Transaction Log restore. Because we are doing a simple restore, you can leave this first radio button selected.

T-SQL Commands

In this example script, I include the commands to relocate the files, even though I am just placing them right back where they were:

RESTORE DATABASE [DBAMDT]
FROM DISK = N'D:\Temp\DBAMDT.bak'
WITH FILE = 1
, MOVE N'DBAMDT'
    TO N'd:\Program Files\Microsoft SQL
Server\MSSQL.2\MSSQL\Data\DBAMDT.mdf',
 MOVE N'DBAMDT_log'
    TO N'd:\Program Files\Microsoft SQL
Server\MSSQL.2\MSSQL\Data\DBAMDT_log.LDF'
, REPLACE
GO

The commands that move the file are the WITH MOVE predicates for both the database and log files. You can remove those lines in a restore operation to the same locations. The WITH FILE = 1 sets the backup in the file in case you used the append option on the backup when you took it. You can leave that line as well if there is only one backup in the file or on the tape. I have also set the example script here to replace the database in case it already exists with the REPLACE option.

Restore Example, Full or Bulk-Logged Recovery Model

To restore a database in the full or bulk-logged recovery model, you first restore the database the same way as above, but in the Options panel, you select the second radio button to leave the database in a restoring mode. With that complete, the database will be restored but unusable because you have not "closed it out." Right after that, you restore the log files the same way from earliest to latest and close out the last log file restore with the first radio button.

Using the script method, you add the NORECOVERY option to the WITH statement, which changes the example restore script as follows:

RESTORE DATABASE [DBAMDT]
FROM DISK = N'D:\Temp\DBAMDT.bak'
WITH FILE = 1
, REPLACE
, NORECOVERY
GO

Here it is the NORECOVERY option that leaves the database in the restoring mode. Next you restore each Transaction Log, starting with the earliest, one at a time, just as in the graphical method. I show you a complete exercise in a moment.

Restoring the Database Using Transaction Log Backups

In the full or bulk-logged recovery models, you take log backups more frequently than you do each database backup. It is important to keep the backups straight, so I usually name the log backups in a sequential manner starting with 1. That makes restoring them fairly intuitive. This is important because many times when you are restoring a database, there has been some type of issue, and being clear and straightforward is a good thing in those circumstances.

Graphical

In this example, I have already restored the database and left it in the "recovery" mode, ready for Transaction Log restores, picking up where we left off in the database restore example earlier.

Move back to the Restoration Operation panel and select the backup files and sets just as before, as shown in Figure 3-11.

Figure 3-11.

[View full size image]

In this example, pick the Transaction Log backup file. You will do each log restore operation one at a time.

Next select the Options section and select any options that you want to change. In this example, you have just one log file to restore, so at this point select the first radio button and restore the log, as shown in Figure 3-12. That sets the state of the database ready for use, and closes out the restore operation.

Figure 3-12.

[View full size image]

T-SQL Commands

To restore the database log files using a script, restore the database backup first, using the WITH NORECOVERY qualifier as I explained earlier. When that completes, use the following example script to restore the DBAMDT database we have been working with:

RESTORE LOG [DBAMDT]
FROM DISK = N'D:\Temp\DBAMDTTL-1.bak'
WITH FILE = 1
GO

Point-in-Time and Other Special Restore Operations

The point-in-time recovery is much the same as a regular restore, but you stop the restore early and close out the restore process. Although the process is simple, as I mentioned earlier these restores are more difficult than just implementing the process because it is often impossible to tell what the state of the data is at any given point in time for an entire application. There are times, however, when this kind of restore is useful, such as during testing or controlled data changes.

Graphical

In the previous graphic, you may have noticed the To a point in time… button near the top of the Transaction Log backup. That is the only option you need to set to get the restore to the time you are looking for. Once again, you need to be in the full or bulk-logged recovery model for this to work.

T-SQL Commands

There is a modifier to the WITH predicate on the example restore script I showed earlier that will stop the backup at a certain point in time:

RESTORE LOG [DBAMDT]
FROM DISK = N'D:\Temp\DBAMDTTL-1.bak'
WITH FILE = 1
,   STOPAT = 'Nov 22, 2005 12:02 AM'
GO

The STOPAT modifier to the WITH predicate will set the time the restore stops.

I have one final note before I close out this section on backups and restore operations. Never trust your backups. Whatever methods you use to backup and restore your databases, test them regularly. Restore your backups on a test system, noting the duration each time.


Consistency Checks and Shrinking the Database

There are two other broad categories of maintenance that you need to keep up with for each database. The first deals with the physical state of the database; the other deals with the fragmentation of the database indexes.

In the first category, there are two primary tasks to perform: checking the consistency of the data files and shrinking the database. Checking the consistency of the database means that you will verify that the internal storage structure of the database is correct. To understand how to interpret the results of the maintenance and later the index commands, it is useful to know a little about the internal structure of the objects in SQL Server 2005.

The internal storage of objects within a SQL Server database starts with the page. A page is a unit of 8 kilobytes of storage on the physical disk. The first part of this 8K (96 bytes) is used to tell the database engine what kind of data is stored in the page (tables or indexes), how much room is left in the page, and also contains a unique number for the page location.

The rest of the page is filled with rows of data. As you enter more data or index entries, more rows get filled, until you are out of room in that page. A new page is created, with the same structure of header and data.

SQL Server 2005 calls eight contiguous pages of data or indexes an extent. The extent is the lowest unit that SQL Server allows you to work with, and it is how much the reporting we will see in a moment is done.

Sometimes you do not have enough data in a table to take up eight pages in a row, and because SQL Server 2005 manages at the extent level, it will group different tables on single pages into the same extent. This type of storage is called a "mixed extent."

As you create small tables that become large ones, or add and remove lots of data, the possibility exists that the structure of the pages can become damaged. This is what you are monitoring for in this phase of your maintenance.

Checking Consistency

To check the consistency of the pages and extents in your database, you can use the Database Consistency Check Commands (DBCC) statements. There are other tasks in addition to checking consistency that these commands can perform, and I explain those later.

DBCC commands can repair some damage as well as report it. Usually, however, this is not the best idea. The reason for this is that most DBCC commands verify structure, not data. Depending on the DBCC command, you might correct the structure but lose data. That puts the database in an inconsistent state, and you cannot go forward that way.

If you do find that there is structural damage, make sure you have a current backup handy. You can then repair the damage, and if there is data loss, you can always restore to a known good state.

DBCC CHECKDB

The most comprehensive test you can run on your database is the DBCC CHECKDB command. It actually encompasses some of the same checks that the other DBCC commands provide, but I recommend you work from a "top-down" approach so that you can quickly locate any errors in your database, drilling down to the individual objects that might have problems.

In its simplest form, the CHECKDB command checks the structure of the entire database, and presents a report with the results. The format is DBCC CHECKDB ('Database Name'), with other options that I explain in a moment. I will run the basic command on our test database and show you some of the abbreviated results here:

DBCC CHECKDB ('DBAMDT')

DBCC results for 'DBAMDT'.
Service Broker Msg 9675, State 1: Message Types analyzed:
14.
Service Broker Msg 9676, State 1: Service Contracts
analyzed: 6.
Service Broker Msg 9667, State 1: Services analyzed: 3.
Service Broker Msg 9668, State 1: Service Queues analyzed:
3.
Service Broker Msg 9669, State 1: Conversation Endpoints
analyzed: 0.
Service Broker Msg 9674, State 1: Conversation Groups
analyzed: 0.
Service Broker Msg 9670, State 1: Remote Service Bindings
analyzed: 0.
DBCC results for 'sys.sysrowsetcolumns'.
There are 548 rows in 5 pages for object
"sys.sysrowsetcolumns".
...
DBCC results for 'ServerInfo'.
There are 1 rows in 1 pages for object "ServerInfo".
CHECKDB found 0 allocation errors and 0 consistency errors
in database 'DBAMDT'.
DBCC execution completed. If DBCC printed error messages,
contact your system administrator.

I have removed some of the output where you see the three periods because the tables being checked are the system tables. There are quite a few of those.

You can see in the output that there are no problems in the structure of the tables, and you can even see how many rows each table has in each page.

Normally, this command is not a drag on the database's performance, but if there are a lot of large tables, you can check just the tables and not the columns by adding the WITH PHYSICAL_ONLY predicate to the end of the command. That produces a shorter output and takes a lot less time to run.

If you do discover errors, it is time to run a few of the other DBCC commands to locate where it is. When you find them, you can return to the CHECKDB command to try and fix the errors. Sometimes you can correct the errors without data loss.

You may decide that the data loss of restoring the database is a better choice than attempting to repair the database and then hoping things all work out. Even if you do that, you may want to move the damaged database to another location so that you can try and recover what you lost in the time between the last backup and the restore operation.

To correct the problems you find, you first want to try and fix the structure without harming the data. To begin, get everyone out of the database and set it to single-user mode. You can do that with the following commands:

ALTER DATABASE DBAMDT
SET SINGLE_USER
GO

This DBCC command runs the fastest and tries to fix minor errors:

DBCC CHECKDB ('DBAMDT', REPAIR_FAST)
GO

If the errors are corrected, you will get a message stating details. The REPAIR_FAST option does not fix indexes, however, because that takes a lot longer. To include the indexes, but still protect the data, you can use this command:

DBCC CHECKDB ('DBAMDT', REPAIR_REBUILD)
GO

If you find that neither of those commands work, you can try the most drastic method:

DBCC CHECKDB ('DBAMDT', REPAIR_ALLOW_DATA_LOSS)
GO

Remember, even if this works, you may have lost data in the process, so you cannot trust the database. This part of the DBCC command set will not be here foreverMicrosoft has already announced plans to remove it in future versions of SQL Server. The preferred method of dealing with a corrupt database is to restore it.

DBCC CHECKTABLE

The DBCC CHECKTABLE command checks the integrity of a particular table or index. If the CHECKDB command finds errors, you can run this command on a particular table to try and deal with it there. The format is DBCC CHECKTABLE (' Table or Index Name'). Here is a sample run in our example database:

DBCC CHECKTABLE ('ServerInfo')
GO

DBCC results for 'ServerInfo'.
There are 1 rows in 1 pages for object "ServerInfo".
DBCC execution completed. If DBCC printed error messages,
contact your system administrator.

You have the same WITH NOINDEX, WITH PHYSCIAL_ONLY, WITH REPAIR_FAST, WITH REPAIR_REBUILD, and WITH REPAIR_ALLOW_DATA_LOSS predicates to work with on the CHECKTABLE command.

You can also run this command on an index name rather than a table name. Unless the index is large, however, there are other ways to deal with them than using DBCC commands. I explain those in the next section.

DBCC CHECKALLOC

The DBCC CHECKALLOC command is a subset of the CHECKDB command and investigates the space allocation structures for an entire database. Here is an example:

DBCC CHECKALLOC('DBAMDT')
GO

Once again, you have the same WITH NOINDEX, WITH PHYSCIAL_ONLY, WITH REPAIR_FAST, WITH REPAIR_REBUILD, and WITH REPAIR_ALLOW_ DATA_LOSS predicates to work with on the CHECKALLOC command.

Other DBCC Commands

You can use other DBCC commands, most of them similar to the reporting commands we have just seen. Most of these commands do not have a repair component, however. DBCC CHECKFILEGROUP validates a filegroup, and DBCC CHECKCATALOG examines a catalog by database.

A couple of other commands begin to delve into the data side of the equation but still deal with structure. DBCC CHECKCONSTRAINTS ensures that all constraints are consistent within the table, and DBCC CHECKIDENT checks identity values, which are sequential numbers automatically inserted in a column. We used an identity data type in our database. The interesting thing about this DBCC command is that you can use it to change data.

Let's assume that you set a particular table to have an identity column, but you want to renumber all of the values it currently has. Perhaps someone hard-coded a number in the field and it served as the primary key for the table. You can run the following command to renumber the values:

DBCC CHECKIDENT('ServerInfo', RESEED, 1)
GO

The 1 at the end of the command is the value you start the numbering with. You should take extreme care with this command because you can break all manner of relationships in the database. For the most part, SQL Server will prevent you from doing that, but it can happen.

Shrinking the Database

I want to show you one more DBCC command. There are two methods of dealing with the size of a database. The first is to set a finite size for the database when you create it. You can do that with the CREATE DATABASE statement or by accessing the Properties panel of the database using SQL Server Management Studio as I showed you earlier. In the File area, you can click the button with the three periods on it to change the file properties, as shown in Figure 3-13.

Figure 3-13.

[View full size image]

You can do the same thing in code. The following script sets our database not to grow higher than 100 megabytes:

USE [master]
GO
ALTER DATABASE [DBAMDT]
MODIFY FILE ( NAME = N'DBAMDT', MAXSIZE = 102400KB )
GO

You can also set the database to grow automatically, as I did when I originally created the database, as you can see in the previous graphic.

As you use the database, it will grow as data is added. As data is erased, space is freed up but not claimed by SQL Server. You can have SQL Server deal with claiming the space (called shrinking the database) or you can do it as part of your maintenance plan. I usually advocate doing it as part of the maintenance plan, because having SQL Server taking care of it causes extra database activity without your control. You certainly do not want to cause an application delay due to maintenance.

To allow SQL Server to shrink the database all the time, stay in the Properties panel and open the Options section. Change the Auto Shrink option to True and you are all set.

To shrink the database on command, use the following DBCC command:

DBCC SHRINKDATABASE ('DBAMDT', 10)
GO

The 10 at the end of the command is optional. It sets the percentage of space you want left in the database when the command is through. You do this so that the database does not have to spend time asking for more space when it starts up again. In some commands, you will see the reverse of this number, called a fillfactor. That is how much room you want to use for the operation.

Do not expect this command to "defrag" a database. You will not reclaim all the empty space in the database, especially if it has a lot of indexes or uncommitted log entries, but this is a safe, fairly quick command to run during maintenance.

For the DBCC commands that report on data, your users can be in the database. It is best if the user activity is low to shrink the database. For the repair commands, they have to be out.


Index Fragmentation

Designing indexes is outside the purview of this book, but as the administrator you will be responsible for their physical components. To understand how to maintain indexes, you should know a little about how they work.

The concept behind using an index is not complicated; it is similar to the index in most any book. The index in a book is a listing of certain words or phrases with a location of the pages where you can find them. Without an index, you would need to read the entire book not only to locate the word, but to be sure to find everywhere that it appears. As you can imagine, the longer the book the longer this takes.

The key is to know what to place in the index. If you just repeated every word with its page number, the book would be twice as long, and it would take just a little less time to locate your word as it would just to read the book again. So normally the index just contains the words you are most likely to look up. Picking what to index in a book is the job of an editor, and picking the things to index in a database is the job of a database developer.

In SQL Server 2005, there are two kinds of indexes. The first is a clustered index, which changes the table itself to physically store the data in the order of the columns you choose. For instance, in our ServerInfo table of the DBAMDT database, you might pick the ServerName column to be the clustered index for the table. That would cause the table to be reordered with the value "AlabamaSvr1" coming before "WashingtonSvr1" on the disk. Because the table is physically arranged this way, the clustered index has two unique qualities: The table is the index, and you can only have one of them. For this kind of index, it is best to use sequential, seldom-changing values. The reason for that is that you do not want the disk to have to move everything down to insert a new row earlier in the sequence, because physical operations are expensive in computing terms.

The second kind of index is more like the book example and is called a nonclustered index. It is a separate structure and contains ranges of values and pointers to where those values are found.

The difference between the idea of an index in a book and the index in a database is that the information in the book is static. Because it never changes, the index is a write-once operation. In the database, however, data is always changing. For the most part, the pointer part of the index tracks the movement of the physical location pretty well; when new data is inserted, however, new rows need to be added to the index. This leads to a great deal of fragmentation as the indexes are written and changed. Each insert, update, and delete operation might affect an index entry. Reads, of course, do not.

Earlier I explained how data and indexes are stored in the pages and extents on the physical drive. In the case of data, the pages are not always rearranged or changed after they are created. In the case of an index, however, it would not be efficient to have its data stored in lots of locations around the disk; it would be best if they were all stored together. But because of the page structure, you are not always guaranteed where a particular row is stored. So to properly maintain the indexes, you need a way to determine this fragmentation and physically reorganize the indexes into a more contiguous data arrangement.

Because indexes are so vital to the performance of an application, they are one of the items that need the most maintenance. To maintain the indexes, you need to check the fragmentation and then physically reorder them. In the "Automation" section, I explain how to set this process to fire off automatically, but you should still understand how to do this process manually. The reason is that as the database gets larger, it will take longer to defrag and reorder the indexes, which can adversely affect performance. You may find yourself optimizing different indexes at different times. You might even determine that it would be best to completely drop the index and re-create it.

SQL Server 2005 uses another construct called statistics to help the indexes understand how effective they will be when called. The default settings on a new database automatically create these statistics and automatically keep them up to date. Although this takes some time, the tradeoff is usually worth it. If the database does not have these settings, you should include commands to create and update them in your maintenance plan. I cover these commands in the DBCC command explanation dealing with re-indexing.

To find out the schedule you need for defragmenting your indexes, check the fragmentation level and then test the index defragmentation for timing. From there, automate the testing.

Before we do can do all that, we need an index to work with. We actually already have an index on the ServerInfo table that we created with the first part of our DBAMDT table scripts. The commands for creating the primary key in that script created a clustered index, and because we set that to be an identity value we will always have a sequential number that is well suited to a clustered index. Here is a script to create another index on the same table from our project:

USE [DBAMDT]
GO
CREATE NONCLUSTERED INDEX IX_ServerInfo_ServerName
 ON dbo.ServerInfo(ServerName)
GO

Do not worry about the syntax, because we are not concerning ourselves with index design for now. This script adds a nonclustered, named index called IX_ServerInfo_ServerName consisting of the ServerName column. It also allows duplicate values to exist in the ServerName column, which we need because we might have servers from different locations with the same name. I have added this script to the end of the CREATE_DBAMDT_Tables.SQL file that is part of the original solution we began earlier in the book.

Now that we have an index to work with, let's take a look at the tools you have to find out information about the indexes and to maintain them.

Checking Fragmentation

You can check the fragmentation of the indexes in two main ways. The first uses a new concept called a dynamic management view, and the second uses more DBCC commands.

Dynamic Management Views

Dynamic management views are really just special functions that Microsoft has built in to SQL Server 2005 that act like a view. They contain much more information than other methods and show data as it occurs in the server environment. I show you several of these useful views throughout the book.

The first couple of views I explain really show how powerful they are, but they deal more with design than maintenance. The first is sys.dm_db_index_operational_stats, and it shows you the current I/O performance statistics for the indexes. If you are helping a developer see the real-time information on the indexes, this process is really worth the effort. To use it, you need at least two pieces of information: the database ID number and the table or index ID number. Because you probably do not keep those in your head, you can call another function to get them for you. Let's look at an example script that gets the information for our test table:

SELECT *
FROM sys.dm_db_index_operational_stats(DB_ID(N'DBAMDT')
, OBJECT_ID(N'DBAMDT.dbo.ServerInfo')
, NULL
, NULL)
GO

The first two parameters for this view are the database and object ID numbers, and as I mentioned I got those by asking another couple of functions, DB_ID() and OBJECT_ID(), and embedding them in the script.

The next parameter is the index ID. If you leave this at NULL as I have here, you will get all the indexes. The next parameter deals with the number for the index's partition, and once again you most probably want them all, so I have left the value at NULL.

The output this view creates is fairly complicated. Your developers will probably have you create it for them in a testing or production system to see how the indexes are affecting I/O. This function is less useful for administration as it is in design, but it does have fragmentation information that you can use.

The next view is more helpful for our purposes, and shows the index activity. It is called sys.dm_db_index_usage_stats and uses a similar format:

SELECT *
FROM sys.dm_db_index_usage_stats

Although this view also deals with performance, it also contains useful information for the DBA. It shows the index type, and also the percent and level of fragmentation, which you can read periodically to determine how often the index becomes fragmented. If you record that the index is fragmenting at the rate of 10 percent a day in the production environment, you might want to schedule an optimization every two or three days, depending on how long the maintenance runs.

Dynamic views, as well as some tables, return object numbers rather than object names. You can use the SELECT OBJECT_NAME(Object Number) command to find out the name.

The most useful dynamic view for us is sys.dm_db_index_physical_stats. This view shows you the size and fragmentation information for the data and indexes of the specified table. That is more in line with what we are after. The format of this view looks like this:

SELECT *
FROM sys.dm_db_index_physical_stats(DB_ID(N'DBAMDT')
, OBJECT_ID(N'DBAMDT.dbo.ServerInfo')
, NULL
, NULL
, 'LIMITED')
GO

You have seen the database and object ID parameters before, and the next two, index ID and partition number, are also familiar. The new parameter is at the end, and there are three values it can take that you should be aware of.

The first is LIMITED, the second is SAMPLED, and the third is DETAILED. The differences have to do with how "deep" a scan for the information is and how many locks they take on the database as they do it. As you might imagine, LIMITED is faster and less intrusive than DETAILED, but is not as accurate. For daily maintenance you can stick with LIMITED; on weekly maintenance, you might want to use SAMPLED; and when you are doing investigation on an issue, you would used DETAILED.

The output for this view is similar to what might see in the previous example, but is presented from the physical aspect, which is what we are after. In all of the views I have shown you here, you will get back a significant amount of data. Most of the columns have more to do with design than optimization, but the number you should pay most attention to is the fragmentation level. In all cases, you want to watch for one magic number: 30 percent. Before the fragmentation gets to that level, you will perform one kind of correction method, and when it exceeds it you will perform another. I show you which to use when in a moment.

Let's take a look at a sample output from the command shown previously:

database_id object_id index_id partition_number
index_type_desc      alloc_unit_type_desc      index_depth
index_level  avg_fragmentation_in_percent fragment_count
avg_fragment_size_in_pages page_count
avg_page_space_used_in_percent record_count
ghost_record_count version_ghost_record_count
min_record_size_in_bytes max_record_size_in_bytes
avg_record_size_in_bytes forwarded_record_count
----- ------ ------ -------- -------------------------------
------------------------------ ------ ------ --------------
---------- ------------- ---------- --------------- ---------
---------- ------------- ------------ ------------ ----------
------ ------------
7  2073058421 1 1 CLUSTERED INDEX    IN_ROW_DATA        1
0  0    1   1   1 NULL    NULL   NULL  NULL   NULL
NULL   NULL  NULL
7  2073058421 2 1 NONCLUSTERED INDEX     IN_ROW_DATA
1  0  0    1   1  1   NULL   NULL   NULL   NULL    NULL
NULL   NULL   NULL

(2 row(s) affected)

I have no fragmentation in the small index we created earlier. Remember, we have only one row in the indexed table so far, and we created the index after we created the table. Right now the index is one for one and not really useful.

DBA 101: Index Use

Although index use has more to do with design than with our purposes here, if a table has only a thousand rows or less, SQL Server decides that it is quicker to read the whole thing (called a table scan) than it is to find the data in the index (called a bookmark lookup) and then locate the pointer to the real data. In some of the tools discussed in Chapter 5, "Monitoring and Optimization," you might notice this scanning behavior, which is normally a bad thing. In the case of a small table, however, this is not bad at all. Understanding the data in your system is vital to designing good indexes.

By examining the output of these dynamic views, you can get a picture of how the indexes are being used and how fragmented they are. Scan at peak usage to get a true evaluation and then develop your index maintenance plan from there. You will also want to scan periodically as your data and systems change to verify your setup.

DBCC Commands

The method in SQL Server 2000 and below was to use DBCC commands to find information about your indexes and then defragment or rebuild them. Although you can still use these methods in SQL Server 2005, Microsoft recommends that you use the dynamic views that I just showed you. On test systems that I have run this command on, I have seen one to five minutes for a 10GB database, and up to almost four hours on an 800GB database with several thousand indexes.

One of the reasons that the dynamic views are better to use is that they calculate the information for you. In the DBCC commands, you have to evaluate the results yourself. Let's take a look at an example of using the DBCC SHOWCONTIG command on our sample table:

DBCC SHOWCONTIG ('ServerInfo')
GO

DBCC SHOWCONTIG scanning 'ServerInfo' table...
Table: 'ServerInfo' (2073058421); index ID: 1, database ID:
7
TABLE level scan performed.
- Pages Scanned................................: 1
- Extents Scanned..............................: 1
- Extent Switches..............................: 0
- Avg. Pages per Extent........................: 1.0
- Scan Density [Best Count:Actual Count].......: 100.00%
[1:1]
- Logical Scan Fragmentation ..................: 0.00%
- Extent Scan Fragmentation ...................: 0.00%
- Avg. Bytes Free per Page.....................: 7975.0
- Avg. Page Density (full).....................: 1.47%
DBCC execution completed. If DBCC printed error messages,
contact your system administrator.

You get a lot of information back, but you will need to do a little math to sort it all out. Let's take it line by line. First, we have the number of pages the table takes, which is one. The page lives on one extent, as you might imagine. In the case of a more fragmented setup, you would divide these two numbers to find out how many extents the pages take. This number is also found in the line marked Avg. Pages per Extent.

The extent switches show you how many times the system had to switch to a new extent to read all the data, normally an expensive proposition. The next line shows how physically contiguous the data isthe closer to 100 percent (as I have here) the more contiguous the data.

It is the next two lines that show the clustered index information. It is best if this number is closer to zero.

The final two lines show how much room is left in the table. Normally, you want to fill the tables as much as possible so that SQL Server does not have to move to another page (called a page split) to read the data. If that table will have a lot of inserts, however, you want to leave a little room for that. Once again, understanding the data and how it is used is vital to design.

As you can see, it is a bit more difficult to decide when to defrag using this command. Numbers have to be interpreted and must be taken in conjunction with each other to understand what to do. The dynamic management views do all the work for you and present the data in an easier format.

Correcting Fragmentation

After you have examined your indexes, you will want to correct them using one of two methods: a reorganize or a rebuild. You can use the ALTER INDEX command or the older DBCC commands.

ALTER INDEX Method

The ALTER INDEX command allows you to either reorganize the index or completely rebuild it. Reorganizing an index changes only the leaf level of the index tree, takes the least locks, and causes the least amount of disruption. You can perform this maintenance while users are online.

DBA 101: Index Trees

A "leaf" level is the lowest level of the index. Think about the roots of a tree and how they dig into the ground. At the top level (or root), SQL Server keeps the main page of the index. Next come lots of index entries in "branches" (called the intermediate level), and at the very bottom level are the "leaves" or leaf level.

Rebuilding an index takes far more locks and takes more time but completely rewrites the index.

After you do your index evaluation, you can decide whether to use a reorganize or a rebuild. If the fragmentation level is under 30 percent, use ALTER INDEX REORGANIZE; anything over that, use the ALTER INDEX REBUILD command.

The ALTER INDEX REORGANIZE command is simple to run:

ALTER INDEX IX_ServerInfo_ServerName
ON dbo.ServerInfo
REORGANIZE
GO

The first parameter specifies the index, and the ON parameter specifies the table. This command re-indexes at the leaf level and does not interfere with the users significantly, so it can be run while there are users in the system. It is also quick.

The ALTER INDEX REBUILD command can also take a simple format:

ALTER INDEX IX_ServerInfo_ServerName
ON dbo.ServerInfo
REBUILD
GO

This command takes longer to run and blocks users while it runs. For that reason, you want to make sure you have got low activity on the database before you run it.

For more control over the rebuild operation, you can specify several options in the WITH predicate. For instance, to rebuild all the indexes on the ServerInfo table, leave 20 percent of the space free, and not update the statistics, you use the following syntax:

ALTER INDEX ALL
ON dbo.ServerInfo
REBUILD WITH (FILLFACTOR = 80
, STATISTICS_NORECOMPUTE = ON);
GO

In this case, the word ALL replaces the table name, and the FILLFACTOR sets the index to leave 20 percent of the space free in each page, by filling it to 80 percent before it starts a new one. The STATISTICS _NORECOMPUTE = ON section sets the statistics not to rebuild along with this operation. There are more options for this command that I explain throughout the book, but these show the more common operations.

Your maintenance plan should consist of either the ALTER INDEX REBUILD statement for each table or a series of ALTER INDEX REORGANIZE and then ALTER INDEX REBUILD statements for individual tables and indexes, based on a schedule that you determine using the evaluations from the dynamic views I explained earlier. You normally find the latter method in larger databases.

In the "Automation" section that follows, I show you how to run scripts or use graphical tools to do the maintenance on a periodic basis.

DBCC Commands

Although Microsoft recommends the ALTER INDEX methods shown previously, you can still use DBCC commands to maintain your indexes. The two commands that you can use are DBCC INDEXDEFRAG and DBCC DBREINDEX. The names are fairly intuitive, but the first defragments the index and the second performs a complete rebuild of the indexes.

DBCC INDEXDEFRAG

The DBCC INDEXDEFRAG command defragments your indexes and has the least impact on the system for index maintenance. It is simple and quick to run and takes very few locks on the tables and indexes as it runs. You do, however, have to specify one index at a time for the command. That can be useful if you want to maintain the indexes on a separate schedule, but is more work if you want to do them all at once.

In SQL Server 2005, the ALTER INDEX command handles locks differently than the DBCC commands. Because of that, you have to change the index to allow the older locking scheme before you run the DBCC command:

ALTER INDEX IX_ServerInfo_ServerName
ON dbo.ServerInfo
SET (ALLOW_PAGE_LOCKS = ON)
GO

The parameters are the index name, followed by the table name (as you can see above). The ALLOW_PAGE_LOCKS setting allows page-level-locking behavior by the DBCC commands. Now you can run the defragment the index using the DBCC INDEXDEFRAG command. The parameters for this command are the database name, followed by the table name, and finally the index name:

DBCC INDEXDEFRAG ('DBAMDT'
, 'dbo.ServerInfo'
, 'IX_ServerInfo_ServerName')
GO

Statistics

Earlier in this section, I mentioned that SQL Server 2005 uses tracking numbers called statistics to determine a query plan and the efficacy of an index. The default options setting for a new database on SQL Server is to both create and update statistics for you. If that is not true on your database or you have just run the DBCC INDEXDEFRAG command, your statistics need to be rebuilt manually. You can update all the statistics on a table by using the UPDATE STATISTICS command:

UPDATE STATISTICS dbo.ServerInfo
GO

DBCC DBREINDEX

The DBCC DBREINDEX command is similar to dropping and re-creating an index. In fact, when you drop and re-create an index, you have a lot more flexibility because you can change the definition of the index at the same time. In either case, this command can take a long time to run on a large table or when it creates a large index. You are creating hundreds, thousands, or more rows of new data, which causes a significant impact to the I/O component of your server.

This command is also more intrusive because users must be out of the tables for it to run. The advantage is that it "deep cleans" the index and updates the statistics at the same time:

DBCC DBREINDEX ('dbo.ServerInfo'
, IX_ServerInfo_ServerName
, 80);
GO

The first parameter is the table name, and the second is the index. If you leave off the second parameter, you will rebuild all the indexes on a table. The final parameter is the fillfactor, which sets how full to create the page where the index lives. Remember that the higher this number, the less room you leave for new entries, but the faster the access is for that particular set of pages. Like most things, it is a tradeoff.

For the most part, the ALTER TABLE command is far more efficient and effective for maintaining your indexes. It is also simpler to work with. If that is not reason enough, Microsoft plans to remove these DBCC commands in future versions of SQL Server.

Auditing the Server

The next part of your maintenance is to audit your server. I want to stress the difference here between monitoring and auditing. When you monitor a system, you collect information regarding performance or security, with an eye toward solving a problem, enhancing performance, or predicting growth. Auditing a system ensures that you are following a certain set of practices or tracking changes. In essence, it is all about enforcement.

The logical question then is what you need to audit. The three primary areas you want to audit are changes, security, and errors. There are several tools you can use to audit a system, so I explain the methods that you can use to perform the audit to find the information for each of these areas.

Most of these tools have a graphical component, and many are available by just right-clicking an object such as a server or database and selecting Properties from the menu that appears. I do not cover that but instead focus on commands so that you can automate the process.

The last question is how to audit. We are creating the DBAMDT database to record information into a database as part of the book, so the logical place might seem to be in tables and views within that system. There are, however, reasons that you might want to create paper for this exercise. I normally try not to have paper for anything I do not have to, but legal requirements, distribution constraints, and so forth might make having a binder with the auditing information more useful than electronic methods. That is not to say you could not have an electronic system that creates the paper, just that the paper might still be necessary. The other consideration is that if the auditing system is stored on a server that just had a complete failure, you cannot get to the auditing information to rebuild it.

For this exercise, I assume a paper-based output and do not store the data in the DBAMDT database, but that is not to say that you could not do that on your own. The tables and views within the DBAMDT database are extensible for situations like this.

Auditing the Platform

I begin the auditing on the outermost part of the server, the platform. The platform information involves the hardware and software that makes up the server, the operating system, and SQL Server.

You do not have to actively audit these components. I normally only record this information on build and then after each major change. The important thing is to make sure you re-audit the system after the change and keep history on the items that do change. The history tracking and change management is another reason that having the information in a database as well as on paper is useful.

Hardware Documentation

Documenting the hardware is one of the most often overlooked parts of an audit, but it is very important to maintain. Having the information regarding memory and I/O as well as hardware types is invaluable to correlate with performance counters later on, so that you can quantify that adding that extra memory increased the speed of the application by a certain percentage. Knowing that you have increased storage space twice in the last year could give you great insights for budgeting.

Gathering hardware documentation is a manual task. Certain software utilities will get this information from you if you are on a particular set of hardware, but the operating system and SQL Server 2005 commands will not always provide you with perfect information for the hardware. The best way to get this information is by looking at the purchase orders or build sheets for the server components.

If you are tracking this by using reports created from an electronic system you have created, you will want to enter the data in a consistent format such as gigabytes, megahertz, and so on.

Operating System Documentation

The operating system is likely to change the most often, specifically with service packs, new services, hot fixes and so forth. Whereas newer versions of Windows have multiple tools to show information regarding the system, most all versions work with the Windows Microsoft System Diagnostics, or WINMSD. Here is a command that you can run inside CMD.EXE that create a text file of the system information for you:

WINMSD /F

The file this command creates will be located in the directory where you execute the command and will have the system's name as the title.

If you are entering this into an electronic system, you can either cut and paste or create a program to parse the file and store the parts within the database.

You should run this command whenever you change the operating system in any material way, such as when you apply a service pack or add new software.

Versions

The version of the operating system is stored in the file created by WINMSD or you can type VER at the command line to retrieve it. In SQL Server, you can get the information graphically within the Management Studio or you can type the following command in sqlcmd or any other query tool:

EXEC xp_msver
GO

This extended stored procedure returns a great deal of information about the version of SQL Server on your system as well as operating system information and even how many processors the system is reporting.

The version of SQL Server that this command produces is in a numeric form rather than in English (such as "SQL Server 2005 Service Pack 2"). The reason for this is that a numeric representation allows for a better granularity. Several Web sites will help you create a mapping for the numbers to the specific release and service pack, and as time goes on you will learn to recognize them.

Auditing Security

One of the most important tasks for an audit is to enforce security. I cover security in greater depth in the next chapter, so here I explain what you need to capture in more detail than how to capture it.

A complete security audit on a Windows system is beyond the scope of this book and includes such variables as borders, exposure points, surface area, and more. For the purposes of this type of audit, the more important items are the users and the items they have access to.

Users

You can use several external tools to enumerate the users on a Windows system, but you can use the NET USER command to send the local accounts to a file:

NET USER > USERS.TXT

This does not show what the users have access to on the server or the rights they have. For that, you can use the Windows Resource Kit tools, Windows Scripting Host languages, or you can manually enumerate the shares using NET SHARE to find the shares, NET SHARE sharename to find the permissions at the share level, and CACLS to find the permissions at the file level.

Why all this work for the operating system when you are interested in the databases? Because the file system is ultimately where the database files are stored, including backups. If you do not encrypt or otherwise protect the backups and you store them on disk, they are at risk. In addition, if users can access the raw SQL Server 20005 data files, they can potentially copy and mount them on another system.

For SQL Server, I explain in great detail how to enumerate the users and their permissions on a system in the next chapter, but you can run the following set of commands in a script for sqlcmd to locate the users in a particular server and database and what they are allowed to do from a broad perspective:

SELECT *
FROM sys.server_principals
GO
SELECT *
FROM sys.server_permissions
GO
USE databasename
GO
SELECT *
FROM sys.database_principals
GO
SELECT *
FROM sys.database_permissions
GO

Replace the word databasename with the name of the database you are interested in, copy from that line down, and paste a new section of it for each database, and then send this information to a file using the -o switch on the sqlcmd command, as I demonstrated in Chapter 2, "Architecture and Tools." That creates a great set of documentation for you automatically.

Securables

Both the Windows command-line and T-SQL commands in the previous examples will enumerate many of the items that are exposed by your server, which Microsoft calls securables. You may have additional items, however, that have not been shared out or granted permissions. Only enter as many of those as you intend to give permissions to later. There is no point in enumerating every single item in a system that has an exposure, especially for an audit.

DBA 101: Reviewing an Audit

Although collecting all kinds of information about your system is useful, it does not help if you never review it. Set aside some time each month to evaluate the information you gather and try and find methods and tools that automate as much of it as possible. The easier something is to do, the more likely it is that you will do it.

In the next chapter, I explain securables in greater detail. For an audit trail, what I have explained here usually suffices.

Log Files

I have focused a great deal on all the manual and disjointed ways that you have to gather and collect information regarding your server. Microsoft Windows and SQL Server 2005 both provide logging, and you can use built-in or Resource Kit tools to read them.

You do not need to store log files, unless you are required by legal, medical, or military reasons to do so. You should review them, however, and document any issues they expose.

Windows Event Logs

Microsoft includes several "event logs" in each operating system from Windows 2000 and up, also including Windows XP. You may have several types of these, but you will always have at least three: System, Security, and Application.

Things that affect the overall operating system will exist in the System and Security logs. SQL Server 2005 will include a subset of its logging in the Application Log. You can find the application to read the logs in different places on the Start menu depending on your operating system, but you can right click My Computer and select Manage and find the logs on all versions under System Tools, then Event Viewer. You can see that in Figure 3-14.

The main engine creates entries under the name MSSQLSERVER in the Source column.

Figure 3-14.

[View full size image]

If you are in charge of the system that runs SQL Server, make sure you right-click each event log category and change the properties so that the logs rotate on the schedule you want. The default settings allow the logs to fill (and then stop collecting) before the rotation period is complete.

You do not have to save the entries from these logs unless they pertain to a problem that you are having. You can use several Resource Kit tools to read through or search the logs from a command line.

Error Logs

SQL Server 2005 includes logs of its own that you can read within SQL Server in Management Studio or a text editor. These logs are created each time SQL Server is started and rotate through seven logs, numbered sequentially with 1 being the earliest and 6 being the latest before the current log. There are logs on both the engine and the SQL Server Agent. Each log set is by instance, so each instance has its own set.

To read them in SQL Server Management Studio, drill down to the Management item in the Object Browser and double-click the SQL Server Logs item. You will find all the informational messages as well as any errors in the various logs under that item.

The SQL Server Agent logs are located underneath the Error Logs object in the SQL Server Agent item on the Object Browser. They are numbered the same way as the engine logs.

Both sets of logs are stored on the file system for the default instance of the server in C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG. You can read them with any text editor.

The logs contain the most information on any item in SQL Server 2005. You will see some of the same entries in the Windows event logs, but the SQL Server logs have a superset of that information. Make it a point to review these logs and record any abnormalities you find in your audit.

Service Packs and Hot Fixes

Like all software, SQL Server 2005 ships with its fair share of errors in the code. Some of these are functional issues, others deal with security, and still others cause performance or stability problems. Microsoft releases new software to correct these issues in service packs. Hot fixes are smaller patches that affect the platform only for certain conditions.

When new service packs are released, you should read the notes that accompany them. Microsoft explains what the service pack fixes and, more important, what it affects. The reason this is important is that SQL Server 2005 is a development platform, which means that many of its features and services will be integrated into your developer's code. You cannot just apply a service pack without speaking with them first. This is especially true for hot fixes, which can have far greater consequences for installing or not installing them.

After you install a service pack, your system will report a new version as a number after the first decimal point in the version information I explained earlier. After you update, make sure you run a system audit report again.

Locating Service Packs and Hot Fixes

Microsoft delivers all of their service packs over the Web (www.microsoft.com/sql), and you might also get them on a CD if you are a member of Microsoft TechNet or the MSDN program. You can visit their site to find when the next service pack is due. You can also subscribe to several mailing lists that will alert you when a service pack is released.

You should install service packs not only on servers but on any system that has the SQL Server 2005 tools installed. Again, make sure you check with the user of that system so that you do not break anything that they have running.

Installing Service Packs and Hot Fixes

Before you install any modifications to your server, you need to have a complete backup. That means not only all databases (including system databases) but also the operating system, because the service pack might update files there as well. You only need to do this if you have a problemwhich you will not find out until you are done. In other words, you always want a full backup before you begin. I have only had one or two updates go badly in my entire career, but I was extremely pleased I followed this advice when it did.

Some service packs are specific to the edition, such as SQL Server 2005 Express; others cover all editions. Some cover just the database engine; others cover only Reporting Services or Analysis Services. When you visit Microsoft's site, you will select the type and edition you want to update as well as the language you installed.

The service pack downloads and installs with just a couple of clicks. Sometimes there are further information messages and selections when you start the install, so do not just click through the process without carefully reading each choice.

When your system is updated, test it to make sure that everything works as advertised.

Growth Analysis

Growth analysis is something that you need to do both before and after the system is installed. In the first chapter, I explained how to estimate a system's size, including the I/O, memory, CPU, and network subsystems.

In Chapter 5, I cover a comprehensive set of tools and processes you can use to watch your system. As part of that discussion, you will find many ways to get the current size and impacts of your database.

In this section, the emphasis is on watching the size of the database and using a few simple methods to estimate how the system will grow over time.

Estimating Database Sizes

There are a few methods of estimating the size of a database that you have not installed yet. If the system is already installed at another location, you can check to see the size of the database it is using. As long as the site approximates yours, you can use that as a starting point.

If there is no other installation of this type of system in existence elsewhere, you can still estimate the size from a similar type of system. Although the other system may store data a bit differently, as long as the major parts of the software work in a similar fashion and you are storing similar data types, you can use that as a starting point.

Finally, you can use an estimating method to create a model of the new database. To do that, determine the table layouts with all column types. For each column type, count the maximum bytes stored in each column. Add those together and then estimate the number of rows each table will have. Sum those table numbers and that gives you a rough count of how big the database will be.

Finding Current Database Size

If you are using Management Studio, you can get a graphical representation in the Summary panel of the database by using the Reports menu I explained earlier. If you want to use a command, you can find out the size of our sample database with the sp_heldb stored procedure:

EXEC sp_helpdb 'DBAMDT';
GO

You can locate the size for all the databases on the system with the same stored procedure by not specifying the database name:

EXEC sp_helpdb;
GO

Planning for Growth

To plan for growth, you need the current database size over time. That is part of both the monitoring and auditing processes.

If you know the database will grow at a rapid rate, run sp_helpdb frequently. When you know the growth pattern has settled into a predictable rate, run the command monthly. Record the values in a table or a spreadsheet and use either straight-line average to predict growth or use the regression-analysis features in a spreadsheet to plot a growth curve.

After you have the predictions, the key is to begin working with the developers early. You have two choices for adding space to the database once it is in place. The first is to back up the database and then restore the backup to the larger drive. You can use the second method if you have used filegroups to separate your database objects. In that case, you can move the tables with an ALTER T-SQL statement on the table to move it to a new location. I do not cover the syntax of that process here because it depends heavily on your database design, which this book does not cover.

The important thing in the auditing section of what you are doing is to make sure that you include the database sizes in your reports. The size today is not useful for predicting growth if you do not have a historical record of the size.


Automation

You will use many applications for automation. Scheduled tasks, imports for the business intelligence system, and most important, maintenance are all things that you can set up to use the automation features of SQL Server 2005.

The automation features in SQL Server 2005 use a service called SQL Server Agent and the msdb system database in conjunction. The service runs in the background all the time, and the msdb database stores the schedules and tasks that the SQL Agent performs.

Another component of automation is an operator. An operator is a set of contact information stored in the msdb database. Within each job, you can send information to an operator using this information.

The SQL Server Agent

The SQL Server Agent runs as a service on the server with the name SQLAgent for the default instance. Each instance has its own SQL Server Agent Service, using the naming convention of SQLAgent$InstanceName.

The service runs either under a user account or the Windows System account. When you installed SQL Server 2005, you selected this account, and you selected whether it will start automatically or manually. It is normally not a good idea to run SQL services as the Windows System account.

You want to make sure the service is running under a user account so that the jobs you run will have access to user-privileged activities such as connecting to a drive, sending mail, and so forth. I try to use an account that has the least privileges possible. I also use a different account for each instance.

The primary reason for user accounts and for keeping them distinct is for tracking. In the various logs that I explained earlier and in much of the monitoring that I show you later, there are references to the security principal associated with the activity. It is important that you can track the activity back to the affected account.

By default, the SQL Server Agent is not started automatically in Windows, although you may have changed that during the installation. You can use the SQL Server Configuration Manager tool I explained in Chapter 2.

All the service identities can be changed using SQL Server Configuration Manager, but the Reporting Services account is managed separately. When you alter the account for that service using the SQL Server Configuration Manager, you are redirected to the Reporting Services management tool, which I cover in Chapter 10, "Analysis Services."

SQL Server Agent uses the msdb database to store all of its information. There are also dynamic views that show some SQL Server Agent activity, which I cover in Chapter 5.

To control the SQL Serve Agent, open SQL Server Management Studio and locate the SQL Server Agent item in the Object Browser pane. Right-click that object and select Properties from the menu that appears, as shown in Figure 3-15.

Figure 3-15.

[View full size image]

This panel has several areas, beginning with the General settings. Here you can set the SQL Server database engine and the SQL Server Agent services to start automatically if they stop. Keep in mind that if either service terminates, there is a reason. It might be that someone accidentally stopped one of them; but if the services stopped due to an error, it is not always a good idea to just restart them. If something bad happened, SQL Server may have terminated itself to protect its databases. Having the service restart automatically might cause more damage than it saves you time. Set these according to your preference. In any case, the SQL Server Agent service must be running to automate anything.

The SQL Server Agent uses the msdb database. If you choose to auto-start the SQL Server Agent, you probably want to choose the database option, too, because both are needed. You do not have to restart the SQL Server Agent if you just want to select the database option, however.

You can also set the location for the error logs on this section as well include a net send address for notifications when the agent sends messages to the error log.

The Advanced settings involve forwarding errors to a central server. I show you how to set that up in the "Take Away" section.

Certain actions only take affect when the CPU is at an "idle" condition. Some servers always have a certain level of activity, so the CPU would never show an idle state. This section allows you to change the value and duration for that condition, which you can see in Figure 3-16.

The next section covers the Alert System.

Figure 3-16.

[View full size image]

There are two kinds of mail systems in SQL Server 2005. The first is SQL Mail, which uses a standard Mail API (MAPI) call down to the mail system installed on the server. This means that to use this type of mail, you must install a Microsoft Exchange client locally on the machine and configure a mail profile that the SQLAgent service account has access to. Sensing the limitations of this configuration, Microsoft created a complete mail system within SQL Server 2005, called Database Mail. Before you can use the Database Mail system, you need to move over to the Management item in the Object Browser, right-click the Database Mail item there, and select Configure Database Mail. A wizard will guide you from there.

Several items in this tab deal with operators, and I cover those in a moment.

In the Job System section, you can change the time a particular job will wait without response to determine a job has failed. You can also set a proxy account for the jobs to run in case the account in the job does not have proper permissions, and I explain that choice further in the chapter on security.

If you cannot connect to the local server using the server's name or IP address, you can set alias information in the Connection section.

In the History section, you can change the number of rows that SQL Agent uses in the msdb database history tables. You can change how many rows the entire table consumes with the Log Size setting, and you can also change the number of rows per job. I suggest you leave this at the default unless you know you are going to run into trouble with it, and then experiment with it until you get what you want. In this section, you can also automatically remove agent history by time, if you know the information will become "stale" after a certain period.

Jobs

Although jobs are just one component of automation, they serve as the heart of the process. Jobs are packages of steps that you can run by using graphical tools, on a schedule, using commands, and in response to alerts that I explained earlier.

Before I explain how to create a job, I explain job categories. Job categories are named groupings that you can place jobs into. To examine, edit, or create new job categories, open the SQL Server Management Studio and right-click the Jobs object. Select Manage Job Categories from the menu that appears, as shown in Figure 3-17.

Figure 3-17.

You will see a button on this panel for creating new categories. After you have created categories for your jobs, right-click the Jobs item in the Object Browser and select New Job… from the menu that appears. You are placed in the General section of the new job definition, shown in Figure 3-18.

Figure 3-18.

[View full size image]

In this section, you name and select a category for the job. You can also set a verbose description for the job, which is always a good idea. Jobs are easy to create and have a short name. Providing more detail here ensures that obsolete jobs are removed later and current jobs stay in place. You can also enable and disable jobs here.

Jobs are made up of smaller parts, called steps. In the Steps section, you can review, edit, and add new steps. Steps can be anything from T-SQL code to replication operations, operating system commands, and more. Once in the Steps area, click the New Step button to open another panel, similar to Figure 3-19.

Figure 3-19.

[View full size image]

In the General section of the Steps panel, you set the name of the step, which account it runs under, the database, and the type of step. In this example, I have selected a T-SQL command to show me the space in my databases. You can open a script file, paste the contents of a set of scripts in memory, and even parse the statements to make sure they are syntactically correct, all from this screen.

The Advanced section, shown in Figure 3-20, sets the success and failure conditions. Using this panel, you can create a simple program flow, such as "do the next step" if the step is successful, and "send an alert" if it fails. You can also set timeouts and even control the account to use for a particular step.

Figure 3-20.

[View full size image]

You can also record the activity of each step. I have set the output of the commands in this example to be sent to a text file, allowing me to automate the sizing check for the database that I spoke about in the last section of this chapter.

After you have added all the steps you want, click the OK button to return to the New Job panel. The next section, Schedules, allows you to set an amazing array of timed options for the job, including multiple options for recurrence.

The Alerts section sets up the alerts for this job. I explain those further in a moment. Alerts can place notifications from the job in the event logs.

The Notifications section sends the ending status of the job to an operator, which we will set up in a moment. You can send a notification when the job completes, when it is successful, or when it fails. Although the temptation is to send the notification only when the job fails, that is not always the best course of action. If the SQL Agent service fails, you will not get a failure notice, and you will think everything is okay when it is not. Although it is sometimes difficult to deal with, it is better if the job sends a notification when it runs, regardless of the outcome. That way, when you do not get any mail, you will know to check the server. Of course, if you have hundreds of jobs, this advice does not always hold.

You can send notifications to an e-mail, a pager, using a net send command, and by writing to the Windows Application event log.

The net send command only works if the recipient is online. Do not rely on this method alone to send notifications.

You can also make the job run once and then go away by combining a one-time schedule and the Automatically delete job selection you see in the Notifications section, choosing whether it deletes on completion, success, or failure.

Click the OK button to create the job with the steps you defined.

To manage the jobs, check the results they send when they run. You can also check them as they run by using the Job Activity Monitor located directly underneath the Jobs item in the Object Browser.

You can right-click any job and select View Job History from the menu that appears. The entries you see here are stored in the msdb database, and you made selections along the way that affect how many of these entries you will see.

Another management requirement is checking the efficacy of your jobs. Over time, you will develop lots of jobs, and sometimes a job becomes obsolete but is not removed. In Chapter 5, I explain how to check the jobs and their histories automatically.

With your new job created, you can use several methods to start it. The simplest is to right-click the job name and select Start Job from the menu that appears.

Your job will also run in response to an alert, which I show you in a moment. Your job will also run using any schedules that you created for the job.

You can control jobs using two stored procedures that Microsoft provides: sp_start_job and sp_stop_job. The simplest form is to call the stored procedure with the name of the job as a parameter, like this:

sp_start_job'JobName'

The sp_start_job stored procedure has more parameters that can set the error flag for the return code, the job name, the server name, and even the step to start with. Here is the full syntax:

sp_start_job
  'job_name'
  , error_flag
  , 'server_name'
  , 'step_name'

Alerts

I covered the definition of alerts in Chapter 2. You may recall that alerts have an error level, a severity, and text. Alerts are entered into the event logs, and SQL Server 2005 reads those logs and responds to them based on the settings you define in this area.

Alerts can do more than just fire on an event. You can also create alerts for a Performance Monitor condition, which I demonstrate in Chapter 5. If you are using Windows Management Interface (WMI) programming, you can also create alerts on WMI events.

The interface here is simple. You can create a new alert by right-clicking the Alerts item in the Object Browser and then selecting New Alert… from the menu that appears, shown in Figure 3-21.

Figure 3-21.

[View full size image]

Once inside, you create the conditions and responses for the alert. You can also set various notification options in this set of panels.

Operators

An operator is a collection of contact information stored in the msdb database. The contact information includes a name, a network account identification, an e-mail account, and pager information. You can also set a schedule for notifications so that you do not page an operator at two in the morning to tell him or her that a backup just finished.

Operators are simple to create. Right-click the Operators item in the Object Browser and select New Operator from the menu that appears, as you can see in Figure 3-22.

Figure 3-22.

[View full size image]

You can see in this example I have set myself up to be able to receive e-mail, network notifications, and pager alerts, although I have given myself a fairly easy schedule.

The Notifications section of this panel displays the alerts that the operator will receive. This is the corollary to the Operators section in the alerts and jobs we just examined. What you assign there shows up here.

At least one operator should be set to receive all the built-in system alerts, so after you create an operator you should return to the Alerts area and set one for those.

The History section of this panel shows the e-mails, pager, and network notifications this operator has received.

Creating a Maintenance Plan

After learning all the concepts and manual steps for performing maintenance, you may be surprised to learn there is a simple, flexible wizard that will create an entire plan and schedule it for you, complete with notifications.

It is common to see someone learn only about the Maintenance Plan Wizard and not probe any further into what it does. I think that understanding the individual steps in a maintenance plan will help you know when to use the wizard, and when to deviate from it. It is great to start with a wizard and then edit the plan it creates to suit your own needs.

To create a maintenance plan using the wizard, right-click the Maintenance Plan item in the Object Browser and select Maintenance Plan Wizard from the menu that appears. You will be led through several steps where you can set the actions, schedule, and order of the maintenance I have described in this chapter. When you complete the wizard, you can double-click the maintenance plan to edit any of the steps within it. I cover a complete exercise of this process in Chapter 8, "Integration Services."

You can also create a plan without a wizard. You can right-click the Maintenance Plan item again and select New Maintenance Plan… from the menu and start with a "blank slate." You are presented with a graphical tool, shown in Figure 3-23, where you can drag actions from a toolbox and create a comprehensive plan.

Figure 3-23.

[View full size image]

When you drag an item from the toolbox to the pane on the right, you can double-click it to fill out the properties for that item. In the case of the backup operation, you enter the type of backup, database name, and other information for the operation. When you are done placing items, you can drag the conditional arrows to them to create the maintenance flow.

Using these objects and conditional flow connectors is an efficient, clean way to perform all the maintenance I have explained in this chapter. You do not have to remember the commands to defragment a database because the toolbox has an item for that. It is, however, important to understand all the manual methods for your maintenance so that you can use the most effective methods when called for. In other words, the tools should assist you, not do the work for you.

Although it takes extra time, make sure you use the conditional logic on all steps. As mentioned earlier, you only need a recovery strategy and maintenance when things do not go well. Of course, when you need it, it is too late to create it.


Take Away

We have covered a lot of important ground in this chapter, because maintaining the system is one of your most important jobs. Because there are many things to maintain and audit, automation forms the key for being effective at that job. In this section, I give you an idea of "what you need to do when" as part of your plan.

In many installations, SQL Server 2005 may be used in smaller implementations. If you are in charge of an entire data infrastructure, however, you need to be able to check multiple servers in many locations. Microsoft has a mechanism to help you control maintenance and reporting across the enterprise. In the second part of this section, I show you how to tie several servers together for centralized control.

Maintenance Checklist

Certain tasks, such as backing up the database or performing a rebuild on an index, have schedules dictated by factors other than the calendar. You will develop a schedule for your backups and index maintenance, but it will not have the granularity of a single day or be something that everyone has in common. What most sites do have in common for daily and weekly schedules are checks and audits.

Your situation might also dictate a different schedule from what I propose here. The important thing is to use this checklist as a spring-board for your own checklist.

Daily

Run backups (full, differential, filegroup, and log).

Check services and instances to ensure they are running.

Check SQL Server 2005 Logs (engine and SQL Agent).

Check Windows event logs for SQL Server or SQL Server Agent

entries.

Verify backup and maintenance steps.

Weekly

Check Jobs history and any logging they create.

Audit for security changes.

Performance tuning.

Check Microsoft and other Web sites for updates and issues.

Quarterly

Audit for database growth (monthly if the growth is rapid).

Perform a test disaster recovery.

On Demand

Run backups (full, differential, filegroup, and log).

Perform index and table maintenance.

Add and edit users.

Restore databases.

Create databases.

Tune the system.

Apply service packs.

Enabling Multiple-System Automation

In many environments, you will be asked to manage several servers. Even if you are in a large environment where you do not manage all the servers, it helps to group the servers into regions or even by purpose.

You can set the SQL Server Agent to control and report the status of jobs on other servers. The server that will act as the main focal point is called the master server. The process involves creating an operator and starting the SQL Server Agent as the master server. You can then "enlist" other servers into the scheme so that they report up to the master server.

It is as easy as running another wizard. Open SQL Server Management Studio and right-click the SQL Server Agent in the Object Browser. Select Multi Server Administration and then Make this a Master… from the menu that appears.

You will be asked to set primary contact information for a new operator, called MSXOperator, on your server. You will then be guided through selecting the servers you want to report to this one. The wizard will check the version and connection information on the "target" server and then create the new operators on both systems. The systems will be enlisted and you can now report back to the master server.


Chapter 4. Security

Unlike the ones that preceded it, this chapter deals with only one subject. Security has many facets, from the exterior of the system (called a surface) to the rights and privileges the users have on the objects the databases contain. Even so, we only scratch the surface (no pun intended) on the tools and features you have at your disposal to secure your system.

Throughout the chapter, I refer to "least privilege," which is a concept common to secure systems and applications. It means to allow only who is necessary to do only what is necessary, and it is the basis for planning an effective security strategy. In the military, this is called a "need-to-know" basis. In fact, many military concepts carry through to platform and database security.

In the preceding chapter, I explained how to create a maintenance plan, and in this chapter, you will find many concepts to create a security plan. You do not have to create a plan to implement good security, but having security documentation is useful for two reasons: It provides an audit trail, and it forces you to think through the process. Although optional for some, in other cases you may actually be required by compliance policies to have this documentation.

The security plan includes the surface area that is exposed to security risks and what you are doing to secure those areas. I show you how to identify both of those areas.

Common problems in database security are authorization issues, lack of auditing and enforcement, and data exposure. In this chapter, I will help you deal with these issues, and in the "Take Away" section, I show you a few useful methods for creating an automated auditing system.

As you read through this chapter, I ask you to "do as I say, not as I do." I explain how to configure your system a certain way or create complex passwords and other security measures, and then I use simple passwords or less-secure settings in my examples. The reason is that I want to make sure the examples are clear, so I might use the word password as a password. That is, of course, a bad idea, but it is for clarity. Keep that in mind as you work through the examples in this chapter.

There are two sections in this chapter: securing the system, and securing the application's database. These are actually two separate discussions; but in your role as a DBA, you will be responsible for both. I do not cover things such as routers and network demilitarized zones (DMZs) because those areas are normally handled by other groups in IT. I focus instead on the items that you are more likely to be responsible for.

Security has its own jargon, just like all the other IT disciplines. So that you can take what you learn here and refer to other publications, you need to learn a little of that jargon. Anything in the database that can be secured independently such as a table or stored procedure is called a "securable." Anything that can perform actions such as a user account, role, or service is called a "principal." Although I use the more familiar names of user and table throughout the rest of the book, in this chapter I make an effort to use the security jargon so that you can get used to it.

I first deal briefly with the areas outside the system that have security implications and then focus more in depth on the areas that DBAs are commonly asked to manage. I then describe the security within an application. Even there, the discussion does not deal with just one set of concepts, because there are multiple ways to secure access to data.

In addition to the platform and data access, you will often need to secure the data itself. In the preceding chapter, I explained how to place a password on a backup operation, but in some situations that is not enough. SQL Server 2005 enables you to encrypt data within a database. I explain data encryption in the second part of this chapter.


Security and the Data Path

In Chapter 2, "Architecture and Tools," I explained the system architecture by showing the path a query takes from a data perspective. It makes sense to follow that same procedure here, because knowing where the packets go affects what you have to be concerned about from a security standpoint. This overview will help you navigate the rest of the chapter.

Let's start with the concept of endpoints. An endpoint is some combination of transport, such as TCP/IP or HTTP, and a payload, such as a T-SQL query and its results in a Tabular Data Stream (TDS) or a Simple Object Access Protocol (SOAP) call. In effect, it is the interaction between the user (principal) and the database services. The server listens using the Browser Service, which redirects the request to the proper port for a particular instance. You do not have to use this service if your application is using the default instance port (1433) or knows which port to connect to.

You can encrypt the transmissions across the channel and use a certificate between the two systems. A certificate is a piece of software that guarantees an identity. SQL Server 2005 can generate a self-signed certificate for the session, which protects against most "man-in-the-middle" attacks. You can also use a certificate from a certification authority, which provides better protection. In either case, to use the certificate by default, you need to set the Force Protocol Encryption switch on the server using the SQL Server Configuration Manager.

Once the connection is configured, the principal is authorized to an endpoint using either a Windows login or a login created and managed by SQL Server 2005, both of which grant entry into the server. After that, the principal is authenticated (using a user identity value) directly through their account or by virtue of a group (called a role) to perform actions in or on the database.

If you are not using Windows accounts but you are using Windows 2003 Server, SQL Server 2005 can enforce the password policies controlled by the operating system. Otherwise, you can set up a few simple policies within SQL Server 2005 such as password length, complexity, and lockout durations. You can also force the principal to change their SQL Server 2005 login password the first time he or she connects, and you can disable accounts without deleting them.

You can secure the following objects, which Microsoft calls scopes:

  • Server

  • Database

  • Schema

  • Other objects such as tables and views

Each of these scopes contains other objects, which we explore throughout the chapter. As the administrator, you can grant access to change a table (DDL-type statements) or to view, edit, add, or delete the values (DML-type statements) the table holds. Normally if a principal has the ability to use DDL statements, that principal also has the ability to use DML statements.

DBA 101: DDL and DML

In the Structured Query Language (SQL), there are two types of commands (called statements) that you can use. The first is called Data Definition Language (DDL). These commands include CREATE, DROP, ALTER, CONTROL, TAKE OWNERSHIP, and GRANT. They allow a principal to add, delete, change, or allow access to constructs such as tables and views inside the database, or even the database itself.

The second set of statements is called Data Manipulation Language (DML). These statements include SELECT, INSERT, UPDATE, and DELETE. Granting these statements to a principal allows the principal to add, change, and delete data in tables. Another set of DML statement are the EXECUTE and EXECUTE AS statements, which allow principals to run a stored procedure or function. To secure the metadata within the server, you can also grant or deny the text that shows the object's T-SQL creation using the VIEW DEFINITION statement.

In the sections that follow, I explain these statements and how you use them.

Security principals of the "user" type do not own objects within a database; instead, the schema object holds all database objects. Using a fully qualified name in the format servername.database.schema.object, you can grant or deny access on a single object to a principal with fine granularities. Using a schema allows you to bring users accounts in and out of the system without "orphaning" other objects.

Your developers have a couple of other measures to control security, which do not require you to set privileges and rights on securables. They can use tables within the application to implement their own security, or they can use an application role, which I describe shortly. If they use one of these other constructs, you need to be aware of what they are doing, but you will not always be in control of the application security. You will, however, still be in charge of overall data security, so it is best to understand the way the application implements security.

Let's explore the outermost security layer and some of the tools and techniques you can use to develop and implement your security plan. This process involves thinking about of every place that the data travels and identifying each intersection in that path as a security risk. Documenting the path and the data exposure points forms the core of your security plan. The rest of the plan details what you have done about those risks.


Platform Security

Most DBAs are not responsible for the network or operating system security. Even so, you should be aware of the risks these components have and where your boundary is. This is called a "border" in security parlance, and because SQL Server 2005 is so integrated into the Microsoft operating system, you need to be aware of crossover points.

The first part of good platform security is controlling physical access to your system. This involves what should be commonsense measures such as locating the server cabinet and network hardware in a separate rooms and controlling access to those items. It also includes securing the backup media, wherever it ends up. If you send your backup tapes offsite, check on their security policies, too. Securing the platform requires that you control who has access to the physical components it has.

Securing the Network

Although you may not have to configure routers and other network hardware, you will certainly live within the borders of that security, and sometimes even outside it. Make sure there is a security plan for those devices in place.

There are networking considerations within SQL Server 2005 that you do have control over, including the protocols and ports that SQL Server listens and talks on.

You can configure the ports that SQL Server uses. By default, you need to protect TCP port 1433 and UDP port 1434 for the default instance. Even better, you should change those ports, using the SQL Server Configuration Manager tool.

Just changing the port number for SQL Server does not make it totally secure. To have even higher security, you should control the port's access using your network hardware and software such as firewalls and routers.

To make the change, open the SQL Server Configuration Manager tool, expand the SQL Server 2005 Network Configuration item, and then click the Protocols for MSSQLServer object, as shown in Figure 4-1.

Figure 4-1.

[View full size image]

On the right pane of the tool, you will see the protocols listed. You can double-click these protocols to get their properties, which show different settings different based on the protocol. For TCP/IP, you can set the port number, timeouts, and even which IP addresses the server will listen on if you have multiple network cards on the server.

If you are using TCP/IP and want to use the Browser Service so that the port is dynamically set by SQL Server 2005 on startup, access the IP Addresses tab within the properties view and set the TCP Dynamic Ports to 0.

If the only application that uses SQL Server is located on the same system as the database, you can disable all protocols on the server with the exception of Shared Memory. It is not often that you will see this arrangement, and most often not a good idea to use SQL Server as a client to itself, but certain applications such as kiosk systems are often set up this way.

If at all possible, use the system only for SQL Server. Do not share files, printers, or other services from the system, to be truly safe.

You should also place your server behind a firewall and block the following NetBIOS- and SMB-related ports on the firewall:

  • UDP/137 NetBIOS name

  • UDP/138 NetBIOS datagram

  • TCP/139 NetBIOS session

  • TCP/139 - SMB

  • TCP/445 - SMB

Further along in the chapter, I explain how to encrypt the network traffic during SQL Server communications with the client. This provides a high degree of safety between the client application and the database.

You can also guarantee who the sender is by using a certificate. A certificate is an electronic "signature" that contains data specific to a user or organization. It identifies who the sender of the traffic is. You can have SQL Server 2005 create a certificate for you or you can use a certifying authority to provide the certificate.

You may want to consider another mechanism for highly secure network needs. Certain network cards (called cryptocards) can establish a software key with a partner card and encrypt the data transport with that key, in complete transparency to any application. The advantage of this setup is that you do not have to do anything further to secure the transmissions to and from the server, because the card will only talk to those that are secured in a similar fashion. The primary consideration in this scenario is that all cards, including any used in replication or clustering, need the same level of security. These cards are vendor specific and have their own setup requirements.

Both the software encryption and card security methods help protect against "man-in-the-middle" attacks. In this type of security breach, a system intercepts the packets from the source and replaces its own payloads after the packet headers. The server (or workstation) interprets those packets as coming from the original source and is compromised. Using encryption and certificates prevents this because the two systems have independent ways of verifying the source of the packet.

You should also consider using Active Directory if you are not already, with strong policies. It is not always the DBA's call for what the network uses, but using Active Directory services has benefits in addition to a higher level of security than NTLM authentication.

Securing the Hardware

Although you might not be responsible for overall network security, you may be responsible for the physical hardware where SQL Server is installed.

Unless you have in a high security level requirement, the primary hardware defense is the physical security I mentioned earlier. Although not strictly hardware, you should always update the firmware on your system because many of these updates affect both performance and security. Before you update firmware, take a full backup of the system and create an exit plan. After any firmware update, record the change in your security plan as an audit item.

If you are in a high-security environment, you may have further considerations, such as reinforced walls that block electronic signals all the way up to TEMPEST equipment that blocks almost all electromagnetic wavelengths. Other measures you can take are removing or disabling all bootable media devices, disabling USB ports, and setting strong passwords on the system's BIOS.

Securing the Operating System

For a production installation of SQL Server 2005, always use a secure operating system such as Windows 2000 Server or higher. As you will learn later in this chapter, there is a significant security value in using even more recent operating systems such as Windows 2003 Server or higher, because SQL Server 2005 hooks into the enhanced security subsystems those operating systems provide.

You also want to use the same operating system in the testing and staging environments that you use in production. These systems often house databases containing at least a subset of production data. This brings back the central point of security: data awareness. To properly secure your data, you have to consider all the locations the where the data is accessed.

Always keep the operating system patched to the latest levels and monitor the Microsoft support site for security hot fixes. Of course, make sure that you test the patches on your testing environment before you put the patch into production. This is especially important if you have separate teams managing the operating system from the database software. I have seen DBAs fight a problem for several days only to discover the operating system team applied a patch that affected the system.

File System

As explained in Chapter 2, your data lives on the file system of the server. SQL Server 2005 has the best performance and security when you use the NTFS file system, and NTFS is required for advanced features such as the Encrypted File System (EFS). EFS is a feature of the Windows operating system and is not under the control of SQL Server 2005.

If you are using EFS on the data partitions, be aware that the account you used to start the SQL Server 2005 database engine is the account that is used to encrypt the data in that partition. That means that if you detach the database and transfer it to another server as I described in the preceding chapter, you must decrypt the data files with the SQL Server database engine startup account before you transfer it; otherwise, it will not work.

If you have an extremely strong protection scheme but leave the data or backups on drives that are accessible by the wrong people, your other security measures will not help. The bad guys can simply pick up the data files, attach them to another database server, and they have your entire system.

Service Accounts

The important thing to consider in the Windows accounts involved with controlling SQL Server 2005 services is that they should have no more rights and privileges than they absolutely need. This concept is called least privilege. In most cases, a Domain or Enterprise Administrator account is not needed to run SQL Server 2005. The security normally required for the SQL Server 2005 accounts lies somewhere between an "enhanced" user that can run services and a local administrator account. For clustering, replication, and enterprise administration features, you need domain accounts with local administration privileges on the server. If you create a normal set of Windows users and then assign those to the service accounts during the installation or by using the SQL Server Configuration Manager tool, SQL Server 2005 gives them all the rights they need.

Earlier in the book, I mentioned that I use the same accounts to start the various services that SQL Server 2005 uses, such as the accounts used to control the database engine and the SQL Server Agent. I also mentioned that I use a different set of accounts for each instance of SQL Server 2005 that I have installed. There are good security reasons for this practice. Having separate accounts allows you to control what privileges each service has. As you will learn in this chapter, the rights you grant these service accounts have a great deal to do with how secure your server is.

Each service account needs to be documented in your security plan and should be set with strong passwords (of high complexity and more than 14 characters in length). Do not use the service accounts for anything else, and do not allow anyone to log in with the account on the network.

The service account that runs the SQL Server database can be used to create a certificate for encryption. Once encrypted, you must use this account to decrypt the data. For that reason, it is best not to change this account after it is in production.

You can change the accounts and startup types of the service accounts for SQL Server 2005 in many places, but the two you will probably use the most are the Services control MMC and the SQL Server Configuration Manager. I have left the Configuration Manager open from the previous example, and in Figure 4-2, I have clicked on the SQL Server 2005 Services item in the left pane.

Figure 4-2.

[View full size image]

If you have the tool open on your system, you will find many of the security settings that I explain in this chapter. Keep that tool open to refer to as we go along.

Securing SQL Server

I start with an overview of the security scheme for SQL Server 2005 with an analogy, and then I will spend the rest of the chapter explaining the details of each component described in the overview.

To help visualize the security scheme within SQL Server 2005, it is helpful to think of the server as a building, with rooms and objects within the rooms.

Endpoints, which are the first connections the server exposes, are similar to the doors of the building. There is usually more than one door to a building, and each can be secured separately. For instance, the public might be allowed into the building via one door but not another. The same holds true for SQL Server endpoints. Endpoints include the standard network connections that you do not control and HTTP endpoints that you can.

To enter any building, you need a key. Having this key gives you access to the building, but not to any particular room. This is similar to a SQL Server Login account. SQL Server logins allow you to connect to the server, but nothing within it. You can leverage the accounts within the Windows operating system or create all new accounts just for SQL Server 2005 to get this entrance key.

Once inside the building, people are given another key to access a particular room. The room is similar to a database, and the second key is similar to a database login. To access multiple rooms, a person needs a key for each one. To allow access to multiple databases, you need to create a database login for each one, too. So just like a building, you only need one SQL Server login, but for each database (like the rooms), you need a database login account as well. On my test system, I have a woodyb Server Login account, and a WoodyB account in many of the databases. These are tied to each other so that once I log in to the server, I do not have to provide further database authentication.

Finally, each room in a building contains objects, such as a chair, a computer, or a pencil. Just because a person has access to a room in the building does not automatically grant that person the right to use any of these objects. Even if they do have the right to use an object, the rights may be limited. In other words, someone might be allowed to sit in the chair, but that does not mean that person can take it.

You can compare the objects in the room to the tables, views, and other objects in a database. Database logins (principals) have to be granted access to the database objects individually, and even then you will need to specify what the principal is allowed to do with them.

Of course, someone might "own" an office in a building and therefore have the right to do anything he or she wants with the objects in that room, even granting others access to the things in the office. In the SQL Server analogy, you can allow SQL Sever 2005 principals to "own" a database, and then the principals can do anything they want with it. They can even grant access to the database to other SQL Server logins and allow them to access objects within their database.

Just like a building, the central concept for any secure computer system is that you should only expose what is absolutely necessary. This holds true for SQL Server, too. When you enable a connection method or feature that you are not using, it is easy to forget to monitor it for a security breach. For instance, in previous versions of SQL Server, an extended stored procedure called xp_cmdshell allowed principals to run operating system commands. Because this command runs under the security account that the Database Engine Service uses, some savvy principals ran a NET GROUP operating system command to add themselves to the Windows Administrators group.

SQL Server 2005 comes with features such as these turned off. You can see the current settings on your server by entering the command sp_configure in a query window.

If you have upgraded a previous version of SQL Server to 2005, the settings from the features and services I describe here are retained from the previous installation. Make sure that you go back and plug any holes that you find described in this section (after you finish the upgrade). I discuss the post-installation steps you should take in Appendix A, "Upgrading to SQL Server 2005."

You should run the sp_configure command periodically and record the output as part of your security audit to ensure the settings have not been compromised.

Functionality

The SQL Server 2005 Database Engine Service, Reporting Services, Integration Services, Analysis Services, and Notification Services all provide features that you should be familiar with from a security standpoint. I have already mentioned xp_cmdshell within the database engine, which is turned off by default.

Another feature to be aware of is the Common Language Runtime (CLR). This feature allows your developers to write managed code stored on the server and use them as database objects.

The developers write and compile a dynamic linked library (DLL) called an assembly in the language of their choice. If they have access to do so, they can enable the assembly directly from Visual Studio. If they are not using Visual Studio or it is not configured directly to your database, they can tell you where it is on the server, and you can include the assembly into the database for them.

This provides a great deal of power but opens several new vulnerabilities. Microsoft helps you mitigate those problems in three ways. The first is that CLR integration is disabled by default. Of course, if you need or want this functionality, you want to enable the feature. You can enable the CLR integration feature with the following query:

EXEC sp_configure 'clr enabled', 1
GO
RECONFIGURE
GO

You can also use a graphical tool that I show you in a moment. Either way, you need to have the ALTER SETTINGS permission, which is already enabled for the sysadmin and serveradmin server roles that I show you a little later. If you are logged on as a local administrator account on the server, you already have this permission.

After you have enabled the CLR, you can include an assembly into the database. The following code creates a new assembly in the database from a file called c:\temp\test.dll:

CREATE ASSEMBLY helloworld
FROM 'c:\temp\test.dll'
WITH PERMISSION_SET = SAFE

Your developers can then create stored procedures, principal-defined functions, principal-defined types, or triggers that reference this assembly. In the line beginning WITH PERMISSION_SET, you find the second method that you can use to secure the CLR.

You can set three CLR security levels. The first, as I have in this example, is SAFE. This level allows only access within SQL Server and does not allow access to the file system, the network, or the registry. This is the most secure security level. The next level is EXTERNAL_ACCESS, which allows access to the file system, the network, the registry, and system variables. This security level runs within the context of the Database Engine Service account but can also be set in code to impersonate another principal.

The least secure level is UNSAFE, which not only has all of the access from the other levels but can also call unmanaged code, which does not respect the memory space of SQL Server. This security level can even compromise the stability of SQL Server and should be used with extreme caution.

The third way that you can secure the CLR is up to the development staff. They can implement stringent standards to ensure that they write only those functions they need and that those functions do not do dangerous things. You and your developers should work together to ensure that nothing is enabled that is not needed. Because the configuration is basically invisible to the principals of the system, it is up to you to make sure that they have what they need, but no more.

Connections

Various applications make connections to SQL Server 2005. All use the same client and server libraries explained in Chapter 2.

Endpoints

Both the Database Engine and Analysis Services allow connections into SQL Server, as does Notification Services and Reporting Services. Recall from Chapter 2 that network transmissions flow from the client to the server and then back to the client. On the server side, the clients connect to various "endpoints," which allow access into SQL Server 2005. Each endpoint is associated with a protocol.

You can allow or deny the network protocol with the SQL Server Configuration Manager as described earlier, but even if the protocol is denied, SQL Server 2005 still creates an endpoint for it. When a connection attempts to use a denied protocol, the endpoint closes the connection.

Five endpoints are associated with each protocol type. The following chart describes the default endpoints and what each is used for.

Protocol | Endpoint Name | Use
DAC | Dedicated Admin Connection | This connection can be used even if the server is not responding to other requests, and is used for administrative tasks only.
Named Pipes | TSQL Named Pipes | Principal connections originated from Named Pipes (Windows networking) connections.
Shared Memory | TSQL LocalMachine | Principal connections originated on the server.
TCP/IP | TSQL Default TCP | Principal connections originated from the Transmission Control Protocol.
VIA | TSQL Default VIA | Principal connections originated from the Virtual Interface Adapter protocol.

You can create new endpoints with the CREATE ENDPOINT statement. You can change who has access to an endpoint with the ALTER ENDPOINT statement. By default, everyone has access to all endpoints with the exception of the Dedicated Admin Connection; only members of the sysadmin fixed server role have access to that.

Here is a script that will deny access to the WoodyB account to the default Named Pipes endpoint:

/* I have the brackets [] here because there are spaces in
the name of the endpoint.*/
USE master;
DENY CONNECT
ON ENDPOINT::[TSQL Named Pipes]
TO WoodyB;
GO

You can create a fine-grained security level to a database by only allowing certain accounts and groups connect through an endpoint. In the section where I describe SQL Serveer Logins, I show you a graphical method of managing account access to endpoints.

DMO and SMO

Two other connection methods used by client software are Distributed Management Objects (DMO) and the newer Server Management Objects (SMO). These application programming interfaces (APIs) still use endpoints but allow a different kind of access into SQL Server 2005.

DMO and SMO are primarily used for administrative tasks. In fact, the SQL Server Management Studio uses SMO. You can secure these APIs with the same endpoint, service, and protocol management as the other access methods I have described.

Once again, include any programs written using DMO or SMO in your security audit. In Chapter 5, "Monitoring and Optimization," I explain how you can determine which type of application is accessing your server.

Web and XML

Both the SQL Server 2005 Database Engine and Reporting Services allow access via HTTP. The SQL Server database engine also allows access to the server using SOAP calls.

To enable this type of access to your server, you need to create an endpoint. When you create the HTTP endpoint, you specify the type of requests the server will answer. These choices include SOAP, T-SQL, or Service Broker requests. Any application that supports XML calls over SOAP can send and receive XML data to SQL Server 2005. Unlike Web services from an HTML server, SQL Server 2005 does not allow anonymous connections, so you have less of a security concern than traditional Web servers.

There are quite a few parameters for creating an endpoint, which should be coordinated with your developers, such as specifying whether Web Service Description Language (WSDL) is respected by the server. I do not cover the syntax for creating this type of endpoint here because its configuration depends heavily on what the developers want to allow as a request and how they want to format the return values. We create a few in Chapter 7, "Notification Services and the Service Broker."

Your part will be working with developers to create the endpoints and then tracking those endpoints for your security audit.

For the audit, you can get a list of endpoints that your server is presenting with this query:

USE master
GO
/* Get the system endpoints */
SELECT *
FROM sys.endpoints;
GO
/* Get the http endpoints */
SELECT *
FROM sys.http_endpoints;
GO

Reporting Services works primarily through Web interfaces using HTTP and SOAP, so you need to secure those like any other Web-interfaced application. I cover the configuration of Reporting Services in greater detail in Chapter 10, "Analysis Services."

Database and SQL Mail

SQL Server can send maintenance and other notifications using various kinds of electronic mail accounts. You can also use mail to send queries to SQL Server 2005, and the server can send back the results as a reply.

There are two methods in SQL Server 2005 for using mail within the server. The first, called SQL Mail, is an older programming interface in SQL Server. It requires the server to have Microsoft Outlook installed, and for the account that starts the Database Engine Service to have a mail profile associated with it, because it uses the Microsoft Mail API (MAPI) for its connections. You are also limited with this method to 32-bit editions of SQL Server 2005, and if you want to send and receive attachments, you need to be a member of the sysadmins server role.

SQL Mail is not activated on your server by default. To use it, you must install an e-mail client first and then enable it using the Surface Area Configuration Manager tool, which I explain in a moment.

The principals must be in the same domain as the SQL Server 2005 database engine account to use SQL Mail so that they can authenticate properly. The SQL Mail interface will not allow the accounts to do more than they can from any other client tool. Make sure you are aware of the applications that are using this method and keep up with the service packs for Microsoft Outlook. Because applying a service pack has the possibility of breaking an application, you need to form a testing and exit strategy before you apply it.

With all these limitations and drawbacks, I mention SQL Mail only because you may have a legacy application that still uses this method of communication with SQL Server. You should migrate these applications as soon as possible to the more recent Database Mail functionality.

The Database Mail security implications involve controlling who has access to the mail profiles you create. To add an account to a profile, you use the sysmail_add_Principalprofile_sp stored procedure.

This script adds the WoodyB account to the primary profile I created in Chapter 3, "Maintenance and Automation," called MainProfile:

EXECUTE msdb.dbo.sysmail_add_Principalprofile_sp
 'WoodyB'
,'MainProfile'
GO

You can list the principals that have Database Mail privileges with the sysmail_help_profileaccount_sp stored procedure:

EXECUTE msdb.dbo.sysmail_help_profileaccount_sp
GO

Include this information in your security audit and, as always, do not grant access to mail to any account that does not need it. You should also be aware of programs that use this feature.

Remote Servers

Using the remote query and linked server features in SQL Server 2005, principals on other systems can reach into your server from theirs to transfer data back and forth using a single query. If the principals exist in both systems, you treat their accounts just as you do any local account. You also need to verify the security strategies used in their environment, because including these accounts widens your security boundary.

This is part of the reason for creating a documented security strategy and a security audit. You may have a need to access a remote server from your own, and the administrator there may want to verify your security. Having a plan enables you to provide a level of trust between the two environments.

If the principal does not exist in your system, the developers can set up the credentials in the query. This feature uses an OLE DB or ODBC call just as an ordinary application would. In this situation, you need to set up the same security as you would with any application.

There is an older mechanism for using another SQL Server system from a local server connection, called a remote server. For this to work, you access the properties of the server and then the Connections page. Ensure the Allow remote connections to this server box is on, which is the default. If you do not need this feature, uncheck this box.

Services

Most of the engines and features of SQL Server 2005 are controlled by services. There are services associated with the database engine, Reporting Services, Integration Services, Analysis Services, Notification Services, the SQL Server Agent, the Full Text Search Service, and the SQL Server Browser Service. Each of these is essentially a principal running software on your server.

You can control these services in various ways. The first way is to use the Windows Control Panel > Administrative Tools applet, where you can set the account and password the service will use, as well as the startup type (automatic, manual, or disabled) for the service. You can also use the SQL Server Configuration Manager to control these services. Your security strategy here is not to start any services that you do not need.

SQL Server Surface Area Configuration Manager

The surface area of SQL Server includes the things that expose themselves to the network and those services and functions that are called from the local system. You can control these surfaces using a tool called the SQL Server Surface Area Configuration (SAC) tool. This tool ships with all editions of SQL Server 2005 and changes the settings for a server. It works with multiple instances, clustering, and even virtual servers, and you can use it to copy a configuration from one server to another.

Locate the SAC tool on the Start menu under the Microsoft SQL Server 2005 menu in the Configuration Tools item, shown in Figure 4-3.

Figure 4-3.

[View full size image]

Once inside, you can work with services and connections or features. Selecting Surface Area Configuration for Services and Connections brings up the panel shown in Figure 4-4.

Figure 4-4.

[View full size image]

You can work with the system either by instance or by component across all instances. This allows you to set the remote connections for all instances at once or just on one instance.

Each feature you install on the server can be controlled here from a service or, in some cases, a remote connections standpoint. (Not all features have remote connections.)

Returning to the main panel of the SAC tool, select Surface Area Configuration for Features, as shown in Figure 4-5.

Figure 4-5.

[View full size image]

This panel shows two parts of SQL Server 2005 installed on this test system: the database engine and Analysis Services. Clicking each one of these objects allows you to select check boxes that enable or disable features associated with that function. You can also use the items you find here as entries for your security audit. If you have other features installed, such as Reporting Services, they show up here, too.

There is also a command-line version of this tool, called SAC.EXE. If you took the defaults during the installation of SQL Server 2005, it is located in the C:\Program Files\Microsoft SQL Server\90\Shared directory. There are several switches you can use to control what you would like to export or import. Use the command SAC /? at the operating system command prompt to see them all.

Let's look at a simple example using the SAC.EXE tool on the default instance that will transfer all of the settings from one server to another. After you have configured the SQL Server 2005 default instance the way you want, open a shell on the operating system of the first server and use the following command:

SAC out c:\temp\settings.xml

This creates an XML file in the c:\temp directory called settings.xml, which contains the settings you currently have on the first system. Copy that file to the c:\temp directory on the server where you want to deploy the settings and type the following command in an operating system shell on that system:

SAC in c:\temp\settings.xml

This assumes, of course, that you have a directory called c:\temp on both systems. You can also store this file as an attachment for your security audit.


User Security (Principals)

Entry to SQL Server 2005 and the database objects on it is controlled with user accounts, called security principles. You have two kinds of security accounts to work with: accounts that you create in Windows and those that you create using SQL Server 2005. I describe those further in a moment. Let's take a look at some of the broader concepts you should be aware of no matter which accounts you choose.

Password and Account Policies

Policies are the formal or informal conditions you place around passwords and accounts. Enforcing these policies helps you protect against a principal specifying a simple or easily broken password, or leaving a password unchanged for an extended period of times.

If you use Windows accounts for SQL Server security, you do not have to control these settings. The operating system or networking layer affects how strong the passwords are and how often they are changed. This relieves the burden of securing the accounts from SQL Server. Using this type of security, you are only responsible for securing the databases and their objects.

There are times, however, when you need to allow access to a database but the principal either is not in your domain or is using a different type of security altogether. For those, you can create SQL Server 2005 accounts. Although they do not offer the same level of protection as the operating system, you can still apply polices to them. I show you how in a moment.

The first policy that you need to consider for principals is password complexity. This means that the password the principal selects should contain upper- and lowercase alphabetic characters, and not a dictionary-definable word. Complexity also involves using nonalphabetic characters such as punctuation and numbers. Used in combination, this policy makes the password more difficult to derive.

One note of caution here: Substituting obvious punctuation for characters is still considered a dictionary-defined word. For example, password and pa33word are easily discovered with modern password-breaking tools. Spaces are an interesting case. Although they are allowed in a Windows password and make it easier to create complex passwords, they are not counted by the Windows system as a complex character. They also are not counted as an alphabetic character. There is no harm in creating a complex password with multiple spaces in it, but it is best not to end a password with a space because some applications may trim it.

No password should consist of repeating characters. More than three or four repetitions of a character allows a password-cracking algorithm to find a numeric pattern in the password.

The next line of defense is the length of the password. A short password, even if it is complex, is still easily derived. The longer you make the password, the harder it is for a system to guess. Without delving too far into the Windows security model, passwords greater than 15 characters are usually best. This is due to the fact that the older LAN Manager hashing algorithms breaks into two seven-character parts. Setting a password to a higher value than 14 characters causes a curious side effect that stores the hash incorrectly, making it almost impossible to track back to the login by the password-cracking algorithms.

You should only allow a password to remain in effect for a short period of time. Requiring the principals to change their passwords every few months gives less time for the password attacks, and if the password is ever compromised, only allows a discrete period of breach.

Enforcing a historical aspect to the password also increases security. This means not allowing a principal to rotate through a few passwords, developing another pattern so important to the password-breaking programs. Setting a length of time of around five passwords or more creates a fairly secure environment. Along with this requirement, the password complexity can often be set not to allow the principal to change just one or two letters of the password to cheat this system.

The final area for password and principals to enforce is the reset policy. If an attempt is made against an account multiple times, you can enforce how many times the password can be entered incorrectly before the account is locked. When the account is locked, you can control how the account is unlocked. You can require a manual account unlock or you can set a length of time the principal must wait before their account is automatically unlocked.

Both policies have advantages and disadvantages. If you require a manual unlock, you need to be responsive to the principals when they contact you; if you use the timed method, your principals will have to wait and the system will be less secure.

All of these policies fall within the concepts of using passwords for security, which itself is the subject of several books. There is quite a bit of math behind creating and deriving passwords, including the formulas given by the U.S. government publication called Federal Standard 1037C.

That reference uses this formula to calculate optimal password length: (log S )/(log N ), where S is the size of the password space and N is the number of characters available. The password space (S) formula is LR /P, where L is the maximum lifetime of a password and R is the number of guesses per unit of time. Most passwords are not this advanced; if you need a password formula, however, this is a good one to use.

The longer you allow an attacker to attempt the password, the more probable the attempts will eventually succeed. Keep these policies in mind as I explain the security on the server and the databases and its objects.

If you are using the Windows Server 2003 operating system or higher, you can use some of the same policies as the operating system, but in general the operating systems has a higher level of security options than SQL Server created accounts.

Whatever your account policies, add them to your security audit. It is useful to have this information available for new DBAs and others who create accounts. You can also present this information to security auditors.

User Accounts

As mentioned earlier, there are two methods to allow access to SQL Server 2005 and the databases. The first uses the Windows operating system, and the second uses SQL Server 2005 to create and manage the accounts. When you install an instance of SQL Server, you select the security access that you want to allow. You can select Windows accounts or you can allow both Windows and SQL Server 2005 generated accounts. If your principals are all on your NTLM or Active Directory domain or are trusted by them, you should use Windows accounts. It simplifies the security and places it in a stronger framework. If you have principals that will not be part of your NTLM or Active Directory domain, you will need to create accounts in SQL Server 2005.

Windows Logins

With your policies documented and your account strategy laid out, you can create the accounts using graphical methods or in code. I only cover creating the user in Windows briefly because a full discussion of Windows security is beyond the scope of this book and different versions of Windows have a different place to create the users. I do, however, want to cover the basics of creating a Windows user in case you are asked to manage the Windows server along with your database.

There are different tools to use depending on whether the server is part of an Active Directory domain or whether it uses local security. For Active Directory, you control the user accounts and policies from a domain controller. If the SQL Server 2005 system is not part of a domain, you control the Windows accounts from the SQL Server system.

In either case, you are leveraging the accounts from the Windows operating system, so you need that account set up first.

Creating Windows Accounts

Let's take a look at the places you create and manage user accounts in an Active Directory environment. Locate a domain controller, and then from the Start > Program Files menu, click the Administrative Tools item. From there, select Active Directory Users and Computers. Once inside, right-click the Users item and select New and then User from the menu shown in Figure 4-6.

Figure 4-6.

[View full size image]

In the panel shown in Figure 4-7, create the user by filling out all of the fields. Certain parts of them are filled out for you automatically.

Figure 4-7.

Click the Next button and then enter a password for the user. You can set a few minor policies here, but the policies I have been talking about are controlled by another tool that I show you in a moment. For now, enter the password and click Next to continue to the panel shown in Figure 4-8.

Figure 4-8.

When you move on from that panel, you will get a summary of the account information. Click Finish to complete the process.

You can place the user in any groups by right-clicking the account name and selecting Properties from the menu that appears. You can also double-click the group name and select multiple users that way.

You can use Windows or Active Directory groups in to SQL Server 2005 just as you would an individual user, but I rarely recommend it. Although it is easier to drop a user into a Windows group and manage privileges that way, you lose some tracking and granularity.

A different tool sets up more complex account and password policies. From the Administrative Tools item on the Start > Programs menu, access the Active Directory Domain Security Policy program. You'll get a screen similar to Figure 4-9.

Figure 4-9.

[View full size image]

Once inside, you can set up the advanced security settings I explained earlier.

If you are not using Active Directory, the SQL Server system will be part of a Windows NTLM domain (from the Windows NT days) or a stand-alone server.

In the case of a stand-alone server, right-click the My Computer object and then Manage from the menu that appears to set up your users. Once inside the tool, expand the Local Users and Groups item and then right-click the Users object. Select New User from the menu that appears and enter the information about the user. You can see that in Figure 4-10.

Figure 4-10.

When you have set up the user information, access the Local Security Policy item from the Administrative Tools area of the Start menu, as shown in Figure 4-11.

This tool provides similar functionality to the Active Directory Domain Security Policy tool but with fewer options.

Figure 4-11.

[View full size image]

Connecting Windows Accounts to SQL Server 2005 Logins

After you have the account created in Windows, you can use it within SQL Server 2005 by tying it to a SQL Server login account. In this section, I show you the entire process using the SQL Server Management Studio.

Using the Management Studio shown in Figure 4-12, expand the server's name in the Object Explorer panel, and then Security. Right-click the Logins object and select New Login from the menu that appears. You can see the results in Figure 4-13.

Figure 4-12.

[View full size image]

Figure 4-13.

[View full size image]

In the General section, you either type the account name using the NTLM format (DOMAIN\Username) or click the Search button to locate the account in NTLM domains or Active Directory domains. You can also select a Windows group for the user. I normally recommend against this practice for tracking; if your developers are using application-based security, however, this is an easy way to set up groups of users if they need them.

The second selection on this panel sets the default database for the principal. This is an important option to set because anything the principal accesses, creates, or deletes without reference to a database happens in the database set here, and the default database is master. Because you should not change anything in the master database, make sure you set a database for the principal on this screen.

You can also change the default language for the principal on this screen. Here it is set to (default), which means the language you installed the server under. If you choose a different language in this selection, make sure the database code page supports the characters in that language.

The Server Roles section, shown in Figure 4-14, sets the roles (groups) that this account will belong to. Server roles are called "fixed roles" because you cannot add or delete them. Each of these roles has specific rights and privileges, which I explain in a moment. A principal can belong to as many server roles as you want, but some of them overlap from a permissions standpoint.

Figure 4-14.

[View full size image]

In the User Mapping section, shown in Figure 4-15, you can set the databases the principal can access, as well as the default schema they will use in that database. You can also set the database roles the principal belongs to. Database roles are similar to server roles in that they are groups that have assigned permissions and rights, but you can create more database roles in addition to the "fixed" roles on this panel.

Figure 4-15.

[View full size image]

You do not have to assign the principal to a database in this panel. You can skip it entirely and then access the Security item on the Database object and assign the principals from there. Until you assign the principals a database, they will be able to log in to SQL Server but not access any database on the server.

If the SQL Server 2005 login is part of the administrative server roles, including sysadmin or serveradmin, they do not need to be granted access to a database or a database role. Those fixed server roles have implicit access to all the databases, so you do not need to add those accounts to any database.

In the Securables section, shown in Figure 4-16, you set the endpoints the principal has access to and what level that access has.

Figure 4-16.

[View full size image]

The Status section in Figure 4-17 allows you to grant or deny access to the database engine. Denying access trumps any other permission the login account has, even through a role membership.

Figure 4-17.

[View full size image]

You can also enable or disable the login. Disabling the account is useful if you want to temporarily suspend access to the system but not destroy the account. If a principal will be gone for an extended period of time, you can disable their login until the principal returns.

If the login is SQL Server based, and you are using Windows 2003, you can enforce account policies from the operating system onto the SQL Server 2005 login. If the principal violates that policy and locks his or her account, such as trying the incorrect password too many times, you can unlock the account on this panel.

You can also script the login using the CREATE LOGIN command in T-SQL. This example script creates the same login account as the previous screens:

USE [master]
GO
CREATE LOGIN [LOGCABINPRESS\Buck]
FROM WINDOWS
WITH DEFAULT_DATABASE=[DBAMDT]
GO
EXEC master..sp_addsrvRolemember @loginame =
N'LOGCABINPRESS\Buck'
, @Rolename = N'sysadmin'
GO
USE [DBAMDT]
GO
CREATE PRINCIPAL [LOGCABINPRESS\Buck]
FOR LOGIN [LOGCABINPRESS\Buck]
GO
USE [DBAMDT]
GO
EXEC sp_addRolemember N'db_owner', N'LOGCABINPRESS\Buck'
GO

You can also use the sp_addlogin stored procedure, but this method is included for backward compatibility only because it does not set the advanced options available in SQL Server 2005. Here is a sample script using sp_addlogin:

EXEC sp_addlogin '[LOGCABINPRESS\Buck]'
, 'password'
, 'DBAMDT'

This script only creates the principal and sets a password and a default database. The only other options on this stored procedure are the default language, an optional SID value, and encryption. You should use either the graphical methods or the newer CREATE LOGIN statement.

SQL Server 2005 Logins

If you selected the ability to create SQL Server logins during your installation, you are able to add accounts that do not exist in the Windows environment at all. These accounts are maintained completely within SQL Server 2005. Although using Windows-based accounts is a more secure practice, it is not always possible. In Web applications, you may have multiple database principals that you do not want to license on the Windows system that houses the application. If you allow SQL Server logins, you can create and manage those principals independent of the operating system, saving that operating system license cost.

I explain the process for creating a SQL Server 2005 login but do not repeat the entire graphical process here because there is only one difference between creating a Windows account login and a SQL Server controlled login. The primary difference is here, on the main screen, shown in Figure 4-18.

Figure 4-18.

[View full size image]

When you select the SQL Server authentication radio button, you will need to add a password and confirm it.

If you are using Windows 2003 or higher as the operating system, you can also have SQL Server 2005 enforce some of the Windows account policies. If you try to create a password for the principal on this screen and it does not follow the policies on the Windows 2003 system, you will get an error, as I did on this test system. You can see the error in Figure 4-19.

Figure 4-19.

[View full size image]

You can also set a certificate for this principal, which "signs" the data the principal sends. Instead of certificates, you can also set up an asymmetric or symmetric key to sign the data. These types of security methods encrypt data and provide some level of guarantee that the application can use to ensure that the sender of the data is really who they say they are.

Roles

Roles in SQL Server 2005 are similar to groups in an operating system. They serve as a collection object that you can grant rights and privileges and add logins to. Roles simplify security management because you can grant rights to the roles and as logins change the roles do not have to.

SQL Server 2005 has roles at both the server and database levels. That means there is one set of server roles, but each database has its own roles. You get several "fixed" or predefined roles as part of the installation, with rights and privileges already associated with them. You can create more database roles but not server roles, although in the section that follows on permissions I explain how you can grant logins and other roles many server-level permissions that fill any gaps you find in the server roles.

Let's examine the fixed server roles and what they can do.

Fixed Server Roles

For the most part, server roles allow a login to handle the management tasks associated with SQL Server 2005. You can add as many principals as you like to the server roles, but once again you should not add a principal to a role with higher privileges than he or she absolutely needs. If you need to have a principal out at a remote site with the ability to perform only a certain action, do not put that principal in a role that includes that right along with others he or she does not need. You can use a fine-grained approach by applying special permissions I explain later.

Most of the role names are self-describing, so I do not focus on the obvious here. Instead, I have developed a chart that will help you understand what rights the server role has and what kind of login is usually placed in it.

I start out with the server roles that have the most privileges. If a role already has a privilege, you do not need to include a login in another role that also has that privilege. For instance, if a login is placed in the sysadmin server role, that login does not need any other role.

Server Role Name | Rights Granted | Type of Principal or Operation
sysadmin | CONTROL SERVER | Full system administration
serveradmin | ALTER SETTINGS SHUTDOWN ALTER SERVER STATE ALTER ANY ENDPOINT ALTER RESOURCES VIEW SERVER STATE | Limited administration
setupadmin | ALTER ANY LINKED SERVER | For a Windows administrator, or a developer with enhanced rights
securityadmin | ALTER ANY LOGIN | Security administration, developer, or application administrator
processadmin | ALTER SERVER STATE |
 | ALTER ANY CONNECTION | Local DBA, developer, or network/security administrator
dbcreator | CREATE DATABASE | Developer or DBA
diskadmin | ALTER RESOURCES | Storage administrator or process
bulkadmin | ADMINISTER BULK OPERATIONS | Extract, transform, and load principal or process

To add a login to a fixed server role using SQL Server Management Studio, open the Security item in the Object Explorer. Expand Security and then Server Roles, and double-click the role you are interested in. Click the Add button to add logins to the role.

You can also add a login to a fixed server role in code by using the sp_addsrvRolemember stored procedure in the master database. The following example places the Windows WoodyB account into the disk-admin fixed server role:

EXEC master..sp_addsrvRolemember N'[LOGCABINPRESS\WoodyB]',
N'diskadmin'
GO

Fixed Database Roles

Fixed database roles work the same as the fixed server roles, with the exception that they manage not only a database but can control access to data as well.

In the "Server Role" section, I showed you which type of principal you might want in a role. In this chart, I focus instead on the main rights a database role grants a login account.

[Pages 211 - 212]

Fixed Database Role | Rights Granted | Permits
db_owner | CONTROL | Any operation in the database, including deleting it
db_securityadmin | ALTER ANY APPLICATION ROLE ALTER ANY ROLE CREATE SCHEMA VIEW DEFINITION | Security management
db_accessadmin | ALTER ANY USER CONNECT (With GRANT) CREATE SCHEMA | DB Login management
db_backupoperator | BACKUP DATABASE BACKUP LOG CHECKPOINT | Backup management
db_datareader | SELECT | Select data from any table
db_datawriter | DELETE INSERT UPDATE | Add or edit data in any table
db_ddladmin | ALTER ANY ASSEMBLY ALTER ANY CERTIFICATE ALTER ANY CONTRACT ALTER ANY EVENT NOTIFICATION ALTER ANY DATASPACE ALTER ANY FULLTEXT CATALOG ALTER ANY MESSAGE TYPE ALTER ANY REMOTE SERVICE BINDING ALTER ANY ROUTE ALTER ANY SCHEMA ALTER ANY SERVICE ALTER ANY SYMMETRIC KEY ALTER ANY TRIGGER ALTER ANY XML SCHEMA COLLECTION CHECKPOINT CREATE AGGREGATE CREATE ASSEMBLY CREATE CONTRACT CREATE DEFAULT CREATE FUNCTION CREATE MESSAGE TYPE CREATE PROCEDURE CREATE QUEUE CREATE REMOTE SERVICE BINDING CREATE ROUTE CREATE RULE CREATE SCHEMA CREATE SERVICE CREATE SYMMETRIC KEY CREATE SYNONYM CREATE TABLE CREATE TYPE CREATE VIEW CREATE XML SCHEMA COLLECTION REFERENCES | All DDL operations
db_denydatareader | Denied: SELECT | Prevents the principal from reading data in any table
db_denydatawriter | Denied: DELETE Denied: INSERT Denied: UPDATE | Prevents the principal from writing data in any table

To add a login to a fixed database role using SQL Server Management Studio, open the Database and then the Security items in the Object Explorer. Expand Roles and then Database Roles, and double-click the role you are interested in. Click the Add button to add logins to the role. Notice that you can also change the schema that the role owns in this panel, but it is best to leave these settings for the fixed database roles alone.

To add a principal in code, use the sp_addRolemember stored procedure. This example adds the WoodyB Login to the db_accessadmin fixed database role in the DBAMDT database:

USE [DBAMDT]
GO
EXEC sp_addRolemember N'db_accessadmin',
N'[LOGCABINPRESS\WoodyB]'
GO

Principal-Defined Database Roles

You can add your own database roles to any database. Using SQL Server Management Studio, open the Database and then the Security items in the Object Explorer. Expand Roles and then right-click Database Roles. Select New Database Role… from the menu that appears. The main panel is shown in Figure 4-20.

Figure 4-20.

[View full size image]

All you need to do to complete a database role is to name it and assign an owner to it.

You can also add logins to the role from this panel and assign securables and the permissions and rights this role will have on them. A more common practice is to approach the security for the role from the other direction by accessing the securable's Security section in its Properties panel. From there, you can assign which roles access the securable.

To create a database role in code, use the CREATE ROLE command. This example creates a new database role called Test in the DBAMDT database and sets the ownership to the db_accessadmin principal:

USE [DBAMDT]
GO
CREATE ROLE [Test] AUTHORIZATION [db_accessadmin]
GO

To assign a login to a database role in code, use the sp_addRolemember stored procedure. This example adds the WoodyB Login to the Test group in the DBAMDT database:

USE [DBAMDT]
GO
EXEC sp_addRolemember N'Test', N'[LOGCABINPRESS\WoodyB]'
GO

To grant the database role rights to an object, treat it just as any other principal. I explain object security further in the next section.

Application Roles

Application roles are used by developers to run applications with no principal context. In other words, an application role has no principals assigned to it.

You create the application roles the developers need, and assign them a password. You grant the application roles the permissions and rights that the developers need. The developers "call" the application role and provide the password. The developers then do some work and close the connection. The application role is then "dead" while it waits for another call. In this case, no principals are associated with the action, other than the account the developers used to connect to the database and call the application role.

Developers can use a few other parameters with the sp_setappRole stored procedure, such as encrypting the password and storing a local cookie on the client machine. You should encourage them to use the highest level of security possible when they use application roles.

To create an application role using SQL Server Management Studio, open the Database and then the Security items in the Object Explorer. Expand Roles and then right-click Application Roles. Select New Database Role from the menu that appears.

Just as in the principal database role, you name the role and set an owner for it, and you can also assign the rights to securables from this panel. The differences are that you need to set and confirm a password and you do not assign other principals to it.

To create an application role in code, use the CREATE APPLICATION ROLE command. Here is an example of a role called Test with a password of password, using the default access of the dbo schema:

CREATE APPLICATION ROLE Test
WITH PASSWORD = 'password'
, DEFAULT_SCHEMA = dbo;
GO

Here is an example of code being called with the example application role:

EXEC sp_setappRole 'Test', 'password';
SELECT *
FROM sometable
GO

For enhanced security, your developers should use the encrypt function on the password.


Object Security (Securables)

The overall security scheme in SQL Server 2005 is the intersection of principals and securables. Principals, as you have just seen, include principals and roles. Securables are the objects you can secure, such as databases and the objects they contain. The security for your data and objects involve giving the principals rights and permissions to the securables. The security for the system objects works in a hierarchy, with the server at the top level and the database objects at the bottom. Let's examine the permissions process and then work our way down the hierarchy to learn how to create a security matrix.

Permissions

All database object permissions intersect with three basic commands and two classifications of those commands. The commands are GRANT, DENY, and REVOKE, and the two classifications are Data Definition Language (DDL)-related statements and Data Manipulation Language (DML)-related statements.

The GRANT permission allows a principal to perform an action, the DENY permission prevents them from performing it, and REVOKE takes away a privilege without affecting the role or other specifically GRANTed permissions. The basic format for applying permissions using T-SQL is this:

  1. GRANT (or DENY or REVOKE) some permission

  2. ON some securable

  3. TO some principal

  4. WITH some option

  5. AS another principle

I flesh out this format as we move along, but you can use that little outline to create the most complicated security matrix.

There are also graphical methods for assigning permissions, which many people find a lot easier to use. Using the SQL Server Management Studio, right-click a securable such as a database or a table and select Properties from the menu that appears. Navigate to the Permissions section and add any principals you want from there. You can see that in Figure 4-21.

Figure 4-21.

[View full size image]

Graphical methods are useful but not practical for bulk situations. To grant or deny massive amounts of rights to multiple principals, you will normally create large scripts. The rest of this section shows you how to build your security statements from the basic format shown previously. This example sets the same permissions as the previous graphic:

use [DBAMDT]
GO
GRANT INSERT ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck] WITH GRANT OPTION
GO
GRANT REFERENCES ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck] WITH GRANT OPTION
GO
GRANT ALTER ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
GRANT CONTROL ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
GRANT DELETE ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
GRANT SELECT ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
GRANT UPDATE ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
GRANT VIEW DEFINITION ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO
DENY TAKE OWNERSHIP ON [dbo].[ServerInfo]
TO [LOGCABINPRESS\Buck]
GO

Although it may seem like a lengthy process, creating a script like this is far quicker than clicking through hundreds of tables, stored procedures, and views to set up security. Although the syntax for adding permissions to securables for principals looks simple, it is in the layering where it is easy to become confused.

To properly create your security layout, you need to understand the permissions hierarchy within SQL Server. I cover the full hierarchy in a moment, but for this part of the discussion, the important separation is the base unit of a table and the items that call or reference a table, such as a view or a stored procedure. If a principal is granted the ability to select from a view but not from the table it references, the principal will not see any data. If, however, a single principal creates all the objects in a database, granting a higher-level object permissions implies the permissions below it. In other words, if a developer creates a view and a table and then grants other principal rights to the view, the other principal can use the view even without explicit rights to the table. The developer can still deny or revoke the rights for the principal to the table if he wants.

To help keep this layering straight, I normally create a matrix to help sort it out. Another useful tool is using the Permissions panel I showed you earlier. There is a button there called Effective Permissions that will show you the "real" permissions the principal has. For instance, if you grant the principal the right to use the view but deny them the right to the table it references, you have removed their ability to see the data. Clicking this button helps you see what they really have.

Let's examine the permissions and see what each provides. Not all of these permissions apply to every object, so I show you a chart in a few moments that shows where each is used.

DDL-Related Permissions

Although these commands are not categorized strictly as DDL, they do affect the DDL operations a principal can perform. You can use this chart to begin the layering of the security for each object.

Right | Gives the Principal the Ability To
ALTER | Change the structure of the securable.
CONTROL | Have total control of the securable.
IMPERSONATE | Have one principal to use the rights of another.
REFERENCES | Access the securable "below" this one, such as a view or stored procedure to an underlying table.
TAKE OWNERSHIP | Take ownership of the securable.
VIEW DEFINITION | Examine the metadata about the securable.

DML-Related Permissions

These commands affect how the principals are allowed to access data, such as viewing, inserting, editing, and deleting data. This chart shows the data access part of your security layers.

Right | Gives the Principal the Ability To
CONNECT | Connect to the securable.
DELETE | Delete the securable.
EXECUTE | Run the securable.
INSERT | Insert data into the securable.
SELECT | Retrieve data from the securable.
UPDATE | Change the data in the securable.

The Security Hierarchy

Not all of the rights you have just seen apply to every securable. The securables on the system range from the server, which contains logins, endpoints, and databases, down through the database, which contains objects such as tables and stored procedures. Each of these is secured using the GRANT, DENY, or REVOKE commands, followed by the permission, and then the securable you want to protect.

Server

The server object is at the top of the hierarchy, because you can conceptually consider it as the engine for all of the objects it contains. Most of the rights and permissions at this level deal with maintenance and control of the system rather than the data it contains. Earlier I explained that you cannot add new fixed server roles to help with these functions, but the server object has several control permissions associated with it. Here is a list that separates the permissions for each of the objects the server contains. Some of these permissions, such as VIEW ANY DEFINITION, apply not only to one category (Server) but to others (Databases):

  • Server

        ALTER ANY EVENT NOTIFICATION
    VIEW ANY DEFINITION
    VIEW SERVER STATE
    ALTER ANY LINKED SERVER
    ALTER RESOURCES
    ALTER SERVER STATE
    ALTER SETTINGS
    ALTER TRACE
    AUTHENTICATE SERVER
    CREATE TRACE EVENT NOTIFICATION
    EXTERNAL ACCESS ASSEMBLY
    SHUTDOWN
    UNSAFE ASSEMBLY
    VIEW ANY DATABASE

  • Logins

        ALTER ANY CREDENTIAL
    ALTER ANY LOGIN

  • Endpoints

        ALTER ANY CONNECTION
    ALTER ANY ENDPOINT
    CONNECT SQL

  • Databases

        ADMINISTER BULK OPERATIONS
    ALTER ANY DATABASE
    CONTROL SERVER
    CREATE ANY DATABASE
    CREATE DDL EVENT NOTIFICATION
    CREATE ENDPOINT

The CONTROL SERVER is the highest permission you can grant, somewhat equivalent to adding a principal to the sysadmin fixed server role. You should grant only those rights a principal needs to do the jobdo not use the "shotgun" approach of granting more rights than are needed simply because it is easier.

Database

The database securable has permissions that are similar to the Server object. Most of them have to do with controlling the database. A few have to do with data access, but most of those are down at the object level, which I explain next.

The permissions on the database range from the specific, such as the ALTER permission, down to the general, such as ALTER ANY APPLICATION ROLE. This allows your security to be as granular as you need it to be.

Here is a list of the database securable permissions:

  • ALTER

  • ALTER ANY APPLICATION ROLE

  • ALTER ANY ASSEMBLY

  • ALTER ANY ASYMMETRIC KEY

  • ALTER ANY CERTIFICATE

  • ALTER ANY CONTRACT

  • ALTER ANY DATABASE EVENT NOTIFICATION

  • ALTER ANY DATASPACE

  • ALTER ANY FULLTEXT CATALOG

  • ALTER ANY MESSAGE TYPE

  • ALTER ANY REMOTE SERVICE BINDING

  • ALTER ANY ROLE

  • ALTER ANY ROUTE

  • ALTER ANY SCHEMA

  • ALTER ANY SERVICE

  • ALTER ANY SYMMETRIC KEY

  • AUTHENTICATE

  • BACKUP DATABASE

  • BACKUP LOG

  • CHECKPOINT

  • CONNECT

  • CONNECT REPLICATION

  • CREATE AGGREGATE

  • CREATE ASSEMBLY

  • CREATE CERTIFICATE

  • CREATE CONTRACT

  • CREATE DATABASE

  • CREATE DATABASE DDL EVENT NOTIFICATION

  • CREATE DEFAULT

  • CREATE FULLTEXT CATALOG

  • CREATE FUNCTION

  • CREATE MESSAGE TYPE

  • CREATE PROCEDURE

  • CREATE QUEUE

  • CREATE REMOTE SERVICE BINDING

  • CREATE ROLE

  • CREATE ROUTE

  • CREATE RULE

  • CREATE SCHEMA

  • CREATE SERVICE

  • CREATE SYMMETRIC KEY

  • CREATE SYNONYM

  • CREATE TABLE

  • CREATE TYPE

  • CREATE VIEW

  • CREATE XML SCHEMA COLLECTION

  • DELETE

  • EXECUTE

  • INSERT

  • REFERENCES

  • SELECT

  • SHOWPLAN

  • SUBSCRIBE QUERY NOTIFICATIONS

  • TAKE OWNERSHIP

  • UPDATE

  • VIEW DEFINITION

Database Objects

The permissions for a database securable are based on the securable itself. For instance, you can get data from a table by SELECTing it, but you EXECUTE a stored procedure.

You can use the following chart to help you build a security statement. Place the GRANT, DENY, or REVOKE verbs in front of the object, add the permissions possible next (if you are GRANTing the permission), then add the principal name, and finally add your choices from the permissions possible. Adding a WITH GRANT OPTION means that the principle can also grant the rights to others, and adding AS sets the role that the permission is derived from.

[Pages 224 - 226]

Object | Permissions Possible
Application role | ALTER VIEW DEFINITION
Assembly | ALTER EXECUTE REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Asymmetric key | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Certificate | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Contract | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Endpoint | ALTER EXECUTE CONNECT TAKE OWNERSHIP VIEW DEFINITION
Full-text catalog | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Login | IMPERSONATE ALTER VIEW DEFINITION
Message type | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Remote service binding | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITIONM
Role | ALTER TAKE OWNERSHIP VIEW DEFINITION
Route | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Service | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Symmetric key | ALTER REFERENCES TAKE OWNERSHIP VIEW DEFINITION
Schema | ALTER DELETE EXECUTE INSERT REFERENCES SELECT TAKE OWNERSHIP UPDATE VIEW DEFINITION
Table | ALTER CONTROL DELETE INSERT REFERENCES SELECT TAKE OWNERSHIP UPDATE VIEW DEFINITION
View | ALTER CONTROL DELETE INSERT REFERENCES SELECT TAKE OWNERSHIP UPDATE VIEW DEFINITION
Function | ALTER CONTROL EXECUTE REFERENCES SELECT TAKE OWNERSHIP VIEW DEFINITION
Procedure | ALTER CONTROL EXECUTE TAKE OWNERSHIP VIEW DEFINITION
Queue | ALTER CONTROL REFERENCES VIEW DEFINITION
Synonym | CONTROL DELETE EXECUTE INSERT SELECT TAKE OWNERSHIP UPDATE VIEW DEFINITION
Type | TAKE OWNERSHIP EXECUTE VIEW DEFINITION REFERENCES
XML schema collection | TAKE OWNERSHIP ALTER EXECUTE VIEW DEFINITION REFERENCES

To put everything together, I have found it is easiest to use a security matrix. Take a spreadsheet and list the principals down the first column. Across the rows, list the securables you have been given by the developers. Below the securables, list the permissions for that object. Here is an example.

Principal | Table: Client | Table: Event | Table: Method | Table: Class
Jane Doe | RU | CR | CR | UD
John Smith | CR | CR | UD | UD

In this example, the C stands for create (or insert), the R stands for read, the U stands for update, and the D stands for delete. This is often called a CRUD matrix, and the developers might have already created it for you.

With this matrix in hand, you can now put the commands, the objects, and the permissions together to create the security. Remember that GRANT allows, DENY prevents (and trumps all other permissions), and REVOKE removes the permission but does not deny it if the principal has the right in another way, such as through a role membership.

Because Jane Doe in the example CRUD matrix needs read and update on the table called Client (which uses a schema called Teachers), you would run this code:

USE [SchoolDB]
GO
GRANT SELECT, UPDATE
ON [Teachers].[Client]
TO [LOGCABINPRESS\DoeJ]
GO

With a little work, you can also have the spreadsheet create the commands for you, ready to cut and paste into your script.

You can also right-click a securable and grant rights as I mentioned earlier; if you have several to do, however, the scripts are faster. You can also save those scripts out to audit security later or to compare against your CRUD matrix.


Application-Based Security

Sometimes applications are coded such that the SQL Server 2005 security is less of an issue. This type of program relieves you of the responsibility of managing how the principals access data. This type of programming access is called application security.

Although the developers are largely responsible for setting up application security, it does not hurt to have a little background on what tools they use.

Application Security Methods

Developers use three basic tools to enforce application security. The first involves using one or more accounts to access the data on behalf of the principals. In this case, the accounts for the principals are often set up in standard tables in SQL Server, and the application checks these tables and accesses data based on the contents of the table. This is a secure way of using SQL Server because there are fewer accounts and privileges to manage. It does, however, place a higher burden on securing the application on the developer.

The second method is to use an application role. In this method, you create roles within the database and grant them rights just like any other account or role. The difference is that no database login accounts are associated with the role.

Finally, developers can start out with a lesser level of permissions and then "jump up" to a higher level of permissions in code. In effect, one account can act like another.

Whenever an application runs against the database, it has an authentication based on the rights and permissions granted to the SQL login account, the fixed server roles, the database login, and the database roles the account belongs to. The combination of all these rights and permissions form the execution context of the activity, all denoted by a number.

In addition to this execution context, the developers can switch a process to another account, on-the-fly. The two primary methods they can use are the EXECUTE AS and the SET PRINCIPAL features. In either case, the activity runs as if the account referenced by either of these commands called it.

You might think that this creates havoc in tracking the application chain. In fact, SQL Server 2005 records all logging and traces (more on that in the next chapter) with both accounts. The account that originally called the code is referenced, and the account that has the proper permissions granted to run the code is also referenced. SQL Server logging shows all the context switches used in the application.


Encryption, Certificates, and Key Management

Let's assume you create a paper request to ask a manager at your company for some information and put it in your company's outbox. The messenger takes the message and heads off to the other building to get the answer for you. Meanwhile, someone notices the messenger leaving and writes his own response. When the messenger returns, he removes the original response and place his own in the inbox. You read the message but have no way of verifying that the sender is really the head office. This is called a "man-in-the-middle" ploy, and electronic signals that travel across untrusted networks are vulnerable to this type of attack.

To protect your applications, you need to be able to scramble the data so that it cannot be read. But to protect yourself from a "man-in-the-middle" attack, you need to have some way of verifying who the original sender is. In the real-world example, you might do one of two things. You could get a third party who puts a special mark on the document to verify that it has not been tampered with. Another method might be to place a special code in the document, and when the head office reads the message, they use that code to scramble the letters in the document. When you receive the document, you use another code to unscramble the message.

The electronic equivalents of these tools are the symmetric and asymmetric key and the certificate. SQL Server 2005 has both of these methods, and you can create them for your developers to use.

Keys

An asymmetric key has two parts. The first part is called the public key, which is a password known to the receiver of the message. The second part of the key is the private key, which is a password known only to the sender. At the core, both of these are just really long strings of numbers. The data is run through one set of numbers to encrypt the data, and through the other to decrypt it. Because only one of the keys are exposed at any one time, it makes the security fairly high. To create an asymmetric key, think of a good password and a name for the key.

Next, select the type of algorithm you want to use when the key encrypts the data. You have three to choose from: RSA_512, RSA_1024, and RSA_2048. Each gains in strength but takes longer to encrypt and decrypt. Here is an example script for creating an asymmetric key called Asym_LogCabinPress in the database with the password of This1sToo Easy:

CREATE ASYMMETRIC KEY Asym_LogCabinPress
WITH ALGORITHM = RSA_512
ENCRYPTION BY PASSWORD = 'This1sTooEasy'
GO

The I in the Is part of this password is actually a number 1, because using all characters does not meet the complexity requirements of my test system. You can also create a key from a file, an executable, or even a CLR assembly that contains security code.

A symmetric key is the same on both sides. The data is encrypted and decrypted using the same key. With a symmetric key, you have even more choices for the encoding algorithm:

  • DES

  • TRIPLE_DES

  • RC2

  • RC4

  • DESX

  • AES_128

  • AES_192

  • AES_256

This example script creates a symmetric key call Sym_LogCabinPress with the same password I used earlier and the DES encryption algorithm:

CREATE SYMMETRIC KEY Sym_LogCabinPress
WITH ALGORITHM = DES
ENCRYPTION BY PASSWORD = 'This1sTooEasy'
GO

After you have created the keys, you can see them under the Security item in the database on the Object Explorer in SQL Server Management Studio.

Certificates

A certificate is a piece of software that is coded either to a third party (called a certifying authority) that verifies the identity of a principal. To use a certificate from a third party, follow the instructions they provide for your site.

You can also "self-certify," by having SQL Server generate its own certificate. To do that, you need to have a master key for the database. This is a password used to guarantee the certificate. Here is an example of the T-SQL syntax to create a master key:

CREATE MASTER KEY
ENCRYPTION BY PASSWORD = 'This1sTooEasy'
GO

You can now create a certificate. In this example, I am creating a certificate called Cert_LogCabinPress:

CREATE CERTIFICATE Cert_LogCabinPress
WITH SUBJECT = 'Self-Certification',
 EXPIRY_DATE = '01/01/2007 v
GO

I have also placed a date on the certificate that sets the expiration, called EXPIRY_DATE. Certificates can also be authorized for certain accounts. Here is another certificate that only the WoodyB principal can use:

CREATE CERTIFICATE Cert_ForBuckWoody
WITH SUBJECT = 'Buck Woody',
AUTHORIZATION WoodyB
GO

DBA 101: Passwords

Using the same passwords or even the same pattern of passwords for anything is a really bad practice in production. If someone were to guess one of the passwords, they would be able to get everything I have created here. For clarity, I am using the same password to focus on the concepts.

To put all this together into a single concept, let's look at a simple example a developer might use. The aim is to insert some data into the database such that it is unreadable to anyone. The process is as follows:

  1. The developer asks you to create keys and certificates.

  2. You provide the developer the names of those objects and the passwords on them.

  3. The developer writes code to encrypt the data using the objects.

  4. The developer writes code to decrypt the data using the objects.

I have already shown you the first step. Let's assume we have given the developer the passwords, and now the developer wants to encrypt some data in a field called FName in a table called Test using the certificate.

To use the certificate, the developers use the EncryptByCert function, which has two parameters: the certificate name and the string of characters to be encrypted. They first use the encryption function to set up the data and then insert it into the table. Here is the script:

USE DBAMDT
GO
INSERT INTO Test (FName)
VALUES (EncryptByCert ( Cert_ID('Cert_LogCabinPress'),
N'Buck Woody'))
GO

Now let's have a look at that data:

SELECT FName
FROM Test
GO
___________________________

amsdjdknosdj dkm asklf doihjf sakifnsaofhuhwe9uqhrp oisd
[f0as spij

That is not a misprint. The data really looks like that because it is encrypted. To see the data properly, the developer reverses the process using the DecryptByCert function:

SELECT
CONVERT(nvarchar(max)
, DecryptByCert(Cert_Id('Cert_LogCabinPress')
, FName))
FROM Test
GO

The developer has to use the CONVERT or CAST functions to bring the data from VARBINARY (which is how the data is encrypted) to the NVARCHAR characters that you can read. There are three parts in this script. The first is the DecryptByCert function, which decodes the data. The second is the column name that the developer wants (in this case, Fname). If you create a certificate using a password, you also need to include it as a third parameter.

Encrypting data by using the two keys we created earlier follows a similar process, with one exception. Before the developers can decrypt using the key, they need to open it. Here is the code that does that:

OPEN SYMMETRIC KEY Sym_LoginCabinPress
DECRYPTION BY PASSWORD N'This!sTooEasy'
GO

To use asymmetric encryption, the developer then uses the EncryptByAsymKey and DecryptByAsymKey functions, and for the symmetric keys they use the EncryptByKey and DecryptByKey functions. When they are done, they use the CLOSE SYMMETRIC KEY statement.

In the earlier examples, I created new keys and encrypted them with a password. You can also create keys and certificates and encrypt them with each other. You could create an asymmetric key and then encrypt your symmetric keys with it. Because there is a slight performance penalty for using the higher levels of encryption, this gives you a greater security function without the cost of asymmetric decryption.

Keep your keys safe and make sure you track the duration of your certificates. They should be put into place for no more than a year at a time to be truly secure.


Take Away

In this chapter, I have talked a great deal about creating a security plan and a security audit. You can combine these elements into a single document or once again store them in a database and report them from there. The security plan is a simple document that shows the protocols, services, and features (the surface) your system exposes to the world. The audit shows what you have done about those risks, and what their current state is.

In this section, I show you a few stored procedures that you can use to document your system, in addition to the ones I have explained throughout this chapter.

Security Auditing

I have not had you open your DBAMDT database in a chapter or so. The reason is that I do not want to give you lots of typing to do to create your system. Instead, I want to show you how to begin to create and fill out your own tables and create your own stored procedures to fill them. In each chapter, I show you a few more metadata mining procedures and views, and you can use the procedure I outline here to create your own monitoring system.

This chapter is tailor made for your security audit. As I mentioned at the beginning of the chapter, you really should keep a permanent record somewhere that you can deliver to others, but that does not preclude your storing the data electronically first. You can adapt the DBAMDT database to hold security data, too.

As I explained when we first created the monitoring database, you want to monitor some things less frequently than others. In fact, anything you can derive from a function, view, or system stored procedure that does not change does not have to be stored in the database. The reports that show the database can just run those queries to show the information.

But even if things change slowly, when they change you want a historical record of when it happened and what it changed to and from. The process to create this record is what we started at the beginning of the book, when we created the DBAMDT database. You will recall we created a table, a view, and a stored procedure to track the changes on a server.

To continue the process, all you need to do is to define what metadata you want to track, create a table for it, and then create any views, functions, and stored procedures you need to enter and display the data. You might create tables for file sizes, query activities, and other internal activities on the server. In this chapter, I have explained quite a few views and functions that have to do with security. In a moment, I list a few more.

To get the data from the system views and functions into your own tables, you first need a table to hold the data. You should design these tables to have the structure you plan to report on and any linking columns such as primary or foreign keys. For a view, you can use the INSERT INTO T-SQL command and a subselect to get what you want. I show you how to do that in a moment. For a stored procedure, the process differs a bit. Because you cannot select only certain columns from all stored procedures, you have to put everything from the stored procedure into a temporary table and then select what you want from there just as you do in a view.

Let's look at an example of getting information from a view into a permanent table. In this case, I have decided that I want to store information from the sys.sql_logins system view that shows the SQL Server logins on the server. I want to historically track the non-Windows accounts, but all I care about are the names of the accounts, when they were created, and their default database. The following script creates a table called SQLLogins in the DBAMDT database to hold the data:

USE DBAMDT
GO
CREATE TABLE [dbo].[SQLLogins](
[SQLLoginKey] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS
NOT NULL,
[Created] [datetime] NOT NULL,
[DefaultDB] [varchar](55) COLLATE
SQL_Latin1_General_CP1_CI_AS NOT NULL,
[DateRecorded] [datetime] NOT NULL CONSTRAINT
[DF_SQLLogins_DateRecorded] DEFAULT (getdate()),
CONSTRAINT [PK_SQLLogins] PRIMARY KEY CLUSTERED
(
[SQLLoginKey] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO

I have created a column called DateRecorded that has a default value of a function that returns the current date (GEtdATE()). That provides the historical date I need for when the entry is created.

With the table all set, I can now use the INSERT statement to fill it with data. The basic format is the INSERT command followed by the table name and columns, and then the SELECT statement that retrieves the data. Here is my code:

USE DBAMDT
GO
INSERT dbo.SQLLogins (
 Name
, Created
, DefaultDB)
SELECT
 name
, create_date
, default_database_name
FROM sys.sql_logins
GO

This is just an example; you can use the same two-step process with any view or SELECT-accessible function. But what about those that are not, such as stored procedures that you cannot run a SELECT against? This example uses the xp_msver stored procedure to illustrate the stored procedure data mining process.

/* Look at the structure of the output from this
extended procedure: */
EXEC xp_msver
GO
/* Now use the output to create a temporary table with that
structure */
CREATE TABLE #ServerInfoHolder
(IndexNumber int
, Name varchar(55)
, Internal_Value varchar(55)
, Character_Value varchar(200)
)
/* And put the data from the stored procedure there.
This works the same way for a function. */
INSERT
INTO #ServerInfoHolder
EXEC xp_msver
/* Now you can look at the data in whatever format you want,
and use an INSERT statement to place it where you need it.
*/
SELECT Name, Character_Value
FROM #ServerInfoHolder
GO

There are other ways to do this, but the advantage here is that the data is available for many uses. You could "shred" out the columns into different insert operations and so forth.

This chart shows a few more security-related views and functions you can query for your base tables.

Table or Function | Shows
sys.server_permissions | Server-level permissions.
sys.database_permissions | Database-level permissions.
sys.fn_builtin_permissions | System permissions. This is useful to audit if anyone has changed the built-in defaults.
sys.server_Principals | Shows all logins on the server.

You can also build your own auditing by placing a DDL trigger on a table. Anytime someone modifies that table, you can have the code insert a row in your auditing table.


Chapter 5. Monitoring and Optimization

In this chapter, I explain the tools and methods you can use to monitor and optimize your database servers. Monitoring is more than just observing the results of activity. Monitoring helps track down issues, determines growth patterns, and aids in optimization. I explain what you should monitor and why, and I discuss the tools at your disposal to help you along the way.

Before you begin monitoring the system for performance, growth, or other changes, you should have an idea of its original state, or at least the state it was in when you inherited it. These measurements are called a baseline, and having one is a bit like insurance: invaluable only when you need it. I show you what information to capture to form a baseline for your database servers. I also give you practical advice on broad concepts you can use to enhance your troubleshooting skills.

Systems are made up of applications, networks, clients, and servers. Although the developers have the biggest impact on performance, in the second half of the chapter I explain what steps you can take to optimize the database platform and environment. I also show you some useful tools to help you and your developers make the system more responsive.

The "Take Away" section explains one of the methodologies that I use to monitor and optimize the systems I deal with on a daily basis. Using this outline, you can take some of the mystery out of a poorly running application.

The job of a database administrator (DBA) can be boiled down to just three primary tasks: create the system, make it safe and secure, and make it go fast. In the previous chapters, I have shown you how to create the system and how to make it safe and secure. In the first part of this chapter, I cover most of what you will need to monitor your system, which is an integral part to making it go fast.

Your managers (hopefully) spent quite a bit of time planning before they decided on a particular architecture, software vendors, and tools for your organization. They probably spent more than a little money to buy it all.

If you have read this far in the book, you have probably spent a lot of time to learn how to properly size a system, and you have learned how to configure it to be responsive. The developers spent quite a bit of effort writing the best code they know how, using the latest coding techniques, languages, and tools.

With all this time, effort, and money invested, everyone has an expectation that the system will be responsive. When quality or performance issues occur, the users want to know why. They ask the managers, and the managers ask the technical team. Sometimes the answer is not easy to find. If the answer is hard to come by, it is often because there has not been enough time, effort, and money spent on periodic monitoring and tuning the system.


Monitoring Your System

Whether you are looking at a system to determine growth patterns, locate a problem, or determine where a performance bottleneck is, you first have to know where to look. Fortunately, even though most enterprise systems are complex, computers have only four basic parts: processor(s), memory, storage, and network interfaces. There are a lot more components in the hardware architecture of a system, but they all affect or are affected by one or more of these parts.

Growth occurs as an increase in storage space and memory use and shows up as more traffic on the network interfaces and greater load on the processors. Problems occur in one of the four parts of the system, or more often in a combination of them. Bottlenecks follow the same pattern.

All of these components change over time. But before you can see a change, you have to know either what the system is supposed to look like or what it looked like in an earlier state. That is where a baseline comes into play.

Setting a Baseline

You may be fortunate enough to start with a "fresh" system, one that you have built yourself with lots of planning, an adequate budget, and enough time to implement. More probably you inherited an incorrectly sized system that was rushed into production and whose use has grown far beyond its original purpose. Either way, monitoring begins with a baseline. A baseline is a set of measurements from some earlier point in time, preferably right after you build and configure the system.

The baselining process follows a simple outline:

  1. Find a representative set of measurements that move with activity.

  2. Understand the metrics associated with those measures.

  3. Set up a controlled amount of the activity.

  4. Collect and record the metrics.

Although the outline is simple, the steps to implement it can become quite complex. The reason is that most applications have multiple interactions that flow through various paths. In addition, the interactions often do not correlate in a linear fashion. For instance, in a given application, memory does not climb at the same rate as connections.

To accurately monitor a system, you need a thorough understanding of the application and platform architecture to find representative measurements and the metrics associated with those measures.

To monitor for growth or to find issues, you can use your test or production environments, but you will find that the best way to monitor for performance is in a controlled environment.

A complete set of baselines includes a stasis test, where the system has no connections; an in-use test, where an average load is placed on the system; and a stress-test, where the system is placed under heavy use. The basic idea in measurement is running several identical tests and taking the average of the metrics you collect. That forms the first part of your baseline.

Baseline metrics belong in permanent form, just like your security audit. Once again, I advocate placing the metrics in the DBAMDT database or something like it and then creating permanent reports from there. Whether you store the data electronically or not is up to you, but you need to compare lots of numbers during a performance-tuning exercise, and the electronic format favors that process. In any case, you have to record the data somewhere.

What to Monitor

Earlier I mentioned that there are four parts in a computer system, but none of them stand alone. The processor is affected by its architecture, cache, the bus, and even the motherboard it is connected to. Memory is also affected by architecture and is connected to a bus and the motherboard. Network cards are affected by the bus speed and caching. Probably the most complex part is storage, also called I/O. In fact, storage is often called a subsystem because it may not even be controlled directly by the server.

Taken together, these affected parts form the major components of the system. To find performance problems or functional issues, you are looking for any deviation from the baseline or other standard within the components. When you locate those deviations, you can break down each of the parts of the system, from the client systems forward, focusing on any parts that specifically affect the SQL Servers. You can then trace the components to the parts they affect and drill down on the activity there. It is a matter of starting with the general and moving down to the specific.

Consider an application that has a full client installed on a workstation connecting directly to SQL Server 2005. To find a problem, inspect the interface the client software uses, the connection methods to and from the server, and monitor the four primary parts within the server that the client activity produces.

When you troubleshoot, monitor in conjunction with the application suite, taking combined metrics on both the client and server(s) and performing tests in tandem. It is important to combine the testing because the problem might exist in two places at once.

For performance tuning and baselining, you can monitor SQL Server and its affected parts. In both cases, you should monitor two classes of objects: platform and database activity.

Monitoring the platform involves more than just collecting database metrics. You also need to record information about the hardware, operating system, and other layers on the system that affect the growth or performance of the system, because the two are related. In this section, I show you how to use a combination of operating system utilities and mechanisms in SQL Server 2005 to monitor both kinds of activity. You can also use third-party software hardware solutions to measure platform activity, but I stick to the tools installed by default.

Monitoring database activity is easy to do but difficult to interpret properly. There are several ways to monitor database activity, from embedding trace code in the application to using tools on the server that watch the activity and report on it. Interpreting the data you receive from the tests needs to be done in conjunction with the development team because only they know what the application is doing and why.

To develop a list of the specific objects to monitor, focus on the purpose of the monitoring. The primary reasons you are monitoring the system is either for growth prediction and management, activity as it relates to performance, changes to the structure (Data Definition Language, DDL), or to find and correct problems or issues. Most of the monitoring techniques can be used for multiple purposes, but monitoring for growth is slightly different from monitoring for performance, and so on.

Growth

One of the most common purposes of monitoring is to measure and predict growth. This exercise requires a baseline, taken when the system is new or when you inherit it. The baseline provides the "starting point" for growth predictions. Monitor each system and collect metrics from the four categories of parts: processor, network interface, storage, and memory.

Specificlly, you want to monitor the following objects and collect their values.

Object Class | Value to Capture
Processor | % of utilization
Network | Traffic, total in bytes
Storage | Total transfers, in operations or blocks
Memory | Megabytes or gigabytes in use
Database | Size (for each database)

By taking a snapshot of these items and tracking the same measurements over time, you can begin to plot a growth trend. That much is obvious, but the important decisions are how often to take the measurements and how many to take.

If you are not sure where to start, monitor with a short interval (5 or 10 seconds) for 24 hours, for a week. After that, take the 24-hour measurements once a week for a month, and then once a month for two months.

With those numbers in hand, first check the daily range. Arrange the numbers by time and then create a graph from them. The numbers show a trend, either flat, up, or down. If the graph shows a general trend upward, arrange the numbers from lowest to highest values. Plot the points again along a graph and evaluate how often they change. You might see a consistent growth pattern or one where the numbers rise in slopes and plateaus.

If the numbers rise in a predictable pattern in an upward trend, take the first measurement of the week, the last measurement of the week, and predict from that pattern what the growth pattern should be for the next three weeks. If your predictions are correct, they will match the weekly numbers you captured after the daily measurements. Follow the same pattern to test against the monthly numbers.

If the metrics originally show a flat line, you need to measure for a longer period of time. Measure weekly, but for three months or more. Follow the same process as above, using the weekly trends to predict the next weeks and months.

If the measurements show a downward trend, chances are that the system was experiencing a significant change or archiving operation. Wait a week and then begin the daily measurements again.

Like all monitoring, this is not a one-time process. You should repeat the exercise each year, testing your theories along the way. The longer you monitor, the more accurate your predictions become.

Activity and Performance

Monitoring to examine activity is usually done to determine performance on an application. You have two choices when you monitor for performance. The first is to record these historical values just as you did in monitoring for growth.

Object Class | Value to Capture
Processor | % of utilization, specific to the processes involved in the application
Network | Bytes read and written
Storage | Read and write operations
Memory | Amount SQL Server wants to use and amount SQL Server is granted
Database | Database activities, connections, and locks

After you have the metrics, organize them by time. Calculate the minimums, maximums, and averages. To make the averages meaningful, you also want to include a standard deviation of the set. The closer the standard deviation is to zero, the more you can trust the average.

To make the performance monitoring meaningful, correlate the metrics with activity. SQL Server makes this process easy, as you will see in a moment. I show you a complete performance-tuning exercise in the "Take Away" section at the end of the chapter.

Data Definition Changes

Another purpose for SQL Server 2005 monitoring is tracking changes on the tables, views, and other objects in the database. SQL Server 2005 includes DDL triggers, and you can capture the changes to another table or even prevent those using triggers. I show you how to do that in a moment.

Issues

Most of the time, you will not be pressured to monitor your system. The only time management or the users will be interested in the results of monitoring involves an issue or problem. When things stop working, everyone is suddenly concerned about the state of the system.

There is not a chart of objects to monitor this time, because the problem might exist in only one part or in multiple parts of the system. A better approach is to use basic troubleshooting skills to find the objects that show movement, and monitor the specific objects that are showing the issue. You can then drill down on the item(s) that caused the problem. Let's take a look at that process.

Troubleshooting

Troubleshooting skill is one of the most valuable assets you have as a DBA. The process is not difficult from a conceptual standpoint: Evaluate the symptoms and determine the cause of the issue. That is easily stated, but hard to implement.

You can make troubleshooting much simpler by knowing a lot about the system. The more you know about how something operates, the quicker you can narrow down the places you need to look for interactions that might be causing the problem. So, the first tool you have in troubleshooting is educating yourself about how SQL Server 2005 operates. You are doing that now, by reading this book.

Beyond that, it is a process of elimination. Most problems fall into only a few categories. It is either an issue with the physical environment or layout, something on the server, or in the application software. Often, the more complex issues are located in all three areas, and as you fix one you will find the next.

Locating issues within the physical environment starts with running an entirely different application that takes the same path as the one having the trouble. If you see similar problems with a different application, chances are it is in the environment, not the specific application where the problem was reported. Work backward from the database, taking out one variable at a time until you locate the cable, drive, memory chip, or other cause for the problem.

Finding issues on the server platform is easier if you established baselines and performed audits on your system earlier. If you know what the system looked like when it was running correctly, you can compare to the current state to locate the differences. After all, if something was running correctly once before and now it is not, the problem normally lies in the change. Find the change and you will most likely find the problem. If you did not take a baseline, you will need to test each part of the system separately, eliminating the most likely suspects first.

Finding an application problem is the most complicated type of issue to resolve. If there has been a recent upgrade or change, always start there; in a large application, however, it may be that the users have only now exercised an option or reached a threshold where the error occurs.

In most cases, the application will not be the first place that the team looks, so it is up to you to rule out the physical environment and the platform as the cause for the problem. With that established, you can use the SQL Management Studio and SQL Server Profiler tools to show the activity in the database. Depending on the type of issue, you can add counters in System Monitor that will correlate to the problem. Again, it is a process of elimination.

Other problems occur within a mixture of the application and the platform. A prime example of this is in issues involving locks. Locks occur when an application reads or writes data. If an application is reading data, it is important that the data stays consistent, so a lock is taken on the data during the read. This prevents updates to the data until the read operation is complete. This can cause concurrency issues if the code is not written properly.

SQL Server 2005 mitigates this problem by allowing a copy of the data to be held while the original data is updated, so reads do not block writes. Even so, you can still run into an issue where one update depends on another, called a deadlock. In this case, SQL Server picks a "victim" connection and terminates it, sending the application a special signal in code. Developers can watch for this signal and deal with it. If you are experiencing locking issues, monitor with the tools I describe for both locks and deadlocks. Although some locking is unavoidable, excessive locks and deadlocks are good places to start troubleshooting applications.

Regardless of the problem, use the tools and procedures I describe here to monitor what is happening and where it is happening. From there, you can start to answer why it is happening.

Tools

Because SQL Server 2005 is integrated into the Microsoft Windows platform, you can use several operating system commands in conjunction with SQL Server-specific tools to create a complete picture of your server's load. In fact, some SQL Server tools such as SQL Server Profiler actually integrate operating system monitoring tools into their operations.

Several programs included with SQL Server 2005 provide measurements on the platform's activity. Most of these have interfaces that also enable developers to include monitoring within their applications.

Let's examine the tools from the operating system and SQL Server that show you the general and specific activity on the server.

Operating System Commands and Programs

Because SQL Server 2005 runs only on the Microsoft platform, it has a distinct advantage in being able to rely on operating system consistency and its programming interfaces. Many of these are implemented by programs rather than commands, but some commands can show activity that has an effect on SQL Server.

You can also write your own programs for monitoring, even if you are not a professional developer. Using Windows Management Instrumentation (WMI) components in the operating system, you can create simple scripts to show you information about the environment on the servers involved.

The script format is to declare a WMI object and then query it using the WMI Query Language (WQL). WMI scripts to find out information work using classes. Here is an example script that shows the memory for the system using the Computer System class:

' Use the current system
strComputer = "."
' Get the management object
Set objWMIService = GetObject("winmgmts:" _
 & "{impersonationLevel=impersonate}!\\" _
 & strComputer & "\root\cimv2")
'Set the query
Set colSettings = objWMIService.ExecQuery _
 ("Select * from Win32_ComputerSystem")
' Iterate through the objects you are interested in
For Each objComputer in colSettings
 Wscript.Echo "System Name: " & objComputer.Name
 Wscript.Echo "Total Physical Memory: " & _
 objComputer.TotalPhysicalMemory
Next

To show this script in action, enter and save it as a text file called memory.sc and then invoke the scripting engine on it from the operating system command line:

wscript memory.sc

SQL Server also has WMI classes you can use, for both the SQL Server engine and Reporting Services, in addition to others. Search Books Online for "WMI" to find a wealth of scripts that provide information as well as controls for your installation.

Windows Event Logs

The first place within the operating system that you can find SQL Server information is in the Application event log. The event log display program is located in the Administrative Tools area in the Start menu on the Windows server operating systems. You can right-click My Computer and select Manage to locate them on all current Windows operating systems.

Once inside the Event Viewer, locate the Application log. You can filter events by category or date, but I normally do not filter anything out at first. Sometimes the problem or performance information is lurking in places other than SQL Server, so I want to make sure and catch those, too.

Events come in types. The types are Information, which is just as it says; Warning, which indicates there may be a problem or that one is impending; and Error. If you are trying to find a problem, you might want to investigate the most recent Error types, and then work back through the stack with the arrow icons to where the first Warning type dealing with that error occurred. When you locate an item that you find interesting, double-click it.

Figure 5-1 shows that the properties of an event contains when the event happened, the event ID, and the application name that reported the event. The description has whatever the developer wants you to know about the event, so some of these are more robust than others. The event also carries a data payload that you can see in bytes or words.

Figure 5-1.

[View full size image]

Task Manager

One of the easiest tools to use to show activity on your server is the Windows Task Manager. Right-click your Start menu bar in any blank area and select Task Manager from the menu that appears. Click the Processes tab to show the individual tasks that are running on your system. Check the Show processes from all users box to see the tasks you do not own. I've done that on my test system in Figure 5-2.

Figure 5-2.

[View full size image]

Your System You can click the headings at the top of the panel to sort the results. On this test system, SQL Server is taking most of the memory, which is to be expected.

You can also sort by the processor utilization to show what is taking most of the resources on the system, but this number is a gross level and not always useful. Switching to the Performance tab shows a graph of the processor use.

At the bottom of the panel shown in Figure 5-3, there are two measures that are important to examine. If the Total value in the Commit Charge (K) area is larger than the Total value in the Physical Memory (K) section, the system is using more memory than it physically has. At that point, the server is paging, which means it is shifting contents of memory to the hard drive and back again. Because physical disks are several orders of magnitude slower than solid-state memory, that factor alone causes an incredible slowdown in performance, and can even cause timeouts that create issues.

Figure 5-3.

[View full size image]

Task Manager is a great place to start when you are diagnosing an issue that is causing a system-wide failure. It illuminates or eliminates areas for you to check.

System Monitor

One of the most useful tools in your arsenal is the Windows System Monitor, which is sometimes referred to as Performance Monitor, from the older version within Windows that looked and operated in a similar way. It works using objects, such as Processor or Memory, and counters, which are things that are recorded about the objects, such as % Used or Total. The number recorded in the counter at a given instant for an object is called a value.

The System Monitor works by reading special registry keys that the operating system controls. Applications, including the operating system, write values to these registry keys every second. This three-way handoff allows the most efficient way to monitor a system.

SQL Server 2005 includes multiple counters that you can monitor, and the advantage is that you can combine measurements into one tool to show cause and effect.

System Monitor has three modes. It can show the values you select in real time, it can collect them and then alert you when an upper or lower boundary is reached, and it can record the information to a log file.

The real-time modes of System Monitor show the values in a graph, a histogram, and report formats. These formats show the state of the objects, but when you include more than a few objects, all of the formats becomes incomprehensible. I normally monitor in this mode only when I have found a particular area to focus on.

Let's take a quick tour of the tool using real-time monitoring. Open the

System Monitor on your SQL Server 2005 system from the Start menu under Administrative Tools, then Performance. If you are using a nonserver Windows operating system, it might be located under a different submenu.

On most operating systems, the tool will automatically show three or four counters loaded and running, as you can see in Figure 5-4. You can click the X in the icon bar to remove those, because you will pick the items you want from a list in a moment.

Figure 5-4.

[View full size image]

To add objects and then counters to monitor, click the plus sign (+) in the icon bar.

Monitor one computer from another by choosing the Select counters from computer radio button and then entering the name of the server, prefaced by two backslashes. You need to have administration privileges on that system to add the counters remotely, which is why I suggest for this example that you run the tool directly from your test server.

In the Performance object selection, you can choose which objects you want to monitor, and in the selection below that you can select as many counters from the object as you want. You can also choose to monitor all counters; but because monitoring takes a performance toll, it is better to be more specific.

On the right side of the panel shown in Figure 5-5 are the instances associated with the counter. An instance is the count of items within that object, such as the processor number. Not all objects have more than one instance.

Figure 5-5.

For the CPU object, you could choose to monitor a single processor at a time or the combined total of all processors. This might matter if you discover that a query is not parallelizing as you think it should. The Microsoft Windows operating system takes advantage of symmetric multiprocessing (SMP), but SQL Server is one exception that can use processor affinity, meaning that you can choose to run SQL Server 2005 on a particular set of processors, as I explained earlier in the book.

You can click the Explain button to open a small box with a simple explanation of the counter, but it does not often help much. It is better to review the previous chapters to see how the hardware and operating system affect SQL Server activities to understand what each counter does. You can also search the Web to find more information about a counter, but make sure to cross-reference what you find. Counter values can be interpreted differently based on what they are paired with and how often they are measured.

As you click the Add button for each counter, you will notice the graph begin to move behind the Objects panel. To see some interactions on the platform, add the following objects and counters, taking the default instances of each.

Object | Counter | Meaning
Processor | % Processor Time | Shows the percent of time during monitoring that the processor is in use.
Memory | Available Mbytes | Shows how much memory is left. Reverse the number from the total memory to find out how much you are using.
LogicalDisk | % Disk Time | Shows the percentage of read and write time during monitoring.
Network Interface | Bytes Total/sec | Shows how many bytes are transferred in and out.
SQLServer:Databases | Active Transactions | Shows all the transaction activity for the databases selected in the instances area.
SQLServer:General | User Connections | Shows the number of users that Statistics are connected to the server.

That is enough for now; what we care about at the moment is to see the movement and interplay between objects.

Log in to SQL Server Management Studio and run one query, or just click around in the Object Explorer. Size and arrange the screens so that you can see both of them as you do. Notice the activity levels in the System Monitor as the values of the counters rise and fall.

Change the format in the System Monitor by clicking the button that looks like a bar graph in the icon bar. You will see the view change to a histogram, which shows movement up and down over time. This is a useful view for items such as memory use or drive space, or other items that lend themselves to a "full or empty" kind of display.

Now click the button that looks like a notepad. This view shows a report of the output, but other than spot monitoring, you may not find a lot interesting here. From time to time, I have used the output to show others a particular set of values, especially if they are slowly changing.

Right-click the main part of the System Monitor display and select Properties from the menu that appears. From the Graph tab, you can change the scale, and in the General tab, you can change how often you take the sample.

These instant-feedback views have some limited uses, but to locate an elusive issue, it is more effective to use an alert. An alert is a condition you set when a value goes high or low for a specified period of time. The alert can send a notification or run an operating system command. Using an alert with the notification function alone tells you that the threshold is exceeded. Pairing this function with the sqlcmd command and scripts allows you to watch for a condition and take corrective action.

Let's create an example alert that opens the Windows program called Notepad when the processor exceeds a certain amount of use. For this example, I use 7 percent of processor for 1 second. Change the time or amount as needed on your test system if that is too little a change to monitor for.

Begin by clicking the Performance Logs and Alerts item on the left side of the System Monitor, and then right-click the Alerts object and select New Alert Settings from the menu that appears.

On the first panel, shown here in Figure 5-6, enter any name and description for the alert. Add a counter of Processor, % Processor Time. Set it to alert when it is over 7 percent, or whatever your system shows as a higher-than-average percentage.

Figure 5-6.

[View full size image]

The sampling rate is how often the system will check for the value. Set the interval for every second and then set the account that will run the alert.

Move to the Action tab as shown in Figure 5-7 to set a program that runs when the threshold is reached. In this example, I have set the action to log to the Event Viewer and to run the notepad.exe application. You could also have the alert start up a recording session of System Monitor, so that you can begin a more in-depth monitoring session automatically only when a problem occurs.

Figure 5-7.

The Example command-line arguments box gives you the text output for the parameters on the command-line version of this tool. Using these parameters at the command line allows you to set up this same alert in text rather than this graphical method.

On the Schedule tab, you can set the alert to monitor at a certain time or on a schedule. Set this to Manual.

Save the alert, right-click it, and select Start from the menu that appears. If the Notepad program does not start, check the Application event log and look for a message from SysMon. It may be that your path differs from the example steps I have here.

Alerts are useful to find a condition and trigger actions based on thresholds. Another useful function of the System Monitor is the Log file function. This function records the values that System Monitor collects and sends them to a file, either in binary format or in ASCII.

To use the Log function, select the objects and counters you want to monitor and designate a file to send the results to. Set a time to start and stop the logging, and you can then read the file into another session of System Monitor using the binary log format, a spreadsheet such as Microsoft Excel when you use a comma-separated value format, or even read either file into SQL Server Profiler.

No matter which function within System Monitor you use, there is an amazing array of counters that you can add for SQL Server. Search for "Monitoring Resource Usage (System Monitor)" in Books Online to see a list of objects and counters broken down by area. Start with the "outer objects" such as Processor, Memory, Logical Disk, and Network Interface. Use the General Statistics object in SQL Server to locate what is moving when you see the issue or are trying to locate the bottleneck. When a counter does not correlate to an issue or performance event, remove it. When a counter does move with these two items, add more detailed objects and counters affected by that area.

You can obtain additional programs in SQL Server or Windows "Resource Kits" from Microsoft that show SQL Server activity as it relates to the operating system. Check the Microsoft SQL Server Web site at www.microsoft.com/sql to find out more about these tools.

SQL Server Management Studio

Although you will not often monitor for historical purposes using the SQL Server Management Studio, it does have features that provide good feedback on a query. Each query you run has several properties associated with it that you can use to track metrics around it.

One of the most useful features within Management Studio is the reporting output that is available from the Summary pane. If that is not showing on your system, click the View item in the menu bar and select Summary. Click the name of the server you are interested in from the Object Explorer, and a button called Report lights up in the Summary tab. There are several reports here that show just about everything that is happening on your server. You can even export these reports to Microsoft Excel or PDF formats. For most descriptive monitoring, the reports here cover everything you need.

Management Studio is also one of the command-based tools used to call System Views and Functions that show historical metrics and metadata, but I leave that discussion for later. Let's examine how you can use the SQL Management Studio to monitor your server.

The Management Object

You can also use two items in the Management item in the Object Explorer to monitor your server. In the Object Explorer pane, drill down to the Management object and expand it.

To get a graphical representation of the activity on your server, right-click the Activity Monitor item and select View Processes from the menu. This brings up a panel similar to Figure 5-8 that contains information on the users and locks in the system. You can double-click any username to see a short output of the query they are running, as well as several columns describing the objects they are accessing, locks, client system names, and much more.

Figure 5-8.

[View full size image]

At the upper left of this panel, you can select views that show locks by process or object, and you can also right-click the menu of Activity Monitor to get those same views.

Just above the Activity Monitor is the SQL Server Logs object. SQL Server 2005 tracks a great deal of information about its state in seven "rolling" logs. A new log is created every time SQL Server is started, and the previous logs are renamed in numeric format. If you right-click the SQL Server Logs object, you can click Configure and set the amount of logs SQL Server 2005 collects. The logs show a huge amount of data about the internal workings within SQL Server 2005. They are the first place to start when you have a platform issue involving the server or the SQL Server software.

Open any SQL Server log and you can also view the Windows System, Application, and Security logs. This is useful because you can correlate what's happening in the operating system with events in SQL Server 2005. You can see an example of that in Figure 5-9.

Figure 5-9.

[View full size image]

The SQL Server log files, unlike the Windows event log files, are just text. You can read them with any editor, and if you took the defaults when you installed the server, they are at C:\Program Files\Microsoft SQL Server\MSSQL.n\MSSQL\LOG\ERRORLOG and ERRORLOG.n . Replace the first n with the instance you want to examine, and the second n with the number of the log you are interested in. This is useful for automated monitoring using scripts from the operating systems or from other applications.

Query Properties

So far you have seen the activity the clients produce. You can also monitor your own transactions in SQL Server Management Studio by using Query Properties. This is useful for developers because they can snip out the T-SQL from their code and run it here to see what it does.

To see how this works, open Management Studio and then open a query window. In this case, we will use the AdventureWorks database because it has a lot of data already. If you did not install the sample databases, use any database you want, substituting the proper columns and database names where appropriate. Make sure that you click the View item in the menu bar and select Properties. Enter and run the following query:

USE AdventureWorks
GO
SELECT a.LoginID, a.Title, c.City
FROM HumanResources.Employee a
INNER JOIN HumanResources.EmployeeAddress b
ON a.EmployeeID = b.EmployeeID
INNER JOIN Person.Address c
ON b.AddressID = c.AddressID
ORDER BY a.LoginID

When that query runs, you will get some feedback on how long the query took, how many rows were returned, the ID of the process that ran the query (called the SPID), and so forth.

As you can see in Figure 5-10, there is nothing earth shattering here, but it can be useful to have simple, quick metrics on what the query does.

Figure 5-10.

[View full size image]

Execution Plans

Of more use is the ability to examine the execution plan associated with a query. An execution plan is a graphical representation of the internal path the engine takes to satisfy the query. When the plan is generated, you can examine it by hovering over each step, which provides even more information. Knowing the path gives the developer clues as to what is taking the most time, and what they can do to optimize the path. I do not spend a lot of time on the query here because we are more interested in the administration part of the system, but it is useful to show the developers they have this tool available and to look at your own queries to gauge how efficient they are.

Let's look at the path that example query takes on the Adventure-Works database. If you are still in Management Studio from the last query, close the Properties view and select Query and then Include Actual Execution Plan from the menu bar. Now run the same query again.

When the query finishes (it will take slightly longer this way), you will get a new tab in the bottom panel, called Execution Plan. Click that to see the graphical output from the query engine, as shown in Figure 5-11.

Figure 5-11.

[View full size image]

I will come back to the graphical elements and what they mean in a bit. For now, hover the cursor over each icon. You will see a large amount of text that explains metadata about that step in the query.

You can save these plans out to a file by right-clicking anywhere in the graphical area of the output. You can display saved plans in another session of Management Studio by using the File and then Open items from the menu. You can use that output in group meetings when you want to optimize performance.

Query plans are great tools for optimization, especially when the developers test the T-SQL they include in their applications. I explain a little more about execution plans in the next section.

DDL Triggers

You also have the ability to watch for data structure changes on the system. You can create DDL triggers in Management Studio that perform actions when someone attempts to change an object in the database. You can have the trigger log the action, notify you, or even prevent the change.

To show how this works, let's create a table that shows a change log on DDL operations:

USE DBAMDT
GO
CREATE TABLE dbo.Tmp_ChangeLog
     (
     DateChanged datetime NOT NULL,
     UserName varchar(55) NOT NULL,
     ChangeText varchar(255) NOT NULL
     ) ON [PRIMARY]
GO

This script creates a table with three columns: one for the date and time that the change occurred, one for the principal making the change, and another for the text message regarding the change.

Now let's create a trigger on the ServerInfo table that populates the ChangeLog values:

CREATE TRIGGER ChangeLogTracking
ON DATABASE
FOR ALTER_TABLE
AS
INSERT INTO ChangeLog
SELECT GETDATE()
, CURRENT_USER
, 'Table Changed.'
GO

We are using two system functions here, one for the current date and time (GEtdATE()) and another for the principal's name (CURRENT_USER). The last part enters the words Table Changed, so that we can see that a change has occurred.

Now let's change one of the columns in the ServerInfo table; don't worry, we won't really change it:

BEGIN TRANSACTION
ALTER TABLE ServerInfo
ALTER COLUMN LastUpdated
DROP DEFAULT

And, finally, we read from the change log to see what has been changed:

SELECT *
FROM ChangeLog

Now let's put all that back the way it was. Add this text to the end:

ROLLBACK
GO

And now press F5 to make it all go. The BEGIN TRANSACTION and ROLLBACK statements bracket what we have done, so the change is not permanent. Here is the output on my test system:

_________________
2005-10-20 08:46:44.920 dbo Table Changed.

This script is not incredibly useful, because it allows changes and does not really tell us what has changed. You could make it more useful by adding some metadata from the EVENTDATA() function that would show the text from the change script the user tried to run. I do not spend a lot of time on that function here because it returns data in an XML format, and you will need to use a bit of XPath query functions to get the parts you want. If you are interested in learning more about that, search for "EVENTDATA()" in Books Online for more information.

You can use two scopes for DDL triggers: database and server. You need to have server-level permissions for the server scope and database-level permissions for the database scope.

Here is a table that shows the events you can trigger on for the database:

ALTER_APPLICATION_ROLE | ALTER_ASSEMBLY | ALTER_AUTHORIZATION_DATABASE
ALTER_CERTIFICATE | ALTER_FUNCTION | ALTER_INDEX
ALTER_MESSAGE_TYPE | ALTER_PARTITION_FUNCTION | ALTER_PARTITION_SCHEME
ALTER_PROCEDURE | ALTER_QUEUE | ALTER_REMOTE_SERVICE_BINDING
ALTER_ROLE | ALTER_ROUTE | ALTER_SCHEMA
ALTER_SERVICE | ALTER_TABLE | ALTER_TRIGGER
ALTER_USER | ALTER_XML_SCHEMA_COLLECTION | CREATE_APPLICATION_ROLE
CREATE_ASSEMBLY | CREATE_CERTIFICATE | CREATE_CONTRACT
CREATE_EVENT_NOTIFICATION | CREATE_FUNCTION | CREATE_INDEX
CREATE_MESSAGE_TYPE | CREATE_PARTITION_ FUNCTION | CREATE_PARTITION_SCHEME
CREATE_PROCEDURE | CREATE_QUEUE | CREATE_REMOTE_SERVICE_BINDING
CREATE_ROLE | CREATE_ROUTE | CREATE_SCHEMA
CREATE_SERVICE | CREATE_STATISTICS | CREATE_SYNONYM
CREATE_TABLE | CREATE_TRIGGER | CREATE_TYPE
CREATE_USER | CREATE_VIEW | CREATE_XML_SCHEMA_COLLECTION
DENY_DATABASE | DROP_APPLICATION_ROLE | DROP_ASSEMBLY
DROP_CERTIFICATE | DROP_CONTRACT | DROP_EVENT_NOTIFICATION
DROP_FUNCTION | DROP_INDEX | DROP_MESSAGE_TYPE
DROP_PARTITION_FUNCTION | DROP_PARTITION_SCHEME | DROP_PROCEDURE
DROP_QUEUE | DROP_REMOTE_ SERVICE_BINDING | DROP_ROLE
DROP_ROUTE | DROP_SCHEMA | DROP_SERVICE
DROP_STATISTICS | DROP_SYNONYM | DROP_TABLE
DROP_TRIGGER | DROP_TYPE | DROP_USER
DROP_VIEW | DROP_XML_SCHEMA_COLLECTION | GRANT_DATABASE
REVOKE_DATABASE | UPDATE_STATISTICS |

And here are the server-level triggers you can create:

ALTER_AUTHORIZATION_SERVER | ALTER_DATABASE | ALTER_LOGIN
CREATE_DATABASE | CREATE_ENDPOINT | CREATE_LOGIN
DENY_SERVER | DROP_DATABASE | DROP_ENDPOINT
DROP_LOGIN | GRANT_SERVER | REVOKE_SERVER

Using DDL triggers is a great way to control and document the changes that happen in your database.

You might be wondering about the performance impact of triggers in general, and DDL triggers in specific. There is a slight hit on the system each time a trigger is fired because it adds to the transactions that run each time. Triggers can also cause other triggers to fire, and in addition they can refer back to each other. SQL Server handles this fairly well, and you can also control the trigger flow, although this is normally something the developers are more concerned with. The short answer is that if DDL triggers are used properly, the performance impacts can be managed. DDL operations do not happen as often (if you do not count temporary tables), so as long as the trigger logic is not overly complex, performance impacts are normally low.

System Views and Functions

SQL Server 2005 tracks an amazing amount of information about itself. The data it collects is accessed through system functions, stored procedures, dynamic views, and catalog views. These are the objects that you will use most often to show the activity within your system.

In previous versions of SQL Server, several system databases stored a great deal of metadata. These have been replaced by objects owned by the sys schema. Most of the older tables are reproduced in this schema, and new views have been added. Here is a script that shows you all of these objects:

SELECT name
FROM sys.all_objects
WHERE schema_id =
(
SELECT schema_id
FROM sys.schemas
WHERE name = 'sys'
)
ORDER BY name

You also have a set of views that are owned by the schema called INFORMATION_SCHEMA. These are also older views that show common system objects. You can find those in the graphical tools or by replacing the 'sys' value in the WHERE line with 'INFORMATION_SCHEMA'.

The newer dynamic views are not actually views in the SQL Server sense. When you select data from them, you are calling a system function, which talks to programs written against the database engine. There are dozens of these views, and I only show a few here. To see the descriptions for all of them, search on the topic "Dynamic Management Views and Functions" in Books Online. You can also run this script to get their names:

SELECT name
FROM sys.all_objects
WHERE name LIKE 'dm%'
ORDER BY name

In Books Online, each dynamic view is categorized by functions, but I describe a few by the type of monitoring you need to do. Many of these functions and views can be used to form the basis of your DBAMDT configuration. Remember that these views are "materialized," meaning that they show the state of the server at that moment. If you want to show a particular value over time, you need to create automation such as a job to populate the DBAMDT tables from the views and functions.

Server, Database, and User Information

Let's start off by having SQL Server 2005 give us some of the same information I explained in the section on performance counters. First, let's take a look at the methods to get at this data.

When you use the sys or INFORMATION SCHEMA objects, you can treat them just like a table.

SQL Server 2005 provides the sys.dm_os_performance_counters system view, which shows the SQL Server objects, counters, and values installed on your server. Here is how you can call the view:

SELECT * FROM
sys.dm_os_performance_counters
GO

An older version of this particular data is available in the system view called sys.sysperfinfo.

You can use the SELECT command against a system function, too:

SELECT @@VERSION
GO

or

SELECT GETDATE()
GO

Other information is available through the use of system stored procedures. To see their results, use the EXECUTE or EXEC command:

EXEC sp_who
GO

Remember, the data is only good for that moment in time. Even so, they provide a great way to get metrics without ever leaving Management Studio. The rest of these functions work the same way, by using a SELECT statement, or an EXEC statement for the stored procedures.

Here is a list of the most common server information views, functions, and stored procedures:

Function Name | Shows
@@CONNECTIONS | All connection attempts since the server was started.
@@CPU_BUSY | How busy SQL Server has been, in milliseconds, since the server started. This is for all processors, so this number can be a bit difficult to interpret.
@@IDLE | The opposite of @@Processor_BUSY.
@@SERVERNAME | Name of the server.
@@SERVICENAME | Service name this instance is using.
@@VERSION | Version of SQL Server running.
EXEC sp_help be used for ' object' | Shows information about the object. Can be used on just about any object. Note that there are several dozen sp_help* stored procedures, all worth a look in Books Online.
EXEC sp_lock | Shows information about locks in the system.
EXEC sp_monitor | Server activity in general.
EXEC sp_who | Shows currently logged on user information, as well as the locks they hold on database objects.
sys.dm_io_virtual_file_stats (NULL, NULL) | Disk activity for the server. Replace the NULLs with a specific database ID and file ID to get more-specific information.
sys.dm_os_ performance_counters | SQL performance counters.
sys.dm_os_sys_info | Operating system information such as memory and processor use.
sys.dm_os_tasks | Task-level information. It is a little obscure but very useful when you are tracking down an issue.

And here are a few of the database views and functions.

Function Name | Shows
@@SPID | Current process ID
CURRENT_USER or USER_NAME() | Name of the current login
sys.dm_clr_loaded_assemblies | CLR assemblies that are loaded in the database
sys.dm_db_index_usage_stats | How well the indexes are being used
sys.dm_db_mirroring_connections | Mirrored databases
sys.dm_exec_connections | Lots of user connection information and statistics
sys.dm_exec_query_optimizer_info | A great deal of information about the query optimizer's statistics
sys.dm_exec_requests | Information similar to the monitoring in the Activity Monitor object in Management Studio
sys.dm_tran_active_transactions | Current transaction information

Many other functions, stored procedures, and dynamic views can help you develop a complete picture of your server's activity. I cover more of these in the subsequent chapters.

SQL Server Profiler

When you need to see exactly what is happening on your server at a low level, SQL Server Profiler is the tool you are after. You can find it in the Start menu under Microsoft SQL Server 2005, and then Performance. I covered this tool briefly in Chapter 2, "Architecture and Tools," but I explain more about it here because it is the best utility to use to get at server operation details.

You can think of SQL Server Profiler as a complete recorder for the SQL Server's activities. The basic process is quite simple. You create a trace or recording session by selecting various items to watch. You turn on the trace, which records everything you have asked for to the screen, a file, or a table in a database. After you have collected everything you want to see, you turn off the trace. If you have captured the trace to a file or table, you can load it up later and "replay" it, even against another server. Otherwise, you lose the information once you exit the tool.

To include items to watch, you select them from an event category. Event categories are logical groups of the things you can capture, such as locks or SQL statements. Within these event categories are event classes. These are the individual actions you capture, such as lock acquisitions. For instance, one event class is Lock:Acquired. If you include that event class in a trace, the Profiler will add a line to the trace every time a lock is acquired.

There are performance implications for the Profiler. When you are running the tool, SQL Server has to take a little time to record what it is doing. You normally do not run Profiler all the time, you just turn it on to find a problem or record test data, or perhaps during the design phase of an application to ensure your developers have a good picture of what the engine is doing with their code.

That last reason might seem a bit strange. After all, if a developer writes a certain SQL statement in code, should not that be what runs in the database engine? Not always. In Chapter 2, I explained the process that a query follows once it is submitted to SQL Server. You may recall that there is more than one component that "interprets" what the code wants and reforms it to a more optimized set of code. So what the developer asks for might not always be what shows up in the engine. Not only that, in certain code paradigms, the developer provides the ability for the user to make multiple selections, and the developer builds a code string from there. They might not be aware that a certain paring of values causes poor performance from the engine. Using Profiler, you can show them exactly what is happening on the server when the user makes those selections.

Sometimes while you are watching a Profiler session, you will want to add or change your selections. You can use a "pause and modify" feature that we experiment with in a moment to change what you are capturing.

Let's open Profiler and examine a simple session between one client and the server. I limit the items we are tracking in this example so that we can focus on how the process works.

The Profiler is located in the SQL Server section of the Windows Start menu, under the Performance Tools item. Just as you saw in Chapter 2, when the tool first opens, you are presented with a blank screen. Click File in the menu bar and then New Trace.

The first panel sets the name and any options, such as sending the trace to a table or file, and whether you want to end the session automatically at a certain time. In this case, I am going to monitor my test server for performance while I run a large script, and I will start and stop it manually. I have selected the template called TSQL_Duration in Figure 5-12.

Figure 5-12.

[View full size image]

Templates, as you may recall, contain predefined event and filter settings. You can create your own templates by setting the events and filters the way you want and saving the session as a template.

I will take the defaults for this session based on the template. I click the Run button and then wait for a moment to activate my script. I do that because I want to clearly see the beginning of the script execution. After I run the script, I return to Profiler and stop the session.

In the example output shown in Figure 5-13, the longest section was 75 milliseconds long, and it involved the part of my code that ran a PIVOT statement. I can now zero in on that statement for optimization.

Figure 5-13.

[View full size image]

You can also use the Profiler tool for troubleshooting. After you have narrowed down the issue using the other methods I have described to something within the T-SQL code, monitor the server while the users attempt to reproduce the error. The event classes you care more about in this case involve the T-SQL and stored procedure code. It is interesting to see how SQL Server 2005 interprets the statements you write in code a little differently than the way you type it in. To see an example of this, take a Profile trace during a complex operation. Check the syntax in the TextData column.

This is another example of all those layers I spoke about in the earlier chapters that receive and parse the T-SQL to find the quickest way to respond to the query.

Another useful feature is the ability to synchronize the trace you are capturing in SQL Server 2005 with a Windows System Monitor log file. To do that, you need to have a few things lined up. You need to make sure that you save your System Monitor activity to a log file, either in binary or text format. On your SQL Profiler session, you want to make sure you include the StartTime and EndTime columns. You also have to save the trace to a file.

The basic process is to start the log activity in Windows System Monitor, start a trace to file in Profiler, and then run the applications against SQL Server. When the activity is complete, you stop the trace collection and the System Monitor logging. Open the trace file in Profiler and import the System Monitor log into the session. The times will correlate automatically, and you will be able to see both screens in Profiler.

This is a great feature, so let's take a look at a simple exercise. I will start the Windows System Monitor logging with the Processor and Memory objects I explained earlier in the chapter. In this case, I will use the basic counters associated with them, but normally you would include all the counters that you suspect will correlate to the database activity. I start the System Monitor logging when I am done assigning the counters to it.

Next, I create a new trace in SQL Profiler, this time using the Standard (default) template, which includes the StartTime and EndTime columns. I run the trace when I finish.

Next, I run the T-SQL statement and let it complete. When it is finished, I stop the trace and save the results to a trace file. I then stop the Windows System Monitor logging.

Finally, I click the File menu item in SQL Profiler and open the trace file I just created. Then, from the File menu again, I select Import Performance Data and locate the log file that I saved from System Monitor.

Now all I have to do is click the middle panel, shown in Figure 5-14, where I see the resources affected the way I am observing in the field. Moving the red bar in the center panel causes the display in the top panel jump to the statement that correlates with the time index on that counter. This way I know exactly what was happening when the statement ran.

Figure 5-14.

[View full size image]

You may have noticed if you followed along with the example that you can save the trace out to other formats, such as XML and even XML for Playback. The playback feature allows you to record activity on one server or database and "play it back" to another. To do that, you just save the activity as XML for Replay and then open that same file on the other server's Profiler, using the XML file type. Profiler will read the content of the file and determine that it is for playback and give you an item in the menu bar called Replay. It has several options, including breakpoints and run to cursor. This is useful to capture testing scripts.

There are security considerations with SQL Profiler because it shows both the queries and the results of the query. Previously, you needed administrative rights to use this tool, but in SQL Server 2005, you can grant the right to use Profiler to others. The easiest way to do that is to use Management Studio to locate the principal (role or login) that you want to allow for monitoring. Move to the Securables page and add the server type. Look for the trace items and grant the principal rights to those. Make sure the principals have the right to connect to the server and databases they are interested in monitoring, too.

SQL Server Profiler can also be used to monitor SQL Server Integration Services (SSIS) and SQL Server Analysis Services. I explain more about that feature in the chapters that deal with those subjects.

SQL Trace

Using Profiler is probably one of the most comprehensive ways to monitor your server. As you have seen, you can collect almost any information about the system from the SQL Server engine activities all the way to queries and results. The next step is to automate the Profiler so that you do not even have to open it to get the trace data. There are automation methods for Profiler, but Microsoft has also included another tool to get the same results, called SQL Trace. SQL Trace is a series of system functions you or your developers can call to start and stop Profiler traces within T-SQL code.

Several stored procedures deal with creating, managing, and using SQL Trace, but you only need four to complete a SQL Trace session. The simple outline is as follows:

  1. Create a new trace using sp_trace_create.

  2. Add events by using sp_trace_setevent.

  3. Add any filters to limit the results with sp_trace_setfilter.

  4. Start, stop, and then close the trace using sp_trace_setstatus.

Although that is a simple outline, there are a few parameters to the various stored procedures you will need to complete it. Let's examine each of the parameters for each of the stored procedures.

Use the sp_trace_create stored procedure to create the trace. The first parameter for this stored procedure is an ID number for the trace, which is set by SQL Server, so I normally just create a variable for it.

You need to capture the return value from the function so that you can use it in the rest of the process. Declare a variable and then store the result of the function there, like this:

 This one holds SQL's ID
DECLARE @TraceID INT
 This one holds this trace's ID
DECLARE @trcID INT
EXEC @trcID = sp_trace_create @TraceID OUTPUT
, 4
, 'c:\temp\test.trc'

The next parameter sets any options you want for the trace session. There are three that you can set.

Value | Meaning
2 | Sets the trace rollover. Requires another parameter called max_file_size. When the value in max_file_size is reached, a new trace file starts. They are numbered sequentially based on the name you choose.
4 | Stops the session if SQL Trace cannot write to the output file.
8 | Sends the last 5MB of the trace session to the SQL Server 2005 "black box," which is a kind of dump for everything the engine is doing. If you set this option, you cannot set the others.

If you want to use both the trace rollover and stop options at the same time, use the number 6 here.

The next parameter is the filename, surrounded by single quotes, like this:

'c:\temp\test.trc'

You can set the file to use a local drive or a network location. Using SQL Trace, you will always send the output to a file. I show you how to read data from the trace file in a moment.

The next parameter on the sp_trace_create stored procedure is the maximum file size for the trace (max_file_size), in megabytes. If you used the trace rollover option earlier, you create a new file when the trace session reaches this size. If you did not specify that option, the session will end when the file reaches this size.

The next parameter sets the time the trace will stop. If you do not specify a date and time, the trace will run until you stop it in code. Set this value by specifying the date and time in single quotes.

The final parameter deals with rollover files again. This parameter sets how many rollover files you will have. If you set the rollover option, the maximum file size option and this option, you could for instance create seven separate trace files of 3MB each.

The sp_trace_setevent stored procedure defines the events you want to capture, just like we did earlier using SQL Server Profiler. In this stored procedure, there are four parameters. The first sets the trace ID from the sp_trace_create procedure (which in this example is stored in the @trcID variable), the second sets a number that represents an event, the third sets the columns you want to see about that event, and the fourth turns that event on or off in the trace session. By juggling these parameters, you or your developers can create adaptive trace sessions that branch out based on what is happening in the code. Developers can even turn on tracing options in their applications so that the user could select a trace themselves.

There are over more than 100 events you can capture and more than 60 columns you can collect about them. I do not reproduce them all here, but I do include the ones I show you in the complete example.

Books Online contains all the parameters under the index item sp_trace_setevent. Here are a few of the events you might normally capture.

Number | Event Class | Meaning
13 | SQL:BatchStarting | T-SQL batch has started.
12 | SQL:BatchCompleted | T-SQL batch has completed.
40 | SQL:StmtStarting | T-SQL statement has started.
41 | SQL:StmtCompleted | T-SQL statement has completed.
42 | SP:Starting | Stored procedure has started.
43 | SP:Completed | Stored procedure has completed.

And here are the columns you might want to see in relation to those event classes:

Number | Column | Meaning
13 | Duration | Time in milliseconds for the event.
11 | LoginName | SQL Server login name of the client.
1 | TextData | Text of the command.

Just as in the Profiler tool, the event classes collect data, and filters limit the collection. The sp_trace_setfilter stored procedure has several parameters associated with it. The first is the trace ID from the sp_trace_create procedure. The second sets the column ID to apply the filter on, and the third parameter sets the logical operator you want to use. Here is the list of those.

Number | Operator
0 | Equals
1 | Not equal
2 | Greater than
3 | Less than
4 | Greater than or equal to
5 | Less than or equal to
6 | Like (allows for wildcards such as %)
7 | Not like (allows for wildcards such as %)

The last parameter is the value you want to use the logical operator on. For instance, to trace only in the AdventureWorks database, use the column ID of 35, which is the database name, the logical operator of 0 for equals, and the value 'AdventureWorks'.

So far, I have explained how to create a trace and add events and filters to it. At this point, the SQL Trace session is in a stopped status. In fact, you can only make changes when the trace is stopped. Now that the trace is defined, you can control it with the sp_trace_setstatus stored procedure. This one takes only two parameters: the trace ID and the state you want that ID to have. The three options for the status are stop (0), start (1), and close (2). The difference between stop and close is that close removes the ID from the memory pool, so you will not have access to it again. You do that when you are all finished so that you release those resources back to the server for other work.

To read the trace file in code, use the fn_trace_gettable function. It takes two arguments: the name of the trace file on the file system and the number of rollover files you want to read in. If you did not use the rollover feature, just specify default as the last value. Here is an example that reads the trace contents of a test file called c:\temp\test.trc:

SELECT *
FROM
fn_trace_gettable('c:\temp\test.trc', default);
GO

If you want to store that permanently in a database table, you can use the SELECT INTO syntax.


System Optimizations

Optimizing a system is one of the most difficult, time-consuming activities you are asked to do, but it is one that if done properly creates the most value in your job. After all, time is money. Because I have separated monitoring from the optimizations, this section defines what you can do after you have discovered the bottleneck.

You will find that optimizing a system is often an exercise in moving bottlenecks around. Just like in a crowded city, when you widen a street the next intersection becomes a problem. Fixing that intersection moves the bottleneck two streets over. That is why monitoring and optimizing form a continuous cycle. Do not be surprised to fix one performance issue only to discover that you have hit a different slowdown. Follow the monitoring and optimization process several times, and at some point you will find that you are no longer affecting performance by the changes you are making. At that point, your system is in balance, and beyond a redesign there is not a great deal you can do to make it any faster.

All of this changes when a new variable such as new functionality or new hardware is added to the equation. Then you start the whole process over again.

Platform

Although you will get the most impact from optimizing the application, optimizing the platform is one of the least expensive and most basic things you can do. The optimizations involve the same four parts that are involved in monitoring your system: processor, memory, storage and the network interface.

Start with the basics. Monitor the system parts and ensure that it has enough headroom to satisfy the queries as you monitor. Before you add parts to the system, however, ensure that the application is as optimized as possible. It may be that you do not need more processors or memory, just a better-designed query.

That is not to say that a little extra hardware is not needed. If you followed the guidelines from Chapters 1 and 2 when you designed the system, you should have what you need, but things always change over time. Monitoring and being familiar with new technologies on the system is required to ensure that you have good performance over the life of the system.

It is not only the hardware that needs to be configured. Ensure that the operating system and SQL Server 2005 settings are tuned in accordance with your needs. This involves the SAC tool mentioned earlier as well as the SQL Server Configuration Manager.

Indexes

More than almost any other change in a database system, Indexes provide the greatest tuning benefit. There is a balancing act involved with indexes, because you pay for them on a write, but they benefit you greatly during a read. The first set of decisions involves what to index. You need to have indexes that benefit the most reads but not too many that will adversely impact writes.

The next decision is what kind of index to use. As I mentioned earlier in the book, you have two types of indexes you can use. The first is the clustered index, which physically orders the table. Because this index places the table data in a certain location on the disk, you can only have one of these. The second type of index is the nonclustered index. This type of index is another object that is similar in structure to the table, containing ranges of data and pointers to their locations.

The final decision is when and how to maintain the indexes. If you try to perform your re-indexing operations in the middle of a production cycle, you can expect the system to slow down. If you followed the monitoring guidelines from earlier in the chapter, you will know when your peak activity hours are.

If yours is a large shop with complex systems, there may be no downtime available for index and other maintenance. In Chapter 6, "High Availability," I explain how you can balance your maintenance among systems so that you need little or no application downtime.

Choosing an Index Strategy

Choosing what to index is often left up to the developers. That makes a lot of sense because they are the most familiar with the type of data in the database, and the data access patterns. In complex systems, however, the access patterns are anybody's guess, because they can be used in so many ways. In that case, it is best for the developers to work with the DBAs to find out the patterns.

The focus on the data access patterns is important because indexes slow down inserts but speed up reads. By monitoring what the users are doing, you can find out where to place an index. In general, you should place an index on each of the most-often used columns in a query. Create the index with the most restrictive column declared first. This makes the selectivity on the index high, which means that it points to the data the user is asking for quickly.

If a column or set of columns is highly sequential and does not have new values inserted earlier in the sequence, you may want to consider a clustered index. Clustered indexes are faster than nonclustered indexes, but you can only have one. A clustered index becomes expensive if the values have to shift around to accommodate inserts at an earlier point.

Another consideration is to create a "covering" index. This is not a different kind of index, it just describes an index that contains every column in a query. If a user requests data from a "covered" query, the data is read directly from the index without a trip to the table at all.

There are often times when you do not need an index at all. If a table is less than a few thousand rows and you are sure that it will not grow larger than that, you may not need an index. The Query Optimizer will look at a table that size and just scan it anyway. If the queries against a table are not highly selective, such as in the case where the selection is on Male or Female and the values are evenly distributed, an index is not useful there either. Check to ensure that there is not an index already covering the columns you are about to index. Having double indexes on a column is a waste of space and maintenance time.

Creating indexes is an intricate task, and unless you are familiar with the data access patterns, it is easy to do incorrectly. In a few moments, I show you a useful tool that will help you and your developers select and create good indexes.

Full-Text Indexes

Full-text indexes enable you to quickly locate text within char, varchar, and nvarchar data types, by using a special engine. It is useful for searching for phrases and other patterns of words. You can even use it to search binary data such as Microsoft Word or Excel documents that you have stored in the database using a varbinary(max) or image data type. You can use several language tools to search the data, such as FREETEXT, FREETEXTTABLE, CONTAINS, and CONTAINSTABLE. You can use full-text indexes for queries against linked servers, against various sets of columns, and XML data. You can also specify the language to be used for word breaking in an index, which is how SQL Server 2005 knows that a word is broken out within the data.

A full-text index is not controlled the same way other indexes are. Full-text indexes are controlled by the Microsoft Full-Text Engine for SQL Server Service. Full-text indexes are stored in a full-text catalog, one per database. You need to store the full-text catalog on a drive directly connected to SQL Server 2005.

Some restrictions apply as to where you can create a full-text index. To create one, you need a single, unique, not-null column in that table. This column is what SQL Server 2005 uses to locate the text you are looking for.

Because the indexes are managed separately, you need to run a process to "populate" them, or fill them with data. A full population completely rebuilds the index, and depending on how large the data is this can take a long time. When a full population is complete, you can use a change tracking-based population, which is an update operation on the index and completes much faster. If your table has a timestamp column, you also have the option of an incremental timestamp-based population, which updates the index based on the timestamp column.

To create a full-text index, you can use a graphical method or with T-SQL commands. Creating a full-text index using SQL Management Studio is the simplest method because the entire process is wizard driven.

When your table meets the requirements for a full-text index, right-click it and select Full-Text Index and then Define Full-Text Index from the menu that appears. Select the unique column and the columns you want to index, and the wizard will lead you through creating a full-text catalog. As the final step, you will be able to set up a population schedule that will run automatically. When the wizard completes, you will have a full-text index, fully populated with data.

To create the full-text index in code, you first need a full-text catalog, which you create with the CREATE FULLTEXT CATALOG command. You can use several options, but you do not need to specify any to create one. I normally set the path for the catalog and declare the catalog to be the default, if applicable. Here is a script that creates a full-text catalog called DefaultFullTextCatalog on the C: drive in the Catalogs directory. It also sets this catalog as the default for the database:

USE DBAMDT;
GO
CREATE FULLTEXT CATALOG DefaultFullTextCatalog
IN PATH 'c:\Catalogs\'
AS DEFAULT;
GO

The new full-text catalog is now available for new full-text indexes. This next script creates a full-text index on a table called dbo.Documents for the Description column. The unique key for this table is called DocID. The end of the script contains a statement to automatically keep up with the data changes when new text is placed in the column:

CREATE FULLTEXT INDEX ON
dbo.Documents(Description)
KEY INDEX DocID;
WITH CHANGE_TRACKING AUTO
GO

The WITH CHANGE_TRACKING option has three possible choices: MANUAL, AUTO, and OFF. The MANUAL setting means that you will have to use the SQL Server Agent to set up jobs to schedule the population, AUTO does it for you when data is entered, and OFF means that the server will not track changes in the index.

If your developers are using WRITETEXT and UPDATETEXT statements, the changes are not detected with change tracking.

To populate the full-text index in code, use the ALTER FULLTEXT INDEX statement. The following script updates the full-text index the previous scripts created:

ALTER FULLTEXT INDEX ON dbo.Documents
START UPDATE POPULATION
GO

The UPDATE population repopulates only the changes because the last full population. In addition to the UPDATE type of population, you can also use FULL, which completely rebuilds the full-text index, or INCREMENTAL, which requires a timestamp column.

You should time the full-text index operations using either the Windows System Monitor with the full-text index counters in the SQL Server objects or by using SQL Profiler. When you know how long these operations take, you can create or alter your population schedules.

Data Structure

Although not as obvious as the other optimizations, you can greatly affect the speed the server responds to queries by setting up data structures properly. Although this book does not specifically address database design, the basic premise is that the database should reflect the data access.

It matters how much data there is. The more data there is, the more page-split operations there are, where SQL Server has to move to the next storage unit (page) to return all the data requested. You can do little about the amount of data in the database, but you can make sure that you keep the database maintenance up-to-date so that it is as contiguous as possible.

It also matters where you place the data. If two tables are accessed at the same time quite often, it is best to separate them onto their own physical devices. Separating data files, log files, indexes, and the tempdb database onto their own devices helps a great deal, if you are outside a storage area network (SAN) environment.

Another place to look is how normalized the database is. Normalization describes how segmented the data is into separate tables. To oversimplify a bit, if a database is fully normalized, each individual data set is in its own table. As long as you are selecting or inserting data in only a few tables at a time, the operations are fast. The more joins you have to make to insert or select the data, the greater the odds that the operation will slow down. That is not to say that normalized databases are slow; it is just that normalization does have performance implications. There are times, such as in reporting systems, that it makes sense to denormalize the data to make its retrieval faster.

Most of the actions you need to take are difficult to untangle and find. Data dependencies and program interactions make it hard to even know where to start. Even developers might find it hard to make changes to the structure after the database is in use.

SQL Server 2005 has a tool that will help with data changes, called the Database Tuning Advisor (DTA). The operation is similar to SQL Profiler; you run the tool while users or testers access the database. After a representative period of activity, you stop the tool and examine the recommendations.

Just like SQL Profiler, it is vital that you operate the applications as much as you can during the DTA evaluation period. The more activity you have, the longer you have it, and the more parts of the application the users exercise, the better the results.

The DTA also has multiple reporting options to show you how the database performed, what changes are suggested, and more. You are also able to save the recommendations as a T-SQL script, so that you can review the changes, alter them, or execute the ones you want. This is the method I use most often.

You can get to the DTA from several places, including from the Windows Start menu in the SQL Server 2005 item, from the SQL Profiler after a trace run, and in the Management Studio.

The process is similar to using the SQL Profiler: Open the tool and select the options, monitor database activity, and stop the tool. The difference is that SQL Profiler shows you the results of the activity, with as much or as little detail as possible. The DTA does not display all the activity unless you ask it to, and it recommends and can implement changes to your database. You also get an estimate of improved performance.

Before you use this tool, there are a few caveats with its use. As I stated earlier, the more representative the activity, the better the recommendations are. In fact, if you monitor only one or two statements out of a whole, you might do more harm than good. Second, you should perform a full change review with as many people as you can. It is common to see changes take place on a large system only to have some hidden workaround break the entire process. Finally, take a backup before you start. As I stated in the earlier chapters, I do not recommend going into any situation without an exit strategy. Ask yourself, "What am I going to do if this does not work?" If the answer is "I am not sure," don't do it.

With that in mind, let's walk through an example of using the tool to watch some activity on a test server to see its capabilities. To begin, you need either a trace file from SQL Profiler or a T-SQL script in text. If you are going to tune a large system, you will want the trace file; and if your developers want to see how a group of statements might act in a test, you should use the script file. Microsoft calls both of these a workload. On my test system, I will use the SQL Profiler trace I created with that tool earlier.

When you open the tool, you are asked to connect to a server. I recommend that you do all tuning on a test system, because an improvement on one will normally represent an improvement on another, unless the architecture (such as 64 bit versus 32 bit) is completely different. When you connect, you are presented with the General panel, as shown in Figure 5-15.

Figure 5-15.

[View full size image]

Here you are asked to name the file and select the location of the workload file, or if you sent the trace to a table, you can specify the table and database name where that is stored. You are also able to set the database and tables that you want to tune. The reason you might want to tune a particular table or set of tables, and you do not want the tool to recommend that you remove or create indexes and other objects that the workload does not affect.

The next tab, shown in Figure 5-16, shows the Tuning Options panel.

Figure 5-16.

[View full size image]

You can limit the time the engine will run to tune the system; but the longer you allow it to run, the better. In this case, I have not limited the engine at all. You can also select the options that the engine will use to decide what recommendations to make. The first set of these options asks what kind of indexes the optimizer should consider in tuning. The second set of options deals with the physical portioning. In some cases, you will have the ability to change the partitions; in others, you will not. If you know that you are not able to change the partitions on the system, select No partitioning.

In the last set of options, you can tell the system how much free reign it has with the physical layout. Allowing the system to suggest any changes with the Do not keep any existing PDS is a powerful, and in some cases dangerous, option to choose. Setting this option in effect allows the engine to second-guess the database designers. I usually set this option to see what choices the system makes, but I rarely implement them. There are just too many variables that may not have been captured in the workload to allow the system to have carte blanche on the physical design.

After you have set all the options the way you want, click the Start Analysis button in the icon bar. As the workload processes, you are placed in a panel showing the progress of the activity. When it completes, you are shown the list of recommendations for the various options you selected.

As shown in Figure 5-17, you can select or de-select any of the options that you want to show up in the final change scripts. When you have made your selections, click Actions and then Save Recommendations… in the menu bar. The tool creates a script file with all of the statements needed to implement the changes, in the proper order. You should never select the Apply Recommendations… item, even if you are sure that is what you want to do. Creating a script file allows for a controlled change and documentation for your exit strategy.

Figure 5-17.

[View full size image]

On the Reports tab shown in Figure 5-18, you can view several tuning reports. If you right-click inside any of them, you can copy the data to the clipboard or send them to a file. Saving these reports electronically allows you to collaborate with your developers on any proposed changes.

Figure 5-18.

[View full size image]

In the Management Studio query window, you can select Analyze Query in Database Engine Tuning Advisor from the Query item in the menu bar. The process to use this method is the same as you have just seen, with the exception that you do not have to provide a workload file. The query from the Management Studio is already loaded, ready to execute.

Applications

If you optimize a platform to a perfectly balanced state, you may be able to gain up to 15 percent speed increase on the same hardware. Applying indexes properly can gain you much more. The biggest factor in performance, however, is within the applications written against the server. The DBA does not often have control over the queries used by the application, but you can help your developers gain an awareness of how they can use SQL Server 2005 in the most effective way.

Most applications benefit from a "selection review." What this means is that each query in the application should affect no more data than is absolutely necessary. In the case of data selection, the query should return only the data needed to satisfy the request. In the case of an insert or deletion, the query should only touch the tables it has to in order to make the change.

That sounds like common sense, but in a surprising number of cases the developer takes a shortcut by returning a huge amount of data to the client to be processed there. In a single instance, this is not even noticeable; multiplied by thousands of requests, however, this practice can adversely affect the performance of the system dramatically.

Another application change your developers can make is by using transactions properly. By pooling connections and other aggregation methods, the programs are able to make fewer trips to the database. It is all about taking as light a touch on the engine for as little time as possible from the applications.

Although not directly a programming issue, it is important to track the applications that use the database. Often, the application has been properly tuned, the servers, workstations, and network are configured correctly, but the application still performs poorly. On these occasions you should check to make sure only the authorized applications are using the database. It is tempting to hook other applications, especially for reporting, into the same database that has been tuned for a particular application. If you have secured the database, this should not happen, but performing a security audit should turn up the guilty applications. Remove those and you have got a fast, easy, and inexpensive performance upgrade.

Refactoring Queries

Refactoring a query means to try all the possible statements and methods of accessing the same data. Developers evaluate all these to ensure first of all that the data is consistent with what they want and that the query runs faster. After using the various queries, the developer eventually settles on a query that has the best performance.

Using SQL Server Management Studio, you can see the statistics for a query as I explained earlier in the chapter. I also explained how to show an execution plan, and this function is far more useful to refactor the queries. There are two modes to show the plan, graphically and in text. In this example, I have opened Management Studio and created a new query window. I typed in a rather lengthy query and then selected Query and then Include Actual Execution Plan from the menu bar. After that, I ran the query, just as I explained in the earlier example. After the query ran, I opened the Execution Plan tab. In Figure 5-19, I right-clicked within that panel and then selected Properties from the menu that appeared.

Figure 5-19.

[View full size image]

When you run a query this way, you can hover over any icon in the graphical panel to show lots of information about that step, but in some cases the output obscures the entire window. It is often more convenient to open the properties, as I have here, and then click each icon to show what is happening on the server.

To analyze the output, you (or the developers) start on the far-right side of the graphical panel. You are looking for the highest-"cost" item within the panel. You are also looking for the proper use of indexes. Unless there are fewer than a thousand rows or so, the query should use an index. There are many elements within the plan, and none of them are bad or good by themselves. In general, bookmark lookups are good, table scans are bad, and an index seek operation is better than an index scan operation. But that is just in general and depends a great deal on the query.

You can save an execution plan out to a file, and read it into another query window, even on a different system. Right-click inside the Execution Plan graphical window and select Save Execution Plan as from the menu that appears. You can save it as a SQL Server execution plan with an extension of .sqlplan, or you can just save it with an XML extension, because that is the format for the file. You can read it in to an XML editor or into another copy of Management Studio with the File and then Open menu items. You can use this feature to mail the plan around for discussion and review.

You can also show the same information in text. At the top of a query, insert the following lines:

SET SHOWPLAN_ALL ON
GO

Now run the query again. You will get a text output similar to Figure 5-20 that some find a little easier to read than the graphical icons.

Figure 5-20.

[View full size image]

Although you might not use the execution plans yourself, you need to be aware of security implications. Make sure that you grant this right only to developers who need it.

To grant a developer the right to view the query plan, use the following command:

GRANT SHOWPLAN TO username
GO

Leveraging the Platform

As a final application optimization step, developers should always leverage the SQL Server 2005 platform. This means studying the features, structure, and limitations of the platform, so that the programs use the latest, most-efficient methods to access the data.

Companion books to this one will help the developers take advantage of the latest coding techniques and the newer features and syntax that SQL Server 2005 offers. Although it needs to be handled with tact, always question whether the code is using everything it can to perform at its best.


Take Away

As you have seen throughout this chapter, you can collect a lot of metadata information from SQL Server. You can use any of the system views, functions, or tables I have explained in this chapter to create the metrics you are looking for. Often, the type of monitoring you are interested in will dictate the collection method you need to use. For instance, if you need immediate feedback on a query, you can use SQL Server Profiler. If you need to collect information on an ad-hoc basis in code, you can use SQL Trace. For long-term monitoring for performance or growth, you can use the sys.dm_os_performance_counters table or dynamic views, or even the Windows System Monitor. Do not limit yourself to one tool, because each has advantages depending on what you need to collect.

It is important to combine the monitoring with an understanding of the application and the performance problems before you try to interpret the results. Do not get caught making an absolute statement about an interpretation you have made only to be shown another variable that contradicts the statement.

After you collect the data, you need to be able to analyze the results. There are a lot of ways to show the data such as reports and Web presentations, but if you are preparing the data for business users, you cannot go wrong by putting the results in a spreadsheet. That is the approach I use in the exercise that follows.

A Monitoring Exercise

In this section, I show you a pared-down example of a monitoring exercise I perform on servers that are showing performance issues. In practical uses, I include far more information. In this example, I use the System Monitor counters I explained earlier.

Object | Counter
Processor | % Processor Time
Memory | Available Mbytes
LogicalDisk | % Disk Time
Network Interface | Bytes Total/sec
SQLServer:Databases | Active Transactions
SQLServer:General Statistics | User Connections

I normally use the logging feature in System Monitor and send the output to a tab-delimited file. I set the collection interval to five seconds if I am monitoring for a single day, and five minutes if I am monitoring longer.

Next, I create an Excel workbook with three tabs: Observations, Interpretations, and Recommendations. I read the System Monitor results into the tab called Observations. I change the format on the columns to remove the server name. I also color the blocks differently for Processor, Memory, Disk, Network, and SQL Server counters.

On the Interpretations tab, I create formulas that reference the rows in the Observations tab for maximum, minimum, average, and standard deviation values. I evaluate each of the numbers in the columns and color them red for values I consider outside the norm, green for values that are good, and blue for those that are indicators, such as active transactions. I include the minimums, maximums, and standard deviations so that the averages are useful. Below the numbers, I explain why I feel the numbers are good or bad and the other factors that affect the numbers. I also explain a little about the testing environment, and sometimes I even include server information. It is important to understand the interactions between the numbers. I have only included a few counters here, but in practice I use far more to diagnose an issue.

On the Recommendations tab, I explain what to do to fix the issues. For instance, if the active connections move up at a certain rate and at the same time the memory numbers fall below what is physically installed on the server, the server is paging and the server needs more RAM.

If I am monitoring more than one server, I include each of them in blocks below each other in the tabs I mentioned. This is especially important if the servers interact within the system.

It is surprising to find that you can tell management over and over that a particular part is a problem and you will be ignored. Show the same results on a spreadsheet and your suggestions will be adopted.


Chapter 6. High Availability

The concept of high availability is quite simple: Systems must stay up as much as possible (in some cases, constantly). This requirement is not only aimed at unexpected events such as system failures but also for planned downtimes such as maintenance or upgrades.

Microsoft provides a wide array of features within its operating systems and in SQL Server 2005 that can help you accomplish that goal. The mechanism you choose depends on what level of availability you are after and which parts of the system you want available.

For duplication of data across several servers, you can copy data automatically from one place to another using replication. Several types of replication bring data from one server to many or to both of them bidirectionally, pulling the data from its source or pushing it to a new location. To send an entire database to another server on a short time interval, you can use log-shipping techniques to copy and apply the Transaction Log from one database to another, keeping both in sync. For immediate time requirements, you can copy the database automatically to another server using database-mirroring technology. You can even duplicate an entire server setup, something Microsoft refers to as "clustering." With a cluster, the system uses multiple computers connected to the same database to maintain constant uptime.

In this chapter, I cover what you need to know about the options you have for implementing a highly available system, and I explain how you manage and monitor it. There is a lot to cover, so I overview the main concepts and show you practical examples along the way.

It is often difficult to find a highly available system to practice on because the production systems are sensitive to change and the components that make up the system are expensive. In the "Take Away" section at the end of this chapter, I show you how to set up an entire cluster on a single system. You can use the same technology to practice or test the other high-availability mechanisms, too.

High availability is keeping your systems operational as long as possible. The requirements may even state that the system needs to be available all the time. There are many reasons to implement a highly available system, but the primary one is protecting your systems against a hardware failure. The effort to implement a highly available system often begins after a crisis at the organization or after a crisis at another firm makes the news. The question is asked, "What can we do to prevent this?"

Another reason for high availability is that the system might be so large that it would take a long period of "downtime" to implement maintenance, which the organization cannot afford. The organization might also want to keep the system available during changes in an application or when performing upgrades.

Although all these goals for high availability are easy to state, they are often full of vague terms such as systems, operational, and all the time. It is important to ensure that the organization understands the costs and benefits for a high-availability system. There are several ways to reach the goals you have for high availability after you define the terms.

The place to start is to define what the organization desires to have as "uptime." When asked that question, most often the response is that the organization wants 100 percent uptime; but in fact, this really is not often a necessity. Most organizations do, in fact, have a time when a particular system can be unavailable. If that is not true, you probably already have a highly available system in place.

What the organization wants most of the time is the guarantee that the systems will not go down during regular operational hours, which might be another thing entirely. At some firms, this might be 24 hours a day, 7 days a week; but many firms have a window of time where there are no operations, no product shipped, or no employees at work. At the least, you might be able to define which systems have these kinds of windows and how long they are.

To find out the true uptime needs, detail the functions of your systems and examine the uptime requirements separately. Shipping might have a longer downtime than a product line, or salespeople might be off during holidays. You can design the uptime for each function within the system and apply the appropriate high-availability solution accordingly.

After you have defined exactly how much time the organization wants to make sure their systems are available, you can begin to plan out your strategy to meet that goal. With that goal in mind, you begin to look for current downtime windows to meet it. It might just be a matter of moving your maintenance processes into those windows.

Let's assume for a moment that there is no activity on the system from 1 a.m. until 6 a.m. That is a pretty big window, and you might be able to get all of your maintenance and upgrades done in that time period. If you followed the guidelines in Chapter 1, "Installation and Configuration," for buying a server-class machine with hot-swappable parts and a properly configured RAID storage unit, you may have all the redundancy you need. Perhaps you do not have a five-hour window conveniently available in the middle of the night. You might have five hours throughout the day, in various slices. You can design your maintenance and upgrade time to fit within those windows.

DBA 101: Emergency Recovery Time

To find out your emergency recovery time, test your backups periodically and hardware replacements, time the process, and communicate that information to the organization. The organization owners can create a plan for that downtime when the crisis occurs.

On the other hand, your system might have activity on it at all hours of the day and night, especially if it is used from multiple geographic locations simultaneously. In this case, you do not have enough time to perform maintenance. In high-use situations, when the system stops, it means a stop in production. In this case, you have a real need to protect not only the data your system holds but also the functionality of the system. If your system has these constraints, you need to consider how you will implement some form of high availability.

You have various options for high-availability technologies, each a bit more complex than the other and costing more in time and money to implement. A highly available system also requires more knowledge to operate, more planning, and more attention. You should understand what each of the technologies do for you as well as what is involved in setting one up and maintaining it to explain to the organization what the costs and benefits of the system are.

SQL Server 2005, paired along with the Windows operating system, provides several mechanisms for making the system highly available. You can copy data from one location to another automatically using a process called replication or log shipping. These methods are useful when the data is allowed to have a certain level of latencythat is, when the data can be a few minutes or hours old without severely impacting the application. Neither of these is connection agnostic. The applications must be able to detect that the original system is unavailable and switch to a different server.

In database mirroring, the entire database is duplicated on another system. Once again, the applications must be able to detect the switch if the original server name is unavailable, although if the application is compiled using ADO .NET 2.0, it has the capability to automatically switch when the primary server goes down.

The next level of availability, called clustering, provides more protection. In addition to protecting the data in real time, two or more servers protect a single identity. Applications connect to the shared identity (called a virtual server), so that if one server is unavailable, another continues to provide the shared identity to the network. Clustering requires some shared components, as I point out later, but provides a maximum degree of safety for the data. Not only that, you can take one system (called a node) in the cluster offline to perform maintenance or apply service packs. When that is complete, you can bring the node back online and repeat the process with the other node. Depending on how robust the code is, the users never have to leave the application while this happens.

These implementations are not without cost, however. At the least, you need another system, and that means more licenses for operating systems, SQL Server 2005, and backups and other utility software. In the case of clustering, you also have to adhere to strict hardware requirements.

Let's examine how you can install and manage the various levels of high availability.


Near-Real-Time Availability Mechanisms

The first set of mechanisms you have at your disposal involve creating an infrastructure that provides a near-real-time availability of the data. These processes move or copy data from one system to another within a disconnected architecture. The interval you choose for data latency might be almost immediately or span hours, days, or even longer.

There are three ways to reach the objective of providing a copy of the data to an application while the server is unavailable. The first is to use various coding techniques to either stage the data the application needs on another server or to request it from a federated set of resources. I do not cover those methods here because they involve programming techniques and architectures. The way you configure and manage SQL Server 2005 while using this mechanism does not change at all from what you have learned in this book.

The second mechanism is replication. Replication involves a series of stored procedures and database extensions in the server that "watch" database changes as they occur and store those changes in another location. Depending on the type of replication you implement, the changes are either immediately propagated to another system or picked up at certain intervals. Replication works best when subsets of data are needed by other systems, such as a copy of inventory data sent to a salesperson's local database on their laptop, or when one system depends on data from another.

The third mechanism is log shipping. In log shipping, the Transaction Log backups are copied to another server and restored to a copy of the original database there. You can even use log shipping to copy the changes to more than one server.

Replication

We begin the discussion of replication with a look at how it works from a high level. As the database administrator (DBA), you will often be asked to design, implement, and monitor the process, so you need to understand all of the components it uses, their relationships to each other, and how to maintain them.

Replication in SQL Server 2005 is described in terms of the publishing industry. In this metaphor, a publication such as magazine or newspaper is produced by a publishing company. Although a lot of paperwork is generated at a publishing company, only a certain portion of it is meant for publication. Publications such as newspapers are made up of articles. Some larger publishing houses do not keep printing presses and distribution trucks at the same location where the articles are written. They have a remote distribution center to send out all of the newspapers or magazines because that building can be located closer to rail and ports or to major highways. At the end of a newspaper subscription is the subscriberthe person who reads the publication.

In SQL Server 2005, the same kind of verbiage is used to describe the replication schema. A server's database contains articles, such as views or tables, which make up a publication. Although an article is the lowest level of work within the SQL Server 2005 replication architecture, just like in the publishing house example, the published data is the publication, not the article.

The distributor is the system that stores the publications and sends them on their way. The distributor can be located on the same server as the publisher or another to lighten the load.

The subscriber is the system or systems that receive the data. SQL Server 2005 can publish to all kinds of subscribers, even Oracle or Microsoft Access. Subscribers can pick up the data from the distributor, called a pull subscription, or the publication can be sent to them automatically, called a push subscription.

The metaphor carries even further. Controlling all these movements is a series of software agents, which are responsible to get the data from one place to another. You can monitor these agents to see just where the data is at any given moment.

The publishing house metaphor begins to break down a bit when we talk about the types of replication. SQL Server 2005 can send everything in the publication to the subscribers, called snapshot replication, only the changes, called transactional replication, or it can send and receive changes, called merge replication.

With those concepts in mind, let's take a closer look at the design considerations for a replication architecture. We will finish up with a simple replication example.

Types of Replication

You need to start your design by describing which systems need to see what data. After you have defined those parameters, you can create a layout that will meet the subscriber's needs. For instance, your high-availability requirements might allow for the primary server to be offline for maintenance as long as the parts inventory information is still available. In that case, you can create a publication for the parts inventory data to be pushed out to a shop-floor server working as a subscriber.

Within the data needs for the subscriber are the types of data refresh that they need. In the previous example, the entire set of data needs to be current when the primary system is taken down. In that case, all the data can be sent to the subscriber at once.

But perhaps that data is too large to be sent all at once, or another need dictates that the subscriber should be kept up to date with more frequent changes. In that case, you want to create a replication architecture that sends only the changes to the subscriber, possibly more often. You might also need to make data changes at the subscriber, and you want to send those changes back to the original publisher. In that case, you need to merge the data together.

Let's take a look at the various types of replication to see how they solve each of these problems.

Snapshot Replication

The first type of replication is a snapshot, where the entire publication is sent from the distributor to the subscribers. It is the simplest type of replication to create and implement. In fact, all replication types start out with a snapshot because the data needs a starting point to implement any changes.

I've seen cases where the original publication is so large that sending it to a distributor and then on to the subscriber over the network is implausible. Microsoft has published several white papers that show you how to implement the initial snapshot publication using a tape backup or another media.

If you are bringing a small about of data to the subscriber or the data changes a great deal but not often, a snapshot is the way to go. The data in this case is read-only at the subscriber, because the whole data set is updated when the replication completes.

Transactional Replication

In transactional replication, the Transaction Log is monitored for changes, and those changes are sent on to the distributor. The subscribers either pull the changes or the changes are pushed to them. The process begins with an initial snapshot, and then changes are gathered as they occur and are stored on the distributor. The changes are available to the subscriber on the schedule that is set up for the subscription.

By default, the data from a transactional replication publication is read-only, although you can create an updateable subscription that allows you to propagate the changes at the subscriber back to the publisher. The trickiest part of setting up this arrangement is how to deal with a "clash," where the same data element on both systems is changed.

Merge Replication

Merge replication starts with an initial snapshot, and changes at the publisher are sent to the distributor and then on to the subscriber. The subscriber can change the data, and the results are sent back to the publisher.

This brings up the question, "what happens if the same data is changed at both places?" This is called a "conflict," and you can handle it in three ways. The first method states that the publisher always wins. The second states that the subscriber always wins. In the third case, you are asked to resolve each conflict individually through a SQL Server 2005 interface.

Replication Servers

Within replication, you can install each function on the same server or separate servers. I explain the various layouts in a moment, but first let's examine what part each function plays in replication.

Publisher

The publisher is the system that holds the data you want to publish, or the publications. In most cases, this is the same system that holds the production data.

When you configure a system to use replication and become a publisher, substantial changes occur on the server. The Replication Wizard runs several processes that affect the server's operation and new tables and stored procedures are created.

Distributor

The distributor holds all the data for the publications. This is actually a two-step process, because the data is sent here first and then is picked up by or sent to the subscribers. The data stays in place until all the subscriptions for it are fulfilled.

It is important to understand that this is another copy of the data in the publications. That means if you place both the publisher and distributor functions on the same server, the publication data is stored there twice. If you are working with small publications, that might be acceptable, but with larger data sets, you are not only increasing your storage requirements, you are causing more I/O on your publisher.

Depending on how many subscribers hit the system, the distributor also has an increased network load. It is best if the distributor is connected on a fast, dedicated connection to the publisher and then across other connections for the subscribers.

Subscriber

The subscriber is the server that either picks up the publication or has it delivered. Subscribers can be any other edition of SQL Server (even the mobile editions) or ODBC-compliant systems such as Oracle or IBM's DB2.

You should know a great deal about the subscriber, because that will determine the type of replication you can create and how you set up the schedules. Of special importance is the type of connection from the subscriber to the publisher. The slower this link is, the more creative you will have to be in your design.

In a moment, I show you how you incorporate of these design decisions into a replication schema.

Replication Agents

All of the data tagging and movement within the replication system is handled by various stored procedures and jobs, called agents. Each agent has a specific task and is created and operates on specific servers. The agents are controlled by the SQL Server 2005 Agent Service, which runs in the background on the server.

DBA 101: Check the Power Cord

One of the most common errors in a replication schema is that the SQL Server Agent Service is not running. Always look for the most common causes when you run into a replication issue.

With the SQL Server Agent Service running on the instance of SQL Server 2005 that is participating in replication, the following agents are implemented.

Agent Name | Purpose | Server
Distribution | Used with transactional and snapshot replication, this agent moves the initial snapshot to the subscriber and also moves the data from the distributor to the subscriber. | Runs on the distributor for push subscriptions and on the subscriber for pull subscriptions.
Log Reader | In transactional replication, the Log Reader agent looks for entries marked for replication in the transaction log on the publisher database. | Runs on the distributor, reads from the publisher.
Merge | In merge replication, this agent applies the snapshot and then reconciles the incremental data changes from there on out. Although you can configure it differently, the changes are sent from the subscriber first and then from the publisher. | Runs on the distributor for push subscriptions and on the subscriber for pull subscriptions.
Queue Reader | Used to move data from the subscriber back to the publisher when you enable transactional replication with queued updates. | Runs on the distributor.
Snapshot | Used with all replication types, this agent prepares the schema and stores all the files and objects required to move the entire publication from the publisher to the subscriber. | Runs on the distributor.

Several jobs run in the background, too. You should be aware of what they do so that you can diagnose and correct any issues.

Job | Purpose | Frequency
Agent history cleanup: distribution | Empties the replication history tables | 10 minutes
Distribution cleanup: distribution | Removes the replicated transactions from the distribution databases | 10 minutes
Expired subscription cleanup | Removes expired subscriptions from the publisher | Once a day at 1 a.m.
Reinitialize subscriptions having data validation failures | Finds subscriptions with errors and marks them for re-initialization | No default schedule
Replication agents checkup | Ensures that history is written to the operating system event viewer log | 10 minutes
Replication monitoring refresher for distribution | Refreshes cached queries used by the Replication Monitor | Constantly

Replication Architecture Design

How you set up the disparate parts of SQL Server replication is highly dependent on what you want to accomplish, and you do not want to create a schema based from too little information. You should ensure that you fully understand what the organization wants to accomplish by implementing replication before you start your design.

When you understand what the needs are, you need to evaluate your systems to find the best layout. There is a six-step process I use to create an effective replication design:

  1. Set the replication's goals.

  2. Determine and document the environmental factors.

  3. Determine the frequency needed to satisfy the goals.

  4. Document the process for the different types of subscribers.

  5. Create the design.

  6. Develop a fallback strategy.

Let's explore each item in this outline a little further.

Replication Goals

First, you need to define what the goal for the process is. Possible goals include copying parts of the inventory catalog to a salesperson's laptop, moving data between suppliers or plants, or sending reporting data to another server. Enumerating these goals make decisions as to the type of replication and the systems on all sides of the process.

Environmental Factors

The environment of your system also helps you understand which replication types to implement. If you have a slow network connection to work with between the systems, you need to have a smaller and perhaps more frequent distribution layout. This means that on a slower link, you might not be able to do snapshot replication once a week on a 300MB publication.

Frequency

The frequency for your replication schema is dictated by two factors. The first is what your environment can handle, and the second is the organization's needs. The organization's needs are covered in the first step of the outline, and the second step details what your physical infrastructure can handle. In this step, you combine the two and record each publication and the times that they need to go.

Answering the frequency question will also help you decide on the direction of the publication. If the subscriber is connected to the network erratically (such as a salesperson's laptop), you will not have a clear timeframe to push the data. In that case, the publication is pulled from the client on demand. If the system you are publishing to is a reporting system, however, you have control over the schedule and you can push the subscription.

Subscriber Information

The next step is to detail the type of subscriber you are working with. This gives more insight into the type of subscription (push or pull).

It also helps you understand how much disk space you need to leave on the distributor. For instance, if the subscriber is in a remote location with a poor link to the system, connects infrequently, and is unreliable, you need to plan for enough space on the distributor's hard drive to store the publication for an extended period of time.

Let's look at a concrete example. Assume that your subscriber is that poorly connected unreliable system and that you need to send a 3MB publication to them once a day. When you detail out this step, you realize that it is possible that the subscriber could miss up to three days of publications. In the interim, the distributor will need to retain 9MB of space on its drive to service the request when the subscriber connects. You need to work through this exercise for each of your publications to have an effective plan.

Creating the Design

Based on what you find in these three areas, you can begin to create your design to implement the solution. Let's look at a few examples to see how a design effort works. The process is easy to follow: Take the previous steps in the outline and apply it to your situation. Each step in the process is iterative and will further refine your layout.

You may find that you need only a few publications from a single server that can also act as the distributor. Your needs might also dictate multiple publishers, one distributor, and a single subscriber, or any other arrangement of servers. Following the outline helps you design the proper sizes and layouts for your schema.

Develop a Fallback Strategy

As you can imagine, in a complex replication schema, things can go wrong in lots of places. Microsoft SQL Server 2005 contains a plethora of graphical and command-based monitoring mechanisms, which you must stay on top of to correct problems.

Never wait until you have a problem to develop a strategy to deal with it. For each publication, document what you will do in case it fails. You can retry the replication, wait until the next replication cycle, or even break the replication and re-create it, sending an entirely new snapshot. The goals from the first step and the factors from the second will dictate the route you take. Whatever the case, the time to create your strategy is before you implement it. You should also test that fallback before you implement in production to ensure that the strategy is viable.

Implementing Replication

With the terms, theory, and processes in mind, let's work our way through an entire exercise. The requirements from the organization state that we need to provide current inventory levels from the Manufacturing database on the NodeA server in the headquarters cluster to the Tampa, Florida, site in the United States. We have determined that the CurrentInventory table in the CLUSTEREDSQL instance of SQL Server 2005 holds the data that we need and that we need to send it in total to the Tampa server each day.

The data is less than a few thousand rows, and it changes by more than 50 percent each day. That means we need a snapshot replication type. If the data changed less frequently or needed to be more up-to-date, we might choose to implement a transactional replication type.

We've evaluated the environmental factors and determined that we have fast network links between powerful servers. The subscriber system is a duplicate of the publisher system. It is constantly connected to the same network backbone.

Our fallback strategy is to first allow the replication to retry, waiting two cycles for a success factor. Barring that success, we will break and recreate the replication.

Setting Up Replication: A Step-by-Step Example

Using the information from the outline I explained earlier, I will proceed through the physical process of setting up the replication schema.

In this example, I'm replicating to and from the same server, but the same process applies to working with multiple servers. Working with a single server allows you to practice along with this exercise on a test machine.

First, we need to configure the distributor and the publisher. I open SQL Server Management Studio, locate my server, and right-click the Replication object in the Object Explorer. The results are shown in Figure 6-1.

Figure 6-1.

[View full size image]

With that menu item open, I select Configure Distribution. The first panel that comes up is an introduction to the wizard. I click Next on that panel to bring up the screen shown in Figure 6-2.

Figure 6-2.

On this panel, I select the system that acts as the distributor. The outline we followed earlier dictated that this system acts as the machine that creates the distribution databases, share locations, and files that the replication schema needs. When I have made this selection, I click Next to show a panel that has a selection for the SQL Server Agent Start. I leave this one at the default for this test system and then move on to the next panel, shown in Figure 6-3.

Figure 6-3.

Each publication, regardless of type, needs an initial snapshot of the data to get started. This provides the initial data numbering that replication needs to store to track what goes next. It is a directory that is shared to all of the replication agents. In this panel, I have set that to my S: drive, which has plenty of space.

With that selection made, I click Next to continue to the panel shown in Figure 6-4.

Figure 6-4.

This panel sets up the distribution database the distributor needs to track what it is doing. I can also set up the location for the distributor databases and logs here. When that is complete, I select Next to show the screen in Figure 6-5.

Figure 6-5.

This panel shows the servers on the network and allows you to select the publishers that use this system as a distributor. In this example, I am using the same server for all three functionsdistributor, publisher, and subscriberbut in production, you would set the servers based on the layout you created in your outline.

After I choose the server name, I click Next to continue.

In Figure 6-6, I can begin the process of creating the distributor, or just save it out to a file to create the distributor later. Using a script, I can re-create the process on multiple servers with just a little editing. In this case, I create the distributor as I have detailed in the previous screens. I click Next to continue, and the wizard recaps the process for me, giving one more chance to click the Back button. I click Finish, and I am shown the progress screen shown in Figure 6-7.

Figure 6-6.

Figure 6-7.

When that process completes, I have a distributor set up and ready. I now need to create the publisher, articles, and publications that I want to replicate. Once again, I right-click the Replication item in the Object Explorer and select publisher Properties from the menu that appears, which brings up the screen in Figure 6-8.

Figure 6-8.

[View full size image]

First, I need to select the database I want to work with. In this case, I want the Manufacturing database. After I do that, I click the OK button. That process marks the database for the Replication objects I need to continue, as shown in Figure 6-9.

Figure 6-9.

[View full size image]

Next, I need to create the publication for the inventory data. Under the Replication object in the Object Explorer, I right-click the Local Publications item and then select New Publication… from the menu that appears.

The screen shown in Figure 6-10 is the introduction to the wizard. I click the Next button to continue.

Figure 6-10.

[View full size image]

In Figure 6-11, we are shown the database list from the previous step. In my case, I only enabled the Manufacturing database, so that is what is selected. I will select that database and click Next to show Figure 6-12.

Figure 6-11.

Figure 6-12.

[View full size image]

In this screen, I select the type of replication I need. From my outline, I know that I need a snapshot type of replication, which is shown and described on this screen. You can also read about the other types here. After I have made that selection, I click Next to continue.

In Figure 6-13, I have selected the table I need to send to the subscriber. Clicking the Article Properties button allows me to set up the properties of the items I want to replicate. When I have made my selections, I click the Next button to bring up the next screen, as shown in Figure 6-14.

Figure 6-13.

[View full size image]

Figure 6-14.

[View full size image]

This panel lets me limit the data that is stored in the article, which I do not need in this case. I want all the rows, so I just click Next to continue.

In Figure 6-15, I can create the initial snapshot of the data or schedule it to run later. The schedule runs using a Snapshot Agent, which I explain later. In this case, I create the snapshot now and click Next to bring up the next panel.

Figure 6-15.

[View full size image]

Although you see only the result in Figure 6-16, I actually had to click the Security Settings button to select the account that the agent uses to talk between the servers. Remember that this account needs access to both systems, and the ability to work with the data on both sides. Pressing that button allows you to set up a Windows or SQL Server account to do this.

Figure 6-16.

[View full size image]

In my case, I have a Windows account I created with the proper permissions and rights on both sides of the replication schema that I can use for the transfer. With that complete, I click Next to continue.

As shown in Figure 6-17, I have the ability to immediately create the publication or create a script. I will select the immediate choice and click Next to bring up the screen shown in Figure 6-18.

Figure 6-17.

[View full size image]

Figure 6-18.

[View full size image]

At the final screen of the wizard, I name the publication. You can use any name you like, but I try to keep them descriptive, short, and use either underlines or upper- and lowercase. This is not required, but it is better when you work with scripts not to have lengthy names with spaces in them. When I click Finish, the final panel displays, as you can see in Figure 6-19.

Figure 6-19.

[View full size image]

I now have a distributor and a publisher, along with an article and publication. All that is left is to create the subscriber, the subscription, and the schedule.

Once again, I am using the same system for the subscriber as the one I used for both the distributor and publisher. Although this is not the most common way you will replicate, I actually have used this in production.

Once again, we are in the SQL Server Management Studio, in the Object Explorer under the Replication item. This time I right-click the Local Subscriptions item. I select New Subscriptions from the menu that appears.

It is important to remember I am doing this on the subscriber.

I have skipped the introduction screen once again by clicking the Next button. That brings up the screen shown in Figure 6-20.

Figure 6-20.

[View full size image]

In Figure 6-21, I have selected the publishing server, and then the InventoryLevels Publication. Once I have done that, I click Next to continue.

Figure 6-21.

In Figure 6-22, I select whether I want to pull the subscription or have it pushed to me. Because our initial outline stated that we have a good connection and a powerful subscriber, I select Push here. With that selection made, I click the Next button to bring up the screen shown in Figure 6-23.

Figure 6-22.

Figure 6-23.

Here I need to set the server that acts as the subscriber and the database where I want the publication to end up. It is important to remember that I might have published an entire database. In that case, I have to make other choices to allow people to read the data or make updates to it. But in this case, I am only pushing a single table across, so my TampaManufacturing database might have other tables that are both read and write to the local users. That is a common arrangement, and in fact it is the one I have here.

When I complete my selections, I click Next to continue to the screen shown in Figure 6-24.

Figure 6-24.

Once again, I need to select the security I will use, this time for connecting to the subscription. I will use the same account, but this time I will click the ellipses button on the first line and show the security panel you can see in Figure 6-25.

Figure 6-25.

[View full size image]

You have quite a few options here that you can choose for the Distribution Agent. I will use the account I explained earlier and click the OK button. When I do that, I am placed back at the previous graphic, and I click Next there to show the panel shown in Figure 6-26.

Figure 6-26.

Here I am setting the schedule for the replication synchronization. In this example, I am using a manual replication, but I rarely do that in production. Most of the time, I run on a schedule; and if I am using a transactional replication, I select Run continuously to keep two systems in sync. Again, your outline will define your choices.

I then click Next to continue to the screen shown in Figure 6-27.

Figure 6-27.

Here I set when the initialization starts. I have selected this to happen immediately. With that selection made, I select Next to move to the next screen, shown in Figure 6-28.

Figure 6-28.

On the summary screen and the final screens, I continue through the displays and watch the progress of the subscription.

If you do not select to immediately create the subscription, you can right-click it later to initialize it. You must have the subscription initialized before you synchronize it. This is done for me by the wizard because that is what I selected earlier.

I have now completed the entire process to create the distributor, publisher, subscriber, and all the articles and publications, and also the initialization of the process. At any point in the process, we might want to see what the current status is. It is a simple matter of right-clicking the publication and selecting View Synchronization Status. That brings up the panel shown in Figure 6-29.

Figure 6-29.

[View full size image]

When I click this menu item, I am shown the status of the publication, as you can see in Figure 6-30.

Figure 6-30.

[View full size image]

Often, you need more information about what is happening within replication. There are several log locations and other methods to evaluate the process, but a simple method is to either select the Launch Replication Monitor menu item in the graphic shown two steps earlier or click the Monitor button on this panel.

If you click that button or the menu item in the Object Explorer, you are shown a panel similar to Figure 6-31.

Figure 6-31.

[View full size image]

This panel allows you to control many aspects of publication; you can view the details, and you can also start the synchronization process. Because I selected a manual synchronization earlier, I need to kick that off now.

Let's stop for a moment and understand the difference between initializing and synchronizing. Initializing the subscription sets up all the plumbing, and synchronizing replicates the data.

In Figure 6-32, I have pulled up a query on both the source and destination databases after I completed all these steps. There is only one row, but they are identical.

Figure 6-32.

[View full size image]

Replication is one of those concepts like the game of chessfairly simple to implement but can become quite complex in the implementation. Just as in a chess game, the components are to understand what all the pieces do and then evaluate what your goals are. From there you develop a strategy.

If you implemented this example along with me on your system, try breaking it and finding the solutions from what you have learned here. If you cannot fix it, try checking Books Online for more information to locate the internals of the process.

Log Shipping

Another form of copying your data from one location to another is called log shipping. In this process, the Transaction Log from one database is backed up to disk and then copied and applied to another database. You could actually manually create this kind of process, but Microsoft provides a wizard to lead you through the setup and jobs that monitor it for you.

When you apply the Transaction Logs, the other database (called the secondary database) can be in Standby or Recovery mode. In Standby mode, users can query the secondary server database in a read-only fashion, useful to offload some work from the primary server. In Recovery mode, the database is not available to the users for data access. This is more useful to protect the data in the case of a problem with the primary server. In either case, switching from the primary to the secondary server as the main database system is a manual process. Switching back is also a manual process, and requires a little planning and forethought to do properly.

There is another function within the log shipping scenario that "watches" the processes to ensure they are working and to display the current status. This monitoring process can live on another server or on the primary server.

There are four jobs that make up the log shipping operation, as follows:

  • Backup jobs

  • Copy jobs

  • Restore jobs

  • Alert jobs

The backup job backs up the primary database, logs history and monitoring information, and deletes old information. It runs every two minutes by default, but you can change that interval during the setup. The copy job runs on the secondary server and copies the log backup files from the primary database to the secondary server. The restore job applies the copied database log files on the secondary server. The alert job is responsible to send out alerts if there is an issue with the process, and they run on the monitoring server (if you have one) or on the primary and secondary servers.

The normal process for setting up log shipping using the SQL Server 2005 wizards involves ensuring that you have a few things in place ahead of time. You need an edition of SQL Server 2005 that supports log shipping (Standard, Workgroup, or Enterprise). The database you want to ship needs to be in the Full or Bulk-Logged Recovery Model. You will need a share location on the secondary server to place the logs on, and you will need to establish an account between the two servers to facilitate the data transfer.

If the primary server fails, apply any remaining Transaction Log backups on the secondary server. Use the WITH RECOVERY mode of the restore operation to give the secondary server the ability to read and write data. Any code you have needs to be able to point to the new server name, because the two servers retain their original identities.

If you want to switch back to the primary server, you have a couple of options. The first is to back up the database on the secondary server and restore it over the top of the original database on the primary server when it comes back online. To follow this method, you will need to get everyone off of the secondary server, back up the database, restore it to the primary server, and then change the applications to point back to the primary server. This is the process you should follow to use log shipping as a recovery method. Make sure you test the process and document the interval the application is unavailable.

The second method is more useful if you are using the system as a failover mechanism in the case of upgrades or maintenance. In this case, you know ahead of time that the server will be taken offline. Have the users log off of the application, and back up the Transaction Log manually, using the WITH NO RECOVERY mode of the log backup operation. This leaves the database ready to accept the Transaction Log backup of the secondary server when you are ready to "fall back" to the primary server when your maintenance is complete.

Restore the log backup on the secondary server with the WITH RECOVERY mode of the restore operation to bring it up to the latest state and point your applications to the new server. When the primary server is ready to return to service, reverse the above process.

Setting Up Log Shipping

Let's take an example of the entire process using the wizards. On a production server, you will have at least two servers involved in log shipping; if you want to, however, you can follow along on your test server. I demonstrate running the setup on a single server.

First, ensure that your test database is in the Full Recovery Model by right-clicking it and selecting Properties. With that complete, on this test system (called \\SQL), I have created a share for the backups called \\SQL\Backups. Everything is now ready to start the wizard. I right-click the database name and select Tasks from the menu and then Ship Transaction Logs. You can see the results in Figure 6-33.

Figure 6-33.

[View full size image]

Because I am demonstrating the process on a single system, I have two instances (SQL\S2K5DE1 and SQL\S2K5DE2). I will Log Ship from the LSMaster database on the first instance to the second. Inside the main panel, I click the Enable… check box. You can see that in Figure 6-34.

Figure 6-34.

[View full size image]

With that selection made, I click the Backup Settings button to set up the log backups for transfer.

Inside the panel shown in Figure 6-35, I set up that network path for the share, and set the delete and alert job time schedules. I leave the backup job name alone, because it is descriptive enough. I click the OK button to leave this panel.

Figure 6-35.

[View full size image]

Back on the main panel, I am now ready to set up the destination for the process. I click the Add button to show the screen shown in Figure 6-36.

Figure 6-36.

[View full size image]

This panel has three tabs. In this first (shown here), I have picked the second instance on my server, set the database name, and because it does not exit yet, I have instructed the wizard to create a backup and apply it for me.

The second tab, called Copy Files, has locations for the copied log files and a schedule and name for the deletion jobs. I have set mine to use the C:\temp directory, but if you are using this in production, you will want a dedicated location on the server that has enough disk space and is properly secured.

The last tab on this panel sets the mode the database is in when the restore is completeStandby or No recovery. Remember that only Standby allows queries against the shipped database. This panel also sets the recovery and warning intervals as well as the name of the restore job.

With those selections complete, I click the OK button to return to the main panel and OK again to begin the process. The wizard completes and the log shipping begins.


Real-Time Availability Mechanisms

Although near-real time availability is applicable to many situations, sometimes you need to make a system available in real time. A real-time highly-available system is essential where emergency services or medical systems are involved, or in utilities such as power plants. You also need real time high-availability for global systems, where a central server stores data needed by other servers. In these cases, the system cannot be down, not even for maintenance. Because you need to maintain and upgrade your system, you need to be able to provide a seamless method for the applications to get at the same data regardless of a particular server's state.

Because a server can and probably will develop a requirement to be offline at some point, the only alternative is to provide multiple servers to service the application's data requests. Using coding techniques, developers can write the application to request data from a federated set of servers. Federating means that each SQL Server 2005 system retains its own identity and another system brokers the data requests to an available server. There are many ways to design this type of architecture, but only a few require any special configuration for the SQL Server systems.

The other methods of implementing multiple servers are database mirroring and clustering. In database mirroring, a copy of the entire database is available on another server, in real time. Once again, the servers retain their individual identities. The application must "know" which server they point at for data. It is a simpler coding change, however, to implement database mirroring than federating the data layer.

The most comprehensive mechanism for providing real-time high-availability is the database Cluster. Using a combination of the Windows operating system and SQL Server 2005, two or more servers present a separate, single name to the applications, called a Virtual Server. Applications are coded only to talk to the Virtual Server, and the Clustered systems behind it ensure that one of them will continue to provide services such as the database engine and Analysis Services under the guise of the Virtual Server's name and IP address. This is the most costly mechanism to implement, and it requires special hardware to make it happen.

Database Mirroring

Database mirroring is similar to log shipping but is more automatic for the failover. It can be used instead of clustering. It requires no special hardware but does require another database instance to implement. Database mirroring cannot mirror system databases.

Although database mirroring is immediate and fails over automatically, client programs need to be able to switch to a new server in order to complete the recovery. Although most programs need to be recoded to have this feature, some of the Microsoft .NET development tools have a special setting within the connection string that can detect the failover and set the application to use the new server.

There are three components within database mirroring: the principal server, the mirror server, and an optional witness server. The principal server holds the original database, the mirror server holds the copy, and the witness server keeps track of the transfer and state of the system. You can use the SQL Server 2005 Express Edition as the witness server so that you do not incur additional license costs.

In the earlier releases of SQL Server 2005, database mirroring was not enabled by default. If you have one of these releases, you can start the Database Engine service with trace flag number 1400 to enable the Mirroring feature. Check Books Online for more information about this trace flag. You can use the DBCC TRACEON (1400) command or start it with the service by adding T1400 to the startup line of the service.

Database mirrors can also interchange roles, making the primary server the mirror and the mirror server the primary. In most cases that I have seen database mirroring implemented, it is as a failover measure and not to fail back and forth this way.

You can work three modes for database mirroring. If you are using a witness server and you have the Transaction Safety Setting (more about that in a moment) to Full, you are using the High-Availability Operating Mode. If you have the Transaction Safety Setting to Full but you do not have a witness server, you are operating in High-Protection Operating Mode. If you have the Transaction Safety Setting off, you are operating in High-Performance Operating Mode. All of these have to do with how often the transactions are committed and how quickly you fail over. In the first mode, you are protected and the failover is fast, and in the last you are not as current.

You do not need any special hardware to use database mirroring, but it is a good idea to have a fairly robust network connection between them. Unlike replication, in database mirroring only two systems are directly involved. In other words, you cannot replicate from one principal server to multiple mirror servers. You also want to make sure that the times on the servers are synchronized either to each other or to a central time server. Microseconds do not matter as much, but seconds do.

Database mirroring communications work through TCP endpoints. Endpoints are essentially a direct connection to the database engine. You need to create these on both the principal and mirror servers so that you can make the connection between them. I show you how to do that next.

Setting Up Database Mirroring

You can follow a graphical setup on your system by simply right-clicking the database and selecting Mirror from the menu that appears. When you use this set of panels, it is not always clear what you need to set up first. In this section, I explain how to create the entire process using scripts, so that we can talk a little about the process. In these scripts, I call the principal server PRINCIPAL and the mirroring server MIRROR

Both systems need to be able to connect to each other using TCP/IP. You also need a file share location on the principal server that both systems can read and write to. I call that \\PRINCIPAL\DBMIRROR on my test system

We begin on the principal server first. We need a set of keys to encrypt the data between the servers. This is not required, but I normally set one up. Create a key and a certificate, and then store that certificate on the share:

 PRINCIPAL
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'password';
CREATE CERTIFICATE principal_cert
AUTHORIZATION DBO
WITH SUBJECT = 'Principal';
BACKUP CERTIFICATE principal_cert
TO FILE = '\\PRINCIPAL\DBMIRROR\principal_cert.cer'
GO;

Next, we need a connection that the process uses for the principal server to connect to the mirror server, and vice versa. I explained endpoints earlier in the book when we discussed connections, and database mirroring is one of the places this is used. In this example, the authentication is set with the certificate we just made for the mirror_ep endpoint:

CREATE ENDPOINT mirror_ep
STATE=started
AS tcp (listener_port=5022)
FOR database_mirroring (role=all, authentication=certificate
principal_cert);
GO;

It is set to start automatically, and the port is 5022. Next, the database needs to be backed up so that it can be restored. In this example, the database name is MIRRORTEST:

BACKUP DATABASE mirrortest
TO DISK = '\\PRINCIPAL\DBMIRROR\MIRRORTEST.BAK'
WITH FORMAT
GO;

The system needs a common account between the servers that can use the certificate. In this example, we create a login, then a database user, and grant the account the certificate and the ability to talk to the endpoint:

CREATE LOGIN mirror_user
WITH PASSWORD = 'password';
GO;
CREATE USER mirror_user
FOR LOGIN mirror_user;
GO;
CREATE CERTIFICATE mirror_cert AUTHORIZATION mirror_user
FROM FILE = '\\PRINCIPAL\DBMIRROR\principal_cert.cer';
GRANT CONNECT ON ENDPOINT::mirror_ep TO mirror_user;
GO;

Finally, set the database to have a partner from the mirror system, using the TCP port:

ALTER DATABASE mirrortest
SET PARTNER = 'tcp://MIRROR.mydomain.mycompany.com:5022';
GO;

Now we move over to the MIRROR server and run the following script to create the certificates to work with:

 MIRROR
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'password';
GO;
CREATE CERTIFICATE mirror_cert AUTHORIZATION DBO WITH
SUBJECT = 'Mirror';
BACKUP CERTIFICATE mirror_cert TO FILE =
'\\PRINCIPAL\DBMIRROR\ mirror_cert.cer';
GO;

Now create an endpoint on the MIRROR server to talk with the PRINCIPAL server. In this case, I will just use the same names:

CREATE ENDPOINT mirror_ep
STATE=started
AS tcp (listener_port=5022)
FOR database_mirroring (role=all, authentication=certificate
mirror_cert);
GO;

And now the same login and user account, granted it access to the endpoint:

CREATE LOGIN mirror_user
WITH PASSWORD = 'password';
GO;
CREATE USER mirror_user
FOR LOGIN mirror_user;
GO;
CREATE CERTIFICATE mirror_cert AUTHORIZATION mirror_user
FROM FILE = '\\PRINCIPAL\DBMIRROR\principal_cert.cer';
GRANT CONNECT ON ENDPOINT::mirror_ep TO mirror_user;
GO;

We will restore the database to the mirroring server, and leave it set to a recovering state, ready to apply changes from the principal server:

RESTORE DATABASE mirrortest
FROM DISK = '\\PRINCIPAL\DBMIRROR\MIRRORTEST.BAK'
WITH NORECOVERY
GO;

Finally, set up the restored database to participate in the mirroring:

ALTER DATABASE mirrortest
SET PARTNER = 'tcp://PRINCIPAL.mydomain.mycompany.com:5022';
GO;

The server is now mirroring the two databases. To check the mirroring, you can check a table on the PRINCIPAL server:

SELECT *
FROM sys.database_mirroring
GO;

Clustering

Of all the methods of high availability, only clustering meets the requirements of securing the data to multiple locations in real time and providing application-agnostic switching. In this section, I explain the concepts around a Microsoft cluster implementation and then describe how to install, configure, and manage SQL Server 2005 on that cluster.

The original meaning for a computer cluster was a group of systems that coordinated processing among themselves. This means that a computer cluster in the classic sense shared processing, storage, memory, and applications. To do this required a dedicated set of hardware and an operating system designed to manage the interaction between the systems, deciding which process would be handled by which system and so forth. Under this type of cluster, the focus was often more on sharing the processing load (called load balancing) than the redundancy that was inherent in the system.

Microsoft uses the term cluster to define a failover system, one that allows a single system within the cluster, or node, to leave the cluster without interrupting operations. Little if any application load balancing occurs in a Microsoft cluster without special coding.

Microsoft calls its new 64-bit true-clustering software a "compute cluster" to indicate that it does true application load balancing. This chapter focuses on the failover cluster technology found in Windows 2000 and Windows 2003.

To use a physical example of Microsoft clustering, think of the window at a doctor's office. You, as the patient, always approach a single window to make your requests. There are usually two or more attendants behind the window, each capable of handling your request, but only one that speaks with you. If the first attendant is not available, the next steps up to help you. You do not concern yourself with which attendant answers the question, as long as the question gets answered.

In the case of a cluster, the "window" from the preceding example is a name (called a virtual server) and IP address provided to the network for all the attachments from the applications. The "attendants" are the servers (called the nodes) involved in the cluster. The "patients" in this example are the applications.

It is a bit beyond the scope of a single chapter to cover everything to know about Microsoft clustering, but the basic concepts are not difficult. I cover these basics in this chapter, but before you implement a cluster, you should learn more about installing and configuring a cluster using Microsoft operating systems from the Microsoft clustering Web site.

To keep the examples simple, I explain a two-node cluster, although depending on the software you use you might have up to eight nodes in a single cluster. I explain more about which versions of SQL Server supports clustering in a moment.

In a two-node cluster, two computers of similar hardware share three things: A network card each for the "public" network that all clients can see, a network card each configured for a "private" network that only the two nodes can see, a shared SCSI bus for some of the hard drives that both nodes can see, and three sets of drives. The first drive is connected locally to each system, like any other server. Each node can see only its own version of this drive or set of drives.

The second drive or set of drives is set up on the shared SCSI bus that both nodes can see as a "quorum," which is the place where a set of data files are stored to show who "owns" the cluster resources at any given moment.

Windows 2003 has the ability to store the quorum files on multiple systems, not just on a shared drive. This allows a greater degree of redundancy, but it is a bit more complicated to administer. In this chapter, I will stick with a single quorum drive.

The third drive or set of drives is also on a shared SCSI bus and is used for data that each node needs to keep in common. For a database application such as ours, this is the drive where the data files and log files go. Although you can use the same SCSI bus that the quorum drives use, I recommend that you use a second controller for this purpose.

From the software side, you need an edition of the operating system that supports clustering. As of this writing, that is Microsoft Windows 2000 Advanced Server (two nodes), Microsoft Windows 2000 Data Center (four nodes), Windows 2003 Enterprise Edition in either 32- or 64-bit (eight nodes), or Windows 2003 Data Center (eight nodes).

There are several other requirements for a Microsoft cluster, both from a hardware and software standpoint. Microsoft maintains a "Hardware Compatibility List" on their Web site that is mostly ignored during regular server installations. You cannot ignore this list for a cluster because the hardware is examined during installation. If it does not meet the requirements, the installation aborts I explain the specifics further in a moment.

After the hardware is assembled and configured, you install the cluster from the Windows CDs. In Windows 2000, you install the clustering software from the Windows Components section in the Add and Remove Programs of the Control Panel. In Windows 2003, it is pre-installed for you and just needs to be configured. In any case, a wizard will guide you through selecting the network cards used for the public and private networks, the drives for the local, quorum and shared disks, and will automatically create a cluster group that shares various resources. Clustered resources include network shares, IP addresses, applications, and more.

When the installation of the clustered operating system is complete, you can install and configure applications and resources within the cluster, which will present themselves on the virtual server. To cluster an application, it needs to be cluster enabled or cluster aware, such as SQL Server 2005. These applications are written to understand and use the facilities of the cluster and have capacities to fail over if the primary node is unavailable.

Within each cluster, nodes are defined as either active or passive. An active node retains its identity and can share resources like any normal server in addition to the cluster. A passive node lurks in the background, its only purpose to provide the cluster resources in case of a failure. In a cluster, you could have an active/active arrangement, where two nodes are active, or an active/passive arrangement, where one node is active and the other passive.

The ramifications for these arrangements have to do with licensing and resources. Having an active node fail over to another node incurs another license and more hardware. If you have a node fail over to a system where SQL Server is already running, you will need to ensure you have enough memory and hard drive space to support both instances of SQL Server on the system.

The licensing information I have described has to do with Microsoft products and only in a general sense. Always check with your software vendor to ensure you are within licensing compliance for your installation.

Clustering Requirements

Setting up a cluster is not difficult, but it has more requirements than setting up a standard SQL Server. You face three main design decisions when you create the cluster: hardware, operating system, and platform.

Hardware Requirements

As I mentioned earlier, your choices with the hardware decision are guided by the Hardware Compatibility List (HCL) from Microsoft. The primary difference between the hardware guidelines you follow for a standard SQL Server installation and those on the HCL is that the HCL defines makes and models of computers, not just requirements for their architecture. You might be tempted to stray from the list, but you will have to use hardware from the list if you want support from Microsoft.

On a general level, you need two servers, and it is a good idea to make sure they are similar. In fact, I recommend that you pair the systems, so that when you have issues, you can eliminate a difference in configuration as an issue. This is not a requirement, but you will find it easier to manage two (or more) servers that have the same configuration, especially in a clustered environment.

Within those servers, you need at least one shared SCSI bus. This is a special type of controller that can see two servers on the same channel. Once again, you want to pair these cards by manufacturer to avoid configuration problems.

The shared SCSI bus is required for at least one drive that is shared among all the nodes in the cluster, called a quorum. The quorum drive stores files that ensure that only one server acts as the primary node in the cluster. It has to be available for the cluster to work. You do not need a lot of space on the quorum drive, just around 500MB or so. You also do not want to store anything else on that drive, and it is a good idea to have it mirrored or protected in a RAID set. Most often, you will see the drive letter set for the quorum set to Q:, but that is only a convention.

For a SQL Server cluster, you need at least one more shared bus between the nodes in the cluster. Because a SCSI device can have several IDs per controller, you can use the same controller with a different drive ID for this storage; for higher performance and security, however, you can install another shared SCSCI bus card in each node for the drives. The SQL Server cluster needs these drives for data and log files. The same layouts that I explained in Chapter 1, "Installation and Configuration," apply to a cluster.

The next set of hardware you need to ensure you have for the cluster is at least two network cards for each node. The first network card provides a method for the client machines to access the nodes, called the "public" network. Although not a hardware requirement, you need a standard set of IP addresses for the public network cards on all your machines. You need another IP address for the cluster itself. This is what the users access, and what the nodes present as a single network resource to keep the cluster alive.

In the case of the SQL Server cluster, you need another IP on the public network for that, too. This is the address that SQL Server presents to the network for the clients.

The other network card is used for a "private" network between the nodes of the cluster. Each node pings the other to make sure they are still available. If a node does not answer back on the private network within a specified period of time, usually in a few seconds, the node is removed from the cluster. If the node was the primary, the cluster's resources are transferred to another node, keeping the cluster alive. This second card will need an independent IP network, and it is best to keep them on either a second set of switches or, if you are using only two nodes, connected directly to each other with a cross-over cable.

There are other hardware components and configurations that are important in a cluster, such as Error-Correcting memory and hot-swappable parts, but they are not technically required for the cluster to work. You should seriously consider these measures because the whole purpose of a cluster is to enable high availability, and anything you do to ensure that will help. You do not want to put a cluster into place only to have a cascade failure of cheap memory chips.

OS Requirements

Not every operating system supports a cluster. At this writing, you can choose from Windows 2000 Advanced Server or Windows 2000 Datacenter, and Windows 2003 Enterprise Edition and Windows 2003 Datacenter.

You also need an Active Directory domain, a DNS server, and everything set up with the latest security patches; and all nodes need to be on the same domain. If possible, I recommend that you build the entire cluster connected only to each other, and then connect the nodes to the Internet to upgrade them. The reason is that if you install the operating system and then patch it, you will have a lot of CD switching that you will have to do to install the cluster. This largely goes away on Windows 2003, but I still follow the same process.

Platform Requirements

Just as certain operating systems support clustering, only certain editions of SQL Server 2005 support clustering. You need either SQL Server 2005 Standard Edition for up to two nodes, and Enterprise Edition for the maximum that the operating system supports.

You do not have to install anything else to make the cluster work for SQL Server. All you have to do is ensure all nodes are online and log on to the primary one. Start the installation program, and the entire process is handled automaticallyassuming that your cluster meets all the requirements. If it does not, the installation program will let you know what problems it encounters; you can check the documentation to see what you can do to fix the problem.

In the next section, I lead you through installing the software on a cluster. If you do not have a cluster to practice on, check out the information in the "Take Away" section at the end of the chapter. If you do not have a lot of cash, you can get evaluation versions of the virtual machine software, the operating system, and even SQL Server 2005 to practice.

Installing SQL Server 2005 on a Cluster

To install SQL Server 2005 on a cluster, you need to have one of the operating systems I described earlier installed and configured as a cluster. If you do not have access to a system that is clustered, you can follow the instructions in the "Take Away" section at the end of this chapter to build a two-node cluster using virtual machine technology.

It is important to keep a few terms straight: remember that a virtual machine is the same thing as a physical server in this chapter, and a virtual server is what the cluster presents to the users as a single computer. So a virtual server runs on a virtual machine. On a physical set of systems on your actual network, you will only work with a virtual server.

After the operating system is clustered and configured with all current service packs, ensure the cluster is running. You need to configure the Microsoft Distributed Transaction Coordinator service (or MSDTC) to run as a clustered application. On all nodes, ensure that the service is set to automatic and then open an operating system command prompt. Type comclust.exe and press Enter. Allow that process to complete before you continue.

Next, open the SQL Server 2005 installation program on the primary node and start the setup program from the installation media. The server will detect the cluster and begin the installation.

As you can see in Figure 6-37, when the installation program starts, it proceeds like a normal installation, but when you reach the components installation screen, you can choose to create a SQL Server failover cluster for the database engine and Analysis Services, as shown in Figure 6-38.

Figure 6-37.

[View full size image]

Figure 6-38.

[View full size image]

The database and log files need to be available to both systems. When you create databases on the system, they should go on the shared drive for the cluster, but not the quorum drive. You will set that in a moment.

When you are past the selection portion of the installation, you are asked to provide the name of a "virtual server." This name is the "window in the doctor's office" from the example I mentioned earlier. It is the name that the cluster will ensure is available for client requests. As you can see in Figure 6-39, I have named the Windows 2003 cluster SQLCLUSTER and the SQL Server ClusteredSQL.

Figure 6-39.

[View full size image]

Next, you are asked to set the address the SQL Server Instance will listen on for client requests. In Figure 6-40, I have typed in my public address in this example (10.1.1.5) and clicked the Add button to add the address to the list in the bottom panel.

Figure 6-40.

[View full size image]

In the panel shown in Figure 6-41, you are asked which cluster group you want the instance to belong to. I have set mine to the only group I have, and set the data files to go to the shared drive on this cluster.

Figure 6-41.

[View full size image]

From there, you are asked to select the nodes you want to include in the cluster. You do not have to install SQL Server on each node, but it will only be available on the ones you select. After that panel, you are asked to select an account that will control the SQL Server 2005 services, just as in a single-server installation. You are also asked for the startup groups for each service. After that selection, the installation proceeds through and completes a standard set of setup screens.

Managing a Clustered SQL Server Instance

After your SQL Server 2005 system is installed, managing the SQL Server is not all that different from managing a stand-alone SQL Server. You still use the SQL Server Management Studio for the relational database engine and Reporting Services and Business Intelligence Development Studio for Analysis Services design work. The only difference is that you connect to the name of the cluster rather than the name of the machine.

Once inside the tools, you can follow the same procedures I have shown you throughout the book for all tasks such as creating databases, backing up and restoring them, tuning, and security. In fact, the only time you will notice that you are using a cluster is in some of the properties of the server such as "IsClustered" show the status of the SQL Server system.

There are differences in the way you manage the cluster, however. You cannot just shut down or reboot the SQL Server whenever you want, in the way that you are used to. If you shut the primary node down, the secondary node will take over, called a failover event. Although that is not fatal, it is better to handle the interruptions in a controlled manner. You also face differences when you apply service packs to SQL Server or the operating system.

Cluster Maintenance

For the most part, a cluster has the same maintenance processes as a stand-alone system such as backups, defragmentation, and monitoring. When you need to work with the cluster nodes, or recover from a failover, or even install service packs, however, you will manage the system using the Cluster Administrator, under the Administrative Tools section of the Start menu.

If you are connecting to this tool on any node other than the primary, you may be prompted to connect to the cluster when you start it. Once inside the tool, you can view the groups defined within the cluster. You can see my system's screen in Figure 6-42.

Figure 6-42.

[View full size image]

Groups (in my case, I have one called cluster group) contain resources, which are services, programs, or shares presented on the virtual server name. You can see the other the resources I have that deal with SQL Server in Figure 6-43, including the cluster name (SQLCLUSTER) and the IP address (10.1.1.10).

Figure 6-43.

[View full size image]

To work with one of the resources, right-click it. That will bring up a menu that allows you to fail the resource, causing it to move to the next node in the chain; take it offline, which stops the resource but does not start it on another node, create a new resource, delete or rename it, or work with its properties. I have selected Properties to show the panel in Figure 6-44.

Figure 6-44.

[View full size image]

From this panel, you can change items such as which nodes will support the resource, set or change any dependencies that the resource has, and view or alter the parameters for the resource. If the resource is operating properly, you should not change anything here.

If you need to install a service pack on a node but not stop its operation, you should follow a specific process. First, ensure that all nodes are functional. Then, right-click the group that contains the resources and select Move Group from the menu that appears. You'll get a result similar to Figure 6-45.

Figure 6-45.

[View full size image]

After the resources are moved to another node, ensure that everything is functional. Then right-click the primary node and select Pause Node from the menu that appears to bring up the screen shown in Figure 6-46.

Figure 6-46.

[View full size image]

When the node is paused, apply the service packs and reboot that node. When it is back online, connect to the cluster, right-click the paused node, and select Resume Node from the menu that appears. Now reverse the process, and move the group back to the primary node. Pause the second node, log on to it and perform your maintenance, and reboot that node. Connect to the cluster again and resume the second node. Repeat this process for all the nodes in the cluster.

To place another node in the cluster, right-click the name of the cluster and select New and then Node from the menu that appears. You will be led through a series of wizard panels that will connect to and enlist the server as a new node in the cluster. The new node will need to be on the domain and meet all of the requirements for clustering.

Adding a node this way does not install SQL Server on it. You must run the install program again, answering the questions the same way that you did during the first install. Near the end of the installation program, the system will detect the addition of the node and allow you to select it. You will not really install SQL Server 2005 again on the nodes that already have it; the program will just install the software on the new node.

If you want to remove a node from the cluster, right-click it and select Evict Node from the menu that appears. If you do this, the node will no longer participate in the cluster.


Take Away

This chapter helps you understand what you need to know about working with highly available systems, but you cannot learn everything just by reading a book, any more than you can learn to play the piano by reading a book. Most people learn by doing.

The purpose of a cluster is to prevent a loss of accessibility. But to learn how all that works, you need to be able to fail the system and test things. You cannot "play around" on the production system because you would expose the company to a data interruption while you are testing a failover. Because of the cost and difficulty of setting up a copy of the production cluster, you might not have a development or test environment.

That is where virtual machine (VM) technology comes in. A VM is literally thatsoftware that emulates hardware. What the VM does is to run a program whose sole purpose it is to provide a processor, a BIOS, video output, hard drives, network cards, and inputs such as a mouse and keyboard. It "carves out" parts of your processing power, memory, and disk space to emulate computer hardware. When that environment is running inside a window on your screen, to any other software it looks like a computer. You can install any operating system on it, turn it on and off, and so forth. When you are "inside," you are working with a full computer. This allows you to run Linux on Windows, Windows on Linux, and so forth. Most of these VM software packages emulate an Intel or AMD processor.

Newer processors include "hooks" and other technology specifically designed to work with VM software. Both Intel and AMD are including more and more of these features in their architectures with each release.

There are a few major software vendors in the VM space, most notably Microsoft and VMWare. VMWare has three primary versions as of this writing, from Workstation to GSX out to a version called ESX. Each progression allows more memory, CPUs, and other features, and of course the higher versions cost more than the less-capable ones. Microsoft's offerings are Virtual PC for workstation-level use and Virtual Server for higher features.

I have used all these vendors and versions, and each works well. In the following example, I use Microsoft's version because I have found the process for creating some of the shared components a little easier, but I have built clusters with VMware's offerings, too. Regardless of which vendor you choose, pick a version that can share a SCSI bus between two or more machines. Most of the "personal" or "workstation" versions do not do that.

You need a single computer to create the VMs on. I recommend an inexpensive, highly capable workstation-class computer, something you might build yourself. I use a powerful x64 system with several gigabytes of RAM and a few inexpensive large hard drives that cost me only a few hundred dollars. With that, I have set up entire networks, including clusters, replication, database mirroring, and even transfers to other database systems such as Oracle and MySQL.

Before you begin, make sure you are comfortable with creating virtual machines on your platform, whether you are using VMWare, Microsoft, or some other vendor. I do not spend the time going through each individual step for that.

Setting Up a Two-Node Cluster Using Virtual Machine Technology

I use Microsoft's Virtual Server for this example, and I give you a simple outline of the process so that you can extrapolate to whatever VM software you have. Whatever you use, the only requirement is that it supports multiple disks, network cards, and a shared SCSI bus between two or more machines.

You also need to meet the software requirements for a cluster. If you do not have an MSDN license for the server software and SQL Server, you can use evaluation versions of both for the exercise. In this example, I am using Windows Server 2003 Enterprise Edition with SQL Server 2005.

Planning the Requirements

Before you design any system, even a test one, you need to have a layout planned. I create a plan for my test system and then develop the VM servers based on that plan.

My system will be a two-node, active-active cluster. I plan to have only one large database on a single shared drive. Because a cluster needs to be located in a domain, I will make the first node of the cluster the domain controller and also house the Active Directory and DNS there.

That plan means I will need the following inventory of hardware.

Device | Use
Computer, 512MB of RAM | Node A
Network card | To connect to the Internet
Network card | To connect to the "public" network between the two machines
Network card | To connect to the "private" network between the two machines for the heartbeat
Drive, 10GB | Primary drive for Node A
Drive, 500MB | Quorum drive for both systems
Drive, 9.5GB | Shared drive for both systems
Computer, 512MB of RAM | Node B
Network card | To connect to the "public" network between the two machines
Network card | To connect to the "private" network between the two machines for the heartbeat
Drive, 10GB | Primary drive for Node B

I like to create a graphic that shows the layout. Although the text descriptions are the final word, the picture shown in Figure 6-47 helps me keep everything straight.

Figure 6-47.

Here is the basic process to create the cluster:

  1. Create a virtual machine and tie it to the hardware required to fulfill the requirements you just createdin Microsoft's Virtual Serve, you create the hardware first and then tie it to the machine; in VMWare, you create the machine and then add the hardware.

  2. Install the operating system on the first server; assign all the network cards and drives, with the exception of the shared drives and the "WAN" NIC.

  3. Install the VM "tools" or "extensions" to set up all the devices properly.

  4. Promote the first server to a domain controller, adding DNS and Active Directory.

  5. Assign the shared drives in the operating system.

  6. Shut down the first machine.

  7. Create a second virtual machine and tie it to the hardware required to fulfill the requirements you created.

  8. Install the operating system on the second server; assign all the network cards and drives, with the exception of the shared drives

  9. Install the VM "tools" or "extensions" to set up all the devices properly.

  10. Assign the shared drives in the operating system.

  11. Start the first machine again.

  12. Join the second machine to the domain created on the first machine.

  13. Install Microsoft cluster software on the first machine, picking the quorum and shared drives.

  14. Join the cluster from the second machine.

  15. Connect the third NIC on the first machine so that you can reach the Internet.

  16. Apply all patches required.

  17. Install SQL Server on the cluster.

Creating the Hardware

With the requirements and this outline in mind, I can now begin to build the "hardware" to support the cluster. First, I will set up the network cards I need by opening virtual server administration screen as shown in Figure 6-48 and creating three virtual networks, similar to creating a router connection, with the following information.

Figure 6-48.

[View full size image]

Network Name | Connected To | Use
Private | Internal guests only | Heartbeat network.
Public | A physical network card on the server | Allows client traffic to the cluster.
WAN | A physical network card on the server | I will set this up when I am done so that I can download all the service packs and connect to the Internet.

Now I will make the three virtual drives needed for the cluster. The results are shown in Figure 6-49.

Figure 6-49.

[View full size image]

Drive Name | Size | Use
PrimaryNodeA | IDE, 10GB, expandable | Primary drive for Node A
PrimaryNodeB | IDE, 10GB, expandable | Primary drive for Node B
Shared | SCSI, 10GB, fixed | Will be divided into a Q drive and an S drive for both systems

With all the virtual equipment ready, I can set up the two machines and tie them to the network cards and drives.

Throughout this process, I had to switch back and forth between the virtual CD to install the extensions and the physical CD to install things from the Windows 2003 CD. Make sure you understand how to do this on your platform before you begin.

Creating the Servers

In Figure 6-50, I create a new virtual machine with the name NodeA. I give it 512MB of RAM and set it to use the drive I created earlier called PrimaryNodeA, using an IDE bus. I also add a network adapter to the public network I created.

Figure 6-50.

[View full size image]

Next, I add a shared SCSI adaptor so that I can connect it to the quorum drive. I give it a SCSI ID of 6 so that this node has a different address from the other node I create later. You can see that process in Figure 6-51.

Figure 6-51.

[View full size image]

And as a final step for this machine, I create another network card for the "heartbeat," connecting it to the private network. I am now ready to install the operating system on the first node. You can see the beginning of that installation in Figure 6-52.

Figure 6-52.

[View full size image]

Installing the Operating Systems

I set the name for this first system to NodeA. I will install the operating system with default options, other than the IP addresses on the cards. I will configure those cards with a 10.1.1.1 address for the public connections (as you can see in Figure 6-53), and a 10.2.1.1 address for the private heartbeat connection. I disable the second network card until I start the cluster installation, but you do not have to do this. I just find that it speeds the wizards along to only deal with one card.

Figure 6-53.

[View full size image]

Next, I will log in to the system and install the Microsoft Virtual Server extensions. Now I set that SCSI drive I created on the shared SCSI bus into two drivesa Q: drive with 500MB of space and an S: drive with 9.5GB of space. You can see those drives in Figure 6-54.

Figure 6-54.

[View full size image]

When that is complete, I run the dcpromo command to make this system a domain controller, and allow it to create a DNS server. You can see in Figure 6-55 that the Domain Controller installation process is just ending.

You cannot really create a functional cluster as I have here with only two nodes. The reason is that the clustering function depends on DNS, Active Directory(AD), and so forth. If you take the primary node down to simulate a failure, you take the DNS and AD offline, so the cluster will not work. You can mitigate this by making the second node a domain controller, installing another DNS pointing the first and vice versa, and so on. You can also make the primary node the system without the AD and DNS, and then practice failing only the non-DNS system.

Figure 6-55.

[View full size image]

After I shut the first machine down completely (very important for the Microsoft VM), I will build the second machine just like the first, with the following exceptions:

  • I name it NODEB.

  • I use the address 10.1.1.2 for the public card.

  • I use the address 10.2.1.2 for the private card.

  • I do not set it up as a domain controller, or as a DNS server.

I progress with setting that server up, I install the extensions, and format the Q: and S: drives, just like before. Now I start up NODEA again.

Installing the Cluster

I enable that second NIC (the one with 10.2.1.1) on NODEA, and now I have everything set up to install clustering. I start the Cluster Installation Wizard by accessing the Clustering Services Manager from the Start menu. You can see the wizard's first screen in Figure 6-56.

Figure 6-56.

[View full size image]

I name this cluster SQLCLUSTER, and set the quorum to use the Q: drive and published the S: drive as a shared resource. I set the cluster management tools to use an IP address of 10.1.1.10.

When that wizard completes, I start up NODEB and then add a node (NODEB) to the cluster using the same wizard. This time, I join the SQLCLUSTER cluster rather than create a new one.

Now the cluster is complete. All there is to do now is to install the service packs and SQL Server 2005 on it, as I described earlier in the chapter. If you follow this process, you can manually fail the cluster to the second node by using the Cluster Administrator, without having to take the first node down. Try connecting to the cluster using a client machine with the SQL Server Management Studio to see that the resources continue to respond even after the cluster fails to the second node.


Chapter 7. Notification Services and the Service Broker

In the last chapter, I explained how you can use a feature called Replication to send data from one system to another. That discussion was framed in regard to high availability, but of course you can use Replication for many other applications that require data interchange.

In this chapter, we examine a more targeted approach for data distribution. Using Notification Services, you can trigger specific datagrams to users based on conditions in your database, from notifying someone that a stock price has changed to alerting that a machine has just gone offline. Any type of event that you track in your database is available for use in Notification Services. Unlike Replication, you can use Notification Services to send data to an e-mail address, a cell phone, or using the Simple Messaging Service (SMS) to any device that conforms to that standard.

There are a couple of ways you can use Notification Services. You can treat it primarily as a task that the maintenance database administrator (DBA) is responsible for or you can enable it for your developers. In this chapter, I focus on the DBA side of things, but you should be aware of its many applications.

You can use SQL Server 2005 as part of a robust service-oriented architecture (SOA). An SOA allows you to build applications in a new way, distributing the load across multiple servers using messages between them. SQL Server 2005 provides the Service Broker as a programmable object that you can use as a store-and-forward mechanism. Although you might not write these programs, as the DBA you will be responsible to set up and manage this part of the framework for an SOA.

In the second part of the chapter, I explain how you can help develop and manage an SOA using the Service Broker. I show you a sample application using the Service Broker in the "Take Away" section at the end of the chapter.

In the early days of computing, users did not get immediate responses from their programs. Developers wrote code in either machine language or something close to it, saved it on a media such as magnetic tape or punch cards, and submitted it to a computer. The computer would run the program and create an output, either more punch cards or a printout. The users would receive the printouts as their results, disconnected from the computer. This is called "batch processing"; and most of us could not imagine working this way any more.

In modern applications, users connect to the database either directly or through a data-access layer in the system. They input data and receive immediate feedback using a video screen. This book was produced with a program using that computing method, and you might even be reading it that way.

But there are times when batch processing or disconnected computing is useful. In some applications, the user is not near the computer when the results are tabulated or created but may still need the data. Other systems might also need access to the data but not all at once. The feedback between the data and the user does not have to be immediate.

The value in this paradigm is that you can spread out the system over large areas and balance the loads between the components. Not needing the immediate feedback lowers the amount of resources a system requires.

You are already using at least one disconnected system in your daily routine. E-mail, for instance, is created on one computer at a certain time, stored on another computer at a later time, and picked up and read on still another computer at a later time. The e-mail server provides the reception, storage, and delivery of the message as a service. In effect, e-mail is a part of an SOA. In an SOA, a server provides interfaces and data that are available to any program that knows how to talk to the service. The service can also be coded to accept input, or perform some other action based on the connection.

SOA systems are not new; in fact, Microsoft provides many SOA mechanisms in programs such as BizTalk or as separate add-ons to SQL Server 2000. In Microsoft SQL Server 2005, Microsoft includes the Service Broker directly integrated into the database engine.

If Microsoft already has SOA systems available in other products, why include it in SQL Server 2005? The reason has to do with the requirements that an SOA has. If you are going to allow systems to be disconnected, your SOA has to guarantee that the traffic between the systems is encapsulated into messages so that the sending program is identified, so it can receive the proper answer from the service. You also have to make sure that the messages are ordered properly. For instance, the program might send the third line of a purchase order before it sends the header. The service needs to be aware of the order and the encapsulation so that it can respond to the purchase order only when it is complete, just as in a database transaction.

To manage all this, SOA systems use a database. In other mechanisms, you need to manage not only the SOA system but a complete database as well. By including the SOA within the database engine, the data, metadata, tracking data, and mechanisms are all contained within the same architecture. There is just one system to learn, implement, and manage.

Another type of SOA is a "push"-oriented data system. I explained a little about this type of data movement in the last chapter when we examined SQL Server's Replication features. In that environment, data is either sent or picked up by another database. Although that is a useful feature, you will often need to send data to a medium other than a database, such as e-mail or an SMS device. In SQL Server 2005, Microsoft includes the Notification Services system that can do just that.

Many people in your technology department are normally involved in creating an SOA, simply because of the vast array of technical skills required to implement and maintain it. As the DBA, you will be asked to manage and maintain the system, working with developers, business analysts, and others.


Notification Services

Notification Services is one of those products that does what it says: It provides a means of notifying a user that an event has occurred in a database. The system uses several components to accomplish this goal, from the databases that store data about the events to the control files and databases that shepherd the process.

Let's begin by examining a few applications for Notification Services and how you can distribute them. The most obvious uses involve applications that require immediate feedback for a user. These include changes in prices, levels, inventory, and any other information that is time sensitive.

Other uses for Notification Services might not be as obvious. Marketing studies show that clients are most frustrated when they feel that a company is not responsive. You can code your applications to notify a customer as to which stage an order is in, such as when it is complete, when it leaves the building, or any other action that makes a change to a status code in a database. You can also set up a notification that an order was received and when it was placed.

For the system administrator, you can use Notification Services to help reactively manage your systems. Normally, a reactive mode of managing anything in IT is a bad idea, but using Notification Services you can build in the look-ahead logic so that a server watches objects (like backups or other maintenance) for you and can alert you when a threshold is reached.

So just what is Notification Services? It is an infrastructure within SQL Server 2005 that involves a service, instances, and applications, and a set of programs your developers create to interface with the system. In this chapter, I focus on the Notification Services instances and applications.

The easiest way to create the system is programmatically using notification management objects (NMOs). You can find several examples of this type of programming if you installed the sample and applications on your server.

You can also create and manage Notification Services using the native tools within SQL Server 2005 using XML files. We take a look at the structure of these files and examine the results of its implementation so that you can see how to manage and maintain it. In practice, most applications are coded using NMOs, but you can always create the instance and application XML files with an Export from the Object Explorer right-click menu in SQL Server Management Studio.

Notification Services Architecture

There is a lot of excellent information in Books Online regarding Notification Services, but it is not arranged in a holistic view of the system. There is a good reason for that. Notification Services uses several components within the SQL Server 2005 platform to accomplish its goals. In addition to all the capabilities that Microsoft delivers, your developers can extend every part of the system with custom programs and interfaces. To get a picture of how this all fits together, let's start at the back of the system and work forward to the user who receives the notification.

Because Notification Services has so many components, I am going to cover it three times. In the first, I give you a broad overview, then I explain it again with a little more detail, and then I cover the various basic components by explaining the Instance Configuration File (ICF) and the Application Definition File (ADF). Even with all of this information, you are only seeing a quick overview of this topic.

In Notification Services, several terms are used that already have meanings within SQL Server, such as instance, publisher, subscriber, and so forth. In this chapter, pay close attention to the definitions that Notification Services uses for these common terms.

At the back of the process is SQL Server 2005, which stores the data the users need to see and all of the metadata that Notification Services requires to operate. On top of SQL Server is the NSServer.exe program, which runs all of the instances, which make up the notifications for the users. To create these instances, you have two options: Your developers can create them with programs that use NMOs, or you can create them using an XML document called an ICF.

With an instance created, you create one or more Notification Services applications within it that listen for events, which are changes that your users are interested in knowing about. Events might involve database activities but also can track changes in Analysis Services, files on a file system, and other activities.

These events are matched with subscriptions, which are sent to the subscribers (users) over various protocols to devices such as cell phones or e-mail. Once again, your developers can create Notification Services applications using NMOs or you can create them with an XML document called an ADF detailing the parts of the application.

If you create the instances and applications using XML files, you import them using SQL Server Management Studio or the NSControl.exe command-line program.

That is the first overview, which leaves out quite a bit of detail. Let's examine the process a bit further. After the instance and applications are created and running, the instance uses a program called NSServer.exe that runs in the background watching for events using an event provider. Event providers watch for database changes, file system changes, and other events. It then applies matching rules to pair up the subscriptions and the events and then sends the data on to a provider. A provider is a piece of software that receives the data and formats it for delivery.

Providers are called by the application, and a generator matches up subscriptions to the events. Subscriptions are groups of data that a subscriber (the user) cares about. Notification Services populates the entire subscription, which is read by a distributor. Distributors use special-delivery services to talk not only to other databases but to file systems, e-mail and telephones, and any other kind of communication system your developers create.

The distributor formats the output using a content formatter and then sends it on to the delivery protocol provider, which can send data over SMTP, HTTP, and SMS. The subscriber receives the data in that format, over that protocol.

As the third explanation of the process, let's take an even deeper look at two of the primary components that you will work with: the ICF and the ADF. Using these XML files, you set up the instances and applications that enable your developers to create a complete Notification Services system. All the pieces I have been talking about begin to come together as you examine these files.

Instances and the Instance Configuration File

An instance of Notification Services is a collection of applications that run as a unit on a SQL Server. You can have one or more instances on a single SQL Server. You can create an instance programmatically using NMOs or by using SQL Server Management Studio and an XML file called an ICF. That is the approach I take here.

DBA 101: XML

If you are not familiar with XML, it is not difficult to learn. It is an ASCII file with special characters (called tags) in front of the text you want to mark off in some way. It is the same principle as HTML, but the tags in XML are not set by an independent body. You can make up any tag you want. All you have to remember is to "start" the tags (in this format: <thing>) and "end" or close the tag (using the format </thing>), add a bit of header information, and keep the tags nested properly (in which case, the document is said to be well formed).

In XML, the words that are tagged are called elements, and any other information about those words are called attributes. Elements are enclosed by tags, and attributes go inside the brackets. Here is an example of an element called name with the value of Buck, and an attribute of first:

<name type="first">Buck</name>

Another item that you will see in XML files is the comment. It looks like this:

<! This is a comment >

Comments are actually a form of directives. Directives are special characters (such as question marks) that tell the XML parser to act in a certain way (in this case, ignoring the text that follows).

An important difference from HTML to keep in mind is that XML is case sensitive. Buck is not the same as buck. When you are matching the tags, this often comes back to bite you.

There is a lot more to know about XML, of course, but these basics will help you read the files I show here.

The ICF contains the name of the instance, a database for its control, the name of the applications that can talk to the instance, and encryption and delivery information. You can also place a version stamp in the file, which I recommend for production applications. The format for the file is detailed in Books Online under the topic "Instance Configuration File Reference."

I display the minimal ICF file here so that we can talk a little about the parts that are required. In a moment, I show you the results of importing an ICF file into the database using SQL Server Management Studio. If you want a useful example of Notification Services, it is best to consult the examples from Microsoft, if you installed them. Setting up a complete system with an interface and databases and explaining it all would take up much more than a single chapter and would involve you in parts of the system that a DBA does not normally handle. I do recommend you use the examples, however, because they will shortcut all the development for you and allow you to concentrate on managing a Notification Services system.

Here is the minimal version that you need to fill out for a basic instance. Each section is commented out ( <! comment >) so that we can more easily discuss it:

<?xml version="1.0" encoding="utf-8"?>
<NotificationServicesinstance
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.microsoft.com/MicrosoftNotificationServices
/ConfigurationFileSchema">

<! Notification Services instance Name >
<instanceName></instanceName>

<! Database Engine instance >
<SqlServerSystem></SqlServerSystem>

<! Applications >
<Applications>
 <Application>
  <ApplicationName></ApplicationName>
  <BaseDirectoryPath></BaseDirectoryPath>

<ApplicationDefinitionFilePath></ApplicationDefinitionFilePath
>
 </Application>
</Applications>

<! Delivery Channels >
<DeliveryChannels>
 <DeliveryChannel>
  <DeliveryChannelName></DeliveryChannelName>
  <ProtocolName></ProtocolName>
 </DeliveryChannel>
</DeliveryChannels>

</NotificationServicesinstance>

It looks like there is a lot going on here, but it is actually quite easy to follow. The first five lines of the file represent an XML header, which details what the file is and how it will be used. That does not change for any of your instances. The rest of the sections control the creation of the instance, similar to the old .INI files of earlier Windows programs.

<! Notification Services instance Name >

The first section, specified in the <instanceName> element, sets the instance name for the system as well as the service it creates to run this instance. From here on out, the applications and other parts of this notification system's grouping will refer to this name. I usually keep these short and as descriptive as possible. The instance Name becomes the name of the service that runs in the operating system, prefaced by NS$. For example, the following entry creates a service called NS$AWInventory:

<instanceName>AWInventory</instanceName>

<! Database Engine instance >

The <SqlServerSystem> element sets the name of the SQL Server instance (no relation to the Notification Services instance set up with this file) where the Notification Services instance will run. It is the name of the server where you want the service to run.

<! Applications >

Each instance holds one or more applications that can talk to it. The <Applications> element is a parent element that contains other information that relates to the applications that will run under this instance. In effect, it is pointing to the ADF that you will also need to create prior to importing the instance using the ICF.

If you try to import an ICF before the ADF is complete, you will get an error.

This section includes the name of the application and its location and name on the hard drive. You can store these files in a shared folder, as long as the SQL Server and service accounts have access to it.

DBA 101: XML Hierarchies

If you read down the file until you see the "opening" tag of <Applications> and the "closing" tag </Applications>, you will see that there are other items between them. In XML, this is how you nest items to make them "children" of other elements.

In this example, I create only one application for the instance, but you can have as many as the resources for your system allows. It is best to plan out the entire system before you begin this process so that the instances are broken out logically to contain the applications that make the most sense to group together.

The first child element in the <Applications> tag is <Application>, which is another parent tag. You repeat this element for each application that the instance will host. Within that parent element is the <ApplicationName> element, which sets the name of that particular application.

Each application needs a directory to store its ADF. This file, which I create in a moment, holds the same kind of information created for the instance but sets up the application.

The next element, <ApplicationDefinitionFilePath>, sets the name of the ADF file. If you store only the filename here, the file needs to be stored directly in the BaseDirectoryPath location.

The reason you have two elements available is because in actual production, you will normally have several applications that run on a single instance. You want to keep these applications separate so that various developers can have the access they need to work with only the applications they are assigned to. Keeping the applications in separate directories allows you a greater level of security.

<! Delivery Channels >

The instance controls how the system communicates with the outside world. The first parent element, called <DeliveryChannels>, begins the section describing all the ways that this instance can communicate.

You open the delivery method with the <DeliveryChannel> element, and then type in the <DeliveryChannelName> element to have the type you want, from e-mail to file. Within that instruction, you need to specify the protocol the channel will use. You do that with the value of the <ProtocolName> element.

There are lots of ways to send information, each with its own protocol settings. So that we can maintain the overview process here, I will direct you to Books Online using the topic search I mentioned earlier to learn about the various methods you have available.

With all of the basic elements defined, we need to create the ADF.

Applications and the Application Definition File

One or more applications are contained within a single instance of Notification Services. When you create an application, you will also create or specify a database that stores the events, the subscriptions, notification data, and other application metadata about the application. If you do not have a database, you can have the XML file (or the NMO program) create one for you.

Once again, we will look at a basic XML file and examine the elements it contains:

<?xml version="1.0" encoding="utf-8" ?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.microsoft.com/MicrosoftNotificationServices
/ApplicationDefinitionFileSchema">

 <! Version >
 <! History >
 <! Database Definition >
 <! Event Classes >

 <! Subscription Classes >
 <SubscriptionClasses></SubscriptionClasses>

 <! Notification Classes >
 <NotificationClasses></NotificationClasses>

 <! Event Providers >

 <! Generator Settings >
 <Generator>
  <SystemName>%SystemName%</SystemName>
 </Generator>

 <! Distributor Settings >
 <Distributors>
  <Distributor>
   <SystemName>%SystemName%</SystemName>
  </Distributor>
 </Distributors>
 <! Application Execution Settings >
 <! Important: At minimum, you should define
  a vacuuming schedule and turn off some or all
  distributor logging. >

</Application>

This file has quite a few subelements, so once again I recommend that you look up the topic in Books Online using the search topic "Application Definition File Reference." There you will find all of these elements hyperlinked, in alphabetic order. Using this minimal template and those search results, you can quickly build the file you want.

In this minimal file, the first four lines set the header for the XML file and tell SQL Server 2005 how to handle it.

<! Version >

The first section here is the <Version> element. Within that are various child elements that set a version number and build. The <Version> element is optional, but I normally include it so that I can track where I am in the build process. It is a good habit to get into.

<! History >

The History section is similar to the Version section, in that it provides tracking data for your system. You can include data here to track what changes you have made to the ADF.

<! Database Definition >

In the <Database Definition> elements, you set the name, schema, and physical structures for the application. If you do not provide one, the system creates a database using the defaults for the server. In a moment, I create a Notification Services application and show you the types of things that end up in this database.

<! Event Classes >

It is in the Event Classes section that things really become interesting. In this section, beginning with the <EventClasses> parent element, you set up one or more <EventClass> elements that define what events the system responds to.

<! Subscription Classes >

In the Subscription Classes section, using the parent tag of <SubscriptionClasses>, you set up one or more subscriptions for the application using <SubscriptionClass> elements. Here you describe the name, the schema for the subscription, any indexes you want on the tables, and event and schedule rules. All of these elements set up how often the events are captured and what can be subscribed to.

<! Notification Classes >

Using the <NotificationClasses> parent tag in this section, you set up one or more notifications for the application using <NotificationClass> elements. These elements define where your application stores the notifications, what filter is used to format them, and what protocols the transport uses. You can also set whether the notifications are sent one at a time or if they use a "digest," which groups them all together to be sent at the end of a specified period.

All of these settings are similar to what you will see in a newsgroup; and if you think about it that way, you will have the concept.

<! Event Providers >

As I explained earlier, an event is a change in data that you care about. For instance, a stock price change or a project change would be an event. Microsoft delivers several of these with SQL Server 2005, and you can also write your own. Here are a few of the event providers that your system can use.

Analysis Services | MDX statements
File Watcher | XML output, requires a schema document
SQL Server | Runs a SQL statement to detect changes; Can be non-Microsoft because it uses linked servers
Custom | Uses IEventProvider and IScheduledEventProvider

The most prevalent event that I have seen used is the SQL Server event. This allows you to watch a table for changes and deliver the information to a user via a subscription.

Each of the event types has specific elements they require to be able to process them.

<! Generator Settings >

The generator is the component within Notification Services that matches changes in data (events) to those who should learn about the changes (subscribers). It runs in the background on your server using the NSServer.exe program.

In this section, you use elements to define the name of the system that will act as the generator and set the amount of threads it will use.

<! Distributor Settings >

The distributor is the component within Notification Services that formats and sends the data. You can store the distributor on one or more servers to help balance the load on your system. You use elements in this section to set the name of the distributor, the threads it uses, and an optional duration setting.

<! Application Execution Settings >

In this section, you set up all of the information around the processing of your events, such as limiting the amount of time that is spent on a particular task as well as the order in which applications are processed. An important element to set is the <Vacuum> value, which sets how often the cleanup process runs.

With all of those components in mind, let's take a look at how all of it fits together within the framework of a complete application. I have a single application that I plan to set up to watch an inventory level. I have created the ICF and the ADF and placed them in a directory on my local test system. I will import those into my instance of SQL Server 2005.

Installing the Instance and Creating the Application

In Figure 7-1, I have opened SQL Server Management Studio and right-clicked the Notification Services object in the Object Explorer.

Figure 7-1.

[View full size image]

Within that panel, I specify the location of the ICF, as shown in Figure 7-2.

Figure 7-2.

[View full size image]

With that set, I click the OK button to process the file. My system shows the screen in Figure 7-3 while it processes.

Figure 7-3.

[View full size image]

Now I will enable the system by right-clicking it in the Object Explorer again. This process sets the distributors, generators, and subscriptions for the system. You can see that in Figure 7-4.

Figure 7-4.

[View full size image]

When that process completes, I right-click the service and select Tasks,… and then Register from the menu that appears, as shown in Figure 7-5.

This brings up a panel where I create the service, set the authorization for the service, and set how the service will access the database. You can see that in Figure 7-6.

I receive feedback that shows that the system is creating registry entries, creating the service, and adding new performance counters. With all that complete, I again right-click the name of the service and select Start from the menu that appears. After confirming that I want the system to start, the databases I specified in the ICF (AWInventoryNSMain) and the ADF (NSMetaData) are created, and the service starts.

Figure 7-5.

[View full size image]

Figure 7-6.

[View full size image]

Figure 7-7 shows a list of the tables within the instance database called AWInventoryNSMain.

Figure 7-7.

[View full size image]

This database contains the elements I explained earlier within the ICF document. There are a few other metadata tables here, especially involving the time settings such as time zone and locales.

The tables in the application database called NSMetaData also contain the elements I explained in the ADF and other metadata. You can see all of that in Figure 7-8.

The service reads these databases and accesses the information, ready for the client programs to subscribe, similar to what I explained for replication.

Figure 7-8.

[View full size image]

Security

There are two parts to the security of your Notification Services system. The first is to secure the locations where the ICFs and ADFs are located.

The reason you want these files secure is that those who can access them can change them to reflect more data in the Events section than you might want. The small files I described earlier are not representative of what you create in a production environment. The production files are much larger, so you might miss the addition of a new event or a change to a current one.

After you have your instance running, you can alter it later by re-importing the ICF and ADF. That is where the danger comes in on leaving these areas unsecured. If your developers are using NMOs to code the application rather than the control files, this is less of an issue.

The second part of securing the system is against the subscription data. To secure this data, you simply use Windows accounts or SQL Server accounts.

Monitoring and Performance Tuning

Notification Services can be used on servers that scale up, and they can also be used on a cluster. Another method for tuning the application is to spread it out onto multiple servers, including the generators and other parts of the application. That is why it is important to plan out the system ahead of time.

You can monitor a Notification Services system in SQL Server Management Studio by right-clicking the instance you are interested in and selecting the Properties item. Once inside, click the Applications item in the left pane, and in the Components for that application, check the Status column, as shown in Figure 7-9.

Figure 7-9.

[View full size image]

You can also check the Subscriber status from this resource box. Another monitoring tool is a suite of Performance Monitor objects and counters that you can access from the System Monitor in Windows, as you can see in Figure 7-10.

Figure 7-10.

[View full size image]

You can monitor all parts of the system based on what you are interested in examining, from the subscriber subsystem to the collection methods.


Service Broker

In replication, data is exported from one server to another, but it always stays in the format of a database. In Notification Services, data is delivered to subscribers, using phones, e-mail, or other formats. Although you can send data between systems using these technologies, they are often used in a unidirectional method, sending data from a server to another system or user. Earlier I explained that there are distinct advantages in using a disconnected system such as replication and Notification Services, which are in effect store-and-forward mechanisms. The SQL Server 2005 Service Broker extends this concept to the entire application.

With the Service Broker, you can build an entire infrastructure that is "loosely coupled." That means that the system does not receive immediate feedback from each application that the tasks are complete, just that the task has been received.

Microsoft uses the analogy of the Post Office to explain the Service Broker. For the DBA learning about an SOA, this is a great way to think about the process. I use that analogy to describe what happens at a physical Post Office and then extend it to show the parts of the Service Broker. In the "Take Away" section, I show you a simple Service Broker program example that you can extend to do useful work on your production servers.

Using the Post Office to Understand the Service Broker

Let's assume you want to place an order for a new computer through the mail. You write a letter, place it in an envelope with instructions for the final location (the address), and drop it into any receptacle that the Post Office recognizes as a pick-up point. The message is picked up and brought to the local Post Office, sorted, and sent to the next Post Office in the chain all the way down the line until it reaches the Post Office closest to the recipient. That Post Office delivers the message to the recipient, who opens the envelope and reads your instructions. They initiate several actions in multiple departments, and you receive a confirmation message and then a follow-on package from the company, which completes the transaction.

Most of us use this kind of system every day, without considering the advantages it provides. For one thing, it provides a distribution of function. You write the message and receive the goods, without any knowledge or concern about the route the message will take. The Post Office carrier picks up the envelope without knowledge of its contents or final destination. The Post Office sorts the message and routes it without knowledge of the message. The recipient reads the message without concern for its previous route or future destination. The process is then reversed. As long as everyone does his or her part, there is no need to have an active governing system that needs to know about every component, from the routes to the message content. In other words, one person does not have to follow the letter from beginning to end.

Another advantage is that a single message might cause a flurry of activity at the receiving end. If you order something through the mail, that triggers several departments at a company, such as purchasing, shipping, marketing, and so on. All with just one small order that you sent: you do not have to send an individual message to each department to have them start working on your order. In fact, each department is working on several orders at one time.

The Service Broker works the same way. Each component has a specific function, and items can be "dropped off" at each point with the guarantee that the component will perform its function having no knowledge or control of any of the other parts. Developers can write complex systems by coding their individual parts with the Service Broker and letting each work on multiple activities at once.

Let's take a look at each of the parts of the Service Broker as it applies to the Post Office analogy. Along the way I explain how the two relate and some of the information you will need to implement and maintain your own system.

Just as in the case of Notification Services, the Service Broker introduces entirely different uses for the same terminology used in replication and elsewhere in IT parlance. Watch this section for new definitions of words such as type and service.

The Postal System and the Service Broker

As a conceptual "umbrella" to the whole process is the idea of a postal system. Someone (in the United States, Benjamin Franklin) originally institutes and lays out the rules for getting messages from sender to receiver. In the case of the Service Broker, the SQL Server 2005 database engine works in terms of conversations, which are similar to a single-direction motion in the postal system. For instance, the postal carrier picks up your mail from your mailbox and carries it to the local Post Office. That is a conversation in Service Broker terminology.

Happily, the message in the Service Broker is the same as the message in the postal system, referring both to the enclosure (envelope) and the payload inside it. The only difficulty is making sure you define whether you are talking about the whole thing or just the payload inside. I try to make that clear as we move along.

In a simple example where the recipient is in the same town that you are, the message leaves the local Post Office and moves to the recipient's mailbox. That is another conversation. Taken together, the two conversations along with the message form what Microsoft calls a conversation group.

In T-SQL, you can examine the conversation group with the command GET CONVERSATION GROUP. You will often use this command to get the status of a particular part of the system.

When your letter gets to the Post Office, it is stored there until it moves on. In the Service Broker, this is handled by storage called the queue.

Once inside the Post Office, the letter moves from place to place to be properly sorted. Those internal processes that keep everything in the right place in the Service Broker are called dialogs. The Post Office stamps a "cancellation" across the message to show when it was received. If you are in the United States, April 15 (the day our tax forms are due) is one date you want to make sure you see on your envelope. Dialogs in the Service Broker do the same thing. They stamp each message with an order number so that they are guaranteed to be correct. This is important in transaction-type applications because one line in an order might change the one prior to it.

The Post Office has employed several people to do all the moving, sorting, and delivering. In the Service Broker, the employees are called services, and activations make them work. We discuss those further in a moment.

The Post Office actually handles more than just one type of letter. You might send a postcard, or letters of various sizes. You might send a package, too, and even specify special handling instructions. In the case of the Service Broker, this is referred to as a message type. Because applications can send everything from text to music files, each is treated a little differently. Most of the issues I have run into using the Service Broker involve a mismatch of types.

The letter is finally read by the recipient. In the Service Broker, the system sends an EndDialog message, which ends the conversation.

That is the broad overview of the physical layout; now let's drill in a little to the traffic flow from the sender to the receiver.

The MailMessage Types and Contents

The postal process starts with the mail message itself. Whenever you look at a piece of mail, it has at least two components: the envelope and the contents of the message inside. The envelope has a required size, it must have the addresses for both the sender and receiver in certain places and be readable, and it must have the right amount of postage. You form a contract between yourself and the Post Office that you will follow the rules and they will deliver the letter.

EnvelopeMessage Types

To send a letter, you have to subscribe to the rules they lay out for envelopes and postage. In the Service Broker, these are analogous to message types. A Service Broker message type has two basic values: XML and NONE.

In the XML type, SQL Server attempts to check the validity of the document prior to processing it. Anything that can use text can use the XML type. You can also specify that you want to process it with a particular XML schema document to verify the contents.

If you are sending anything other than text, such as a binary file, you can use the message type of NONE. That type instructs SQL Server not to check the validity of the contents.

The message type is specified when you send the message with the SEND and RECEIVE T-SQL commands, and you can create new ones with the CREATE MESSAGE TYPE command in T-SQL. I show you this process in the "Take Away" section at the end of the chapter.

Contents

You can send entire books in a single pass through the postal service, but that is not always a good idea for the Service Broker. You are working with a computer system that uses your regular network lines, so it is usually best to break down the messages into smaller pieces.

You create the message in your originating program, called the initiator. Most of the time the documents are passed around using XML files.

The SenderInitiators, Contracts

When you send mail, you are responsible for writing the message on some type of paper, for recording the address properly (both yours and the recipient's) on the envelope, and using proper postage. The postal service guarantees that if you follow those rules, they will deliver the letter. That is a contract, and in the Service Broker you have exactly the same type of arrangement.

The user or a program, called the initiator, enters the information they want to send or receive. The program runs a stored procedure, which creates a message and begins a conversation with the proper Service Broker service program, which I explain in a moment.

The Postal CarrierPhysical Transport, Protocols, and Remote Service Bindings

Although it is invisible much of the time, you are still using some method to move this data from client to server. In the Post Office, postal carriers drive vans or walk a route to pick up and deliver mail. In the Service Broker, physical transports (such as the network infrastructure) and protocols (such as TCP/IP) handle the movement from one location to another.

That might seem obvious, but it is important to consider it in the overall picture. If the Post Office ignored the fact that they use vehicles in their work, they might not perform the routine maintenance they need or plan for redundant vehicles. It is the same in the Service Broker. You need to be aware that there are physical parts to your system and plan for maintenance and backups in case they fail.

You can secure the entire communication process with certificates if you want. This provides a high level of encryption so that the message is difficult to break.

The Post OfficeEndpoints, Queues, and Activations

I have been explaining the Service Broker using Microsoft's postal service analogy, but I have moved in the direction of the client to the receiver. I have done that on purpose so that you can see how it all fits together, but in the postal service as well as the Service Broker, that is not how things really get built. Before a user ever sits down to write a letter, the Post Office has to be built first. That is the same way that it is in the Service Broker; you normally build the middle first and work out from there.

Each Post Office has a set of doors that only the postal carriers are allowed to use. In the Service Broker, this is called an endpoint. Endpoints are network connections directly into a service. The conversation groups I described earlier send messages between services using endpoints. Much of the DBA's time is spent creating and tracking endpoints.

In the Service Broker, the queue is similar to a single storage location at the Post Office. A queue is the data repository for all the messages. Unlike the Post Office, one message can be sent from a queue to multiple locations. I guess it is more like the Post Office than I care for; many people do get lots of the same junk mail.

In the Post Office, an employee is assigned to work on certain kinds of mail, in certain areas of the Post Office. In the Service Broker, each queue is associated with a stored procedure or a managed program in the CLR layer, called a service program. When a message is placed in the queue, the service program is activated and begins to process the message. Most of the work the developer does is within these stored procedures. In effect, these stored procedures are the workers in the Post Office.

This brings up another advantage in having Service Broker running directly within the database. This architecture removes the need for multiple services to be installed on the operating system. Because the queue is associated with a stored procedure, the stored procedure "wakes up" each time there is work to do, but not before. Nothing has to run in the background, constantly watching for work to do.

Another important part of this analogy is that whereas a postal worker might send the message to the final recipient, the worker might also send it to another postal worker for some additional processing. In the Service Broker, the stored procedures might do the same thing. Depending on what the stored procedure code is, the message might be processed and a final message sent or sent to yet another procedure for more processing.

The RecipientCalling Programs

When the processing is complete, the conversation is terminated. Your original calling program can either collect the status from the Service Broker or move on without checking.

Security

Unlike other database applications, the Service Broker is based on passing messages between systems. Because of that, you are not as concerned with individual user accounts. It is a bit more like the application security explained in Chapter 4, "Security," because a single account is used to access the system on the user's behalf.

In the Service Broker, you focus your security on two areas: dialog and transport. Dialogs have to do with the messages, and transport has to do with the network. Implementing both makes the system secure.

Dialog security is set up by creating a user in the application database and then a certificate for that user. From there, you can use a Service Binding mechanism to associate the user and the Service Broker. The application uses that user to send and receive data.

For transport security, you set up a server login and a certificate in the master database and then send the certificate to the calling programs and the DBA for the application. The system uses this login to process the transactions.

Monitoring and Performance Tuning

To manage and monitor the system, you get four new dynamic management views and three Performance Monitor objects and counters.

The dynamic management views are as follows:

sys.dm_broker_activated_tasks | Shows all of the stored procedure activations
sys.dm_broker_connections | Shows all of the Service Broker network connections
sys.dm_broker_forwarded_messages | Shows the messages that are in the process of forwarding
sys.dm_broker_queue_monitors | Shows the queue monitor that manages activation for a queue

To display the Performance Monitor values, add the following objects and counters to your trace.

SQL Server, Broker Activation Object | Shows stored procedure activations
SQL Server, Broker Statistics Object | Shows general Service Broker information
SQL Server, Broker / DBM Transport Object | Shows the interaction between the Service Broker and database mirroring

The two most important tuning strategies for the Service Broker are planning the layout and optimizing the stored procedures that act as the service programs in the application. Proper planning for the infrastructure is normally why you have implemented the Service Broker to begin with. When you analyze a solution, always ask whether an asynchronous architecture is possible or preferable, and then implement the layout accordingly.

Like most applications, the biggest bang for the effort is in tuning the T-SQL code that runs within the stored procedures. One strategy that you might want to alert the developers about is the use of the WAITFOR statement. This instructs the stored procedure to wait for an event to occur prior to looping through the code. Adding this statement is important in a disconnected architecture because the message might not be complete when the service program is activated.


Take Away

Both Notification Services and the Service Broker are more about the programming constructs than administration. Other than the security implications and a few more Performance Monitor objects and counters, there is not a lot for the DBA to do in the system from day to day.

I thought it might be useful to work through a simple example of a Service Broker application because it is the one that is the most accessible from standard T-SQL statements. Creating your own application helps you to understand the process, and you can even use this simple program as a starting point for a more advanced application.

I will keep the comments down to what I think you should know along the way, because seeing the entire application process laid out is useful to understanding the process quickly. You should be familiar with the terms I explained in the chapter to follow along. If you are interested in extending this system, create the application on your test system just as I have here. Then review the T-SQL statements that make up the process to expand what the system can do.

Service Broker Example

In this section, I create a simple example of adding an employee to a database. The requirement is for a junior human resources worker to be able to add an entry into the company's employee system. Because her manager wants to review the entry prior to the employee receiving a permanent ID number, we have decided to create an application that takes her entry and places it in a "holding" table until it is reviewed.

On my test system, I created a database called ServiceBrokerExample that has one table called Employee. That table has one column called EmployeeInfo, with an xml column setting, because that is what my application expects. Here is the code for all that in case you want to try it on your system:

 Create the database
CREATE DATABASE ServiceBrokerExample
GO

 Create the table
USE [ServiceBrokerExample]
GO
CREATE TABLE [dbo].[Employee](
      [EmployeeInfo] [xml] NULL
) ON [PRIMARY]
GO

With that all set, I begin the process by creating a message type for the system. I instruct the type to check to make sure my data is in proper XML format, but in production you will also often reference a full XML schema document:

 Create the Message Type
CREATE MESSAGE TYPE
 [ServiceBroker/Example/Employee/AddEmployee]
 VALIDATION = WELL_FORMED_XML
GO

I then create a contract for the system, for each party to use. You can see that it uses the AddEmployee message type that I just created:

 Create the Contract
CREATE CONTRACT
 [ServiceBroker/Example/Employee/AddEmployeeContract]
 ([ServiceBroker/Example/Employee/AddEmployee]
 SENT BY INITIATOR
 )
GO

Here are my "postal workers" that read the data from the queue and insert it into the database. I am using an XML conversion function to move the data along and place it into the database. There are two stored procedures here: one to do the inserts, and the other to check and empty the queue. Do not let the complexity stop you; read through each section line by line to see what's happening here:

 Create the Insert SP
CREATE PROCEDURE [dbo].[AddEmployee]
      @MB xml
AS
INSERT INTO Employee(EmployeeInfo)
VALUES (@MB)
GO

And now the stored procedure that reads the queue:

 Create the Update SP
CREATE PROCEDURE [dbo].[ProcessEmployee]
AS
BEGIN
 BEGIN TRAN
  DECLARE @CH uniqueidentifier
  DECLARE @MB varbinary(max)
 dequeues the message
  WAITFOR
  (
  RECEIVE TOP(1) @CH = conversation_handle, @MB =
message_body
  FROM EmployeeQueue
  ),
  TIMEOUT 1500
  process the message

  EXECUTE ProcessEmployee @MB
  END CONVERSATION @CH
COMMIT TRAN
END

With the workers in place, I need to set up the "Post Office"the queue that will store all the data. When I create the queue, I assign the service program (in this case, my stored procedure) that is assigned to process it:

 Create the Queue
CREATE QUEUE EmployeeQueue
 WITH STATUS = ON,
 ACTIVATION
 (
 PROCEDURE_NAME = ProcessEmployee,
 MAX_QUEUE_READERS = 5,
 EXECUTE AS SELF
 )
GO

I am almost there. Now I create the service that responds to the requests from the conversations; I will also tie that to the contract I created earlier:

 Create the Service
CREATE SERVICE AddEmployeeService
 ON QUEUE [EmployeeQueue]
 ([ServiceBroker/Example/Employee/AddEmployeeContract])
GO

The system is now ready, and I can examine all the objects using the SQL Server Management Studio. With the server ready for Service Broker conversations, I can set up a full sample event. I am only sending a snippet of the code I would really use as the XML document, but this snippet makes the code easier to read:

 Begin the dialog
DECLARE @CH uniqueidentifier
DECLARE @EmployeeName XML
SET @Employeename = '<name>Buck</name>'

BEGIN DIALOG CONVERSATION @CH
 FROM SERVICE AddEmployeeService
 TO SERVICE
'[ServiceBroker/Example/Employee/AddEmployeeService]'
 ON CONTRACT
[ServiceBroker/Example/Employee/AddEmployeeContract];

 SEND ON CONVERSATION @CH
    MESSAGE TYPE
[ServiceBroker/Example/Employee/AddEmployee] (@EmployeeName)
GO

To check the results, I query the dynamic management views I mentioned earlier, as well as the destination table:

 Look at the results
SELECT * FROM sys.dm_broker_activated_tasks
SELECT * FROM sys.dm_broker_connections
SELECT * FROM sys.dm_broker_forwarded_messages
SELECT * FROM sys.dm_broker_queue_monitors
GO

SELECT *
FROM
Employee
GO


Chapter 8. Integration Services

It seems strange that the minute you get data into a database, you are deluged with requests to take it out and put it into another one. It is also common to import data from other databases, files, and systems into your databases. Microsoft's SQL Server Integration Services (SSIS) serves as the primary engine and design environment for everything from simple imports and exports to a full-fledged extract, transform, and load (ETL) platform for the most demanding business intelligence (BI) landscape.

However, it is not just about moving data back and forth. SSIS has an intuitive graphical design environment that you can use to create powerful programs that perform maintenance on your databases, talk to other systems such as FTP servers or Web services, read and write XML files, or even all of that in the same package.

In this chapter, I explain how you navigate this powerful new feature in SQL Server 2005. I explain the architecture and the tools to create and manage solutions and then take you down the path of the ETL features that you have in the product. I use a lot of tables and lists for these explanations because I want you to see the breadth of what you have available as a toolset. I encourage you to do further research on the items I show you in the information you will find in this chapter. SSIS is one of those areas that have a lot of "corners" to explore.

Because this tool replaces Data Transformation Services (DTS) in previous versions of SQL Server, I explain what you need to know to make the transition in the "Take Away" section at the end of the chapter. I cover a little more information about upgrading a DTS package in Appendix , "Upgrading to SQL Server 2005."

One of the primary tasks of a database is to move data into and out of the system. Other than the data-entry screens of an application, you have several tools in SQL Server 2005 available to do that, such as the bulk-copy program (bcp) or by running a query from the sqlcmd command-line utility with an output flag. But these tools are not really suited for complex or fine-grained control when you import and export data, and they do not allow the data to be changed in any meaningful way during their operation. These tools also are not well suited to data migrations when you need to import or export a table from the database between another database or nondatabase format.

SSIS is a tool that has a simple wizard-driven interface for quick imports and exports and a powerful design environment and development model capable of serving as the ETL function of a full BI implementation. That is a big requirement, and SSIS handles it by providing various objects that you can connect to make powerful workflows for your system.

Integration Services is an engine, a service, a development environment, and a programming application interface designed to move data through a workflow that you design. It is the data movement and manipulation engine of SQL Server 2005 that you can use in simple and complex applications. In general, it is an ETL system.

Most ETL systems focus only on moving data, usually through a staging area. Using database tables, the data is stored "raw" prior to its transformation. Staging tables have open fields that allow any ASCII characters of almost any length. This allows the data to come over from the source system quickly, and the transform process deals with any problems once it reaches the ETL system. The source systems and the BI landscape are considered separate entities.

Microsoft takes a different view of this process.

Rather than bringing the data into a set of open tables, using SSIS you can transform the data into its final form directly within the extract process. This is called a pipelined procedure. Integration Services is set up to perform transforms with components built right in to the toolset and contains a complete coding model for any custom transforms along the way. That is not to say you cannot create a staging area, just that you should give serious thought to transforming as early in the process as you can. My experience has always been to transform as late in the process as possible, but the in-line transformation enhancements in Integration Services makes the pipelining process convenient. The real benefit is that you can do things either way.


Integration Services Architecture

Because other ETL tools view the migration of data from their sources to various levels of storage, they view the ETL process as a series of data paths. This colors the way the tools implement their design process. Most of the code and toolset they provide are slanted toward data manipulation only. They are also often geared toward complex data warehousing environments, so they have a steep learning curve.

Because Microsoft views the data as a single path, the toolset focuses not only on the data manipulation, but the extract process, the operating system, Web services, and more. Although it has many powerful BI features, it contains many wizards that simplify tasks such as copying a database or importing a text file to a table.

SSIS has two primary areas that you will work with: the control flow and the data flow. I explain these concepts as we explore the components of the system and then I demonstrate them fully in the section describing the Business Intelligence Development Studio, one of the tools you can use to work with Integration Services.

Integration Services Components

The components within SSIS include the outer objects such as packages, which hold other objects, and two primary objects to process and work with the data. The first is the control flow, which contains tasks you want to perform before, in between, or after the data is moved. The second is the data flow, which has connections, sources, transforms, and destinations to work with the data.

Within the design environment, you will see two sequential tabs with those objects as the title. You will actually create the control flow first, and it will contain at least one data flow object, even though you work with them on what appear to be equal tabs. We come back to that in a little while, but it is important to remember the distinction that a control flow contains data flow objects.

Packages

The first object we will explore is the package. This is the collection of the control flow and data flow objects in a single unit. The problem with this term is that it was also used in the previous version of SQL Server to refer to the data movement tool there, called Data Transformation Services (DTS). To confuse matters even further, when you design a SSIS system, the largest container is called a solution, which is made up of one or more projects. Projects contain packages.

You can store a package in a SQL Server database, the SSIS Package Store, or even as a file.

To help separate these terms, keep in mind that solutions and projects are part of the design effort; an SSIS package runs on a server and is separate from a DTS package. I will make sure I am clear when I use these terms as I explain the SSIS objects. For now, think of the package as the outermost container of the rest of the objects.

Control Flow

The control flow view of the process involves the tasks performed before the data is imported or exported and the tasks performed after the transfer occurs. Although you can work with data in this process, it is normally done in another area.

Available tasks in this area include downloading a file from a foreign system using File Transfer Protocol (FTP), sending a notification that a transfer process is starting, or making a database call to trigger a process on another server. You can also create tasks that link into an Analysis Services system to prepare a transfer there. As we work through the various scenarios, I explain more about these components.

There are three types of objects within the control flow. The first are the containers, which hold units of work. Containers deal with loops and sequences. If you want to iterate through a series of tasks, you can create either a Foreach loop, which works through a defined number of steps, or a For loop, which works through a series of steps until some condition is true or false. Both of these are types of containers.

Another container is the sequence. This container provides order to the processes it contains and can even act like an entire package within a package.

The second type of object in the control flow is the task. Tasks are the individual units of work for the control flow. Remember, the control flow contains the work your package does around the data movements.

Tasks include items such as scripting, Analysis Services, XML manipulations, downloads, and more. Tasks are the building blocks of your package. By putting tasks together, you can create complex operations that are easy to follow because they are represented graphically.

One of the most important tasks is also the second major component within SSIS: the data flow task. This task moves data from one place to another. I cover the data flow task separately because it has its own types of tasks within it.

You can create one or more tasks within a single control flow. That means you could load data from one location, receive a downloaded XML file, read it in, and transform all that data into a final form.

Here are the control flow tasks you have available and a quick synopsis of what they do.

[Pages 404 - 405]

Task Name | Task Type | Short Description
File System task | Data Preparation | Reads, copies, deletes, and moves directories and files.
FTP task | Data Preparation | Upload and download files on an FTP server, also create and delete directories.
Web Service task | Data Preparation | Executes Web service methods, such as connecting to an HTTP server to query a database.
XML task | Data Preparation | Reads, compares, merges, and validates XML documents. Can also reformat XML documents or run an XPath query against them.
Execute Package task | Workflow | Runs another Package as part of the workflow.
Execute DTS 2000 Package task | Workflow | Runs a SQL Server 2000 DTS package as part of the workflow.
Execute Process task | Workflow | Runs an application or batch file.
Message Queue task | Workflow | Send and receive messages between SSIS and Microsoft Messaging Queue (MSMQ).
Send Mail task | Workflow | Sends an e-mail, with or without attachments.
WMI Data Reader task | Workflow | Runs Windows Management Instrumentation (WMI) Query Language statements (WQL) that allow you to query the operating system on everything from logins to processors.
WMI Event Watcher task | Workflow | Runs WQL statements to determine the status of an object in the operating system.
Bulk Insert task | SQL Server | Copies data into a SQL Server table or viewrequires that the SQL Server database allows these types of inserts since they bypass the transaction log.
Execute SQL task | SQL Server | Runs a set of SQL statements using OLEDB, ODBC, ADO, ADO.NET, SQLMOBILE, or even Excel connections on any database that supports those connection methods.
Transfer Database task | SQL Server | Copies or moves an entire SQL Server database from one server to another.
Transfer Error Messages task | SQL Server | Transfers custom error messages from one SQL Server to another.
Transfer Jobs task | SQL Server | Transfers SQL Agent jobs from one SQL Server to another.
Transfer Logins task | SQL Server | Transfers SQL logins (other than sa) from one SQL Server to another.
Transfer Master Stored Procedures task | SQL Server | Transfers user-created stored procedures from the master database of one SQL Server to another.
Transfer SQL Server Objects task | SQL Server | Copies objects such as tables, views, or even schemas from one SQL Server to another.
ActiveX Script task | Scripting | Runs ActiveX scriptsthe Script task is preferred because they can be precompiled.
Script task | Scripting | Runs scripts created in the Visual Studio for Applications (VSA) environmentprovides full programmatic extensions to the package.
Analysis Services Processing task | Analysis Services | Runs Analysis Services processing tasks against cubes, dimensions, and mining models.
Analysis Services Execute DDL task | Analysis Services | Allows you to create, alter, or drop cubes, mining models, and other multidimensional objects.
Data Mining Query task | Analysis Services | Runs Data Mining Extensions (DMX) prediction queries based on data mining models.
Back Up Database task | Maintenance | Backs up a SQL Server database.
Check Database Integrity task | Maintenance | Checks the integrity of one or more SQL Server databases.
Execute SQL Server Agent Job task | Maintenance | Runs a SQL Server Agent job.
Execute T-SQL Statement task | Maintenance | Runs a T-SQL statement (SQL Server only).
History Cleanup task | Maintenance | Deletes entries in the msdb history tables for things such as backup and restore operations.
Notify Operator task | Maintenance | Sends notifications to a SQL Server Agent operator.
Rebuild Index task | Maintenance | Rebuilds indexes on SQL Server tables and views.
Reorganize Index task | Maintenance | Reorganizes indexes on SQL Server tables and views.
Shrink Database task | Maintenance | Shrinks a SQL Server database.
Update Statistics task | Maintenance | Updates the statistics on a table or indexed view.
Custom tasks | Maintenance | Allows you to create your own task type using any .NET language that supports COM.

The third type of objects are the precedence constraints. They connect containers and tasks in packages into an ordered control flow. Using precedence constraints, you can control the sequence execution for tasks and containers and specify conditions that determine whether tasks and containers run.

Data Flow

Inside the control flow (although viewed separately in the design environment) is the data flow. The data flow contains connections to data systems and three basic objects and tasks: sources, transformations, and destinations.

Each source object represents various places that you can pull data. You can read from databases, files, Web services, Excel files, XML documents, and more.

You can also apply transformation tasks, which include things such as aggregations, merges, splits, various mathematical and string manipulations, and more. You can place these tasks directly after the source system objects to create the pipeline architecture, or after data arrives at the destination. You can have as many transformation tasks as you want.

When you are finished importing and transforming the data, you place it into a destination. The destination does not have to be a SQL Server database. You can read data from an FTP site, transform it into all lowercase, and then record the data out into an XML file. The data never has to pass through a SQL Server database at all.

The Connection Manager

Before you can read data from a source or write it to a destination, you must have a connection to work with. Connections involve the libraries that talk across protocols to systems and the security authentication (if any) that those connections require.

If you have never programmed before, this might be a new concept, because it seems unnecessary to tell the system you want to extract data and also that you want to connect to that data. In practice, connections are useful because you can reuse them. Not only that, but connections need to be abstracted from the data transfer because you might want to make two connections to a system with different security or configuration information but perform the same data operations on both.

There are a lot of data sources you can access, right out of the box. Here is a quick list:

  • ADO

  • ADO.NET

  • Analysis Services

  • Excel

  • File

  • Flat file

  • FTP

  • HTTP

  • Microsoft .NET Data Provider for mySAP Business Suite

  • MSMQ

  • Multiple files

  • Multiple flat files

  • ODBC

  • OLE DB

  • SMO

  • SMTP

  • SQL Server Mobile

  • WMI

In the following sections, I show you how to create connections for both source and destination data.

Sources

With the connection created, SSIS can read from several types of data sources. Here is a chart of the types you have available.

Source | Short Description
DataReader | Reads data from a .NET Framework data provider
Excel | Reads data from an Excel worksheet
Flat file | Reads data from an ASCII file
OLE DB | Reads data from an OLE DB connection
Raw file source | Reads bytes from a file
Script component | Uses a script to extract, transform, or load data
XML Source | Reads and shreds data from an XML document

If you are connecting to a SQL Server database, you will most often use the OLE DB data source.

In a moment, I put all this together in a simple package so that you can see the interactions between the connections and the data sources.

Transforms

When you bring data from another system into a database, you will often need to change it. Most often, it is a data-type issue.

For instance, when you are reading from a flat file, the data is necessarily all text. There is no notion of dates, numbers, or binary representation because the ASCII set does not support a "type" attribute. When you put the data into a database, however, it will have a type. In some cases, the database can just "figure out" what to do, such as importing the string "10/23/1963" as a date. That is called an implicit conversion, and other than the convenience factor, it is not always a good thing. The system might implicitly convert a telephone number string into a numeric format, and subtract the phone number from the area code.

To prevent this kind of thing, you should at least implicitly transform the data types from the source system to the destination. In other cases, you might want to completely change the string, even if the data types match. In applications such as BI systems, you might have a need to change the case of text or add ten days to a date.

So far, I have been describing changing the contents of a single field within the extract. In more complex applications, you need to do much more than that. SSIS includes several types of transformation operations, covering everything from the single-field changes to creating fields that do not even exist in the source system.

I am describing these transformations so that you can plan your ETL process. Even if you are not setting up a complex BI system, you still need to have a plan for the data path. The reason for that plan is that you do not own the data. The minute you change the data from what it originally was in the source system you have added meaning, which is not something the DBA should do. I explain that concept in greater depth in Chapter 10, "Analysis Services," but for now you should ensure that you present your plan to change data to the data owners for their sign-off. Only the owners can set the meaning for the data.

Let's take a look at some of the more common types of transformations you are likely to use and then compare that to the transformation tasks available in SSIS.

Homogenized

One of the most common transformations is to homogenize data. That simply means making the data from multiple sources look the same by changing case, order, data types, and so forth.

Assume that you have three source systems at various plants. One might represent part numbers separated by dashes, another might use forward slashes, and still another might use periods to separate the numbers. When you bring that data in, you may want to strip out all of those characters and simply run the numbers together. That is a homogenization transformation.

Cleansing

Another more difficult transformation is cleansing. If you import much data, you will certainly encounter sources that have allowed "dirty" data into their systems. This can include foreign characters, strings where numbers are required, or any number of data corruptions. In this case, a judicious review of what your goals are is essential. Do you ignore the rows with the bad data? If you do, there is a possibility you will violate the integrity of the system you are designing. For instance, if you delete a "bad" part number from an import, you might not be able to import the larger assembly that depends on it.

In addition to ignoring the data, which really is not a transform anyway, you could set up a logical rule that changes the bad data to an acceptable form. This is possible if you know what the possible errors are and what they should map to, but this is not always possible.

The last choice you have if you cannot cleanse the data is to write it to an "exceptions table" or to a logging file. Although technically a transform, you should at least employ some sort of detection logic so that you do not stop an entire import process because of a single row of bad data.

Aggregating

An aggregation transformation is used with numeric values. In this case, you add numbers or counts of data elements and store the result into a single field. For instance, instead of recording every part from the source system, you might record that there were ten part number 12345s, five number 34215s, and so on.

Aggregations are common in analytical systems because by definition they are more concerned with strategic numbers than the detail.

Merging

A merging transformation type is used when you need to combine fields. You may find that your source systems have broken a single field into multiple parts. This is quite common in project management systems, where the Work Breakdown Structure (WBS) numbers are used as a "database in a database," meaning that each character in the number has positional relevance. In that case, you may want to combine the fields into one field.

Conversion

In global systems, you may need to convert data from one language or format to another. This might be as simple as changing one character to another or as complex as looking up a currency rate on a Web service and changing the inputs to dollars or euros.

Joining

Much like the merge transformation, joining transformations involves combining fields. The difference is that a joining transformation adds two or more fields together or even appends data to an existing field and does not make any other changes to it.

Materialized

One of the most interesting transformations is when you create a new field from the data that is imported. This is called a materialized transformation. Possible uses include creating totals by performing mathematical operations on merged or joined columns or adding a location field to the import based on where it came from.

Most every BI extract includes this type of transform because they include a timestamp field on the import. Because that data was not in the original file, it is a materialized transform.

SSIS Transforms

You can achieve the types of transformations I have just described by using one of the transformations that SQL Server provides or by combining the output of one to the input of another.

It is beyond the scope of this chapter to describe each of the transformations and how you can use them, but here is a chart that gives you a quick rundown of the variety of capabilities SSIS transformations provides.

Transformation Name | Short Description
Aggregate transformation | Creates aggregations
Audit transformation | Creates information about the environment available in a package
Character Map transformation | Applies string functions to character data
Conditional Split transformation | Routes rows to different outputs
Copy Column transformation | Adds copies of input columns to the output (useful for combinations)
Data Conversion transformation | Converts data types
Data Mining Query | Runs data mining prediction queries transformation
Derived Column transformation | Creates new columns with the results of expressions
Export Column transformation | Inserts data into a file
Fuzzy Grouping transformation | Standardizes column of data
Fuzzy Lookup transformation | Uses lookup tables and creates a fuzzy match
Import Column transformation | Reads data from a file
Lookup transformation | Uses a lookup table by an exact match
Merge Join transformation | Uses FULL, LEFT, or INNER joins to combine data
Merge transformation | Merges data
Multicast transformation | Distributes data to multiple outputs
OLE DB Command transformation | Runs SQL commands for each row in an import
Percentage Sampling transformation | Creates a sample set using a percentage specifying a sample
Pivot transformation | Flips a data set on its axis
Row Count transformation | Counts rows as they pass and stores the count in a variable
Row Sampling transformation | Creates a sample set by specifying the number of rows in a sample
Script Component | Uses scripts to extract, transform, or load data
Slowly Changing Dimension transformation | Configures a slowly changing dimension's updates
Sort transformation | Sorts data
Term Extraction transformation | Extracts terms from text
Term Lookup transformation | Uses lookup tables and counts terms extracted from text
Union All transformation | Merges all data
Unpivot transformation | Unflips a data set on its axis

Destinations

The destinations for the data are similar to what I explained for sources, but SSIS includes a few more destinations than sources. The reason is that SSIS is used a great deal for BI applications, so you will see destinations for dimensions, data models, and partitions. I cover those objects more in Chapter 10. Here is a list to help you with the type of destinations you have available:

  • Data mining model training

  • DataReader

  • Dimension processing

  • Excel

  • Flat file

  • OLE DB

  • Partition processing

  • Raw file

  • Recordset

  • Script component

  • SQL Server Mobile

  • SQL Server

Precedence Constraints

To connect containers to tasks or tasks to other tasks, you use the next object in the control flow, precedence constraints. Precedence constraints are conditions of failure, success, completion, or any mix of all three.

Essentially, you "tie" a container or task to another container or task. In this way, you can create your logic for the workflow, such as "Do A if this task succeeds, B if it fails, or C when it completes." By using the proper constraint precedence arrangements, you can create AND and OR conditions on a task. This creates the flow for the package.

You have a few choices for the precedence constraints. Most of the time, you will use the execution results from the task to enforce the constraint. For instance, to enforce the following workflow: "If the execute T-SQL command is successful, move on to the next task, but if it fails, send an e-mail," you would create two precedence constraints: one for the success tied to the next task, and the other for the failure tied to a Send Mail task.

You can also code your precedence constraints to use an expression evaluation. This would mean something like "If A = B, then True" to evaluate whether to branch off to one task over another. The variables I discuss in a moment are useful for this type of workflow requirement.

Precedence constraints attach to either an input or an output of another object. Although you might think a destination would not have an output, it does. Many objects, such as destinations, have error outputs.

Not all inputs can be linked to all inputs, but don't worry: The interface will stop you if you try to attach to the wrong place.

Event Handlers

Another concept that is borrowed from programming that you may not be familiar with is the event. Conceptually, the idea of an event is not hard to follow. When something happens, that is an event. The more interesting part is that certain SSIS objects can "raise" events, which you can watch for and respond to. You can watch events from a package, a task, or a container.

To watch and respond to the events, you create an event handler. Here are the types of event handlers and what they watch for.

Event Handler Name | Watches For
OnError | An error in execution
OnExecStatusChanged | Execution status changes in an object
OnInformation | Information during execution, but not errors
OnPostExecute | An execution's completion step
OnPostValidate | An execution's validation step
OnPreExecute | Just before an execution step
OnPreValidate | Just after the validation starts in an execution step
OnProgress | The progress of an execution step, if available
OnQueryCancel | The cancellation of a query
OnTaskFailed | Task failure
OnVariableValueChanged | Changes in a variable
OnWarning | Warnings during execution, but not errors

Event handlers are great for debugging your process, too. You can watch for status changes, errors, and so on when the package is running, and when you have got everything worked out, you can remove them.

Other Components

Even with all those objects, there is still more that you can do with SSIS. To really extend the Package into an enterprise process, you need two other facets: logging and extensibility. I cover the logging options in the "Management" section a little later, because it allows you to maintain your system as well as develop against it.

Variables

Another handy concept borrowed from programming is the variable. A variable is a name set aside to hold a value that can change while the package is running. It is similar to a notepad off to the side that you can write on and read while your workflow completes.

Variables are stored in a central location, called a namespace. SSIS has some built-in variables and stores them in the namespace called System Variables, which you can also read. The variables you create for yourself are normally stored in a namespace called User Variables. In this section, I focus on the user variables, but do not overlook the valuable information that the system variables hold. I show you how to get to those later.

When you create a variable, it applies to only the area that contains it. For instance, if you create a variable in a task, only that task and any objects contained within it can see or change the variable. This is called scope.

If you create a variable in a container, all the objects within that container can reference the variable. Creating a variable in the package essentially creates a "global" variable that all objects can see. Although my developer friends cringe when they see me do it, I normally create variables with the scope in their name.

To set a variable, you can right-click an object and select that option from the menu that appears. You can also work with variables in the Expressions property page in the Business Intelligence Development Studio, which I describe next.

Tools

Because the SSIS feature has so many facets, you can use more than one tool to work with it. You can use several command-line tools to manage and deploy packages. SSIS also includes a full programming model, so your developers can create entire applications using only code. In this chapter, I focus on the tools that you will be asked to use as the DBA.

Creating a Project with the Business Intelligence Development Studio

To design SSIS packages, you will use the Business Intelligence Development Studio. This is a Visual Studio-inspired application that has an integrated graphical design interface, a debugger, context-sensitive help, and even source control.

You will find the Business Intelligence Development Studio in the Start menu in the SQL Server 2005 item. In this section, I open the interface and explain the environment. You can follow along on your own testing system.

You have already seen much of this interface when we explored the SQL Server Management Studio, so I do not cover things such as the pushpins on the menus or the panel placements. Those work here the same as in the SQL Server Management Studio. There are, however, a few other items that are unique to a design environment.

In this section, I am not creating a real project; we are just exploring the interface. While you read through this section, watch for all the objects I explained earlier in the chapter.

When you first open the Business Intelligence Development Studio (which I call the Development Studio from here on), you are placed in the Start page, which is actually in the main section of the environment, similar to the Summary area in SQL Server Management Studio. You can see the screen layout in Figure 8-1.

Figure 8-1.

[View full size image]

This HTML-enabled area has hyperlinked items you can click, showing the latest news from the Microsoft development world and introductions to various parts of the tool. One of the main panels within this screen allows you to create various projects, such as a Reporting application, a BI application, and an Integration Services application. Click the Create: Project link on the Start page to bring up the dialog shown in Figure 8-2.

Figure 8-2.

[View full size image]

Here you are able to set the directory for the project and create a subdirectory for the project automatically. This is a practice that I recommend, along with storing your projects in a central location.

After you make your selections, you are dropped right into the interface without connecting to a server first. This is because you are working in a development environment. You could create an entire project without connecting to a database at all.

In Figure 8-3, you can see the screen I am presented with on my test system.

Figure 8-3.

[View full size image]

If your screen does not look like this one, you can click the Window menu bar item and then select Reset Window Layout.

On the far-right side of the screen are the Solution Explorer and Properties panels, just as I showed you earlier when we spoke about the SQL Server Management Studio. In a moment, I show you how to add data sources and data source views and how you use those, but for now "unpin" those panels to hide them. Next slide out and "pin" the Toolbox on the left side of the screen.

As you can see in Figure 8-4, this is where having multiple monitors really comes in handy. With a large-enough monitor, you could set all of the menus to "Float," and then have a large central area to work with the package. In this case, just keep the menus in the tabbed interface.

Figure 8-4.

[View full size image]

The large area in the center-right of this screen has four tabs, containing the control flow, the data flow, event handlers, and the Package Explorer view. That is all encompassed inside one tab referenced by the name of the project. At the bottom of the screen is the Connection Manager area. This is where we will make the connections to the databases that are used by the other objects. On the left-hand panel is the Toolbox for the control flow items.

To begin the project, simply drag items from the Toolbox into the panel on the right that you are interested in working with. Because the control flow is what is showing here, drag a File System task into that part of the screen.

In Figure 8-5, you can see that there is an error on this task, indicated by the X in a red circle. It also has a precedent constraint highlighted. In this case, it is green, which means it is the success precedent. I come back to that in a moment.

Figure 8-5.

[View full size image]

Right-click the object and select the Properties item from the menu that appears. The tab on the right shown in Figure 8-6 slides out to show all the properties of the object and allows you to set or edit them.

Figure 8-6.

[View full size image]

Close that panel and double-click the File System Task object. You can also right-click an object and select Edit from the menu that appears to do the same thing. The results are shown in Figure 8-7.

Figure 8-7.

[View full size image]

In this case, the File System task expects information about what it needs to do. As I explained earlier, the File System task can copy, move, delete, or create files and folders on the operating system. This example I have created copies a file called test.txt in the c:\temp directory to a file called new.txt in the c:\temp directory (and overwrites it if it already exists). Most of those instructions are visible in this panel, but the directories are not.

The reason is that when you click the SourceConnection entry box to set the start file, you have to either specify a file connection or create a new one. Although I'm not showing that dialog box here, selecting the item marked <…new connection .. > brings up a panel with a browse button to set the path or filename. Once those are created, they show up in the bottom part of the environment and are available to any of the other objects you work with.

Click the OK button to close that item and create the next item in the workflow we are developing. In this case, we will send an e-mail that the process is complete. Drag the Send Mail task onto the control flow.

You can see the object in Figure 8-8. With the Send Mail task created, it must be configured for use. Double-click the object to bring up the editor for it, shown in Figure 8-9.

Figure 8-8.

[View full size image]

Figure 8-9.

[View full size image]

In this case, the Mail item on the left is selected to show that the system will attempt to mail a message containing the text of the file called new.txt from the file connection created a moment ago; notice how we are already reusing connections. Clicking the name of the SMTP Connection allows you to set up the SMTP server the task connects to. Click OK to close out the task.

Next we turn our attention to the precedence constraints. We need to tell the system to create the new.txt file and then send out the mail only if that is a successful event.

To set the constraint, click once on the File System task and then click once on the green arrow pointing downward from it. You do not have to hold the mouse button down; the line is now "locked" to your pointer until you click another object. Move the arrow until it connects to the Send Mail task, and then click once more. You can see the results in Figure 8-10.

Figure 8-10.

[View full size image]

Now double-click the green line, which brings up the dialog box shown in Figure 8-11.

Figure 8-11.

This is the Precedence Constraint Editor. Here you can set whether the step evaluates to a constraint such as success, failure, or completion; or whether the step evaluates an expression, which contains the variables I explained earlier in the chapter. You can also set the evaluation to both the constraint and the expression or the constraint or the expression. Selecting the Expression option opens up the Expression text box below. Click OK to continue.

If you want to add another precedence constraint, right-click the object and then select Add Precedence Constraint from the menu that appears.

Figure 8-12 shows the complete picture of the package as it stands now.

Figure 8-12.

[View full size image]

In that screen, you can see the two tasks I have created, the success precedent constraint, and the connections for the files and the mail system. Although this seems perfectly clear now, if you begin to add more File System tasks or Send Mail tasks, it will not be obvious what you are trying to accomplish in the package.

To add more meaning to the graphics, click the File System object in the text that is displayed, wait a moment, and then click it again. The text will highlight, which allows you to type a more meaningful title, as shown in Figure 8-13.

Figure 8-13.

[View full size image]

With everything complete, it is time to save the project. I normally click the "multiple disks" icon in the icon bar, but you can also use the menu bar to save your work. The project is now saved, the objects are all configured, and we are ready to run the package. In the Development Studio, this is known as debugging, and you can start it by pressing the F5 key or you can click Debug and then Start Debugging in the menu bar.

On my test system, the File System task worked properly and is colored green. The Send Mail task has failed, however, and is indicated with a red box, which you can see as the darker box in Figure 8-14.

Figure 8-14.

[View full size image]

To find out why it has failed, I click the Progress tab at the top of the package.

You can see in Figure 8-15 that my test system is behind a firewall that blocks SMTP traffic. Although the actual failure message text is not helpful in this case, the steps taken by the package are clearly diagrammed so that you can trace each activity within SSIS. This is useful when trying to find an issue with your package's execution. You will also get feedback on how long the process takes, which is valuable in performance monitoring.

Figure 8-15.

[View full size image]

Command-Line Utilities

The graphical tools are by far the easiest method to work with SSIS, but sometimes you will need to use the command line to access the system. In that case, you have four tools at your disposal.

Tool | Purpose
Dtutil | Manages SSIS packages
Dtexec | Executes SSIS packages
Dtexecui | Displays a graphical user interface to load and run SSIS packages
Dtswizard | Launches the Import and Export Wizard

Wizards

Although you have a powerful environment to work with for designing projects, you might only have modest data transport needs, such as moving a table to a file or reading data from an Excel spreadsheet into a database. You will be happy to learn that you do not have to learn an entire new development environment to do that. Microsoft provides a simple wizard-driven interface that walks you through the process to move data to or from your database. In fact, if you followed the Maintenance Wizard process I demonstrated earlier in the book, you have already used SSIS to set up a package. In your role as a DBA, you will often create a package using a wizard and then expand the package in the Business Intelligence Development Studio.

I do not show you every screen a Transfer Wizard uses, but I show you where the process starts and hit some of the highlights. Before I start, you should know that you can bring in data from other SQL Server databases, text files, and Microsoft Office applications such as Access or Excel. Using OLE DB providers, you also have access to Oracle, IBM's DB2, and even MySQL databases.

For this example, I export one table from the AdventureWorks database to an Excel file. I will work through the wizard and then save the Package so that I can add to it, using the concepts I explained in the Development Studio.

I begin the process by right-clicking the database name and selecting Tasks and then Export Data from the menu that appears, as shown in Figure 8-16.

Figure 8-16.

[View full size image]

I am presented with a splash screen, and click the Next button to continue. I am then asked for the source of the data. I select my test server, the SQL Native Client, and I will use Windows Authentication for the connection. I then set the database to AdventureWorks, as you can see in Figure 8-17. If you are also using this wizard as a starting point, you are not limited to your own server or even SQL Server 2005. You can select any source that your server is configured to talk to.

Figure 8-17.

[View full size image]

With those selections complete, I click the Next button to move on to the next panel. This panel looks similar to what I just set, with the exception that this is the output, not the input for the process. I will set my destination to an Excel file on my local hard drive, shown in Figure 8-18.

Figure 8-18.

[View full size image]

The next panel asks whether I want to send a whole table or view to the Excel spreadsheet or just a query. In this case, I am going to specify a query because I do not want to include all the information from the Contacts table in my database. I select Write a query to specify the data to transfer and click Next to continue.

The next panel presents an area for me to type my query. Here is what I entered:

SELECT
  [Title]
  ,[FirstName]
  ,[MiddleName]
  ,[LastName]
  ,[Suffix]
  ,[EmailAddress]
  ,[Phone]
FROM [AdventureWorks].[Person].[Contact]

And then I click Next to continue. The next panel that shows allows me to select or de-select the data to transfer and also gives me an opportunity to preview the data. I click Next to move on past that screen.

The next screen allows me to run the package immediately and offers the option to save the package to SQL Server or to a file. I deselect the Run Immediately option and select the option to save the package to a file.

I click Next to move to the last screen. I am immediately forced to save the package using a protection level. I cover protection levels in the next section, but for now I will save the package with a simple password. After I have entered that password, I click Next to move on to the Filename screen.

In the next screen, I am asked to name the package and then set its location. I do that, and then click Finish to complete the process. On the final screen, I am asked to review what I have selected. When I click Finish on that screen, I get a progress report of the process.

Your system may create the spreadsheet and fill it with data. To follow along with the next example, delete the spreadsheet before you continue.

When you create a package as I have here, you have two options for working with it. The first is that you can navigate to the file location and double-click the filename. When you do that, you are placed in the Execute Package Utility, which gives you a stripped-down control panel for the package. If you just need to run the package and perhaps set a few variables, this is the easiest way to do it.

The second method is to open a new solution in the Business Intelligence Development Studio and then add the file as a new SSIS package. That is what I have done in Figure 8-19.

Figure 8-19.

[View full size image]

From here I can add any variables, connections, and other tasks. In this screen, you can see that the control flow has two tasks and two connections, as you can see in Figure 8-20.

Figure 8-20.

[View full size image]

As I described earlier in the chapter, the data flow is shown as a task within the control flow. Double-clicking the Data Flow task moves to the Data Flow tab, shown in Figure 8-21.

Figure 8-21.

[View full size image]

We have already seen the methods for including tasks in the Control Flow area, and the Data Flow tasks are much the same, with properties, actions, and precedence constraints. What is new here is that there are source tasks and destination tasks. In this case, the wizard chose to use an OLEDB connection to get the SQL Server table data and an Excel connection for the spreadsheet output. It then created a DataReader source to use the OLEDB connection and an Excel destination to use the Excel connection.

Double-clicking the DataReader source brings up the panel shown in Figure 8-22.

Figure 8-22.

[View full size image]

Here you can view the connection this source uses, the type of data access, and then any parameters associated with that data access. Because the wizard created a SQL command access method, it requires a query, which is displayed here. The section just below the Connection Manager output on the left called Columns, which you can see in Figure 8-23, shows the columns available in this source. Pay close attention to these because they become the output that the Excel destination will take as input.

Figure 8-23.

[View full size image]

I close that panel and repeat the process for the Excel destination.

Figure 8-24 shows that the Excel workbook is treated as a database, and the data will go to a spreadsheet that is treated as a table. In this case, it is a spreadsheet called Query.

Figure 8-24.

[View full size image]

Just below the Connection Manager topic on the left side of the panel is the Mappings topic. This is where the outputs of the source are tied out to the inputs of this destination, which you can see in Figure 8-25.

Figure 8-25.

[View full size image]

So far, we have only examined the path the data takes. Now I want to transform the data to have a different value for the Suffix column in the spreadsheet. I plan to use that field in the spreadsheet to set the value of the source system's location as Tampa for this import.

To create a transform, you delete the connections between the objects, select the type of transformation you want from the toolbox, connect the output of the source to the transformation, and then output that transformation to the destination. Let's walk through this process step by step.

First, I click the green line connecting the two Data Flow tasks and delete it, as you can see in Figure 8-26.

Figure 8-26.

[View full size image]

Next, I drag the Derived Column transformation task onto the screen between the source and destination. This is the task I use to replace the Suffix field with the value of "Tampa." You can use the charts shown earlier in the chapter to find other types of transformations.

When the Derived Column object is on the screen, I drag the green line from the source to it and then drag the green line from the Derived Column object to the destination. In effect, I am tying the output of the source to the transformation and then the output of the transformation to the destination, as shown in Figure 8-27.

Figure 8-27.

[View full size image]

Now I double-click the Derived Column object. Once inside, I pull down the Derived Column selection in the bottom half of the screen and select Replace 'Suffix'. I give the column the same name, enter a value of "Tampa" (quotes are required for text values), and leave the data type at the default. You can see that in Figure 8-28.

Figure 8-28.

[View full size image]

There are a lot of other changes we can make in the previous panel, but I will move on so that you can see the process as a whole. The only step that is left to complete the transformation is to edit the Destination object and set the mappings. Because I did not change any of the names in the last step, it looks the same as before, as you can see in Figure 8-29.

Figure 8-29.

[View full size image]

I will now save the package and then run it by pressing F5. When the process completes, I open the spreadsheet and examine the results. I've highlighted the Suffix column in Figure 8-30.

Figure 8-30.

[View full size image]

You can explore a great deal more in this simple example, but it does demonstrate all of the concepts I have explained. Save this project. We will come back to it one more time to add some logging components.


Security

With all the data moving in and out of your system, security is a big concern. I explain the security within SSIS in this section; before I do, however, you need to be aware of some other security aspects.

When you create your import schema, it is tempting to focus on only the technical challenges and design solutions. But you should resist the urge to jump right in and begin coding. Make sure you follow the process of gathering the requirements, documenting the process, and then implementing the solution.

When you do that, you should include each source system and the data path. You might think that if you are pulling production data from the human resources system and then stripping out the names and address before you move it into your system that you are safe. But if you leave the data on an FTP server or Web service before you bring it over, you need to identify that location as a risk factor and protect it accordingly. In most of the systems I have consulted on, I have found major security breaches on the "staging" areas.

After you have the data within the destination, standard SQL Server security takes over. I have covered what you need to be concerned about in that area earlier in this book. Within SSIS, you can apply three roles to allow your developers access to create and manage packages.

Role | Rights
db_dtsadmin | Package administrator
db_dtsltduser | Execute only the SSIS packages the user has been given permission to
db_dtsoperator | Execute as well as backup and restore packages

You can also increase security by protecting the package when you create it, and even the objects it contains. Access the Properties panel of the object or package to set the following protection levels.

Protection | Short Description
Do not save sensitive | This level does not encrypt anything, but it prevents properties that are marked sensitive from being saved with the package. Other users have to add the blank data themselves.
Encrypt all with password | Encrypts the package using a password, and other users must supply to the password to access or run the package.
Encrypt all with user key | Encrypts the package using a key based on the current user profile, and only that profile can load or run the package.
Encrypt sensitive with password | Unlike the "Do not save sensitive" level, this level does save data marked sensitive, but the data is encrypted by using a password. If the password is provided, the package opens. If not, the package is opened without the sensitive data.
Encrypt sensitive with user key | Encrypts sensitive information using keys based on the current user profile.
Rely on server storage for encryption | If you save the package to the msdb database, the package is secured using SQL Server database roles.


Monitoring and Performance Tuning

Monitoring SSIS is different from monitoring other SQL Server activities because it involves a mix of what Microsoft provides and what you create. In addition, you might need to monitor parts of the source and destination systems to find any bottlenecks.

System Monitoring

You can use several Windows System Monitor objects and counters to monitor SSIS. The object you want to monitor is SQL Server:SSISPipeline. You can add the following counters to the System Monitor logs you create for your other monitoring.

Counter | Short Description
BLOB bytes read | Number of bytes of binary data read from all sources
BLOB bytes written | Number of bytes of binary data written to all destinations
BLOB files in use | Number of binary files the data flow engine is using to spool
Buffer memory | The amount of physical and virtual memory in use
Buffers in use | The number of buffer objects in use
Buffers spooled | The number of buffers written to the disk subsystem
Flat buffer memory | The total bytes of memory that the flat buffers use
Flat buffers in use | The number of flat buffers in use
Private buffer memory | The amount of memory in private buffers
Private buffers in use | The number of buffers that transformations use.
Rows read | The number of rows in the source
Rows written | The number of rows sent, but not necessarily written, to a destination

Package Logging

You should monitor the system using Windows System Monitor, but if you are looking for bottlenecks or problems, you will only get part of the picture with that tool. Because SSIS is a development tool, you will be responsible for creating your own logging.

You can go a bit overboard with logging if you are not careful. In complex systems, especially ones deployed to other servers, it is a worthwhile effort to use variables to set whether the system create more logs or fewer. That way, you can turn the logging on when you are looking for a problem and turn it off when you are in production.

Here are the logging providers you can use.

Provider | Where Written
SQL Profiler log provider | Log files that you can open in SQL Profiler
The SQL Server log provider | The sysdtslog90 table in a database
The Windows Event log provider | The Application Log in the Windows event log
The XML File log provider | An XML file
The Text File log provider | An ASCII text files using comma-separated value (CSV)

To create logging for the package from the preceding section, open the project you saved earlier and click SSIS from the menu bar, as shown in Figure 8-31.

Figure 8-31.

[View full size image]

Once inside, you must choose the type of log provider you want to work with. I am going to set this example to send the output to the Windows event logs because that is how I normally check the condition of my servers. I can then use filters to display the events I am interested in. I click the Add button to send this selection for the log locations. You can select as many logs as you like, but there is a slight impact to performance if you go overboard with the logging. You can see my selections in Figure 8-32.

Figure 8-32.

[View full size image]

Next I drill down to the tasks in this package. In Figure 8-33, I select both tasks and all their events. This is for demo only; do not select everything blindly in production. Only log what you need to know about on a consistent basis.

Figure 8-33.

[View full size image]

With the selections in place, I switch tabs to Providers and Logs and select the Windows event log provider I created a moment ago. You can create a different provider for each task if you want. You can see the results in Figure 8-34.

Figure 8-34.

[View full size image]

I click OK to close this panel and then run the package by pressing F5.

If your run fails, ensure that you have deleted the Excel file before you try it again.

When the process completes, I open the Application Log in the Windows Event Viewer and check for the SQLISPackage source. In this panel, I can see just how many rows were written in this operation. Figure 8-35 shows the event on my test system.

Figure 8-35.

[View full size image]

Performance Tuning

There really is not any magic to performance tuning: It is a combination of proper system sizing and configuration, asking only for the data you need, and optimizing the logic in your code. It is no different here than in any other application.

Good performance begins with the system's sizing and configuration that I explained in Chapter 1, "Installation and Configuration." You need to have adequate memory and a fast I/O system in combination with an efficient network connection and adequate processors for the system where the packages run.

Because your source and destination data may be located on another system, the network component deserves more scrutiny. The fastest, most efficient system will not help the overall speed of the system if the network connection is the bottleneck. Although you might not have the ability to add network capacity, your data flow plan should include a timing component so that you can send less data more often.

You should also work hard to ensure that you do not get data you do not need. In the example project I just created, the suffix is sent even though it is not needed. A better design is to send only the fields that are required from the source and create the derived columns on the destination.

The last component is to ensure that your logic on any transformations is as optimized as possible. There is no substitute for a peer review in this situation. Although it is sometimes painful to have others review your process, it is one of the only ways to ensure that you have explored all possible ways to implement the logic. Whether you are the expert in the office or the junior member on staff, it is still valuable to review your code.

You can take some concrete acts to optimize the system for the SSIS process. Ensure that all the tables you read from have as many covering indexes as the system can tolerate. Quick queries make quick sources. Ensure the destination tables have as few indexes as possible, or suspend them if you can. This is not always possible, but if you can it will save the updates on the indexes while the import is running.

If possible, use stored procedures when you are calling for data from SQL Server systems. Stored procedures can be compiled into the memory space on the server more efficiently than dynamic SQL statements.

I want to mention one final note on performance tuning for SSIS. In many situations, you can scale up solutions by using 64-bit processors. Although SSIS runs on 64-bit processors, the Itanium version does not include the Business Intelligence Development Studio, and it will not install on that platform. That is not normally an issue because it is more efficient to design projects on a workstation expressly used for that purpose, but it is something to keep in mind. The other issue with the 64-bit processors is to make sure that you select the proper providers for the platform. If you are not careful, you can find yourself using 32-bit providers on a 64-bit platform and thus negating the performance gains you thought you might get. Check the topic "Integration Services Considerations on 64-bit Computers" in Books Online for more information.


Take Away

SSIS is an amazingly powerful new tool to work with importing and exporting data into your SQL Server databases or Analysis Services system, but it is not Microsoft's first attempt at that process. In previous versions of SQL Server, Microsoft included a product called Data Transformation Services, or DTS. If SQL Server 2005 is the only version you have used, you do not have to worry about "unlearning" anything or changing your paradigm.

But if you have used DTS, you probably have a few packages lying around that you need to either upgrade or migrate to SSIS. In this section, I explain a few of the best practices and other items that you need to know about moving to SSIS from DTS.

SQL Server Integration Services and DTS

When you upgrade a SQL Server 2000 system to SQL Server 2005, you will probably have DTS packages. SQL Server 2005 includes a Data Transformation Services "runtime" that allows many packages to operate. This layer is included to run many of the older DTS packages, but it does not allow you to design them.

I say "many" because some complex logic (such as the data mining task) is not supported in the runtime layer. In those cases, it is best to bite the bullet and redesign the package in SSIS without having to use the runtime incompatibilities. In the Appendix, I talk more about upgrading your system so that you can make the determination about how many of these you will have to redesign. Microsoft has done an admirable job in getting the system to work with a great deal of the DTS packages, but you will need to run the Upgrade Adviser to see what you have got to work with.

You can also run the DTS package as a task within the SSIS framework. You have the same limitations as when you use the runtime layer, but you can add functionality around the DTS package using this method.

You can migrate DTS packages to SQL Server 2005 using the Package Migration Wizard. This tool connects to a server and reads the packages it has. It then attempts to map the tasks it contains to SSIS. When it cannot map the task exactly, it picks something that it feels is the closest match. For this reason, it is important to review the process when it is complete.

The Package Migration Wizard only runs in the Standard Edition of SQL Server 2005 and higher.

There are few important points to know about this process even before you begin. The Migration Wizard is a copy of a DTS package. Whether you are bringing the package from another server or a SQL Server 2000 instance on the same server, the original package stays the same. The second thing is that some tasks do not map properly to SSIS, such as the following:

  • ActiveX script

  • Analysis Services

  • Custom tasks

  • Data-driven query

  • Dynamic properties

  • Execute packages tasks that load Meta Data Services

  • Parallel data pump

  • Transaction settings

  • UDL connections

Let's take a look at an example DTS package migration. I have SQL Server 2000 running on my system with a simple DTS package that performs a similar function to our earlier example. In this case, it exports the authors table from the pubs database to an Excel file. You can see that package in Figure 8-36.

Figure 8-36.

[View full size image]

In the SQL Server Management Studio, I have right-clicked the Data Transformation Services object in the Management, Legacy item in the Object Browser. That brings up a menu for the Migration Wizard, which you can see in Figure 8-37

Figure 8-37.

[View full size image]

After the welcome screen, I am asked to provide the connection for the DTS package, shown in Figure 8-38.

Figure 8-38.

[View full size image]

When I click the Next button, I am asked for the destination of the SSIS package. I enter my SQL Server 2005 instance name, and then click Next to continue. In this panel, I am asked to set which DTS packages I want to bring over. Because I only have one on this test system, I select it. You can see that selection in Figure 8-39.

Figure 8-39.

[View full size image]

On the next screen, I am asked to provide a logging location for this operation. I enter a file on my temp directory, and click Next to continue. At the last screen, I am asked to review the process and click Finish to finalize my choices. The process runs, and in this case it is successful.

Now I will open the Business Intelligence Development Studio to review the choices the Migration Wizard made, as shown in Figure 8-40.

Figure 8-40.

[View full size image]

It is important to remember that the DTS package is migrated as it stands into an SSIS package. One of the most common errors is not to realize that all the original connections are preserved. If you are exporting from a SQL Server 2000 system to an Excel file, the sources and destinations remain the same. I have seen many administrators sit in amazement that the data they just entered into their server did not make it to the destination, only to realize that the source system is still pointed at the original server.


Chapter 9. Reporting Services

The first phase in implementing a solid IT infrastructure is to develop a data management strategy. The organization needs to know what data is stored, how it is stored, where it is stored, for how long it is stored, and a description of its security. The second phase of the infrastructure implementation is the reporting layer. Using a solid data foundation allows the organization to form a reliable, consistent way to report data from all its sources. That is the focus of this chapter. The third layer, which I cover in the next chapter, is analytics.

Among all the disciplines in database technology, the one that crosses the most skill lines is reporting. To implement a complete reporting system, you need people who know the data and the best ways to access it, infrastructure talent, and people gifted with design.

Reporting touches every part of the data cycle. From online transactional processing (OLTP) to online analytical processing (OLAP) systems, reporting interfaces with the organization's data at every stage. One of the most common problems in reporting infrastructures is showing different results on reports that are developed from the same base data. This occurs because so many stages are involved, and sometimes coordination between them becomes an issue.

As the database administrator (DBA), your part of this process is to manage, secure, and maintain the reporting system. In days past, the reporting system was separate from the database layer. Microsoft SQL Server 2005 now includes a full HTTP-based reporting system that you can use to fulfill both the OLTP and OLAP requirements for data output to screen, printer, or other binary formats such as Excel and PDF files. Users can access reports over using their browser, an application, or even have reports delivered to them.

In this chapter, I show you what you need to know to install, manage, and monitor a reporting system. In the "Take Away" section at the end of the chapter, I show you how to connect a report to an OLAP system, something you will use in the next chapter.

Reporting Services, much like SQL Server Integration Services (SSIS), is less a single product than a full suite of features based on the SQL Server 2005 platform. It is a complete reporting system, enabling you to design, create, manage, and distribute reports. There are a lot of components in Reporting Services, each capable of multiple configurations.

I have found that the best approach for understanding Reporting Services is to explore the architecture and then examine the aspects of what the product can do. I spend a little time explaining how to design and create reports and then more on how you can distribute them and manage the entire application. In this chapter, I cover some, but not all of the things you will be called on to do.

Just like in SSIS, as the DBA you will not normally be called on to design and develop full reporting applications. In fact, because reports have so many capabilities, your developers will (read that should) have dedicated at least one resource to creating them.

Those development professionals can use the Business Intelligence Development Studio, which is a full design environment, power users can arrange reports based on a model created for them using a more simplified interface in the Web-based Report Builder, developers can use a full report development suite in Visual Studio, and everyone can create reports using a special application of XML called the Report Definition Language (RDL). We take a quick tour of all of these methods in this chapter.

That is not to say the DBA does not ever create reports, but in complex environments, it is often a full-time job just managing and keeping up with them. Where your talents will come into play is in assisting the users, report professionals, and developers create reports that access the data in the most efficient way. You will need to explain to them how the architecture works, what the data flow looks like, and how security fits into the whole picture.

You can run Reporting Services on every SQL Server 2005 edition starting with the Express; to get higher security and performance features, however, you want to install the software on the Standard Edition or higher. You can use all of the editions as a source of data for your reports, and other sources such as XML files and other database systems can be accessed as data sources.

Before you start working through the examples in this chapter, make sure that Reporting Services is installed on your test server. If you chose the same options shown during the installation exercise in Chapter 1, "Installation and Configuration," you will already have an instance of Reporting Services running. If not, you can put the installation media in the drive and walk through the installation screens to add it. You can also use the Add or Remove Programs applet in the Windows Control Panel to add features to SQL Server 2005, but you still need to have the CD or other SQL Server 2005 source ready. Just check the boxes for Reporting Services; the installer will do the rest.

In either case, when you install the feature, you are asked whether you want to create the standard configuration for Reporting Services. If you select that option, the installer creates all the services, directories, Web sites, and databases you need to run the product.

If you do not choose that option, after you have installed the feature, you can tailor the settings for these objects. You will need to create a Web site, secure the file locations, and choose a service account and other options before you can process reports. I walk you through the screens where you bring all those settings together in Reporting Services, but read through the entire chapter first so that you can decide on the layout that works for your situation.

Although you can install everything on one system, in production this is not a good idea, for performance reasons. I explain the various layouts a system can have a little further on in the chapter. For now, you can set up your test system to have all components.

On my test system, I have taken the default configuration of Reporting Services, and I am running Windows 2003, Enterprise Edition.

Because Reporting Services depends on an HTTP server, you will run into differences based on the version of Internet Information Services (IIS) you are running. The Windows 2000 version of IIS has fewer security features that interact with Reporting Services, and so some settings differ. I point those out in the "Management" section a little later.

To allow access to the system and secure Reporting Services, you need to open the SQL Server Surface Area Configuration (SAC) tool. In the following screen, I have navigated to the Surface Area Configuration for Features area of the tool and set my system to allow HTTP connections. In fact, on this system I have opened up all the options for Reporting Services. I have allowed HHTP access, and I also selected Windows Integrated Security for authentication. Make your choices here based on the level of security you require. You can see my settings in Figure 9-1.

Figure 9-1.

[View full size image]

Next, navigate to the Microsoft SQL Server 2005 item in the Windows Start menu, open Configuration Tools and look for the Reporting Services Configuration item. Click that to start configuring your server, as shown in Figure 9-2.

Figure 9-2.

[View full size image]

You are asked to connect to the Reporting Services instance name. When you enter that information and connect, you are presented with a step-by-step set of tasks to configure your Reporting Services instance, as shown on the left side of Figure 9-3. If you took the defaults on installation, your system will already be configuredat least for the most part.

Figure 9-3.

[View full size image]

On this system, I have two more tasks to complete, indicated by yellow icons. For now, I close the tool, but we will revisit it in the "Management" section a little later. To verify that you can access your Reporting Services instance, open a Web browser and enter the following string:

http://SERVERNAME/Reports

or

http://SERVERNAME/Reports$INSTANCENAME

Replace SERVERNAME with the name of your server, and INSTANCENAME with the name of the SQL Server 2005 instance where you installed Reporting Services if you installed a named instance of SQL Server. If all goes well, you will see a screen similar to Figure 9-4.

Figure 9-4.

[View full size image]

If you do not see this screen or something like it, first check your spelling. You may have the instance name wrong. You can open the IIS Manager in the Administration Tools from the Windows Start menu to see whether the Web site name shows up there. You may also have to preface the Web site with https:// rather than http:// if your IIS system has been secured to use Secure Sockets Layer (SSL).

If you still cannot seem to make it work, make sure your Web server is installed and running. Try entering just the name of the server in a Web browser. If that does not return anything, check the IIS configuration. Correct that issue, and then run the Reporting Services Configuration tool again, and verify the operation there. With all of that corrected, you should be able to see the Web site open up.

When you are able to see the Reporting Services Web site, we are ready to start.


Reporting Services Architecture

Let's begin at the "front" of the system by briefly exploring a report. A report is a text file that contains a definition and at least one data source. The definition contains a description of the report, such as formatting, layouts, sections of data, and other report elements. The data source contains all the connection information for data along with a query that pulls it back to fill out the report.

When the report is created, it is stored in a Reporting Services database on a Reporting Services server. This database, called the catalog, is named starting with the words ReportServer, with the name of the instance attached, if applicable. Multiple Reporting Services servers can point to this same database, which provides one point of scale-out for the system.

Reporting Services uses another database, called the ReportServicestempDB (again with the name of the instance attached, if applicable) to hold the results of reports or perform sorting and other temporary operations to process the reports when called. That is an important distinction, and something we should take a little time to understand.

Although my developer friends and Microsoft might cringe to hear me say it, Reporting Services has a lot in common with a mail-merge operation in an application such as Microsoft Word. In a mail-merge process, you create a "master" document, which has some static data, formatting, and perhaps a picture or two, and combine it with "fields," which are just placeholders where other data will go. You then attach to some sort of data, such as a spreadsheet, database, or even a text file, and then combine the data with the master document. The master document is used only once, and the fields in it are replaced with the individual bits of data from the data location into a new document or set of documents. The master document never changes; if more data is added in the data location, however, any new mail-merged copies will have the same document with different data.

That is similar to how Reporting Services works. The "master document" in this case is called a report definition, and the "data location" is called a data source, which is the connection you make to whatever information the designer wants to include in the report.

The source data is not stored in the Reporting Services database, but the report definition is. When the report is called, the data source is combined with the report definition to produce a report.

Users will call the final product a "report." That is why Reporting Services makes the distinction for the master document, calling it a report definition. It is easy to get these terms mixed up, especially during design discussions.

The difference between a mail-merge operation and report-processing operation is that Reporting Services can talk to far more data sources than Microsoft Word, even using different data sources within the same report, and has a powerful storage engine. It can also schedule reports to run at a specific time, or even distribute the reports to users based on a schedule.

Another difference is that you can provide not only a static report to the users but a report model, which is a kind of framework for a report. The users can use that framework to create their own reports using a Web-delivered tool. It is a great way to deliver a type of "ad-hoc" reporting while maintaining control over how much data the user can pull down from the system.

The engine for Reporting Services is composed of a Windows service and a Web service. The Windows service, called SQL Server Reporting Services, runs and manages the reports for the instance where it is installed. The Web service provides a connection to the Reporting Services server from the clients that need the report and is stored and processed in IIS.

The Reporting Services processing layer handles more than just connecting to the data and filtering it through report definitions. The server(s) also "render" the report, which is the process of encoding the output into multiple possible formats such as HTML, Excel, PDF documents, and more. Reporting Services handles security for the process and also monitors and tracks report processing progress.

Users can access reports directly from the Reporting Services Web interface, or they can use a report created by a developer right inside their application. Yet another way that users can receive reports is on a scheduled basis, in their inboxes on e-mail or on a personal digital assistant (PDA) device. Regardless of how they receive the report, they get it from one or more Reporting Services servers.

Deployment Models

Your deployment will fall somewhere between two extremes. On the one side, you have a simple setup designed to move the reporting processing load from your production application servers. At the other extreme is a setup involving multiple servers accessing data from multiple databases.

In any model, you have to make a decision about the source data. If you are accessing text files or Excel spreadsheets, you do not have to deal with much contention for the data sources. If you are accessing the OLTP system, you will impact the performance while the system reads the data for the reports. A better design is to place a copy of the production data on another database server using database mirroring or replication. That provides reasonably current data without impacting the production system. Check Chapter 6, "High Availability," to find multiple strategies to copy data from one server to another.

For a small application, you can set up Reporting Services on a system separate from your reporting database server and either code the application to use the Web service it provides or point the users to the server's name in a Web browser. This deployment model splits the data from the processing, formatting, and delivery of reports, as shown in Figure 9-5.

Figure 9-5.

If your needs are slightly more complex, you can add more Reporting Services servers, each with its own report and tracking databases. In this deployment model, you have to direct the users to the server that runs their particular report. This layout is shown in Figure 9-6.

Figure 9-6.

At the higher end of the spectrum, you can balance the load among multiple Reporting Services servers. You have two methods of doing this. The first is that you can install Reporting Services on multiple servers, taking the "files only" option and not choosing the standard configuration as we did earlier. In that case, you return to the configuration tool I described in the preceding section and set up various servers to talk to a single Reporting Services report database. That system then accesses the source data for the other servers, and they present the report. This layout is shown in Figure 9-7.

Figure 9-7.

[View full size image]

To balance the Web servers, Microsoft provides a set of checklists for configuring the servers to connect to the database and to talk with each other. You can find that checklist on the Web on Microsoft's Reporting Services site or in Books Online under the topic of "Configuring a Report Server Scale-Out Deployment." This type of setup involves steps that change over time as the .NET Framework is updated, so I do not reproduce that information here. It is best that you check a current source for that process.

Whichever layout you choose, plan it thoroughly before you begin. The best process to follow is to design the system with the ability to grow and then monitor it proactively. If you keep your design flexible at the outset, you can add components to your layout as needed. This is important not just for the servers but the design of the reports. Everyone, from the report developers to the administrators, should keep things such as data sources as object oriented as possible. Keeping the components independent and well documented allows you to grow the system with minimum impact.


Report Design and Creation

A bewildering array of information is available on how to design effective reports. The skills you need to design reports include knowledge of graphic design, programming, the source data systems, and even the organization's structure, to name just a few.

I do not cover the complete design process here because our focus is more on administration. You should, however, be familiar with the various types of reports developers create and what reporting tools are available. I guide you through those and then we create a simple report using a wizard and another using the other tools.

Report Types

Reports are stored as a text description using RDL (more on that in a moment), but they are broken down into conceptual types based on how they are created, processed, and accessed.

Web Based

The default mode for accessing a report is to use the Reporting Services Web site. Earlier I had you test your Reporting Services by accessing the site. You can access the Web site using the address I showed you earlier:

http://SERVERNAME/Reports

or

http://SERVERNAME/Reports$INSTANCENAME

This Web site is arranged like a file system using folders, which contain reports or other folders. I cover that a little more in a bit.

Application

It is not always convenient for the users to leave their application and open a Web browser to look at reports. A more common method for accessing reports is through an application. Developers write code to access the reports using .NET methods by invoking a proxy Web service. When they connect to the proxy service, the application instantiates a method against that service, which interacts with the reports.

Developers can also use the RS.EXE command-line program to control reports. This is used less often but is useful for scripting applications.

Another way that developers can embed a call to Reporting Services in their applications is to use the Simple Object Access Protocol (SOAP). Developers can invoke this kind of call in the application by using a SOAP application programming interface (API) in their language of choice. This is the most portable method and allows you to use even non-Microsoft-based platforms to call the reports.

Ad Hoc

A hot-button topic in report design is whether or not to allow "ad-hoc" reporting. Ad hoc means that end users define what they want to see and how they want to see it. The users get a development environment, which enables them to connect to data sources and create their own reports. A true ad-hoc environment is rarely implemented because a careless query can completely immobilize a system. Users have to be carefully educated even in environments where this feature is used in a limited fashion.

Users, on the other hand, desperately want ad-hoc capabilities in their report environment. Many times, they are not able to fully articulate their reporting needs during the application design process, or they want the ability to explore the data freely. These are real needs, despite the desire on the part of the developers to more tightly control the process.

Reporting Services provides a feature to help. Developers can create a report model, which is a predefined report structure stored on the Reporting Services instance. The report model contains friendly "business" names for the fields referenced in the data source, the relationships the data sources have with each other, and groups of items that the users can select from. The users connect to the Reporting Services Web site, and a special ActiveX control is downloaded to their machine. The control gives them a design environment where they can add text and formatting, pull down and lay out the individual fields from the data sources, and even navigate through the data the report can access. Developers can also enable the report model to be able to drill down through the data in the report, if it is viewed on the Web or in an application.

Even though the developers can create report models that the users can access, they need to test each scenario to determine the impact to the system. Leaving too much access or allowing connections to production data still runs the risk of creating performance issues. Report models provides a layer of protection, but if you are not careful, you can still create a design that adversely affects your system.

Report Definition Language

Report definitions are stored in a specialized application of XML called Report Definition Language (RDL). The beauty of storing the reports this way is that it allows an open platform for Reporting Services. It also provides developers a way to build a report on one system and transfer it to Reporting Services seamlessly.

RDL is not the transport protocol for the report definition; it is the payload. Developers can load the RDL through several methods. Your developers can find a complete set of examples for creating and uploading RDL programmatically in the Samples directory if you installed them when you installed SQL Server 2005. You can also load RDL files in the Reporting Services Web interface.

I do not reproduce the XML element set for RDL here because it is lengthy and deals with the layout of the report. You can find a complete list of the RDL elements that Reporting Services supports in Books Online under the topic of "Report Definition XML Elements." You can find other definitions of these elements on the Web, too. Microsoft also includes a complete entity-relationship diagram under the topic "Report Definition XML Diagram" in Books Online and on their Web site.

Design and Development Tools

There are a few ways to interact with Reporting Services, from command-line tools to full integrated development environments (IDEs). In this section, I deal with the graphical tools used to create reports, and in the "Management" section I explain the graphical tools you can use to manage the process.

This is not an exhaustive coverage of these tools because we are not focusing on designing reports. Instead, it gives you a quick tour of the interfaces your users and developers have available.

Report Designer

The Report Designer is a template item called Report Server Project within the SQL Server Business Intelligence Development Studio. You can find that in the SQL Server 2005 item in the Windows Start menu. You can see the result on my test system in Figure 9-8.

Figure 9-8.

[View full size image]

I do not cover the screen layout for this tool because we already explored it briefly in the preceding chapter and we cover it in even further depth in the next.

The audience for this tool is the report developers in your company. They will use this Visual Studioderived interface to create "canned" or static reports, or the report models that the users have to work with in the Report Builder tool.

Model Designer

Also found in the SQL Server Business Intelligence Development Studio is the Report Model Designer. Once again, the audience is the report developer, who performs the same steps as creating a static report, with the exception of layout and formatting. When the report model is deployed to the Reporting Services server, select users can connect to the model and create their own reports, based on the data sources and restrictions the model contains.

This is the primary tool you will find for the Business Intelligence landscapes based on Microsoft SQL Server 2005. In the "Take Away" section of this chapter, I demonstrate how a report model is created with this tool.

Report Builder

The Report Builder is a full development environment for the users. That is not to say that all users need to create reports; most just want to view reports created for them. For those who need an ad-hoc view of data, the developers can create report models in the Business Intelligence Development Studio, which are then deployed to a server. Users connect to the Reporting Services Web interface and click the Model folder. Once inside, they can click the Report Builder button on the Web page that contains the models, shown in Figure 9-9.

Figure 9-9.

[View full size image]

This button starts a download of the Report Builder program, as an ActiveX component. When the download is finished, users have a complete development studio to create their own layouts, formats, and destinations for the report based on the limitations created in the report model, as shown in Figure 9-10.

Figure 9-10.

[View full size image]

When users create the report, they can save it to any of the folders they have been given rights to, from their own to those created by the developers. I show you how to grant those rights in the section on security.

Creating Reports

I want to stress here that we are not going to spend a great deal of time in the layout and creation of reports. There are so many concepts in that area that we would lose track of where we were. This chapter is more concerned with the management and deployment of those reports, but we do need a few to work with. In this section, I show you how to create a test report using a menu-driven wizard, and then I show you a more useful production report that will help you watch your databases for growth.

Although I do not cover creating report models in this section, you can follow the same process as the other two reports to create one. Select Report Model from the Create New Project menu item after you have completed these exercises to create and deploy a model on your test server.

Using the Report Wizard

The easiest way to create a report is to use the Report Wizard in the SQL Server Business Intelligence Development Studio. Let's create a report from the AdventureWorks database showing a product listing using the Report Wizard.

To begin, open the BI Development Studio and select the Project… item from the Create link in the left side of the screen. From the Visual Studio Installed Templates area in the right panel of the dialog that appears, click the Report Server Project Wizard item and then give the project a name in the lower box, shown in Figure 9-11.

Figure 9-11.

[View full size image]

When you click the OK button, the first panel of the wizard explains what you are about to do. Click the Next button to move on.

The first item the wizard creates is the data source. In this case, you can select the New Data Source radio button and name it Adventure-Works. I like to set the data sources to the generic name of the source I am connecting to. You might be tempted to set the source to AWInventory or some other specific name, but in fact the connection is to the database itself. After you have entered a name, leave the Type box set to Microsoft SQL Server and click the Edit button.

This brings up a panel where you can select the name of the server, the database, and the credentials that you want to assign this data source. Make those selections for your test system and click the Test Connection button.

When that is successful, click the OK button to return to the data source creation panel of the wizard. You will see the connection string filled out for you. Instead of clicking the Edit button next time, you can just type in a string like this if you are familiar with the format.

Selecting the Make this a shared data source check box makes this connection available outside of this report; otherwise, the complete connection information is part of the RDL. Click Next on this panel after you have made your changes.

The next panel that appears is the query designer. We have made our connection to the database, and now we need to select the specific data we want to include in the report. You have two options for entering data on this panel. The first is to type a Transact-SQL (T-SQL) statement in the bottom text box. If you already know what you are looking for, this is the fastest way. If you are not sure of the table names or the exact query you want, click the Query Builder button.

Once inside this panel, you are not much better offat least at first. You are placed by default in a text editor for T-SQL. Click the "pencil" icon in the upper left, and you are placed in a more graphical selection area, much like the query designer in Microsoft Access or Excel. Click the small icon with a plus sign in it to bring in tables from your data source and make your selections for the columns they contain.

In this case, you are looking for the Product (Production) table and the Name, ProductNumber, and ListPrice columns. When you have finished, click the OK button, as shown in Figure 9-12.

Figure 9-12.

[View full size image]

When you return to the Design the Query panel in the wizard, the T-SQL that is derived from the graphical interface is entered for you. If you want any sort options, it is often better to put them in the report so that you take the lightest touch on the source system, but in this case I have added an ORDER BY statement to my query, so that it ends up with this text:

SELECT Name, ProductNumber, ListPrice
FROM  Production.Product
ORDER BY Name

Click the Next button to move on to the next panel. In the Select the Report Type panel of the wizard, you are asked to set the layout for the report. If you are following along, just leave the defaults here and click Next to continue.

The Design the Table screen allows you to set page breaks and grouping, which we do not take time to explore here. Click Next to continue.

The Choose the Table Style panel gives you some template options for the report. Choose any option you like and then click Next to continue.

The Choose the Deployment Location panel sets the server that will process the report and a folder that the users will see if they use the Web browser method of looking for the report.

Make sure you check the name of the server in the location, because the wizard may not detect the name properly. In my case, because the report server is on a named Instance of SQL Server called SQL2K5DE1 on the server named SQL, the string becomes http://sql/ReportServer$SQL2K5DE1. Entering the wrong name does not cause the preview to fail, but the report will not build properly if the server name is incorrect.

Click Next when you are sure of the server's name. Finally, you are asked to name the report. Type in AW Inventory and select the Preview Report box. Click the Finish button when you have made those changes.

If everything is set up properly, you are dropped back into the Business Intelligence Development Studio with a preview of the report. You can see the results in Figure 9-13.

Figure 9-13.

[View full size image]

If you want to make any changes to your report, you can click the Layout tab and set any fonts, spacing, color, and other options. When you are all done with those changes, make sure you select the Build and then Deploy items from the menu bar.

When the build and deploy operations complete successfully, enter the Report Server name in the browser in this format:

http://SERVERNAME/Reports

or

http://SERVERNAME/Reports$INSTANCENAME

You should see a link to the folder you created there, and you should see a link to the report within that, as shown in Figure 9-14.

Figure 9-14.

[View full size image]

Creating a Report Manually

Let's build another report, this time without using the wizard. This report will be one that you can use in production. We create a query for the data source that shows the tables in a database, along with the space they take and the space taken by their indexes. Using this report, you can audit how fast your tables are growing. This is a common thing to do on newly developed databases.

It is not difficult to create a simple report manually, following the same progression that we did in the previous section. Here is the checklist we follow in this section:

  1. Create a project.

  2. Set the target.

  3. Add a data source.

  4. Add a new report item.

  5. Create the dataset.

  6. Set the layout.

  7. Build and deploy the report.

Let's begin. If you are still in the Business Intelligence Development Studio, click the File and then Close Project item from the menu bar. Save the project if you are prompted. Click the Project… from the Create link in the left pane of the main screen.

Select the Report Server Project item and name the project Table Sizes as I have in Figure 9-15.

Figure 9-15.

[View full size image]

This time you are dropped right into the project. Right-click the name of the project (Table Sizes) in the Solution Explorer, which you will normally find on the right slide-out panel. Select Properties from the menu that appears and enter the name of the Reporting Services instance in the TargetServerURL box, shown in Figure 9-16.

Figure 9-16.

[View full size image]

Remember, this is the name of the Reporting Services instance that processes the reports, not the site where the users view the report. This name should have ReportServer somewhere within it.

Right-click the Shared Data Source item just below the project name (Table Sizes) and select Add New Data Source from the menu that appears. Follow the same procedure that we did in the previous section to add a database name. In this case, pick any database that you are interested in creating an audit for. In my case, I have chosen to connect once again to the AdventureWorks database, naming the data source AdminAdventureWorks for clarity.

Next, right-click the Reports folder in the Solution Explorer and select Add and then New Item from the menu that appears. If you select Add New Report, you will start the wizard process again.

When you click the New Item menu object, you are presented with a list of choices. Select Report from the list and name the report Table Sizes.rdl. Click the Add button when you have finished. You can see the results in Figure 9-17.

Figure 9-17.

[View full size image]

Now you are placed within the report designer for the Business Intelligence Development Studio. We need to create a dataset to work with, which is the result from a query. Click the New Dataset… drop-down menu item in the center top of the screen in the Dataset box.

You are placed in the Dataset properties box where you have lots of control over what you want to pull back and how you want to get it. For this report, enter the name TableSizeQuery in the Name box and then enter the following text in the Query string text box:

SELECT
 INFORMATION_SCHEMA.TABLES.TABLE_NAME "Table Name",
 sysindexes.rows "Rows",
 ((SELECT sum(dpages)
  FROM sysindexes
  WHERE indid < 2 and id = sysobjects.id)
 +
 (SELECT isnull(sum(used), 0)
  FROM sysindexes
  WHERE indid = 255 and id = sysobjects.id))
 *
 (select v.low / 1024
  from master..spt_values v
  where v.number=1 and v.type=N'E')
 "Data Space (K)",
 (SELECT sum(used)
  FROM sysindexes
  WHERE indid in (0, 1, 255) and id = sysobjects.id)
 *
 (select v.low / 1024
  from master..spt_values v
  where v.number=1 and v.type=N'E')
 -
  ((SELECT sum(dpages)
  FROM sysindexes
  WHERE indid < 2 and id = sysobjects.id)
 +
 (SELECT isnull(sum(used), 0)
  FROM sysindexes
  WHERE indid = 255 and id = sysobjects.id))
 *
 (select v.low / 1024
  from master..spt_values v
  where v.number=1 and v.type=N'E')
 "Index Space (K)"
FROM
 INFORMATION_SCHEMA.TABLES
 INNER JOIN sysobjects ON
 INFORMATION_SCHEMA.TABLES.TABLE_NAME = sysobjects.name
 INNER JOIN sysindexes ON
 sysobjects.id = sysindexes.id
WHERE
 INFORMATION_SCHEMA.TABLES.TABLE_TYPE = 'BASE TABLE' AND
 sysindexes.indid < 2
ORDER BY
 "Data Space (K)" DESC
, "Index Space (K)" DESC

That is a lot of typing, but it is worth it. Figure 9-18 shows what my test system looked like for those entries.

Figure 9-18.

[View full size image]

When you click OK on this panel, you are placed in the Data tab of the Business Intelligence Development Studio. You can click the exclamation icon to run the query and see the results.

Now click the Layout tab in the Business Intelligence Development Studio. You may have to auto-hide many of the panels to get any usable space to work with the report. Open the Toolbox panel (usually found on the left side of the screen) and drag the Table object onto the report, as you can see in Figure 9-19.

Figure 9-19.

[View full size image]

We need four columns in our table, and we only have three. Click the top gray bar of any of the columns to select it and then right-click in that selected area. Add a column to the left or right.

You have three main areas to work with in a table: the header, the detail, and the footer. Whatever you put in the header and footer will repeat at the top and bottom of the report. The detail lines show up once for each row of data. In the upper-left cell of the header, double-click and enter the text Table. In the second cell of the header, double-click and enter Rows and then Data Space and Index Space for the third and fourth cells in the header.

In the first cell of the detail, right-click and select Properties from the menu that appears. Click the drop-down menu for the Value item and then select the formula =Fields!Table_Name.Value. This item comes from our query; it is the first part of the SELECT statement.

You can also select the Datasets tab at the bottom of the toolbox and drag the fields from there, or you can enter the formulas by typing them in the detail cells.

Repeat this process for the remaining three columns, selecting =Fields!Rows.Value, =Fields!Data_Space__K_.Value, and =Fields!Index_Space__K_.Value for the Value items in those cells. Your report design screen should look similar to Figure 9-20.

Figure 9-20.

[View full size image]

You can format your cells in any way you see fit, and add graphics, text, and other objects to the report if you like. I am going to bold the header cells and leave it at that for this exercise. Save your work by clicking the multiple disks icon and then click the Preview tab to see the results.

When you are satisfied with the report, click the Build and then Deploy items from the menu bar. If you have set your server's name properly, you should be able to view the report on the Web site that we accessed earlier for the Inventory report, as on this test system screen in Figure 9-21.

Figure 9-21.

[View full size image]


Reporting System Management

In a sense, all the previous information in this chapter deals with the management of the Reporting Services system. A great many of the choices you will need to make to manage and maintain the system depends on the layout you chose early in the layout process. If you chose a single server that connects to your production data, you have a simpler task ahead of you than if you have multiple servers set up in a Web farm.

In this section, I show you the components you need to manage, no matter where they are. First, I explain the tools you need to use to manage each part. After you are familiar with the environment each tool offers, we will put that to good use in each of the sections that follow.

Reporting Services Configuration Tool

You will use several tools in managing your systems, including the SQL Server Management Studio, the Windows Control Panel, and others. The primary tool you will use to set up and configure Reporting Services servers is the Reporting Services Configuration tool that I showed you in the first part of the chapter. I return to that tool here to talk about the components within the system.

Starting and Stopping Reporting Services

You have got a few options for starting and stopping Reporting Services on your server. The first is to use the Services Control Panel applet in the Windows Server operating system, as shown in Figure 9-22.

Figure 9-22.

[View full size image]

This method is useful if you are already in the Services applet or if you want to stop the service on a remote computer. Another method to stop and start the service is to use the SQL Server Configuration Manager, shown in Figure 9-23.

Figure 9-23.

[View full size image]

This is a good way to control the services from an entire SQL Server perspective, or if you are in the tool to stop more than just Reporting Services.

If you are working with the server strictly from a Reporting Services aspect or you want to change other configuration parameters, the best place to view, stop, and start Reporting Services is using the Reporting Services Configuration Management tool. You should see a screen similar to Figure 9-24, and the first icon in the list shows the server's status and allows you to control its state.

Figure 9-24.

[View full size image]

If you want to work with a different Reporting Services server, just click the Connect icon and supply the server's name and your credentials.

Virtual Directories

A virtual directory is a term from Microsoft's Web server, IIS. It serves as a pointer that the system uses to present a physical directory as a Web location. For instance, the site http://MyServer/MyDirectory can be "pointed" using a virtual directory to a location on the server's hard drive called c:\www\public. Anything you put in the physical directory shows up on the Web site.

The second icon in Reporting Services Configuration Management sets the Web site virtual directory that you can use for the Web-based management of the system. On this test system, I have the default directory set up; if you are setting up the system for the first time without using the original installation wizard, however, you can specify the site in Figure 9-25. If you want to use a SSL for more security, this is also the place to set that. If you want to make that choice your IIS, you need to have the SSL setup complete before you can access the system using the HHTPS protocol.

Figure 9-25.

[View full size image]

The second virtual directory shown in Figure 9-26 sets the site that the system will use to present the reports to the users.

Figure 9-26.

[View full size image]

The information here looks similar to the previous screen, but notice that it does not have a selection for SSL.

On my test system, I have also opened the Internet Information Services Microsoft Management Console (IIS MMC) from the Windows Start menu and expanded the Reporting Services virtual directory there. You can see that in Figure 9-27.

Figure 9-27.

[View full size image]

I have right-clicked the name of this site and selected Properties from the menu that appears. That brings up the panel shown in Figure 9-28 where I can see the physical directory of the Reporting Services Web site.

Figure 9-28.

Unless you are familiar with IIS management, do not make any changes to these settings. You can make your site unavailable by making the smallest adjustments. One of the most useful settings you should leave selected is the Log Visits check box. This setting places an entry in the IIS log system, which will aid you in finding any problems. There are ways to configure that to send to a text file or a database, and this forms an invaluable part of troubleshooting for IIS.

The screens I have just shown you are for the Windows 2003 Server version of IIS. If you are running Windows XP or Windows 2000, your screens will look slightly different.

Service Accounts

The next icon in the Reporting Services Configuration Manager deals with the Windows service that runs Reporting Services.

In Figure 9-29, you can see the four choices for the identity of the service account. The first, in the Built-in account pull down, is the Local System account. Because this account cannot make network connections and is not treated as a user account, this is normally a bad choice.

Figure 9-29.

[View full size image]

The second option, Network Service, has slightly more privileges, but is not the best choice because of its limited operating system rights. The same holds true for the Local Server, which has the added disadvantage that it cannot use networking protocols.

The better choice is to create a Windows account that you can use for this service and grant it the rights needed to talk across the network, connect to an SMTP server, and run as a service.

DBA 101: Service Accounts

Although my test system shows the Administrator account, it is a bad practice to use the Administrator account to start services in production.

The next icon is used to set the identity for the IIS 6 and higher application pool, as shown in Figure 9-30.

Figure 9-30.

[View full size image]

The application pool allows IIS to run code in isolation, with different rights and privileges. What that means is that a particular application will not bring down the entire IIS server, just the application assigned to its "pool." Unless you need to change this setting, leave it at the default.

Report Server Database

In the next icon area, you can control the database that Reporting Services uses to store the reports. You can see that selection in Figure 9-31.

Figure 9-31.

[View full size image]

You also set the credentials in this area that the system uses to connect to the database. The default is to use the service accounts you specified in the previous panels. You can change that to a Windows or SQL Server account.

There is also a button here that you can use to upgrade the database. This is for systems where you had an earlier version of Reporting Services installed and you want to retain the reports it has. After you install SQL Server 2005 Reporting Services, you point this panel at the older database and then click the Upgrade button. The system will apply the changes for the newer version and apply SQL Server 2005 schemas to the database layout.

Encryption Keys

If you want to ensure the highest level of security on your system, encrypt the contents using this area of the Reporting Services Configuration Manager. This is shown in Figure 9-32.

Figure 9-32.

[View full size image]

If you want to secure your system, click the Change button to start the process. When the process completes, it has generated a key that this server will use to encrypt its data. Click the Backup button right away and supply a password and a filename for the key. If you lose this key, you cannot decrypt the data later. Store that file in a safe place.

Initialization

The Initialization area lets you create all of the objects and settings you need to set up an instance of Reporting Services. You can remove the instance by clicking the Remove button, as shown in Figure 9-33.

Figure 9-33.

[View full size image]

E-Mail Settings

Reporting Services can send out reports using multiple channels. One of the most useful channels for reports such as Key Performance Indicators in Business Intelligence is e-mail. In the screen shown in Figure 9-34, you can set the login and location for your e-mail server.

Figure 9-34.

[View full size image]

Execution Account

The Reporting Services service accounts are not used to authenticate to other servers, other than for certain data sources. Instead, the reports can impersonate a user or use a Windows account or this execution account to retrieve graphics files and other documents. You can set that account in the screen shown in Figure 9-35.

Figure 9-35.

[View full size image]

Report Manager

If you have been following along with the examples in this chapter, you have already accessed the Web-based Report Manager several times. The location for the manager is in the following format:

http://SERVERNAME/Reports

or

http://SERVERNAME/Reports$INSTANCENAME

The Report Manager provides a rich environment to do everything from viewing the reports to managing the security for them. In this screen, I have connected to my Reporting Services instance Web site from my local workstation. I clicked the Show Details button to arrange the items on my Reporting Services instance, as you can see in Figure 9-36.

I demonstrate this tool further in a bit.

Figure 9-36.

[View full size image]

Working with Report Manager Objects

The layout in Report Manager is similar to a file system. You have objects, such as reports, or folders that contain objects and other folders. In my case, someone (me, actually) published a report directly to the root of the site, something I would rather not have.

I could either move the item to an existing folder or create a new one using the Report Manager. Because I do not have a convenient place for the reports, I create a new folder called User Reports using the New Folder button at the top of the page, as shown in Figure 9-37.

Figure 9-37.

[View full size image]

With that folder created, I click the check box for the report and then click the Move button, specifying the new folder as the destination.

Each object and folder in the Report Manager has a Properties tab associated with it. You can click this tab to set the security and other information for the report or folder. In Figure 9-38, I have selected the report I just moved and then accessed the Properties tab to find out more about it.

Figure 9-38.

[View full size image]

Each object also has a security property associated with it, which you can manage from here. I cover security in greater depth in a moment.

Back on the Home screen of the Report Manager, you will also find a button to upload a report or other files. If your developers create reports using a tool other than the ones Microsoft provides, you can take its RDL and load the report into your system here. You can also upload binary files such as logos and other objects the users need access to; unless you have a driving need to store those types of files on the Report Server site, however, it is best not to.

The Report Manager can also control the subscriptions the user has for report distribution. I cover that a little further on in the chapter.

Another useful feature in the Report Manager is the ability to control the entire site from the Web. Click the Site Settings link in the upper right of the screen to manage the server. You can see the result in Figure 9-39.

Figure 9-39.

[View full size image]

On this screen, you can allow users to create their own folders for publishing reports, set the number of filled out, static reports (called snapshots) the users can save, and set limitations for the duration of the reports such as timeouts. You can also enable logging here, as well as manage security for the site and set the jobs and schedules for the reports.

SQL Server Management Studio

If you are connected to the same network as the Reporting Services instance, another tool you can use for management is the SQL Server Management Studio. When you connect to this tool, you specify that you want to connect to Reporting Services in the Server type box, shown in Figure 9-40.

Figure 9-40.

Once inside, you have a more familiar interface to work with than the Report Manager Web site. This environment uses the noun-verb arrangement where you right-click objects (nouns) to do something with them (verbs). You can see that arrangement in Figure 9-41.

Figure 9-41.

[View full size image]

Right-click any object to get to its properties, such as security and history. Right-clicking the objects you see here is the primary method of interacting with this tool.

Another advantage in using SQL Server Management Studio is that you can register Reporting Services as well as database engines, Integration Services, and more in the same panel. When you need to manage an "end-to-end" solution, SQL Server Management Studio is often the best tool to use. Let's take a look at the first task you should do with Management Studio.

Backing Up and Restoring Reports

We are going to add to the Report Server registration within SQL Server Management Studio, because to back up the data for the reports we need to use the database engine. Although it might seem cumbersome to use another part of the tool to do this kind of management, it makes sense from a task standpoint. Normally, the people in charge of the reporting system are not in charge of the database systems. If you are, it is a simple matter of clicking the Connect button just above the Object Browser and selecting Database Engine from the menu. Enter the name and credentials for the server that holds the Reporting Services databases.

From there, you maintain and back up the databases in the same way that you do with any production database. There are some subtle differences, however, in the way that you set up and manage the databases.

The Reporting Services databases start with the words ReportServer and are created during the install or configuration process. You will have at least two: one for the Reporting Services site, which holds the report definitions and binary objects used by the system; and the other for the temporary files created by the system.

These temporary items are not just the sort operations and other "scratch" areas; it also holds the copies of the filled-out report. Recall that what the users call a report is the combination of the report definition and the source data. As such, this database can become quite large, which is just one of the reasons you do not store it on the same server as your OLTP data.

Another difference in the way you handle these databases is that you should not change the structure or add objects to the database unless directed to by Microsoft.


Reporting Security

Report security is another one of those "it all depends" kind of discussions. It depends because there is more than one way to set up the security, so there is more than one way to manage it. I will try and de-mystify the process, at least from a high level.

The primary concept is that Reporting Services is Web based, so the security within Reporting Services first goes through the IIS. To explore this security, open your Windows Start menu and select Administrative Tools and then Internet Information Services (IIS) Manager. Note that this item might be in a different menu location based on the operating system you are using.

Drill down in the Web Sites object and expand the Default Web Site item. If you took the default installation, you will find two virtual directories there, one starting with the word Reports and the other with the words ReportServer. Right-click the Reports site and select Properties. From there, click the Directory Security tab, as shown in Figure 9-42.

Figure 9-42.

[View full size image]

Click the Edit button under the Authentication and Access control field to set the security for the Report Manager (in the Reports site) or the SOAP access for running reports from application (in the Report-Server site), as shown in Figure 9-43.

Figure 9-43.

You have several options, from using Windows integrated authentication for those systems connected to your network, to Basic, which pops up a box on the user's screen and sends information in clear text. Digest authentication is not used in SQL Server Reporting Services.

You can also configure your trusted or other Internet Explorer site to prompt the user for credentials even with Windows authentication selected.

Security Layers

Think of the IIS security as the front desk in the lobby, with the directories for the reports as doors behind the receptionist. The IIS server is the first layer of security. That security only allows users access to the systemnot any particular report. From there, the folders and objects (such as reports) are configured to allow those accounts access.

When the security method is set in IIS, there are three categories of methods that you can use to secure the folders and reports. The first, and by far the simplest for you, is to use application authentication. In this method, your developers use a Windows account that the application accesses on behalf of the users. This account has full permissions on all folders and objects, but the name and password are known only to the developers. You lose a great deal of control in this method, but in a strong development shop this is commonly implemented.

The second method you can use to secure your reports is to tie Windows accounts to roles within Reporting Services and then tie those roles to folders and objects. Roles are groups that you can use or create, and this is the normal method for securing the system.

The final method is to manage security using a security extension, which is a program your developers can create to fold in to the authentication methods trusted by Windows. I do not spend any time on that process here because it completely depends on how they write and implement it. If you are interested in seeing a practical example of this type of security, look up "Security Extension Example" in Books Online.

Regardless of which method you choose, you can use SQL Server Management Studio or the Report Manager to implement and manage the roles. On my test system, I have seven accounts created, named for what they do in a fictional company.

Full Name | Windows Login
Aaron Accountant | Accountant
David DBA | DBA
Dianne Developer | Developer
Lewis LineOperator | LineOperator
Mary Manager | Manager
Susan Shipper | Shipper
Thomas Tech | Tech

Let's take a look at assigning a couple of these accounts to roles within Reporting Services, and then we will assign those roles to folders and reports.

Managing Security with SQL Server Management Studio

You can connect to a Reporting Services instance either by registering it or connecting to it after you are in the Management Studio. Once inside, you can drill down to the Security item and then expand the Roles object. You can see the results of that process in Figure 9-44.

Figure 9-44.

[View full size image]

Several roles are already created for you, with most of the common tasks that the users need to do. Double-click the role name to view and set what the privileges the role contains. Figure 9-45 shows a view of what the Browser Role can do.

Figure 9-45.

[View full size image]

It is best not to change the roles that come configured with Reporting Services. A better practice is to create a new role and assign the types of things the user can do in a new object.

After the roles are created, you can assign them to objects and to user accounts at the same time. On my system, I have a report called Contacts in the User Reports folder. I will right-click that folder and then choose Properties from the menu. Once inside, I select the Permissions panel, as you can see in Figure 9-46.

Figure 9-46.

[View full size image]

This folder inherits the security from the one above it. That is a important concept in Reporting Services security. As you create folders underneath each other, by default they retain the security from the parent. You can change the security for each successive folder.

Objects in the folder inherit the folder's security. You can change the security for any object, as I do here for this folder. To make the change, I click the radio button that starts with Use these roles… and then click the Add Group or User button. That process allows me to enter the name of a Windows group or user and assign them to one or more Reporting Services roles for that object. I will allow Mary Manager to browse, publish, and build reports in and to that folder by adding her Windows login to this panel and assigning her to the Reporting Services roles that have those rights.

If you are interested in performing this on your own test system, create some Windows accounts and experiment with the test reports we created earlier. Figure 9-47 shows the results.

Figure 9-47.

[View full size image]

That brings all the security together. The IIS server authentication "discovers" the user who is accessing the site, and the roles that account is assigned to set the rights he or she has on the folders and objects.

If you have users who need to access reports but they are not members of your company, the simplest method to provide reports is to create a low-privilege Windows account and then have the developers create reports that request credentials. When users run the report, they provide both the Windows account and the credentials requested by the report.

Managing Security with Report Manager

Although the Management Studio has the richer interface for managing security, you can also connect to Report Manager with a browser and assign roles and users there.

You have the same abilities using the Web-based Report Manager, but things are in a little different arrangement. To see the properties for an object, click it in the Report Manager and then click the Properties tab link near the top of the page, shown in Figure 9-48. From there, you can set whatever options are available, including security.

Figure 9-48.

[View full size image]

In this example, I set Dianne Developer to have a high level of privilege on the entire site. I have navigated to the Reports Web site on my Reporting Services server and clicked the Site Settings link in the upper right of the page. Once inside that page, I clicked the Configure site-wide security link under the Security heading. Once there, I clicked the New Role Assignment button. You can see the results in Figure 9-49.

Figure 9-49.

[View full size image]

I enter the name of the Windows account, select a role assignment, and then click the OK button to continue.

Although you can use this tool in much the same way as the SQL Server Management Studio, you do not have as many protections or helps built in. Still, it is a great interface when you cannot connect through other means.


Reporting Delivery System

There are two main paradigms for delivering the reports to your users in Reporting Services. The first is to use the default Web site within Report Manager, as we have been doing. The other is for the developers to use the SOAP to connect to the Web site and retrieve reports. In some cases, you will employ both. There is also an "object model" within the Windows Management Interface (WMI) that your developers can use to call the service and preview reports.

If you are using Reporting Services as part of a portal, you might be tempted to use Report Manager. I have found that after a period of time, the site becomes difficult to navigate. Any enterprise-level application will normally use SOAP or WMI models to display the reports.

Subscriptions

Users can also subscribe to a report, which means that the report is delivered to them on a file share or through e-mail rather than the user connecting to the server via the Web or through an application. Users can set and manage their own subscription using Report Manager.

The developers can also create data-driven subscriptions that run a query to determine the subscription information. As the DBA, you can create a subscription for users from SQL Server Management Studio, by expanding the folder that contains the report and right-clicking the Subscriptions folder underneath it. The report must have a data source that uses stored credentials or no credentials. If you send the subscription via e-mail, the Report Server needs to have the e-mail information filled out as we did earlier in the chapter.

The reports can be scheduled for the subscriptions, too. You can use the Shared Schedules section of Report Manager or the object with the same name in SQL Server Management Studio.


Monitoring and Performance Tuning

Because Reporting Services uses everything from IIS to e-mail and the files system, there a lot of places to monitor and tune for performance.

Just like performance tuning in any major application, it is a matter of divide and conquer. The process I have explained in other chapters still holds: Detail the parts of the application, monitor for bottlenecks, determine the components within that part of the system, and tune for performance.

You can find many resources for monitoring from the client through the IIS layer, so I do not repeat those here. The important elements for performance within Reporting Services have more to do with the databases that it uses as well as the source databases and the Reporting Services connection to them.

For the source databases, you should follow two basic rules: Always use a separate database for reporting source data than the OLTP system, and optimize those databases for reads. That means a high fillfactor and a lot of indexes. The idea behind these concepts is to reduce contention and to allow the fastest access possible to the source data.

Make sure you have maintenance plans set up on both Reporting Services databases as well as the separate source database. Other than standard database tuning, you want to optimize the path between the Reporting Services servers and the source database. If you have split several Reporting Services servers to talk to a central catalog server, ensure the path is optimized between those systems, too.

You have several additional objects and counters available in Windows System Monitor (PerfMon) to watch the Reporting Services server. They are divided into the two mechanisms I mentioned at the beginning of the chapter: the Windows service and the Web service.

For the Web service, you can watch the following counters in the MSRS 2005 Web Service object. Cache objects show memory pressure, actions per second show the load on the server. The more times things are received directly from cache, the better.

Counter | Used to Show
Active Sessions | Active sessions; includes all browser sessions generated from report subscriptions, whether they are currently active or not.
Cache Hits/Sec | The number of requests for all cached reports.
Cache Misses/Sec | The number times that a report was not in cache.
First Session Requests/Sec | The number of new user sessions started from cache.
Memory Cache Hits/Sec | The number of times that reports are retrieved directly from memory cache.
Memory Cache Misses/Sec | Reports that could not be retrieved from the in-memory cache.
Next Session Requests/Sec | Reports that are opened in an existing session.
Report Requests | Reports that are active and being processed by the report server.
Reports Executed/Sec | Successful report executions.
Requests/Sec | All types of requests sent to the report server.
Total Cache Hits | Requests for reports from the cache since the service started.
Total Cache Misses | Report requests that were not in the cache since the service started.
Total Memory Cache Hits | Reports returned directly from memory cache since the service started.
Total Memory Cache Misses | Cache misses since the service started.
Total Processing Failures | Report processing failures since the service started.
Total Rejected Threads | Processing threads switched from asynchronous processing to synchronous, serial processes.
Total Reports Executed | Total number of reports that ran successfully since the service started.
Total Requests | Total number of all requests made to the report server since the service started.

This table shows the counters for the Windows service in the MSRS 2005 Windows Service object. The same logic holds true, activities per second show load and the cache objects show memory use. More items found in cache show proper memory sizing.

Counter | Used to Show
Active Sessions | Active sessions in the report server database.
Cache Flushes/Sec | Number of times the cache flushes.
Cache Hits/Sec | Requests for cached, re-rendered reports.
Cache Misses/Sec | Requests that fail to return the report from cache.
Delivers/Sec | Report deliveries from any extension.
Events/Sec | Events such as SnapshotUpdated and TimedSubscription that are processed.
Memory Cache Hits/Sec | Number of times reports are retrieved from in-memory cache.
Memory Cache Misses/Sec | Number of times that reports cannot be retrieved from in-memory cache.
Next Session Requests/Sec | Requests for reports that are open in a session.
Report Requests | Reports that are active and handled by the report server.
Reports Executed/Sec | Reports successfully generated.
Total App Domain Recycles | Application domain cycles since the service started.
Total Cache Flushes | Report server cache updates since the service started.
Total Cache Hits | Reports processed directly from memory cache since the service started.
Total Cache Misses | Reports not processed directly from memory cache since the service started.
Total Deliveries | Reports delivered for all extensions by the Scheduling and Delivery Processor.
Total Events | Events processed since the service started.
Total Memory Cache Hits | Cached reports returned directly from memory cache since the service started.
Total Memory Misses | Cache Cached reports returned directly from memory cache since the service started.
Total Processing Failures | Processing failures that have occurred since the service started.
Total Rejected Threads | Processing threads switched from asynchronous processing to synchronous, serial processes.
Total Reports Executed | Total number of reports that ran successfully since the service started.
Total Requests | Total number of all requests made to the report server since the service started.


Take Away

There are several uses for Reporting Services, including processing application reports, merging data from multiple sources into a single view of data, and, as we have done in this chapter, system status reporting. But one of the most powerful uses of this technology is as a destination for Business Intelligence data. In the next chapter, I explain how all the features that SQL Server 2005 provides come together with Analysis Services to enable a comprehensive, strategic view of the data locked in your application databases.

One of the key components of that system is the ability to show everything from metrics that show how an individual site or plant is doing against a defined requirement, called a Key Performance Indicator (KPI), all the way out to a reporting showing the results of multidimensional analysis on the entire organization.

In this section, I explain how you can use Reporting Services as the "front end" of a Business Intelligence (BI) layout, called a landscape. I create a simple report using a wizard, much as I showed you earlier in the chapter, but with a few changes for an Analysis Services data source.

The design for a Reporting Services deployment changes for a BI landscape. Although you still want to follow the guidelines for evaluating the load on the Reporting Services servers from the database and source data standpoints, the source data you are pointing at with this application involves consolidated, aggregated data rather than multiple text files, Excel sources, and the like. You are much more likely to use a single server to access the data for the reports.

In fact, you may not connect to a database server directly at all. In a BI layout, you will more often use the Analysis Services server to provide data to the report.

In this case, I use a wizard that connects to an Analysis Services cube of data. In the next chapter, I describe creating the cube. Because you do not have the cube I am working with configured on your system, I explain each step as I go.

In Figure 9-50, I have opened the Business Intelligence Development Studio and selected the Report Services Project Wizard. I will name this project Cube Access.

Figure 9-50.

[View full size image]

In the first few steps of the wizard, I am asked to select the data source, just as we did in the report project earlier in the chapter. This time, however, I need to pick the Microsoft SQL Server Analysis Services type of source. I will name the data source AW Cube. In Figure 9-51, I enter the Analysis Services server name and select the cube I deployed from there.

Figure 9-51.

[View full size image]

I click Next to design the query and then click the Query Builder button. The data selections shown in Figure 9-52 are much different than what we dealt in relational data. For now, I make the selections to show a promotion name that ran, along with the date that items shipped, as well as the revenue. I am trying to show a correlation between the promotions that we paid for and the return we received.

Figure 9-52.

[View full size image]

In Figure 9-53, I select the Matrix type of report, and then lay out the fields I chose in the last step. Depending on how the users want to see the data, you can set the columns and rows to your liking. In my case, I have set the promotion across the top and the dates on the left, with the dollar figures in the detail area.

Figure 9-53.

[View full size image]

The next few screens set the formatting and layout, and for this example I take the defaults. On the last entry panel, I am asked to set the server name and a folder to hold the report. I set the names there and then finish the report. When I am dropped back in the designer, I build and then deploy the report.

From the report shown in Figure 9-54, it seems that the Volume Discount 11 to 14 promotion seemed to be the big winner. More analysis is warranted to ensure that other factors did not play a big part of the increases shown, but that is the point of a good BI system: It always forces more questions.

Figure 9-54.

[View full size image]


Chapter 10. Analysis Services

Analysis Services is an engine that provides near-real-time and real-time business information for your users. Business intelligence (BI), also known as business information, is the process of pulling useful information from the data your systems store. In this chapter, I explain what BI is along with some common business intelligence scenarios and how you can use SQL Server 2005 features to implement them. The first section of the chapter is not meant to be an exhaustive treatment of the subject; but you do need a background on what a BI system is and a basic understanding of how it works to manage it properly. I give you an overview that will bring you up to speed on the major concepts.

To create the physical layout that forms a BI "landscape," you will use what you have learned in previous chapters. Everything from sizing a system, how Integration Services works with the process, and how you can present your data by using what you know about Reporting Services is involved with a BI implementation.

The second part of the chapter deals with administering the BI system. I also explain the security implications that are unique to Analysis Services, as well as the monitoring and optimization techniques you can use with it.

There is a great deal of information to cover on Analysis Services. In this chapter, you will get a good overview of the concepts involved as well as a rundown on the parts of the product and how you can manage them. I also explain a little about the maintenance for a multidimensional database, and I finish up with a complete example of the entire process.

Although this chapter introduces many new concepts, if you want to implement a full BI system, you should read up on Analysis Services in Books Online and then check out a few good books on the subject. Even then, it is best to get experienced help in creating a BI system.

The best definition I have heard for BI is the act of turning data into information. That is a different process from storing, securing, and controlling application data as explained throughout the book. Using BI is more strategic than an online transaction processing (OLTP) system.

A good way to visualize this difference is to think of how you interact with your bank. Your bank can tell you how much money you have in each of your accounts, they can move the money around for you, and they can even report it in paper money or coins. You can have the bank change the money into another currency, they allow you to remove the money you have, and they accept new deposits. There is almost no limit to the way they can manipulate the cash and credit in your accounts.

What the bank cannot do is tell you what the amount in the accounts means. They just do not know whether having a certain amount in an account is good or bad. They will not tell you that you need to move money from one account to another, or to invest in a certain real estate transaction. The interpretation of the amounts and how the money should be used is up to you.

An OLTP database-based system is similar to what the bank does. It stores data; allows inserts, updates, and deletes; and can even change the format of the data. However, a standard database application does not try to say what the data means. Just like the bank, interpretation is up to the end user.

A complete BI system does, however, try to determine the meaning of data. By using various tools and combinations of the data along with numeric analysis, the system explores trends, aggregations, and key measurements to show the users how the data should be interpreted. That is a tall order, but as you will see, there are various mechanisms that allow the users to automatically derive meaning, or information, from data.

Microsoft SQL Server 2005 contains everything you need to collect, store, analyze, and present data in this way. Microsoft's implementation of BI goes further than many systems try to present. In most BI systems, the focus is on strategic, analytic, corporate-level data. Microsoft extends the BI definition to include strategic, tactical, and even operational data.

OLTP systems have functions that almost everyone in the company uses, but there is a different audience for the BI system. The user for BI systems is most often a strategic decision makerin most organizations, the middle to top-tier management staff. These users need special training to work with a BI system because the system is more of a tool than just another way of creating output.

A defining measure of a "deep" BI system is that it is strategic and not as much tactical in use. Strategic decisions involve whether to expand the business, what products or services to offer, and how best to meet those goals. Tactical decisions involve pricing, personnel, and production or service rates. Although SQL Server 2005 can be used for tactical and even operational data, the focus is usually on strategic.


Business Intelligence Scenarios

I am focusing in this book more on maintenance and optimization of SQL Server 2005 features, but I do spend a little time in this chapter on designing BI systems. Before you implement a BI solution, you should learn how to complete a thorough design process that always begins with finding out what questions need to be answered. What I give you here is a brief overview of the process so that you can understand how all the pieces fit together in Analysis Services. I do that by explaining a little theory and then working through a complete scenario in the "Take Away" section at the end of the chapter.

All along, I have been using the words business intelligence system to describe the tools that provide the ability to make strategic decisions. Many people equate a particular function or tool with BI, but a proper BI system is made up of several tools, an effective data strategy, and a good design.

The first part of that design is a proper data structure on the OLTP systems. You must have a stable, secure landscape that stores the day-to-day operational data, called a source system, before you begin. If you try to build strategic information from a faulty origin, you will end up enabling bad decisions.

The next layer in the BI system is a robust reporting system. This is either a software package that accesses the source system directly or an entire subsystem that extracts data from the source system into a reporting database. In the latter case, the database is physically structured differently than the original data to facilitate reporting. In some cases, this is called an operational data store (ODS).

The definitions I use in this chapter might differ from the ones you will see in other books or vendor documentation. In the many years I have used BI systems, everyone seems to describe things a little differently. The concepts I present here follow the best practices of BI systems as I have seen them implemented.

Many smaller firms use a combination of the source system and a reporting system as their BI system. Spreadsheets or specialized reporting tools provide a good basis for manual analysis.

At the next layer are the data marts. These systems collect data from several source systems and combine the data into a single format. To do that, data must often be homogenized so that it reports the same meaning. For instance, in one source system, the product called Buckster's Coffee uses the product code 12345, and in another source system, the product number is identified as A13GFF.

To show how many of these things are sold, you need a way to put them together. One strategy to homogenize the two part numbers is to convert them to an entirely different part number with an Integration Services process that is coded to replace the part number based on a rule. Another method for homogenizing the data is to create a third table to act as a joining entity. Both approaches are valid based on your reporting needs from the data mart.

As the organization grows, it requires more formal analytical tools. These tools are essentially "blank slates" on the screen that allows data selection and presentation. The reason a tool is needed rather than just a report is that the analysis is on more than one axis, or dimension.

To understand how dimensions work, it is helpful to think of a standard spreadsheet. A spreadsheet (as well as a paper report) shows data on two axes at a time. Imagine a typical spreadsheet with rows and columns. In the columns, you can visualize store locations. Along the rows, you can visualize product names. In this case, the store locations and product names are dimensions of what you want to see.

The intersection of the rows and columns (the cells) shows a number indicating the amount of a product sold at a store location. In other words, where the dimensions intersect each other, you have a fact about that dimension. In this example, you can easily imagine the number 300 in a cell where Tampa and Buckster's Coffee intersect. That particular fact means that 300 cases of Buckster's Coffee sold at the Tampa location.

Now imagine that you also want to see a further breakdown of the data, involving when those products were sold, perhaps by month or quarter. That is a little harder to represent with a spreadsheet, although you could add a set of rows that shows each month, and perhaps even another for the quarters. The spreadsheet is still useable, but is becoming more cluttered and harder to interpret.

But now, imagine you want to know the name of the products and the location they sold at, by month and quarter and by salesperson. A paper report with this information becomes cumbersome to navigate, and a spreadsheet would need several tabs to show all the permutations.

You need something that enables you to "flip" through these all dimensions, showing calculated facts about a view of the dimensions dynamically. This is also called "slice and dice." This component of a BI system is called online analytical processing (OLAP). OLAP systems store dimensions and facts in a structure called a cube, because a cube shape represents data with more than one axis. OLAP systems collect data from the ODS, data mart, or source system layer and aggregate the various facts and dimensions into possible combinations. This is all done online, in an investigative manner.

Another part of the BI system is a facility for data mining. Data mining involves interactively searching the data for statistically relevant patterns. A data mining system uses sophisticated algorithms that remove false correlations and show the user things such as products that sell well together, how effective a marketing campaign is, and more.

At a higher level, many mid- to lower-level managers want to know how they are doing against an established norm, such as growth over last year's sales and so forth. The system that provides this information is called key performance indicators (KPIs).

All of these disparate parts make up a BI system. When I use the term system here, I am describing the conceptual parts of a group of data and software that provide strategic, consolidated, analytic data to management users. The physical parts of a BI system are called the BI landscape. The landscape includes the source system software and databases, reporting servers, processing engines, and display mechanisms.


Implementing Business Intelligence in SQL Server 2005

Microsoft SQL Server 2005 provides a complete set of tools to move data from source systems through to the analysis and then on to the presentation layer. In fact, you can implement a BI system entirely within the toolset that ships with SQL Server 2005 without having to buy anything else.

SQL Server 2005 handles source systems for relational OLTP data, SQL Server Integration Services (SSIS) provides the extract, transform, and load (ETL) process from the source system into the ODS or data mart layers, SQL Server Analysis Services (SSAS) serves as an OLAP and data mining engine with multidimensional storage, and SQL Server Reporting Services (SSRS) is a great presentation tool. Using built-in Web technology, you can extend the presentation to an active tool for the users.

That is not to say that you should try and do all that on one server. In fact, even a small BI implementation needs a separate server for the three primary parts of the system: the ETL engine and database, the Analysis Services system, and the Reporting Server. In larger situations, you have even more separate servers, based on load. Although it is beyond the scope of this single chapter to detail a large-scale system, you can follow the same process that I explained in Chapter 1, "Installation and Configuration," on sizing a database landscape and the information you find here to create a good BI landscape strategy.

Before you begin, you should have a good grasp on all the previous information from this book. The concepts involving sizing, monitoring and optimization, maintenance and security, all the way out to Reporting Services and SSIS are used when you use SQL Server 2005 to create your system. For more information on a particular part of the system, review the chapters that deal with those subjects.

SQL Server Database

At many stages of the BI landscape, you can use SQL Server 2005's relational database engine. Your OLTP applications make up the source system data that form the core of the analytical data further down the line. Certain spot measurements that form small parts of your BI implementation are available directly from source systems, so long as they do not place an undue burden on performance.

You can also use the database engine store the data for your ODS. This is the near-concurrent data your users need for reporting, so that they do not place an additional load on the OLTP system. Data marts can also be stored this way.

Analysis Services, as I explain further, has its own database engine. The demands for processing multidimensional storage differ from those required by relational data processing used by OLTP applications. Although Analysis Services has its own storage, the database that holds the dimensions and facts is a standard relational engine, even if the tables are structured a bit differently. SQL Server 2005 works well for these kinds of tables.

Source Systems

Most OLTP applications store their data using a database engine, whether it is SQL Server or some other Relational Database Management System (RDBMS). As you have seen throughout this book, SQL Server 2005 works well in that area. If you are using SQL Server 2005 to implement a BI solution, the advantages go even further, because the source for the BI data is "native" and several wizards within Integration Services and Analysis Services can automatically detect relationships from the database structures and rules it has. That is not to say that you cannot use other systems for your source data, it just means that the SQL Server 2005 systems will work that much better in the landscape.

Whether you use SQL Server 2005 or not, the important considerations on the physical plant of the source systems are the extract level and schedule and where you place the logic for the transformations and homogenizations you will need for the next levels in the system.

The extract level is the amount of data you will extract from the source system at a single pass. The more data you get, the more locking and other performance considerations you face. Closely tied to that decision is the extract schedule. The more frequent the extraction cycle, the less data you need to get during that cycle. On the other hand, the more frequent the schedule, the more often you interrupt production time.

Another important factor to consider is where you transform and homogenize the data. You are eventually going to need the data in the same data type, and by the time it is combined with other systems, it needs to have the same meaning. I personally favor transforming the data as late in the process as possible; in your implementation, however, it may need to be transformed on the way out of the source system. If you need to do that, make sure you logically and even physically separate that process so that you do not adversely affect production.

The Operational Data Store and Data Marts

The ODS and the data mart database structures are still relational. That means that both of these layers can be implemented in a standard SQL Server 2005 database.

The primary differences in an ODS or data mart database and an OLTP database is that the former are used in a different way. Users in these systems are reporting out of the database, so it has less-constant inserts and more reads. That means that indexes and denormalized tables are more advantageous.

DBA 101: Denormalization

In normal OLTP databases, the data structures follow certain rules, called normalization. These rules require that each unique piece of data is stored only once, that there are relational tables to join these unique data elements, and so forth. Normalization provides a stable, rapid-insert environment.

In reporting instances, it is quicker to prejoin the data so that the joins are not necessary. This relaxation of relational rules is called denormalization. It is acceptable to do this in a reporting system because data is not often edited by the end user, so the risk of having an inconsistent system is reduced.

Even though users are not entering data into the ODS or data mart system, it does get loaded periodically. Because the load represents all the changes since the last load, the loads might be adversely affected by a lot of indexes. During the load, it is common to remove the indexes or bypass them and then rebuild the indexes when the load completes.

Another difference between this layer and the OLTP source system is that the data can grow quite large in an ODS or data mart layer. In the OLTP system, you store the data once for each row. If that row changes, you only record the end result. In an ODS or data mart, you often store both events. You need to know that you sold two items yesterday and today you adjusted it to three. Because of this historical recording, and because you are integrating several sources systems at this layer, your data will grow, most often by a great deal.

Although this chapter does not deal with BI design, as the database administrator (DBA) you need to consider data archiving at this level. Most of the systems I deal with that are "in trouble" have massive amounts of data and no clear archiving strategy.

Online Analytical Processing and Data Mining Databases

Before the OLAP system builds multidimensional database aggregations, the data needs to be in a format that allows the system to "pull" the information it needs. The data is stored in a relational database but has a completely different layout from the original data.

It is beyond the time we have in this chapter to completely describe how to design these "base" tables, but it is worth understanding what the shape of the data looks like so that you can administer it properly.

Earlier I mentioned a spreadsheet with rows and columns that indicated the dimensions of data you are interested in knowing more about. I also mentioned that the numbers inside the spreadsheet are called facts about those dimensions. Those two constructs for the core of what you need to know about the cube origins.

In the spreadsheet I mentioned, the data looks like this.

 | Buckster's Coffee | Tampa's Best Blend | Mocha Rico | Morning Glory Tea
Tampa | 300 | 150 | 150 | 79
Seattle | 500 | 170 | 100 | 400
London | 278 | 150 | 100 | 230
Brugge | 320 | 100 | 80 | 234
Mexico City | 234 | 200 | 90 | 125
Penang | 250 | 350 | 100 | 600

This format also looks like the result from a standard query from a set of relational tables or from a view.

To add other dimensions and analyze data from this layout, you need a way to break out the dimensions from the facts. You can do that with a star schema. A star schema is a set of two kinds of tables: dimensions and facts. By using a foreign key in the fact table to a primary key in the dimension tables, you can get the data ready for processing by an OLAP system. Let's add the time dimension to that example to see how a star schema would look for this data.

Region Dimension

RegionKey | Region
1 | Tampa
2 | Seattle
3 | …

Product Dimension

ProductKey | Product
1 | Buckster's Coffee
2 | Tampa's Best Blend
3 | Mocha Rico
4 | …

Time Dimension

TimeKey | Date
1 | 01/01/06
2 | 01/02/06
3 | …

Fact Table

RegionKey | ProductKey | TimeKey | Units
1 | 1 | 1 | 300
1 | 2 | 1 | 150
1 | 3 | 1 | …

By joining these tables together using the keys, you can see how you can produce the same output as the earlier query and also how you can add other dimensions. In a moment, I explain how Analysis Services uses these tables to create a cube.

The star schema can also be extended to include dimensions that point to other dimensions. This is called an extended star schema or sometimes a snowflake schema (from the shape that the tables take). You can use SQL Server's relational engine to store the star or snowflake schemas, as I show you in the example in the "Take Away" section at the end of the chapter.

SQL Server Integration Services

You can use SSIS as I have already described in two main capacities. The first is to pull data from the source systems into the ODS or data mart layers. The sources can range from almost any relational system such as Oracle or IBM's DB2 all the way to word processing documents, text files, and spreadsheets.

The second way you can use SSIS is to put the data into the star or extended star schemas. Of all the tasks in a BI system, this is one of the most tedious processes. Although it is a time-consuming task, it is not that difficult to design. It is a matter of carefully evaluating the source of the data for the star schema and then using SSIS steps to fill out the tables accordingly.

Although I have focused on the star and snowflake schemas in this overview, you do not have to map the tables this way to use them in Analysis Services. In fact, you can map an Analysis Services directly to an OLTP database. Analysis Services differs from other BI solutions in that it focuses on dimensions and facts, not a particular schema.

Once again, one of the most important factors is the schedule. You will need to consider the ETL process from the source systems and the creation of the dimensions and facts that form the base of the cubes to ensure that you do not overload one part of the SSIS infrastructure. You may have to use more than one SSIS server, depending on how much data you are trying to integrate.

Later in the chapter, I explain how you can use SSIS to process cubes and perform maintenance tasks.

SQL Server Analysis Services

The heart of the business intelligence system and landscape is SQL Server Analysis Services (SSAS). It provides the OLAP processing and distribution functions as well as the data mining engine and KPI mechanisms.

It is simple to create a cube or data mining display using SSAS. If the data is properly structured into dimensions and facts with the proper relationships, just a few clicks creates an entire cube structure. Microsoft includes "Intellicube" technology that determines all the relationships it needs from the fact and dimension tables.

SSAS can also create levels of the data such as region, country, and state (called attributes) automatically. It also has a function to create new data by combining columns in a dimension called calculated members. All in all, it has everything you need to take the dimensions and facts and turn them into a BI presentation.

Microsoft includes many wizards to help you create these objects. These wizards simplify the process and hide many of the complex underpinnings of the process, but that can be good and bad. It is good because you can quickly assemble a cube or data mining solution quickly, but it is bad because if not designed properly, the information may not be useful. If you have issues later with the design and you have relied on the wizards to create everything, you might not understand how to correct the problem. Because these systems can generate so much processing and storage, you can create an inefficient process quite easily. Fortunately, the wizards in SQL Server 2005 create structures and objects that you can edit later.

Even so, you should experiment. This chapter gives you enough background to enable you to be a little more curious and readies you to intelligently read up in other references. Always practice on your test system and do not be afraid to explore. Microsoft has done a great job with the wizards; in large part, they just automate the minutia. When they are done, the model is fully extensible.

Business Intelligence Development Studio

To develop your Analysis Services objects, the Business Intelligence Development Studio carries on the same type of development interface you see in Visual Studio or the SQL Server Management Studio. Just like the SQL Server Management Studio, you use this tool to create the content for your system.

If you installed Analysis Services, you will find the tool on the Windows Start menu under the Microsoft SQL Server 2005 item. Once you open it, you will see a screen similar to Figure 10-1.

Figure 10-1.

[View full size image]

The windows, tabs, and menu items work the same way in this tool as they do in the SQL Server Management Studio; as the name implies, however, the primary use is to develop the BI system. Whereas the Management Studio tool lets you jump right in and create queries, open tables, and generally explore, everything in the Business Intelligence Development Studio requires a project. A project here has the same meaning that it has in Management Studio: It is a collection of connections, databases, and outputs that belong to the same process.

You can also use the Business Intelligence Development Studio to create Integration Services and Reporting Services projects as well as database migration and other templates. We will do that in a few moments.

In the "Take Away" section, I show you how to create a simple Analysis Services solution. If you have not created an Analysis Services database by the time you review that section, you will not be able to follow along on your system, so I will provide a few more screenshots in this section than normal.

Using the Business Intelligence Development Studio, the first place to start is to create a project, as shown in Figure 10-2. This is just a collection of Analysis Services connections and objects that are grouped together. It is similar to creating a project in SQL Server Management Studio, except there it is optional; here it is not.

Figure 10-2.

[View full size image]

The first object in a project is a connection to your data. In Analysis Services, this is called a data source. Data sources can connect to everything from SQL Server to Oracle databases, from text files to spreadsheets, and many others.

Although the screen looks quite involved, it is a fairly simple process to set up a connection to a source. You can work through an entire example in the "Take Away" section at the end of the chapter. You can create as many connections as you need for a project.

The next step is to create a data source view, as shown in Figure 10-3. This is a further refinement of one or more data sources into a single view. It is a logical representation of the data that the Analysis Services database will use. It helps to insulate the users from the structure of the underlying source data.

Figure 10-3.

[View full size image]

After you have created one or more data source views, you can create a cube. You can create a cube structure by adding in the dimension and fact tables from the star-schema type databases.

After you have done that, you can refine the structure by adding hierarchies, calculated members, and other options. Microsoft calls this collection of dimensions, facts, and other elements a Unified Data Model (UDM). The UDM simplifies how the users think about the data. You can see a sample UDM in Figure 10-4.

Figure 10-4.

[View full size image]

The next step is to create the storage for the OLAP database. You design storage using partitions, which describes the storage type for the database (more on that in a moment) and how many aggregations the engine will process. You can see these choices in Figure 10-5. I cover the choices for storage in the "Management" section a little later in the chapter.

Figure 10-5.

[View full size image]

You can create other limitations for the views on the data, called perspectives, shown in Figure 10-6. Perspectives allow you to limit the data based on groups of people that need to see it. After you create the perspectives and make any final refinements, you apply security to the various objects using roles.

Figure 10-6.

[View full size image]

With everything defined, refined, and secured, you need to build and deploy the cube and then process it. The building process shown in Figure 10-7 reads the UDM definition, creates the multidimensional database, and calculates the aggregations. Deploying places the solution on the production server and publishes the results for access. Processing fills the aggregation tables with data. I show you an example of that process in a moment.

Figure 10-7.

[View full size image]

Accessing the System

After the cube has been built, deployed, and processed, you can access it from Analysis Services in one of three ways: inside the Business Intelligence Development Studio, using an application that your developers create, or through a third-party application.

Using the Business Intelligence Development Studio is primarily for the designer's review before they release the solution to the users. The last subtab in the Cubes screen allows you to pull the measures (facts) and place them in the center part of the screen. You can drag the dimensions to the top and left side of that panel shown in Figure 10-8 to create a multidimensional view of the data.

Figure 10-8.

[View full size image]

Although this is an easy way to move through the cube and other BI data, you will not install the Business Intelligence Development Studio on every user's system. The end users will access the data using other tools, in what is referred to as the "clicks" paradigm.

The clicks paradigm refers to how many clicks it takes a user to receive the BI data. A "0-click" application is where the data finds the user, normally in an e-mail or over a pager. The user does not have to click anything to get the data. In a large organization, this is normally the largest distribution of the data, but it is often tightly targeted. It is usually used for KPIs or a single metric.

A "1-click" presentation requires the user to access a static report on the BI data, such as on a portal or corporate share. A "2-click" presentation involves a similar view, but it allows the users to change parameters such as a date range or a region selection.

A "3-click" presentation is where a full tool is used. This type of tool, whether locally developed or provided by a third-party vendor, has a "pallet" of the measures and dimensions from the Analysis Services system. It looks similar to the Business Intelligence Development Studio but is for presentation, not design. This is normally reserved for the highest levels of management because it provides a great deal of strategic value and takes a higher level of training to use properly.

In all of these presentations, developers can access the data using an application programming interface (API). Query languages are available for cube presentation called multidimensional expressions (MDXs) and for data exploration called data mining expressions (DMXs).

You can access an Analysis Services application using an XML implementation called XML for Analysis (XMLA), manage the server using analysis management objects (AMO), and code stored procedures for it using the Common Language Runtime (CLR). As an aside, Microsoft Office products such as Excel also hook into Analysis Services 2005.

The list of features and concepts go much further, but this overview explains what you will need to consider as you prepare to manage it. I explain more about that part of the process in the next section.

SQL Server Reporting Services

In the preceding chapter, I explained how Reporting Services works and its many uses. I mentioned briefly that it provides a great way to view the output from Analysis Services. I also mentioned in the preceding section that you can develop a 1- or 2-click report to show the data from Analysis Services.

When you create a report against Analysis Services using Report Services, one of the advantages you have is the variety of outputs you have access to. You can present or export the data to the Web using HTML applications, send it to an image, save it as a PDF file, or send it on to an Excel spreadsheet or comma-separated value (CSV) file. Using a three-step process, you can send the report to a 2-click presentation, out to a 1-click presentation on a Web site or portal, and then on to a 0-click application to other users.

Once again, you must consider all the aspects of a BI system for presentation. SQL Server Reporting Services has many wizards that can guide you through creating a report that attaches to Analysis Services, as I demonstrated in the preceding chapter. Each tool should be suited to the presentation the user needs. Reporting Services is a passive viewing method of the data because it does not allow a great deal of control during presentation. In contrast, a 3-click tool is more interactive, allowing the user to move through the data more actively. That is not to say that Reporting Services does not deserve a place in the BI presentation layer, just that it is better suited to static data than interactive mining. I have seen BI implementations grow over time to use hundreds of reports to show the data when a single data browser tool is a better choice.


Analysis Services Maintenance

So far, I have explained the types of things a BI developer focuses on. As the DBA, you will need to concentrate on a few areas for the maintenance of the system: storage, account management (security), job processing, synchronizing, backups, and then monitoring and optimization. Before I explain those, it is important to understand a little about the way the product works.

Just like the design portion of this chapter, multiple factors affect how complex the maintenance feature descriptions become. I cover the base concepts in this section, but based on your design you should keep in mind other maintenance considerations, especially in the partitioning and processing areas.

During many of the maintenance tasks, you will leave the Business Intelligence Development Studio and open the SQL Server Management Studio. I point out the switch as we go because the interfaces have a similar look and feel.

Infrastructure

The Analysis Services in SQL Server 2005 is implemented as a Windows service. You can have more than one instance of the software running as multiple services, similar to the SQL Server 2005 services that make up instances. You can use the SQL Server Configuration Manager to control the services and ports for Analysis Services. If the server is used as an Analysis Services server, you need to set that service to begin automatically on system startup. That may seem to be an obvious point, but I have responded to calls that the system was not functional only to discover the service was not running.

Analysis Services listens on TCP/IP ports for client connections. The default ports are 2383 for the Microsoft SQL Server 2005 Analysis Services default instance, port 2382 for the redirector for other Microsoft SQL Server 2005 Analysis Services instances, and 2725 if you upgraded Analysis Services from Microsoft SQL Server 2000 Analysis Services instances.

In previous versions of Analysis Services, processing and caching were shared between the server and the client. This is now changed with SQL Server 2005 Analysis Services, which is based entirely on server-side processing and caching. This allows for thin-client applications but takes a heavier toll on your server. In a moment, I detail some performance counters you can use to monitor the system so that you can proactively grow it as the load increases. The point is that the server you specify for Analysis Services needs to be a large-capacity server from the processor and storage standpoint.

Storage

Understanding the storage for a BI system can be a little difficult at first, especially if you depend on only one source for your information. Books Online, for instance, describes OLAP storage in various sections, and sometimes it is not clear how everything fits together. The reason is that the storage does not involve a single concept like a relational database does. In a standard BI implementation, the data is spread into three places. Let's take a look first at where the data lives, and then I will explain a little more about how you handle each one of them.

The first set of data for your BI landscape is the source systems, normally stored and controlled in relational database engines. As I explained earlier, an ELT process copies and changes data from the source systems to a reporting structure, either an ODS or data mart layer. Those databases are also stored and controlled by relational engines.

It is the next layer that gets a bit murky. In most of the graphics that explain a BI system (mine included), there seems to be only the source system, the ODS, data marts, and on to the data warehouse, and then to the OLAP system. You might think that the data, both in the star-schema form and the multidimensional aspects, along with the aggregations are in the same place.

In fact, the star-schema data is stored and managed separately from the multidimensional data and the aggregations. Storage for the dimensions and facts is relational, and is designed, stored, and managed in a relational database, using relational database tools such as SQL Server Management Studio. The Universal Data Model (UDM) elements and aggregations are designed using the Business Intelligence Development Studio, stored in Analysis Services, and managed using a combination of Business Intelligence Development Studio and SQL Server Management Studio. I show you all that in a moment.

Storage for the information within Analysis Services differs from the relational engine because it can be placed in different locations based on storage and performance needs. Storage is controlled by partitions, which is a logical and physical group of storage types, based on measure groups. Because of this, BI structures are processed at the partition level.

When you create the BI solution, you assign it to a partition and then design the storage for it. The screen to do that is shown in Figure 10-9.

Figure 10-9.

[View full size image]

When you click the ellipses next to the aggregations, you are presented with three basic choices for the storage: MOLAP, ROLAP, or HOLAP, as shown in Figure 10-10. These choices have differences in where they store data and how much space the data takes in those locations.

Figure 10-10.

MOLAP stands for multidimensional OLAP, which stores the aggregations in Analysis Services. This is the fastest option but takes the most space to process and during the processing affects the load on the Analysis Services system.

ROLAP stands for relational OLAP, and this option stores the aggregations in the relational system. This has the advantage of lightening the storage and processing load on the Analysis Services system, but at the expense of the relational system. You will often see this arrangement in smaller implementations.

HOLAP stands for hybrid OLAP, meaning that some of the aggregations are stored in the multidimensional system and some in the relational system. This forms a good compromise for both storage and processing.

There are degrees of each of these settings, as displayed in the previous graphic. These nuances have to do with the level of caching the setting provides. The more caching, the more drive space and processing time is needed, but the faster the results for the users when they request them. It is all a tradeoff, with the leftmost setting (ROLAP, low caching) the most beneficial for processing time and space, and the rightmost setting (MOLAP with high caching) the most beneficial for response time.

The diagram in Figure 10-11 shows the entire process and storage options laid out in a single view.

Figure 10-11.

The source data is located on the far left of the diagram in a RDBMS. An ETL process pulls the data through various layers (such as ODS and data marts) to either the data warehouse or some other RDBMS designed to store the facts and dimensions. If you select MOLAP, the aggregations and metadata needed by the OLAP engine are stored in the multidimensional database management system. If you select ROLAP, the aggregations are stored in the RDBMS. If you select HOLAP, it is stored in both.

With those concepts in mind, let's examine how you create and control the storage for the aggregations. Remember, you use the same techniques for the relational data that you have learned throughout this book.

To design an Analysis Services project, you use the Business Intelligence Development Studio. To manage the storage, we will switch back to SQL Server Management Studio.

Instead of choosing a database engine server type of connection on startup, change to an Analysis Services connection, type in the server name as shown in Figure 10-12, and press Enter.

Figure 10-12.

In Figure 10-13, I have expanded all the objects in this screen so that you can see the similarities and differences between this view and the interface in Business Intelligence Development Studio.

Figure 10-13.

[View full size image]

This is the location where the administrator can create or set data sources, create views, and even design cubes. You can do all of those things in the Business Intelligence Development Studio, too, but SQL Server Management Studio also provides access to back up and restore the databases and control security. If you click Connect and select Database Engine as the server type, you can manage all of the levels of the BI landscape from one location, assuming they are in Microsoft SQL Server.

To work with the storage for the Analysis Services system, drill down to the database name you are interested in and then open the Cubes object. Open the cube, and then drill down through the Measure Groups. Open the Facts in the cube, and then open the Partitions object. If the cube has been processed, you will have at least one partition. Right-click the name of any partition, as shown in Figure 10-14.

Figure 10-14.

[View full size image]

From here, you can create a new partition, process the partition, merge it with another, change the aggregation methods, or delete it. To set the storage location for the partition, select Properties from the menu. In the General section shown in Figure 10-15, you can view and set the storage location.

Figure 10-15.

[View full size image]

There is a bit overlap here between what you are able to do in the maintenance phase versus the design phase. It is important that you decide early in the process where you will design and initially create the cube's storage. For the relational parts such as the source system data and the dimensions and facts, you will design, create, and manage the system using SQL Server Management Studio, assuming those objects are stored in SQL Server databases.

For the Analysis Services objects, you can design data sources, data source views, dimensions, measure groups, and partitions in both Business Intelligence Development Studio and SQL Server Management Studio. Significant management tasks such as backing up and restoring are done in SQL Server Management Studio. My recommendation is that you do all design in Business Intelligence Development Studio, including the partition design. The process flows better there. Most management tasks should be accomplished using SQL Server Management Studio, because you have a single place to connect to all the databases in the process.

One of the primary tasks you will be asked to perform is to process the storage after it has been designed and built. I show you how to do that manually using SQL Server Management Studio and automatically using an Integration Services package in a moment. I will explain the various options for processing the partition. This is another area that can be accomplished using Business Intelligence Development Studio; because I consider this a management task, however, I recommend using Management Studio. I have found it cleaner to use Business Intelligence Development Studio for design and SQL Server Management Studio for management and maintenance.

The one exception is in the security area. It is not that you cannot control the security from SQL Server Management Studio, because it is easy to do from there. The consideration for security actually deals more with the way BI is used and controlled. Often in a BI system, the developers are more familiar with the business needs and the rights that groups within the business should have. If the administrator is not part of the design process, it is a two-step task to have them create and control the security. For example, the developer might create a measure that deals with employee salaries. In some cases, a carefully crafted query might be able to show the breakdown of the salaries in the company, so the group that is allowed to see that information must be cleared through the human resources department. The same information can be designed in such a way as to show only an overall figure, and so a wider group is allowed to view it. Unless the administrator is part of that design discussion, the administrator might not be able to assign the rights properly. I explain the security process further in the next section.

Another option you have in dealing with the storage options is to enable a writeback operation on a partition. A writeback operation allows the users of the UDM to alter the data in the fact tables. In fact, the data is not altered at all, but a separate table is created to hold the changes, which the users can view along with the "real" data.

To enable a partition for writeback, drill down to the partition and right-click the Writeback object you see below it. Select Enable Writeback… from the menu that appears. The screen shown in Figure 10-16 will open.

Figure 10-16.

[View full size image]

From here, you can set the table for the altered data and its corresponding data source. After you have created a writeback in a partition, you can disable it if you find it is not being used or you can convert it to a new dimension from the same right-click menu.

Security

The security for Analysis Services is a bit harder to find than the obvious locations in an RDBMS. It is also a bit different in the concepts than in the SQL Server 2005 RDBMS. The reason for this is that the security in Analysis Services is more report based, because a majority of the work in an OLAP system involves queries, not writing data to the database.

In an OLTP system, you focus more on what the users need to do and then build your objects security plan around those needs. When you build objects in Analysis Services, you design the data structures with the most permissive rights in mind. In other words, you design a cube to have everything possibly needed in it. The security in Analysis Services is based on limiting the access from the total cube. Using various methods, you show only the dimensions, facts, and other data to the groups of people that need to see them.

Another difference in Analysis Services security is that it is based entirely on roles. Roles are Windows user or group accounts grouped together based on a security need. There is no platform-based security as there is in the RDBMS, so you do not create any users in the system that are controlled by Analysis Services.

In many ways, using roles based on Windows authentication makes implementing security much simpler. The process involves just three steps: Create the roles, assign Windows users or groups to them, and assign the roles the object permissions.

Let's take a look at an example of creating a role and assigning rights to a cube. This role needs to have the rights to look at any data in a cube called Sales By Time but will not use any writeback operations. This role will also not process or change the project.

As I mentioned earlier, I recommend Business Intelligence Development Studio to handle the security for the project, but you can certainly use SQL Server Management Studio in your implementation, regardless of who handles the security. In this case, I use the Business Intelligence Development Studio.

Open the Business Intelligence Development Studio, and open the project you are interested in. I am using the example project I created in the "Take Away" section at the end of this chapter. To begin, we need to create a role. Once inside the project, open the Solution Explorer, right-click the Roles object, and select New Role. You are immediately placed in a panel that allows you to enter data about the role. The first thing to do is change the name of the role, because it defaults to Role.role. Right-click the name and change it to what you like, but keep the extension of .role. When you do that, you will be asked whether you want to change the object name, too. Although you can have different names for the object than how you refer to it, I recommend you keep them the same unless you have a different need.

When you rename the role, you should enter some descriptive text about what the role is used for. I have entered some information about the Role on this system in Figure 10-17.

Figure 10-17.

[View full size image]

At first glance, it appears that there are only a few options for a role. You can assign it Full Control permission, which means that it has administrative rights to the solution and can make any change or even delete it. You can assign the role the Process database permission, which means it can start a process operation. The only other permission is Read Definition. This permission allows the role to read the structure of the solution. In practice, this is the permission most of the roles will have. We will come to the data part of the permissions in a moment.

The next subtab is Membership, shown in Figure 10-18. This tab controls which Windows accounts are placed in this role.

Figure 10-18.

[View full size image]

Click the Add button to place any Windows accounts your system has access to in this role.

The next subtab, called Data Sources, sets what this role can see and what rights they have to that source. In this case, I have assigned one data source name and given this role the right to read the data. You can see that tab in Figure 10-19.

Figure 10-19.

[View full size image]

The Cubes subtab shown in Figure 10-20 allows you to set the rights this role has within the cubes. You have a bit more control here because you can allow the role to write data back to the cube or drill through the cube to its source detail. Be careful with this choice because it has performance implications on the network, the Analysis Services system, and even the source system.

Figure 10-20.

[View full size image]

The Cell Data subtab displayed in Figure 10-21 deals with the content of each set of cells within the cube. In this example, I have restricted this role to the count of the Product Class cells. This level of control is powerful and also allows you to set writeback operations or even contingency-based cell access. These are design decisions more than administrative; your developers deal with the way the data is aggregated and displayed long before you get to this screen.

Figure 10-21.

[View full size image]

The Dimensions subtab in Figure 10-22 controls which dimensions this role can access and whether they can read them.

Figure 10-22.

[View full size image]

The Dimension Data subtab in Figure 10-23 is similar to the Cell Data subtab. Although it gets a bit confusing for the terminology because of its multidimensional nature, you can think of this subtab as dealing with the horizontal partitioning of the data. Of course, a dimension exists horizontally and vertically, as well as along a Z-axis, but you can extrapolate the idea.

Figure 10-23.

[View full size image]

Finally, if you have added data mining structures to your project, you can control the access to them through the Data Mining subtab. You can see that screen in Figure 10-24.

Figure 10-24.

[View full size image]

As I mentioned earlier, you can also control and set the security for the model using SQL Server Management Studio. Right-click the Roles object in the database and then select New Role from the menu that appears. You are presented with the same data I just explained in a single set of panels in Figure 10-25.

Figure 10-25.

[View full size image]

Security for an Analysis Systems project is less about the mechanics, which is quite straightforward, than it is about the design. You will need to work closely with your BI developers if you are asked to manage the security.

Job Processing

After you design and create measure groups, partitions, dimensions, cubes, mining models, mining structures, and databases, you build and deploy them to a production Analysis Services server. In your experimentation thus far you have probably been using the Business Intelligence Development Studio on the server; in production, however, BI developers will use the tool on their own workstations. Before the objects are available on the server to be filled with data, the deployment process creates or updates those objects on the server using a deployment script.

Building the solution creates all the structures and metadata on the local system. Deploying the solution loads those objects to the production server. Deploying the objects is a matter of a few clicks within the Business Intelligence Development Studio. You can see an example of that process in the "Take Away" section at the end of the chapter.

After the project is built and deployed, it needs to be processed. Processing the project involves filling the aggregations with data and creating all the links, metadata, and other database activities in the cube. Until the project is processed, there is no data for the users to see. Even after the data is initially filled in, its source changes over time. To bring the project back to a current data state, you process it again. If the structure of the project or an aggregation calculation is changed, you need to process it again, too.

You do not have to process the entire solution at one time. In most cases, only a few things will change over time, so you can process those as the need arises. You can process at the following levels, from the top down:

  • Database

  • Cube

  • Measure group

  • Partition

  • Dimension

  • Data mining structure

You can process each of those objects, at least to a certain degree. The degree, or processing option, affects how much of the structure and data of each object is processed, which impacts time and computing resources. The more depth you use on the processing, the more recent and complete the data is, but the longer it takes. In many instances, you do not have to reprocess everything, just what has changed.

[Pages 552 - 553]

Option | Used For | Description
Process Default | Cubes, databases, dimensions, measure groups, data mining models, data mining structures, partitions | This is one of the most useful options because it automatically detects the state of the object and then performs whatever processing is needed to fully populate it.
Process Full | Cubes, databases, dimensions, measure groups, data mining models, data mining structures, partitions | Completely processes the object, even if it is current. Drops all the data, examines the structure, and repopulates the data.
Process Incremental | Measure groups, partitions | Adds new facts data and processes the partitions associated with the facts.
Process Update | Dimensions, data mining models | Rereads the data and updates the associated dimension attributes.
Process Index | Cubes, dimensions, measure groups, partitions | Creates or, if needed, rebuilds all processed partitions' indexes.
Process Data | Cubes, measure groups, partitions | Processes data but does not build aggregations or indexes.
Unprocess | Cubes, databases, dimensions, measure groups, data mining models, data mining structures, partitions | Drops the data in the object and the objects that belong to it. This option does not load any data; it is just used to free up disk space.
Process Structure | Cubes and data mining structures | This option processes all the cubes' dimensions if that has not already been done. If it has been, this option creates the new cube definitions. In the case of a data mining structure, it populates the structure with data. It is similar to the Process Full option, but it does not process the full data mining models, which the higher level does.
Process Clear Structure | Data mining structure | This processing level is only used to remove "training data" from a data mining structure. You will not often set this level of processing.

By a careful selection of what you need to process, how deep you need to process it, and when, you can create an effective level of currency on the system.

It is important to define what level of currency you want for your system. If you ask your users, they will of course want the latest data, all the time. Because of the amount of data a BI system holds, staying current as of the last transaction is not practical. You will need to work with your developers and users to see what part of the system needs from a currency standpoint. Some data needs to be as close to real time as possible, and others can be weeks or even months old. It all depends on what the data is used for.

You can process each object completely or incrementally using SQL Server Management Studio from the Object Explorer, or you can use Business Intelligence Development Studio. You can also run an XML for Analysis (XMLA) script, or you can use AMOs.

Let's take a look at processing the project manually and then examine one of the automated methods. We begin in SQL Server Management Studio. Right-click the object you are interested in and select Process from the menu that appears. The result on my test system is shown in Figure 10-26.

Figure 10-26.

[View full size image]

Select the processing level you want. If you are interested in seeing what will be affected by the processing, click the Impact Analysis button. When you have reviewed that, click OK. You will get a status screen to show you what the system is doing.

Although this is certainly a simple process, you do not want to have to process each object manually. Although you have several choices to automate the process, as I mentioned earlier, one of the most common is to use SSIS. I explained SSIS full in Chapter 8, "Integration Services," so I do not cover the basics again here.

Using Business Intelligence Development Studio, create a new project. Select an Integration Services project template and name it, as shown in Figure 10-27.

Figure 10-27.

[View full size image]

Move to the Control Flow subtab shown in Figure 10-28 and drag an Analysis Services processing item onto the main screen.

Figure 10-28.

[View full size image]

Right-click the bottom part of the screen shown in Figure 10-29 to create a connection to the Analysis server.

Figure 10-29.

[View full size image]

Fill out the screen that comes up with the server name and security information for the package step. When that is complete, double-click the step, and in the General section, enter the screen name and describe it. Click the Analysis Services object in the left part of the panel and set the processing for the objects you want, with the options you are looking for, as shown in Figure 10-30.

Figure 10-30.

[View full size image]

From there, follow the SSIS processes described in Chapter 8 to set up your complete maintenance with alerting, logging, reporting, and the like. You can use scripting or AMO methods in the package if you want a different grain on the process than the Analysis Services Processing object in SSIS. The advantage for using SSIS this way to automate your Analysis Services maintenance is that you can include lots of other logic and actions in one graphical location.

Synchronizing

In a large-scale BI implementation, you often purchase and deploy three sets of systems: development, test, and production. These systems do not always match in size and speed, primarily because of cost. In some cases, you will place the development and testing systems together, but you should never develop or test on a production system.

You will normally purchase and deploy your development landscape first. You will have a few false starts, things you want to do differently, and rapid changes to the system. As the project progresses and shows promise, you will set up your testing system. Normally, you will take greater care in this system to make fewer changes, and do things the "right" way, based on what you have learned in development. When the testing is complete, you are ready to move to production.

In any kind of a sizeable system, it takes a long time to design and deploy the solution. In addition, you will sometimes want to "refresh" your development environment from production so that you have real data to work with. What's needed is a simple way to move an entire project from one place to another. Microsoft calls this process a synchronization and provides a wizard to help you with the process.

To run the wizard, open the SQL Server Management Studio and connect to an Analysis Server. Right-click the Databases object and select Synchronize from the menu that appears. From there, the wizard guides you through selecting the source server and database and imports the database into the server you are on.

The interesting part about this process is that it can happen live. In other words, users can query the database on the destination server while the new one is synchronized. When the synchronization is complete, the new schema is available to the users.

You can also synchronize the system with an XMLA script:

<Command>
 <Synchronize>
 <Source>...</Source>
 <SynchronizeSecurity>...</SynchronizeSecurity>
 <ApplyCompression>...</ApplyCompression>
 <Locations>...</Locations>
 </Synchronize>
</Command>

Backups

Recall that the BI system has different methods of storage for the various layers. The source systems are backed up by the maintenance set up for them independently. For the ODS, data mart, and data warehouse layers as well as the dimensions and facts storage, you still have the same maintenance and recovery scenarios and tools that the OLTP systems have, because they all use a relational database back end. If these are stored in Microsoft SQL Server 2005, you can refer back to Chapter 3, "Maintenance and Automation," for a complete explanation of how to manage those.

The only part of the storage left involves the UDM information, security, and other metadata within the Analysis Services project and the aggregations. These are all stored within the Analysis Services database.

Referring back to the previous graphic showing the aggregation storage in Analysis Services, the location of aggregations is based on which model you chose for the partition's storage (MOLAP, ROLAP, or HOLAP) and the caching options you selected. If you are using MOLAP storage, the backup of the Analysis Services database is larger because the aggregations and potentially the caching are stored there. In the case of HOLAP storage, the backup will be slightly smaller because the aggregations are spread throughout the relational and multidimensional storage engines. With ROLAP storage, the aggregations are placed in the relational storage, so the Analysis Services database backup is much smaller.

Analysis Services database backups are sent to a file, regardless of which storage method you use. You can also restore the Analysis Services database backup to the same server or another or even to another database within any Analysis Services server. Backups are taken at the database level.

In all cases, you can use manual or automated methods for the backups. For the manual method, open SQL Server Management Studio and connect to the Analysis Services server. Right-click the database name you are interested in and select Backup… from the menu that appears. You are presented with a screen similar to Figure 10-31.

Figure 10-31.

[View full size image]

From here, you can set the location for the backup and set compression or security options. You can also back up remote partitions, which are Analysis Services partitions stored on other Analysis Systems servers.

You have several methods available to automate this process, but one of the simplest I have found is to stay within the previous screen and click the Script button on the upper-left side of the screen. That creates an XMLA script of your choices that looks something like this:

<Backup
xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">
 <Object>
 <DatabaseID>Sales By Time</DatabaseID>
 </Object>
 <File>c:\backups\Sales By Time.abf</File>
 <AllowOverwrite>true</AllowOverwrite>
</Backup>

You can copy this script and use it in an AMO or other connection method, but a great way to automate your maintenance is to use a package within Integration Services.

To do that, open the Business Intelligence Development Studio and create a new Integration Services project or use a current one. Create a new task, dragging the Analysis Services Execute DDL item onto the palette. Double-click the item to bring up the screen shown in Figure 10-32.

Figure 10-32.

[View full size image]

On the first panel, set the name of the item. Click the DDL item in the left side of the panel to bring up the panel shown in Figure 10-33.

Figure 10-33.

[View full size image]

In this panel, choose the connection you want to use (you will need to create one if you do not have one already) and set the SourceType to Direct Input. In the SourceDirect item, click the three dots (ellipses) and paste the code you created in the last step. Save the item and then enter any other maintenance and notification logic you want in the package. By including other items that connect to a relational database, you can create a complete maintenance setup for your entire BI system.

Monitoring and Performance Tuning

Happily, the same monitoring techniques apply as I described in Chapter 5, "Monitoring and Optimization," specifically using Windows System Monitor and SQL Server Profiler. The only difference is that you capture different objects and counters in System Monitor and different trace items in SQL Server Profiler. I do not cover the same ground for the process of collecting here, but let's examine a few of the specifics.

Here are a few of the objects and counters you should collect for Analysis Services.

[Pages 563 - 564]

Object | Counter | Description
Analysis Server: Connection | Current connections | Shows the number of clients. Monitor for growth.
Analysis Server: Connection | Failures/sec | Failed connection attempts. If you see a growing rate of these, check the worker thread pool.
Analysis Server: Connection | Requests/ sec | Incoming requests. Monitor for growth.
Analysis Server: Query | Avg time/query | Response time in milliseconds. Monitor for growth.
Analysis Server: Query | Current queries | Number of active queries. Monitor for growth in conjunction with Avg time/query counter.
Analysis Server: Query | Data bytes/sec | Bytes read from data files. Monitor with Map reads for growth.
Analysis Server: Query | Data reads/sec | Number of logical reads per second from data files.
Analysis Server: Query | Map bytes/sec | Bytes read from the map file. Monitor for growth.
Analysis Server: Query | Map reads/sec | Number of logical reads per second from map files. Monitor with Data bytes/sec for growth.
Analysis Server: Query | Bytes sent/sec | Bytes sent from the AS server to clients. Monitor for high values. Indicates network issues.
Analysis Server: Query | Current worker thread pool | Threads in the worker thread pool. Monitor for growth.
Analysis Server: Query | Current worker threads active | The number of active worker threads.
Analysis Server: Query | Current process thread pool | Process threads required to execute queries not found in the query results cache. Monitor for growth; add disks or processors if correlated with a slowdown.
Analysis Server: Query | Current process threads active | Active query threads.
Analysis Server: Query | Current process thread queue length | Length of the processing thread queue.
Analysis Server: Agg Cache | Current bytes | Query results cache size. Compare to cache hits and misses for changes over time.
Analysis Server: Agg Cache | Direct hits/sec | The queries answered from an existing cache entry. Watch for decrease.
Analysis Server: Agg Cache | Filter hits/sec | The queries answered by filtering an existing cache entry.
Analysis Server: Agg Cache | Evictions/sec | Rate of evictions from the query results cache, showing memory use. Reduce parallel operations if a significant increase occurs during a slowdown.
Analysis Server: Agg Cache | Lookups/sec | Cache lookups. Consider with cache misses and hits.
Analysis Server: Agg Cache | Misses/sec | Cache misses. Lower is better, except during startup.
Analysis Server: Proc | Current partitions | The number of partitions in this phase of processing at one time. Shows resource use.
Analysis Server: Proc | Rows read/sec | Data retrieved from the relational database. Lower numbers indicate the AS server is slowing down.
Analysis Server: Proc | Rows created/sec | Aggregation row creation rate. Monitor for decrease.
Analysis Server: Proc | Memory size bytes | Shows the rough size of the current rows in memory. Shows memory use.
Analysis Server: Proc | Memory size rows | Shows the rough size of current rows in memory.
Analysis Server: Proc | File bytes written/sec | MOLAP partition file bytes written. Monitor for activity.
Analysis Server: Proc Aggs | Current partitions | The number of partitions processing at one time. Shows memory pressure.
Analysis Server: Proc Aggs | Rows created/sec | Creation rate for aggregation rows. Monitor for decrease.
Analysis Server: Proc Aggs | Rows merged/sec | Merged and inserted rows into an aggregation. Monitor for decrease.
Analysis Server: Proc Aggs | Memory size bytes | Shows the rough size of the current rows in memory. Shows memory use.
Analysis Server: Proc Aggs | Memory size rows | Shows the rough size of current rows in memory.
 |  | Shows memory use.
Analysis Server: Proc Aggs | Temp file bytes written/sec | Temporary file bytes written. Indirectly shows process buffer space; lower is better.

You should include these counters along with the others I explained in the earlier chapters for overall system health.

For the SQL Server Profiler trace items, the primary objects are those shown here.

Event Class | Meaning
Query Begin | Shows the query start
Query End | Shows the query end
Query Subcube | Shows any queries that touch items within other cubes

You should collect at least the start and stop times for the queries, and if you are interested in specific activity you can capture the text from the items as well as the source connection information and the database name. You can capture several other items to show security, commands, and even notification events. Check Books Online under the topic of "Analysis Services Event Classes" for a full description.


Take Away

In this section, I provide a simple BI solution from end to end so that you can become familiar with the process. Microsoft also includes a similar exercise that you can access from the Windows Start menu in the SQL Server 2005 area. I advise that you start with this example and then go through the Microsoft tutorial. I focus on the entire process and what you have learned in this chapter, and the tutorial focuses more on the tools and mechanics of the process. Using both will help reinforce the concepts and help you get familiar with the tools.

Because we need a common, large database to pull from to adequately demonstrate BI concepts, I will use the AdventureWorks data warehouse example that ships with SQL Server 2005. You will only have the AdventureWorks and AdventureWorksDW databases if you have installed them on your test system. If you would like to follow along with this example and you do not have these databases, just put the installation CD back in the drive and install it on your test system now.

Before you create a BI system, you need a thorough understanding of the business and how it works. I normally do not quote directly from Books Online because you are reading this text for things you do not find there, but in this case I do, because it has the best description of this example company:

Adventure Works Cycles, the fictitious company on which the AdventureWorks sample databases are based, is a large, multinational manufacturing company. The company manufactures and sells metal and composite bicycles to North American, European, and Asian commercial markets. While its base operation is located in Bothell, Washington with 290 employees, several regional sales teams are located throughout their market base.

In 2000, Adventure Works Cycles bought a small manufacturing plant, Importadores Neptuno, located in Mexico. Importadores Neptuno manufactures several critical subcomponents for the Adventure Works Cycles product line. These subcomponents are shipped to the Bothell location for final product assembly. In 2001, Importadores Neptuno became the sole manufacturer and distributor of the touring bicycle product group.

Coming off a successful fiscal year, Adventure Works Cycles is looking to broaden its market share by targeting their sales to their best customers, extending their product availability through an external Web site, and reducing their cost of sales through lower production costs.

This description contains several important BI concepts. For one, it is a large company. That means that our audience for the system will be large, at least for a BI system. A larger audience means people to train on the system, larger and faster hardware, and possibly a heavier load on the networking infrastructure.

Notice also that the company is multinational. That means dealing with multiple currencies, legal definitions, collations, and more.

There are also regional sales teams. That means that there will be a need for granular analysis. Managers at the regions want to know about their site's performance, and managers at headquarters want to know how the entire company is doing.

You can see the questions emerging from a definition of the company and the things managers want to know. From that description, you can begin to develop the formal questions that need to be answered. From those questions, you can investigate the type of data you need to collect, which in turn points you to where you need to get that data. From that information, you can develop a data path, which shows where the data originates (called a source system), where it ends up (called the presentation), and any changes (called transformations) that you do to consolidate and aggregate it. In this chapter, I limit the questions we want to answer to sales of products by time and district.

The AdventureWorks company uses SQL Server 2005 components and features to run their operations and provide a BI landscape. Let's take a look at the source of the data and a little about how it is structured in the data warehouse.

The AdventureWorks database is divided up into several schemas, which have to do with customers, vendors, and employees stored in the People schema, personnel management stored in the HumanResources schema, manufacturing information stored in the Production schema, source vendor information stored in the Purchasing schema, and sales information stored in the Sales schema. These are arranged in typical normalized tables, where unique information is generally stored only once and is referenced to each other through the use of key fields.

The information stored in the AdventureWorks database is used and supported by the company for OLTP applications and reporting. It fills the needs of the first part of the BI system because it provides a source system and a reporting infrastructure.

As the company grew, they needed more analytical mechanisms. Using the SQL Server 2005 database engine, they created a new database called AdventureWorksDW, for their data warehouse. The data in this database is pulled form the OLTP system periodically. One of the decisions you will make when you create your BI system is when and how often you will pull data from the source systems into your reporting subsystem or another storage system for use by Analysis Services.

The data in the AdventureWorksDW database is also structured differently than the OLTP layout. In this database, the data is stored in the dimensions and facts needed for Analysis Services. As you will recall from the spreadsheet example, the columns and rows of the database such as Products and Store Location become dimensions, and the numbers associated with them become the facts. Using the same methodology I showed you earlier in the chapter, the AdventureWorks database forms the source for the star and snowflake schemas found in the AdventureWorksDW database. This database has 16 dimensions, six fact tables, and a few other tables used in KPIs and data mining. In this case, we are going to use a single fact table and a few associated dimensions, just to get familiar with the process.

To begin, open the Business Intelligence Development Studio. Click File and then New and then Project from the menu bar. Leave the project type as Business Intelligence Projects and leave the template selection on Analysis Services Project. Name the project Sales By Time. You can also leave the option set to create a subdirectory for the project.

With that panel filled out as shown in Figure 10-34, click OK to start the process. After the tool creates the directories and sets up the environment, you will be dropped into the project view. On the right side of the tool, you will see a panel called Solution Explorer. It is arranged (conveniently enough) by the steps you take to create the solution. First, you need to connect to a database, arranged by dimensions and facts. In our case, that is the AdventureWorksDW database.

Figure 10-34.

[View full size image]

Right-click Data Sources, and then click New Data Source from the menu that appears. You are presented with a wizard (which you can cancel when you are more familiar with the process) that guides you through selecting a database. You can see the process starting in Figure 10-35.

Figure 10-35.

Select Next, and you are placed into the connection definition screen. Because you do not have any connections defined yet, click the New… button on the panel. The next screen has selections for the type of provider (leave it at the default for this exercise), the server and database names, and the security credentials for the connection. Remember, the database you want to connect to is the one that is prepared for use with your Analysis Services system, which in this case is the Adventure-WorksDW database.

Click OK, and you can review the selections that will be used for the connection, shown in Figure 10-36. Click Next to continue. The next panel sets the credentials the solution will use for the project. This might be a different account from the one you used a moment ago. Set this one to Use the credentials of the current user. This means the solution will check what rights the user that is logged on to connect to the source database. Click Next and give the connection a name. In this case, you can leave the defaults. Click Finish to create your connection. You can repeat the process if you need to include sources for more dimensions or facts.

Figure 10-36.

[View full size image]

You now have a connection to the database. Now you need to create either a view of that data or to create a cube. In this case, let's add a simple view of the data. Having a view of exactly what we are looking for simplifies the creation of a cube.

Right-click the Data Source Views object in the Solution Explorer panel on the right. Select New Data Source View from the menu that appears. Once again, a wizard opens to help you through the process. Click Next to continue.

On the Data Source Selection panel, select the Data Source we made in the last step. If this is your first cube it will be the only one. Click Next when you have done that. You are placed in a panel shown in Figure 10-37 where you can select the tables you want to include in this view. Select the dboFactInternetSales and then the arrow icon to move the selection to the right panel.

Figure 10-37.

[View full size image]

This is where the power of the Intellicube technology really comes into play. Click the Add Related Tables button. After a few moments, the Analysis Services Wizard adds seven tables to the panel, all derived from the relationships and naming in the dimensions and facts.

The wizard adds more than we care about, because we only want to know about sales by product, region, and time. Remove all the tables from the panel on the right, with the exception of the following:

  • dbo.FactInternetSales

  • dbo.DimProduct

  • dbo.DimTime

  • dbo.DimSalesTerritory

Figure 10-38 shows what the final selection looks like.

Figure 10-38.

[View full size image]

Now click the Next button. Leave the name for the view at the default value. Click the Finish button. The system will process and create the view. You are placed in the graphical representation of the view in a navigation pane, as shown in Figure 10-39.

Figure 10-39.

[View full size image]

Note a couple of interesting things about this view. On the left side of the screen, you can rapidly move through the diagram by clicking the table names. We have a simple diagram in this case, but in more complex uses the diagram can become quite large. In the lower-right corner of the main graphic panel, you can see a small double arrow. Move your cursor over that graphic and click it, holding down the mouse button. This provides a small dynamic view that allows you to quickly move to any section of the diagram. Click any table name or connecting arrow and more information about it shows up in the right side in the Properties panel.

Now right-click the Cubes object in the Solution Explorer and select New Cube from the menu that appears. After the Cube Wizard starts, click the Next button. In the panel shown Figure 10-40, you can have the wizard build the attributes and hierarchies for you, just the attributes, or neither. Notice also that you can build the cube using a template and not a connection to any database, in case it is not available. For this example, leave the defaults.

Figure 10-40.

Click Next to continue, and then select the Data Source View we created a moment ago. Click Next when the wizard completes examination of that data source. In the panel that appears, you are asked to check the boxes next to the facts and dimensions. In this case, the wizard automatically designated them properly. What it did not guess was the time dimension, so select DimTime from the pull-down menu, as shown in Figure 10-41.

Figure 10-41.

After you have set the time dimension, click Next to continue. Because we defined a time dimension that has multiple attributes, the wizard wants to know how we will break them into levels. In our case, we are interested in years and months. The source table contains many more breakdowns than that, so this panel allows us to assign which fields provide the levels we want. In this case, select Year to map to the CalendarYear field, and Month to map to the EnglishMonthName field, as shown in Figure 10-42.

Figure 10-42.

In the next panel, you can set the measures (facts) that you want to include in the cube. In many cases, you will store lots of facts in a single fact table, especially when they occur in the same time period. To keep this example simple, we want only the FactInternetSales item, all the keys, and the SalesAmount facts. De-select everything except those items, as shown in Figure 10-43.

Figure 10-43.

[View full size image]

Click the Next button. The wizard will detect any hierarchies in the data. When that is complete, you can review the layout before it is finalized. Click the Next button, leave the name of the cube at the defaults, and click the Finish button. The system creates the cube and drops you into the graphical panel displayed in Figure 10-44, which is similar to the data source view you worked with earlier.

Figure 10-44.

[View full size image]

Although this graphical view works the same way as the data source view graphic, you have different items to work with in the main panel; and if you look at this busy screen closely, you will notice several subtabs within the main tab. These views show the dimensions, allow you to work with calculated members, create perspectives, and more. To keep this example simple, let's process the cube as it is and then take a look at the results.

At the moment, the cube is defined and designed. However, there is no OLAP database for the project, and so there is no data to display. To create the database, we need to deploy it. Before we do that, we need to design the storage the database uses. As explained earlier in this chapter, you need to make a tradeoff between storage size and the responsiveness of the system. The more responsive the system is, the more aggregations it uses. The more aggregations it uses, the more space it takes.

In this case, click the subtab marked Partitions. Next to the Molap 0% item, click once on the button that has three dots on it, as shown in Figure 10-45.

Figure 10-45.

[View full size image]

When you click that button, another wizard appears. Click Next and leave the slider selection at the MOLAP selection, as shown in Figure 10-46.

Figure 10-46.

Click Next to continue. Click the Count button on the next panel to have the wizard automatically figure out how many calculations it needs to make (see Figure 10-47).

Figure 10-47.

On the panel shown in Figure 10-48, you can limit how much space or time you want the wizard to spend calculating the aggregations. If your test system is short on space, enter a number you can live with for the space; otherwise, pick a percentage gain you want to use for optimizing the aggregations. The higher the numbers you place here, the higher the performance.

Figure 10-48.

Click the Next button and allow the wizard to complete. When it finishes, it will display the amount of space it will use and the number of aggregations it created, as shown in Figure 10-49. Click Next. Change the selection on the next panel to Deploy and process now, and then click the Finish button.

Figure 10-49.

The system will pause for a moment or two and then present you with a Run selection. The options here determine how much of an impact to the system the process will take. In this example, leave the Process Full option set and click the Run button. The system will then create the aggregations. When that process completes, you can close the processing panel.

Now we can view the fruits of our labor. Click the Browser subtab in the Business Intelligence Development Studio tool, in the Adventure-Works cube. It is the last subview in the screen. Open the measures you see and drag any fact into the center part of the panel on the right. Now expand a dimension and drag an item from into the thin bars across the top and sides of the section on the right, as shown in Figure 10-50.

Figure 10-50.

[View full size image]

In this example, I have placed the Sales Amount fact from the Fact-InternetSales measure in the center area. I have also added the Model Name dimension from the Dim Product dimension in the far-left side of the vertical bar. After that, I added the Sales Territory Region from the Dim Sales Territory dimension next to that same bar. Along the top horizontal bar, I added the Calendar Year dimension from the Ship Date dimension.

This quickly shows me the sales for the various regions for all the products I am interested in, by month. Even with this simple example, I can quickly change the view to see the due dates and sales by month, use other languages (they were included in the source start schema), and more.

You can certainly do more with this example. Take some time and add a data mining exercise to this cube to see whether you can determine any patterns in the data and to what level of certainty they show. You can also add KPIs to this project to show any deviations from the norms you establish within the solution.

If you are interested in reviewing a real-world example of using a SQL Server 2005 implementation for large-scale BI, search inside Microsoft's TechNet site for "Project REAL." It is a complete set of white papers detailing that effort.


Appendix A. Upgrading to SQL Server 2005

Depending on the kind of person you are, you have probably arrived here from one of two places. You may be the type of person who likes to get all the information you can and then make a plan to implement what you have learned. If that sounds like your personality, you probably read the entire book and are here now to find out how to put it all to work.

On the other hand, you might be the kind of person who eats dessert first. When you get new software, you toss the manual to one side and start loading. Perhaps you even tried that with SQL Server 2005 and maybe you hit a glitch or two. So, you bought this book and turned to this chapter first.

Good newsthis section can help personality types. If you are here after reading the previous chapters in the book, you have gained some insights into how you want to implement your platform. You have learned all about the design decisions you need to consider. If you are here first, you can use this information as the first part of your system planning, as well as a guide for upgrading your test systems. This Appendix does not cover the same information contained in other chapters, so if you are reading this first, make sure you check out the other chapters to gain a complete understanding of all the choices you have during the upgrade.

Although I mention features such as Analysis Services, Notification Services, Reporting Services, Replication, and Clustering, this Appendix focuses on upgrading the database engine. Each feature has several upgrade caveats based on how it is installed now, and most features can be migrated or moved to SQL Server 2005 versions without disturbing the original system. In some cases, you do not have to upgrade the feature at all, keeping them on the same server as the 2005 version (which is called a side-by-side installation). You can find more about upgrading features such as Analysis Services and Data Transformation Services in Books Online after the base system is upgraded. Search on "Upgrading" and the feature name to find out more. Reporting Services and Analysis Services largely upgrade in place with few issues.


General Guidelines

Your upgrade strategy depends on what you are trying to accomplish. If you are just replacing a previous version of SQL Server on the same system, that is often an easier process than changing hardware and applications at the same time as the database platform upgrade. I help you deal with each of those situations in a moment, but you can follow some general guidelines prior to starting the upgrade.

In most cases, but not all, you must have some downtime for the system before you begin your upgrade. Sometimes that is not possible. If you need to maintain availability of the system, you will need to follow the planning steps I mention here and then select the strategy that gives you the option of presenting the system during the upgrade.

As with any change to your production infrastructure, make sure you have a solid understanding of the options available to you, and then you need a good plan to implement the change. This Appendix will help you do that; when you upgrade other features or special installations such as clustered systems, however, you need to do more research. In all cases, start with planning.

Planning

You cannot substitute for planning the upgrade. Most often, a Relational Database Management System (RDBMS) is used in a critical area of your organization, and rushing headlong into the process is asking for trouble. Even if you are a seasoned veteran with lots of experience, at least read through this section to see what you are up against.

First, contact all the affected parties or their managers (called the stakeholders) and explain that you are planning an upgrade. After you read this Appendix, you will know more about what impacts the upgrade will have, and you can communicate that to them. After explaining what you know and soliciting their input, you will almost certainly find out things you did not know. One group might tell you that their application vendor will not support SQL Server 2005, and another might tell you they need to re-architect an in-house application to take advantages of the new features it provides. With that information, you are able to better coordinate your upgrade. It is also a matter of business professionalism to keep everyone informed.

Your planning document should include the following elements:

  • Description of the change

  • Timetable

  • Impacts

  • Costs

  • Expected benefits

  • Exit strategy

Document Your Current Systems

In a perfect world, you already have your current systems completely documented. Because we do not live in a perfect world, you may need to either create documentation from scratch or freshen up the documentation you already have. In either case, you need to collect at least the following information:

  • Server name

  • Server location

  • Hard drive layout and configuration

  • Amount of physical memory

  • Exposed network ports and security constraints

  • Operating system version and service packs

  • Utility programs

  • SQL Server version

  • Database metadata

  • Maintenance plans

  • DTS packages

  • A full script of the current permission schema (needed for restored or attached databases)

  • List of stored procedures that reference system objects

Normally, this type of documentation is necessary because you are asked to gather it for the upgrade program. In the case of SQL Server 2005, Microsoft provides a tool that scans your system for any issues it can detect and tells you what needs to be corrected prior to the upgrade. I show you how to use that in a moment. But you still need the documentation.

Documenting your systems often leads to some new discoveries, which can help you for budgeting and other purposes. The information you are gathering here is for the planning process as well as the next phase: the exit strategy.

Create an Exit Strategy

Although Microsoft has done a great job of making the upgrade process as seamless as possible, things can still go wrong. Before you begin the upgrade, you need to create an exit strategy.

Also called a disaster recovery plan or backup strategy, an exit strategy is essentially what you will do if things do not go as planned. In small implementations of applications that depend on SQL Server, the exit strategy can be as simple as ensuring that you have a good backup and restoring that backup in case of an issue.

Even then, you need to document the process so that you can communicate it to the people affected by the upgrade. The exit strategy for a "simple" implementation should be documented and tested, to ensure that you know what you plan to do, that it will work, and how long it will take. You might also want to include a "risk factor" that explains which steps have the most potential (however small) to fail. A little paranoia here is a good thingassume the worst about any changes to a production system and you are safer than expecting everything to go well.

In a more complex environment, the exit strategy will necessarily be more complicated. Although this might seem like more of a formal process than you need, it is not. Having a plan is like having insurance. It never seems worth it until you don't have it.

The following table is a sample of the start of an upgrade plan that includes a simple restore as an exit strategy.

Step | Duration | Risk Factor
Notify the users | 5 minutes | 0
Restrict access to the system | 0 minutes | 0
Take a full backup | 1 hour | 0
Perform the upgrade | 2 hours | 2
Test the results | 30 minutes | 2
Exit strategy for failure - restore system | 2.5 hours | 3
Test the results | 30 minutes | 2
Allow access to the system | 5 minutes | 0
Notify the users | 5 minutes | 0

You add more columns to this chart as you move along in the planning. If you total the durations on the chart, it may add up to more than you originally thought to perform the upgrade on a simple system. If these sample numbers are accurate, for instance, your system will be down for the better part of an entire day for the upgrade.

If your system involves multiple departments or time zones, is a clustered system, or has multiple databases, your exit strategy has to include not only a risk factor but also a business impact. A business impact is a rating on how badly the business is hurt if some process is not resumed within a set period of time.

For instance, if an application processes payroll, the business impact might be less than if the application processes parts at the plant. This depends, of course, on how close you are to the time that you need to cut payroll checks! So for more complex environments, you will create the same type of documentation but for each part of the application. You will also need to have formal meetings with the process owners so that they are aware of the possible impacts to their function.

Documenting the steps like this forces you to defend them. Your company might not want a high level of downtime or may not tolerate a high degree of risk, and with documentation you might be able to justify another way to implement the solution.

Requirements

Other than the hardware requirements for any SQL Server 2005 system that I explain in Chapter 1, "Installation and Configuration," the primary thing you need to upgrade is the space of the database storage that you are using now. Depending on the type of upgrade you plan to do, SQL Server needs more room to do the conversion between the databases, and you will need it to retain the backups in case of a failure.

There are ways around this requirement. One is to back up some of the databases and upgrade the system only with the databases that have the most critical objects. You can then restore the databases that you backed up to the new version. SQL Server 2005 automatically upgrades a database during the restore process.

With these space requirements, you might be tempted to use Windows-based compression on the drives that hold the databases. Although this works in earlier versions (but is always a bad idea), it does not work in SQL Server 2005.

What Works and What Doesn't

As with all software, sometimes a simple upgrade is not possible. For the most part, Microsoft does a great job with the hardware and software that you have when you move to SQL Server 2005, but there are certain things that either do not upgrade well or do not upgrade at all. In this section, I explain well-known issues that you are likely to encounter.

That is not to say that you will not face any issues other than what I explain here. It is absolutely vital to have that exit strategy complete before you attempt any upgrades, because anything can happen.

Versions

You can upgrade from SQL Server version 7 or SQL Server version 2000. If you have something earlier than that, you can create Data Manipulation Language (DML) scripts of the objects and then use the Bulk Copy Program (bcp) to copy out the contents of the tables into text.

The following table lists which version upgrades to which.

[Pages 587 - 589]

You Can Upgrade From | To
SQL Server 2000 IA64 (64-bit) Developer Edition | SQL Server 2005 IA64 (64-bit) Enterprise Edition
 | SQL Server 2005 IA64 (64-bit) Developer Edition
SQL Server 2000 IA64 (64-bit) Enterprise Edition | SQL Server 2005 IA64 (64-bit) Enterprise Edition
SQL Server 2000 Developer Edition SP3 or later | SQL Server 2005 Developer Edition
SQL Server 2000 Enterprise Edition SP3 or later | SQL Server 2005 Enterprise Edition
SQL Server 2000 Evaluation Edition SP3 or later | SQL Server 2005 Evaluation Edition
SQL Server 2000 Personal Edition SP3 or later | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
 | SQL Server 2005 Express Edition
SQL Server 2000 Standard Edition SP3 or later | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
 | SQL Server 2005 Standard Edition
SQL Server 2000 Workgroup Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
SQL Server 2005 Developer Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
SQL Server 2005 Evaluation Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
 | SQL Server 2005 Express Edition
SQL Server 2005 Express Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
SQL Server 2005 IA64 (64-bit) Developer Edition | SQL Server 2005 IA64 (64-bit) Enterprise Edition
 | SQL Server 2005 IA64 (64-bit) Standard Edition
SQL Server 2005 IA64 (64-bit) Evaluation Edition | SQL Server 2005 IA64 (64-bit) Enterprise Edition
 | SQL Server 2005 IA64 (64-bit) Developer Edition
 | SQL Server 2005 IA64 (64-bit) Standard Edition
SQL Server 2005 IA64 (64-bit) Standard Edition | SQL Server 2005 IA64 (64-bit) Enterprise Edition
 | SQL Server 2005 IA64 (64-bit) Developer Edition
SQL Server 2005 Standard Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
SQL Server 2005 Workgroup Edition | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
 | SQL Server 2005 Standard Edition
SQL Server 2005 X64 (64-bit) Developer Edition | SQL Server 2005 X64 (64-bit) Enterprise Edition
 | SQL Server 2005 X64 (64-bit) Standard Edition
SQL Server 2005 X64 (64-bit) Evaluation Edition | SQL Server 2005 X64 (64-bit) Enterprise Edition
 | SQL Server 2005 X64 (64-bit) Developer Edition
 | SQL Server 2005 X64 (64-bit) Standard Edition
SQL Server 2005 X64 (64-bit) Standard Edition | SQL Server 2005 X64 (64-bit) Enterprise Edition
 | SQL Server 2005 X64 (64-bit) Developer Edition
SQL Server Desktop Engine (MSDE) 2000 | SQL Server 2005 Workgroup Edition
 | SQL Server 2005 Express Edition
SQL Server Desktop Engine (MSDE) 7.0 SP4 | SQL Server 2005 Express Edition
SQL Server 7.0 Desktop Edition SP4 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Workgroup Edition
SQL Server 7.0 Developer Edition SP4 | SQL Server 2005 Enterprise Edition
 | SQL Server 2005 Developer Edition
SQL Server 7.0 Enterprise Edition SP4 | SQL Server 2005 Enterprise Edition
SQL Server 7.0 Evaluation Edition SP4 | Upgrades not supported
SQL Server 7.0 Standard Edition SP4 | SQL Server 2005 Standard Edition
 | SQL Server 2005 Enterprise Edition

As with any upgrade, you cannot upgrade a system to a higher version without considering the licensing issues. You are not allowed to replace a lower version with a higher one without paying for the license. For the most part, the upgrade tools prevent this anyway, but it is something you need to keep in mind.

Features

Although SQL Server 2005 added many amazing new features, not all previous components made the cut. English Query, a Natural Language Processing (NLP) system, is no longer a part of SQL Server. If you have never used this handy feature, you are not aloneMicrosoft did not see enough use in the community to include it in the new packaging.

Unless your databases are set to the 80 compatibility mode (more on that later), you will not be able to use the older ANSI 89 join syntax. For instance, this kind of statement does not work on a 2005 database:

SELECT column1, column2
FROM table1, table2, table3
WHERE table1.key=table2.key AND table2.key=table3.key

If you are still writing these kinds of joins, you are not taking full advantage of the speed of the newer T-SQL format anyway. If you need them, you can set the compatibility mode to 80 and they will still work.

Another feature that has changed dramatically from previous SQL Server versions is Data Transformation Services. I have an entire chapter in the book devoted to its replacement, SQL Server Integration Services (SSIS). The two tools differ completely. There are a few ways you can deal with the change, but to take full advantage of the power in the new platform, Microsoft suggests that you "migrate and modify" the packages. In practice, I have found it is better to completely rewrite the packages in SSIS because you have so many new tools within it that you can approach the problems they solve in new ways. If you have a lot of packages or they are particularly complex, that is difficult to do, so Microsoft includes a layer within SSIS that can still run older DTS packages. There is not enough room here to cover all the changes that you need to take into account, so I refer you to the SSIS section of this book to learn more.

Depending on your version of operating system, the Microsoft Database Access Components (MDAC) may change when you upgrade to SQL Server 2005. Because the MDAC level is also affected by service packs to both the operating system and SQL Server, I refer you to Microsoft's SQL Server site to find the latest version that is installed. This is one of the most important things to communicate to your developers before you upgrade, because the MDAC level is important in the way they write applications against the database.

Other Information

You need to know a few other things about upgrading your system. English versions upgrade to any locality, but localized versions upgrade only to that locality. If you are in a multinational organization, this can be an issue if you are using the upgrade as an opportunity to standardize on a language; keep this restriction in mind.

System catalog direct updates are not supported in SQL Server 2005. That is a bad practice anyway, but in this case the code that calls it will break. Along those same lines, code that references system objects does not work in SQL Server 2005.

You need to have your developers scan the new reserved words (a standard practice) to ensure that they have not named any database objects with those words or used them in their code.

Although it is a rare problem, the database ID (an internal number given to each database) of 32767 is reserved. You do not normally set this number; it is an automatic value assigned on database creation. The Upgrade Advisor (more on that later) will catch that error anyway.

If you have placed data or log files on compressed or read-only drives, they will not upgrade properly. It is normally bad practice to do this to begin with because it dramatically affects performance.


Upgrade Strategies

You can implement SQL Server 2005 in a few ways in your organization. You can upgrade your current servers "in place," or you can migrate the databases and other objects from an older version on one server onto SQL Server 2005 on another system. This is useful if you are upgrading your hardware at the same time. You can also keep both versions on one server (called a side-by-side install). The strategy you choose should be based on a cost/benefit analysis for your situation.

If you need to allow users to access their current applications during the upgrade, you can either follow the migration or side-by-side upgrade methods I show you in a moment. You can have the users hosted on a separate system while you copy the items to the new, and then either "catch up" the transactions or use a differential output from the older system to the new to handle the time difference for the changeover.

Migration

In a migration upgrade, you copy the data and database objects from an earlier version of SQL Server to 2005. This method provides a lot of flexibility when you do not want to bring over all the data or objects from an older system. Even within this method, you have options.

Copy Database Wizard

The first method to migrate databases from an earlier version of SQL Server to the latest is to use the Copy Database Wizard within SQL Server 2005. You can access that tool by opening the SQL Server Management Studio and then connecting to the older database server or instance. Drill down to the Databases object in the Object Browser. Right-click any database and select Copy Database from the menu that appears. The wizard guides you through the process of selecting the objects, data, and security information you want to migrate (and even lets you schedule the process).

This process enables you to move all or part of the database from one server to another with a lot of control. It hides much of the complexity of the process from you.

Backup and Restore

You can also back up your older databases on one system and restore them to the new one. This is a more useful feature when you are going to wipe the old system clean and re-install the operating system or the database software.

SQL Server 2005 will restore an older database (7.0 and 2000) using the standard RESTORE DATABASE command or with the graphical tools I explain in this book.

Detach and Attach

You can use the sp_dbdetach stored procedure on a SQL Server 7 or 2000 server, copy the mdf and ldf files to the SQL Server 2005 system, and then use the sp_dbattach stored procedure to copy and migrate the databases to a new server.

You can also do the same thing graphically by right-clicking the name of the database in Enterprise Manager or SQL Server Management Studio and selecting Detach Database from the menu. You then copy the mdf and ldf files to the new server, right-click the Databases item in the Object Browser in the SQL Server Management Studio, and select Attach Database from the menu. Provide the name of the database and the location of the files and the system will "adopt" the databases.

Side-by-Side

You might decide not to upgrade at all. You might want to keep your older system just as it is and add SQL Server 2005 into your landscape. Many organizations coincide major platform releases with new versions of their internal applications. The reason this works out is that major changes in the underlying platform create new architecture possibilities. It may be that a rewrite makes more sense than just tacking on a new back end to the system. New advances in hardware, especially with lengthy upgrade schedules, might mean that it is time to upgrade the hardware as well as the database engine. If you do that, you can recycle the old hardware after the cutover to the new system. That strategy also allows you to run a parallel system (old and new), a common practice in complicated mission-critical environments.

Although this might sound expensive, it can actually save you money. SQL Server 2005 includes new features that enable you to consolidate multiple functions, such as reporting, onto fewer servers. If it is time to replace several servers, you may be able to re-architect the system such that you can use fewer of them.

Another advantage in keeping your old system is that you do not have to put as much effort into an exit strategy. Because the older system is still available, your exit strategy normally contains just the steps to enter any data back into the old system that you have been entering on the new. It is one of the safest, but possibly more expensive, ways to upgrade.

The Microsoft SQL Server 2005 tools enable you to connect to and manage earlier SQL Server versions, so you only have to install one toolset on your local workstation. You can manage most features in the older versions the same way as SQL Server 2005, with a couple of exceptions. You will not be able to create new objects when they are not included in SQL Server 2005. This means that you can still manage the older maintenance plans when you are connected to version 7 or 2000, but you cannot create new ones using SQL Server Management Studio. Likewise, you can manage DTS packages on those versions but not create new ones in SQL Server Management Studio. Newer features found in SQL Server 2005, such as the Reports view in Management Studio, are not available against an older system because they do not have the subsystems needed to make them work.

Rather than deprecating your current hardware, you may decide to keep two versions of SQL Server running on it. You can install SQL Server 2005 in addition to a current version of SQL Server, but you can only have one named instance on a system at a time.

To install a single-server side-by-side configuration, install SQL Server 2005 as normal, but specify a new installation during the initial screens and set the name of the instance to a name that is not currently used on the system. You will still have the older tools and binaries, along with the SQL Server 2005 tools and binaries. The directory structure for SQL Server 2005 differs slightly from what you will find in the previous versions, so read through Chapter 1 to find out where everything ends up.

In-Place

In many cases, you want to replace the version of SQL Server on your current system with SQL Server 2005. This is called an in-place upgrade. For simple software packages, you start the installation or upgrade program, and the binaries for the programs are replaced with new ones. In more complex software applications, you have to complete many steps to upgrade a system, starting with preparing the older system to be upgraded. An SAP application, for instance, can take days or even weeks to upgrade.

SQL Server is used as the back end to many other programs. As such, it falls into the "more complex" category. SQL Server is composed not just of the database engine function, but a "Swiss knife" of features and applications. Previous upgrades took a high level of reading, studying, and planning to implement.

Although you still need to be concerned about some things for the upgrade, Microsoft provides a handy tool to at least discover all the issues you have, and in most cases it even tells you what you can do to fix them.

Before you start the upgrade, make sure you are upgrading the edition you have to either the same or a higher edition. Also make sure that you have covered whatever licensing issues you need to deal with. Just because you can physically upgrade something does not mean that it is legal to do so.

Upgrade Advisor

The Upgrade Advisor program examines your SQL Server installation and creates a report that shows you what issues the upgrade will have. The report also includes any information available to help you solve the problem or at least where to find out more information. You can run the SQL Server Upgrade Advisor from any machine that can connect to your current SQL Servers.

You can find the Upgrade Advisor on the installation media. When you insert that media, the auto-start menu gives you an option to run it. When you start the Upgrade Advisor, you will see a screen similar to the one shown in Figure A-1.

Figure A-1.

[View full size image]

The first thing you should do is click the Check for Updates link. The tool works from a series of control files that were built during what Microsoft called the Community Technical Preview (CTP). They watched the upgrade process on thousands of machines to drive the Upgrade Advisor, building on the knowledge of those trial upgrades. This process continues, so that as Microsoft learns more about what steps are needed to upgrade a component, they include it in the control files. By downloading the latest tool, you get the most current information for your upgrade process.

When you download the latest version, click the Launch Upgrade Advisor Wizard link to start the analysis. The first page is the welcome screen, shown in Figure A-2.

Figure A-2.

This is standard stuff for Microsoft installations and upgrades, and if you are going to analyze more than one SQL Server, you can select the box not to show this screen again. Click the Next button to continue.

Figure A-3 shows the screen that sets the instance and components that you want the wizard to examine. If you know you have not used a feature even though it is installed, you can leave it out of the process.

Figure A-3.

The next screen shown in Figure A-4 uses the server name you entered on the previous screen and sets the instance name on that server along with the credentials you need to connect to it.

Figure A-4.

Figure A-5 shows the part of the wizard that sets the databases that should be part of the analysis. You can also capture trace files from the server for the wizard to evaluate. This is a handy feature.

Figure A-5.

Suppose you have an application that was written a long time ago and that has not been updated. There may be things such as ANSI 89 joins or other calls that are coded into the application. If you take a trace file on the server that hosts the database for the application, the wizard will tell you if it encounters anything to be concerned about. You need to make sure the trace file is long enough to guarantee some level of completeness.

The same idea goes for the next check box. The wizard can examine any of your batch scripts that call SQL Server commands to ensure that they do not have any issues.

After you have made your choices, select Next to continue.

Figure A-6 shows the Data Transformation Services package selection. The wizard can evaluate packages on the server or the ones you have stored in files. If you have complicated DTS packages, you are almost sure to have things to deal with here.

Figure A-6.

Figure A-7 shows the screen that allows you to move backward and change any selections you have made. It is also the screen that lets you send the upgrade report to Microsoft. This is how the product gets better. The wizard states that some personal information may be sent, but it is not like it is going to collect your personal address. The reason they have this disclaimer is in case the server name, IP, and other information is sensitive to you. If you are nervous about it, cut it off; if you are not, select it.

Figure A-7.

In Figure A-8, I have started the wizard on a test system. During the run, you will get feedback on the process. This test system is an IA64-bit SQL Server 2000 system with Service Pack 4 installed.

Figure A-8.

Figure A-9 shows that this system has two sets of issues. It seems there are some problems with the SQL Server engine and other issues with Data Transformation Services. I click Launch Report to see what the issues are.

Figure A-9.

Figure A-10 shows the Reporting portion of the tool. In this screen, I selected the DTS area to see what I need to take care of. The wizard presents me with a great deal of information, the ability to learn more, and even a check box to indicate I have fixed the issue.

Figure A-10.

[View full size image]

This brings up a couple of very important points. The first is that the tool lets you save the report using the Export Report link at the bottom of the screen. You can use that to create a "to-do" list that also fits in the checklists I showed you earlier in this Appendix. The other point is that after you correct the issues, you should run the wizard again to ensure that you did not create new issues by fixing the old ones.

Figure A-11 shows the SQL Server engine report, which has even more information, including a time indicator for when the problem should be addressed. Although you can leave "informational" things uncorrected going into an upgrade, you should deal with as much as you can prior to starting.

Figure A-11.

[View full size image]


Post-Upgrade Steps

If you performed an upgrade (not a migration to another server), you need to run two checks on each system you have upgraded: a system configuration check and a security check. Work through the steps shown here even if you are certain everything worked properly.

Also, tune the system for performance after you have upgraded it, even if you migrated, restored, or attached the database from an older version to SQL Server 2005.

Whenever you upgrade, back up, or attach a database from an older version to SQL Server 2005, the databases are set to an earlier compatibility mode. This is a safe way to handle things, because arbitrarily making the changes to a higher version might break an application that depends on earlier features. To change the setting, it is a simple matter of opening the properties of the database in SQL Server Management Studio and setting it to 90.

System Configuration Check

Along with the databases, check the server settings within SQL Server Management Studio to make sure that the memory, CPU, and other settings are not just what they were, but are optimized for what you need now. See the Chapter 5, "Monitoring and Optimization," to learn more.

Another check to make is any target and linked server registrations your system has. They may still work, but you should run any queries that depend on them to make sure.

If you created any extended stored procedures in Windows 2000, make sure that they are instantiated with the fully qualified file path in the registration. Earlier versions of SQL Server allowed you to leave the path off as long as it could locate the dynamic link library (DLL), but SQL Server 2005 will not. Even better is to remove those stored procedures and rewrite them in the Common Language Runtime (CLR) layer. The CLR provides a greater degree of flexibility and is much safer.

Security Checks

I describe the Surface Area Configuration (SAC) tool and the Configuration Manager in many chapters in this book, with the most depth in Chapter 4, "Security." These tools deal primarily with security, but also with what SQL Server presents to the network from the ports, protocols, and other standpoints. Make sure you review these tools and then run them to open or secure your system appropriately. This is a departure from previous versions of SQL Server, which were largely open by default.

If you used the backup and restore or detach and attach methods to migrate the databases to a new installation, the security accounts in the database will no longer be connected to the security accounts on the server, even if the names are the same. There are a few ways to deal with this, and I cover those in Chapter 4.

Performance Tuning

When you are finally done, make sure you run through one maintenance cycle. The upgrade testing is not complete until you ensure that your maintenance runs, and you may even want to re-create the maintenance using SSIS so that you have a more effective, flexible plan. I cover that in Chapter 8, "Integration Services."

Another reason to run the maintenance on your system is that the statistics and indexes may not be accurate for the new system. All of the configuration settings on the new platform have an impact to these performance systems, so make sure you present the most optimized view of the data that you can for your users by running full maintenance before you let them on.

With the system upgraded, optimized, and running, you are ready to begin. Make sure you review the sections of this book from time to time that apply to what you are working on and keep your system safe, secure, and optimized.


Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]


A

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

abstraction objects, schemas
access
Analysis Services
Reporting Services
accounts
policies
Reporting Services
security
services
SQL Server Agent
user
Windows
connecting
creating
logins
ActiveX Database Objects (ADO)
Activity, monitoring
ad hoc reports
adding
logins
passwords
roles
ADF (Application Definition File)
ADO (ActiveX Database Objects)
ADO.NET
agents, replication
aggregation
alerts
automation
Algebrizer
ALTER INDEX command
ALTER permission
AMO (analysis management objects)
analysis management objects (AMO)
Analysis Services
BI
Analysis Services
databases 2nd 3rd
implementing
Integration Services
Reporting Services
maintenance
backups
infrastructure
job processing 2nd
monitoring 2nd
security 2nd
storage
synchronization
analysis, growth
APIs (application programming interfaces) 2nd
connections
Application Definition File (ADF)
application programming interfaces (APIs) 2nd
connections
ApplicationName element
applications
architecture
client/server
distributed computing
monolithic
service-oriented
three-tier
communication processes
definition files
installation
operating system variables
post-installation configuration
layers
monitoring
Notification Services
optimizing
reports
roles
security
tools
Applications element
architecture 2nd
client/server
CPUs
distributed computing
Integration Services
command-line utilities
components 2nd
tools
wizards
monitoring
monolithic
Notification Services
objects
communication processes
databases
physical footprints
data files
operational files
queries
replication
Reporting Services
service-oriented
SOA
three-tier
Attach Database
attaching databases 2nd
auditing
reviewing
security
servers
automation
alerts
jobs
maintenance
multiple-system
operators
SQL Server Agent


B

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

backups
alternate methods of
Analysis Services
bulk-logged recovery models
copying
databases 2nd
differential
full backups
full recovery models
locations
mirroring
passwords
reports
security
simple recovery models
strategies
Transaction Log
baselines, configuring
bcp (Bulk Copy Program) 2nd
BI (business intelligence) 2nd
Analysis Services 2nd
databases 2nd 3rd
implementing
Integration Services
Reporting Services
binaries
building variables
Bulk Copy Program (bcp) 2nd
bulk-logged recovery models 2nd
business intelligence (BI) 2nd
Analysis Services
databases 2nd 3rd 4th
implementing
Integration Services
Reporting Services
Business Intelligence Development Studio 2nd
business logic layers


C

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

caches
calculus, relational
calling jobs
central processing units (CPUs), configuring
certificates
CHECKDB command
cleansing data
clients
architecture
network libraries
CLR (Common Language Runtime) 2nd 3rd 4th
Analysis Services
security
clustering 2nd 3rd
indexes
installing
requirements
two-node (on VM) 2nd
comma-separated value (CSV) files
command-line utilities, Integration Services
command-prompt utilities
bcp
sqlcmd
commands
ALTER INDEX
CHECKDB
CREATE APPLICATION ROLE
CREATE FULLTEXT CATALOG
CREATE LOGIN
CREATE ROLE
DBCC CHECKALLOC
DBCC CHECKCATALOG
DBCC CHECKCONSTRAINTS
DBCC CHECKDB
DBCC CHECKFILEGROUP
DBCC CHECKIDENT
DBCC CHECKTABLE
DBCC DBREINDEX
DBCC INDEXDEFRAG
DBCC SHOWCONTIG
DENY
GRANT
MIRROR TO
NET USER
operating systems
RESTORE
REVOKE
sp_configure
T-SQL
Common Language Runtime (CLR) 2nd 3rd 4th
Analysis Services
security
communication processes
complete restores
components
Integration Services
control flow
data flow
event handlers
packages
precedence
variables
monitoring
WMI
configuration
applications
architecture
client/server
distributed computing
monolithic
service-oriented
three-tier
baselines
clustering two-node (on VM) 2nd 3rd
databases (mirroring)
editions (SQL Server)
Enterprise
Express
Mobile
software requirements
Standard
Workgroup
files
hardware
CPUs
I/O
memory
networks
landscapes
departments
enterprise
farms
log shipping 2nd
post-installation
recovery models
replication 2nd
reports
SAC 2nd 3rd 4th
SQL Server Configuration Manager
systems
checks
selecting use of
sizing
Connection Manager
connections
accounts
containers
methods
security
consistency checks
constraints, precedence
containers, connecting
control flow, Integration Services
conversion, Integration Services
Copy Database Wizard
copying
backups
bcp
log shipping
counters, performance
CPUs (central processing units), configuring
CREATE APPLICATION ROLE command
CREATE FULLTEXT CATALOG command
CREATE LOGIN command
CREATE ROLE command
CVS (comma-separated value) files


D

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

DAOs (Data Access Object)s
data definition changes, monitoring
Data Definition Language (DDL) 2nd
permissions
triggers
data files
data flow
data layers
Data Manipulation Language (DML) 2nd 3rd
permissions
data mart databases
data mining
databases
expressions (DMXs)
data paths, security
Data Source Name (DSN)
data sources, Integration Services
data structures, optimizing
Data Transformation Services (DTS) 2nd
Integration Services
database administrators (DBAs)
maintenance
Database Definition elements
Database Mail connections
DATABASEPROPERTYEX function
databases
attaching 2nd
backups 2nd
BI 2nd 3rd 4th
Copy Database Wizard
creating
data mart
data mining
detaching 2nd
engines
maintenance
auditing servers
backups
consistency checks
growth analysis
hot fixes/service packs
index fragmentation
recovery models 2nd
restoring 2nd
mirroring
monitoring
Notification Services
architecture
monitoring
security
objects
permissions
security
recovering
Reporting Services
architecture
creating reports 2nd
delivery systems
deployment models
design
managing
monitoring
Report Manager 2nd
Reporting Services Configuration tool 2nd
security
SQL Management Studio
tools
types of reports
restoring
roles
fixed
principal
security
sizes
estimating
searching
system views
tools
Books Online 2nd
command-prompt utilities
SQL Server Configuration Manager
SQL Server Management Studio 2nd
SQL Server Profiler 2nd
DB-Library
DBAs (database administrators)
maintenance
DBCC CHECKALLOC command
DBCC CHECKCATALOG command
DBCC CHECKCONSTRAINTS command
DBCC CHECKDB command
DBCC CHECKFILEGROUP command
DBCC CHECKIDENT command
DBCC CHECKTABLE command
DBCC DBREINDEX command
DBCC INDEXDEFRAG command
DBCC SHOWCONTIG command
DDL (Data Definition Language) 2nd
permissions
triggers
decision matrix
default software versions
definition files, applications
Deliver Channels
delivery systems, Reporting Services
demilitarized zones [See DMZs (demilitarized zones).]
DENY command
departments
deployment models, Reporting Services
design [See also configuration.]
BI systems
Analysis Services
databases 2nd 3rd 4th
implementing
Integration Services
Reporting Services
replication 2nd
Reporting Services
creating reports
types of reports
destinations, Integration Services
Detach Database
detaching databases 2nd
devices
backups
network security
differential backups
differential restores
directories, virtual 2nd
disaster recovery strategies, creating
disks, backup locations
distributed computing architecture
Distributed Management Objects (DMOs)
connections
distributors
Notification Services
replication
DLL (dynamic link library) 2nd
security
DML (Data Manipulation Language) 2nd 3rd
permissions
DMOs (Distributed Management Objects)
connections
DMXs (data mining expressions)
DMZs (demilitarized zones) 2nd
documentation
Books Online 2nd
hardware
operating systems
SQL Server 2005 upgrades
DSN (Data Source Name)
DTS (Data Transformation Services) 2nd
Integration Services
duplication of data
dynamic link library (DLL) 2nd
security
dynamic management views


E

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

e-mail, Reporting Services
editions (servers)
Enterprise
Express
Mobile
software requirements
Standard
Workgroup
EFS (Encrypted File System)
elements
ApplicationName
Applications
Database Definition
Event Classes
History
instanceName
SqlServerSystem
SubscriptionClasses
Version
emergency recovery time
Encrypted File System (EFS)
encryption
PGP
Reporting Services
transmissions
endpoints 2nd
connections
engines
database
Reporting Services
Enterprise Edition (SQL Server 2005)
enterprise landscapes
environments, replication
error logs
estimating database sizes
ETL (extract, transform, and load) platform
Event Classes elements
events
handlers
logs 2nd
Notification Services
architecture
monitoring
security
providers
execution
queries
Reporting Services
exit strategies, creating SQL Server 2005 upgrades
Express Edition (SQL Server 2005)
Expression Service
Extensible Markup Language [See XML (Extensible Markup Language).]
extract, transform, and load (ETL) platform


F

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

fallback strategies, replication
farms
features, SQL Server 2005 upgrades
File Transfer Protocol (FTP)
files
ADF
CVS
data
definition
ICF
instances
LDFs
log
auditing
separating
transactions
MDFs
moving
operational
security
Filestream Agent process
fixed database roles
fixed server roles
footprints, physical
data files
operational files
For loop
Foreach loop
formatting
accounts (Windows)
databases
passwords 2nd
projects
reports 2nd
roles
stored procedures
tables
views
fragmentation, indexes
frequency of replication
FTP (File Transfer Protocol)
full backups
full recovery models 2nd
full text queries
full-text indexes
functionality, security
functions [See also commands.]
DATABASEPROPERTYEX
monitoring


G

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

generators, Notification Services
goals of replication
GRANT command
grouping data files
growth
analysis
monitoring


H

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

handlers, events
hardware
configuration
CPUs
I/O
memory
networks
documentation
network security
requirements
security
HBA (host bus adapter)
HCL (Hardware Compatibility List)
help, Books online 2nd
hiding tables
hierarchies
security
databases
servers
XML
high availability
log shipping
near-real-time
real-time
clustering 2nd
mirroring databases
replication
History element
homogenizing data
host bus adapter (HBA)
hot fixes


I

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

I/O subsystems, configuring
ICF (Instance Configuration File)
IDE (integrated development environment)
IIS MC (Internet Information Services Microsoft Management Console)
implementation [See also configuration.]
BI
replication
in-place SQL Server 2005 upgrades 2nd
indexes
fragmentation
full-text
optimizing
trees
infrastructure, Analysis Services
initializing Reporting Services
installation
architecture
client/server
distributed computing
monolithic
service-oriented
three-tier
clustering
editions (SQL Server)
Enterprise
Express
Mobile
software requirements
Standard
Workgroup
hot fixes/service packs
instances
landscapes
departments
enterprise
farms
operating system variables
post-installation configuration
systems
selecting use of
sizing
Instance Configuration File (ICF)
instanceName element
instances
clustering
managing
two-node (on VM) 2nd
named
Notification Services
integrated development environment (IDE)
Integration Services (SSIS)
architecture
command-line utilities
components 2nd
tools
wizards
BI
DTS
monitoring
performance tuning
security
interfaces
APIs
connections
MAPI
tools
Books Online 2nd
command-prompt utilities
SQL Server Configuration Manager
SQL Server Management Studio 2nd
SQL Server Profiler 2nd
VIA
WMI
Internet Information Services Microsoft Management Console (IIS MMC)


J

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

J2EE (Java2 Enterprise Edition)
JDBC (Java Database Connection)
jobs
Analysis Services 2nd
automation
joining transformation


K

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

keys
KPIs (key performance indicators) 2nd


L

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

landscapes
departments
enterprise
farms
layers
applications
architecture
client/server
distributed computing
monolithic
service-oriented
three-tier
security
LDFs (Load Definition Files)
leveraging platforms
libraries
client networks
server networks
Load Definition Files (LDFs)
locations, backups
Lock Manager process
locks
log files
auditing
events
packages
separating
shipping
transactions
log sequence number (LSN) restores
logical unit numbers (LUNs)
logins
fixed server roles
SQL Server 2005
Windows
long-running queries, searching
loops
For
Foreach
loosely coupled distributed computing architecture
LSN (log sequence number) restores
LUNs (logical unit numbers)


M

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

Mail API (MAPI)
maintenance 2nd
Analysis Services
backups
infrastructure
job processing 2nd
monitoring 2nd
security
storage
synchronization
backups
alternate methods of
differential
full
locations
Transaction Log
checklists
consistency checks
growth analysis
high availability
log shipping
near-real-time 2nd 3rd
real-time 2nd
hot fixes/service packs
index fragmentation
planning
recovery models
bulk-logged 2nd
full 2nd
simple 2nd
restoring
alternate methods
complete
differential
LSN
Point-in-Time 2nd
Transaction Log 2nd
servers
management
clustering
dynamic views
Reporting Services
Report Manager
Reporting Services Configuration tool 2nd
SQL Management Studio
SQL Server management Studio
System Monitor
Task Manager
tools
Books Online
command-prompt utilities
SQL Server Configuration Manager
SQL Server Management Studio 2nd
SQL Server Profiler 2nd
Management object
MAPI (Mail API)
mark restores
MARs (Multiple Active Result Sets)
master databases
materialized transformations
MDAC (Microsoft Database Access Components) 2nd
MDFs (Menu Definition Files)
MDXs (multidimensional expressions)
memory, configuring
Menu Definition Files (MDFs)
merging
replication
transformation types
Meta Data process
methods
application security
connections
metrics
Microsoft Database Access Components [See MDAC (Microsoft Database Access Components).]
Microsoft Distributed Transaction Coordinator service (MSDTC)
Microsoft Tape Format (MTF)
Migration Wizard
migration, SQL Server 2005 upgrades
MIRROR TO command
mirroring
backups
databases
Mobile Edition (SQL Server 2005)
models
deployment
recovery
bulk-logged 2nd
configuring
full 2nd
simple 2nd
UDM
modes
WITH NO RECOVERY
WITH RECOVERY
modifiers, STOPAT
modifying
accounts
startup
system databases
monitoring 2nd 3rd
Analysis Services 2nd
baselines
components
Integration Services
Notification Services
real-time
Reporting Services
Service Broker
tools
operating system commands
system views
monolithic architecture
moving data (database engines)
MSDTC (Microsoft Distributed Transaction Coordinator service)
MTF (Microsoft Tape Format)
multidimensional expressions (MDXs)
Multiple Active Result Sets (MARS)
multiple-system automation


N

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

named instances
NAS (network attached storage)
Natural Language Processing (NLP)
near-real-time availability
replication
NET USER command
network attached storage (NAS)
networks
client libraries
configuring
security
server libraries
NLP (Natural Language Processing)
nonclustered indexes
NORECOVERY option
Notification Services
architecture
monitoring
security


O

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

Object Linking and Embedding for Databases (OLE DB)
objects
architecture
communication processes
databases
Management
Report Manager
security 2nd
hierarchies
permissions
ODS (operational data store)
OLAP (online analytical processing) 2nd
OLE EDB (Object Linking and Embedding for Databases)
OLTP (online transaction processing) 2nd
online analytical processing (OLAP) 2nd
Open Database Connectivity, (ODBC)
operating systems
commands
documentation
monitoring
requirements
security 2nd
file systems
hardware
networks
service accounts
variables
versions
operational data store (ODS)
operational files
operators
automation
optimization 2nd [See also configuration.]
applications
data structures
indexes
platforms
options
NORECOVERY
REPAIR_FAST


P

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

packages
Integration Services
logging
PageManager Process
Parser
partition views
passwords
adding
formatting
path security
performance
Analysis Services 2nd
counters
Integration Services
monitoring
components
configuring baselines
system views
tools
Notification Services
optimization
applications
data structures
indexes
platforms
Reporting Services
Service Broker
tuning
permissions
ALTER
DDL
DML
PGP (Pretty Good Privacy) encryption
physical files, moving
physical footprints
data files
operational files
planning
execution plans, queries
for growth
maintenance
SQL Server 2005 upgrades
platforms
auditing
ETL
leveraging
monitoring
optimizing
requirements
security
connections
functionality
hardware
networks
operating systems
SAC 2nd
schemes
services
PocketPC, SQL Server 2005 Mobile Edition
Point-in-Time restores 2nd
policies, accounts
post-installation configuration
post-upgrade steps, SQL Server 2005 upgrades
precedence, Integration Services
predicates, WITH
presentation layers
Pretty Good Privacy (PGP) encryption
principals, security
accounts
passwords
roles
private keys
procedures, stored 2nd
creating
sp_setappRole
processes, communication
processors, CPUs
Profiler 2nd
projects, creating
properties
Query Properties
Status
protocols
FTP
SOAP 2nd
providers, events
public keys
publishers, replication


Q

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

queries
architecture
database engines
execution plans
full text
long-running
refactoring
WQL
Query Execution process
Query Optimizer process
Query Processor
Query Properties


R

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

RAID (redundant array of inexpensive disks)
RAM (random access memory)
RDBMS (Relational Database Management System) 2nd
RDL (Report Definition Language) 2nd
RDOs (Remote Data Objects)
real-time availability
clustering
database mirroring
real-time monitoring
recovery
databases
emergency time
models
bulk-logged
configuring
full 2nd
simple 2nd
redundant array of inexpensive disks (RAID)
refactoring queries
relational calculus
Relational Database Management System (RDBMS) 2nd
Remote Data Objects (RDOs)
remote server connections
REPAIR_FAST option
replication 2nd
design 2nd
implementing
merge
snapshot
transactional
Report Builder
Report Definition Language (RDL)
Report Designer
Report Model Designer
Report Wizard
Reporting Services 2nd
architecture
delivery systems
deployment models
design
tools
types of reports
managing
Report Manager
Reporting Services Configuration tool
SQL Management Studio
monitoring
reports 2nd
security
reporting systems
reports
requirements
clustering
SQL Server 2005 upgrades
RESTORE command
restoring
bulk-logged recovery models
databases 2nd
alternate methods
complete
differential
LSN
mark
Point-in-Time 2nd
Transaction Log 2nd
full recovery models
reports
simple recovery models 2nd
reviewing audits
REVOKE command
roles
applications
fixed database
fixed server
principal-defined database
routers, security


S

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

SAC (Surface Area Configuration) 2nd
security 2nd
SAN (storage area network)
hard drives
SAS (serial attached storage)
schemas
schemes, security
connections
functionality
SAC 2nd
services
scopes
scripts, DML
searching
database sizes
long-running queries
security
Analysis Services
applications
auditing 2nd
backups
certificates
checks (post-upgrade steps)
data paths
encryption
PGP
Reporting Services
transmissions
hierarchies
databases
servers
Integration Services wizards
keys
Notification Services
objects 2nd
passwords
platforms
connections
functionality
hardware
networks
operating systems
SAC 2nd
schemes
services
Reporting Services
Service Broker
users
accounts
passwords
roles
selecting systems
separating log files
serial attached storage (SAS)
Server Management Objects (SMOs) 2nd
servers
architecture 2nd
communication processes
data files
databases
objects
operational files
physical footprints
queries
editions
Enterprise
Express
Mobile
software requirements
Standard
Workgroup
maintenance
auditing servers
backups
consistency checks
growth analysis
hot fixes/service packs
index fragmentation
recovery models 2nd 3rd
restoring 2nd
monitoring
network libraries
remote connections
replication
Reporting Services
architecture
creating reports 2nd
delivery systems
deployment models
design
managing
monitoring
Report Manager 2nd
Reporting Services Configuration tool 2nd
security
SQL Management Studio
tools
types of reports
roles
security hierarchies
systems
selecting use of
sizing
views
tools
Books Online 2nd
command-prompt utilities
SQL Server Configuration Manager
SQL Server Management Studio 2nd
SQL Server Profiler 2nd
upgrading
creating exit strategies
documenting
features
guidelines
in-place 2nd
migration
planning
post-upgrade steps
requirements
side-by-side
strategies
versions 2nd
Service Broker 2nd 3rd 4th
monitoring
security
service-oriented applications (SOAs)
service-oriented architecture (SOA) 2nd
services
accounts
Reporting Services
security
Notification Services
architecture
monitoring
security
packs
security
shrinking databases, consistency checks
side-by-side SQL Server 2005 upgrades
Simple Messaging Service (SMS)
Simple Object Access Protocol (SOAP) 2nd
simple recovery models 2nd
sizing
databases
estimating
searching
systems
Smart Phones, SQL Server 2005 Mobile Edition
SMOs (Server Management Objects) 2nd
SMP (symmetric multiprocessing)
SMS (Simple Messaging Service)
snapshot replication
SOA (service-oriented architecture) 2nd
SOAP (Simple Object Access Protocol) 2nd
SOAs (service-oriented applications)
Sort Manager process
SOS (SQL OS) process
source systems (BI)
sp_configure command
sp_dbdetach stored procedure
sp_setappRole stored procedure
speed, CPUs
SQL (Structured Query Language)
SQL Mail connections
SQL Management Studio
SQL Manager
SQL OS (SOS) process
SQL Server 2005
creating exit strategies
features
in-place 2nd
logins
migration
post-upgrade steps
requirements
side-by-side
strategies
upgrading to
documenting
guidelines
planning
versions 2nd
SQL Server Agent
SQL Server Configuration Manager
SQL Server Integration Services [See Integration Services (SSIS).]
SQL Server Integration Services (SSIS)
SQL Server Management Studio 2nd 3rd
SQL Server Profiler 2nd 3rd
SQL Trace
sqlcmd utility
SqlServerSystem element
SSIS (SQL Server Integration Services) [See Integration Services (SSIS).]
stakeholders
Standard Edition (SQL Server 2005)
starting, Reporting Services
startup, modifying
Status property
STOPAT modifier
stopping Reporting Services
storage
storage area network (SAN)
hard drives
stored procedures 2nd
creating
sp_dbdetach
sp_setappRole
strategies, SQL Server 2005 upgrades
Structured Query Language (SQL)
subscribers, replication 2nd
SubscriptionClasses element
subscriptions, Reporting Services
Surface Area Configuration (SAC) 2nd
security 2nd
symmetric multiprocessing (SMP)
synchronization, Analysis Services
System Monitor
systems
architecture [See architecture.]
configuration checks
consistency checks
databases [See also databases, ; master databases.]
monitoring
operating
optimization
applications
data structures
indexes
platforms
reporting
selecting use of
sizing


T

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

T-SQL (Transact SQL) 2nd
Execution Engine
tables 2nd [See also databases.]
creating
hiding
tabular data stream (TDS) 2nd
tapes, backup locations
Task Manager
tasks, connecting containers
TDS (tabular data stream) 2nd
text
full-text indexes
queries
Text Manager process
third-party restore tools
three-tier architecture
tightly coupled distributed computing architecture
tools
Books Online
command-prompt utilities
Integration Services
monitoring
Reporting Services
Report Manager 2nd
Reporting Services Configuration tool
SQL Management Studio
SAC 2nd 3rd
SQL Server Configuration Manager
SQL Server Management Studio 2nd 3rd
SQL Server Profiler 2nd
System Monitor
Task Manager
Third-party restore
Transact SQL (T-SQL) 2nd
Execution Engine
Transaction Log
backups
restores 2nd
Transaction Manager process
transactional replication
transactions
transforms
transmissions, encryption
trees, indexes
triggers
DDL
troubleshooting
Books Online 2nd
high availability
log shipping
near-real-time 2nd 3rd
real-time 2nd
maintenance
Analysis Services 2nd 3rd 4th
auditing servers
backups
consistency checks
growth analysis
hot fixes/service packs
index fragmentation
recovery models 2nd 3rd
restoring 2nd
tuning performance [See also optimization.]
Analysis Services 2nd
Integration Services
Notification Services
Reporting Services 2nd
Service Broker
two-node clustering on VM, configuring 2nd
types
of replication
of reports


U

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

UDFs (user-defined functions)
UDM (Unified Data Model)
UDM (Universal Data Model)
UDTs (User-Defined Types)
Unicode character sets 2nd
Unified Data Model (UDM)
Universal Data Model (UDM)
Upgrade Advisor 2nd
upgrading
high availability
log shipping
near-real-time 2nd 3rd
real-time 2nd
to SQL Server 2005
creating exit strategies
documenting
features
guidelines
in-place 2nd
migration
planning
post-upgrade steps
requirements
side-by-side
strategies
versions 2nd
user-defined functions (UDFs)
User-Defined Types (UDTs)
users
databases [See also databases.]
information (system views)
security
accounts
auditing
passwords
roles


V

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

variables
building
Integration Services
operating systems
Version element
Version Store process
versions
default software
operating systems
servers
Enterprise
Express
Mobile
software requirements
Standard
SQL Server 2005 upgrades 2nd
Workgroup
VIA (Virtual Interface Adapter)
views
creating
dynamic management
monitoring
virtual directories 2nd
Virtual Interface Adapter (VIA)
VM (virtual machine)
clustering


W

Previous Page
Next Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

Web
endpoints
reports
Web Service Description Language (WSDL)
Windows
accounts
connecting
creating
event logs 2nd
logins
System Monitor
Task Manager
Windows Management Interface (WMI) 2nd
Windows on Windows (WOW)
WITH NO RECOVERY mode
WITH predicate
WITH RECOVERY mode
witnesses
wizards
Copy Database
Integration Services
Migration
Report Wizard
WMI (Windows Management Interface) 2nd
WMI Query Language (WQL)
Workgroup Edition (SQL Server 2005)
WORM (Write-Once Read Many) drive jukeboxes
WOW (Windows on Windows)
WQL (WMI Query Language)
Write-Once Read Many (WORM) drive jukeboxes
WSDL (Web Service Description Language)


X

Previous Page

Index

[A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

XML (Extensible Markup Language) 2nd
endpoints
hierarchies
XML for Analysis (XMLA) 2nd
XQuery Execution Engine